apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Christian Haudum	cba31024d4	Extend scheduler queue metrics with enqueue/dequeue counters (#8891 ) What this PR does / why we need it: Better o11y of the scheduler. This change yields new metrics with potentially high cardinality on the scheduler. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	3344d59fb5	Extract scheduler queue metrics into separate field This allows for easier passing of the metrics to the scheduler instantiation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752 ) What this PR does / why we need it: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. Special notes for your reviewer: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	22b672986a	Revert "Implement hierarchical queues for query scheduler" (#8796 ) Reverts grafana/loki#8691 These changes caused problems with the scheduler in our dev environments. I will conduct further testing and include the reverted changes in https://github.com/grafana/loki/pull/8752	2 years ago
Christian Haudum	4b74a5b815	Implement hierarchical queues for query scheduler (#8691 ) ### What this PR does / why we need it: This PR is the first step towards hierarchical queues within the query scheduler. Hierarchical queues can be used to ensure query fairness across multiple actors within a single tenant, such as individual (human) users using the same tenant. The change is described in the LID ["Query fairness across users within tenants"](https://grafana.com/docs/loki/next/lids/0003-queryfairnessinscheduler/). * Extract metrics for `RequestQueue` into their own `metrics` field, so that they can extended and passed more easily in the future. * Move mapping for tenant queues into its own data structure (ordered map) so its functionality can be reused. It also simplifies the code inside the `tenantQueues` struct. * Add the `LeafQueue` struct that implements hierarchical queues. Sub-queues are dequeued in a round-robin manner using the same implementation of `Mapping` that is also used by the tenant queues. ### Special notes for your reviewer Part of https://github.com/grafana/loki/pull/8585 Contains the changes from PR https://github.com/grafana/loki/pull/8722 This PR does not yet make use of the new hierarchical queues, it only adds the relevant data structures. Wiring up will be done in a follow up PR. While the new `Mapping` data structure adds an unsignificant overhead to the time/op I consider the simplification an overall better tradeoff. ```console $ benchstat scheduler-bench-old.txt scheduler-bench-new.txt name old time/op new time/op delta GetNextRequest-8 18.0µs ± 2% 20.5µs ± 3% +13.89% (p=0.008 n=5+5) QueueRequest-8 56.6µs ± 0% 53.5µs ± 3% -5.48% (p=0.008 n=5+5) name old alloc/op new alloc/op delta GetNextRequest-8 1.60kB ± 0% 1.60kB ± 0% ~ (all equal) QueueRequest-8 35.6kB ± 0% 29.8kB ± 0% -16.37% (p=0.008 n=5+5) name old allocs/op new allocs/op delta GetNextRequest-8 100 ± 0% 100 ± 0% ~ (all equal) QueueRequest-8 819 ± 0% 806 ± 0% -1.59% (p=0.008 n=5+5) ``` #### Complexity of operations \| Operation \| Function \| Complexity \| \| --------- \| -------- \| ---------- \| \| Insert \| `Put(key string, value tenantQueue) bool` \| `O(1)`\| \| Select \| `GetNext(idx QueueIndex) tenantQueue` \| `O(1)` (best case) `O(n)` (worst case) \| \| Remove \| `Remove(key string) bool` \| `O(1)` \| --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	a5652c8027	Clean up scheduler code to prepare upcoming changes (#8643 ) In preparation of https://github.com/grafana/loki/pull/8585 this PR contains some cleanups. Changes in this PR do not change the implementation, but only rename functions and data structures to make them more generic and/or suitable for extending. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916 ) What this PR does / why we need it: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](`5550cd65ec/docs/sources/configuration/_index.md`) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. NOTE: this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki-private/issues/83 Special notes for your reviewer: Files: * [docs/sources/configuration/index.template](`5550cd65ec/docs/sources/configuration/index.template`): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](`c32e5d0acb/docs/sources/configuration/_index.md`): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](`5550cd65ec/tools/doc-generator`) directory with documentation generation tool. Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Gerard Vanloo	bcdf17b0a5	Configurable TLS MinVersion and CipherSuites (#7227 )	3 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724 )	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257 ) * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Trevor Whitney	99e046c42f	Default max_outstanding_per_tenant to 2048 (#5204 ) * Default max_outstanding_per_tenant to 2048 * Update changelog	3 years ago
Dylan Guedes	634baa8aa8	Copy `cortex/pkg/querier/stats` package dependency into Loki (#5047 )	4 years ago
Cyril Tovena	6ef934b24c	Fix deadlock in disconnecting querier (#5063 ) * Fix deadlock in disconnecting querier Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Uses in-memory GRPC server for tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	4 years ago
Dylan Guedes	a4f218305d	Copy `cortex/pkg/frontend` package dependency into Loki (#5044 ) * Move `cortex/pkg/frontend` to `loki/pkg/lokifrontend/frontend`. * Update modules.txt with removed Cortex frontend. * Use `pkg/lokifrontend` instead of cortex `pkg/frontend`.	4 years ago
Dylan Guedes	07b583bb83	Copy `cortex/pkg/scheduler` package dependency into Loki (#4962 ) * Fork pkg/scheduler from cortex to Loki. * Replace cortex/pkg/scheduler with loki/pkg/scheduler. * Remove unused cortex subpackages from modules.txt. * Run `make format` to fix new loki/pkg/scheduler imports. * Update new scheduler .proto generated files.	4 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944 )	4 years ago
Ed Welch	2b5f3000b6	Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS (#4424 ) * Fork the frontend and scheduler so we can add support for discovery via the ring. * Add a ring to the query scheduler which is then used by the frontend worker and querier workers to find the scheduler address as an alternative to using DNS * update some of the forked code to use dskit grpcclient and grpcutil since #4312 was merged after the original fork was created. * remove query scheduler dependency from querier Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * migrate logging package and make linter happy Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add SafeReadRing for instances when scheduler is not enabled on same instance * Doc changes from code review A few doc fixes Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * go mod tidy on go-kit dep * update changelog * remove ReadRing in favor of SafeReadRing Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Mauro Stettler	4b59556342	Update cortex to newer version (#4478 ) * update cortex and thanos to newer version * pin k8s.io/apimachinery * fixes for newer libs * add missing method to mock wal data collector * fixing test by using different gokit level pkg * make check-mod	4 years ago
Nick Pillitteri	79a7bdbabc	Chore: Use services and modules from grafana/dskit (#4196 ) * Update Cortex vendor to bcf15611345da65e03604a08a0213aaaaac49e54 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use services and modules packages from grafana/dskit Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Update promtail test to set default weaveworks server network type Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Punctuation Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Marco Pracucci	94041307b1	Upgrade Cortex (#3466 ) * Upgrade Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated vendor/modules.txt Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Owen Diehl	306cc72438	Revendor/cortex (#3446 ) * Upgrade Cortex in Loki Signed-off-by: Marco Pracucci <marco@pracucci.com> * proto compat * gofmt * changelog for deprecated querier.split-queries-by-day Co-authored-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Michel Hollands	4f62f36af7	Update cortex to 1 6 (#3131 ) * Update Cortex to recent master (1.5.0+, 35e698bb56d6). Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Ignore set scheduler address. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Make linter happy. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * Upgrade cortex to 1.6 Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Work around issues in go-openapi libraries Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Clean up go.mod and go.sum Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago

18 Commits (bde65667f7c88af17b7729e3621d7bd5d1d3b45f)