57 Commits (38e2d7be4a6f1a65e7454b1fe7bdc44e9688a8b6)

Author	SHA1	Message	Date
Juraj Michálek	0ed536cd7a	Add query-frontend option to select request headers in query logs (#11499) ... **What this PR does / why we need it**: Adding feature present in mimir, specifically https://github.com/grafana/mimir/pull/5030. Adds a config option to the query-frontend to specify a list of request headers to include in query logs. For example, setting -frontend.log-query-request-headers="X-Grafana-Org-Id" and sending a query with X-Grafana-Org-Id:1 results in query log lines that include header_x_grafana_org_id=1. **Which issue(s) this PR fixes**: Fixes #11422	1 year ago
Matthew Penner	a91f3f11b3	frontend: Use `net.JoinHostPort` to support IPv6 addresses (#10650) ... Signed-off-by: Matthew Penner <me@matthewp.io> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Karsten Jeschkies	8822ae660d	Use correct error write function (#11487) ... **What this PR does / why we need it**: This fixes the last known regression for `protobuf` encoding. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
Karsten Jeschkies	f67fff3eb2	Approximate `quantile_over_time` (#10417) ... **What this PR does / why we need it**: This change shards `quantile_over_time` queries using t-digest or DDSketch approximations. It can be enabled with `querier.shard_aggregations=quantile_over_time`. Outstanding - [x] Replace generic return type of `StepEvaluator` with interface `StepResult`. - [x] Send mapped query with quantile sketch expression from frontend to querier over the wire. - [x] Serialize sketches. See https://github.com/influxdata/tdigest/issues/34 - [x] Add feature flag. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284) ... **What this PR does / why we need it**: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` *All* queriers will handle requests for a tenant if neither limits are applied. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) 26f097162a/pkg/loki/modules.go (L895) but the [docs](26f097162a/pkg/validation/limits.go (L276)) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Sandeep Sukhani	e28f7f3363	update dskit to latest version (#11287) ... **What this PR does / why we need it**: Update dskit to latest version	2 years ago
Karsten Jeschkies	05c4b77ed0	Define QueryResponse and QueryRequest protobufs. (#10956) ... **What this PR does / why we need it**: This will be a follow up to https://github.com/grafana/loki/pull/10688 finally switching over to protos. Hang on tight! Everything is behind the feature flag `-frontend.encoding=protobuf` which will disable the transcoding to HTTP and HTTPgRPC. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118) ... **What this PR does / why we need it**: This just updates the Loki build imaage changed in #11114. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Shantanu Alshi	fb6f6d2e2f	Stop using global logger in modules (#11051) ... **What this PR does / why we need it**: Refactors Loki to inject logger from the constructors instead of using global logger variable. It is generally considered a good practice. **Which issue(s) this PR fixes**: [Fixes #<issue number>](https://github.com/grafana/loki-private/issues/810) **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR -->	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025) ... **What this PR does / why we need it**: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Karsten Jeschkies	6948c4a179	Turn frontend Tripperware into a Middleware. (#10688) ... **What this PR does / why we need it**: Currently, a request to Loki's frontend API goes through these conversions: ``` http.Request ↓ limitedRoundTripper queryrangebase.Request ↓ queryrangebase.Middlware … ↓ queryrangebase.Request ↓ limitedRoundTripper http.Request ↓ grpcRoundTripperAdapter httpgrpc ↓ grpcRoundTripperAdapter http.Response ↓ limitedRoundTripper queryrangebase.Response ↓ limitedRoundtripper http.Response ``` Since `httgrpc` and `queryrangebase.Request` are Protobufs there's no good reason to encode and decode them to HTTP responses/requests. Furthermore, the encoding to HTTP makes it harder for us to encode query plans. Thus the conversions are changed to the following: ``` http.Request ↓ queryrangebase.Request ↓ queryrangebase.Middlware … ↓ queryrangebase.Request ↓ httpgrpc ↓ queryrangebase.Response ↓ http.Response ``` In order to achieve this the `http.RoundTripper` is pushed to the outside. Only the serialization layer from `http.Request` to `queryrangebase.Request` and `http.Response` to `queryrangebase.Response` will be an `http.RoundTripper`. Everything else is either a `queryrangebase.Handler` or `queryrangebase.Middleware`. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Christian Haudum	27411cff08	Introduce worker queue in bloom gateway (#10976) ... Instead of calling the bloom store directly on each and every request to filter chunk refs based on the given filters, we want to queue requests in per-tenant queues and process batches of requests that can be multiplexed to avoid excessive seeking in the bloom block queriers when checking chunk matches. This PR re-uses the request queue implementation used in the query scheduler. To do so, it moves the queue related code from the scheduler into a separate package `pkg/queue` and renames occurrences of "querier" to "consumer" to be more generic. The bloom gateway instantiates the request queue when starting the service. The gRPC method `FilterChunkRefs` then enqueues incoming requests to that queue. **Special notes for your reviewer**: For testing purposes, this PR also contains a dummy implementation of the workers. The worker implementation - which includes multiplexing of multiple tasks - is subject to a separate PR. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Karsten Jeschkies	9fcc42dc48	Support protobuf `QueryRequest` in querier. (#10858) ... **What this PR does / why we need it**: This is one step to support pure protobuf encoding without `httpgrpc`. It is only on the scheduler and is fully backwards compatible. **Which issue(s) this PR fixes**: This is a sub change of https://github.com/grafana/loki/pull/10688 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Ashwanth	8bb615c2cc	config: loki better defaults (#10793) ... **What this PR does / why we need it**: Updates some of the configuration defaults to provide a better experience for users out of the box. I tried to add notes for non-trivial changes explaining the reason. | configuration | new default | old default | notes | | ------------------------------------------------------ | ----------- | ----------- | -------- | `compactor.delete-max-interval` | 24h | 0 | splits the delete requests into intervals no longer than `delete_max_interval` | | `distributor.max-line-size` | 256KB | 0 | - | | `ingester.sync-period` | 1h | 0 | ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. | | `ingester.sync-min-utilization` | 0.1 | 0 | ^ | | `frontend.max-querier-bytes-read` | 150GB | 0 | - | | `frontend.max-cache-freshness` | 10m | 1m | avoid caching results for data that is still in churn | | `frontend.max-stats-cache-freshness` | 10m | 0 | avoid caching results for data that is still in churn | | `memcached.batchsize` | 256 | 1024 | - | | `memcached.parallelism` | 10 | 100 | - | | `querier.tsdb-max-query-parallelism` | 128 | 512 | avoid over parallelising queries for smaller installations | | `querier.split-queries-by-interval` | 1h | 30m | avoid over parallelising queries for smaller installations | | `querier.compress-http-responses` | true | false | compress response if the request accepts gzip encoding | | `query-scheduler.max-outstanding-requests-per-tenant` | 32000 | 100 | current default is too small, opening a big dashboard could hit this limit | | `validation.max-label-names-per-series` | 15 | 30 | reducing this to avoid blowing up the series count and index size. Also enforces users to think about what labels are being added. | **Special notes for your reviewer**: regarding `TestQueryTSDB_WithCachedPostings`: removed the assert on get calls as this is hard to get right (splits, querySize mw making stats calls) validating additions to the cache and misses is more important here I think. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	0be191309c	config: better defaults for querier max concurrency (#10785) ... **What this PR does / why we need it**: To simply loki configuration, this pr removes `querier.worker-parallelism` as it does not offer additional value to already existing `querier.max-concurrent` Also updates the default value of `querier.max-concurrent` to 4. This is a better default for users that are just starting out. Users can consider increasing it as they scale up their loki installation. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164) ... **What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Christian Haudum	cba31024d4	Extend scheduler queue metrics with enqueue/dequeue counters (#8891) ... **What this PR does / why we need it**: Better o11y of the scheduler. This change yields new metrics with potentially high cardinality on the scheduler. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	3344d59fb5	Extract scheduler queue metrics into separate field ... This allows for easier passing of the metrics to the scheduler instantiation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752) ... **What this PR does / why we need it**: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. **Special notes for your reviewer**: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	22b672986a	Revert "Implement hierarchical queues for query scheduler" (#8796) ... Reverts grafana/loki#8691 These changes caused problems with the scheduler in our dev environments. I will conduct further testing and include the reverted changes in https://github.com/grafana/loki/pull/8752	2 years ago
Christian Haudum	4b74a5b815	Implement hierarchical queues for query scheduler (#8691) ... ### What this PR does / why we need it: This PR is the first step towards hierarchical queues within the query scheduler. Hierarchical queues can be used to ensure query fairness across multiple actors within a single tenant, such as individual (human) users using the same tenant. The change is described in the LID ["Query fairness across users within tenants"](https://grafana.com/docs/loki/next/lids/0003-queryfairnessinscheduler/). * Extract metrics for `RequestQueue` into their own `metrics` field, so that they can extended and passed more easily in the future. * Move mapping for tenant queues into its own data structure (ordered map) so its functionality can be reused. It also simplifies the code inside the `tenantQueues` struct. * Add the `LeafQueue` struct that implements hierarchical queues. Sub-queues are dequeued in a round-robin manner using the same implementation of `Mapping` that is also used by the tenant queues. ### Special notes for your reviewer Part of https://github.com/grafana/loki/pull/8585 Contains the changes from PR https://github.com/grafana/loki/pull/8722 **This PR does not yet make use of the new hierarchical queues, it only adds the relevant data structures. Wiring up will be done in a follow up PR.** While the new `Mapping` data structure adds an unsignificant overhead to the time/op I consider the simplification an overall better tradeoff. ```console $ benchstat scheduler-bench-old.txt scheduler-bench-new.txt name old time/op new time/op delta GetNextRequest-8 18.0µs ± 2% 20.5µs ± 3% +13.89% (p=0.008 n=5+5) QueueRequest-8 56.6µs ± 0% 53.5µs ± 3% -5.48% (p=0.008 n=5+5) name old alloc/op new alloc/op delta GetNextRequest-8 1.60kB ± 0% 1.60kB ± 0% ~ (all equal) QueueRequest-8 35.6kB ± 0% 29.8kB ± 0% -16.37% (p=0.008 n=5+5) name old allocs/op new allocs/op delta GetNextRequest-8 100 ± 0% 100 ± 0% ~ (all equal) QueueRequest-8 819 ± 0% 806 ± 0% -1.59% (p=0.008 n=5+5) ``` #### Complexity of operations | Operation | Function | Complexity | | --------- | -------- | ---------- | | Insert | `Put(key string, value *tenantQueue) bool` | `O(1)`| | Select | `GetNext(idx QueueIndex) *tenantQueue` | `O(1)` (best case) `O(n)` (worst case) | | Remove | `Remove(key string) bool` | `O(1)` | --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	a5652c8027	Clean up scheduler code to prepare upcoming changes (#8643) ... In preparation of https://github.com/grafana/loki/pull/8585 this PR contains some cleanups. Changes in this PR do not change the implementation, but only rename functions and data structures to make them more generic and/or suitable for extending. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	17577b4d47	Shut down query frontend gracefully (#7978) ... **What this PR does / why we need it**: This PR fixes a bug in the shutdown process of the query frontend. In order to shut down the frontend gracefully without cancelling any in-flight operations/sub-requests, the frontend service has to wait until all in-flight operations are finished. Only then its sub-services may be stopped. In order to control the deterministic order of the shutdown of the service and its child services, child services need to be started with their own context (not with their parent context). Otherwise, if the context of the frontend is cancelled, also the contexts of the child services are cancelled, leading to a parallel stopping of the parent and its child services. During the time of the shutdown, the `RoundTripGRCP` function still needs to accept any round-tripping requests. This is, because the frontend round tripper is wrapped into a query range tripper ware, which splits incoming (HTTP) query requests into multiple (gRPC) query requests if needed. **Which issue(s) this PR fixes**: This fixes a race condition where queries fail, because the query frontend shut down before the queriers that processed in-flight request could send the query response to the frontend. This manifested in an increased error rate of queries during restarts (rollouts) of the query frontend. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Danny Kopping <danny.kopping@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Susana Ferreira	86d33a4e4a	Add CI step to check generated documentation to check pipeline (#7938) ... **What this PR does / why we need it**: Follow up from PR: https://github.com/grafana/loki/pull/7916 to add a CI step to check if the configuration flag documentation file `doc/sources/configuration/_index.md` was generated successfully. Requirements: * CI `check` pipeline should fail if the documentation file is updated manually * CI `check` pipeline should fail if registration flags are changed in the code, but the documentation file is not generated. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Robert Fratto	85392a9728	Update Prometheus dependency to latest release (v2.40.4) (#7826) ... Closes #7811, which is needed for Grafana Agent to update to v2.40 and add support for native histograms. I did not add support for native histograms to Loki, sorry :)	3 years ago
Gerard Vanloo	bcdf17b0a5	Configurable TLS MinVersion and CipherSuites (#7227)	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Owen Diehl	580a5693b5	stripes frontend requests in progress (#6679) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Mohamed-Amine Bouqsimi	4f4b42ba73	Add TLS config to query frontend (#6444) ... * added tls config to query frontend * changelog * remove commented code * lint	3 years ago
Travis Patterson	9b2786bde3	Fix delete updates (#6194) ... * Fix delete updates * Update pkg/storage/stores/shipper/compactor/deletion/delete_requests_client.go Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> * Add compactor address to the storage config * review comments * Review feedback Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christian Simon	6c1a62d829	Ensure proto definitions are formatted [2/2] (#6000) ... * Update build image to 0.10.4 * Add a fmt target for proto defintions This uses buf to ensure the proto defintions are more consistently formatted. * Update formatted files	3 years ago
Travis Patterson	9cef86b162	Invalidate caches on delete (#5661) ... * Generate cache invalidation numbers in the delete store * Get cache generation numbers from the store on request * changlog * rename tombstones to something more meaningful * User invisible module * query frontend relies on a compactor to get the cache generation number * fix serialization * source -> name * lint errors * lint errors * log non-200 responses * add jsonnet changes * lint * review feedback * review feedback * client rename	3 years ago
Ed Welch	c158b2c5f5	Loki: Revert #4845 which changed the format of errors from the API (#5772) ... * This reverts the changes from #4845, unfortunately changing the content type from plain text to JSON is a breaking change of the API and we should wait to do this at a major release. Signed-off-by: Ed Welch <edward.welch@grafana.com> * fix tests which were updated after initial change. Signed-off-by: Ed Welch <edward.welch@grafana.com>	3 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	3 years ago
李国忠	2858ebde5c	[bugfix] fix nil pointer (#5527)	3 years ago
Michel Hollands	d8ed87d04c	Update dskit version (#5392) ... * Update dskit version Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * update changelog Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add debug printing to see interfaces in CI. This will be reverted Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment to remember to remove Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Use netutil.PrivateNetworkInterfacesWithFallback for the other rings Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Move import to place indicated by linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Remove debug printing Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Determine local interfaces in tests Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Rearrange import to satify linter Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update build image in CircleCI Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Change changelegog message to show customer visible changes Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add doc tag and use private network interfaces in common config Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Add comment about filtered out non private IP addresses Signed-off-by: Michel Hollands <michel.hollands@grafana.com> * Update the documentation of the interface names settings Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
Kaviraj Kanagaraj	f5d570177a	Fork `cortex/pkg/util` into Loki (#5257) ... * Fork `cortex/pkg/util/validation` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fork `pkg/cortex/util/limiter` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * fork util/extract Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Spanlogger add more helpers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * getting rid of `cortex/pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * With lesser linter errors Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy by removing unused method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove missed util_log Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix gofmt and goimports Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch operator Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Do not touch generated go files Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Format Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linters happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix lambda promtail tools Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Upgrade Dockerfile of lambda promtail Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add tests for `pkg/util` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Trevor Whitney	99e046c42f	Default max_outstanding_per_tenant to 2048 (#5204) ... * Default max_outstanding_per_tenant to 2048 * Update changelog	3 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187) ... * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 years ago
Kaviraj	37d0c6c39f	Fix cancel issue between Query Frontend and Query Schdeduler (#5113) ... * Fix cancel issue between Query Frontend -> Query Schdeduler Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * remove assert check for message type Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Handle the case where Frontend is running without Frontend workers Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make it into `sendRequestCancel()` func Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove debug logs Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix note Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove workers count debug log on test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add bug link to test Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fixes the case where we cancel after enqueuing Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/lokifrontend/frontend/v2/frontend.go Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> * Switch to a buffered channel instead Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add error on unknown response status from scheduler Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Dylan Guedes	634baa8aa8	Copy `cortex/pkg/querier/stats` package dependency into Loki (#5047)	3 years ago
Cyril Tovena	6ef934b24c	Fix deadlock in disconnecting querier (#5063) ... * Fix deadlock in disconnecting querier Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Update pkg/querier/worker/worker_test.go Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com> * Uses in-memory GRPC server for tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 years ago
Cyril Tovena	9a7f5a22f6	Fix race conditions in frontend_scheduler_worker. (#5060) ... Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	3 years ago
Dylan Guedes	a4f218305d	Copy `cortex/pkg/frontend` package dependency into Loki (#5044) ... * Move `cortex/pkg/frontend` to `loki/pkg/lokifrontend/frontend`. * Update modules.txt with removed Cortex frontend. * Use `pkg/lokifrontend` instead of cortex `pkg/frontend`.	3 years ago
Dylan Guedes	c0bec07e0d	Loki: Implement common net interface/instance addr (#4950) ... * Implement common net interface/instance addr. - Implement two new common configurations: instance_interface_names, which is a list of net interfaces used to discover addresses and instance_addr, used to advertise a component address. * Add docs to new common configurations. * Fix lint issue: missing line. * Add missing CHANGELOG entry. * Change default common net interface from `127.0.0.1` to `""`. * Add missing `InstanceAddr` docstring.	3 years ago
Dylan Guedes	07b583bb83	Copy `cortex/pkg/scheduler` package dependency into Loki (#4962) ... * Fork pkg/scheduler from cortex to Loki. * Replace cortex/pkg/scheduler with loki/pkg/scheduler. * Remove unused cortex subpackages from modules.txt. * Run `make format` to fix new loki/pkg/scheduler imports. * Update new scheduler .proto generated files.	3 years ago
Kaviraj	38c42a26c6	Move `cortex/pkg/tenant` dependency to Loki. (#4944)	3 years ago
Christian Haudum	e573a4d1a2	Change error responses from plain text to JSON (#4845) ... * Return HTTP errors consistently as JSON response The API returned errors in plain text where the error message was the response body. However, the content type header was set to `application/json`. To make the API responses consistent, errors are now returned as JSON as well. ``` { "message": string, "status": string, "code": int } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return HTTP errors consistently as JSON response Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Return 404 Not Found responses as JSON Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago