233 Commits (c2f96221dcd89d41bbb119965e5fde2d21131b36)

Author	SHA1	Message	Date
Jack Baldry	8391e557b1	Update doc-validator to v3.0.0 (#9666) ... ## 3.0.0 ### Added - Structured output for use with [`reviewdog`](https://github.com/reviewdog/reviewdog). You can achieve the original error output by piping the output to the following `jq` expression: `jq -r '"ERROR: \(.location.path):\(.location.range.start.line):\(.location.range.start.column): \(.message)"'`. - Suggestions for simple link fixes. In GitHub, [`reviewdog`](https://github.com/reviewdog/reviewdog) comments these suggestions for convenient replacement. - Support for anchors referring to repeated headings. In the case that a page has multiple headings that share the same text, the renderer appends a zero indexed, numbered suffix to the identifier. For example, when there are two headings that are both "Heading text", the first anchor identifier is `heading-text` and the second is `heading-text-1`. - Error when running `doc-validator` on no files. **Notes for reviewer:** This also updates the `make-docs` procedure. Please ensure you can run `make docs` from the `docs/` directory before approving and merging this PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	2 years ago
Salva Corts	73ac208981	Improve docs for empty value in cache compression config (#9649) ... **What this PR does / why we need it**: Follow up PR for https://github.com/grafana/loki/pull/9535#discussion_r1218167670 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	c6fbff26e1	Add config to avoid caching stats for recent data (#9537) ... **What this PR does / why we need it**: When we query the stats for recent data, we query both the ingesters and the index gateways for the stats. ebdb2b1800/pkg/storage/async_store.go (L112-L114) ebdb2b1800/pkg/storage/async_store.go (L126-L127) Then we merge all the responses, which means summing up all the stats ebdb2b1800/pkg/storage/async_store.go (L157-L158) ebdb2b1800/pkg/storage/stores/index/stats/stats.go (L23-L26) Because we have a replication factor of 3, this means that we will get the stats from the ingesters repeated up to 3 times, hence inflating the stats. In the stats cache, we store the stats for a given matcher set for the whole day, then we extract the stats from the cache by the factor of time from the request that is stored in the cache: 336283acad/pkg/querier/queryrange/index_stats_cache.go (L33) 336283acad/pkg/querier/queryrange/index_stats_cache.go (L40) Inflated stats for recent data will be cached, so subsequent stats extracted from the cache will be inflated regardless of the time. This PR adds a new per-tenant limit `max_stats_cache_freshness` to not cache requests with an end time that falls within Now minus this duration. Here's a scenario illustrating this. The graphs below show the bytes stats queried in the sharding middleware. We are running a log filter query that won't match any log, every 5 seconds with a length of 3h.  As can be seen, after enabling the stats cache and configuring`do_not_cache_request_within` to not cache stats for requests within 30m, the bytes stats used in the sharding middleware stopped increasing. In both cases the stats cache hit ration was 100%.  **Special notes for your reviewer**: - Blocked by https://github.com/grafana/loki/pull/9535 - Note that this PR doesn't fix the root issue of inflated stats form the ingesters, but rather buys us some time to work on that. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Dylan Guedes	609bc22933	Distributor: Make key configurable when logging failures (#9659) ... **What this PR does / why we need it**: Make appending `insight=true` key-value pair to log failures configurable. **Which issue(s) this PR fixes**: N/A	2 years ago
Salva Corts	1694ad0f9b	Stats cache can be configured independently (#9535) ... **What this PR does / why we need it**: Before this PR, the index stats cache would use the same config as the query results cache. This was a limitation since: 1. We would not be able to point to a different cache for storing the index stats if needed. 2. We would not be able to add specific settings for this cache, without adding it to the results cache. In this PR, we refactor the index stats cache config to be independently configurable. Note that if it's not configured, it will try to use the results cache settings. **Which issue(s) this PR fixes**: This is needed for: - https://github.com/grafana/loki/pull/9537 - https://github.com/grafana/loki/pull/9536 **Special notes for your reviewer**: - This PR also refactors all the tripperwares in rountrip.go to reuse the same stats tripperware instead of each one creating their own. - Configuring a new cache in rountrip.go is a requirement for https://github.com/grafana/loki/pull/9536 so the stats summary can distinguish before the stats cache and the results cache. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Danny Kopping	a248e3e970	Querier: configurable writeback queue bytes size (#9604) ... **What this PR does / why we need it**: The background writeback process is used to write chunks that have been fetched from the store to cache. Chunks have a target size, but not all chunks will necessarily be this size, so it is tricky to find the correct value for `writeback_buffer` which specifies how many chunks will be held in the queue at any given time. For example: setting `writeback_buffer=10000` (the default) with all chunks being the default target size (`chunk_target_size=1.5MiB`), in the worst case could result in 15GiB of memory being used for this buffer. Practically speaking, the queue never grows so deep, but if the cache Loki is writing to becomes overwhelmed this could happen. Setting a value too low would result in many chunks not being written back to cache, which defeats the purpose. This PR introduces a configurable _byte size_ limit to the writeback cache, which defaults to 1GB in size. With this setting `writeback_size_limit` in place, we get the best of both worlds: a deep queue which can hold many small chunks or a shallow queue which can hold a few large ones without using too much memory so as to overwhelm the querier. If either the `writeback_buffer` or `writeback_size_limit` reaches capacity, any new chunks which are attempted to be added to the queue will be dropped. Two new metrics are added by this PR: - `loki_cache_background_queue_bytes`: gauge, measures the size of the queue in bytes - `loki_cache_dropped_background_writes_bytes_total`: counter, measures the volume of bytes dropped	2 years ago
Dylan Guedes	7d67b6306f	Loki: Implement initial phase of `limited_log_push_errors` (#9556) ... **What this PR does / why we need it**: Adds an initial implementation of `limited_log_push_errors`. This initial implementation encompass: - Runtime per-tenant configuration - Simple per-tenant rate-limiting based on error message size Notable features that will be added in future phases: - Instead of a single error with the final string, give to the manager the list of all entries - Once it supports per-entry error, make the rate-limiting separated per-reason - Hash the entry error and avoid repeating errors by caching seen errors in memory	2 years ago
Dylan Guedes	e81dd8aed2	Loki: Bump `dskit` to inherit new backoff configs (#9484) ... **What this PR does / why we need it**: Bump `dskit` to inherit https://github.com/grafana/dskit/pull/296. It will allow us to configure custom backoff timeouts. **Which issue(s) this PR fixes**: N/A	2 years ago
Ashwanth	f08794d39a	docs: add memberlist section to the configuration page (#9452) ... **What this PR does / why we need it**: `memberlist_config` section is not present in 2.8.x release, must have gotten removed by mistake when we moved to the doc generation tool. This PR adds it back. **Which issue(s) this PR fixes**: Fixes #9423 Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Trevor Whitney	50bc1cee71	Rename usagestats to analytics (#9212) ... **What this PR does / why we need it**: Rename usagestats to analytics. The config section in the yaml is already called analytics, this makes things more consistent. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447) ... **What this PR does / why we need it**: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](a85d01c9cd/docs/sources/upgrading/_index.md) for more details. **Which issue(s) this PR fixes:** Fixes https://github.com/grafana/loki/issues/7276 **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Periklis Tsirakidis	3abd260975	Bump dskit from 528d8a7 to 1725bcc (#9184)	2 years ago
Mohamed-Amine Bouqsimi	62572b4f83	Change GeneratorURL to use configurable Grafana explore URL (#8500) ... **What this PR does / why we need it**: This PR changes the `GeneratorURL` associated with alerts generated by Loki. The new `GeneratorURL` uses a Grafana URL path.	2 years ago
李国忠	98936437a9	[opt] s3: add s3 client http timeout for GetObject or PutObject (#8951) ... **What this PR does / why we need it**: add s3 client http timeout for GetObject or PutObject, why？ The flush queue has been increasing. This is a difficult problem. We have been investigating this problem for half a year. When the querier has a large query, the load on the s3 server is too high and the number of connections is full. It will cause the flush queue of the ingester to increase continuously and cannot be reduced. When this problem occurs, after we restart the s3 server, the flush queue will quickly become 0. We checked with our s3 (object storage compatible with the s3 protocol) team and found that it was because we used the s3 v1 sdk. The v1 sdk did not configure timeout by default, and it would wait indefinitely without configuration. The s3 v2 sdk defaults to a timeout of 30s, which only causes write failures and does not cause loki to get stuck. **Which issue(s) this PR fixes**: Fixes #<issue number> At present, as long as there are large logql queries such as 30 days, I must restart the s3 server, otherwise our ingester queue will continue to increase. We hope that we can break the stuck go routine after the 5m timeout occurs. **Special notes for your reviewer**: ingester flush queue length up 🔝  big query, 30day logql or something happen  pprof goroutine list profile snapshot  **There are other alternatives**: ``` ingester: flush_op_timeout: 10m ``` This `flush_op_timeout` configuration is more about the timeout of loki, not the timeout of s3 client. We will also have various s3 operations such as delete and getObject. We should add an independent timeout for s3, just like the cassandra client has a separate The timeout period can be configured. And the parameter `flush_op_timeout` in the early version of loki represents the timeout of writing `a batch of chunks `to s3, not a timeout of s3. Now it is indeed changed to write to s3 one by one, but it is not ruled out that in the future, it will be changed back to write a batch of chunks to s3 sequentially. In this way, `flush_op_timeout: 5m` is far from enough to write a batch of chunks to s3. ``` cassandra: timeout: 5m ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Michel Hollands	abd6131bba	Update prepare shutdown (#9175) ... **What this PR does / why we need it**: This updates the PrepareShutdown method so it supports GET and DELETE methods as well. This makes it similar to Mimir: https://github.com/grafana/mimir/pull/4718. The status is now stored in a local file. A new config setting had to be added for this file as there is no obvious place to store it. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Salva Corts	422560b6b1	Flag to disable index stats cache (#9177) ... **What this PR does / why we need it**: At https://github.com/grafana/loki/pull/8972 we started caching all index stats requests. If the results cache gets overloaded, it can quickly take down the rest of the loki cell due to all the increased work. This PR adds a new flag so we can easily disable caching index stats requests. **Which issue(s) this PR fixes**: This PR is a follow up for https://github.com/grafana/loki/pull/8972 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Andrey Em	0f98b45d8d	Loki: Add UsageStatsURL to usagestats Config (#8779) ... **What this PR does / why we need it**: Reports used to be sent to a fixed URL usageStatsURL. Now this url can be configurable while using the old url as a default. Updated tests to use config for custom URLs instead of changing package variable. **Which issue(s) this PR fixes**: Fixes #6502 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Ed Welch	edc6b0bff7	Loki: Add a limit for the [range] value on range queries (#8343) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: Loki does not currently split queries by time to a value smaller than what's in the [range] of a range query. Example ``` sum(rate({job="foo"}[2d])) ``` Imagine now this query being executed over a longer window of a few days with a step of something like 30m. Every step evaluation would query the last [2d] of data. There are use cases where this is desired, specifically if you force the step to match the value in the range, however what is more common is someone accidentally uses `[$__range]` in here instead of `[$__interval]` within Grafana and then sets the query time selector to a large value like 7 days. This PR adds a limit which will fail queries that set the [range] value higher than the configured limit. It's disabled by default. In the future it may be possible for Loki to perform splits within the [range] and remove the need for this limit, but until then this can be an important safeguard in clusters with a lot of data. **Which issue(s) this PR fixes**: Fixes #8746 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	2 years ago
Dylan Guedes	9159c1dac3	Loki: Improve spans usage (#8927) ... **What this PR does / why we need it**: - At different places, inherit the span/spanlogger from the given context instead of instantiating a new one from scratch, which fix spans being orphaned on a read/write operation. - At different places, turn spans into events. Events are lighter than spans and by having fewer spans in the trace, trace visualization will be cleaner without losing any details. - Adds new spans/events to places that might be a bottleneck for our writes/reads.	2 years ago
Suruthi-G-K	e7b0d4d26e	feat(cos): Support authentication with trusted profiles (#8939) ... **What this PR does / why we need it**: Add trusted profile authentication in COS client **Which issue(s) this PR fixes**: Fixes NA **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com>	2 years ago
Salva Corts	28a7733ede	Rename config for enforcing a minimum number of label matchers (#8940) ... **What this PR does / why we need it**: Followup PR for https://github.com/grafana/loki/pull/8918 renaming config. See https://github.com/grafana/loki/pull/8918/files#r1151820792. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/699 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Salva Corts	45775c82f7	Implement `RequiredNumberLabels` query limit (#8918) ... **What this PR does / why we need it**: As pointed out in https://github.com/grafana/loki/pull/8851, some queries can impose a great workload on a cluster by selecting too many streams. Similarly to the `RequiredLabels` limit introduced at https://github.com/grafana/loki/pull/8851, here we add a new limit `RequiredNumberLabels` to require queries to specify at least N label. For example, if the limit is set to 2, then the query should contain at least 2 label matchers. This limit can be configured per tenant and at query time.  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/699 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826) ... **What this PR does / why we need it**: Add support for IBM cloud object storage as storage client **Which issue(s) this PR fixes**: Fixes NA **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	2 years ago
Danny Kopping	c9d5a91206	Ruler: Implement consistent rule evaluation jitter (#8896) ... **What this PR does / why we need it**: This PR replaces the previous random jitter with a consistent jitter. While both are random, having the random jitter be applied _consistently_ is essential for evaluating rules on a predictable cadence. If a rule is supposed to evaluate every minute, whether it evaluates at (e.g.) `01:00` or `01:03.234` is irrelevant because the evaluation _instant_ is not adjusted, so it will produce the same result whether run at `01:00` or `01:03.234`. However, if 1000 rules are set to evaluate at `01:00`, this will create a resource contention issue.	2 years ago
Salva Corts	d24fe3e68b	Max bytes read limit (#8670) ... **What this PR does / why we need it**: This PR implements two new per-tenant limits that are enforced on log and metric queries (both range and instant) when TSDB is used: - `max_query_bytes_read`: Refuse queries that would read more than the configured bytes here. Overall limit regardless of splitting/sharding. The goal is to refuse queries that would take too long. The default value of 0 disables this limit. - `max_querier_bytes_read`: Refuse queries in which any of their subqueries after splitting and sharding would read more than the configured bytes here. The goal is to avoid a querier from running a query that would load too much data in memory and can potentially get OOMed. The default value of 0 disables this limit. These new limits can be configured per tenant and per query (see https://github.com/grafana/loki/pull/8727). The bytes a query would read are estimated through TSDB's index stats. Even though they are not exact, they are good enough to have a rough estimation of whether a query is too big to run or not. For more details on this refer to this discussion in the PR: https://github.com/grafana/loki/pull/8670#discussion_r1124858508. Both limits are implemented in the frontend. Even though we considered implementing `max_querier_bytes_read` in the querier, this way, the limits for pre and post splitting/sharding queries are enforced close to each other on the same component. Moreover, this way we can reduce the number of index stats requests issued to the index gateways by reusing the stats gathered while sharding the query. With regard to how index stats requests are issued: - We parallelize index stats requests by splitting them into queries that span up to 24h since our indices are sharded by 24h periods. On top of that, this prevents a single index gateway from processing a single huge request like `{app=~".+"} for 30d`. - If sharding is enabled and the query is shardable, for `max_querier_bytes_read`, we re-use the stats requests issued by the sharding ware. Specifically, we look at the [bytesPerShard][1] to enforce this limit. Note that once we merge this PR and enable these limits, the load of index stats requests will increase substantially and we may discover bottlenecks in our index gateways and TSDB. After speaking with @owen-d, we think it should be fine as, if needed, we can scale up our index gateways and support caching index stats requests. Here's a demo of this working: <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918478-d4b6c2fd-de4d-478a-9c8b-e38fe148fa95.png"> <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918798-a71b1db8-ea68-4d00-933b-e5eb1524d240.png"> **Which issue(s) this PR fixes**: This PR addresses https://github.com/grafana/loki-private/issues/674. **Special notes for your reviewer**: - @jeschkies has reviewed the changes related to query-time limits. - I've done some refactoring in this PR: - Extracted logic to get stats for a set of matches into a new function [getStatsForMatchers][2]. - Extracted the _Handler_ interface implementation for [queryrangebase.roundTripper][3] into a new type [queryrangebase.roundTripperHandler][4]. This is used to create the handler that skips the rest of configured middlewares when sending an index stat quests ([example][5]). **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L179-L186) [2]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L72) [3]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L124) [4]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L163) [5]: f422e0a52b/pkg/querier/queryrange/roundtrip.go (L521)	2 years ago
Karsten Jeschkies	94725e7908	Define `RequiredLabels` query limit. (#8851) ... **What this PR does / why we need it**: Some end-users can impose great workload on a cluster by selecting too many streams in their queries. We should be able to limit them. Therefore we introduce a new limit `RequiredLabelMatchers` which list label names that must be included in the stream selectors. The implementation follows the same approach as for max query limit. **Which issue(s) this PR fixes**: Fixes #8745 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
W.T. Chang	183fe85f1c	Loki: Add route_randomly to Redis options (#8852) ... **What this PR does / why we need it**: Current Redis client only reads from master node, which makes master node under heavy pressure. Usually, one or more read replicas are expected on production environments. This PR helps to increase the utilization of read replicas.	2 years ago
Danny Kopping	528ed22dc2	Ruler: remote rule evaluation hardening (#8785) ... **What this PR does / why we need it**: This PR is part 2 of 2 implementing remote rule evaluation. See [part 1](https://github.com/grafana/loki/pull/8744) for more context.	2 years ago
Danny Kopping	2d93952929	(chore): remove incorrect cache config preample (#8859) ... With the addition of the [config generation tool](https://github.com/grafana/loki/pull/7916), a comment prefix was added for all cache types: `Cache config for index entry writing.` https://grafana.com/docs/loki/next/configuration/#cache_config This appears in the description of each cache config option, and is misleading.	2 years ago
Danny Kopping	5c3d204ebf	Ruler: rule evaluation jitter (#8848) ... **What this PR does / why we need it**: This PR introduces a configurable random sleep before each rule evaluation to prevent contention on the query engine should multiple runs execute concurrently. It's quite likely that rules will be evaluated ~concurrently if they have the same interval.	2 years ago
Christian Haudum	be8b4eece3	Scheduler: Add query fairness control across multiple actors within a tenant (#8752) ... **What this PR does / why we need it**: This PR wires up the scheduler with the hierarchical queues. It is the last PR to implement https://github.com/grafana/loki/pull/8585. When these changes are in place, the client performing query requests can control their QoS (query fairness) using the `X-Actor-Path` HTTP header. This header controls in which sub-queue of the tenant's scheduler queue the query request is enqueued. The place within the hierarchy where it is enqueued defines the probability with which the request gets dequeued. A common use-case for this QoS control is giving each Grafana user within a tenant their fair share of query execution time. Any documentation is still missing and will be provided by follow-up PRs. **Special notes for your reviewer**: ```console $ gotest -count=1 -v ./pkg/scheduler/queue/... -test.run=TestQueryFairness === RUN TestQueryFairness === RUN TestQueryFairness/use_hierarchical_queues_=_false dequeue_qos_test.go:109: duration actor a 2.007765568s dequeue_qos_test.go:109: duration actor b 2.209088331s dequeue_qos_test.go:112: total duration 2.209280772s === RUN TestQueryFairness/use_hierarchical_queues_=_true dequeue_qos_test.go:109: duration actor b 605.283144ms dequeue_qos_test.go:109: duration actor a 2.270931324s dequeue_qos_test.go:112: total duration 2.271108551s --- PASS: TestQueryFairness (4.48s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_false (2.21s) --- PASS: TestQueryFairness/use_hierarchical_queues_=_true (2.27s) PASS ok github.com/grafana/loki/pkg/scheduler/queue 4.491s ``` ```console $ gotest -count=5 -v ./pkg/scheduler/queue/... -bench=Benchmark -test.run=^$ -benchtime=10000x -benchmem goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/scheduler/queue cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkGetNextRequest BenchmarkGetNextRequest/without_sub-queues BenchmarkGetNextRequest/without_sub-queues-8 10000 29337 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21348 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21595 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21189 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/without_sub-queues-8 10000 21602 ns/op 1600 B/op 100 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33770 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33596 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 34432 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33760 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_1_level_of_sub-queues-8 10000 33664 ns/op 2400 B/op 200 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 71405 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 59472 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 117163 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 106505 ns/op 3200 B/op 300 allocs/op BenchmarkGetNextRequest/with_2_levels_of_sub-queues-8 10000 64374 ns/op 3200 B/op 300 allocs/op BenchmarkQueueRequest BenchmarkQueueRequest-8 10000 168391 ns/op 320588 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 166203 ns/op 320587 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 149518 ns/op 320584 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 219776 ns/op 320583 B/op 1156 allocs/op BenchmarkQueueRequest-8 10000 185198 ns/op 320597 B/op 1156 allocs/op PASS ok github.com/grafana/loki/pkg/scheduler/queue 64.648s ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	33e44ed39d	Ruler: remote rule evaluation (#8744) ... **What this PR does / why we need it**: Adds the ability to evaluate recording & alerting rules against a given `query-frontend`, allowing these queries to be executed with all the parallelisation & optimisation that regular adhoc queries have. This is important because with `local` evaluation all queries are single-threaded, and rules that evaluate a large range/volume of data may timeout or OOM the `ruler` itself, leading to missed metrics or alerts. When `remote` evaluation mode is enabled, the `ruler` effectively just becomes a gRPC client for the `query-frontend`, which will dramatically improve the reliability of the `ruler` and also drastically reduce its resource requirements. **Which issue(s) this PR fixes**: This PR implements the feature discussed in https://github.com/grafana/loki/pull/8129 (**LID 0002: Remote Rule Evaluation**).	2 years ago
Ed Welch	eadb9d2c00	Loki: Change the behavior of the 0 value for retention_period to disable retention. (#8753) ... **What this PR does / why we need it**: Issue #8359 exposed a very unfortunate inconsistency in a Loki config where the zero value did not disable a feature. By convention in Loki the zero value for a config typically "disables" it. This PR changes the retention_period to honor the zero value of `0` or `0s` to disable retention. **Which issue(s) this PR fixes**: Fixes #8359 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727) ... **What this PR does / why we need it**: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. **Which issue(s) this PR fixes**: Fixes #8762 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Jan Jansen	84c5e20af1	S3: add support for s3 session token (#8474) ... We want to automate the access account rotation. This is that easy at moment with service accounts in MinIO. Our currently life time of tokens is 3 months with MinIO STS works this fine. Signed-off-by: Jan Jansen <jan.jansen@gdata.de>	2 years ago
justcompile	ea55f285b6	S3 Storage Class Configuration (#8532)	2 years ago
brettlounsbury	4c227bd268	DynamoDB: Accept a KMS key for DynamoDB tables (#8469) ... Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
李国忠	1fc00fa02d	[new] chunk backend: Integrate Alibaba Cloud oss (#8410)	2 years ago
janhorstmann	0e81d78914	Swift: Allow to configure swift with the internal endpoint (#8438)	2 years ago
Hamish Forbes	027c2ef957	Jsonnet: Allows gateway tenant ID to be configured (#7624) ... Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Svilen	2d11ae2071	Fix the ruler alertmanager_client documentation (#7474)	2 years ago
Espen Meidell	8afb2ce3f1	Loki: Add the ability to wait on SIGTERM before shutting down (#8371) ... Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dylan Guedes	f2d7bb4c38	Docs: Improve/update examples page (#8338) ... What this PR does / why we need it: Update our examples page to reflect recent changes to Loki configuration. Motivated by feedback on #6122	2 years ago
Danny Kopping	ebfeba5de3	Ruler: implement rule-based and shuffle sharding (#8092)	2 years ago
Travis Patterson	4b9cc39f9f	fix remaining docs issues (#8263)	2 years ago
Travis Patterson	1c954166b8	Fix documentation linter errors (#8229)	2 years ago
Susana Ferreira	7fc25743eb	Docs fix: add missing sections on index.template file (#8216) ... **What this PR does / why we need it**: PR https://github.com/grafana/loki/pull/7916 which introduced the documentation automation tool for Loki's configuration values, incorrectly removed the last sections of the documentation. This PR introduces them accordingly. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Danny Kopping	fd3d362e0c	Removing duplicated limits file and refactored references (#8210)	2 years ago
Kaviraj Kanagaraj	9b5fa5e100	doc(distributor): Clarify `distributor.max-line-size` flag. (#8165) ... **What this PR does / why we need it**: Add a clarrification of two flags `distributor.max-line-size` and `distributor.max-line-size-truncate`. And clarity on [discarding the logs](https://github.com/grafana/loki/blob/main/pkg/distributor/validator.go#L84-L92) if later is disabled while still former is set to non-zero values. Whereas if `distributor.max-line-size-truncate` is set, [log line is only truncated.](https://github.com/grafana/loki/blob/main/pkg/distributor/distributor.go#L517-L534) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> **Which issue(s) this PR fixes**: Fixes #NA **Special notes for your reviewer**: Figured while working on PR #8153 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Travis Patterson	5e53303cb8	run 'make doc' and commit the changes (#8142)	2 years ago