Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Salva Corts d24fe3e68b Max bytes read limit (#8670) ... **What this PR does / why we need it**: This PR implements two new per-tenant limits that are enforced on log and metric queries (both range and instant) when TSDB is used: - `max_query_bytes_read`: Refuse queries that would read more than the configured bytes here. Overall limit regardless of splitting/sharding. The goal is to refuse queries that would take too long. The default value of 0 disables this limit. - `max_querier_bytes_read`: Refuse queries in which any of their subqueries after splitting and sharding would read more than the configured bytes here. The goal is to avoid a querier from running a query that would load too much data in memory and can potentially get OOMed. The default value of 0 disables this limit. These new limits can be configured per tenant and per query (see https://github.com/grafana/loki/pull/8727). The bytes a query would read are estimated through TSDB's index stats. Even though they are not exact, they are good enough to have a rough estimation of whether a query is too big to run or not. For more details on this refer to this discussion in the PR: https://github.com/grafana/loki/pull/8670#discussion_r1124858508. Both limits are implemented in the frontend. Even though we considered implementing `max_querier_bytes_read` in the querier, this way, the limits for pre and post splitting/sharding queries are enforced close to each other on the same component. Moreover, this way we can reduce the number of index stats requests issued to the index gateways by reusing the stats gathered while sharding the query. With regard to how index stats requests are issued: - We parallelize index stats requests by splitting them into queries that span up to 24h since our indices are sharded by 24h periods. On top of that, this prevents a single index gateway from processing a single huge request like `{app=~".+"} for 30d`. - If sharding is enabled and the query is shardable, for `max_querier_bytes_read`, we re-use the stats requests issued by the sharding ware. Specifically, we look at the [bytesPerShard][1] to enforce this limit. Note that once we merge this PR and enable these limits, the load of index stats requests will increase substantially and we may discover bottlenecks in our index gateways and TSDB. After speaking with @owen-d, we think it should be fine as, if needed, we can scale up our index gateways and support caching index stats requests. Here's a demo of this working: <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918478-d4b6c2fd-de4d-478a-9c8b-e38fe148fa95.png"> <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918798-a71b1db8-ea68-4d00-933b-e5eb1524d240.png"> **Which issue(s) this PR fixes**: This PR addresses https://github.com/grafana/loki-private/issues/674. **Special notes for your reviewer**: - @jeschkies has reviewed the changes related to query-time limits. - I've done some refactoring in this PR: - Extracted logic to get stats for a set of matches into a new function [getStatsForMatchers][2]. - Extracted the _Handler_ interface implementation for [queryrangebase.roundTripper][3] into a new type [queryrangebase.roundTripperHandler][4]. This is used to create the handler that skips the rest of configured middlewares when sending an index stat quests ([example][5]). **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L179-L186) [2]: ff847305af/pkg/querier/queryrange/shard_resolver.go (L72) [3]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L124) [4]: 3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L163) [5]: f422e0a52b/pkg/querier/queryrange/roundtrip.go (L521)		3 years ago
..
log	Define `RequiredLabels` query limit. (#8851)	3 years ago
syntax	Label selector optimizations (#8763)	3 years ago
vector	Refactor LogQL syntax into its own package (#5539)	4 years ago
blocker.go	Allow passing of context to query related limits functions (#8689)	3 years ago
blocker_test.go	Querier/Ruler: query blocker (#7785)	3 years ago
downstream.go	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761)	3 years ago
downstream_test.go	Max bytes read limit (#8670)	3 years ago
engine.go	Add initial implementation of per-query limits (#8727)	3 years ago
engine_test.go	Use 0.28.1 build image and update go and alpine versions (#8583)	3 years ago
evaluator.go	Allow math on errors (#8823)	3 years ago
evaluator_test.go	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	3 years ago
limits.go	Define `RequiredLabels` query limit. (#8851)	3 years ago
mapper_metrics.go	query_frontend_shard_factor uses exponential buckets (#6689)	4 years ago
matchers.go	Refactor LogQL syntax into its own package (#5539)	4 years ago
matchers_test.go	Label selector optimizations (#8763)	3 years ago
matrix.go	Do not insert point when missing when sharding (#5444)	4 years ago
matrix_test.go	Do not insert point when missing when sharding (#5444)	4 years ago
metrics.go	Loki: add lines processed to metrics.go (#8764)	3 years ago
metrics_test.go	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761)	3 years ago
optimize.go	Add logfmt selective label extraction (#6675)	3 years ago
optimize_test.go	fix panics when cloning a special query (#8531)	3 years ago
range_vector.go	Fix min and max aggregations when using instant queries. (#7957)	3 years ago
range_vector_test.go	Fix various linter errors	3 years ago
rangemapper.go	[new feature] logql:support sort and sort_desc (#7989)	3 years ago
rangemapper_test.go	Fix various linter errors	3 years ago
shardmapper.go	Max bytes read limit (#8670)	3 years ago
shardmapper_test.go	Max bytes read limit (#8670)	3 years ago
step_evaluator.go	fixes a badly referenced variable name in StepEvaluator code (#3996)	5 years ago
test_utils.go	[TSDB] Use index sampling to determine shard factor (#6396)	4 years ago
vector.go	LogQL: Vector and Range Vector Aggregation.	6 years ago