loki/queryrange at d24fe3e68be4be16b3654e6aedafaa71ac9d307c - loki

Like Prometheus, but for logs.

prometheus hacktoberfest grafana cloudnative logging loki

History

Salva Corts d24fe3e68b Max bytes read limit (#8670 ) What this PR does / why we need it: This PR implements two new per-tenant limits that are enforced on log and metric queries (both range and instant) when TSDB is used: - `max_query_bytes_read`: Refuse queries that would read more than the configured bytes here. Overall limit regardless of splitting/sharding. The goal is to refuse queries that would take too long. The default value of 0 disables this limit. - `max_querier_bytes_read`: Refuse queries in which any of their subqueries after splitting and sharding would read more than the configured bytes here. The goal is to avoid a querier from running a query that would load too much data in memory and can potentially get OOMed. The default value of 0 disables this limit. These new limits can be configured per tenant and per query (see https://github.com/grafana/loki/pull/8727). The bytes a query would read are estimated through TSDB's index stats. Even though they are not exact, they are good enough to have a rough estimation of whether a query is too big to run or not. For more details on this refer to this discussion in the PR: https://github.com/grafana/loki/pull/8670#discussion_r1124858508. Both limits are implemented in the frontend. Even though we considered implementing `max_querier_bytes_read` in the querier, this way, the limits for pre and post splitting/sharding queries are enforced close to each other on the same component. Moreover, this way we can reduce the number of index stats requests issued to the index gateways by reusing the stats gathered while sharding the query. With regard to how index stats requests are issued: - We parallelize index stats requests by splitting them into queries that span up to 24h since our indices are sharded by 24h periods. On top of that, this prevents a single index gateway from processing a single huge request like `{app=~".+"} for 30d`. - If sharding is enabled and the query is shardable, for `max_querier_bytes_read`, we re-use the stats requests issued by the sharding ware. Specifically, we look at the [bytesPerShard][1] to enforce this limit. Note that once we merge this PR and enable these limits, the load of index stats requests will increase substantially and we may discover bottlenecks in our index gateways and TSDB. After speaking with @owen-d, we think it should be fine as, if needed, we can scale up our index gateways and support caching index stats requests. Here's a demo of this working: <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918478-d4b6c2fd-de4d-478a-9c8b-e38fe148fa95.png"> <img width="1647" alt="image" src="https://user-images.githubusercontent.com/8354290/226918798-a71b1db8-ea68-4d00-933b-e5eb1524d240.png"> Which issue(s) this PR fixes: This PR addresses https://github.com/grafana/loki-private/issues/674. Special notes for your reviewer: - @jeschkies has reviewed the changes related to query-time limits. - I've done some refactoring in this PR: - Extracted logic to get stats for a set of matches into a new function [getStatsForMatchers][2]. - Extracted the _Handler_ interface implementation for [queryrangebase.roundTripper][3] into a new type [queryrangebase.roundTripperHandler][4]. This is used to create the handler that skips the rest of configured middlewares when sending an index stat quests ([example][5]). Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: `ff847305af/pkg/querier/queryrange/shard_resolver.go (L179-L186)` [2]: `ff847305af/pkg/querier/queryrange/shard_resolver.go (L72)` [3]: `3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L124)` [4]: `3d2fff3a2d/pkg/querier/queryrange/queryrangebase/roundtrip.go (L163)` [5]: `f422e0a52b/pkg/querier/queryrange/roundtrip.go (L521)`		3 years ago
..
queryrangebase	Max bytes read limit (#8670 )	3 years ago
benchmarkutils_test.go	Loki Query Frontend (#1442 )	6 years ago
codec.go	Max bytes read limit (#8670 )	3 years ago
codec_test.go	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761 )	3 years ago
downstreamer.go	Allow passing of context to query related limits functions (#8689 )	3 years ago
downstreamer_test.go	fix test increase timeout (#7022 )	3 years ago
extensions.go	TSDB/query-planning-groundwork (#6367 )	4 years ago
limits.go	Max bytes read limit (#8670 )	3 years ago
limits_test.go	Max bytes read limit (#8670 )	3 years ago
log_result_cache.go	Allow passing of context to query related limits functions (#8689 )	3 years ago
log_result_cache_test.go	fix bugs in logs results caching and its tests (#7925 )	3 years ago
metrics.go	adds result cache key version comparison metrics (#7323 )	3 years ago
ordering.go	Loki Query Frontend (#1442 )	6 years ago
prometheus.go	Ruler: remote rule evaluation (#8744 )	3 years ago
prometheus_test.go	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761 )	3 years ago
queryrange.pb.go	Logproto: Extract push.proto from logproto package to the separate module (#8259 )	3 years ago
queryrange.proto	Logproto: Extract push.proto from logproto package to the separate module (#8259 )	3 years ago
querysharding.go	Max bytes read limit (#8670 )	3 years ago
querysharding_test.go	Max bytes read limit (#8670 )	3 years ago
roundtrip.go	Max bytes read limit (#8670 )	3 years ago
roundtrip_test.go	Max bytes read limit (#8670 )	3 years ago
shard_resolver.go	Max bytes read limit (#8670 )	3 years ago
shard_resolver_test.go	Max bytes read limit (#8670 )	3 years ago
split_by_interval.go	Max bytes read limit (#8670 )	3 years ago
split_by_interval_test.go	Max bytes read limit (#8670 )	3 years ago
split_by_range.go	Loki: only log "executing query" once per query in the frontend (#8337 )	3 years ago
split_by_range_test.go	fix panics when cloning a special query (#8531 )	3 years ago
stats.go	fix(stats_collector): Do not collect stats for failed requests. (#6878 )	3 years ago
stats_test.go	fix(stats_collector): Do not collect stats for failed requests. (#6878 )	3 years ago