Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Danny Kopping bcd03150c9 Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535) ... **What this PR does / why we need it**: The config option `query_ingesters_within` defines the window during which logs _could_ be present on ingesters, and as such queriers will send queries to ingesters instead. `split_queries_by_interval` is defined to split queries into subqueries for increased parallelism. Aggressive query splitting within the `query_ingesters_within` window can result in overloading ingesters with unnecessarily large numbers of subqueries, which perversely can impact writes. `query_ingesters_within` is set to 3h by default. In Grafana Cloud Logs we set `split_queries_by_interval` as low as 15m (defaults to 1h), which would result in result in 3*60/15=12 requests. Every querier queries every ingester during this window, so that's 12 requests _per ingester per query_ which has the `query_ingesters_within` window in its time range _(i.e. a query from now to now-7d would include the `query_ingesters_within` window as well, now-3h to now-7d would not)_. However, we _do_ want to split queries so an ingester won't have to handle a query for a full `query_ingesters_within` window - this could involve a large amount of data. To account for this, this PR introduces a new option `split_ingester_queries_by_interval` on the query-frontend; this setting is disabled by default. 		2 years ago
..
limits	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
queryrangebase	Support tail requests with protobuf encoding. (#11426)	2 years ago
benchmarkutils_test.go	Loki Query Frontend (#1442)	6 years ago
codec.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
codec_test.go	results cache: add new stat `query_length_served` to measure cache effectiveness (#11589)	2 years ago
downstreamer.go	Merge quantile sketches as they arrive (#11544)	2 years ago
downstreamer_test.go	Merge quantile sketches as they arrive (#11544)	2 years ago
extensions.go	Optimize series response format by using repeated fileds. (#11498)	2 years ago
extensions_test.go	Define QueryResponse and QueryRequest protobufs. (#10956)	2 years ago
index_stats_cache.go	Extract results cache into new pkg (#11343)	2 years ago
index_stats_cache_test.go	Extract results cache into new pkg (#11343)	2 years ago
instrument.go	Propagate trace ID with HTTP gRPC request. (#11251)	2 years ago
labels_cache.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
labels_cache_test.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
limits.go	Approximate `quantile_over_time` (#10417)	2 years ago
limits_test.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
log_result_cache.go	fix(log results cache): compose empty response based on the request (#11657)	2 years ago
log_result_cache_test.go	fix(log results cache): compose empty response based on the request (#11657)	2 years ago
marshal.go	Benchmark and improve sketch join (#11534)	2 years ago
marshal_test.go	Optimize series response format by using repeated fileds. (#11498)	2 years ago
metrics.go	Use metrics namespace for more metrics (#11025)	2 years ago
ordering.go	Loki Query Frontend (#1442)	6 years ago
prometheus.go	Extract results cache into new pkg (#11343)	2 years ago
prometheus_test.go	results cache: add new stat `query_length_served` to measure cache effectiveness (#11589)	2 years ago
queryrange.pb.go	Approximate `quantile_over_time` (#10417)	2 years ago
queryrange.proto	Approximate `quantile_over_time` (#10417)	2 years ago
querysharding.go	Approximate `quantile_over_time` (#10417)	2 years ago
querysharding_test.go	Optimize series response format by using repeated fileds. (#11498)	2 years ago
roundtrip.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
roundtrip_test.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
serialize.go	Flag categorize labels on streams response (#10419)	2 years ago
serialize_test.go	Restore previous behaviour of no trailing newlines in HTTP error response bodies (#11606)	2 years ago
series_cache.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
series_cache_test.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
shard_resolver.go	Refactor the proto Request interface to return time.Time (#11018)	2 years ago
shard_resolver_test.go	adds tsdb-max-bytes-per-shard limit (#9811)	3 years ago
split_by_interval.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
split_by_interval_test.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
split_by_range.go	Send query plan to querier. (#11246)	2 years ago
split_by_range_test.go	feat(caching): Support caching `/series` and `/labels` query results (#11539)	2 years ago
splitters.go	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535)	2 years ago
stats.go	Send query plan to querier. (#11246)	2 years ago
stats_test.go	Send query plan to querier. (#11246)	2 years ago
views.go	Optimize series response format by using repeated fileds. (#11498)	2 years ago
views_test.go	Optimize series response format by using repeated fileds. (#11498)	2 years ago
volume.go	always align volume range timestamp to end of step (#11136)	2 years ago
volume_cache.go	Extract results cache into new pkg (#11343)	2 years ago
volume_cache_test.go	Extract results cache into new pkg (#11343)	2 years ago
volume_test.go	always align volume range timestamp to end of step (#11136)	2 years ago