Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Owen Diehl acb40ed40e Eager stream merge (#8968) ... This PR introduces a specialized heap based datastructure to merge incoming log results in the frontend. Recently we've experienced an increase in OOMs on frontends due to logs queries which match lots of data. Sharded requests in loki split based on the amount of data we expect and some queries see thousands of sub requests. For log queries, we'll fetch up the `limit` from each shard, return them to the frontend, and merge. High shard counts * limit log lines, especially combined with large log lines (in byte terms) are accumulated on the frontend. Once they all are received, the frontend merges them. This creates opportunity for OOMs as it can hold up a lot of memory. This PR addresses one of these problems by eagerly accumulating responses as they're received and only retaining a total `limit` number of entries. There's still OOM potential due to race conditions between sub requests returning to the query-frontend and the query-frontend merging other sub requests, but this definitely improves the situation. I've been able to consistently run large limited queries that touch TBs of data (i.e. `{cluster=~".+"} |= "a"`) that previously OOMed frontends. --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>		3 years ago
..
queryrangebase	Loki: Improve spans usage (#8927)	3 years ago
benchmarkutils_test.go	Loki Query Frontend (#1442)	6 years ago
codec.go	Expose optional label matcher for label values handler (#8824)	3 years ago
codec_test.go	Expose optional label matcher for label values handler (#8824)	3 years ago
downstreamer.go	Eager stream merge (#8968)	3 years ago
downstreamer_test.go	Eager stream merge (#8968)	3 years ago
extensions.go	TSDB/query-planning-groundwork (#6367)	4 years ago
limits.go	Loki: Improve spans usage (#8927)	3 years ago
limits_test.go	Split index request in 24h intervals (#8909)	3 years ago
log_result_cache.go	Loki: Improve spans usage (#8927)	3 years ago
log_result_cache_test.go	fix bugs in logs results caching and its tests (#7925)	3 years ago
metrics.go	adds result cache key version comparison metrics (#7323)	3 years ago
ordering.go	Loki Query Frontend (#1442)	6 years ago
prometheus.go	Ruler: remote rule evaluation (#8744)	3 years ago
prometheus_test.go	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761)	3 years ago
queryrange.pb.go	Expose optional label matcher for label values handler (#8824)	3 years ago
queryrange.proto	Expose optional label matcher for label values handler (#8824)	3 years ago
querysharding.go	Split index request in 24h intervals (#8909)	3 years ago
querysharding_test.go	Salvacorts/max querier size messaging (#8916)	3 years ago
roundtrip.go	remove redundant splitby middleware (#8996)	3 years ago
roundtrip_test.go	Loki: Add a limit for the [range] value on range queries (#8343)	3 years ago
shard_resolver.go	Loki: Improve spans usage (#8927)	3 years ago
shard_resolver_test.go	Max bytes read limit (#8670)	3 years ago
split_by_interval.go	Expose optional label matcher for label values handler (#8824)	3 years ago
split_by_interval_test.go	Max bytes read limit (#8670)	3 years ago
split_by_range.go	Loki: only log "executing query" once per query in the frontend (#8337)	3 years ago
split_by_range_test.go	fix panics when cloning a special query (#8531)	3 years ago
stats.go	Expose optional label matcher for label values handler (#8824)	3 years ago
stats_test.go	fix(stats_collector): Do not collect stats for failed requests. (#6878)	3 years ago