Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Owen Diehl acb40ed40e Eager stream merge (#8968) ... This PR introduces a specialized heap based datastructure to merge incoming log results in the frontend. Recently we've experienced an increase in OOMs on frontends due to logs queries which match lots of data. Sharded requests in loki split based on the amount of data we expect and some queries see thousands of sub requests. For log queries, we'll fetch up the `limit` from each shard, return them to the frontend, and merge. High shard counts * limit log lines, especially combined with large log lines (in byte terms) are accumulated on the frontend. Once they all are received, the frontend merges them. This creates opportunity for OOMs as it can hold up a lot of memory. This PR addresses one of these problems by eagerly accumulating responses as they're received and only retaining a total `limit` number of entries. There's still OOM potential due to race conditions between sub requests returning to the query-frontend and the query-frontend merging other sub requests, but this definitely improves the situation. I've been able to consistently run large limited queries that touch TBs of data (i.e. `{cluster=~".+"} |= "a"`) that previously OOMed frontends. --------- Signed-off-by: Owen Diehl <ow.diehl@gmail.com>		3 years ago
..
queryrangebase	…
benchmarkutils_test.go	…
codec.go	…
codec_test.go	…
downstreamer.go	Eager stream merge (#8968)	3 years ago
downstreamer_test.go	Eager stream merge (#8968)	3 years ago
extensions.go	…
limits.go	…
limits_test.go	…
log_result_cache.go	…
log_result_cache_test.go	…
metrics.go	…
ordering.go	…
prometheus.go	…
prometheus_test.go	…
queryrange.pb.go	…
queryrange.proto	…
querysharding.go	…
querysharding_test.go	…
roundtrip.go	…
roundtrip_test.go	…
shard_resolver.go	…
shard_resolver_test.go	…
split_by_interval.go	…
split_by_interval_test.go	…
split_by_range.go	…
split_by_range_test.go	…
stats.go	…
stats_test.go	…