Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Sandeep Sukhani 3e1f2fc273 caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>		3 years ago
..
queryrangebase	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
benchmarkutils_test.go	…
codec.go	Convert SeriesVolume response to prometheus-style (#9703)	3 years ago
codec_test.go	Remove unused Value field (#9774)	3 years ago
downstreamer.go	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
downstreamer_test.go	Update Prometheus dependency (#9205)	3 years ago
extensions.go	Series volume endpoint (#9704)	3 years ago
index_stats_cache.go	Add config to avoid caching stats for recent data (#9537)	3 years ago
index_stats_cache_test.go	Add config to avoid caching stats for recent data (#9537)	3 years ago
limits.go	Add log-volume feature flag (#9762)	3 years ago
limits_test.go	Stats cache can be configured independently (#9535)	3 years ago
log_result_cache.go	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
log_result_cache_test.go	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
metrics.go	…
ordering.go	…
prometheus.go	Cache index stats requests (#8972)	3 years ago
prometheus_test.go	Revert "Add summary stats and metrics for stats cache (#9536)" (#9721)	3 years ago
queryrange.pb.go	Series volume endpoint (#9704)	3 years ago
queryrange.proto	Series volume endpoint (#9704)	3 years ago
querysharding.go	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
querysharding_test.go	Cache index stats requests (#8972)	3 years ago
roundtrip.go	Add log-volume feature flag (#9762)	3 years ago
roundtrip_test.go	Fix instant query summary split stats (#9773)	3 years ago
shard_resolver.go	…
shard_resolver_test.go	Label Volume Endpoint (#9588)	3 years ago
split_by_interval.go	Convert SeriesVolume response to prometheus-style (#9703)	3 years ago
split_by_interval_test.go	Remove unused Value field (#9774)	3 years ago
split_by_range.go	Fix instant query summary split stats (#9773)	3 years ago
split_by_range_test.go	…
stats.go	Fix instant query summary split stats (#9773)	3 years ago
stats_test.go	…