Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Sandeep Sukhani 3e1f2fc273 caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>		3 years ago
..
cache	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
client	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
fetcher	Loki: Improve spans usage (#8927)	3 years ago
tests	Add cache usage statistics (#6317)	4 years ago
bigchunk.go	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
chunk.go	Refactoring of the Loki storage (#5833)	4 years ago
chunk_test.go	Refactoring of the Loki storage (#5833)	4 years ago
factory.go	Refactoring of the Loki storage (#5833)	4 years ago
interface.go	[TSDB] store uncompressed chunk size in index (#6665)	4 years ago
json_helpers.go	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago