Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Sandeep Sukhani 3e1f2fc273 caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>		3 years ago
..
astmapper	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
queryrange	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
series	Remove some unused code (#9611)	3 years ago
stats	Update Prometheus dependency to latest release (v2.40.4) (#7826)	3 years ago
worker	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
http.go	Series volume endpoint (#9704)	3 years ago
http_test.go	Remove unused Value field (#9774)	3 years ago
ingester_querier.go	Series volume endpoint (#9704)	3 years ago
ingester_querier_test.go	Remove unused Value field (#9774)	3 years ago
metrics.go	Add tailing metrics (#6059)	4 years ago
multi_tenant_querier.go	Series volume endpoint (#9704)	3 years ago
multi_tenant_querier_test.go	Remove unused Value field (#9774)	3 years ago
querier.go	Series volume endpoint (#9704)	3 years ago
querier_mock_test.go	Series volume endpoint (#9704)	3 years ago
querier_test.go	Remove unused Value field (#9774)	3 years ago
tail.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
tail_mock_test.go	Improve protobuf serialization (#2031)	6 years ago
tail_test.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
testutils.go	Removing duplicated limits file and refactored references (#8210)	3 years ago
worker_service.go	query-scheduler: fix query distribution in SSD mode (#9471)	3 years ago
worker_service_test.go	Querier worker/inflight metrics (#5124)	4 years ago