2288 Commits (46813867758729df3e18763ad7366f141291f3c9)

Author	SHA1	Message	Date
Salva Corts	7e248e1e90	Support non-indexed labels in WAL (#10011) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we support encoding and decoding non-indexed labels for each entry into the chunks. This PR adds support for writing/reading the non-indexed labels to/from the WAL. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	3f161f5c1a	Improve observability for non-indexed labels usage (#9993) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we support encoding and decoding metadata for each entry into the chunks. In this PR we: - Update the bytes processed stats to account for the bytes from those non-indexed labels - Add new stats for bytes processed for those non-indexed labels - Add new ingestion metrics to track ingested non-indexed bytes **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Karsten Jeschkies	efac690328	Use series hash to identify uniques in merge. (#9985) ... **What this PR does / why we need it**: A run of the protobuf encoding for the querier showed a spike in memory usage in the `codec.MergeResponse` method. The routine was allocating a lot of memory for the `series.String()` method which is then hashed. Instead, we can use a `uint64` hash. ``` › go test -v -run=^$ -bench "Benchmark_MergeResponses$" -count=10 ./pkg/querier/queryrange ... │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _MergeResponses-16 4.563 ± 1% 1.197 ± 1% -73.77% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _MergeResponses-16 6578.715Mi ± 0% 2.362Mi ± 0% -99.96% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _MergeResponses-16 40500.1k ± 0% 100.1k ± 0% -99.75% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Trevor Whitney	ae4cd27d83	add sharded to querystats bytes processed (#9934) ... Add a sharded label to the `loki_logql_querystats_bytes_processed_per_second` metric that we use for determining our `LokiQueryRangeReadPerformanceBudgetBurn` metric so we can exclude slow queries with <= 1 shard from burning our SLO budget.	2 years ago
Ed Welch	fa6f9c638a	Loki: add 'post_filter_lines' stat for tracking how many lines are match a queries filter expression(s) (#9983) ... **What this PR does / why we need it**: To better understand query behavior and labeling strategies of users, logging the number of log lines "post filtering" can be very useful, we already log the total_lines processed in a query so this will allow us to see how many lines in the label selector matched what the query was looking for. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	78aad0b805	Fix buffered iterator (#9976) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we added support for writing non-indexed labels from entries into chunks. This PR introduced two bugs: - When the buffered iterator is closed, the buffer to read metadata labels is put back into the pool but not set to nil. Subsequent uses of the iterator may write on top of the buffer that was put back on the pool. This may lead to inconsistent/incorrect results. - Inside the buffered iterator, we were not correctly handling EOFs while reading the number of labels and each label length. This was due to the `lastAttemp` variable not being reset. This PR adds a new test for these two bugs and also fixes them. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/9700 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ed Welch	6d2b3e8b76	Loki: Fix a merge error when getting rules from sharded rulers using the `/prometheus/api/v1/rules` endpoint (#9973) ... **What this PR does / why we need it**: The `/prometheus/api/v1/rules` api endpoint will return all the rules currently loaded and being evaluated by the ruler. Previously when we merged the rule groups from all rulers in a sharded ruler setup, we only used the rule group name as the key in the map used for merging the results. This introduced a bug when multiple rule groups have the same name but in different namespaces, leading to some rules being reported in the incorrect namespace. This is just a visualization bug, the rules are in the correct space and operating as defined, however Grafana uses this API in conjunction with `/loki/api/v1/rules` to track the state of a rule being added or removed. The `/loki/api/v1/rules` endpoint returns rules as they exist in storage. When someone has multiple rule groups with the same name in different namespaces this can lead to the Grafana UI showing rules in a perpetual state of creating and deleting as it's possible for rules to be returned from `/prometheus/api/v1/rules` in the wrong group. Adding the namespace to the key used in the map for merging rule results prevents this issue. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	ce91076dd4	Add non-indexed metadata to chunks (#9700) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694 we support adding metadata labels for each entry in the push payload. In this PR we take that metadata and add it to the entries in the chunk. Supporting serialization and deserialization of those metadata labels. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	88c17ed7ab	Include entry metadata bytes in metrics tracking ingestion stats (#9808) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694 we added support for adding metadata labels to each entry in a push payload. In this PR we update the following metrics to take into account the bytes needed to store those labels: - `loki_distributor_bytes_received_total` - `distributor_bytes_received` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Salva Corts	15af77ba83	Remove Prometheus dependency for push pkg (#9937) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694, we modified the `push` pkg to import Prometheus as a dependency to use the `labels.Labels` type for the entries' non-indexed labels. Having Prometheus as a dependency is problematic since any project importing the `push` pkg will need to import Prometheus as a result. In fact, this is one of the reasons why the `push` pkg was extracted from Loki in the first place (https://github.com/grafana/loki/pull/8259). This PR removes the dependency of Prometheus from the `push` pkg by copying some bits of the implementation for Prometheus' `labels.Labels`. We copy: - The Labels struct definition - The JSON Marshaling and Unmarshaling methods for the labels. We need this so labels are encoded as maps instead of an array of objects. --- **Notes for reviewers:** - To implement the JSON Marshaling and Unmarshaling methods the `push` pkg now depends on `golang.org/x/exp`. I think it should be fine for projects importing the `push` pkg to also depend on `golang.org/x/exp`, right?	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953) ... Reverts grafana/loki#9920	2 years ago
Travis Patterson	23e59fa4a9	reset pipeline caches when tailing rather than making a new pipeline (#9949) ... Recent query optimizations made pipeline creation more expensive with the assumption that it occurred relatively infrequently. To avoid growing pipline caches without bound, integers recreate pipelines on every push. The combination of these things means that tailing high-volume streams can cause ingester failure. This fix adds the ability to clear pipeline caches so we don't need to create a new one each time.	2 years ago
Travis Patterson	6dc722bc9f	correctly handle unpack when reordering stages (#9936) ... The optimization to reorder stages doesn't take into account the `unpack` stage. When line filters come after unpack stages, they shouldn't be moved.	2 years ago
Trevor Whitney	9e22a89bcb	Call FetchChunks in parallel from series index store (#9935) ... This PR changes the invocation of `FetchChunks` in the series index store to happen in parallel. It seems like this was the original intention due to the `// bound concurrency` comment when allocating the `groups` slice. This came out of an investigation of some slow queries, and the traces showed many serial calls to `FetchChunks`.	2 years ago
Dylan Guedes	9ed7fa9df4	WriteFailures: Add metrics to increase monitoring (#9904) ... **What this PR does / why we need it**: Add metrics for our write failures manager. This way we can easily monitor how much logging we're doing and how much we're discarding. **Which issue(s) this PR fixes**: N/A	2 years ago
Karsten Jeschkies	a09cb07e98	Define protobufs for topk and cms. (#9933) ... **What this PR does / why we need it**: This change introduces Protobuf models for the sketch data structures which will be used to shard topk queries. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Christian Haudum	db38b7bccc	Remove unused fallback in index gateway client store (#9916) ... The fallback was once used by boltdb-shipper, but since boltdb-shipper changed to not use the index gateway, the fallback was also never used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Callum Styan	4d3b15ae89	Implement probabilistic topk via count min sketch and min heap (#9882) ... This PR currently implements the following structures: - **Count Min Sketch**: http://dimacs.rutgers.edu/~graham/pubs/papers/cmencyc.pdf - **sketch-bf**: https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/ell2.12482 improvements to the heap usage for Count Min Sketch - HeavyKeeper: https://www.usenix.org/system/files/conference/atc18/atc18-gong.pdf I also read and evaluated the following: - topkapi: https://www.cs.rice.edu/~as143/Papers/topkapi.pdf - space saving algorithm: https://www.cse.ust.hk/~raywong/comp5331/References/EfficientComputationOfFrequentAndTop-kElementsInDataStreams.pdf **Description**: As part of our work to ensure every query type is parallelizable we started exploring probabilistic data structures as a way to reduce memory consumption and allow merging of partial results after sharding of queries. In this PR I have implemented a few sketch structures as part of investigating structures that are applicable to topk queries. Unfortunately while many papers made claims of improved performance over the original count min sketch paper, topkapi and heavy keeper either didn't behave as expected or were not mergeable while keeping the same degree of accuracy. Count min sketch when using conservative updates (which vastly reduces error over counting, especially for low flows), also can't keep the same degree of accuracy, but it is still more accurate after merging than heavy keeper. I've also applied some of the principles outlined in the sketchbf paper related to reducing the # of heap operations required when observing events into the sketch, which sped up processing in my benchmark of 100k unique streams each with 1-2k randomized # of occurrences by nearly 3x. --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Karsten Jeschkies	c2c5249676	Remove goroutines from limiter middleware. (#9923) ... **What this PR does / why we need it**: The `limitedRoundTripper` would spawn thousands if not millions of goroutines. However, they were mostly idle waiting for downstream calls. This would congest the Go scheduler. A semaphore is more suitable to control concurrency as the [example](https://pkg.go.dev/golang.org/x/sync/semaphore#example-package-WorkerPool) in the documentation illustrates.	2 years ago
Salva Corts	aae13c376d	Add metadata to push payload (#9694) ... **What this PR does / why we need it**: We are adding support for attaching labels to each log line. This is one of the series of the PRs broken up to make it easier to review changes. This PR updates the push payload to send labels with each log entry optionally. The log labels are supposed to be in the same format as the stream labels. Just to put it out, here is how it would look for proto and json push payload with same data: **proto(`endpoint`: `(/loki/api/v1/push|/api/prom/push)`, `Content-Type`: `application/x-protobuf`)**(payload built using [push.Stream](4cd1246b88/pkg/push/types.go (L12))): ``` push.Stream{ Entries: []logproto.Entry{ { Timestamp: time.Unix(0, 1688515200000000000), Line: "log line", Labels: `{foo="bar"}`, }, }, Labels: `{app="test"}`, } ``` **v1(`endpoint`: `/loki/api/v1/push`, `Content-Type`: `application/json`)**: ```json { "streams": [{ "stream": { "app": "test" }, "values": [ ["1688515200000000000", "log line", { "foo": "bar" }] ] }] } ``` **legacy-json(`/api/prom/push`, `Content-Type`: `application/json`)**: ```json { "streams": [{ "labels": "{app=\"test\"}", "entries": [{ "ts": "2023-07-05T00:00:00.000000000Z", "line": "log line", "labels": "{foo=\"bar\"}" }] }] } ``` **Which issue(s) this PR fixes**: **Special notes for your reviewer**: We may need to add more thoughtful tests. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Christian Haudum	90fc5109dd	Remove unused package pkg/prom1 (#9927) ... Removes old, unused code. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920) ... **What this PR does / why we need it**: Updated Loki with the latest version of dskit **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Trevor Whitney	9e19ff006c	Add targetLabels to SeriesVolume requests (#9878) ... Adds optional `targetLabels` parameter to `series_volume` and `series_volume_range` requests that controls how volumes are aggregated. When provided, volumes are aggregated into the intersections of the provided `targetLabels` only.	2 years ago
Trevor Whitney	5a570cb921	Query ingesters for SeriesVolume in single binary (#9888) ... This change is necessary to get recent SeriesVolume data from the ingesters when running without an `AsyncStore` in single binary. It changes the queriers to query both the ingester and the store, rather than relying on the `AsyncStore` to query the ingesters.	2 years ago
Travis Patterson	cb2b3ca9ea	Add limit to the series volume endpoint (#9833) ... To protect index gateways, this PR adds a per-tenant limit to the `series_volume` endpoint so the accumulation of series can't grow without bound.	2 years ago
Travis Patterson	e2e695e8cd	Get basic statistics from series volume requests (#9832) ... The `series_volume` endpoint returns stats about the call. This PR wires things up such that basic stats are returned: - Execution time - Number of responses	2 years ago
dependabot[bot]	9ee918d359	Bump google.golang.org/grpc from 1.52.0 to 1.53.0 in /pkg/push (#9871) ... Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.52.0 to 1.53.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-go/releases">google.golang.org/grpc's releases</a>.</em></p> <blockquote> <h2>Release 1.53.0</h2> <h1>API Changes</h1> <ul> <li>balancer: support injection of per-call metadata from LB policies (<a href="https://redirect.github.com/grpc/grpc-go/issues/5853">#5853</a>)</li> <li>resolver: remove deprecated field <code>resolver.Target.Endpoint</code> and replace with <code>resolver.Target.Endpoint()</code> (<a href="https://redirect.github.com/grpc/grpc-go/issues/5852">#5852</a>) <ul> <li>Special Thanks: <a href="https://github.com/kylejb"><code>@​kylejb</code></a></li> </ul> </li> </ul> <h1>New Features</h1> <ul> <li>xds/ringhash: introduce <code>GRPC_RING_HASH_CAP</code> environment variable to override the maximum ring size. (<a href="https://redirect.github.com/grpc/grpc-go/issues/5884">#5884</a>)</li> <li>rls: propagate headers received in RLS response to backends (<a href="https://redirect.github.com/grpc/grpc-go/issues/5883">#5883</a>)</li> </ul> <h1>Bug Fixes</h1> <ul> <li>transport: drain client transport when streamID approaches MaxStreamID (<a href="https://redirect.github.com/grpc/grpc-go/issues/5889">#5889</a>)</li> <li>server: after GracefulStop, ensure connections are closed when final RPC completes (<a href="https://redirect.github.com/grpc/grpc-go/issues/5968">#5968</a>)</li> <li>server: fix a few issues where grpc server uses RST_STREAM for non-HTTP/2 errors (<a href="https://redirect.github.com/grpc/grpc-go/issues/5893">#5893</a>)</li> <li>xdsclient: fix race which can happen when multiple load reporting calls are made at the same time. (<a href="https://redirect.github.com/grpc/grpc-go/issues/5927">#5927</a>)</li> <li>rls: fix a data race involving the LRU cache (<a href="https://redirect.github.com/grpc/grpc-go/issues/5925">#5925</a>)</li> <li>xds: fix panic involving double close of channel in xDS transport (<a href="https://redirect.github.com/grpc/grpc-go/issues/5959">#5959</a>)</li> <li>gcp/observability: update method name validation (<a href="https://redirect.github.com/grpc/grpc-go/issues/5951">#5951</a>)</li> </ul> <h1>Documentation</h1> <ul> <li>credentials/oauth: mark <code>NewOauthAccess</code> as deprecated (<a href="https://redirect.github.com/grpc/grpc-go/issues/5882">#5882</a>) <ul> <li>Special Thanks: <a href="https://github.com/buzzsurfr"><code>@​buzzsurfr</code></a></li> </ul> </li> </ul> <h2>Release 1.52.3</h2> <h1>Bug Fixes</h1> <ul> <li>Fix user-agent version</li> </ul> <h2>Release 1.52.2</h2> <h1>Bug Fixes</h1> <ul> <li>xds: fix panic involving double close of channel in xDS transport (<a href="https://redirect.github.com/grpc/grpc-go/issues/5959">#5959</a>)</li> </ul> <h2>Release 1.52.1</h2> <h1>Bug Fixes</h1> <ul> <li>grpclb: rename grpclbstate package back to state (<a href="https://redirect.github.com/grpc/grpc-go/issues/5963">#5963</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="dba26e15a0"><code>dba26e1</code></a> Change version to 1.53.0 (<a href="https://redirect.github.com/grpc/grpc-go/issues/5983">#5983</a>)</li> <li><a href="2a1e9348ff"><code>2a1e934</code></a> server: after GracefulStop, ensure connections are closed when final RPC comp...</li> <li><a href="e2d69aa076"><code>e2d69aa</code></a> tests: fix spelling of variable (<a href="https://redirect.github.com/grpc/grpc-go/issues/5966">#5966</a>)</li> <li><a href="a6376c9893"><code>a6376c9</code></a> xds/resolver: cleanup tests to use real xDS client 3/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5953">#5953</a>)</li> <li><a href="bf8fc46fa6"><code>bf8fc46</code></a> xds/resolver: cleanup tests to use real xDS client 5/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5955">#5955</a>)</li> <li><a href="3930549b38"><code>3930549</code></a> resolver: replace resolver.Target.Endpoint field with Endpoint() method (<a href="https://redirect.github.com/grpc/grpc-go/issues/5852">#5852</a>)</li> <li><a href="894816c487"><code>894816c</code></a> grpclb: rename <code>grpclbstate</code> package back to <code>state</code> (<a href="https://redirect.github.com/grpc/grpc-go/issues/5962">#5962</a>)</li> <li><a href="e5a0237a46"><code>e5a0237</code></a> encoding: fix duplicate compressor names (<a href="https://redirect.github.com/grpc/grpc-go/issues/5958">#5958</a>)</li> <li><a href="4adb2a7a00"><code>4adb2a7</code></a> xds/resolver: cleanup tests to use real xDS client 2/n (<a href="https://redirect.github.com/grpc/grpc-go/issues/5952">#5952</a>)</li> <li><a href="52a8392f37"><code>52a8392</code></a> gcp/observability: update method name validation (<a href="https://redirect.github.com/grpc/grpc-go/issues/5951">#5951</a>)</li> <li>Additional commits viewable in <a href="https://github.com/grpc/grpc-go/compare/v1.52.0...v1.53.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/grafana/loki/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2 years ago
Karsten Jeschkies	f5992274f4	Switch on pointer `*logproto.SeriesResponse` in JSON serialization. (#9868) ... **What this PR does / why we need it**: Commit b35bbd80d6 introduced a regression in the deserialization code. It would switch on `logproto.SeriesResponse` instead of the pointer type `*logproto.SeriesResponse`. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Dylan Guedes	bc0069756e	Queriers: Stop emitting spans for every objectstorage call (#9857) ... **What this PR does / why we need it**: Stop emitting spans for every call to our object storages (like S3 or Azure). They are causing every chunk fetching to emit a span. Request metrics will still be reported just fine. **Which issue(s) this PR fixes**: N/A	2 years ago
Karsten Jeschkies	9393f3f11a	Test querier handlers reponse format. (#9854) ... **What this PR does / why we need it**: This is a follow up to #9813 that introduced a handler test to verify the response format for the legacy and v1 APIs. The test requires mocking of the engine and query which meant the introduction of a query engine interface. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Sandeep Sukhani	7cbfdb5d6d	cache: fix a mistake in returning cached error in the index list cache (#9853) ... **What this PR does / why we need it**: I did a mistake in returning a cached error in the code pushed in PR #9798. Fixing it in the PR with some tests. **Checklist** - [x] Tests updated	2 years ago
Sandeep Sukhani	e089b4b60d	address feedback on pr 9798 (#9842) ... **What this PR does / why we need it**: PR #9798 got merged by mistake without me addressing the feedback. There is some critical feedback which needs to be addressed, particularly https://github.com/grafana/loki/pull/9798#discussion_r1243526743.	2 years ago
Hamish Forbes	d0baa40df3	Only modify identical timestamps when increment-duplicate-timestamps enabled (#9437) ... When `-validation.increment-duplicate-timestamps` is enabled out of order log entries within the same push request can have their timestamps incorrectly mutated. The code is currently just looking for 2 consecutive entries where the timestamp is not increasing. It should only affect entries where the timestamp is a *duplicate* Test push request payload: ```json { "streams": [ { "stream": { "increment_duplicate_timestamp": "fixed|true|false" }, "values": [ [ "1683665111628000000", "1a: 1683665111628000000" ], [ "1683665111628000000", "1b: 1683665111628000000" ], [ "1683665111628000003", "3: 1683665111628000003" ], [ "1683665111628000002", "2: 1683665111628000002" ] ] } ] } ```` As only the first 2 of these timestamps are the same I would expect all but `1b` to have the timestamps recorded as-is and the logs returned by a query in the order: 1a,1b,2,3. Actual Results: ``` > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="fixed"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="true"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628000001Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000004Z" } > logcli query --forward --limit 100 -z UTC -q --no-labels --since 60m -o jsonl '{increment_duplicate_timestamp="false"}' | jq { "line": "1a: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "1b: 1683665111628000000", "timestamp": "2023-05-09T20:45:11.628Z" } { "line": "2: 1683665111628000002", "timestamp": "2023-05-09T20:45:11.628000002Z" } { "line": "3: 1683665111628000003", "timestamp": "2023-05-09T20:45:11.628000003Z" } ``` `true` and `false` were written to Loki 2.8.2 with the `-validation.increment-duplicate-timestamps` enabled or disabled `fixed` with this patch applied and the flag enabled **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Sandeep Sukhani	2ed3c15c22	index: allow refreshing index file names cache for specific tables (#9798) ... **What this PR does / why we need it**: We have a cache for index files available in the object storage, which helps reduce the number of list calls we make to the object storage. Currently, we only support refreshing the whole index cache, which includes all the files for all tables and users. However, in a large cell with many tables and tenants, it is less likely that all the tables are being referenced. Also, most iterative operations would only need a fresh listing of just a few tables at a time so it is wasteful to always refresh the whole index cache. This PR adds support for refreshing the index cache for only specific tables. I am doing it as part of some compactor improvements to run compaction and retention in separate goroutines to avoid blocking compaction when some expensive delete requests are being processed. To make those changes, I needed a way to cost-effectively refresh the index files in the table being picked up for processing. **Checklist** - [x] Tests updated	2 years ago
Karsten Jeschkies	b35bbd80d6	Support content negotiation between query frontend and querier. (#9813) ... **What this PR does / why we need it**: Currently, the querier sends results to the query frontend in JSON which is then decoded to Protobuf. It is more efficient to send the results as Protobuf. This will also allow to extend the results with custom data structures. The change is backwards compatible through content negotiation. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
cui fliter	4b53909ccf	all: remove repetitive the (#9823) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: cui fliter <imcusg@gmail.com>	2 years ago
Owen Diehl	b251a10234	uses lowercase standard for logging and adds total bytes to shard res… (#9828) ... Minor logging cleanup + helpful to have total bytes on this log line to correlate without needing to derive total bytes from factor * bytes_per_shard	2 years ago
Karsten Jeschkies	125e168a91	Replace flaky server for log level handler test. (#9822) ... **What this PR does / why we need it**: I've noticed that the `TestLevelHandler` were flaky because the code wasn't waiting to the server to be running. Since the test is for a handler it's much simpler to call it directly with a response recorder. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Dylan Guedes	1d20fdbf57	LogPushFailures: Log the tenant id (#9819) ... **What this PR does / why we need it**: Modify the `writefailures.Manager` to - Log the tenant id. Otherwise, it is hard to identify which tenant is responsible for a problematic push - Log stream parsing failures **Which issue(s) this PR fixes**: N/A	2 years ago
Trevor Whitney	d4cfbaac4f	Implement series volume range queries (#9812) ... This PR adds a `series_volume_range` endpoint which allows for series volume queries over time with a specified step, returning timeseries data in the form of a Prometheus matrix response. The existing `series_volume` endpoint still returns Prometheus vector responses, and hardcodes the step to 0.	2 years ago
Salva Corts	6cc581bd26	Add back cache stats for index stats requests (#9816) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9536, we added cache stats for index stats requests. That PR had a bug that inflated the query stats due to reusing the stats context in the query engine. Therefore, we had to revert the PR at https://github.com/grafana/loki/pull/9721. This PR brings back the changes from https://github.com/grafana/loki/pull/9536 but fixes the inflated starts by no longer reusing the same context in the query engine, but rather creating a new one for the shard resolver. I tested it on a dev cluster and seems to be working fine. here's the output for the same query: **Stats with the bug from #9536**: ``` ... Cache.StatsResult.Requests 980 Cache.StatsResult.EntriesRequested 490 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 490 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 43 GB Summary.LinesProcessedPerSecond 93305142 Summary.TotalBytesProcessed 945 GB Summary.TotalLinesProcessed 2059694183 ``` **Stats from _main_** ``` ... Summary.BytesProcessedPerSecond 1.6 GB Summary.LinesProcessedPerSecond 3403718 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207971404 ``` **Stats with fix in this PR** ``` .. Cache.StatsResult.Requests 132 Cache.StatsResult.EntriesRequested 66 Cache.StatsResult.EntriesFound 0 Cache.StatsResult.EntriesStored 66 Cache.StatsResult.BytesSent 0 B Cache.StatsResult.BytesReceived 0 B ... Summary.BytesProcessedPerSecond 4.3 GB Summary.LinesProcessedPerSecond 9468900 Summary.TotalBytesProcessed 95 GB Summary.TotalLinesProcessed 207793816 ``` As can be seen, with the changes in this PR, the summary stats are no longer inflated. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/9536 **Special notes for your reviewer**: I think it's ok to skip reviewing the changes from the commit cherry-picking the changes from https://github.com/grafana/loki/pull/9536 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dylan Guedes	dc5d458c14	Failures logging: avoid logging generic errors (#9796) ... **What this PR does / why we need it**: Modify our write path to not log generic messages with the `writefailures.Manager`. With generic messages I mean: contextCancel errors and contextDeadlineExceeded errors. **Which issue(s) this PR fixes**: N/A	2 years ago
Salva Corts	08e505dd50	Return all errors for a failed push request (#9807) ... **What this PR does / why we need it**: Distributors enforce limits and validate each push payload. before this PR, if a single push request yields multiple errors, only the last error is forwarded to the client. This PR modifies the distributor push handler so we forward all errors from a single push request to the client. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: - I wonder if the use of [MultiError][1] is well-suited for this. [This test case][2] will return 100 errors with the same content. I wonder if we can leverage the MultiError to group errors that have the same content. Printing for example: `1 error like: 'Error parsing labels...'; 100 errors like: 'Max entry size...'`. This would require extra processing but If we do this processing in the Error() method I think it should be ok-ish? - **Edit:** I ended up implementing this in [4b0816f](4b0816f67f) **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) [1]: d4a4728933/pkg/util/errors.go (L32) [2]: f544b93c20/pkg/distributor/distributor_test.go (L75)	2 years ago
Owen Diehl	e2a63e582c	adds tsdb-max-bytes-per-shard limit (#9811) ... Adds the per tenant limit `tsdb-max-bytes-per-shard` which is used in configuring the shard size for tsdb subqueries. This effectively gives control over how big subqueries should be (assuming they're shardable). The default is no different (`600MB`) than the previously hardcoded initial value. This should help us iterate to find optimal shard sizes to improve operations in the long term.	2 years ago
Christian Haudum	a65c99d9bf	Add per-tenant request counter metric to index gateway server (#9797) ... This commit add a counter metric `loki_index_gateway_requests_total` with labels `operation`, `tenant`, `status` for gRPC requests that are served by the index gateway. **What for?** The per-tenant RPS on the index gateway is used to derive the per-tenant shard factor. **Why tracking on the server?** Unlike tracking index gateway RPS on the client side, tracking on the server side does not yield that many series, even in multi-tenant installations with a lot of tenants, because the amount of index gateway instances is relatively small compared to the amount of queriers and frontends. **Special notes for your reviewer**: The previous approach of tracking requests on the client https://github.com/grafana/loki/pull/9781 has been abandoned. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Travis Patterson	8ca035ffbf	Log Volume: Do the prometheus-format conversion in dedicated middleware (#9776) ... This PR moves the Prometheus-format conversion to it's own middleware. This keeps it all in one place and consolidates logic that was in many places. It also sets us up to make range queries as a next step.	2 years ago
Bryan Boreham	6928aac776	loser tree: allow sequences to contain maxVal (#9057) ... **What this PR does / why we need it**: The tree stores the loser in each node, or maxVal if the sequence has ended. If sequences can contain maxVal, we need to take care not to let an ended sequence overtake an active one. Some test cases from Charles who pointed out the issue in #9053. Note this situation cannot happen for real in Loki, since we don't expect log lines with a timestamp that far into the future, but this is supposed to be a re-usable library. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Charles Korn <charles.korn@grafana.com>	2 years ago
Ryo Shindo	ed0d15a6bf	loki: enable PathPrefix for queryHandler endpoints (#8406) ... **What this PR does / why we need it**: Hi, loki developers. I would like to use the option `http_path_prefix` as described in the official documentation. This setting is valid for endpoints defined at 4d55a24ed3/pkg/loki/loki.go (L477-L494) but not for the endpoints defined at 4d55a24ed3/pkg/loki/modules.go (L366-L392) . The current latest version with `http_path_prefix` will return the expected response for the former endpoint, but will return a 404 not found error when accessing the latter endpoint. For example, if you define `http_path_prefix = /xxx`, requests for `/xxx/ready` will work as expected, but requests for `/xxx/loki/api/v1/query` will result in 404 errors. I've created a pull request for this issue because I've been experiencing some inconvenience when using loki. Please check it out. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/4756 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Tests updated --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Sandeep Sukhani	3e1f2fc273	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago