8 Commits (4cae003ecedd474e4c15feab4ea2ef435afff83f)

Author	SHA1	Message	Date
Danny Kopping	e7b782b186	Query-frontend: use the same query split interval for generated cache keys (#11679) ... **What this PR does / why we need it**: Follow up from #11535 (specifically from [this thread](https://github.com/grafana/loki/pull/11535#issuecomment-1887151515)), this PR modifies the results cache implementation to use the same interval for generating cache keys.	2 years ago
Salva Corts	489ac8d529	Extract results cache into new pkg (#11343) ... **What this PR does / why we need it**: This extracts the results cache from `queryrangebase` into its own pkg so we can reuse it in other components such as the bloom-gateway without having to import `queryrangebase`. - Most of the logic inside `pkg/querier/queryrange/queryrangebase/results_cache.go` now lives in `pkg/storage/chunk/cache/results_cache/cache.go`. - Some of the tests in `pkg/querier/queryrange/queryrangebase/results_cache.go` are moved into pkg/storage/chunk/cache/results_cache/cache_test.go. - Note that here we don't have access to the types we use in `queryrangebase` so we created a new set of mock request/response types to test with.	3 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025) ... **What this PR does / why we need it**: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	3 years ago
Callum Styan	557b516b66	Refactor the proto Request interface to return time.Time (#11018) ... Part of our query path improvements. In addition to Karstens new protobuf formats for all the query request and responses, plus our use of just proto grpc rather than httpgrpc, we'd like refactor the proto Request interface to return time.Time for start and end instead of int64. This will allow his protobuf formats to use more `logproto` proto types. There's still one or two tests that I need to fix. --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164) ... **What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	3 years ago
Karsten Jeschkies	b35bbd80d6	Support content negotiation between query frontend and querier. (#9813) ... **What this PR does / why we need it**: Currently, the querier sends results to the query frontend in JSON which is then decoded to Protobuf. It is more efficient to send the results as Protobuf. This will also allow to extend the results with custom data structures. The change is backwards compatible through content negotiation. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Salva Corts	c6fbff26e1	Add config to avoid caching stats for recent data (#9537) ... **What this PR does / why we need it**: When we query the stats for recent data, we query both the ingesters and the index gateways for the stats. ebdb2b1800/pkg/storage/async_store.go (L112-L114) ebdb2b1800/pkg/storage/async_store.go (L126-L127) Then we merge all the responses, which means summing up all the stats ebdb2b1800/pkg/storage/async_store.go (L157-L158) ebdb2b1800/pkg/storage/stores/index/stats/stats.go (L23-L26) Because we have a replication factor of 3, this means that we will get the stats from the ingesters repeated up to 3 times, hence inflating the stats. In the stats cache, we store the stats for a given matcher set for the whole day, then we extract the stats from the cache by the factor of time from the request that is stored in the cache: 336283acad/pkg/querier/queryrange/index_stats_cache.go (L33) 336283acad/pkg/querier/queryrange/index_stats_cache.go (L40) Inflated stats for recent data will be cached, so subsequent stats extracted from the cache will be inflated regardless of the time. This PR adds a new per-tenant limit `max_stats_cache_freshness` to not cache requests with an end time that falls within Now minus this duration. Here's a scenario illustrating this. The graphs below show the bytes stats queried in the sharding middleware. We are running a log filter query that won't match any log, every 5 seconds with a length of 3h.  As can be seen, after enabling the stats cache and configuring`do_not_cache_request_within` to not cache stats for requests within 30m, the bytes stats used in the sharding middleware stopped increasing. In both cases the stats cache hit ration was 100%.  **Special notes for your reviewer**: - Blocked by https://github.com/grafana/loki/pull/9535 - Note that this PR doesn't fix the root issue of inflated stats form the ingesters, but rather buys us some time to work on that. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	3 years ago
Salva Corts	fd16425062	Cache index stats requests (#8972) ... **What this PR does / why we need it**: As described in https://github.com/grafana/loki/issues/8973, we are substantially increasing the load of index stat requests we sent to our index gateways. Many of these requests should be easily re-used by caching them. This PR adds caching for index stat requests by reusing the results cache. Here's a demo ([source][1]):  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8973 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://ops.grafana-ops.net/d/afcaef21-e5ad-49e7-ab06-42a9d7d915eb/index-stats?orgId=1&var-datasource=dev-cortex&var-cluster=dev-eu-west-2&var-namespace=loki-dev-009&var-loki_datasource=Grafana%20Logging&from=1680259907288&to=1680260431814&var-operation=All --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago