Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Ed Welch d66dd30c17 Loki: introduce an exponential backoff on failed query retries. (#10959) ... **What this PR does / why we need it**: Currently any subquery which fails with a 5xx status code is retried by the query frontend immediately, and by default will make a total of 5 attempts to execute the subquery, ultimately failing the query if you exhaust those attempts. We notice there are times when subqueries can fail extremely quickly, mostly around queriers handling various GRPC related issues between components like ingesters or index gateways. In these cases a query will exhaust all the retries in less than a second and fail back to the user. Also in cases where a query is causing a lot of memory pressure on a pool of queriers leading to OOM crashes of queriers, the lack of a backoff on retries can make this behavior quite aggressive and potentially cause disruption if you don't have enough queriers available. This PR introduces a simple exponential backoff on retries of failed queries to allow for the downstream GRPC issues to have time to sort out as well as causing queries causing a lot of memory pressure and OOMing to be slowed down and minimize their impact slightly. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>		2 years ago
..
astmapper	chore(storage): Restructure shipper packages (#10729)	2 years ago
queryrange	Loki: introduce an exponential backoff on failed query retries. (#10959)	2 years ago
series	Upgrade prometheus to 0.47.1 and dskit (#10814)	2 years ago
stats	Upgrade dskit to use packages migrated from weaveworks/common (#10164)	2 years ago
worker	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago
handler.go	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago
http.go	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago
http_test.go	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago
ingester_querier.go	Add basic structure of bloom gateways (#10782)	2 years ago
ingester_querier_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
metrics.go	Add tailing metrics (#6059)	4 years ago
multi_tenant_querier.go	Upgrade dskit to use packages migrated from weaveworks/common (#10164)	2 years ago
multi_tenant_querier_test.go	Upgrade dskit to use packages migrated from weaveworks/common (#10164)	2 years ago
querier.go	config: better defaults for querier max concurrency (#10785)	2 years ago
querier_mock_test.go	Upgrade prometheus to 0.47.1 and dskit (#10814)	2 years ago
querier_test.go	chore: restructure compactor package (#10775)	2 years ago
tail.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
tail_mock_test.go	…
tail_test.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
testutils.go	Removing duplicated limits file and refactored references (#8210)	3 years ago
worker_service.go	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago
worker_service_test.go	Support protobuf `QueryRequest` in querier. (#10858)	2 years ago