apitech/loki - loki - Gitea: Git with a cup of tea

Author SHA1 Message Date

Author	SHA1	Message	Date
Karsten Jeschkies	9fcc42dc48	Support protobuf `QueryRequest` in querier. (#10858 ) What this PR does / why we need it: This is one step to support pure protobuf encoding without `httpgrpc`. It is only on the scheduler and is fully backwards compatible. Which issue(s) this PR fixes: This is a sub change of https://github.com/grafana/loki/pull/10688 Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Dylan Guedes	6ce510dc55	Storage: Introduce new "ObjectExists" API (#10360 ) What this PR does / why we need it: Introduce to our `Store` interface a new function named `ObjectExists`, which can be used by external tools to validate if a given file is present on an object store without downloading the file. Which issue(s) this PR fixes: N/A	2 years ago
Danny Kopping	beed298549	Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140 ) What this PR does / why we need it: Across the various cloud providers' object storage services, there are different rate-limits implemented. Rate-limits can be imposed under multiple conditions, such as server-side scale up (ramping up from low volume to high, "local" limit), reaching some defined upper limit ("absolute" limit), etc. We cannot know apriori when these rate-limits will be imposed, so we can't set up a client-side limiter to only allow a certain number of requests through per second. Additionally, that would require global coordination between queriers - which is difficult. With the above constraints, I have instead taken inspiration from TCP's [congestion control algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This PR implements [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) (Additive Increase, Multiplicative Decrease), which is used in the congestion _avoidance_ phase of congestion control. The default window size (object store requests per second) is 2000; in other words, we skip the "slow start" phase. The controller uses the Go [`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which implements the token-bucket algorithm. To put it simply: - every successful request widens the window (per second client rate-limit) - every rate-limited response reduces the window size by a backoff factor (0.5 by default, so it will halve) - when the limit has been reached, the querier will be delayed from making further requests until tokens are available	2 years ago

Karsten Jeschkies

Support protobuf `QueryRequest` in querier. (#10858 )

**What this PR does / why we need it**:
This is one step to support pure protobuf encoding without `httpgrpc`.
It is only on the scheduler and is fully backwards compatible.

**Which issue(s) this PR fixes**:
This is a sub change of https://github.com/grafana/loki/pull/10688

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [x] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e3ec)

---------

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>
Co-authored-by: Danny Kopping <dannykopping@gmail.com>

2 years ago

Dylan Guedes

6ce510dc55

Storage: Introduce new "ObjectExists" API (#10360 )

**What this PR does / why we need it**:
Introduce to our `Store` interface a new function named `ObjectExists`,
which can be used by external tools to validate if a given file is
present on an object store without downloading the file.

**Which issue(s) this PR fixes**:
N/A

2 years ago

Danny Kopping

beed298549

Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140 )

**What this PR does / why we need it**:
Across the various cloud providers' object storage services, there are
different rate-limits implemented. Rate-limits can be imposed under
multiple conditions, such as server-side scale up (ramping up from low
volume to high, "local" limit), reaching some defined upper limit
("absolute" limit), etc.

We cannot know apriori when these rate-limits will be imposed, so we
can't set up a client-side limiter to only allow a certain number of
requests through per second. Additionally, that would require global
coordination between queriers - which is difficult.

With the above constraints, I have instead taken inspiration from TCP's
[congestion control
algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This
PR implements
[AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease)
(Additive Increase, Multiplicative Decrease), which is used in the
congestion _avoidance_ phase of congestion control. The default window
size (object store requests per second) is 2000; in other words, we skip
the "slow start" phase.

The controller uses the Go
[`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which
implements the token-bucket algorithm.

To put it simply:
- every successful request widens the window (per second client
rate-limit)
- every rate-limited response reduces the window size by a backoff
factor (0.5 by default, so it will halve)
- when the limit has been reached, the querier will be delayed from
making further requests until tokens are available

2 years ago

3 Commits (b1a61b3df60f11b6cee7d44b0fba0d12d59886a0)