apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Danny Kopping	beed298549	Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140 ) What this PR does / why we need it: Across the various cloud providers' object storage services, there are different rate-limits implemented. Rate-limits can be imposed under multiple conditions, such as server-side scale up (ramping up from low volume to high, "local" limit), reaching some defined upper limit ("absolute" limit), etc. We cannot know apriori when these rate-limits will be imposed, so we can't set up a client-side limiter to only allow a certain number of requests through per second. Additionally, that would require global coordination between queriers - which is difficult. With the above constraints, I have instead taken inspiration from TCP's [congestion control algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This PR implements [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) (Additive Increase, Multiplicative Decrease), which is used in the congestion _avoidance_ phase of congestion control. The default window size (object store requests per second) is 2000; in other words, we skip the "slow start" phase. The controller uses the Go [`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which implements the token-bucket algorithm. To put it simply: - every successful request widens the window (per second client rate-limit) - every rate-limited response reduces the window size by a backoff factor (0.5 by default, so it will halve) - when the limit has been reached, the querier will be delayed from making further requests until tokens are available	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601 ) What this PR does / why we need it: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	3 years ago
Totalus	01f0ded7fc	Update swift_object_client.go : Fetch all objects in List() function (#9463 ) What this PR does / why we need it: Allow the Swift Client `List()` function to return all objects that match the query. Which issue(s) this PR fixes: Fixes #9360 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
janhorstmann	0e81d78914	Swift: Allow to configure swift with the internal endpoint (#8438 )	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212 ) The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Cyril Tovena	8f02495453	Refactoring of the Loki storage (#5833 ) * Extract index interface from the series store * moving package around * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moving package around * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moving more things around again Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve composite store for more generics index store * fixes first round of tests and ensure coverage is still good * implement series method * Fixes storage tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test for GetSeries Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Make Loki compile Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint the whole project Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tsdb util package usage Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add more information of why SetChunkFilterer exists for the index Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler/base package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes o2 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Dylan Guedes	7856db62bb	Copy `cortex/pkg/storage/bucket` package dependency into Loki (#5258 ) * Fork cortex `storage/bucket` into Loki. * Replace storage/bucket usage from Cortex to Loki one. * Remove redundant error check. * Revert SwiftConfig struct and remove cortex pkgs alias. * Add missing test files. * Replace `cortex/pkg/util` with our own implementation. * Remove unused cortex packages.	4 years ago
Dylan Guedes	ba6a2fd41e	Get rid of Cortex chunk. (#5239 )	4 years ago
Arve Knudsen	a2dd731586	Chore: Add pkg/util/log package (#5187 ) * Chore: Add pkg/util/log package Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Add changelog entry Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Fix documentation of loki_log_messages_total Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 years ago
Cyril Tovena	181f4b698d	Fixes metric name for hedging object store requests (#5098 ) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	8b71a44a86	Pre-allocate chunk buffer size (#4884 ) * Pre-allocate chunk buffer size Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add doc	4 years ago
Cyril Tovena	03be8dac1e	Allow to disable http2 for GCS. (#4942 ) * Allow to disable http2 for GCS. This also tweaks our iddle connection pooling accross the board and reduce the insane parallelism when fetching chunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * update changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/storage/chunk/gcp/gcs_object_client.go Co-authored-by: Danny Kopping <dannykopping@gmail.com> * update comment Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/configuration/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Reduce MaxIdleConns to a more standard value. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Allow to configure max parallelism for fetching chunks Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes tests with no default config Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Oleg Kovalov	e893e2696c	Update hedgedhttp to v0.7.0 (#4892 ) * Update hedgedhttp to v0.7.0 * Update CHANGELOG * Fix returned results	4 years ago
Cyril Tovena	c33a651645	Add rate limiting and metrics to hedging (#4860 ) * Add rate limiting and metrics to hedging Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Allow to pass a custom registerer Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Cyril Tovena	b27894fbf4	Adds the ability to hedge storage requests. (#4826 ) * Adds the ability to hedge request for all backends Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove race from tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the race Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove the race Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Testing Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More testing Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Setup credentials to avoid auth Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * improve tests Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * gomod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * changelog Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Group the configuration Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	4 years ago
Trevor Whitney	2427fab32d	Configuration: add a common config section for object storage (#4473 ) * add common config for object storage Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * wip Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * add test to compactor config override Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * update changelog Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> * changes from PR review * rename object_store -> storage * add local/filesytem support to common object storage config	4 years ago
Sandeep Sukhani	138ca71dd5	add a method to check whether error is due to a missing object in storage while reading or deleting the object (#4081 )	4 years ago
Cyril Tovena	32e55c3043	Fork cortex chunk storage into Loki. (#3842 ) The chunk package has been forked from this commit for future reference and licensing. `09fe17c0e6`	5 years ago
Jacob Lisi	c3d1937eab	update vendored cortex and add new replace overrides (#3256 ) * update vendored cortex and add new replace overrides Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update upstream function to new package Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update all moved functions Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update math to util_math Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>	5 years ago
Cyril Tovena	33abab5e42	Cortex update pre 1.7 (#3215 ) * Starting to update cortex. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes ruler overrides. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * go.mod Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Owen Diehl	54c0c5febb	Revendor Cortex (#2755 ) * replaces prometheus & revendors cortex * compat * compat * sd refactor * updates docker * removes prom replace * adds delimiters back to object addrs in boltdb	5 years ago
Owen Diehl	4a45ca5e3a	Revendor cortex (#2427 ) * vendoring bump * vendoring compat	6 years ago
Cyril Tovena	47b79f3ea5	Update cortex to latest. (#1982 ) * Update cortex to latest. This brings fixes for traceID in the logger and swift. I've also added a bit of configuration for swift. Fixes https://github.com/grafana/loki/issues/905 Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update go.sum Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	6 years ago

6 Commits (3a4c0e2d47f74ab73db577e7b08f4686cbd8f873)