apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Periklis Tsirakidis	e590963471	operator: Add infra annotations for OpenShift based deployments (#11393 )	2 years ago
Periklis Tsirakidis	d683aedb1b	operator: Add tools to test on Azure (#11340 ) )	2 years ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Christian Haudum	79693d79ef	Fix `getFromThrough()` function for ShortRefs (#11386 ) What this PR does / why we need it: Since chunks don't necessarily have the same duration, we cannot assume that the last item of a slice sorted by `From` time also has the greatest `Through` time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Gregor Zeitlinger	abb63b21b9	Update template_functions.md (#9683 ) My previous fix (part of #9672) was wrong - reverting the change Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Pangidoan Butar	7f79db908b	Update log configuration (#11211 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
akevdmeer	18778cd548	De-duplicate common prefixes as returned for individual buckets (#11317 ) With multiple buckets configured on a single AWS/S3 store, each object is stored on one of the buckets by hashing its object key in `bucketFromKey()`. That is sufficient for `GetObject()` and `PutObject()`. `List()` needs to list each bucket and combine the results. It appends all object keys into a `storageObjects` slice. That works because each key uniquely exists in only one of the buckets. But that is not the case for the common prefixes; a common prefix may exist in multiple buckets. This PR removes duplicates from the common prefixes as gathered from the multiple buckets in `List()`. Without this fix, we find that a repeated common prefix leads to a table being compacted multiple times concurrently which is not safe. E.g. this error results when the compactor concurrently tries to compact a table and a 'losing' execution finds that its clean-up has already been done: `level=error caller=compactor.go:128 table-name=loki_index_tsdb_19683 msg="failed to remove downloaded index file" path=/var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb err="remove /var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb: no such file or directory" `	2 years ago
Christian Haudum	856b57336c	Pass `--build-arg` to docker-driver build command (#11379 ) This fixes an issue with the build system when the Go version is updated in the build image and go.mod relies on the new features of the Go version. See failed CI run https://drone.grafana.net/grafana/loki/31777/28/2 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	19b0160821	Bloom gateway client: Use binary search to determin membership of fingerprints to server instances (#11336 ) Instead of iterating over all fingerprints and determine their respective instance from the ring individually, this new approach takes calculates the token ranges from the instances from the ring and matches and partitions the fingerprints based on that using a binary search to determine membership to a token range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	489ac8d529	Extract results cache into new pkg (#11343 ) What this PR does / why we need it: This extracts the results cache from `queryrangebase` into its own pkg so we can reuse it in other components such as the bloom-gateway without having to import `queryrangebase`. - Most of the logic inside `pkg/querier/queryrange/queryrangebase/results_cache.go` now lives in `pkg/storage/chunk/cache/results_cache/cache.go`. - Some of the tests in `pkg/querier/queryrange/queryrangebase/results_cache.go` are moved into pkg/storage/chunk/cache/results_cache/cache_test.go. - Note that here we don't have access to the types we use in `queryrangebase` so we created a new set of mock request/response types to test with.	2 years ago
Christian Haudum	0e433f304e	Bloom Gateway: process blocks immediately when they are available (#11360 ) At current point in time, the worker calls `w.store.GetBlockQueriersForBlockRefs()` to obtain a list of block queriers sorted by fingerprint in ascending order. However, the shipper used by the store to fetch blocks by its refs, calls a callback function when a requested block is available. Instead of waiting for all callbacks to be called and then returning a single list of block queriers, the store now exposes a new function `ForEach()` that also accepts a callback that is passed to the underlying shipper. This allows to process blocks in the worker as soon as they become available. Some block may be served from cache, some may need to be downloaded. That way, we reduce waiting time. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	9538499e60	Add error return to PopulateSeriesWithBloom (#11359 ) What this PR does / why we need it: Adds an error return value to PopulateSeriesWithBloom Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284 ) What this PR does / why we need it: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` All queriers will handle requests for a tenant if neither limits are applied. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) `26f097162a/pkg/loki/modules.go (L895)` but the [docs](`26f097162a/pkg/validation/limits.go (L276)`) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Paul Rogers	ceda046229	Bloom tokenization performance tweaks (#11338 ) What this PR does / why we need it: The main work of this PR is to improve the performance of the reassemble() function. By changing the assignment of the 'cur' variable to not do addition, and incrementing the 'pos' variable separately, the benchmarks improved as follows: BenchmarkTokens/threeSkip1Chunk/v2-10 88864 13487 ns/op BenchmarkTokens/threeSkip1ChunkAt2/v2At2-10 98607 12256 ns/op (This was validated with multiple runs, on multiple size inputs) The remaining changes in the PR are just to add some benchmarking for PopulateSeriesWithBloom, in case we need to do other comparisons for future improvements.	2 years ago
Poyzan	11ef833ed3	Compact multiple series into a bloom block (#11295 ) What this PR does / why we need it: The existing `Job` abstraction has 1 stream fingerprint and compactor was making a 1bloom and 1 block at the end of compaction. This PR changes the `Job` struct to have a list of stream fingerprints. At the end of compaction n streams are iterated over to create n blooms that are appended to 1 block. Stream fingerprints are added to the Job after checking if they are in owned by that compactor's ring, so it is still benefiting from sharding strategy. Resulting block will only contain series blooms that belong to 1 tsdb table. relates to https://github.com/grafana/loki-private/issues/856 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Logic to control block size will be added in a follow up PR.	2 years ago
someStrangerFromTheAbyss	0b9e573140	Helm: Add extraContainers to write pods (#11319 ) Adds the extraContainers to the write pod: Which issue(s) this PR fixes: This fix will allow my team to add a pods that makes modification to the incoming logs before sending them to loki with minimal network transfer. Special notes for your reviewer: NONE Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [x] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dmitriy Okladin	a106b061cb	jsonnet: add memberlist_ring_enabled check condition for consul_config_map (#11312 ) The jsonnet lib should not generate a ConfigMap for consul if you are using memberlist. Fixes #11311	2 years ago
Rens Groothuijsen	cf84b2e9b8	fluent-bit: Redact basic auth credentials from fluent-bit logging (#11308 ) This redacts the basic auth credentials from the configured URL that is displayed when fluent-bit is started, if these are included. Go's built-in redaction only hides the password, turning a url like `http://user:pass@example.com` into `http://user:xxxxx@example.com`. Fixes #10514	2 years ago
Christian Haudum	fe4d9bea68	Remove worker callback function that is only used in tests (#11356 ) The queue already exposes the functionality to track connected consumers (workers), so there is no need to have callback that is only used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Robert Jacob	cd3a04d6d7	operator: Run unit tests of API module as well (#11328 )	2 years ago
Christian Haudum	2411c575fb	Wait for all workers to stop when stopping blockdownloader (#11353 ) The `blockDownloader` does not wait until all of its worker goroutines are finished. This leads to a flaky test, see #11347 To properly fix the behaviour, the `stop()` function should wait until all workers are done. Closes #11347 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	71ddec7550	Bloom-Gateway: Use pools to reduce allocations (#11324 ) What this PR does / why we need it: This PR addresses some TODOs in the Bloom-Gateway component to reduce allocations by pooling. Note for reviewers: - I'm not really sure about the max size for the preallocated pools. Used 16K for all of them but that's probably too much for pools such as the `responsesPool`, `addressesPool`... Feel free to leave a comment in the PR with an estimate on the size you think it would make sense to set as the max.	2 years ago
Ashwanth	6d011bedbc	config: fix index periodic config unmarshal (#11344 ) What this PR does / why we need it: Unmarshalling for `IndexPeriodicTableConfig` is not working as expected since it contains an inline field that implements `yaml.Unmarshaler` interface. Configuring `path_prefix` is resulting in a error because it tries to Unmarshal to the parent inline field directly. Fix this by implementing `yaml.Unmarshaler` interface for `IndexPeriodicTableConfig` related to: https://github.com/go-yaml/yaml/issues/125 ``` field path_prefix not found in type struct { Prefix string "yaml:\"prefix\""; Period model.Duration "yaml:\"period\""; Tags config.Tags "yaml:\"tags\"" } ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	b05544a939	Remove obsolete empty test in middleware. (#11304 ) What this PR does / why we need it: A check for `len(queryRangeMiddleware) > 0` is always true so we can remove it. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Sven Grossmann	1496eb4752	Fix `or` filter not being stringified correctly (#11337 ) What this PR does / why we need it: This PR adds a missing `String()` call for the `or` statement in line filters. It also fixes a bug where `or` statements were not applied in conjunction with an IPLineFilter expression. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Robert Jacob	5270d10a7e	Fix minimum Go version for module (#11145 ) What this PR does / why we need it: The main module of Loki already uses features of Go 1.21 (`cmp` module, `context.WithTimeoutCause`), so it can't be built with Go versions lower than 1.21 anymore. The module file should reflect this. The `go.sum` file was updated automatically by the Go tool `go mod tidy` after updating the version. Special notes for your reviewer: Go 1.21 module files introduce a new `toolchain` attribute. I have set this to the currently available latest version. --------- Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
William Travis Holton	1d232b824c	Helm: Add swift credentials and container name to common config (#11256 ) What this PR does / why we need it: Add swift to common config via helm. Which issue(s) this PR fixes: No issue Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Paul Rogers	df1a3ce79a	Initial pass at metrics for bloom tokenizer (#11329 ) What this PR does / why we need it: Removes the stubbed out metrics from the Bloom Tokenizer, and puts in metrics that were being captured and deemed useful during initial POC testing. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Periklis Tsirakidis	21a7f96b63	operator: Add support for blocking queries per tenant (#11094 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Salva Corts	10210b884c	Categorized Labels: Return empty JSON object if no parsed nor structured metadata labels (#11325 ) This is a followup PR for https://github.com/grafana/loki/pull/10419 wrt this comment https://github.com/grafana/loki/pull/10419#discussion_r1406285386. If there is no parsed nor structured-metadata labels in a given entry, and the categorize-labels header is set, add an empty object to the entry array. For example, we currently return: ```json { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "agent": "promtail", "filename": "/var/log/nginx/json_access.log", "host": "appfelstrudel", "job": "nginx_access_log" }, "values": [ [ "1701096036028751750", "example line" ] ] } ] } } ``` But we want: ```json { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "agent": "promtail", "filename": "/var/log/nginx/json_access.log", "host": "appfelstrudel", "job": "nginx_access_log" }, "values": [ [ "1701096036028751750", "example line", {} <<<--- WE WANT THIS ] ] } ] } } ```	2 years ago
Joao Marcal	bd505f8e2d	mixins: Adds structured metadata panels (#11087 ) Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Salva Corts	67ecf5007b	Make Blooms-Gateway queue settings configurable (#11321 ) What this PR does / why we need it: This PR adds three new settings to the Bloom-Gateway config: - `worker_concurrency`: replaces the `numWorkers` constant. Named after the scheduler's `scheduler_worker_concurrency` config. - `max_outstanding_per_tenant`: replaces the `maxTasksPerTenant` constant. Named after the frontend's `max_outstanding_per_tenant` config. - `pending_tasks_initial_capacity`: replaces the pendingTasksInitialCap constant. Note to the reviewers: - I think in the future `max_outstanding_per_tenant` should be configurable per-tenant. This would require refactoring the queue pkg and the frontend/scheduler, so I think it doesn't make sense it in this PR. - I moved the `Limits` struct definition to `config.go` so have all the adjustable settings in the same place.	2 years ago
Kaviraj Kanagaraj	2a83cf01c0	chore(jsonnet): Fix the storage config on shipper.libsonnet (#11323 ) What this PR does / why we need it: We found this is wrongly configured which removes some config(`active_index_directory` and `cache_location`) in our Loki clusters Introduced via this PR https://github.com/grafana/loki/pull/11195/files Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Joao Marcal	b29b4565a5	operator: Add tools to test on GCP (#11303 ) Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 ) This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
renovate[bot]	09cb9ae76f	fix(deps): update github.com/grafana/loki/pkg/push digest to `e523809` (main) (#11107 ) [![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| [github.com/grafana/loki/pkg/push](https://togithub.com/grafana/loki) \| require \| digest \| `0a7737e` -> `e523809` \| --- ### Configuration 📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zMS41IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Danny Kopping	e523809216	bug(volume): return 400 error for invalid volume request (#11313 ) What this PR does / why we need it: Making a request to the `/volume{,_range}` endpoints results in a 500 is a LogQL expression is given instead of the expected label matchers. The reason for this is that the error is not wrapped in such a way as to be interpreted by `ClientHTTPStatusAndError` correctly. 5xx responses from the querier are automatically retried, so any invalid request currently results in 5 retries which are unnecessary. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Vladyslav Diachenko	75cfe59596	bloom blocks downloading queue (#11201 ) implemented bloom blocks downloading queue to control the concurrency of downloading the blocks from the storage Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
renovate[bot]	10fe48b815	chore(deps): update alpine docker tag to v3.18.4 (main) (#11167 ) [![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| alpine \| final \| patch \| `3.18.3` -> `3.18.4` \| \| alpine \| final \| minor \| `3.16.7` -> `3.18.4` \| --- ### Configuration 📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Vladyslav Diachenko	c8093e1ee3	Revert "Compression of bloom blocks (#11267 )" (#11307 ) This reverts commit `af177034` (https://github.com/grafana/loki/pull/11267) because the compression/uncompression must be done outside of the client context: https://raintank-corp.slack.com/archives/C056KUQFBFT/p1700494151990649 Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Periklis Tsirakidis	6b944f7c79	operator: Fix custom CA for object-store in ruler component (#11288 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290 ) What this PR does / why we need it: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago
Periklis Tsirakidis	21a07773ba	operator: Add automatic stream sharding support (#11091 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Jan Wozniak	a971dcb7ad	Document caveats for 2.x to 3.x loki helm chart upgrades (#11199 ) What this PR does / why we need it: https://github.com/grafana/loki/issues/9427 discusses a few notable gotchas and differences between `v2.x` and `v3.x` helm charts. In this PR, I would like to propose aggregation of the information from the mentioned issue and appending that to the upgrade guide. Which issue(s) this PR fixes: see also: https://github.com/grafana/loki/issues/9427 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` --------- Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	10a21cfe9b	[Docs] Update Helm installation to include backend (#11191 ) What this PR does / why we need it: Updates the Helm Charts documentation to include `backend` target Fixes broken links Which issue(s) this PR fixes: Fixes #11189 Special notes for your reviewer: I walked through this with @monodot earlier today. Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Poyzan	1d90a8b956	Fix bloom compactor startup duplicate metric registration (#11300 ) What this PR does / why we need it: Follow up from https://github.com/grafana/loki/pull/11285/, where Loki doesn't start up due to duplicate metric registration. ``` 2023-11-22 22:43:32 /src/loki/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:276 +0x178 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.newMetrics({0x2a62040, 0x4001baa660}) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/metrics.go:21 +0xa4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.NewTableManager({{0x4000cdae58, 0x18}, 0x45d964b800, 0x4e94914f0000, 0x0, {0xffff4f32e908, 0x40009c6750}}, 0x24192e0, {0x2a89f50?, 0x40004e7320}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/table_manager.go:92 +0xa8 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.(indexShipper).init(0x4001b9a480, {0x231b106?, 0x0?}, {0x2a80bf8?, 0x40018b9340?}, {0xffff4f32e908, 0x40009c6750}, 0x0?, {0x0?, 0x0?, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:190 +0x28c 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.NewIndexShipper({_, _}, {{0x4000685900, 0x19}, {0x4000cdae58, 0x18}, 0x4e94914f0000, 0x45d964b800, 0x0, {{0x2313cd3, ...}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:155 +0x194 2023-11-22 22:43:32 github.com/grafana/loki/pkg/bloomcompactor.New({{{{0x232535d, 0xa}, {0x232789e, 0xb}, {{...}, {...}, {...}, _}, {_, _}}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/bloomcompactor/bloomcompactor.go:142 +0x6d4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor(0x40006dc000) 2023-11-22 22:43:32 /src/loki/pkg/loki/modules.go:1404 +0x358 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).initModule(0x4000b48318, {0xffffe15d6f7d, 0x4}, 0x403b88?, 0x2313cb3?) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x1a4 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).InitModuleServices(0x4000b1a718?, {0x400071c520, 0x1, 0x4000946ec0?}) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xb4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).Run(0x40006dc000, {0x0?, {0x4?, 0x3?, 0x3ff40c0?}}) 2023-11-22 22:43:32 /src/loki/pkg/loki/loki.go:416 +0x74 2023-11-22 22:43:32 main.main() 2023-11-22 22:43:32 /src/loki/cmd/loki/main.go:114 +0xcdc 2023-11-22 22:43:33 panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "loki_tsdb_shipper_query_time_table_download_duration_seconds", help: "Time (in seconds) spent in downloading of files per table at query time", constLabels: {}, variableLabels: [{table <nil>}]} has different label names or a different help string ``` Plus a sneaky fix to our loki config, removing a deprecated field. See deprecation clean up here: https://github.com/grafana/loki/pull/11038/files Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Shashwat Pathak	f3b32023ee	Update _index.md (#11265 ) Replacing the invalid (404) promtail-ec2.yaml with the valid one. What this PR does / why we need it: This PR does - updates the invalid url of the promtail-ec2.yaml with the valid one. Why it is needed - [While implementing this doc](https://grafana.com/docs/loki/latest/send-data/promtail/cloud/ec2/), the curl won't work. Which issue(s) this PR fixes: Fixes #<issue number> No issue has been raised for this Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Bilal Khan	13b58afdd9	made some typo changes in the caching.md file (#11214 ) What this PR does / why we need it: There were some typo mistakes that I fixed in the caching.md file and it was mentioned in the issue that I have given below. Which issue(s) this PR fixes: Fixes [Issue #9073](https://github.com/grafana/loki/issues/9073) Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Signed-off-by: Bilal Khan <bilalkhanrecovered@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Vitor Bruno de Oliveira Barth	146b1bb10b	Helm: Update MinIO Helm Chart version to 4.0.15 (#11188 ) What this PR does / why we need it: It bumps the version for the MinIO Helm Chart to v4.0.15. The current MinIO version (4.0.12) has a bug where it throws warnings for incorrect types. ``` coalesce.go:237: warning: skipped value for loki.minio.additionalLabels: Not a table. coalesce.go:237: warning: skipped value for loki.minio.additionalAnnotations: Not a table. ``` Which issue(s) this PR fixes: None Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [X] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christian Haudum	b56f36fe6b	Fix bloom compactor startup when using unsupported index type (#11285 ) This PR fixes the check in the bloom compactor startup to ignore all other index types than TSDB. When updating an existing Loki installation that uses BoltDB Shipper in the schema config, you get the following error: ``` invalid tsdb path: /data/boltdb-shipper-cache/XXXXXXXXX_index_19682/write-0-1639160260677624223-1700595900 ││ create index shipper ││ github.com/grafana/loki/pkg/bloomcompactor.New ││ /src/loki/pkg/bloomcompactor/bloomcompactor.go:159 ││ github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor ││ /src/loki/pkg/loki/modules.go:1406 ││ github.com/grafana/dskit/modules.(Manager).initModule ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 ││ github.com/grafana/dskit/modules.(Manager).InitModuleServices ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 ││ github.com/grafana/loki/pkg/loki.(Loki).Run ``` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

1 2 3 4 5 ...

5948 Commits (e590963471c6dc75edbe0151e61ba3ecd7b8f673) All Branches Search

5948 Commits (e590963471c6dc75edbe0151e61ba3ecd7b8f673)

All Branches