apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Christian Haudum	79693d79ef	Fix `getFromThrough()` function for ShortRefs (#11386 ) What this PR does / why we need it: Since chunks don't necessarily have the same duration, we cannot assume that the last item of a slice sorted by `From` time also has the greatest `Through` time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
akevdmeer	18778cd548	De-duplicate common prefixes as returned for individual buckets (#11317 ) With multiple buckets configured on a single AWS/S3 store, each object is stored on one of the buckets by hashing its object key in `bucketFromKey()`. That is sufficient for `GetObject()` and `PutObject()`. `List()` needs to list each bucket and combine the results. It appends all object keys into a `storageObjects` slice. That works because each key uniquely exists in only one of the buckets. But that is not the case for the common prefixes; a common prefix may exist in multiple buckets. This PR removes duplicates from the common prefixes as gathered from the multiple buckets in `List()`. Without this fix, we find that a repeated common prefix leads to a table being compacted multiple times concurrently which is not safe. E.g. this error results when the compactor concurrently tries to compact a table and a 'losing' execution finds that its clean-up has already been done: `level=error caller=compactor.go:128 table-name=loki_index_tsdb_19683 msg="failed to remove downloaded index file" path=/var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb err="remove /var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb: no such file or directory" `	2 years ago
Christian Haudum	19b0160821	Bloom gateway client: Use binary search to determin membership of fingerprints to server instances (#11336 ) Instead of iterating over all fingerprints and determine their respective instance from the ring individually, this new approach takes calculates the token ranges from the instances from the ring and matches and partitions the fingerprints based on that using a binary search to determine membership to a token range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	489ac8d529	Extract results cache into new pkg (#11343 ) What this PR does / why we need it: This extracts the results cache from `queryrangebase` into its own pkg so we can reuse it in other components such as the bloom-gateway without having to import `queryrangebase`. - Most of the logic inside `pkg/querier/queryrange/queryrangebase/results_cache.go` now lives in `pkg/storage/chunk/cache/results_cache/cache.go`. - Some of the tests in `pkg/querier/queryrange/queryrangebase/results_cache.go` are moved into pkg/storage/chunk/cache/results_cache/cache_test.go. - Note that here we don't have access to the types we use in `queryrangebase` so we created a new set of mock request/response types to test with.	2 years ago
Christian Haudum	0e433f304e	Bloom Gateway: process blocks immediately when they are available (#11360 ) At current point in time, the worker calls `w.store.GetBlockQueriersForBlockRefs()` to obtain a list of block queriers sorted by fingerprint in ascending order. However, the shipper used by the store to fetch blocks by its refs, calls a callback function when a requested block is available. Instead of waiting for all callbacks to be called and then returning a single list of block queriers, the store now exposes a new function `ForEach()` that also accepts a callback that is passed to the underlying shipper. This allows to process blocks in the worker as soon as they become available. Some block may be served from cache, some may need to be downloaded. That way, we reduce waiting time. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	9538499e60	Add error return to PopulateSeriesWithBloom (#11359 ) What this PR does / why we need it: Adds an error return value to PopulateSeriesWithBloom Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284 ) What this PR does / why we need it: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` All queriers will handle requests for a tenant if neither limits are applied. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) `26f097162a/pkg/loki/modules.go (L895)` but the [docs](`26f097162a/pkg/validation/limits.go (L276)`) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Paul Rogers	ceda046229	Bloom tokenization performance tweaks (#11338 ) What this PR does / why we need it: The main work of this PR is to improve the performance of the reassemble() function. By changing the assignment of the 'cur' variable to not do addition, and incrementing the 'pos' variable separately, the benchmarks improved as follows: BenchmarkTokens/threeSkip1Chunk/v2-10 88864 13487 ns/op BenchmarkTokens/threeSkip1ChunkAt2/v2At2-10 98607 12256 ns/op (This was validated with multiple runs, on multiple size inputs) The remaining changes in the PR are just to add some benchmarking for PopulateSeriesWithBloom, in case we need to do other comparisons for future improvements.	2 years ago
Poyzan	11ef833ed3	Compact multiple series into a bloom block (#11295 ) What this PR does / why we need it: The existing `Job` abstraction has 1 stream fingerprint and compactor was making a 1bloom and 1 block at the end of compaction. This PR changes the `Job` struct to have a list of stream fingerprints. At the end of compaction n streams are iterated over to create n blooms that are appended to 1 block. Stream fingerprints are added to the Job after checking if they are in owned by that compactor's ring, so it is still benefiting from sharding strategy. Resulting block will only contain series blooms that belong to 1 tsdb table. relates to https://github.com/grafana/loki-private/issues/856 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Logic to control block size will be added in a follow up PR.	2 years ago
Christian Haudum	fe4d9bea68	Remove worker callback function that is only used in tests (#11356 ) The queue already exposes the functionality to track connected consumers (workers), so there is no need to have callback that is only used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	2411c575fb	Wait for all workers to stop when stopping blockdownloader (#11353 ) The `blockDownloader` does not wait until all of its worker goroutines are finished. This leads to a flaky test, see #11347 To properly fix the behaviour, the `stop()` function should wait until all workers are done. Closes #11347 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	71ddec7550	Bloom-Gateway: Use pools to reduce allocations (#11324 ) What this PR does / why we need it: This PR addresses some TODOs in the Bloom-Gateway component to reduce allocations by pooling. Note for reviewers: - I'm not really sure about the max size for the preallocated pools. Used 16K for all of them but that's probably too much for pools such as the `responsesPool`, `addressesPool`... Feel free to leave a comment in the PR with an estimate on the size you think it would make sense to set as the max.	2 years ago
Ashwanth	6d011bedbc	config: fix index periodic config unmarshal (#11344 ) What this PR does / why we need it: Unmarshalling for `IndexPeriodicTableConfig` is not working as expected since it contains an inline field that implements `yaml.Unmarshaler` interface. Configuring `path_prefix` is resulting in a error because it tries to Unmarshal to the parent inline field directly. Fix this by implementing `yaml.Unmarshaler` interface for `IndexPeriodicTableConfig` related to: https://github.com/go-yaml/yaml/issues/125 ``` field path_prefix not found in type struct { Prefix string "yaml:\"prefix\""; Period model.Duration "yaml:\"period\""; Tags config.Tags "yaml:\"tags\"" } ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	b05544a939	Remove obsolete empty test in middleware. (#11304 ) What this PR does / why we need it: A check for `len(queryRangeMiddleware) > 0` is always true so we can remove it. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Sven Grossmann	1496eb4752	Fix `or` filter not being stringified correctly (#11337 ) What this PR does / why we need it: This PR adds a missing `String()` call for the `or` statement in line filters. It also fixes a bug where `or` statements were not applied in conjunction with an IPLineFilter expression. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Paul Rogers	df1a3ce79a	Initial pass at metrics for bloom tokenizer (#11329 ) What this PR does / why we need it: Removes the stubbed out metrics from the Bloom Tokenizer, and puts in metrics that were being captured and deemed useful during initial POC testing. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Salva Corts	10210b884c	Categorized Labels: Return empty JSON object if no parsed nor structured metadata labels (#11325 ) This is a followup PR for https://github.com/grafana/loki/pull/10419 wrt this comment https://github.com/grafana/loki/pull/10419#discussion_r1406285386. If there is no parsed nor structured-metadata labels in a given entry, and the categorize-labels header is set, add an empty object to the entry array. For example, we currently return: ```json { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "agent": "promtail", "filename": "/var/log/nginx/json_access.log", "host": "appfelstrudel", "job": "nginx_access_log" }, "values": [ [ "1701096036028751750", "example line" ] ] } ] } } ``` But we want: ```json { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "agent": "promtail", "filename": "/var/log/nginx/json_access.log", "host": "appfelstrudel", "job": "nginx_access_log" }, "values": [ [ "1701096036028751750", "example line", {} <<<--- WE WANT THIS ] ] } ] } } ```	2 years ago
Salva Corts	67ecf5007b	Make Blooms-Gateway queue settings configurable (#11321 ) What this PR does / why we need it: This PR adds three new settings to the Bloom-Gateway config: - `worker_concurrency`: replaces the `numWorkers` constant. Named after the scheduler's `scheduler_worker_concurrency` config. - `max_outstanding_per_tenant`: replaces the `maxTasksPerTenant` constant. Named after the frontend's `max_outstanding_per_tenant` config. - `pending_tasks_initial_capacity`: replaces the pendingTasksInitialCap constant. Note to the reviewers: - I think in the future `max_outstanding_per_tenant` should be configurable per-tenant. This would require refactoring the queue pkg and the frontend/scheduler, so I think it doesn't make sense it in this PR. - I moved the `Limits` struct definition to `config.go` so have all the adjustable settings in the same place.	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 ) This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	e523809216	bug(volume): return 400 error for invalid volume request (#11313 ) What this PR does / why we need it: Making a request to the `/volume{,_range}` endpoints results in a 500 is a LogQL expression is given instead of the expected label matchers. The reason for this is that the error is not wrapped in such a way as to be interpreted by `ClientHTTPStatusAndError` correctly. 5xx responses from the querier are automatically retried, so any invalid request currently results in 5 retries which are unnecessary. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Vladyslav Diachenko	75cfe59596	bloom blocks downloading queue (#11201 ) implemented bloom blocks downloading queue to control the concurrency of downloading the blocks from the storage Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	c8093e1ee3	Revert "Compression of bloom blocks (#11267 )" (#11307 ) This reverts commit `af177034` (https://github.com/grafana/loki/pull/11267) because the compression/uncompression must be done outside of the client context: https://raintank-corp.slack.com/archives/C056KUQFBFT/p1700494151990649 Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290 ) What this PR does / why we need it: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago
Poyzan	1d90a8b956	Fix bloom compactor startup duplicate metric registration (#11300 ) What this PR does / why we need it: Follow up from https://github.com/grafana/loki/pull/11285/, where Loki doesn't start up due to duplicate metric registration. ``` 2023-11-22 22:43:32 /src/loki/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:276 +0x178 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.newMetrics({0x2a62040, 0x4001baa660}) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/metrics.go:21 +0xa4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.NewTableManager({{0x4000cdae58, 0x18}, 0x45d964b800, 0x4e94914f0000, 0x0, {0xffff4f32e908, 0x40009c6750}}, 0x24192e0, {0x2a89f50?, 0x40004e7320}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/table_manager.go:92 +0xa8 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.(indexShipper).init(0x4001b9a480, {0x231b106?, 0x0?}, {0x2a80bf8?, 0x40018b9340?}, {0xffff4f32e908, 0x40009c6750}, 0x0?, {0x0?, 0x0?, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:190 +0x28c 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.NewIndexShipper({_, _}, {{0x4000685900, 0x19}, {0x4000cdae58, 0x18}, 0x4e94914f0000, 0x45d964b800, 0x0, {{0x2313cd3, ...}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:155 +0x194 2023-11-22 22:43:32 github.com/grafana/loki/pkg/bloomcompactor.New({{{{0x232535d, 0xa}, {0x232789e, 0xb}, {{...}, {...}, {...}, _}, {_, _}}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/bloomcompactor/bloomcompactor.go:142 +0x6d4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor(0x40006dc000) 2023-11-22 22:43:32 /src/loki/pkg/loki/modules.go:1404 +0x358 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).initModule(0x4000b48318, {0xffffe15d6f7d, 0x4}, 0x403b88?, 0x2313cb3?) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x1a4 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).InitModuleServices(0x4000b1a718?, {0x400071c520, 0x1, 0x4000946ec0?}) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xb4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).Run(0x40006dc000, {0x0?, {0x4?, 0x3?, 0x3ff40c0?}}) 2023-11-22 22:43:32 /src/loki/pkg/loki/loki.go:416 +0x74 2023-11-22 22:43:32 main.main() 2023-11-22 22:43:32 /src/loki/cmd/loki/main.go:114 +0xcdc 2023-11-22 22:43:33 panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "loki_tsdb_shipper_query_time_table_download_duration_seconds", help: "Time (in seconds) spent in downloading of files per table at query time", constLabels: {}, variableLabels: [{table <nil>}]} has different label names or a different help string ``` Plus a sneaky fix to our loki config, removing a deprecated field. See deprecation clean up here: https://github.com/grafana/loki/pull/11038/files Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Christian Haudum	b56f36fe6b	Fix bloom compactor startup when using unsupported index type (#11285 ) This PR fixes the check in the bloom compactor startup to ignore all other index types than TSDB. When updating an existing Loki installation that uses BoltDB Shipper in the schema config, you get the following error: ``` invalid tsdb path: /data/boltdb-shipper-cache/XXXXXXXXX_index_19682/write-0-1639160260677624223-1700595900 ││ create index shipper ││ github.com/grafana/loki/pkg/bloomcompactor.New ││ /src/loki/pkg/bloomcompactor/bloomcompactor.go:159 ││ github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor ││ /src/loki/pkg/loki/modules.go:1406 ││ github.com/grafana/dskit/modules.(Manager).initModule ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 ││ github.com/grafana/dskit/modules.(Manager).InitModuleServices ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 ││ github.com/grafana/loki/pkg/loki.(Loki).Run ``` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	af177034ed	Compression of bloom blocks (#11267 ) What this PR does / why we need it: Compress bloom blocks when writing to object storage, and uncompress when reading from object storage. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	e6940691c9	Set query plan when copying LokiRequest (#11291 ) What this PR does / why we need it: The recent change https://github.com/grafana/loki/pull/11246 requires that `LokiRequest.Plan` is always set. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Sandeep Sukhani	e28f7f3363	update dskit to latest version (#11287 ) What this PR does / why we need it: Update dskit to latest version	2 years ago
Karsten Jeschkies	5b97fcfd93	Send query plan to querier. (#11246 ) What this PR does / why we need it: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Sandeep Sukhani	162bbb11be	do not run retention loop when retention is not enabled (#11280 ) What this PR does / why we need it: Do not run retention loop when retention is not enabled	2 years ago
Owen Diehl	6e93d150d8	utilities for reducing ngram allocations in token iterator construction (#11276 ) Previously, the `calculatePrefix` function was used to generate the token buffer used, but this wouldn't necessarily have enough space to append the ngram afterwards without reallocating+copying the token buffer. This PR ensures we allocate enough space for both prefix and ngram in the token buffer.	2 years ago
Sandeep Sukhani	529084935c	use nanosecond precision for timestamp in compacted boltdb-shipper index file names (#11277 ) What this PR does / why we need it: In PR #9884, we split the compaction and retention loop to run them concurrently. Although we make sure we do not work on the same index from compaction and retention loop, there is a chance that one could run immediately after the other and finish quickly enough to build the index with the same name as the previous one because in `boltdb-shipper` index, we use epoch with `Seconds` precision while building the name of the compacted index file. Since compaction uploads the new file first and then deletes the old file, if the index is built with the same name, we end up uploading the file and deleting it afterwards. This PR fixes the issue by using ns precision for the timestamp in the filenames. Special notes for your reviewer: This is not a problem for TSDB since we also add a checksum to the filenames of the index during compaction.	2 years ago
Paul Rogers	5f4a719bb6	tokenizer v1 cleanup (#11272 ) What this PR does / why we need it: Removes all usage of v1 tokenizers, renames v2 to v1 since we never released this in a production way. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	8d34f857bc	Propagate trace ID with HTTP gRPC request. (#11251 ) The changes in https://github.com/grafana/loki/pull/10688 did not propage the trace ID from the context. `Frontend.RoundTripGRPC` would inject the trace ID into the request. That's not done in `Frontend.Do`. This changes extends the `codec.EncodeRequest` to inject the trace ID there. This is more inline with other metadata.	2 years ago
Sandeep Sukhani	6a62b8cf42	compaction: Separate metrics for tracking retention and compaction (#11263 ) What this PR does / why we need it: In PR #9884, we separated the retention loop from compaction to avoid blocking compaction for too long due to some intensive delete requests. Currently, we track retention and compaction using the same metrics. This PR adds separate metrics for monitoring retention operation. I have also updated the Retention dashboard to use the new metrics.	2 years ago
Kaviraj Kanagaraj	30d0030bf9	inflight-logging: Add extra metadata to inflight requests logging (#11243 ) What this PR does / why we need it: logging: Add extra metadata to inflight requests This adds extra metadata (similar to what we have in `metrics.go`) but for queries in in-flight (both started and retrying) Changes: Adds following data 1. Query Hash 2. Start and End time 3. Start and End delta 4. Length of the query 5. Moved the helper util to `queryutil` package because of cyclic dependencies with `logql` package. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Find the screenshots of log entries looks like (both in `retry.go` and `roundtrip.go`) ![Screenshot 2023-11-16 at 13 01 32](https://github.com/grafana/loki/assets/3735252/177e97ed-6ee8-41dd-b088-2e4f49562ba0) ![Screenshot 2023-11-16 at 13 02 15](https://github.com/grafana/loki/assets/3735252/fb328a37-dbe3-483e-b083-f21327858029) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Sandeep Sukhani	c716e498ad	compactor: do not block compation when retention is taking too long (#9884 ) What this PR does / why we need it: Currently, we perform compaction and apply retention in the same loop. Although we have a flag for not applying retention every time we perform compaction, we still see compaction getting blocked when processing some intensive delete requests(processed while applying retention). This PR separates out the compaction and retention to run in a separate loop. I have added a table-locking feature to avoid compaction and retention from processing the same tables at a time. However, compaction and retention would treat locked tables differently, as explained below: * When compaction sees a table is locked: It would skip the table and move on to the following table. However, before skipping, it would check if the table has any uncompacted files and increment the newly added counter called `loki_compactor_skipped_compacting_locked_tables_total` to track how often we are skipping compacting tables which have uncompacted files. * When retention sees a table is locked: It would wait for the lock to be released since we can't skip any tables while processing delete requests. Special notes for your reviewer: * The check for tables with uncompacted files looks for count > 1 because initially, we did not support per tenant index in `boltdb-shipper`, so a table can have a single compacted multi-tenant index file. In a rare case where we have a single file which was supposed to be compacted away, it is okay to have a single uncompacted file for a while. The aim here is to not block compaction for too long in a large cell with too many uncompacted files. * Retention only works on the compacted index, so we first compact down any uncompacted files while applying retention. Checklist - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Owen Diehl	58eaad9947	boundscheck & partitioning fingerprints between blocks (#11237 ) Adds a few utilities for comparing fingerprints to blocks that cover a specific fingerprint range. Will likely need to be refactored with more comprehensive types, but the logic still applies.	2 years ago
Owen Diehl	6db9d778a1	inverts the logic when testing a block against a list of chunks (#11248 ) returns the list of chunks in the bloom which failed the test so we can merge these results across blocks	2 years ago
Owen Diehl	15d98fa7ca	iterable tokenizer w/ comparison benching (#11255 ) Rebuilds tokenizers based on an iterable approach. This ends up being significantly faster -- benchmarks below. This PR doesn't remove the old tokenizers, just adds the new ones. Revamping tokenizer consumers & removing the old variants can be done in a separate PR. Roughly, raw tokenizers complete in 66% of the time and chunkID prefixed tokenizers in ~55-60% of the time. ``` pkg: github.com/grafana/loki/pkg/storage/bloom/v1 BenchmarkTokens/three/v1-10 134811 8648 ns/op 0 B/op 0 allocs/op BenchmarkTokens/three/v2-10 179797 5728 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v1-10 215822 5260 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v2-10 343784 3514 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v1-10 82394 14559 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v2-10 146973 8138 ns/op 128 B/op 1 allocs/op BenchmarkTokens/threeSkip1Chunk/v1-10 152475 7878 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1Chunk/v2-10 251373 4776 ns/op 128 B/op 1 allocs/op PASS ok github.com/grafana/loki/pkg/storage/bloom/v1 10.206s ```	2 years ago
Angus Dippenaar	258a8b3108	Update nix configuration (#8452 ) Fix nix configuration (mainly failing test), and break up various binaries (ie. logcli, promtail, loki, etc.) into their own pakcages. --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Poyzan	d22c1fd39d	[bloom-compactor] Add configs to enable compactor per tenant (#11235 ) What this PR does / why we need it: We want to control bloom compaction per tenant basis. Adding configs to enable/disable bloom compactor. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Poyzan	3f0f8fa5f5	[bloom-compactor] Move meta.json creation at the end of compaction cycle (#11234 ) What this PR does / why we need it: This is a follow up from https://github.com/grafana/loki/pull/11115 Instead of a creating a meta file per bloom creation, create a meta file per compaction cycle. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer:	2 years ago
Karsten Jeschkies	c6f421233b	Increase test coverage for AST serialization to >94%. (#11230 ) What this PR does / why we need it: This is a follow up to https://github.com/grafana/loki/pull/11123 and fixes a few bugs discovered by increasing the test coverage. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	3a7b5d246b	Serialize query AST to JSON. (#11123 ) What this PR does / why we need it: This introduces the visitor pattern to serialize the LogQL AST to JSON. We've chose this pattern because it will be more flexible one the AST is encoded into Protobuf and an actual queryplan. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Ashwanth	32e9ee8209	chore: follow-up to #11151 and #11025 (#11225 ) What this PR does / why we need it: - remove usage of `enforce_metric_name` in ksonnet - https://github.com/grafana/loki/pull/11151 - make `cortex_ruler_client_request_duration_seconds` prefix configurable - updates upgrade guide to remove reference to metrics that loki does not export Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	117f108642	Return error if span context was not found (#11207 ) What this PR does / why we need it: This is a follow up to https://github.com/grafana/loki/pull/10956. `lokigrpc.GetParentSpanForRequest` would ignore `opentracing.ErrSpanContextNotFound`. However, that should be up to the caller to decide whether this error should be ignored or not. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Owen Diehl	265018b729	populate empty resonse for IndexStatsResponse and VolumeResponse (#11209 ) Fixes a panic when the underlying response struct is accessed but not populated after an out-of-bounds request is short-circuited via `NewEmptyResponse` [here](https://github.com/grafana/loki/blob/main/pkg/querier/queryrange/limits.go#L155-L164). This PR ensures the embedded types are not nil in accordance with the other variants in this function.	2 years ago
Christian Haudum	e5b006ce0a	Enable/disable bloom gateway per tenant (#11203 ) This commits adds a per-tenant setting to enable/disable the use of the bloom gateway component for filtering chunks. As an operator, you want to control what tenant can use the filtering of chunks based on search string, because the creation of the bloom filters requires a significant amount of resources. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

1 2 3 4 5 ...

2438 Commits (e590963471c6dc75edbe0151e61ba3ecd7b8f673)