apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Vladyslav Diachenko	a5aa8b315d	[bloom-compactor] downloading chunks in batches (#11649 ) What this PR does / why we need it: Added chunks batches iterator to download chunks in batches instead of downloading all of them at once. Otherwise, when the stream contains a lot of chunks, it can lead to OOM. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Poyzan	44b817451a	[bloom-compactor] Clean up work to create blooms (#11591 ) What this PR does / why we need it: This PR is the result of debugging sprint on bloom-compactor on a dev cell. With these changes, compactor now can create and upload blooms and relevant meta files from scratch. - Queries tsdb correctly with empty matchers to get all series - Seeks to the beginning of the bloom data file before uploading - Adds uuid strings to local file name, as checksum is not available at the time of directory creation - Creates meta object with all fields populated - Additional logging edit: confirmed that it runs as expected in the dev cell. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Poyzan	cc450dcaf9	Add labels to bloom tokenizer metrics (#11542 ) What this PR does / why we need it: Bloom-compactor panics for duplicate metrics registration in bloom-tokenizers. This PR adds labels to bloom-tokenizer metrics, adopting same solution in https://github.com/grafana/loki/pull/11536	1 year ago
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492 ) In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Poyzan	e56e4c92bc	Run compaction over existing bloom blocks (#11486 ) What this PR does / why we need it: Upon changes to TSDB indexes, the existing blooms needs to be amended with newly added chunks, or new series should be added to blocks. This PR compares the existing meta information with the most up to date TSDB information in the job. In case of no changes to indexes, compaction is skipped; if there's any block needs amending, it does that and merges all blocks in fp range; if there's no previous compaction, it compacts from scratch. Special notes for your reviewer: This logic does not support cutting blocks upon topology changes to bloom-compactors. It can create blocks with series outside of the fp range of the compactor. Cutting blocks will be addressed in a follow-up PR.	1 year ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	9538499e60	Add error return to PopulateSeriesWithBloom (#11359 ) What this PR does / why we need it: Adds an error return value to PopulateSeriesWithBloom Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	ceda046229	Bloom tokenization performance tweaks (#11338 ) What this PR does / why we need it: The main work of this PR is to improve the performance of the reassemble() function. By changing the assignment of the 'cur' variable to not do addition, and incrementing the 'pos' variable separately, the benchmarks improved as follows: BenchmarkTokens/threeSkip1Chunk/v2-10 88864 13487 ns/op BenchmarkTokens/threeSkip1ChunkAt2/v2At2-10 98607 12256 ns/op (This was validated with multiple runs, on multiple size inputs) The remaining changes in the PR are just to add some benchmarking for PopulateSeriesWithBloom, in case we need to do other comparisons for future improvements.	2 years ago
Paul Rogers	df1a3ce79a	Initial pass at metrics for bloom tokenizer (#11329 ) What this PR does / why we need it: Removes the stubbed out metrics from the Bloom Tokenizer, and puts in metrics that were being captured and deemed useful during initial POC testing. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 ) This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Vladyslav Diachenko	75cfe59596	bloom blocks downloading queue (#11201 ) implemented bloom blocks downloading queue to control the concurrency of downloading the blocks from the storage Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	c8093e1ee3	Revert "Compression of bloom blocks (#11267 )" (#11307 ) This reverts commit `af177034` (https://github.com/grafana/loki/pull/11267) because the compression/uncompression must be done outside of the client context: https://raintank-corp.slack.com/archives/C056KUQFBFT/p1700494151990649 Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290 ) What this PR does / why we need it: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago
Paul Rogers	af177034ed	Compression of bloom blocks (#11267 ) What this PR does / why we need it: Compress bloom blocks when writing to object storage, and uncompress when reading from object storage. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Owen Diehl	6e93d150d8	utilities for reducing ngram allocations in token iterator construction (#11276 ) Previously, the `calculatePrefix` function was used to generate the token buffer used, but this wouldn't necessarily have enough space to append the ngram afterwards without reallocating+copying the token buffer. This PR ensures we allocate enough space for both prefix and ngram in the token buffer.	2 years ago
Paul Rogers	5f4a719bb6	tokenizer v1 cleanup (#11272 ) What this PR does / why we need it: Removes all usage of v1 tokenizers, renames v2 to v1 since we never released this in a production way. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Owen Diehl	58eaad9947	boundscheck & partitioning fingerprints between blocks (#11237 ) Adds a few utilities for comparing fingerprints to blocks that cover a specific fingerprint range. Will likely need to be refactored with more comprehensive types, but the logic still applies.	2 years ago
Owen Diehl	6db9d778a1	inverts the logic when testing a block against a list of chunks (#11248 ) returns the list of chunks in the bloom which failed the test so we can merge these results across blocks	2 years ago
Owen Diehl	15d98fa7ca	iterable tokenizer w/ comparison benching (#11255 ) Rebuilds tokenizers based on an iterable approach. This ends up being significantly faster -- benchmarks below. This PR doesn't remove the old tokenizers, just adds the new ones. Revamping tokenizer consumers & removing the old variants can be done in a separate PR. Roughly, raw tokenizers complete in 66% of the time and chunkID prefixed tokenizers in ~55-60% of the time. ``` pkg: github.com/grafana/loki/pkg/storage/bloom/v1 BenchmarkTokens/three/v1-10 134811 8648 ns/op 0 B/op 0 allocs/op BenchmarkTokens/three/v2-10 179797 5728 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v1-10 215822 5260 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v2-10 343784 3514 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v1-10 82394 14559 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v2-10 146973 8138 ns/op 128 B/op 1 allocs/op BenchmarkTokens/threeSkip1Chunk/v1-10 152475 7878 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1Chunk/v2-10 251373 4776 ns/op 128 B/op 1 allocs/op PASS ok github.com/grafana/loki/pkg/storage/bloom/v1 10.206s ```	2 years ago
Paul Rogers	c4ed0d0b7f	Tokenizer tests and TokenizeLine updates (#11133 ) What this PR does / why we need it: The thrust of this PR is to ensure we have tests for each major function of the Bloom Tokenizer. In addition, there was some cleanup, in that constants are used to set some common parameters. Lastly, the TokenizeLine() call was updated to correctly tokenize a line when a "skip tokenizer" is utilized. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Poyzan	cc6f0eca98	Return checksum from BuildFrom function (#11178 ) What this PR does / why we need it: Follow up from https://github.com/grafana/loki/pull/11115 We want to know the checksum of the bloom blocks before uploading them to storage. Instead of opening the file and reading it, this PR returns the checksum from where it's created.	2 years ago
Poyzan	fa378ac0ec	Add first step for creating a single bloom from series (#11115 ) What this PR does / why we need it: This is the first step to create a bloom from a series. The logic to get tenants and tables is a temporary hack and will be replaced by the ring logic in https://github.com/grafana/loki/pull/11154 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: It does not handle appending multiple blooms into blocks. It does not handle creating and uploading of meta files. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118 ) What this PR does / why we need it: This just updates the Loki build imaage changed in #11114. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Owen Diehl	6bfd2baec4	Bloom/fuse queries (#11088 ) Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons: * A "query" contains one of more series, with each series needing to test one or more chunks for some search string(s). * Blocks are sorted by fingerprint, which are the result of a hash function. This means series are ~uniformly distributed across the hash keyspace (`uint64` in our case). Because of this, a query for some number of series are likely to be spread out across the block itself. * Blocks are organized into individually compressed "pages". In essence, querying them requires Seeking to the correct offset, decompressing the page, then seeking within the page to the actual offset for a particular series. * Since the series for any query are likely spread out across the block, performing `n` queries decompresses each page `n` times. * By multiplexing many queries into a single pass iteration of the block, we amortize the cost of loading & decompressing Misc: Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.	2 years ago
Poyzan	df9c6fcac4	Add series population to tokenizer (#11070 ) What this PR does / why we need it: I started to use tokenizer module for bloom creation and populating the SBF without the series with chunk ref is weird, as we populate only half of a data structure. I modified it to populate entire `SeriesWithBloom` as it has all the information. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 years ago
Paul Rogers	c7eb757ed8	Initial loki bloom creation library (#10957 ) What this PR does / why we need it: Optimization and benchmarking of various LRUs that sit in front of the bloom filters. Initial creation of a library that can be used to create blooms from chunks Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	0a7737e7c7	adds a merge-builder for building blocks from other blocks+storage (#10974 ) Some minor refactoring + a (simple) `MergeBuilder` struct to build bloom blocks from other blocks and storage, depending on what's already been indexed into the blocks.	2 years ago
Poyzan	cbc0df3241	Add NewBlockOptions constructor (#10955 ) Adds a `NewBlockOptions()` constructor plus a quick fix for a missed err check.	2 years ago
Salva Corts	89829065e7	Use lazy bloom filter (#10896 ) What this PR does / why we need it: When querying bloom filters, use lazy decoders from https://github.com/owen-d/BoomFilters/pull/1 so we don't make a copy of bloom filters.	2 years ago
Owen Diehl	ad0f7ab845	Blooms/merge-iters (#10923 ) Adds a bunch of bloom filterutilities in preparation for merging & extending bloom blocks with new chunks/series	2 years ago
Poyzan	0832256d7b	Add basic structure of bloom compactor (#10748 ) This pull request adds the basic structure for the new bloom compactor component. - Adds new `bloom-compactor` target that runs with with multiple instances joined by a ring - Adds boilerplate functions to index blooms and compact blocks The main goal of this PR is to provide a basic functionality on which future smaller PRs can build upon. Since the code path is completely separate in a new component, it is not used anywhere at the moment.	2 years ago
Owen Diehl	f861068df4	moves boomfilters lib into loki (#10898 ) In an effort to speed up development, this moves a subset of [my fork](https://github.com/owen-d/BoomFilters) of the https://github.com/tylertreat/BoomFilters library into loki itself. I've done my best to preserve original license headers/etc.	2 years ago
Owen Diehl	b52d836867	Blooms/chunk check (#10886 ) [WIP] Adds ability to return the list of chunks needed to be downloaded by a query based on bloom membership results	2 years ago
Owen Diehl	7f02ef3fa7	helpers for tar+gz over bloom blocks (#10849 ) Simple start, but helpful for building & extracting tar+gz archives over bloom block directories.	2 years ago
Owen Diehl	c405db4317	Format + Library for building bloom filter "blocks" from chunks (#10780 ) This PR introduces both a binary format for bloom filter "blocks" and a library for building/querying them. Each "block" consists of two files: 1) `Index`. This is a list of individually compressed pages of stream+chunk information sorted in ascending fingerprint order. Each series lists both the chunks indexed in the bloom filters as well as the relevant offsets for said series in the bloom file. 2) `Blooms`. This is a list of individually compressed pages of bloom filters for each series, again sorted in ascending fingerprint order. There is still more work to do, but I'm opening this PR as an initial pass.	2 years ago

35 Commits (a5aa8b315da22ef94763f9e048b1a77840e724fd)