140 Commits (8eca826795d20a9d089fd1d7ec60ba738ab4b863)

Author	SHA1	Message	Date
Robert Fratto	8eca826795	chore(storage/bloom): support simplifiable regexp matchers (#14622) ... This adds support for basic regexps which can be simplified into a sequence of OR matchers, such as: * `key=~"value" becomes key="value" * `key=~"value1|value2" becomes key="value1" or key="value2". * `key=~".+" checks for the presence of key. This is currently the only way to check if a key exists. Only the cases above are "officially" supported. However, we technically support basic concatenations and character classes due to how regexp/syntax parses and simplifies expressions such as `value1|value2` into `value[12]`. To prevent unbounded cardinality, we limit regexp expansion to 25 matchers; otherwise a regexp like `value[0-9][0-9][0-9][0-9]` would expand into 10,000 matchers (too many!). Closes grafana/loki-private#1106. Co-authored-by: J Stickler <julie.stickler@grafana.com>	7 months ago
Salva Corts	3af0004cb4	feat(blooms): Only write key and key=value to blooms (#14686)	7 months ago
Salva Corts	d1668f6a11	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	7 months ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577)	7 months ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539)	7 months ago
Robert Fratto	e9efcb57f9	chore(storage/bloom/v1): bind bloom page size to max bloom size (#14372) ... Previously, the bloom page size was hard-coded to 256KB. However, 256KB is much smaller than the normal bloom size, leading to nearly every bloom page to be overfilled with a single, massive bloom. Binding the bloom page size to the maximum bloom size allows bloom pages to contain multiple smaller blooms (which are still larger than 256KB) or one maxed out bloom. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	14b468b236	chore: Add `has_labelfilter_before_parser` to metrics.go log line (#14244) ... The value of this field indicates that the query could potentially benefit from bloom filters on structured metadata, if the queried field is a structured metadata field. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Robert Fratto	1b7086632e	chore(storage/bloom/v1): update comments on bloom test (#14168)	8 months ago
Christian Haudum	02b26a85df	chore: Remove outdate TODO file for bloom package (#14158) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6e36041c08	chore(blooms): Remove excessive logging in fused querier (#14152) ... The FusedQuerier emits a large amount of log lines at level WARN if a bloom is empty. Since the introduction of structured metadata blooms, this happens every time a series does not have structured metadata in any of its entries. In the future, an optimisation could be to not write blooms for series at all, if they contain no data. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	39fa9b47e9	chore: Fix linter errors (#14145) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	53cfef3bca	chore(storage/bloom/v1): add ExtractTestableLabelMatchers (#14119) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Paul Rogers	f226b598a4	chore: Final batch of linting cleanup for Go 1.23 and newer golangci (#14120)	9 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038) ... A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Salva Corts	55c64991db	feat(blooms): Add series & chunks per block metrics (#13721)	10 months ago
Cyril Tovena	04613b492f	feat: Instrument metastore GRPC calls (#13598)	10 months ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Salva Corts	aeb23bb7fc	fix: Fix log line for fingerprint not found (#13555)	11 months ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	11 months ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	11 months ago
Owen Diehl	bfa69556af	fix(blooms): skip empty blooms on reads (#13500) ... Read path safeguard to prevent filtering chunks associated with empty blooms while we work on finding the underlying cause of empty blooms in compaction.	11 months ago
Christian Haudum	652ad2455c	perf(blooms): Avoid tiny string allocations for insert cache (#13487) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	04bc3a423c	fix(blooms): ensure tokenizer cache is reset between series (#13370)	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
洪阿南	670cd89aa8	fix: Fix incorrect sorting of chunks in bloom-filtered response since `ChunkRef.Cmp` method is used in reverse (#12999) ... The `v1.ChunkRef.Cmp()` method is used in reverse, and this can lead to chunks in bloom filtered result be sorted in descending but not ascending.	1 year ago
Christian Haudum	75ccf2160b	feat(blooms): Separate page buffer pools for series pages and bloom pages (#12992) ... Series pages are much smaller than bloom pages and therefore can make use of a separate buffer pool with different buckets. The second commit fixes a possible panic. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
MarkDaveny	3a46d3717d	style: fix function names (#12817) ... Signed-off-by: MarkDaveny <peicuiping@aliyun.com>	1 year ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889)	1 year ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807) ... This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1665e853a0	fix(blooms): dont break iterator conventions (#12808) ... Followup to #12806 which exposes skipped pages more explicitly than as an error. * refactors skip logic for bloom pages that are too large * s/Seek/LoadOffset/ for LazyBloomIter * removes unused code	1 year ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	1 year ago
Salva Corts	76ba24e3d8	fix(blooms): Reset error on LazyBloomIter.Seek (#12806)	1 year ago
Owen Diehl	da8ee7e7f2	chore(blooms): Instrument source bytes added to blooms directly (#12782)	1 year ago
Owen Diehl	fb9b0e80a7	chore(blooms): adds more instrumentation to block building (#12779)	1 year ago
洪阿南	0d1ebebd3a	fix: Add a missing `continue` in fuse which may cause incorrect bloom test result (#12650)	1 year ago