118 Commits (bbbd82bc73322d662ba81efeda3884efcdc09708)

Author	SHA1	Message	Date
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	55c64991db	feat(blooms): Add series & chunks per block metrics (#13721)	2 years ago
Cyril Tovena	04613b492f	feat: Instrument metastore GRPC calls (#13598)	2 years ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	aeb23bb7fc	fix: Fix log line for fingerprint not found (#13555)	2 years ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	2 years ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	2 years ago
Owen Diehl	bfa69556af	fix(blooms): skip empty blooms on reads (#13500) ... Read path safeguard to prevent filtering chunks associated with empty blooms while we work on finding the underlying cause of empty blooms in compaction.	2 years ago
Christian Haudum	652ad2455c	perf(blooms): Avoid tiny string allocations for insert cache (#13487) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	04bc3a423c	fix(blooms): ensure tokenizer cache is reset between series (#13370)	2 years ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	2 years ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	2 years ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	2 years ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	2 years ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
洪阿南	670cd89aa8	fix: Fix incorrect sorting of chunks in bloom-filtered response since `ChunkRef.Cmp` method is used in reverse (#12999) ... The `v1.ChunkRef.Cmp()` method is used in reverse, and this can lead to chunks in bloom filtered result be sorted in descending but not ascending.	2 years ago
Christian Haudum	75ccf2160b	feat(blooms): Separate page buffer pools for series pages and bloom pages (#12992) ... Series pages are much smaller than bloom pages and therefore can make use of a separate buffer pool with different buckets. The second commit fixes a possible panic. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
MarkDaveny	3a46d3717d	style: fix function names (#12817) ... Signed-off-by: MarkDaveny <peicuiping@aliyun.com>	2 years ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889)	2 years ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807) ... This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	1665e853a0	fix(blooms): dont break iterator conventions (#12808) ... Followup to #12806 which exposes skipped pages more explicitly than as an error. * refactors skip logic for bloom pages that are too large * s/Seek/LoadOffset/ for LazyBloomIter * removes unused code	2 years ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	2 years ago
Salva Corts	76ba24e3d8	fix(blooms): Reset error on LazyBloomIter.Seek (#12806)	2 years ago
Owen Diehl	da8ee7e7f2	chore(blooms): Instrument source bytes added to blooms directly (#12782)	2 years ago
Owen Diehl	fb9b0e80a7	chore(blooms): adds more instrumentation to block building (#12779)	2 years ago
洪阿南	0d1ebebd3a	fix: Add a missing `continue` in fuse which may cause incorrect bloom test result (#12650)	2 years ago
Owen Diehl	712a2b6967	fix(blooms): fixup bloom testing (#12494)	2 years ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Anton Kolesnikov	36c703dae8	feat: add pattern match line filter (#12398)	2 years ago
Anton Kolesnikov	cc941fe42a	chore: refactor line filter MatchType (#12388)	2 years ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	30ac88bc26	chore(blooms): increase blockpool by factor-of-2 (#12363)	2 years ago
Salva Corts	5c6a031c08	fix(blooms): Not filters should always match (#12358)	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Salva Corts	e4a5733ada	fix(blooms): Extract only line filters before line format expressions (#12334)	2 years ago
Christian Haudum	a54f2dba7a	perf(blooms): Optimize memory usage for blocks that have no compression (#12316) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	0f43bf6958	fix(blooms): fix typo in pool decl (#12302)	2 years ago
Owen Diehl	fe1b357cc2	fix(blooms): leaky ref (#12300)	2 years ago
Christian Haudum	eb03974086	feat(blooms): Make bloom block compression configurable (#12293) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	2edbe1280f	fix(blooms): Correctly skip block page in case it exceeds the max page size for querying (#12297) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4aec20f36b	perf(blooms): Recycle bloom page buffers (#12256) ... Every time when reading a bloom page into memory, a new byte slice with the size of the decoded page was allocated. This PR changes bloom page decoder to use a byte slice from the `BlockPool`. The caller (`LazyBloomIter` of the block querier) is responsible for closing the page whenever a new one is loaded, so the byte slice is put back to its pool. Additionally, this PR changes the max bucket of the `BlockPool` byte slice pool to 64MB so that pages bigger than 16MB are correctly put back to the pool, since [slices exceeding the capacity of the largest bucket are not put back](https://github.com/grafana/loki/blob/main/vendor/github.com/prometheus/prometheus/util/pool/pool.go#L74-L87). --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	4b28f8221b	fix: Read all series in NewTSDBSeriesIter and close index file (#12211)	2 years ago
Christian Haudum	a0fce39169	feat(bloom-gw): Make num multiplexing tasks configurable, and other adjustments (#12169) ... * We want to be able to change the amount of dequeued items per iterations to test different configurations. * Ignore block pages that are magnitues bigger than the target page size. * Increase download queue capacity to avoid blocking when enqueueing Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	f4b2c5d69b	fix(blooms): bloom-gw chunk merging improvements (#12162)	2 years ago
Salva Corts	49f8f7b3ee	refactor: Chunkref as alias to logproto.ShortRef (#12159)	2 years ago
Owen Diehl	9a993ca690	chore(blooms): Ordering & Unless iterator (#12154)	2 years ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101)	2 years ago