95 Commits (bd20171975e913e429048a0a30328811fc4c8a87)

Author	SHA1	Message	Date
Christian Haudum	64215e1849	fix(blooms): Cleanup temp blockdir in bloom compactor (#13622) ... A follow up to #13522 This PR fixes the missing cleanup function call to remove temporary blocks from the filesystem. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	2 years ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	2 years ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	cb1f5d9fca	fix: Defer closing blocks iter after checking error from loadWorkForGap (#12934)	2 years ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	2 years ago
Owen Diehl	fb9b0e80a7	chore(blooms): adds more instrumentation to block building (#12779)	2 years ago
Salva Corts	0ee2a6126a	fix(blooms): Fix findGaps when ownership goes to MaxUInt64 and that is covered by existing meta (#12558)	2 years ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	2 years ago
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Christian Haudum	71602eb10f	chore(blooms): Clean up bloom component configuration (#12387) ... * Remove unused setting and move download parallelism configuration one level up. * Update description of bloom_shipper config block Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	86c768c3cc	feat: Blooms retention (#12258)	2 years ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	eb03974086	feat(blooms): Make bloom block compression configurable (#12293) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	63c88489d0	refactor: Add RF and Tokens to Loki ring Cfg and allow overwriting docs. (#12142) ... Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Salva Corts	4b28f8221b	fix: Read all series in NewTSDBSeriesIter and close index file (#12211)	2 years ago
Owen Diehl	8a8756aa7a	chore(blooms): honors bloom_compactor_enabled for tenants (#12190)	2 years ago
Christian Haudum	f6cfff5356	refactor(bloomstore): Introduce fetch option for blocks fetcher (#12160) ... Requests to the bloom gateway should not wait for blocks to be downloaded into cache first, but should operate on readily available blocks. This PR introduces fetch options for the bloom store that specify how the store should behave when requesting blocks from block refs. `WithIgnoreMissing`: Ignore errors from blocks that could not be found in object storage, and instead return a `nil` value in the response. `WithFetchAsync`: Return only the blocks that are available locally in cache and dispatch downloading of blocks to the blocks downloader, that operates in the background, and make them available for subsequent requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	49f8f7b3ee	refactor: Chunkref as alias to logproto.ShortRef (#12159)	2 years ago
Owen Diehl	30a240b470	fix(blooms): filter out nil blocks (#12150)	2 years ago
Owen Diehl	102e3ea07f	chore(blooms): use the right iter reference in blockLoadingIter.loadNext (#12146)	2 years ago
Salva Corts	f7710ad7aa	feat: (blooms) Configurable number of tokens (#12125)	2 years ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101)	2 years ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840)	2 years ago
Christian Haudum	2f54f8d5be	chore(blooms): Implement BloomStore as a service (#12044) ... The bloomstore service is instantiated once at startup and used as dependency for both the bloom compactor and the bloom gateway. This will prevent instantiating the store in different ways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	1f56da206d	refactor: Simplify fingerprint partitioning in bloom gateway client (#12018) ... Use the recently introduced `tokenRangesForInstance()` function to determine the the ranges for each replication set. For this purpose, the function moves into the `bloomutils` packages. This PR eliminates the structs `addrsWithBounds` and `instanceWithFingerprints` (which had somewhat similar usage) and unifies them into `rsWithRanges` that holds both the fingerprint ranges and the assigned GroupedChunkRefs. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	e835b6192f	refactor(blooms): refactors `blockLoadingIter`, exposing more error cases + additional metrics (#12015)	2 years ago
Owen Diehl	09aa0f80b5	Blooms/contiguous bounds union (#12009)	2 years ago
Owen Diehl	166229818d	[Blooms] Consistent hashing via tokens for bloomcompactor (#12002)	2 years ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979)	2 years ago
Salva Corts	ef40136715	Use TSDB index prefix on blooms directory (#11977)	2 years ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974)	2 years ago
Owen Diehl	f158b5bc3e	correctly set block iter when no overlapping blocks are found (#11973)	2 years ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966)	2 years ago
Christian Haudum	5967ceeb43	Replace min/max token with `TokenRange` in bloom ring utilities (#11960) ... This PR replaces min/max token fields with a `TokenRange` field that uses the `Range[uint32]` type. The `Range[uint32]` uses similar semantics as the `FingerprintBounds` we use for fingerprint ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	6434df7345	Bloom compactor shuffle-sharding (#11947)	2 years ago
Owen Diehl	bd12e16339	[Blooms] Use correct table address function (#11955)	2 years ago
Owen Diehl	3c06b360ec	meta & removal support for bloom compaction (#11941) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917) ... Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	eb8464a641	Bloom compactor: Load blocks lazily in batches (#11919) ... To avoid loading possibly lots of blocks upfront, this PR introduces lazy loading of blocks in batches using an iterator that loads blocks on demand. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	0bb2574040	makes batchedLoader generic + removes unnecessary interfaces & adapters (#11924) ... While reviewing https://github.com/grafana/loki/pull/11919, I figured it'd be nice to make `batchedLoader` generic so we can reuse it's logic. This let me test it easier and remove a lot of now-unnecessary adapter code (interfaces, types)	2 years ago
Owen Diehl	7560067439	Bloom/running (#11918) ... Includes a few small fixes in order to get the compactor running, including * Wires up i/o from `v1` pkg and the `bloomshipper` pkg in order to archive+write blocks remotely after building * config error handling * fix an error iterating chunks * handles gzipping when loading tsdbs from storage * bypasses internal cache in `BloomTSDBStore`'s internal `indexshipper/storage.Client` to get up to date results, which is easier for prototyping * `ClosableReadSeekerAdapter` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	7a95cb8663	Configurable bloom tokenizer and block settings (#11889) ... **What this PR does / why we need it**: This PR makes the max block size configurable and wires up the already existing nGramLen and nGramSkip settings. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago