160 Commits (39c47e5b5a188b763b32e3e92939ccfce01d2a86)

Author	SHA1	Message	Date
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	f3f01845ad	chore(blooms): Add setting to fetch bloom blocks async or synchronously (#14862) ... In case bloom blocks are small, they could be downloaded synchronously to avoid missing blocks on cold start. This is an undocumented setting. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	03f7eab9c0	chore(blooms): Add number of skipped bloom blocks to stats-report log line (#14771) ... We already log the number of processed blocks, so we also want to know the number of skipped blocks, because they have not been available. This will give better insights on a per-query basis, additionally to the metric of overall skipped unavailable blocks `loki_bloom_gateway_blocks_not_available_total`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	8eca826795	chore(storage/bloom): support simplifiable regexp matchers (#14622) ... This adds support for basic regexps which can be simplified into a sequence of OR matchers, such as: * `key=~"value" becomes key="value" * `key=~"value1|value2" becomes key="value1" or key="value2". * `key=~".+" checks for the presence of key. This is currently the only way to check if a key exists. Only the cases above are "officially" supported. However, we technically support basic concatenations and character classes due to how regexp/syntax parses and simplifies expressions such as `value1|value2` into `value[12]`. To prevent unbounded cardinality, we limit regexp expansion to 25 matchers; otherwise a regexp like `value[0-9][0-9][0-9][0-9]` would expand into 10,000 matchers (too many!). Closes grafana/loki-private#1106. Co-authored-by: J Stickler <julie.stickler@grafana.com>	7 months ago
Christian Haudum	0760095a3e	chore: Add bool to metrics.go log line whether query used bloom filters (#14645) ... Better observability of which queries use bloom filters for chunk filtering. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Salva Corts	d1668f6a11	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	7 months ago
Christian Haudum	601f549656	fix(blooms): Check length of tasks before accessing first element in slice (#14634) ... This PR adds a length check for the partitioned series before accessing the first element of the resulting slice. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
jinjiadu	99c423a1f5	chore: fix some function names (#14628) ... Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	7 months ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539)	7 months ago
Christian Haudum	f568dda136	chore: Improve logging of jumphash server selector (#14306) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Christian Haudum	fc8250bfed	chore: Add new field to "stats-report" log line in bloom gateway (#14446) ... This new field reports how many blocks have been processed in total in the multiplexed request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Christian Haudum	91c7d344fe	chore: Make metric for dequeued tasks in bloom-gateway a Histogram (#14413) ... This change allows to observe the distribution of how many tasks are dequeued at once over time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	46f61fd5ad	chore: Remove ring client pool from JumpHashClientPool (#14367) ... The ring client pool has functionality, such as periodic health checking of the servers and removing of stale clients, which is not needed in the JumpHashClientPool. The removal of stale clients was problematic, because it compared the original server list (which is a list of CNAME records from the DNS lookup) with the client cache, which uses the IP of the server as key. Therefore it removed all cached clients on every check interval. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	ebe0bb875e	chore: replace remaining references to ExtractTestableLineFilters (#14135)	8 months ago
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038) ... A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Salva Corts	784e7d562f	fix: Keep blocks referenced by newer metas (#13614)	10 months ago
Christian Haudum	39a52974af	chore: Remove excessive logging from bloom gateway (#13603) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	3ac231728e	fix(blooms): Suppress error from resolving server addresses for blocks (#13385) ... Instead of returning the error, `FilterChunks` returns the full, unfiltered list of chunks. This makes sure that queries do not fail if no bloom gateway instances are available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	f661593cae	refactor(blooms): more easily testable day partitioning (#13331)	11 months ago
Christian Haudum	3b0502ddd3	chore(blooms): Various minor code cleanups (#13332) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	66f97b2aec	fix(blooms): Fix panic in bloom gateway (#13303) ... The slice of block queriers can contain `nil` values, which causes nip pointer dereference when calling `Close()` on them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Bryan Boreham	2b19dac82a	fix: errors reported by the race detector (#13174) ... Fixes #8586 Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Christian Haudum	d4fcef5229	chore(blooms): Some boom gateway cleanups (#13165) ... * Cleanup logic for processing multiple days in single `FilterChunkRefs` request in the bloom gateway. The handler already returned an error if the requested chunk refs spanned across multiple days, but the logic for processing multiple days/tasks was still in place. This commit simplifies the logic to only process a single task per request. * Remove unused package global variable * Remove bloomgateway package comment * Remove duplicate imports * Avoid conversion from *logproto.ShortRef to v1.ChunkRef and istead cast the value of the pointer * Simplify `getFromThrough()` function Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	d6374bc2ce	feat(blooms): Add counter metric for blocks that are not available at query time (#12968) ... When filtering chunks on the bloom gateway, bloom block may not be available and they will be downloaded asynchronously in the background. This new metric `loki_bloom_gateway_blocks_not_available_total` counts the blocks that are not available at query time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	a1b1eeb095	feat(bloom): Skip attempts to filter chunks for which blooms have not been built (#12961) ... Bloom filters are built from `today - MaxTableOffset` to `today - MinTableOffset`, this means that blooms are not available for the most recent period, between `now` and at least `today - MinTableOffset`. To avoid resolving chunks and filtering out no-matches, we can skip chunks for this period completely and return them as-is without filtering. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	738c274a58	fix(blooms): Fix `partitionSeriesByDay` function (#12900) ... The function `partitionSeriesByDay` could yield empty when chunks were outside of the requested time range. This would result in requests for a time range that does not contain any chunks for a series to filter. These requests would return errors, because a subsequent `partitionSeriesByDay` for would remove that day resulting in no task. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889)	1 year ago
Owen Diehl	5a643c7115	chore: reduces span footprint + double recording (#12864) ... Trying to remove some cruft from traces.	1 year ago
Owen Diehl	4c9b22f110	feat(blooms): ignore individual bloom-gw failures (#12863)	1 year ago
Christian Haudum	48bbf983d0	chore(blooms): Remove ID field from task struct (#12851) ... We've seen a few cases where creating the ULID failed for unknown reasons, and the ID is not really used. It was only useful early on in the development for debugging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807) ... This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Michel Hollands	33e82ec133	fix: remove unused parameter causing lint error (#12801) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Christian Haudum	3bf2d1fea0	fix(blooms): Deduplicate filtered series and chunks (#12791) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	4cc3a73172	test: Add mutex and setter/getter for addresses in test (#12793)	1 year ago
Christian Haudum	ecefb49508	fix(blooms): Fix a regression introduced with #12774 (#12776) ... This PR fixes a regression introduced with #12774 where series where wrongly identified as "skipped" even though there were part of blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c36b1142c7	fix(blooms): Correctly return unfiltered chunks for series that are not mapped to any block (#12774) ... This PR fixes a conceptual mistake in the code of resolving blocks on the index gateways. Currently, a series does not resolve to any block is discarded instead of being kept for the response. This change adds the chunks of the skipped series to the bloom querier response. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	ae180d6e07	fix(nix): lambda-promtail vendor hash (#12763)	1 year ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720) ... This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	1 year ago
Owen Diehl	712a2b6967	fix(blooms): fixup bloom testing (#12494)	1 year ago