72 Commits (258f8267477f9c4d1d950a6a0887c7a902af05ff)

Author	SHA1	Message	Date
Christian Haudum	5846ea2ea0	fix(blooms): Match series to newest block only (#15481) ... **What this PR does / why we need it**: While running bloom filters in production we noticed some Loki clusters that showed a very high percentage of missing chunks when querying blooms, thus resulting in lower filter rate. The reason is that old, superseded blocks are still considered up-to-date, because they cover a keyspace that is not covered by newer blocks with smaller keyspaces (due to larger individual series). ``` | series fingerprint keyspace ------------+---------------------------------------------------------------- | o o o o o o o o o o o o o ------------+---------------------------------------------------------------- iteration 1 | 111111111111111111111111111111111111111111111111111111111 iteration 2 | 22222222222222 3333333333333333 444444 iteration 3 | 5555555 6666666 77777777 888888888 9999999999 ... up-to-date | 555555522266666661111177777777333388888888811119999999999 ------------+---------------------------------------------------------------- | x ``` The chart shows the different blocks marked with the numbers 1 to 9 for a subset of the full series fingerprint keyspace. The blocks are generated in multiple successive bloom building iterations. The first block covers a larger keyspace (more series), because the individual blooms in the blocks are smaller in the beginning of the day. Later, the blooms get larger and therefore the block fingerprint ranges gets smaller. However, since we are dealing with fingerprint ranges, not individual fingerprints, the newer blocks cause "gaps" in the range of the previously larger keyspace. In the case above, every block except block 4, are considered up-to-date, since each of them covers a keyspace that is otherwise not covered. When resolving blocks for a series at query time, we consider looking at all up-to-date blocks, which are referenced by the meta files. The series `x` in the chart shows, that it is within the range of 3 up-to-date blocks: 1, 2, 5. However, only the newest block (5) may contain the requested series. This PR changes the block resolver on the index-gateway to only match the newest block to a series, based on the timestamp of the TSDB from with the blocks were generated. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Ned Andreev	aa7b849e75	fix: data race in bloombuild tests (#15360)	5 months ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	6 months ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Salva Corts	6c0f6089f8	chore: use loki comntant for metrics namespace (#15034)	6 months ago
Salva Corts	79cccfd50b	feat(blooms): Add task timining and sizing metrics (#15032)	6 months ago
Salva Corts	f92dde0e41	feat(blooms): Apply task timeout in bloom builder (#14988)	6 months ago
Salva Corts	a8f7851161	fix(blooms): Fix tenants slice on loadTenantTables (#14900)	6 months ago
Salva Corts	fd9d33241d	fix: Improve docs for min and max table offsets (#14890) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Salva Corts	b646861e77	feat(blooms): disk-backed queue for the bloom-planner (#14874) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Salva Corts	bfc289028e	fix(blooms): Copy chunks from ForSeries (#14863)	6 months ago
Salva Corts	94a8572718	refactor(blooms): Extract planner queue into pkg (#14861)	6 months ago
Salva Corts	ab2721d7ee	fix(blooms): Fix strategy logger and add task test (#14857)	6 months ago
Salva Corts	4bfa3807cf	revert: "perf(bloom): Compute chunkrefs for series right before sending task to builder" (#14839)	6 months ago
George Robinson	f1ff3d252b	chore(blooms): fix lint (#14823)	6 months ago
Salva Corts	66e6b1c1df	perf(bloom): Compute chunkrefs for series right before sending task to builder (#14808)	6 months ago
Salva Corts	766938525f	fix(blooms): Do not restart builders when planner disconnects (#14783)	7 months ago
Salva Corts	76d35cc97f	fix(blooms): Skip multi-tenant TSDBs during bloom planning (#14770)	7 months ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577)	7 months ago
Salva Corts	673ede16a5	feat: New bloom planning using chunk size TSDB stats (#14547)	7 months ago
Ned Andreev	99ef9009e5	fix: lint errors (#14574)	7 months ago
Salva Corts	6aa979d29e	refactor: Extract common functionality to test utils pkg (#14549)	7 months ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539)	7 months ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Robert Fratto	99f12e50d3	test(bloombuild/planner/strategies): use require.ElementsMatch in test (#14521)	7 months ago
Salva Corts	ab5e6eaaee	feat: Extract task computing into a strategy interface (#13690)	7 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
dropbigfish	5f78784d4f	chore: fix some function names (#14020) ... Signed-off-by: dropbigfish <fillfish@foxmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	8 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Christian Haudum	bf60455c8e	feat(blooms): Add bloom planner and bloom builder to `backend` target (#13997) ... Previously, the bloom compactor component was part of the `backend` target in the Simple Scalable Deployment (SSD) mode. However, the bloom compactor was removed (https://github.com/grafana/loki/pull/13969) in favour of planner and builder, and therefore also removed from the backend target. This PR adds the planner and builder components to the backend target so it can continue building blooms if enabled. The planner needs to be run as singleton, therefore there must only be one instance that creates tasks for the builders, even if multiple replicas of the backend target are deployed. This is achieved by leader election through the already existing index gateway ring in the backend target. The planner leader is determined by the ownership of the leader key. Builders connect to the planner leader to pull tasks. ---- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Salva Corts	37e33d41b4	fix: flaky test blockPlansForGaps (#13743)	9 months ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Salva Corts	4cbe2a6a85	fix: Use elements match in gapsBetweenTSDBsAndMetas test (#13722)	10 months ago
Salva Corts	450bbce938	fix(blooms): Ship chunkrefs in task payload (#13677)	10 months ago
Salva Corts	784e7d562f	fix: Keep blocks referenced by newer metas (#13614)	10 months ago
Salva Corts	0029d46c23	feat: Track when builders are doing work with a gauge (#13570)	10 months ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	10 months ago
Salva Corts	e506995e59	fix(blooms): Remove backoff from notify planner (#13506)	10 months ago
Salva Corts	2cc901ad69	refactor(blooms): Apply retention in planner (#13484)	10 months ago
Salva Corts	2affa4862e	refactor(blooms): Improve task progress logging in builder (#13394)	11 months ago
Salva Corts	11e19763d0	fix(blooms): Delete outdated metas during planning (#13363)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Salva Corts	0602b90072	refactor(blooms): Better task naming (#13361)	11 months ago