apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
yy	0574159216	chore: typos in comments (#20481 )	3 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490 )	9 months ago
Paul Rogers	24c260e3e4	chore(deps): Update dskit (#16782 )	1 year ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572 )	1 year ago
Christian Haudum	5846ea2ea0	fix(blooms): Match series to newest block only (#15481 ) What this PR does / why we need it: While running bloom filters in production we noticed some Loki clusters that showed a very high percentage of missing chunks when querying blooms, thus resulting in lower filter rate. The reason is that old, superseded blocks are still considered up-to-date, because they cover a keyspace that is not covered by newer blocks with smaller keyspaces (due to larger individual series). ``` \| series fingerprint keyspace ------------+---------------------------------------------------------------- \| o o o o o o o o o o o o o ------------+---------------------------------------------------------------- iteration 1 \| 111111111111111111111111111111111111111111111111111111111 iteration 2 \| 22222222222222 3333333333333333 444444 iteration 3 \| 5555555 6666666 77777777 888888888 9999999999 ... up-to-date \| 555555522266666661111177777777333388888888811119999999999 ------------+---------------------------------------------------------------- \| x ``` The chart shows the different blocks marked with the numbers 1 to 9 for a subset of the full series fingerprint keyspace. The blocks are generated in multiple successive bloom building iterations. The first block covers a larger keyspace (more series), because the individual blooms in the blocks are smaller in the beginning of the day. Later, the blooms get larger and therefore the block fingerprint ranges gets smaller. However, since we are dealing with fingerprint ranges, not individual fingerprints, the newer blocks cause "gaps" in the range of the previously larger keyspace. In the case above, every block except block 4, are considered up-to-date, since each of them covers a keyspace that is otherwise not covered. When resolving blocks for a series at query time, we consider looking at all up-to-date blocks, which are referenced by the meta files. The series `x` in the chart shows, that it is within the range of 3 up-to-date blocks: 1, 2, 5. However, only the newest block (5) may contain the requested series. This PR changes the block resolver on the index-gateway to only match the newest block to a series, based on the timestamp of the TSDB from with the blocks were generated. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051 ) Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
Salva Corts	b4060154d1	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	6c0f6089f8	chore: use loki comntant for metrics namespace (#15034 )	1 year ago
Salva Corts	f92dde0e41	feat(blooms): Apply task timeout in bloom builder (#14988 )	1 year ago
Salva Corts	766938525f	fix(blooms): Do not restart builders when planner disconnects (#14783 )	1 year ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577 )	2 years ago
Ned Andreev	99ef9009e5	fix: lint errors (#14574 )	2 years ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539 )	2 years ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132 ) Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383 )	2 years ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257 ) This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159 ) Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183 ) This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167 ) Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	2 years ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097 ) The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071 )	2 years ago
Christian Haudum	bf60455c8e	feat(blooms): Add bloom planner and bloom builder to `backend` target (#13997 ) Previously, the bloom compactor component was part of the `backend` target in the Simple Scalable Deployment (SSD) mode. However, the bloom compactor was removed (https://github.com/grafana/loki/pull/13969) in favour of planner and builder, and therefore also removed from the backend target. This PR adds the planner and builder components to the backend target so it can continue building blooms if enabled. The planner needs to be run as singleton, therefore there must only be one instance that creates tasks for the builders, even if multiple replicas of the backend target are deployed. This is achieved by leader election through the already existing index gateway ring in the backend target. The planner leader is determined by the ownership of the leader key. Builders connect to the planner leader to pull tasks. ---- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969 ) What this PR does / why we need it: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.` to `-bloom-build.`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.` were removed. See upgrade docs for further information. Special notes for your reviewer*: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671 ) Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	450bbce938	fix(blooms): Ship chunkrefs in task payload (#13677 )	2 years ago
Salva Corts	0029d46c23	feat: Track when builders are doing work with a gauge (#13570 )	2 years ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522 ) The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537 )	2 years ago
Salva Corts	e506995e59	fix(blooms): Remove backoff from notify planner (#13506 )	2 years ago
Salva Corts	2affa4862e	refactor(blooms): Improve task progress logging in builder (#13394 )	2 years ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273 ) Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	fcb9b283ba	fix(blooms): Remove unused arg (#13343 )	2 years ago
Salva Corts	d0f56eeb0a	fix(blooms): Minor fixes and improvements for testing in dev (#13341 )	2 years ago
Salva Corts	8ebce009f3	refactor(blooms): Implement retry in builder (#13306 )	2 years ago
Salva Corts	0ddf1e6113	refactor(blooms): Bloom building integration test (#13296 )	2 years ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172 ) This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Bryan Boreham	2b19dac82a	fix: errors reported by the race detector (#13174 ) Fixes #8586 Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093 )	2 years ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162 )	2 years ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039 ) This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	6e119aaef5	refactor(blooms): Forward created metas from builder to planner (#13133 )	2 years ago
Salva Corts	fa2c78936f	refactor(blooms): Build new metas and blocks (#13074 )	2 years ago
Salva Corts	ab7af0575d	refactor(blooms): Builder retrieves tasks from planner (#13046 )	2 years ago
Salva Corts	8978ecf0c8	feat: Boilerplate for new bloom build planner and worker components. (#12989 )	2 years ago

44 Commits (bbff4331ead29cbe15dd8e7a970bd754e73a0bd8)