apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539 )	7 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038 ) A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273 ) Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172 ) This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093 )	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162 )	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039 ) This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	75ccf2160b	feat(blooms): Separate page buffer pools for series pages and bloom pages (#12992 ) Series pages are much smaller than bloom pages and therefore can make use of a separate buffer pool with different buckets. The second commit fixes a possible panic. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	30ac88bc26	chore(blooms): increase blockpool by factor-of-2 (#12363 )	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	0f43bf6958	fix(blooms): fix typo in pool decl (#12302 )	1 year ago
Owen Diehl	fe1b357cc2	fix(blooms): leaky ref (#12300 )	1 year ago
Christian Haudum	4aec20f36b	perf(blooms): Recycle bloom page buffers (#12256 ) Every time when reading a bloom page into memory, a new byte slice with the size of the decoded page was allocated. This PR changes bloom page decoder to use a byte slice from the `BlockPool`. The caller (`LazyBloomIter` of the block querier) is responsible for closing the page whenever a new one is loaded, so the byte slice is put back to its pool. Additionally, this PR changes the max bucket of the `BlockPool` byte slice pool to 64MB so that pages bigger than 16MB are correctly put back to the pool, since [slices exceeding the capacity of the largest bucket are not put back](https://github.com/grafana/loki/blob/main/vendor/github.com/prometheus/prometheus/util/pool/pool.go#L74-L87). --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	4b28f8221b	fix: Read all series in NewTSDBSeriesIter and close index file (#12211 )	1 year ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101 )	1 year ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966 )	1 year ago
Christian Haudum	eb8464a641	Bloom compactor: Load blocks lazily in batches (#11919 ) To avoid loading possibly lots of blocks upfront, this PR introduces lazy loading of blocks in batches using an iterator that loads blocks on demand. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	681bb57971	Limit bloom block size (#11878 ) What this PR does / why we need it: This PR limits the size of the blocks created by the compactor. The block builder keeps adding series' blooms to a block until the limit is exceeded, meaning that blocks may grow beyond the configured maximum. This is needed so we avoid having tiny blocks which had space for small blooms but later a bigger blooms didn't fit. Blocks are built lazily: the generator returns an iterator that builds one block at a time. Special notes for your reviewer: The maximum size is currently set to 50 MBs. We will make this configurable on a followup PR. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	af66ecec6a	[Blooms] bloomshipper meta integration (#11874 ) Builds on top of https://github.com/grafana/loki/pull/11872, standardizing on the `bloomshipper` types `Meta` and `MetaRef`.	1 year ago
Owen Diehl	509f6f6b66	Bloom/controller wiring (#11831 ) Functionality & testware for building bloom block plans.	1 year ago
Owen Diehl	a56b56c8b0	[Bloom] Keyspace aware bloom creation, library utilities (#11810 ) This PR continues work refactoring the bloom compactor pkg and adds a bunch of utilities for keyspace manipulation/mgmt. A lot of the bloom block logic is contained here as a skeleton. This code is not complete, but is fine to merge as we don't use this yet and should be easier to review before it gets too large.	1 year ago
Owen Diehl	de4f56e42d	[Blooms] Refactoring bloom compactor to isolate state from logic (#11793 ) This PR refactors a bunch of code to separate state+I/O related complexity from logic so we can test and extend it more easily. It also adds more logic for the bloom generator to use.	1 year ago
Owen Diehl	58eaad9947	boundscheck & partitioning fingerprints between blocks (#11237 ) Adds a few utilities for comparing fingerprints to blocks that cover a specific fingerprint range. Will likely need to be refactored with more comprehensive types, but the logic still applies.	2 years ago
Owen Diehl	6bfd2baec4	Bloom/fuse queries (#11088 ) Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons: * A "query" contains one of more series, with each series needing to test one or more chunks for some search string(s). * Blocks are sorted by fingerprint, which are the result of a hash function. This means series are ~uniformly distributed across the hash keyspace (`uint64` in our case). Because of this, a query for some number of series are likely to be spread out across the block itself. * Blocks are organized into individually compressed "pages". In essence, querying them requires Seeking to the correct offset, decompressing the page, then seeking within the page to the actual offset for a particular series. * Since the series for any query are likely spread out across the block, performing `n` queries decompresses each page `n` times. * By multiplexing many queries into a single pass iteration of the block, we amortize the cost of loading & decompressing Misc: Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.	2 years ago
Owen Diehl	ad0f7ab845	Blooms/merge-iters (#10923 ) Adds a bunch of bloom filterutilities in preparation for merging & extending bloom blocks with new chunks/series	2 years ago
Owen Diehl	c405db4317	Format + Library for building bloom filter "blocks" from chunks (#10780 ) This PR introduces both a binary format for bloom filter "blocks" and a library for building/querying them. Each "block" consists of two files: 1) `Index`. This is a list of individually compressed pages of stream+chunk information sorted in ascending fingerprint order. Each series lists both the chunks indexed in the bloom filters as well as the relevant offsets for said series in the bloom file. 2) `Blooms`. This is a list of individually compressed pages of bloom filters for each series, again sorted in ascending fingerprint order. There is still more work to do, but I'm opening this PR as an initial pass.	2 years ago

27 Commits (8958eb91eda56a5334d0bdb683a9fb592cd5d17e)