7 Commits (75cfe5959694d69841a632f39a8f078e9ad3c2e6)

Author	SHA1	Message	Date
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290) ... **What this PR does / why we need it**: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago
Poyzan	cc6f0eca98	Return checksum from BuildFrom function (#11178) ... **What this PR does / why we need it**: Follow up from https://github.com/grafana/loki/pull/11115 We want to know the checksum of the bloom blocks before uploading them to storage. Instead of opening the file and reading it, this PR returns the checksum from where it's created.	2 years ago
Owen Diehl	6bfd2baec4	Bloom/fuse queries (#11088) ... Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons: * A "query" contains one of more series, with each series needing to test one or more chunks for some search string(s). * Blocks are sorted by fingerprint, which are the result of a hash function. This means series are ~uniformly distributed across the hash keyspace (`uint64` in our case). Because of this, a query for some number of series are likely to be spread out across the block itself. * Blocks are organized into individually compressed "pages". In essence, querying them requires Seeking to the correct offset, decompressing the page, then seeking within the page to the actual offset for a particular series. * Since the series for any query are likely spread out across the block, performing `n` queries decompresses each page `n` times. * By multiplexing many queries into a single pass iteration of the block, we amortize the cost of loading & decompressing Misc: Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.	2 years ago
Owen Diehl	0a7737e7c7	adds a merge-builder for building blocks from other blocks+storage (#10974) ... Some minor refactoring + a (simple) `MergeBuilder` struct to build bloom blocks from other blocks and storage, depending on what's already been indexed into the blocks.	2 years ago
Poyzan	cbc0df3241	Add NewBlockOptions constructor (#10955) ... Adds a `NewBlockOptions()` constructor plus a quick fix for a missed err check.	2 years ago
Owen Diehl	ad0f7ab845	Blooms/merge-iters (#10923) ... Adds a bunch of bloom filterutilities in preparation for merging & extending bloom blocks with new chunks/series	2 years ago
Owen Diehl	c405db4317	Format + Library for building bloom filter "blocks" from chunks (#10780) ... This PR introduces both a binary format for bloom filter "blocks" and a library for building/querying them. Each "block" consists of two files: 1) `Index`. This is a list of individually compressed pages of stream+chunk information sorted in ascending fingerprint order. Each series lists both the chunks indexed in the bloom filters as well as the relevant offsets for said series in the bloom file. 2) `Blooms`. This is a list of individually compressed pages of bloom filters for each series, again sorted in ascending fingerprint order. There is still more work to do, but I'm opening this PR as an initial pass.	2 years ago