9 Commits (ea798e0f2a3364b4a76f153faf324b4a9ababc4d)

Author	SHA1	Message	Date
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162)	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039) ... This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Christian Haudum	2edbe1280f	fix(blooms): Correctly skip block page in case it exceeds the max page size for querying (#12297) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago