987 Commits (d9c441efcc159e91a4251c9aca8f8914633c8505)

Author	SHA1	Message	Date
Owen Diehl	9235229b9e	refactor(bloom-gw): todos and comments (#12135)	1 year ago
Owen Diehl	63e0bcb84b	fix(blooms): embeddedcache size panic (#12130)	1 year ago
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120) ... * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	97ae790f36	test: Update chunk/cache/mock to have a GetKeys() function (#12122)	1 year ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101)	1 year ago
Christian Haudum	4cae003ece	chore(blooms): Make blocks cache preloading less verbose (#12107) ... The warnings suggest something going wrong where it's not the case. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	a8aa3f68cb	feat: Bloom filter regexes (#12096)	1 year ago
Christian Haudum	90bcaea724	chore(blooms): Add more instrumentation to bloom gateway and bloom store (#12048) ... * Add metric for size of metas/blocks fetched Getting the distribution of the uncompressed size of metas and blocks, grouped by their source (cache/filesystem/storage). * Move blocks/metas fetched metrics into bloomstore * Add minimal logging to bloom gateway processor * Observe queue duration of tasks The histogram metric gives us the insight about how long tasks spend their time enqueued waiting to be processed. * Remove queueMetrics field from bloom gateway struct The metrics are only used by the queue instance, which is part of the bloom gateway struct as well. * Move various bloom gateway metrics into separate file This commit introduces a new metrics struct that contains both the metrics specific the server/handler and the metrics specific to the worker. This has the advantage that metrics can be instantiated and accessed from one single place. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	77a2580052	fix: Add data race protection for the tm.wg var (#12100)	1 year ago
Christian Haudum	184d1de4db	chore(blooms): Use `bloomshipper.Interval` instead of `model.Interval` (#12079) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	637c292cd8	fix(blooms): Fix issue when loading cache on startup if directory does not exist (#12080) ... The check `!dirEntry.IsDir()` panicked in case `dirEntry` is `nil`, which is the case when the root directory does not exist. The PR also adds more logging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Danny Kopping	b5f488caec	fix(query/query-frontend): correct congestion control stats aggregation (#12072) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Danny Kopping	609fdb3214	feat(querier/query-frontend): track and log congestion control latency (#12058) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Owen Diehl	60f75f3395	fix(blooms): closes chunk iterator (#12053)	1 year ago
Christian Haudum	cb7e016d64	chore(blooms): Populate blocks cache on startup (#12056) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	7c8992f4e4	fix(blooms): fix reference leak and resulting race condition in BloomPageDecoder (#12050)	1 year ago
Salva Corts	8193063a4a	feat: Convert line filter expressions to a set of bloom tests (#12035)	1 year ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037)	1 year ago
Christian Haudum	b9ce005ec1	fix: Ensure working dir for bloomstore exists (#12019) ... Fail startup if directory does not exist or there are not enough permissions. This prevents the bloomstore to fail later in the process once it tries to download and extract blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	32a9a3f45d	chore(blooms): Improve how block directory is extracted (#12030) ... This PR contains two changes: 1) Extracting the block without an intermediate temp file. The commit is cherry-picked from https://github.com/grafana/loki/pull/12021, but that PR got merged into the wrong branch 🙈 2) Strip `.tar.gz` suffix from block file when extracting it into a directory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	b47f2bfbc6	refactor: Limit processing of blocks to requested fingerprint ranges in bloom gateway (#11987) ... This PR limits the amount of data being processed for a single multiplexed iteration to the union of the fingerprint bounds of its requests, instead of looking at blocks from the complete fingerprint range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1dce7b512c	fix(blooms): race condition in `BloomPageDecoder` (#12027)	1 year ago
Owen Diehl	e835b6192f	refactor(blooms): refactors `blockLoadingIter`, exposing more error cases + additional metrics (#12015)	1 year ago
Christian Haudum	b9d8eefe27	test: Use table prefix in bloomstore tests (#12022) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	09aa0f80b5	Blooms/contiguous bounds union (#12009)	1 year ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979)	1 year ago
Salva Corts	ef40136715	Use TSDB index prefix on blooms directory (#11977)	1 year ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974)	1 year ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966)	1 year ago
Owen Diehl	443720f47b	tsdb parsing handles uint (#11969)	1 year ago
Owen Diehl	bd12e16339	[Blooms] Use correct table address function (#11955)	1 year ago
Owen Diehl	3c06b360ec	meta & removal support for bloom compaction (#11941) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Ashwanth	9e7725b31b	feat(metadata): introduce a separate split interval for recent query window (#11897)	1 year ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917) ... Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	eb8464a641	Bloom compactor: Load blocks lazily in batches (#11919) ... To avoid loading possibly lots of blocks upfront, this PR introduces lazy loading of blocks in batches using an iterator that loads blocks on demand. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	7560067439	Bloom/running (#11918) ... Includes a few small fixes in order to get the compactor running, including * Wires up i/o from `v1` pkg and the `bloomshipper` pkg in order to archive+write blocks remotely after building * config error handling * fix an error iterating chunks * handles gzipping when loading tsdbs from storage * bypasses internal cache in `BloomTSDBStore`'s internal `indexshipper/storage.Client` to get up to date results, which is easier for prototyping * `ClosableReadSeekerAdapter` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	681bb57971	Limit bloom block size (#11878) ... **What this PR does / why we need it**: This PR limits the size of the blocks created by the compactor. The block builder keeps adding series' blooms to a block until the limit is exceeded, meaning that blocks may grow beyond the configured maximum. This is needed so we avoid having tiny blocks which had space for small blooms but later a bigger blooms didn't fit. Blocks are built lazily: the generator returns an iterator that builds one block at a time. **Special notes for your reviewer**: The maximum size is currently set to 50 MBs. We will make this configurable on a followup PR. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	3efd201bff	Fix test utility function `MkBasicSeriesWithBlooms` (#11909) ... The utility correctly populated the blooms, but returned incorrect values for `keys`, which are the indexed keys of a bloom. The values need to be copied when appending to the result slice. Additionally, the source for the tokenizer changed to be avoid duplicate keys that are present in all blooms. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	fe53eb6817	fix type errors after introduction of typedcache into bloomshipper (#11902) ... Recent changes causing compilation failures -- this fixes type signatures after refactoring. Also removes test coverage comparisons because they prevent us from fixing `main`	1 year ago
Owen Diehl	3a1bd41ef5	[Blooms] Routing & wiring for bloom building (#11894) ... A few changes wiring up bloom building logic: * A `Router` as the controlling mechanism for iterating tenants+periods, queueing work for parallel computation, determining which fingerprint ranges and tenants are owned by a replica, and waiting for it's completion * Updates the `SimplerBloomController` to accept ownership ranges, periods, and tenants as arguments for use by the `Router` * Builds a `BloomTSDBStore` struct around the `indexshipper/storage.Client` interface in order to iterate relevant TSDBs/etc	1 year ago
Christian Haudum	3b122e4be1	Bloom gateway: Wire up cache config with implementation (#11898) ... Until now, the cache configuration in the bloom gateway did not initialise the cache. This PR wires the config with the actual implementation and passes them to the bloom store. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	9128036145	Bloom gateway: Integrate processor into worker (#11895) ... This PR replaces the iteration of the bloom gateway worker with the code encapsulated in the previously established processor. Since the processor only relies on the BloomStore, rather than on the BloomShipper, the PR also removes unused code from the shipper. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	ba0538c2a5	[Blooms] tsdb iterator implementation for bloom construction (#11886) ... Builds a lazy iterator over a tsdb file for bloom construction. This is just the library utility, not the integration.	1 year ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]*CloseableBlockQuerier` (#11884) ... **What this PR does / why we need it**: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `*CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	7bbbf232d0	Extend BloomStore with `Client(model.Time)` function (#11881) ... **What this PR does / why we need it**: For certain operations, we want direct access to the `BloomClient`. Since every schema period has its own client, the `Client()` function accepts a `model.Timestamp` for which the client should be returned. The function may return an error, if no client for the given time could be found. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	69d152b618	(chore) BloomStore: Clean up `FetchBlocks()` (#11876) ... **What this PR does / why we need it**: * Removes `FetchBlocksWithQueue` and replace it with `FetchBlocks` * Operate on single `BlockRef` when loading a block directory from cache/fs/storage. * Ensure order of responses from the FetchBlocks request --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	af66ecec6a	[Blooms] bloomshipper meta integration (#11874) ... Builds on top of https://github.com/grafana/loki/pull/11872, standardizing on the `bloomshipper` types `Meta` and `MetaRef`.	1 year ago
Owen Diehl	3a92834de0	[Blooms] Stateless archiving (#11872) ... Moves the archival code to an interface based approach, removing the explicit fs dependency. We can now implement with in-memory solutions. Also refactors to use a tar-file iterator and marks tests as parallelizable.	1 year ago
Christian Haudum	73edf7a943	(chore) Bloomshipper: Separate store and client (#11865) ... **What this PR does / why we need it**: This PR removes the `StoreAndClient` interface that was accepted by the `BloomShipper`. Since the `BloomStore` had to not only implement the `Store` interface, but also the `Client` interface, it caused re-implementation of the same methods in different ways. Now the shipper solely relies on the `Store` interface. See individual commit messages for more context. Tests have been rewritten from scratch and placed in their own respective test files for store and client. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago