20 Commits (d5ecf9aa7b2bd05a8ef00030c7c0f8a043996114)

Author	SHA1	Message	Date
Christian Haudum	c74d0eb107	fix(bloom-gateway): Close all block queriers in case of error (#12338) ... This PR fixes the logic for closing block queriers retrieved from the `store.FetchBlocks()` call in case of an error. Since `concurrency.ForEach()` returns on first errors, it happened that the remaining block queriers did not get closed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	8abfc29e44	chore(bloom-gw): Process blocks in parallel (#12203) ... While processing a single block is rather fast, processing a lot of blocks sequentially can lead to problem that single slowly processed blocks lead to high tail latency. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	a0fce39169	feat(bloom-gw): Make num multiplexing tasks configurable, and other adjustments (#12169) ... * We want to be able to change the amount of dequeued items per iterations to test different configurations. * Ignore block pages that are magnitues bigger than the target page size. * Increase download queue capacity to avoid blocking when enqueueing Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	e95cc6cc09	chore(blooms): Adds basic tracing implementation to bloom-gw (#12175) ... Also * removes unused `SyncMap` * adds a `BlockRefFrom` helper	2 years ago
Christian Haudum	f6cfff5356	refactor(bloomstore): Introduce fetch option for blocks fetcher (#12160) ... Requests to the bloom gateway should not wait for blocks to be downloaded into cache first, but should operate on readily available blocks. This PR introduces fetch options for the bloom store that specify how the store should behave when requesting blocks from block refs. `WithIgnoreMissing`: Ignore errors from blocks that could not be found in object storage, and instead return a `nil` value in the response. `WithFetchAsync`: Return only the blocks that are available locally in cache and dispatch downloading of blocks to the blocks downloader, that operates in the background, and make them available for subsequent requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120) ... * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	90bcaea724	chore(blooms): Add more instrumentation to bloom gateway and bloom store (#12048) ... * Add metric for size of metas/blocks fetched Getting the distribution of the uncompressed size of metas and blocks, grouped by their source (cache/filesystem/storage). * Move blocks/metas fetched metrics into bloomstore * Add minimal logging to bloom gateway processor * Observe queue duration of tasks The histogram metric gives us the insight about how long tasks spend their time enqueued waiting to be processed. * Remove queueMetrics field from bloom gateway struct The metrics are only used by the queue instance, which is part of the bloom gateway struct as well. * Move various bloom gateway metrics into separate file This commit introduces a new metrics struct that contains both the metrics specific the server/handler and the metrics specific to the worker. This has the advantage that metrics can be instantiated and accessed from one single place. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b47f2bfbc6	refactor: Limit processing of blocks to requested fingerprint ranges in bloom gateway (#11987) ... This PR limits the amount of data being processed for a single multiplexed iteration to the union of the fingerprint bounds of its requests, instead of looking at blocks from the complete fingerprint range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	38c09a09bd	Bloom gateway: Fix metric for fetched blocks (#11988) ... This PR fixes the metric for fetched blocks Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	fb728a6781	(chore) Cleanup duplicate functions/strucs in bloomgateway package (#11978) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917) ... Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c1d5ec14fd	Update metrics of bloomgateway worker (#11907) ... **What this PR does / why we need it**: Update metrics for bloomgatway worker to improve observability of the query path. ``` dequeueDuration : Summary processDuration : Summary metasFetched : Summary blocksFetched : Summary tasksDequeued : Counter tasksProcessed : Counter blockQueryLatency : Histogram ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	9128036145	Bloom gateway: Integrate processor into worker (#11895) ... This PR replaces the iteration of the bloom gateway worker with the code encapsulated in the previously established processor. Since the processor only relies on the BloomStore, rather than on the BloomShipper, the PR also removes unused code from the shipper. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]*CloseableBlockQuerier` (#11884) ... **What this PR does / why we need it**: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `*CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	098eef7bd4	Blooms/bounds integration (#11848) ... Replaces `Ref`'s min/max fingerprint fields with `v1.FingerprintBounds` and improves a bunch of checks that use it. Builds on top of https://github.com/grafana/loki/pull/11847	2 years ago
Christian Haudum	1191f88190	(chore) Bloom shipper: Extend `Interval` struct with utility functions (#11841) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4411649a0e	(chore) Bloom shipper: Replace `Keyspace` struct with `v1.FingerprintBounds` (#11839) ... The latter struct has more utility functions to compare and operate on bounds. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	6902130c9a	Bloom shipper: Restructure bloom store (#11828) ... **What this PR does / why we need it**: This PR changes the structure of how the bloom client * The `BloomStore` is the top component interface which implements the `Store` and the `Client` interfaces. The store holds a store entry for each schema period. * The `bloomStoreEntry` implements the `Store` and `Client` interfaces. It holds a bloom client for a single schema period. Additionally, the store entry also exposes a fetcher for metas (and in the future for blocks), which is responsible for getting data from cache or storage (if not available in cache). * The `BloomClient` implement the `Client` interface. The bloom client uses an object client for the schema period of the bloom client. * The `Fetcher` can fetch and cache metas. This structure is very similar to what we use for the chunk store. **Note** Before implementing `FetchBlocks()` in the `BloomStore`, I want to implement the new `FingerprintBounds` type also in the shipper and bloom gateway code to be consistent across components, as well as to make use of the new utilities exposed by them. The store implementation probably needs some specific test cases around using the correct store entry for different schema period configs. The code however is mostly taken from the chunk store implementation. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	ecab33a932	Bloom Gateway: Extract processing logic from bloom worker into separate struct (#11812) ... **What this PR does / why we need it**: The processor executes a set of tasks. It cleanly separates I/O from logic and therefore is more testable than the current worker implementation. 9602214abf/pkg/bloomgateway/processor.go (L33-L36) **Note**: The processor is not used yet in the worker. **Certain parts must be refactored when #11810 is merged.** --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago