apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038 ) A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	784e7d562f	fix: Keep blocks referenced by newer metas (#13614 )	1 year ago
Christian Haudum	39a52974af	chore: Remove excessive logging from bloom gateway (#13603 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502 )	1 year ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273 ) Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	3b0502ddd3	chore(blooms): Various minor code cleanups (#13332 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	66f97b2aec	fix(blooms): Fix panic in bloom gateway (#13303 ) The slice of block queriers can contain `nil` values, which causes nip pointer dereference when calling `Close()` on them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288 )	1 year ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172 ) This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	d6374bc2ce	feat(blooms): Add counter metric for blocks that are not available at query time (#12968 ) When filtering chunks on the bloom gateway, bloom block may not be available and they will be downloaded asynchronously in the background. This new metric `loki_bloom_gateway_blocks_not_available_total` counts the blocks that are not available at query time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889 )	2 years ago
Christian Haudum	48bbf983d0	chore(blooms): Remove ID field from task struct (#12851 ) We've seen a few cases where creating the ULID failed for unknown reasons, and the ID is not really used. It was only useful early on in the development for debugging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Trevor Whitney	ae180d6e07	fix(nix): lambda-promtail vendor hash (#12763 )	2 years ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720 ) This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	712a2b6967	fix(blooms): fixup bloom testing (#12494 )	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	cf71ac7114	fix(blooms): Remove blocks not matching any series in task (#12401 )	2 years ago
Christian Haudum	ca667aa66e	chore(bloom-gw): Cleanup tracing implementation (#12368 ) The huge amount of `bloomgateway.ProcessTask` spans caused problem with Tempo's span limit, resulting in dropped/dangling spans. Since the processing time is now recorded also in a metrics.go-like stats log line, we can remove the spans and only add a log event with the summary to the main span of the request handler. This PR also unifies the event logging in the index gateway, and adds the real processing time (not aggregated processing time) to the request stats. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	9af191f3d4	feat(bloom-gw): Add `metrics.go` style log line to bloom gateway `FilterChunks` call (#12354 ) This PR adds a "metrics.go" style log line to the bloom gateway that contains latencies for queueing, processing, post-processing, as well as number of chunks/series requested/filtered. This log line should give operators a better understanding where time is spent for individual requests. Example log line: ``` ts=2024-03-26T13:16:11.869619964Z caller=spanlogger.go:109 component=bloom-gateway tenant=XXX method=bloomgateway.FilterChunkRefs user=XXX traceID=6239d49e1e88f88645bd7f1f6f1b85c8 level=info status=success tasks=1 series_requested=35 series_filtered=31 chunks_requested=103 chunks_filtered=93 queue_time=4.108128936s metas_fetch_time=22.040408ms blocks_fetch_time=1.284575ms processing_time=30.215863002s post_processing_time=16.084µs ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	c74d0eb107	fix(bloom-gateway): Close all block queriers in case of error (#12338 ) This PR fixes the logic for closing block queriers retrieved from the `store.FetchBlocks()` call in case of an error. Since `concurrency.ForEach()` returns on first errors, it happened that the remaining block queriers did not get closed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292 ) A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	8abfc29e44	chore(bloom-gw): Process blocks in parallel (#12203 ) While processing a single block is rather fast, processing a lot of blocks sequentially can lead to problem that single slowly processed blocks lead to high tail latency. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	a0fce39169	feat(bloom-gw): Make num multiplexing tasks configurable, and other adjustments (#12169 ) * We want to be able to change the amount of dequeued items per iterations to test different configurations. * Ignore block pages that are magnitues bigger than the target page size. * Increase download queue capacity to avoid blocking when enqueueing Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	e95cc6cc09	chore(blooms): Adds basic tracing implementation to bloom-gw (#12175 ) Also * removes unused `SyncMap` * adds a `BlockRefFrom` helper	2 years ago
Christian Haudum	f6cfff5356	refactor(bloomstore): Introduce fetch option for blocks fetcher (#12160 ) Requests to the bloom gateway should not wait for blocks to be downloaded into cache first, but should operate on readily available blocks. This PR introduces fetch options for the bloom store that specify how the store should behave when requesting blocks from block refs. `WithIgnoreMissing`: Ignore errors from blocks that could not be found in object storage, and instead return a `nil` value in the response. `WithFetchAsync`: Return only the blocks that are available locally in cache and dispatch downloading of blocks to the blocks downloader, that operates in the background, and make them available for subsequent requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120 ) * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	90bcaea724	chore(blooms): Add more instrumentation to bloom gateway and bloom store (#12048 ) * Add metric for size of metas/blocks fetched Getting the distribution of the uncompressed size of metas and blocks, grouped by their source (cache/filesystem/storage). * Move blocks/metas fetched metrics into bloomstore * Add minimal logging to bloom gateway processor * Observe queue duration of tasks The histogram metric gives us the insight about how long tasks spend their time enqueued waiting to be processed. * Remove queueMetrics field from bloom gateway struct The metrics are only used by the queue instance, which is part of the bloom gateway struct as well. * Move various bloom gateway metrics into separate file This commit introduces a new metrics struct that contains both the metrics specific the server/handler and the metrics specific to the worker. This has the advantage that metrics can be instantiated and accessed from one single place. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b47f2bfbc6	refactor: Limit processing of blocks to requested fingerprint ranges in bloom gateway (#11987 ) This PR limits the amount of data being processed for a single multiplexed iteration to the union of the fingerprint bounds of its requests, instead of looking at blocks from the complete fingerprint range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	38c09a09bd	Bloom gateway: Fix metric for fetched blocks (#11988 ) This PR fixes the metric for fetched blocks Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	fb728a6781	(chore) Cleanup duplicate functions/strucs in bloomgateway package (#11978 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917 ) Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c1d5ec14fd	Update metrics of bloomgateway worker (#11907 ) What this PR does / why we need it: Update metrics for bloomgatway worker to improve observability of the query path. ``` dequeueDuration : Summary processDuration : Summary metasFetched : Summary blocksFetched : Summary tasksDequeued : Counter tasksProcessed : Counter blockQueryLatency : Histogram ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	9128036145	Bloom gateway: Integrate processor into worker (#11895 ) This PR replaces the iteration of the bloom gateway worker with the code encapsulated in the previously established processor. Since the processor only relies on the BloomStore, rather than on the BloomShipper, the PR also removes unused code from the shipper. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]CloseableBlockQuerier` (#11884 ) What this PR does / why we need it: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	098eef7bd4	Blooms/bounds integration (#11848 ) Replaces `Ref`'s min/max fingerprint fields with `v1.FingerprintBounds` and improves a bunch of checks that use it. Builds on top of https://github.com/grafana/loki/pull/11847	2 years ago
Christian Haudum	1191f88190	(chore) Bloom shipper: Extend `Interval` struct with utility functions (#11841 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4411649a0e	(chore) Bloom shipper: Replace `Keyspace` struct with `v1.FingerprintBounds` (#11839 ) The latter struct has more utility functions to compare and operate on bounds. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	6902130c9a	Bloom shipper: Restructure bloom store (#11828 ) What this PR does / why we need it: This PR changes the structure of how the bloom client * The `BloomStore` is the top component interface which implements the `Store` and the `Client` interfaces. The store holds a store entry for each schema period. * The `bloomStoreEntry` implements the `Store` and `Client` interfaces. It holds a bloom client for a single schema period. Additionally, the store entry also exposes a fetcher for metas (and in the future for blocks), which is responsible for getting data from cache or storage (if not available in cache). * The `BloomClient` implement the `Client` interface. The bloom client uses an object client for the schema period of the bloom client. * The `Fetcher` can fetch and cache metas. This structure is very similar to what we use for the chunk store. Note Before implementing `FetchBlocks()` in the `BloomStore`, I want to implement the new `FingerprintBounds` type also in the shipper and bloom gateway code to be consistent across components, as well as to make use of the new utilities exposed by them. The store implementation probably needs some specific test cases around using the correct store entry for different schema period configs. The code however is mostly taken from the chunk store implementation. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	ecab33a932	Bloom Gateway: Extract processing logic from bloom worker into separate struct (#11812 ) What this PR does / why we need it: The processor executes a set of tasks. It cleanly separates I/O from logic and therefore is more testable than the current worker implementation. `9602214abf/pkg/bloomgateway/processor.go (L33-L36)` Note: The processor is not used yet in the worker. Certain parts must be refactored when #11810 is merged. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

41 Commits (adc75389a39e3aaad69303b82b0d68ec3d94485c)