apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	39a52974af	chore: Remove excessive logging from bloom gateway (#13603 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502 )	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273 ) Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	3ac231728e	fix(blooms): Suppress error from resolving server addresses for blocks (#13385 ) Instead of returning the error, `FilterChunks` returns the full, unfiltered list of chunks. This makes sure that queries do not fail if no bloom gateway instances are available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	f661593cae	refactor(blooms): more easily testable day partitioning (#13331 )	11 months ago
Christian Haudum	3b0502ddd3	chore(blooms): Various minor code cleanups (#13332 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	66f97b2aec	fix(blooms): Fix panic in bloom gateway (#13303 ) The slice of block queriers can contain `nil` values, which causes nip pointer dereference when calling `Close()` on them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288 )	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172 ) This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Bryan Boreham	2b19dac82a	fix: errors reported by the race detector (#13174 ) Fixes #8586 Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Christian Haudum	d4fcef5229	chore(blooms): Some boom gateway cleanups (#13165 ) * Cleanup logic for processing multiple days in single `FilterChunkRefs` request in the bloom gateway. The handler already returned an error if the requested chunk refs spanned across multiple days, but the logic for processing multiple days/tasks was still in place. This commit simplifies the logic to only process a single task per request. * Remove unused package global variable * Remove bloomgateway package comment * Remove duplicate imports * Avoid conversion from logproto.ShortRef to v1.ChunkRef and istead cast the value of the pointer Simplify `getFromThrough()` function Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093 )	12 months ago
Owen Diehl	41c5ee21fc	fix(regression): reverts grafana/loki#13039 to prevent use-after-free corruptions (#13162 )	12 months ago
Christian Haudum	fc264310ce	perf: Introduce fixed size memory pool for bloom querier (#13039 ) This PR introduces a fixed size memory pool for bloom pages that are loaded in the block querier. The aim of a fixed size pool of `[]byte` buffers is to reduce the amount of allocations, as well as to control the maximum heap size to prevent OOMing of the bloom gateways. Also, with the current usage of a `sync.Pool`, the query parallelism (`bloom-gateway.worker-concurrency` and `bloom-gateway.block-query-concurrency`) was defined by the `bloom.max-query-page-size` (and vice versa), because the max page size could be loaded `workers * concurrency` times at the same time. Most of the time, though, smaller pages are loaded, and therefore concurrency is not optimized. With the new fixed size memory pool, this problem should be solved. The pool is divided into slabs of different sizes holding different amounts of buffers; a larger amount of small sized buffers and a smaller amount of large sized buffers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	d6374bc2ce	feat(blooms): Add counter metric for blocks that are not available at query time (#12968 ) When filtering chunks on the bloom gateway, bloom block may not be available and they will be downloaded asynchronously in the background. This new metric `loki_bloom_gateway_blocks_not_available_total` counts the blocks that are not available at query time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	a1b1eeb095	feat(bloom): Skip attempts to filter chunks for which blooms have not been built (#12961 ) Bloom filters are built from `today - MaxTableOffset` to `today - MinTableOffset`, this means that blooms are not available for the most recent period, between `now` and at least `today - MinTableOffset`. To avoid resolving chunks and filtering out no-matches, we can skip chunks for this period completely and return them as-is without filtering. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	738c274a58	fix(blooms): Fix `partitionSeriesByDay` function (#12900 ) The function `partitionSeriesByDay` could yield empty when chunks were outside of the requested time range. This would result in requests for a time range that does not contain any chunks for a series to filter. These requests would return errors, because a subsequent `partitionSeriesByDay` for would remove that day resulting in no task. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889 )	1 year ago
Owen Diehl	5a643c7115	chore: reduces span footprint + double recording (#12864 ) Trying to remove some cruft from traces.	1 year ago
Owen Diehl	4c9b22f110	feat(blooms): ignore individual bloom-gw failures (#12863 )	1 year ago
Christian Haudum	48bbf983d0	chore(blooms): Remove ID field from task struct (#12851 ) We've seen a few cases where creating the ULID failed for unknown reasons, and the ID is not really used. It was only useful early on in the development for debugging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807 ) This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Michel Hollands	33e82ec133	fix: remove unused parameter causing lint error (#12801 ) Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Christian Haudum	3bf2d1fea0	fix(blooms): Deduplicate filtered series and chunks (#12791 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	4cc3a73172	test: Add mutex and setter/getter for addresses in test (#12793 )	1 year ago
Christian Haudum	ecefb49508	fix(blooms): Fix a regression introduced with #12774 (#12776 ) This PR fixes a regression introduced with #12774 where series where wrongly identified as "skipped" even though there were part of blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c36b1142c7	fix(blooms): Correctly return unfiltered chunks for series that are not mapped to any block (#12774 ) This PR fixes a conceptual mistake in the code of resolving blocks on the index gateways. Currently, a series does not resolve to any block is discarded instead of being kept for the response. This change adds the chunks of the skipped series to the bloom querier response. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	ae180d6e07	fix(nix): lambda-promtail vendor hash (#12763 )	1 year ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720 ) This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496 ) Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	1 year ago
Owen Diehl	712a2b6967	fix(blooms): fixup bloom testing (#12494 )	1 year ago
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	ff08d2fcc2	chore(blooms): Remove bloom gateway ring (#12484 ) Remove ring in favour of a client side, DNS discovery of the bloom gateway servers combined with consistent hashing using the jumphash algorithm (#12470). Instead of configuring the ring, the bloom gateway client needs to be configured with the `-bloom-gateway-client.addresses` flag, which takes a list of comma separated, DNS-discovery-style strings. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	58c8c61905	perf(blooms): Use jumpmap for client side sharding of the filter requests (#12470 ) What this PR does / why we need it: Use the same client-side sharding mechanism for bloom gateway filter requests that we use for client side sharding in the memcached client. This uses DNS discovery to resolve a list of addresses uses as buckets for the jumphash algorithm to shard GroupedChunkRefs across bloom gateway servers. The ring component is still part of the bloom gateway, but it is not used any more. Special notes for your reviewer: https://arxiv.org/pdf/1406.2294.pdf Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	a6f3b93f7e	chore(blooms): bloom_client records cache locality score (#12458 ) This is a helpful way to measure how effectively we distribute filter queries to bloom-gws. Should mostly converge to 1 once we start using jump-hash instead of the ring for these, but it'll give us a good way to measure that improvement.	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	71602eb10f	chore(blooms): Clean up bloom component configuration (#12387 ) * Remove unused setting and move download parallelism configuration one level up. * Update description of bloom_shipper config block Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	cf71ac7114	fix(blooms): Remove blocks not matching any series in task (#12401 )	1 year ago
Salva Corts	86c768c3cc	feat: Blooms retention (#12258 )	1 year ago
Christian Haudum	15dc2bac04	feat(bloomstore): Support for sharding blocks across multiple different directories (#12375 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	ca667aa66e	chore(bloom-gw): Cleanup tracing implementation (#12368 ) The huge amount of `bloomgateway.ProcessTask` spans caused problem with Tempo's span limit, resulting in dropped/dangling spans. Since the processing time is now recorded also in a metrics.go-like stats log line, we can remove the spans and only add a log event with the summary to the main span of the request handler. This PR also unifies the event logging in the index gateway, and adds the real processing time (not aggregated processing time) to the request stats. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	120b76d63e	chore(blooms): misc fixes (#12369 ) unbounds bloom-gw-client concurrency removes unused log_gateway_requests param adds note wrt bloom-gw results cache keygen issue add msg field to stats log sets bloomshipper download workers count to 16 default	1 year ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	9810e8ec63	chore(blooms): computed fields for bloomgw stats logging (#12367 )	1 year ago
Christian Haudum	9af191f3d4	feat(bloom-gw): Add `metrics.go` style log line to bloom gateway `FilterChunks` call (#12354 ) This PR adds a "metrics.go" style log line to the bloom gateway that contains latencies for queueing, processing, post-processing, as well as number of chunks/series requested/filtered. This log line should give operators a better understanding where time is spent for individual requests. Example log line: ``` ts=2024-03-26T13:16:11.869619964Z caller=spanlogger.go:109 component=bloom-gateway tenant=XXX method=bloomgateway.FilterChunkRefs user=XXX traceID=6239d49e1e88f88645bd7f1f6f1b85c8 level=info status=success tasks=1 series_requested=35 series_filtered=31 chunks_requested=103 chunks_filtered=93 queue_time=4.108128936s metas_fetch_time=22.040408ms blocks_fetch_time=1.284575ms processing_time=30.215863002s post_processing_time=16.084µs ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Christian Haudum	c74d0eb107	fix(bloom-gateway): Close all block queriers in case of error (#12338 ) This PR fixes the logic for closing block queriers retrieved from the `store.FetchBlocks()` call in case of an error. Since `concurrency.ForEach()` returns on first errors, it happened that the remaining block queriers did not get closed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	e4a5733ada	fix(blooms): Extract only line filters before line format expressions (#12334 )	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292 ) A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	62b3f358a6	feat(bloomstore): Wire up new blocks cache (#12275 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago

1 2 3

141 Commits (39a52974af551e80c54dff573643b59cfc62c72f)