884 Commits (fbb8686d03782ee91f30c9a5e30c48c006f9dfca)

Author	SHA1	Message	Date
Christian Haudum	bf0dd9dda6	perf(bloomstore): Cache metas LIST operation (#12414) ... Reduce the amount of class A list operations on object storage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	824f5aa20a	chore: fix submodule for v3 (#12438) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	71602eb10f	chore(blooms): Clean up bloom component configuration (#12387) ... * Remove unused setting and move download parallelism configuration one level up. * Update description of bloom_shipper config block Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Anton Kolesnikov	36c703dae8	feat: add pattern match line filter (#12398)	1 year ago
Ed Welch	a509871c55	chore: remove experimental flags for l2 cache and memcached "addresses" config (#12410)	1 year ago
Salva Corts	86c768c3cc	feat: Blooms retention (#12258)	1 year ago
Anton Kolesnikov	cc941fe42a	chore: refactor line filter MatchType (#12388)	1 year ago
Christian Haudum	15dc2bac04	feat(bloomstore): Support for sharding blocks across multiple different directories (#12375) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	aca2a38ad6	test: gcs object client test data race (#12324)	1 year ago
Christian Haudum	ca667aa66e	chore(bloom-gw): Cleanup tracing implementation (#12368) ... The huge amount of `bloomgateway.ProcessTask` spans caused problem with Tempo's span limit, resulting in dropped/dangling spans. Since the processing time is now recorded also in a metrics.go-like stats log line, we can remove the spans and only add a log event with the summary to the main span of the request handler. This PR also unifies the event logging in the index gateway, and adds the real processing time (not aggregated processing time) to the request stats. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	120b76d63e	chore(blooms): misc fixes (#12369) ... unbounds bloom-gw-client concurrency removes unused log_gateway_requests param adds note wrt bloom-gw results cache keygen issue add msg field to stats log sets bloomshipper download workers count to 16 default	1 year ago
Christian Haudum	19c046f14b	chore(blooms): Make max bloom page size for querying configurable (#12337) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	30ac88bc26	chore(blooms): increase blockpool by factor-of-2 (#12363)	1 year ago
Salva Corts	5c6a031c08	fix(blooms): Not filters should always match (#12358)	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Travis Patterson	c8d2c32002	feat: Introduce special header that tells Loki not to modify query results (#12327)	1 year ago
Salva Corts	e4a5733ada	fix(blooms): Extract only line filters before line format expressions (#12334)	1 year ago
Dylan Guedes	51c54adc4f	feat: Memcached: Add TLS support (#12318)	1 year ago
Christian Haudum	a54f2dba7a	perf(blooms): Optimize memory usage for blocks that have no compression (#12316) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Travis Patterson	d175184fe5	chore: remove shards from wrapper signature (#12322)	1 year ago
Travis Patterson	fed7566ba9	fix: Pass query to pipeline wrappers from the AST (#12305)	1 year ago
Christian Haudum	07fab4b118	fix(bloomstore): Fix race conditions (#12320) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	3b5ed8c3ec	chore: pass shard count to injected pipelines and extractors (#12213)	1 year ago
BiKangNing	75e031d8e8	chore: fix function names in doc strings (#12313) ... Signed-off-by: depthlending <bikangning@outlook.com>	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	0f43bf6958	fix(blooms): fix typo in pool decl (#12302)	1 year ago
Owen Diehl	fe1b357cc2	fix(blooms): leaky ref (#12300)	1 year ago
Christian Haudum	62b3f358a6	feat(bloomstore): Wire up new blocks cache (#12275) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	eb03974086	feat(blooms): Make bloom block compression configurable (#12293) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	2edbe1280f	fix(blooms): Correctly skip block page in case it exceeds the max page size for querying (#12297) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	0f48ce8f18	feat(bloomstore): Introduce specialised cache for blocks (#12257) ... The blocks cache is an in-memory cache that represents the single source of truth of what blocks are available on the file system and manages their end-of-life behaviour. There is a follow-up branch to wire up the new cache, based on this branch. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Cyril Tovena	cdb934ccee	feat(blooms): Dedupe download queue items to reduce queue size (#12222)	1 year ago
Christian Haudum	4aec20f36b	perf(blooms): Recycle bloom page buffers (#12256) ... Every time when reading a bloom page into memory, a new byte slice with the size of the decoded page was allocated. This PR changes bloom page decoder to use a byte slice from the `BlockPool`. The caller (`LazyBloomIter` of the block querier) is responsible for closing the page whenever a new one is loaded, so the byte slice is put back to its pool. Additionally, this PR changes the max bucket of the `BlockPool` byte slice pool to 64MB so that pages bigger than 16MB are correctly put back to the pool, since [slices exceeding the capacity of the largest bucket are not put back](https://github.com/grafana/loki/blob/main/vendor/github.com/prometheus/prometheus/util/pool/pool.go#L74-L87). --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	0d1a3694f5	test: Don't start server until all configured (#12226)	1 year ago
Salva Corts	2544f003cf	chore: Add summary for received filters in bloom gateway (#12158)	1 year ago
Salva Corts	63c88489d0	refactor: Add RF and Tokens to Loki ring Cfg and allow overwriting docs. (#12142) ... Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Salva Corts	4b28f8221b	fix: Read all series in NewTSDBSeriesIter and close index file (#12211)	1 year ago
Sandeep Sukhani	25f15cb326	feat: add support for setting custom grpc interceptors for requests sent to index-gateway (#12193)	1 year ago
Christian Haudum	a0fce39169	feat(bloom-gw): Make num multiplexing tasks configurable, and other adjustments (#12169) ... * We want to be able to change the amount of dequeued items per iterations to test different configurations. * Ignore block pages that are magnitues bigger than the target page size. * Increase download queue capacity to avoid blocking when enqueueing Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	e95cc6cc09	chore(blooms): Adds basic tracing implementation to bloom-gw (#12175) ... Also * removes unused `SyncMap` * adds a `BlockRefFrom` helper	1 year ago
Christian Haudum	f6cfff5356	refactor(bloomstore): Introduce fetch option for blocks fetcher (#12160) ... Requests to the bloom gateway should not wait for blocks to be downloaded into cache first, but should operate on readily available blocks. This PR introduces fetch options for the bloom store that specify how the store should behave when requesting blocks from block refs. `WithIgnoreMissing`: Ignore errors from blocks that could not be found in object storage, and instead return a `nil` value in the response. `WithFetchAsync`: Return only the blocks that are available locally in cache and dispatch downloading of blocks to the blocks downloader, that operates in the background, and make them available for subsequent requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	f4b2c5d69b	fix(blooms): bloom-gw chunk merging improvements (#12162)	1 year ago
Salva Corts	49f8f7b3ee	refactor: Chunkref as alias to logproto.ShortRef (#12159)	1 year ago
Owen Diehl	9a993ca690	chore(blooms): Ordering & Unless iterator (#12154)	1 year ago
Owen Diehl	8f3767c4ad	chore(blooms): block_query_latency_seconds rename (#12152)	1 year ago
Owen Diehl	30a240b470	fix(blooms): filter out nil blocks (#12150)	1 year ago
Owen Diehl	9235229b9e	refactor(bloom-gw): todos and comments (#12135)	1 year ago
Owen Diehl	63e0bcb84b	fix(blooms): embeddedcache size panic (#12130)	1 year ago
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120) ... * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago