2640 Commits (38e2d7be4a6f1a65e7454b1fe7bdc44e9688a8b6)

Author	SHA1	Message	Date
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120) ... * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	97ae790f36	test: Update chunk/cache/mock to have a GetKeys() function (#12122)	1 year ago
Owen Diehl	17988d5965	fix(blooms): measurable compaction progress (#12101)	1 year ago
Travis Patterson	b0478e61c8	feat: add the ability to inject log parsers to the distributor (#12029)	1 year ago
Christian Haudum	4cae003ece	chore(blooms): Make blocks cache preloading less verbose (#12107) ... The warnings suggest something going wrong where it's not the case. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	a8aa3f68cb	feat: Bloom filter regexes (#12096)	1 year ago
Christian Haudum	90bcaea724	chore(blooms): Add more instrumentation to bloom gateway and bloom store (#12048) ... * Add metric for size of metas/blocks fetched Getting the distribution of the uncompressed size of metas and blocks, grouped by their source (cache/filesystem/storage). * Move blocks/metas fetched metrics into bloomstore * Add minimal logging to bloom gateway processor * Observe queue duration of tasks The histogram metric gives us the insight about how long tasks spend their time enqueued waiting to be processed. * Remove queueMetrics field from bloom gateway struct The metrics are only used by the queue instance, which is part of the bloom gateway struct as well. * Move various bloom gateway metrics into separate file This commit introduces a new metrics struct that contains both the metrics specific the server/handler and the metrics specific to the worker. This has the advantage that metrics can be instantiated and accessed from one single place. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	77a2580052	fix: Add data race protection for the tm.wg var (#12100)	1 year ago
Christian Haudum	184d1de4db	chore(blooms): Use `bloomshipper.Interval` instead of `model.Interval` (#12079) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	637c292cd8	fix(blooms): Fix issue when loading cache on startup if directory does not exist (#12080) ... The check `!dirEntry.IsDir()` panicked in case `dirEntry` is `nil`, which is the case when the root directory does not exist. The PR also adds more logging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Danny Kopping	b5f488caec	fix(query/query-frontend): correct congestion control stats aggregation (#12072) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Danny Kopping	609fdb3214	feat(querier/query-frontend): track and log congestion control latency (#12058) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Shantanu Alshi	b1a61b3df6	chore: Change local setup under dev to use tsdb and v13 schema (#12057) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	60f75f3395	fix(blooms): closes chunk iterator (#12053)	1 year ago
Christian Haudum	cb7e016d64	chore(blooms): Populate blocks cache on startup (#12056) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840)	1 year ago
Christian Haudum	2f54f8d5be	chore(blooms): Implement BloomStore as a service (#12044) ... The bloomstore service is instantiated once at startup and used as dependency for both the bloom compactor and the bloom gateway. This will prevent instantiating the store in different ways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	7c8992f4e4	fix(blooms): fix reference leak and resulting race condition in BloomPageDecoder (#12050)	1 year ago
Salva Corts	8193063a4a	feat: Convert line filter expressions to a set of bloom tests (#12035)	1 year ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037)	1 year ago
Christian Haudum	dbea8ba85b	chore(blooms): Various fixes in blooms read path (#12036) ... * Wire up bloom querier metrics to fix nil pointer panic * Use `bloomutils.TokenRangesForInstance` instead of `subRing.GetTokenRangesForInstance` to avoid `zone not set` error Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	b9ce005ec1	fix: Ensure working dir for bloomstore exists (#12019) ... Fail startup if directory does not exist or there are not enough permissions. This prevents the bloomstore to fail later in the process once it tries to download and extract blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	32a9a3f45d	chore(blooms): Improve how block directory is extracted (#12030) ... This PR contains two changes: 1) Extracting the block without an intermediate temp file. The commit is cherry-picked from https://github.com/grafana/loki/pull/12021, but that PR got merged into the wrong branch 🙈 2) Strip `.tar.gz` suffix from block file when extracting it into a directory. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	1f56da206d	refactor: Simplify fingerprint partitioning in bloom gateway client (#12018) ... Use the recently introduced `tokenRangesForInstance()` function to determine the the ranges for each replication set. For this purpose, the function moves into the `bloomutils` packages. This PR eliminates the structs `addrsWithBounds` and `instanceWithFingerprints` (which had somewhat similar usage) and unifies them into `rsWithRanges` that holds both the fingerprint ranges and the assigned GroupedChunkRefs. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	edc9a44f66	test: Finish sync work for mockIngester.pushed var (#12028)	1 year ago
Christian Haudum	b47f2bfbc6	refactor: Limit processing of blocks to requested fingerprint ranges in bloom gateway (#11987) ... This PR limits the amount of data being processed for a single multiplexed iteration to the union of the fingerprint bounds of its requests, instead of looking at blocks from the complete fingerprint range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1dce7b512c	fix(blooms): race condition in `BloomPageDecoder` (#12027)	1 year ago
Owen Diehl	e835b6192f	refactor(blooms): refactors `blockLoadingIter`, exposing more error cases + additional metrics (#12015)	1 year ago
Christian Haudum	b9d8eefe27	test: Use table prefix in bloomstore tests (#12022) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	09aa0f80b5	Blooms/contiguous bounds union (#12009)	1 year ago
Callum Styan	b91bcecb3a	quantile sharding; fix bug related to # of steps (instant query mem usage) and use CollapsingLowestDense store for ddsketch (#11905) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Owen Diehl	166229818d	[Blooms] Consistent hashing via tokens for bloomcompactor (#12002)	1 year ago
Danny Kopping	eb1379afb9	Remove unnecessary ingester fields from metrics.go which only work in single binary (#12005) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Christian Haudum	71fa802c86	Map ring token keyspace (uint32) into fingerprint keyspace (uint64) (#11975) ... In order to compare the keyspace from the bloom gateway's ring with the fingerprints, they need to use the same keyspace. The ring, however, uses tokens within a `uint32` keyspace, whereas the fingerprints use a `uint64` keyspace. Therefore the ranges from the ring tokens need to be mapped into the `uint64` keyspace. This is done bit-shifting the Min value 32 bits to the left and the Max value 32 bits to the left plus adding `(1<<32)-1` to "fill" the remaining values up to the next Min value. The structs used to combined ring instance information and token/fingerprint ranges are yet to be simplified. However, this is not goal of this PR. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Salva Corts	620488656f	Set tracing.profiling-enabled to true by default (#12004)	1 year ago
Danny Kopping	bdea0b6df1	Add o11y for chunk data retrieved from ingesters (#12003) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Kaviraj Kanagaraj	fac5997b18	feat: Support split align and caching for instant metric query results (#11814) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Salva Corts	b7cb85f92b	Tail queue (#11930)	1 year ago
Christian Haudum	38c09a09bd	Bloom gateway: Fix metric for fetched blocks (#11988) ... This PR fixes the metric for fetched blocks Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979)	1 year ago
Christian Haudum	9f86473b0a	Various bloom gateway instrumentation changes (#11983) ... See individual commit messages for details. * Improve help text for chunk removals metric * End counter metric with _total * Fix incorrect observation of failed tasks * Observe chunk removals from block querier * Move chunks/series metrics into bloom querier * Observe total/filtered series and total/filtered chunks in bloom gateway Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	fb728a6781	(chore) Cleanup duplicate functions/strucs in bloomgateway package (#11978) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	ef40136715	Use TSDB index prefix on blooms directory (#11977)	1 year ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974)	1 year ago
Christian Haudum	31711d5558	(chore) Bloom gateway: Improve fingerprint partitioning in client (#11971) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	f158b5bc3e	correctly set block iter when no overlapping blocks are found (#11973)	1 year ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966)	1 year ago
Owen Diehl	443720f47b	tsdb parsing handles uint (#11969)	1 year ago
JordanRushing	cad4b8e749	Reduce Distributor auto-forget unhealthy cycles from 10->2 (#11935) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago