18 Commits (40ee7667244f2e094b5a7199705b4f3dacb7ffaf)

Author	SHA1	Message	Date
Christian Haudum	d4fcef5229	chore(blooms): Some boom gateway cleanups (#13165) ... * Cleanup logic for processing multiple days in single `FilterChunkRefs` request in the bloom gateway. The handler already returned an error if the requested chunk refs spanned across multiple days, but the logic for processing multiple days/tasks was still in place. This commit simplifies the logic to only process a single task per request. * Remove unused package global variable * Remove bloomgateway package comment * Remove duplicate imports * Avoid conversion from *logproto.ShortRef to v1.ChunkRef and istead cast the value of the pointer * Simplify `getFromThrough()` function Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	fbe7c559b5	feat(blooms): Blooms/v2 encoding multipart series (#13093)	2 years ago
Christian Haudum	738c274a58	fix(blooms): Fix `partitionSeriesByDay` function (#12900) ... The function `partitionSeriesByDay` could yield empty when chunks were outside of the requested time range. This would result in requests for a time range that does not contain any chunks for a series to filter. These requests would return errors, because a subsequent `partitionSeriesByDay` for would remove that day resulting in no task. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720) ... This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	cf71ac7114	fix(blooms): Remove blocks not matching any series in task (#12401)	2 years ago
Salva Corts	49f8f7b3ee	refactor: Chunkref as alias to logproto.ShortRef (#12159)	2 years ago
Owen Diehl	fd702f6691	chore(blooms): remove unused helper from bloomgateway (#12148)	2 years ago
Christian Haudum	184d1de4db	chore(blooms): Use `bloomshipper.Interval` instead of `model.Interval` (#12079) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037)	2 years ago
Christian Haudum	fb728a6781	(chore) Cleanup duplicate functions/strucs in bloomgateway package (#11978) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917) ... Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	6871f3c555	Bloom Gateway: Partition requests into multiple tasks before enqueuing (#11768) ... **What this PR does / why we need it**: This PR changes at which point a request is split into multiple tasks in case the chunks boundaries of the request overlap multiple days. This solves the problem that the request handler needs to know for how many responses from the block querier needs to wait. Previously, a single task that spanned across multiple days was executed multiple times (once for each matching day). --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	600e390b16	Extend `TestPartitionFingerprintRange` test case (#11750) ... **What this PR does / why we need it**: The test cases now also cover partially out of bounds blocks as well as inconsecutive block ranges. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492) ... In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389) ... Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	79693d79ef	Fix `getFromThrough()` function for ShortRefs (#11386) ... **What this PR does / why we need it**: Since chunks don't necessarily have the same duration, we cannot assume that the last item of a slice sorted by `From` time also has the greatest `Through` time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181) ... This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago