apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492 ) In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 ) This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

Author

SHA1

Message

Date

Christian Haudum

62176339c4

Bloom querying: Propagate line filters from querier to bloom filter (#11492 )

In order to query the bloom filters with the search strings from the
user query, we need to propagate the line filters through the different
implementations of the store, namely through the GetChunks function.

In order to avoid converting data structures, the line filter from the
syntax tree is used as custom protobuf type, because it is passed from
querier to index gateway and from index gateway to bloom gateway.

So far, the line filters request have not been tokenized to n-grams.
This PR adds the ability to read the n-gram length from the block schema
at query time and tokenizes the line filters from the input just ahead of
searching the block.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

2 years ago

Christian Haudum

c4f5a57bc8

Bloom Compactor: Optimize check for fingerprint ownership (#11389 )

Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for
each Series of a TSDB index is very expensive, because it not only
creates the tenant's sub-ring but also needs to check the fingerprint
against it.

Instead, we can pre-calculate the current instance's token ranges and
check if the (uint32 converted) fingerprint is contained within these
ranges.
 

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

2 years ago

Christian Haudum

d62d4e37d1

Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 )

This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component.
Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. 

---------

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>

2 years ago

3 Commits (b32d99308da6883bc1e7f321bb82483b69ada001)