Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Christian Haudum 5846ea2ea0 fix(blooms): Match series to newest block only (#15481) ... **What this PR does / why we need it**: While running bloom filters in production we noticed some Loki clusters that showed a very high percentage of missing chunks when querying blooms, thus resulting in lower filter rate. The reason is that old, superseded blocks are still considered up-to-date, because they cover a keyspace that is not covered by newer blocks with smaller keyspaces (due to larger individual series). ``` | series fingerprint keyspace ------------+---------------------------------------------------------------- | o o o o o o o o o o o o o ------------+---------------------------------------------------------------- iteration 1 | 111111111111111111111111111111111111111111111111111111111 iteration 2 | 22222222222222 3333333333333333 444444 iteration 3 | 5555555 6666666 77777777 888888888 9999999999 ... up-to-date | 555555522266666661111177777777333388888888811119999999999 ------------+---------------------------------------------------------------- | x ``` The chart shows the different blocks marked with the numbers 1 to 9 for a subset of the full series fingerprint keyspace. The blocks are generated in multiple successive bloom building iterations. The first block covers a larger keyspace (more series), because the individual blooms in the blocks are smaller in the beginning of the day. Later, the blooms get larger and therefore the block fingerprint ranges gets smaller. However, since we are dealing with fingerprint ranges, not individual fingerprints, the newer blocks cause "gaps" in the range of the previously larger keyspace. In the case above, every block except block 4, are considered up-to-date, since each of them covers a keyspace that is otherwise not covered. When resolving blocks for a series at query time, we consider looking at all up-to-date blocks, which are referenced by the meta files. The series `x` in the chart shows, that it is within the range of 3 up-to-date blocks: 1, 2, 5. However, only the newest block (5) may contain the requested series. This PR changes the block resolver on the index-gateway to only match the newest block to a series, based on the timestamp of the TSDB from with the blocks were generated. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>		1 year ago
..
bloomgateway.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
bloomgateway_test.go	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	1 year ago
cache.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
cache_test.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
client.go	refactor: replace experimental `maps` and `slices` with stdlib (#15051)	1 year ago
client_pool.go	chore(bloom): remove unused code from blooms (#14539)	1 year ago
client_pool_test.go	chore: Remove ring client pool from JumpHashClientPool (#14367)	1 year ago
client_test.go	fix(blooms): Suppress error from resolving server addresses for blocks (#13385)	2 years ago
config.go	chore(blooms): Add setting to fetch bloom blocks async or synchronously (#14862)	1 year ago
metrics.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
multiplexing.go	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	1 year ago
multiplexing_test.go	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	1 year ago
processor.go	chore(blooms): Add setting to fetch bloom blocks async or synchronously (#14862)	1 year ago
processor_test.go	chore(blooms): Add setting to fetch bloom blocks async or synchronously (#14862)	1 year ago
querier.go	fix(blooms): Fix check for skipping most recent data when filtering blooms (#15300)	1 year ago
querier_test.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
resolver.go	fix(blooms): Match series to newest block only (#15481)	1 year ago
resolver_test.go	fix(blooms): Match series to newest block only (#15481)	1 year ago
stats.go	chore(blooms): Add number of skipped bloom blocks to stats-report log line (#14771)	1 year ago
util.go	fix(blooms): Exclude label filters where label name is part of the series labels. (#14661)	1 year ago
util_test.go	chore(bloom): remove unused code from blooms (#14539)	1 year ago
worker.go	chore(blooms): Add setting to fetch bloom blocks async or synchronously (#14862)	1 year ago