Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Christian Haudum 5846ea2ea0 fix(blooms): Match series to newest block only (#15481) ... **What this PR does / why we need it**: While running bloom filters in production we noticed some Loki clusters that showed a very high percentage of missing chunks when querying blooms, thus resulting in lower filter rate. The reason is that old, superseded blocks are still considered up-to-date, because they cover a keyspace that is not covered by newer blocks with smaller keyspaces (due to larger individual series). ``` | series fingerprint keyspace ------------+---------------------------------------------------------------- | o o o o o o o o o o o o o ------------+---------------------------------------------------------------- iteration 1 | 111111111111111111111111111111111111111111111111111111111 iteration 2 | 22222222222222 3333333333333333 444444 iteration 3 | 5555555 6666666 77777777 888888888 9999999999 ... up-to-date | 555555522266666661111177777777333388888888811119999999999 ------------+---------------------------------------------------------------- | x ``` The chart shows the different blocks marked with the numbers 1 to 9 for a subset of the full series fingerprint keyspace. The blocks are generated in multiple successive bloom building iterations. The first block covers a larger keyspace (more series), because the individual blooms in the blocks are smaller in the beginning of the day. Later, the blooms get larger and therefore the block fingerprint ranges gets smaller. However, since we are dealing with fingerprint ranges, not individual fingerprints, the newer blocks cause "gaps" in the range of the previously larger keyspace. In the case above, every block except block 4, are considered up-to-date, since each of them covers a keyspace that is otherwise not covered. When resolving blocks for a series at query time, we consider looking at all up-to-date blocks, which are referenced by the meta files. The series `x` in the chart shows, that it is within the range of 3 up-to-date blocks: 1, 2, 5. However, only the newest block (5) may contain the requested series. This PR changes the block resolver on the index-gateway to only match the newest block to a series, based on the timestamp of the TSDB from with the blocks were generated. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>		1 year ago
..
batch.go	refactor: replace experimental `maps` and `slices` with stdlib (#15051)	1 year ago
batch_test.go	chore(bloom): remove unused code from blooms (#14539)	1 year ago
builder.go	fix(blooms): Match series to newest block only (#15481)	1 year ago
builder_test.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
config.go	feat(blooms): Prefetch bloom blocks as soon as they are built (#15050)	1 year ago
metrics.go	chore: use loki comntant for metrics namespace (#15034)	1 year ago
spec.go	feat: Do not add empty blooms to offsets (#14577)	1 year ago
spec_test.go	chore(bloom): remove unused code from blooms (#14539)	1 year ago