Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Sandeep Sukhani 7f298ff72f avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209) ... **What this PR does / why we need it**: Avoid using bloomfilters for chunks deduplication in tsdb `Stats` calls by avoiding fetching duplicate entries. The idea is to split and align queries by [ObjectStorageIndexRequiredPeriod](61794710a7/pkg/storage/config/schema_config.go (L47)) and make each split process chunks with a start time >= start time of the table interval. In other terms, table interval that contains start time of the chunk, owns it. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-12, 11-13. Query with range 6-15 would be split into 6-10, 10-15. query1 would process chunks 5-7, 8-12 and query2 would process chunks 11-13. This check is not applied for the first split so that we do not eliminate any chunks that overlaps the original query intervals but starts at the previous table. For e.g. if the table interval is 10s, and we have chunks 5-7, 8-13, 14-13. Query with range 11-12 should process chunk 8-13 even though its start time <= start time of table we will query for index. The caveat here is that we will overestimate the data we will be processing if the index is not compacted yet since it could have duplicate chunks when RF > 1. I think it is okay since the Stats call is just an estimation and need not be accurate. Removing all the extra processing saves us quite a bit of CPU and memory, as seen from the benchmark comparison between the two implementations: ``` name old time/op new time/op delta IndexClient_Stats-10 187µs ± 0% 34µs ± 1% -82.00% (p=0.008 n=5+5) name old alloc/op new alloc/op delta IndexClient_Stats-10 61.5kB ± 4% 12.5kB ± 2% -79.69% (p=0.008 n=5+5) name old allocs/op new allocs/op delta IndexClient_Stats-10 1.46k ± 0% 0.48k ± 0% -67.28% (p=0.008 n=5+5) ``` **Checklist** - [x] Tests updated		3 years ago
..
index	avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209)	3 years ago
indexshipper	Remove duplicated `loki_boltdb_shipper` prefix from `tables_upload_operation_total` metric (#7040)	3 years ago
series	refactor index storage code to index reader/writer interfaces for better monitoring (#7154)	3 years ago
shipper	compactor: prefetch existing compacted user index files in parallel (#6803)	3 years ago
tsdb	avoid using bloomfilters for chunks in stats calls by avoiding duplicates (#7209)	3 years ago
composite_store.go	refactor index storage code to index reader/writer interfaces for better monitoring (#7154)	3 years ago
composite_store_entry.go	refactor index storage code to index reader/writer interfaces for better monitoring (#7154)	3 years ago
composite_store_test.go	refactor index storage code to index reader/writer interfaces for better monitoring (#7154)	3 years ago
series_store_write.go	refactor index storage code to index reader/writer interfaces for better monitoring (#7154)	3 years ago