986 Commits (7ed63ea7f493dab6c81200ddb7a0e6f26dc41bec)

Author	SHA1	Message	Date
George Robinson	7ed63ea7f4	feat: remove mutexes from wal.SegmentWriter (#13641)	10 months ago
George Robinson	4abb5a404c	feat: Calculate the age of a WAL segment (#13637)	10 months ago
Cyril Tovena	5f5fd4e4ab	feat(rf1): Store index ref in metastore (#13613)	10 months ago
Christian Haudum	2624a4bdd4	fix(blooms): Use correct key to populate blockscache at startup (#13624) ... The cache key for block directories in the blocks cache are the block's address without the directory prefix. This is how the directory is put to the LRU cache after downloading and extracting the block tarball. This PR fixes the incorrect cache key used to populate the cache from disk on startup, which contained the file system directory prefix. Since the cached item from startup is never used, it gets evicted first in case of a full cache, or due to its TTL. This causes also the underlying directory on disk to be deleted, which can however still be referenced from the correct cache key for that directory from a later download. That caused the error `getting index reader: opening series file: open /path/to/block/series: no such file or directory` when trying to query the block, because the correct cache key was still present. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Cyril Tovena	04613b492f	feat: Instrument metastore GRPC calls (#13598)	10 months ago
Cyril Tovena	0b474981dc	feat: Bootstrap metastore for wal segments (#13550)	10 months ago
Christian Haudum	c9950e394d	fix(blooms): Fix eviction of multiple blockcache items (#13573) ... This commit fixes a bug in the eviction of least recently used items. When an element from the doubly-linked-list of the cache is removed, then the element's prev and next items become nil. Therefore the call elem.Prev() always broke the loop's condition for checking entries to be evicted. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
George Robinson	d3d9923410	chore: Rename PendingItem to PendingSegment and clean up flush.go (#13554)	10 months ago
George Robinson	d8eb9dd9be	chore: Remove redundant comments (#13553)	10 months ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Salva Corts	aeb23bb7fc	fix: Fix log line for fingerprint not found (#13555)	10 months ago
George Robinson	8ef86f8d8b	chore: refactor WAL Manager (#13551)	10 months ago
George Robinson	e2cbde884f	chore: Clean up manager_test.go (#13549)	10 months ago
George Robinson	6acb51d485	chore: Use github.com/coder/quartz instead of time (#13542)	10 months ago
George Robinson	a03e3d39d4	chore: Use pool of bytes.Buffer instead of io.Pipe (#13543)	10 months ago
George Robinson	8ca03a2a3d	feat: Remove flush loop and queue from Ingester RF-1 (#13538)	10 months ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	11 months ago
George Robinson	d4179aa04f	feat: Add metrics for Ingester RF-1 (#13510)	11 months ago
George Robinson	07c3c761ab	chore: Improve test coverage of WAL Manager (#13498)	11 months ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	11 months ago
Owen Diehl	bfa69556af	fix(blooms): skip empty blooms on reads (#13500) ... Read path safeguard to prevent filtering chunks associated with empty blooms while we work on finding the underlying cause of empty blooms in compaction.	11 months ago
Christian Haudum	652ad2455c	perf(blooms): Avoid tiny string allocations for insert cache (#13487) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
George Robinson	583f7f3388	chore: Remove un-used errors from WAL Manager (#13496)	11 months ago
George Robinson	aa55c690f9	chore: Small fixes to comments and ErrFull in manager.go (#13494)	11 months ago
George Robinson	bfe97d724f	feat: Add metrics to WAL Manager (#13490)	11 months ago
George Robinson	15c8b45d26	feat: WAL Manager (#13428)	11 months ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	6ea83b45b5	fix: Properly reset wal segment writer (#13468)	11 months ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	11 months ago
Christian Haudum	69b805d8ff	test: Fix flaky congestion test (#13410) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	973aa2d7ba	feat: Use prefixed WAL storage path in Object Storage (#13377)	11 months ago
Vladyslav Diachenko	33bd390f19	chore: adjusted SegmentWriter to the reader conversion (#13342) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Ben Clive <ben.clive@grafana.com>	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	04bc3a423c	fix(blooms): ensure tokenizer cache is reset between series (#13370)	11 months ago
Vladyslav Diachenko	19c050926e	feat: convert WalSegmentWriter to io.ReadSeeker (#13340) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Salva Corts	0c289a83df	refactor(blooms): Do not use sleeps on integration test (#13308)	11 months ago
Owen Diehl	0cb3ff1830	perf(blooms): always return bloom pages to allocator (#13288)	11 months ago
Cyril Tovena	debb5f202e	feat(wal): Benchmark and improve WAL writes using Reset. (#13272)	11 months ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255) ... Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	11 months ago
Christian Haudum	dbc0e2489c	chore(index-gateway): Improve instrumentation of index download/sync (#13194) ... This PR contains changes to improve the instrumentation and obervability of the download/sync operations that happen on the index gateways (index shipper). This includes fixes to the download latency and wait time metrics. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ed Welch	28642b6d32	chore(index-gateway): Download then extract in separate steps (#13202) ... This provides additional timing information about what step takes how long. Additionally, the `awaitReady()` function is gated with a timeout context to avoid overly long waiting times. Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	41fbacdce7	feat(wal): Add sizing information to writer and reader. (#13267)	11 months ago
Christian Haudum	4117b6ca98	perf: Re-introduce fixed size memory pool for bloom querier (#13172) ... This PR re-introduces the fixed size memory pool that was originally introduced with #13039 but reverted with #13162 Additionally, it removes the package-global variable to store the type of allocator that should be used. Instead, the allocator type is passed during the bloom store creation. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	1d6f8d51fc	feat: Introduce a new Object Storage WAL format. (#13253)	11 months ago
Owen Diehl	1086783a1d	fix: separates directory creation from permission checks (#13248)	11 months ago
Karsten Jeschkies	ca917b3ec0	chore: Update go.etcd.io/bbolt to v1.3.8 (#13227)	11 months ago
洪阿南	f98ff7f584	fix: Fix duplicate enqueue item problem in bloom download queue when do sync download (#13114) ... A missing return will cause item enqueued twice in sync download scenario. That cause the response not like expected. Suppose we want to fetch a blocks array: [0, 1, 2]. Enqueued items are [0, 0, 1, 1, 2, 2]，If we want response sorted we wait 3 times response will get [0, 0, 1]	12 months ago
Salva Corts	9c96d26895	refactor(blooms): Delete outdated metas (#13153)	12 months ago