1005 Commits (bbbd82bc73322d662ba81efeda3884efcdc09708)

Author	SHA1	Message	Date
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	1 year ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	8afdfd500c	chore: Rename `HeapIterator` to `MergeEntryIterator` (#13975) ... Stumbled across this iterator interface and was confused about its name. It leaked its implementation details, and additionally that was not even correct any more, since the implementation has changed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
renovate[bot]	246a1dfbe2	fix(deps): update module github.com/ncw/swift to v2 (#13951) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
renovate[bot]	d43b2de1b4	fix(deps): update module github.com/fsouza/fake-gcs-server to v1.47.7 (#13935) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Dylan Guedes	3c9c647922	fix: Read "404" as object not exist (#13901) ... **What this PR does / why we need it**: Modify the s3 client to recognize "NotFound" errors when checking if an object error is "NotFound". I did some tests locally and `Head` calls to objects that doesn't exist gives a "NotFound" instead of "NoSuchKey".	1 year ago
Vladyslav Diachenko	bb257f54b3	feat(loki): add ability to disable AWS S3 dualstack endpoints usage (#13785) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Cyril Tovena	917053a730	feat: Introduce wal segment read path. (#13695) ... Co-authored-by: Ben Clive <ben.clive@grafana.com>	1 year ago
Salva Corts	55c64991db	feat(blooms): Add series & chunks per block metrics (#13721)	1 year ago
Ed Welch	b5584360d3	chore: add congestion control to AWS (#13691) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Trevor Whitney	5a87ccb648	feat: instrument failed chunk encoding/decoding (#13684)	1 year ago
Cyril Tovena	d42476aa58	fix: Handle block offset exceeding chunk length in memchunk.go (#13661)	1 year ago
George Robinson	8a3ae223ba	fix: fix a bug where AppendRequest with no entries triggers flush (#13672)	1 year ago
洪阿南	bd20171975	fix(blooms): Improve error wrap to make ignoreNotFound work when fetching blocks (#13656) ... Fix an issue where the `options.ignoreNotFound` of the fetcher is not effective. The root cause of this issue is that the error returned by `fmt.Errorf()` in `BloomClient` loses the underlying cause. As a result, `S3ObjectClient.IsObjectNotFoundErr()` fails to determine if the underlying cause of the returned error is of type `awserr.Error`.	1 year ago
George Robinson	2dfc13bb73	feat: add loki_ingester_rf1_segment_age_seconds metric (#13653)	1 year ago
George Robinson	9f197b68cc	chore: move timestamps into SegmentWriter (#13644)	1 year ago
benclive	d9c441efcc	feat: Implement GetObjectRange for all storage providers (#13650)	1 year ago
George Robinson	7ed63ea7f4	feat: remove mutexes from wal.SegmentWriter (#13641)	1 year ago
George Robinson	4abb5a404c	feat: Calculate the age of a WAL segment (#13637)	1 year ago
Cyril Tovena	5f5fd4e4ab	feat(rf1): Store index ref in metastore (#13613)	1 year ago
Christian Haudum	2624a4bdd4	fix(blooms): Use correct key to populate blockscache at startup (#13624) ... The cache key for block directories in the blocks cache are the block's address without the directory prefix. This is how the directory is put to the LRU cache after downloading and extracting the block tarball. This PR fixes the incorrect cache key used to populate the cache from disk on startup, which contained the file system directory prefix. Since the cached item from startup is never used, it gets evicted first in case of a full cache, or due to its TTL. This causes also the underlying directory on disk to be deleted, which can however still be referenced from the correct cache key for that directory from a later download. That caused the error `getting index reader: opening series file: open /path/to/block/series: no such file or directory` when trying to query the block, because the correct cache key was still present. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Cyril Tovena	04613b492f	feat: Instrument metastore GRPC calls (#13598)	1 year ago
Cyril Tovena	0b474981dc	feat: Bootstrap metastore for wal segments (#13550)	2 years ago
Christian Haudum	c9950e394d	fix(blooms): Fix eviction of multiple blockcache items (#13573) ... This commit fixes a bug in the eviction of least recently used items. When an element from the doubly-linked-list of the cache is removed, then the element's prev and next items become nil. Therefore the call elem.Prev() always broke the loop's condition for checking entries to be evicted. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
George Robinson	d3d9923410	chore: Rename PendingItem to PendingSegment and clean up flush.go (#13554)	2 years ago
George Robinson	d8eb9dd9be	chore: Remove redundant comments (#13553)	2 years ago
Christian Haudum	7aa7c15f21	chore: Use filesystem backed writer for blooms (#13522) ... The DirectoryBlockWriter and DirectoryBlockReader are used to avoid OOMing of compactors/builders. The tradeoff is that the writer/reader needs to be cleaned up and that it is I/O bound. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	aeb23bb7fc	fix: Fix log line for fingerprint not found (#13555)	2 years ago
George Robinson	8ef86f8d8b	chore: refactor WAL Manager (#13551)	2 years ago
George Robinson	e2cbde884f	chore: Clean up manager_test.go (#13549)	2 years ago
George Robinson	6acb51d485	chore: Use github.com/coder/quartz instead of time (#13542)	2 years ago
George Robinson	a03e3d39d4	chore: Use pool of bytes.Buffer instead of io.Pipe (#13543)	2 years ago
George Robinson	8ca03a2a3d	feat: Remove flush loop and queue from Ingester RF-1 (#13538)	2 years ago
Salva Corts	b44517a9f1	test: Add logging for empty blooms (#13537)	2 years ago
George Robinson	d4179aa04f	feat: Add metrics for Ingester RF-1 (#13510)	2 years ago
George Robinson	07c3c761ab	chore: Improve test coverage of WAL Manager (#13498)	2 years ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	2 years ago
Owen Diehl	bfa69556af	fix(blooms): skip empty blooms on reads (#13500) ... Read path safeguard to prevent filtering chunks associated with empty blooms while we work on finding the underlying cause of empty blooms in compaction.	2 years ago
Christian Haudum	652ad2455c	perf(blooms): Avoid tiny string allocations for insert cache (#13487) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
George Robinson	583f7f3388	chore: Remove un-used errors from WAL Manager (#13496)	2 years ago
George Robinson	aa55c690f9	chore: Small fixes to comments and ErrFull in manager.go (#13494)	2 years ago
George Robinson	bfe97d724f	feat: Add metrics to WAL Manager (#13490)	2 years ago
George Robinson	15c8b45d26	feat: WAL Manager (#13428)	2 years ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	6ea83b45b5	fix: Properly reset wal segment writer (#13468)	2 years ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	2 years ago
Christian Haudum	69b805d8ff	test: Fix flaky congestion test (#13410) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
benclive	973aa2d7ba	feat: Use prefixed WAL storage path in Object Storage (#13377)	2 years ago