1037 Commits (0c39e1529fc0bf1d7c3eff5435f3045a70dda776)

Author	SHA1	Message	Date
Trevor Whitney	7983f94b15	feat: detected field values (#14350)	8 months ago
Robert Fratto	e9efcb57f9	chore(storage/bloom/v1): bind bloom page size to max bloom size (#14372) ... Previously, the bloom page size was hard-coded to 256KB. However, 256KB is much smaller than the normal bloom size, leading to nearly every bloom page to be overfilled with a single, massive bloom. Binding the bloom page size to the maximum bloom size allows bloom pages to contain multiple smaller blooms (which are still larger than 256KB) or one maxed out bloom. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Dylan Guedes	2f56f50cc6	fix: Transform `ObjectExistsWithSize` into `GetAttributes` (#14329) ... - Transform our `ObjectExistsWithSize` API into `GetAttributes` to conform with the existing Thanos API. - Fix we not returning the size when we should on our s3 implementation of `ObjectSize` - Fix not returning `false, nil` to `ObjectExists` when the only error seen is a `NotFound`. If someone calls `ObjectExists` it should return `false, nil` to a `NotExist` error, instead of `false, err`	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	a584fb713d	chore: sync tables without acquiring read lock the whole time (#14179)	8 months ago
Paul Rogers	c874d2cc44	chore: Wrap error to surface more context (#14284)	8 months ago
Dylan Guedes	ac422b3bc3	feat: Introduce new `ObjectExistsWithSize` API to (#14268) ... **What this PR does / why we need it**: Introduce a new `ObjectExistsWithSize` API to our object storage interface. This is the same as `ObjectExists` but with the object size as part of the return value. This is useful to compare the object size present.	8 months ago
Christian Haudum	14b468b236	chore: Add `has_labelfilter_before_parser` to metrics.go log line (#14244) ... The value of this field indicates that the query could potentially benefit from bloom filters on structured metadata, if the queried field is a structured metadata field. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Paul Rogers	a4642b55e9	fix: Update AWS storage timeout error for Go 1.23 behavior (#14226)	8 months ago
Paul Rogers	71d7291c9c	fix: Expand matching for additional variations (#14221)	8 months ago
Paul Rogers	8322e518e6	fix: Add additional validation for timeout while retrieving headers (#14217)	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Paul Rogers	fc90a63636	feat: implement IsRetryableErr for S3ObjectClient (#14174) ... **What this PR does / why we need it**: Adds support for `IsRetryableErr`in the `S3ObjectClient`, similar to what is found in the `GCSObjectClient`	8 months ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Robert Fratto	1b7086632e	chore(storage/bloom/v1): update comments on bloom test (#14168)	8 months ago
Christian Haudum	02b26a85df	chore: Remove outdate TODO file for bloom package (#14158) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6e36041c08	chore(blooms): Remove excessive logging in fused querier (#14152) ... The FusedQuerier emits a large amount of log lines at level WARN if a bloom is empty. Since the introduction of structured metadata blooms, this happens every time a series does not have structured metadata in any of its entries. In the future, an optimisation could be to not write blooms for series at all, if they contain no data. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	39fa9b47e9	chore: Fix linter errors (#14145) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	89c4282654	chore: avoid some allocations in tsdb while reading posting offset table (#14111)	9 months ago
Robert Fratto	53cfef3bca	chore(storage/bloom/v1): add ExtractTestableLabelMatchers (#14119) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Paul Rogers	f226b598a4	chore: Final batch of linting cleanup for Go 1.23 and newer golangci (#14120)	9 months ago
Sven Grossmann	93009d4e8c	fix(detected_fields): always return empty array as `null` (#14112)	9 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
dropbigfish	5f78784d4f	chore: fix some function names (#14020) ... Signed-off-by: dropbigfish <fillfish@foxmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Cyril Tovena	d178f4c7e2	feat: Implement WAL segment ingestion via Kafka with partition ring (#14043) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	9 months ago
Paul Rogers	20ac1bacd8	chore(deps): Downgrade fake-gcs-server (#14078) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Paul Rogers	73cbbb0f22	feat: add retries for s3 ObjectExists calls (#14062) ... **What this PR does / why we need it**: It was determined via careful examination that Loki would issue retries for S3 `GetObject` calls, but would not retry calls to `ObjectExists`. For transient issues (e.g. rate-limiting), it makes sense to retry the command to see if an object does exist. If the object is not found (meaning, a 404 return code, and a successful query of the S3 storage to show the object does not exist), the retries are not leveraged, as it is not needed.	9 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038) ... A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	8afdfd500c	chore: Rename `HeapIterator` to `MergeEntryIterator` (#13975) ... Stumbled across this iterator interface and was confused about its name. It leaked its implementation details, and additionally that was not even correct any more, since the implementation has changed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
renovate[bot]	246a1dfbe2	fix(deps): update module github.com/ncw/swift to v2 (#13951) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	9 months ago
renovate[bot]	d43b2de1b4	fix(deps): update module github.com/fsouza/fake-gcs-server to v1.47.7 (#13935) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	9 months ago
Dylan Guedes	3c9c647922	fix: Read "404" as object not exist (#13901) ... **What this PR does / why we need it**: Modify the s3 client to recognize "NotFound" errors when checking if an object error is "NotFound". I did some tests locally and `Head` calls to objects that doesn't exist gives a "NotFound" instead of "NoSuchKey".	10 months ago
Vladyslav Diachenko	bb257f54b3	feat(loki): add ability to disable AWS S3 dualstack endpoints usage (#13785) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	10 months ago
Cyril Tovena	917053a730	feat: Introduce wal segment read path. (#13695) ... Co-authored-by: Ben Clive <ben.clive@grafana.com>	10 months ago
Salva Corts	55c64991db	feat(blooms): Add series & chunks per block metrics (#13721)	10 months ago
Ed Welch	b5584360d3	chore: add congestion control to AWS (#13691) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	10 months ago
Trevor Whitney	5a87ccb648	feat: instrument failed chunk encoding/decoding (#13684)	10 months ago
Cyril Tovena	d42476aa58	fix: Handle block offset exceeding chunk length in memchunk.go (#13661)	10 months ago
George Robinson	8a3ae223ba	fix: fix a bug where AppendRequest with no entries triggers flush (#13672)	10 months ago
洪阿南	bd20171975	fix(blooms): Improve error wrap to make ignoreNotFound work when fetching blocks (#13656) ... Fix an issue where the `options.ignoreNotFound` of the fetcher is not effective. The root cause of this issue is that the error returned by `fmt.Errorf()` in `BloomClient` loses the underlying cause. As a result, `S3ObjectClient.IsObjectNotFoundErr()` fails to determine if the underlying cause of the returned error is of type `awserr.Error`.	10 months ago
George Robinson	2dfc13bb73	feat: add loki_ingester_rf1_segment_age_seconds metric (#13653)	10 months ago
George Robinson	9f197b68cc	chore: move timestamps into SegmentWriter (#13644)	10 months ago