1051 Commits (f411a0795af67630a0a70a88ce64fa071de50a56)

Author	SHA1	Message	Date
Joao Marcal	b87224647d	feat(storage): AWS backend using thanos.io/objstore (#11221) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	7 months ago
Salva Corts	51c42e8645	feat: Do not add empty blooms to offsets (#14577)	7 months ago
Joao Marcal	5824e3d35c	feat(storage): Azure backend using thanos.io/objstore (#11315) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	7 months ago
Ashwanth	cff9f43dd6	fix(s3): disable client retries when congestion control is enabled (#14588)	7 months ago
renovate[bot]	275c97cec7	fix(deps): update module github.com/fsouza/fake-gcs-server to v1.50.2 (#14313) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	7 months ago
Robert Fratto	bf54cf1496	chore(bloom): remove unused code from blooms (#14539)	7 months ago
Christian Haudum	f568dda136	chore: Improve logging of jumphash server selector (#14306) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ashwanth	11c4fa253c	chore: thanos client remove `object-store.` prefix (#14546)	7 months ago
Ashwanth	b405ae8557	chore: hide docs for thanos clients + restore original cli prefix (#14545)	7 months ago
Joao Marcal	c059ace53e	feat(storage): GCS backend using thanos.io/objstore (#11132) ... Signed-off-by: Joao Marcal <jmarcal@redhat.com> Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Christian Haudum	fdeec618f0	chore: Log errors when processing a download task fails (#14436) ... At the moment, errors are silently ignored when asynchronous blocks downloading fails. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	5f325aac56	fix(storage/chunk/client/aws): have GetObject check for canceled context (#14420)	8 months ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	8 months ago
Christian Haudum	3beb8ff9cf	fix(build): Use Debian Bullseye base image for build image (#14368) ... This change is necessary to allow the binaries built using the Loki build image to run on operating systems with older libc version. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Trevor Whitney	7983f94b15	feat: detected field values (#14350)	8 months ago
Robert Fratto	e9efcb57f9	chore(storage/bloom/v1): bind bloom page size to max bloom size (#14372) ... Previously, the bloom page size was hard-coded to 256KB. However, 256KB is much smaller than the normal bloom size, leading to nearly every bloom page to be overfilled with a single, massive bloom. Binding the bloom page size to the maximum bloom size allows bloom pages to contain multiple smaller blooms (which are still larger than 256KB) or one maxed out bloom. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Dylan Guedes	2f56f50cc6	fix: Transform `ObjectExistsWithSize` into `GetAttributes` (#14329) ... - Transform our `ObjectExistsWithSize` API into `GetAttributes` to conform with the existing Thanos API. - Fix we not returning the size when we should on our s3 implementation of `ObjectSize` - Fix not returning `false, nil` to `ObjectExists` when the only error seen is a `NotFound`. If someone calls `ObjectExists` it should return `false, nil` to a `NotExist` error, instead of `false, err`	8 months ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	a584fb713d	chore: sync tables without acquiring read lock the whole time (#14179)	8 months ago
Paul Rogers	c874d2cc44	chore: Wrap error to surface more context (#14284)	8 months ago
Dylan Guedes	ac422b3bc3	feat: Introduce new `ObjectExistsWithSize` API to (#14268) ... **What this PR does / why we need it**: Introduce a new `ObjectExistsWithSize` API to our object storage interface. This is the same as `ObjectExists` but with the object size as part of the return value. This is useful to compare the object size present.	8 months ago
Christian Haudum	14b468b236	chore: Add `has_labelfilter_before_parser` to metrics.go log line (#14244) ... The value of this field indicates that the query could potentially benefit from bloom filters on structured metadata, if the queried field is a structured metadata field. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Paul Rogers	a4642b55e9	fix: Update AWS storage timeout error for Go 1.23 behavior (#14226)	8 months ago
Paul Rogers	71d7291c9c	fix: Expand matching for additional variations (#14221)	8 months ago
Paul Rogers	8322e518e6	fix: Add additional validation for timeout while retrieving headers (#14217)	8 months ago
Christian Haudum	cdf084fdae	perf(blooms): Remove compression of `.tar` archived bloom blocks (#14159) ... Decompression is a CPU intensive task, especially un-gzipping. The gain of compressing a tar archive of storage optimized binary blocks is neglectable (question: is it?). In this example, the block of ~170MiB is ~3.3MiB bigger when not compressed, which is a ratio of ~2% ```bash $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar 177048 -rw-rw-r-- 1 christian christian 181293056 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar $ ls -las 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz 173732 -rw-rw-r-- 1 christian christian 177897689 Sep 18 13:49 2bc017f79711e12a-2bffc5dcc0e8e964_1726004114913-1726106283939-bc42f529.tar.gz $ qalc '(181293056 - 177897689) / 1024/ 1024' ((181293056 − 177897689) / 1024) / 1024 ≈ 3.238074303 $ qalc '181293056 / 177897689' 181293056 / 177897689 ≈ 1.019086066 ``` After some consideration, we decided to store the encoding of the bloom block in the `BlockRef`. This means, that the changes in this PR do not break compatibility with existing blocks compressed with gzip, although new blocks will not be compressed any more. However, the PR adds support for different compression algorithms, such as gzip, snappy, lz4, flate, and zstd. Compression is not configurable yet. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Paul Rogers	fc90a63636	feat: implement IsRetryableErr for S3ObjectClient (#14174) ... **What this PR does / why we need it**: Adds support for `IsRetryableErr`in the `S3ObjectClient`, similar to what is found in the `GCSObjectClient`	8 months ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183) ... This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167) ... Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Robert Fratto	1b7086632e	chore(storage/bloom/v1): update comments on bloom test (#14168)	8 months ago
Christian Haudum	02b26a85df	chore: Remove outdate TODO file for bloom package (#14158) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6e36041c08	chore(blooms): Remove excessive logging in fused querier (#14152) ... The FusedQuerier emits a large amount of log lines at level WARN if a bloom is empty. Since the introduction of structured metadata blooms, this happens every time a series does not have structured metadata in any of its entries. In the future, an optimisation could be to not write blooms for series at all, if they contain no data. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	39fa9b47e9	chore: Fix linter errors (#14145) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	89c4282654	chore: avoid some allocations in tsdb while reading posting offset table (#14111)	9 months ago
Robert Fratto	53cfef3bca	chore(storage/bloom/v1): add ExtractTestableLabelMatchers (#14119) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Paul Rogers	f226b598a4	chore: Final batch of linting cleanup for Go 1.23 and newer golangci (#14120)	9 months ago
Sven Grossmann	93009d4e8c	fix(detected_fields): always return empty array as `null` (#14112)	9 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
dropbigfish	5f78784d4f	chore: fix some function names (#14020) ... Signed-off-by: dropbigfish <fillfish@foxmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Cyril Tovena	d178f4c7e2	feat: Implement WAL segment ingestion via Kafka with partition ring (#14043) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	9 months ago
Paul Rogers	20ac1bacd8	chore(deps): Downgrade fake-gcs-server (#14078) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Paul Rogers	73cbbb0f22	feat: add retries for s3 ObjectExists calls (#14062) ... **What this PR does / why we need it**: It was determined via careful examination that Loki would issue retries for S3 `GetObject` calls, but would not retry calls to `ObjectExists`. For transient issues (e.g. rate-limiting), it makes sense to retry the command to see if an object does exist. If the object is not found (meaning, a 404 return code, and a successful query of the S3 storage to show the object does not exist), the retries are not leveraged, as it is not needed.	9 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038) ... A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	8afdfd500c	chore: Rename `HeapIterator` to `MergeEntryIterator` (#13975) ... Stumbled across this iterator interface and was confused about its name. It leaked its implementation details, and additionally that was not even correct any more, since the implementation has changed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago