3139 Commits (02b26a85df4cb02be477f014b769b3debbf07175)

Author	SHA1	Message	Date
Christian Haudum	02b26a85df	chore: Remove outdate TODO file for bloom package (#14158) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Cyril Tovena	44958c4b8b	chore(kafka): Refactor configurations and remove old modules (#14161)	8 months ago
Karsten Jeschkies	48e4a346b8	chore: Remove unused quantile merge step evaluator. (#14160)	8 months ago
Christian Haudum	6e36041c08	chore(blooms): Remove excessive logging in fused querier (#14152) ... The FusedQuerier emits a large amount of log lines at level WARN if a bloom is empty. Since the introduction of structured metadata blooms, this happens every time a series does not have structured metadata in any of its entries. In the future, an optimisation could be to not write blooms for series at all, if they contain no data. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
benclive	78b275bf10	fix: Propagate query stats from quantile & topk queries (#13831) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	9 months ago
Christian Haudum	39fa9b47e9	chore: Fix linter errors (#14145) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	5bffc10d85	chore(blooms): Replace extracting line filters with extracting label filters (#14137) ... Replace the logic for extracting line filters with extracting label filters on the bloom gateway client, because otherwise the bloom gateway server would receive requests with no label matchers. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	ebe0bb875e	chore: replace remaining references to ExtractTestableLineFilters (#14135)	9 months ago
Robert Fratto	4de51f9953	chore(bloomgateway): update bloom gateway to test for structured metadata blooms (#14130) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	89c4282654	chore: avoid some allocations in tsdb while reading posting offset table (#14111)	9 months ago
Sven Grossmann	63e84b476a	fix(aggregated-metrics): correctly create logfmt string (#14124)	9 months ago
Robert Fratto	53cfef3bca	chore(storage/bloom/v1): add ExtractTestableLabelMatchers (#14119) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Christian Haudum	e64124ec1a	chore(blooms): Expose series/stream metadata when iterating over blooms (#14097) ... The bloom block format stores metadata from indexing in the index part of the block (series file), which contains information, such as the fields (metadata names) that have been indexed in the bloom filter. This will be needed in the read path (bloom gateway) later on, to check whether a provided label name from the label filter expression was actually indexed or not. In this PR, the newly available metadata is used by the bloom inspect tool (located in ./tools/bloom/inspect/) to list the fields along the series, as well as the individual pages of the series that contain the actual scalable bloom filters. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Paul Rogers	f226b598a4	chore: Final batch of linting cleanup for Go 1.23 and newer golangci (#14120)	9 months ago
Sven Grossmann	93009d4e8c	fix(detected_fields): always return empty array as `null` (#14112)	9 months ago
Cyril Tovena	4da035b6b7	fix(kafka): Fixes writer initialization for arm32 (#14115)	9 months ago
Christian Haudum	8307c42c54	fix(blooms): Fix panic in initialisation of the bloom planner and builder (#14110) ... The targets `bloom-planner` and `bloom-builder` panic at startup on branch `main`. ```console $ make loki && ./cmd/loki/loki -target=bloom-builder CGO_ENABLED=0 go build -ldflags "-extldflags \"-static\" -s -w -X github.com/grafana/loki/v3/pkg/util/build.Branch=main -X github.com/grafana/loki/v3/pkg/util/build.Version=main-b29b4b4 -X github.com/grafana/loki/v3/pkg/util/build.Revision=b29b4b4bb -X github.com/grafana/loki/v3/pkg/util/build.BuildUser=christian@grafana1210 -X github.com/grafana/loki/v3/pkg/util/build.BuildDate=2024-09-11T08:30:11Z" -tags netgo -o cmd/loki/loki ./cmd/loki panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x287b590] goroutine 1 [running]: github.com/grafana/dskit/ring.(*BasicLifecycler).GetInstanceID(...) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/ring/basic_lifecycler.go:132 github.com/grafana/loki/v3/pkg/bloombuild/builder.New({{0x6400000, 0x6400000, {0x0, 0x0}, 0x0, 0x0, 0x0, {0x5f5e100, 0x2540be400, 0xa}, ...}, ...}, ...) /home/christian/sandbox/grafana/loki/pkg/bloombuild/builder/builder.go:92 +0x470 github.com/grafana/loki/v3/pkg/loki.(*Loki).initBloomBuilder(0xc002984008) /home/christian/sandbox/grafana/loki/pkg/loki/modules.go:1750 +0x4b6 github.com/grafana/dskit/modules.(*Manager).initModule(0xc000d2e6f0, {0x7ffe36edcb45, 0xd}, 0xc0027a22f8, 0xc0013ce960) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x1ea github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0xc000d2e6f0, {0xc000dbeed0, 0x1, 0x7510c18f88e2e5ce?}) /home/christian/sandbox/grafana/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xe8 github.com/grafana/loki/v3/pkg/loki.(*Loki).Run(0xc002984008, {0x0?, {0x4?, 0x2?, 0x6457fc0?}}) /home/christian/sandbox/grafana/loki/pkg/loki/loki.go:497 +0x97 main.main() /home/christian/sandbox/grafana/loki/cmd/loki/main.go:129 +0x131e ```` This is because `t.indexGatewayRingManager` is not nil and therefore incorrectly tries to initialize the ring watcher, which fails. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	a2fbaa8e09	feat(blooms)!: Index structured metadata into blooms (#14061) ... Instead of indexing n-grams of the log line content, we index the plain values of the structured metadata keys and values. Resulting tokens are: * `name` * `chunkPrefix + name` * `value` * `chunkPrefix + value` * `name + '=' value` * `chunkPrefix + name + '=' + value` Indexed fields (metadata name) are also extracted into the series metadata. Indexed metadata values cannot be queried with the bloom gateways yet. This PR does not cleanup unused code used for ngram-tokenization, it is scope for a follow up. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
dropbigfish	5f78784d4f	chore: fix some function names (#14020) ... Signed-off-by: dropbigfish <fillfish@foxmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Kason Braley	c5083c7f1f	perf(logql): Micro-optimizations for IP filter (#14072) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
George Robinson	4cd77b9747	chore: use quartz in metastore (#13900)	9 months ago
Cyril Tovena	d178f4c7e2	feat: Implement WAL segment ingestion via Kafka with partition ring (#14043) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	9 months ago
Ravishankar	55e374e85e	fix: skipping label if it contains special symbol (#14068) ... Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	9 months ago
Vladyslav Diachenko	c65721e7ad	feat: track discarded data by usageTracker (#14081) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	9 months ago
Paul Rogers	20ac1bacd8	chore(deps): Downgrade fake-gcs-server (#14078) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Sven Grossmann	0780456662	feat(max-allowed-line-length): add config to set `max-allowed-line-length` in pattern ingester (#14070) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Trevor Whitney	1b3ba530b8	fix: correct _extracted logic in detected fields (#14064) ... Co-authored-by: Sven Grossmann <svennergr@gmail.com>	9 months ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	9 months ago
Paul Rogers	73cbbb0f22	feat: add retries for s3 ObjectExists calls (#14062) ... **What this PR does / why we need it**: It was determined via careful examination that Loki would issue retries for S3 `GetObject` calls, but would not retry calls to `ObjectExists`. For transient issues (e.g. rate-limiting), it makes sense to retry the command to see if an object does exist. If the object is not found (meaning, a 404 return code, and a successful query of the S3 storage to show the object does not exist), the retries are not leveraged, as it is not needed.	9 months ago
Christian Haudum	5395daf898	chore(blooms)!: Introduce a new block schema (V3) (#14038) ... A new schema version V3 for bloom blocks that is incompatible with old schemas. It simplifies some parts of the code and makes the schema more extensible, laying the groundwork for indexing structured metadata fields into bloom filters. ⚠️ This PR breaks the bloom filter functionality! Until the read path is implemented, bloom blocks created with the new schema cannot be queries in the bloom gateways. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Trevor Whitney	4f962ef7af	fix: enable service detection for otlp endoint (#14036) ... Co-authored-by: Sven Grossmann <Svennergr@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	9 months ago
Sven Grossmann	aa1ac99f4d	fix(`detected_fields`): return parsed labels when parsers are passed (#14047) ... Co-authored-by: Galen Kistler <109082771+gtk-grafana@users.noreply.github.com> Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Paul Rogers	02205f480e	chore: Remove call to deprecated rand.Seed() (#14041)	9 months ago
Sandeep Sukhani	4e4174400f	fix: do not retain span logger created with index set initialized at query time (#14027)	9 months ago
Christian Haudum	bf60455c8e	feat(blooms): Add bloom planner and bloom builder to `backend` target (#13997) ... Previously, the bloom compactor component was part of the `backend` target in the Simple Scalable Deployment (SSD) mode. However, the bloom compactor was removed (https://github.com/grafana/loki/pull/13969) in favour of planner and builder, and therefore also removed from the backend target. This PR adds the planner and builder components to the backend target so it can continue building blooms if enabled. The planner needs to be run as singleton, therefore there must only be one instance that creates tasks for the builders, even if multiple replicas of the backend target are deployed. This is achieved by leader election through the already existing index gateway ring in the backend target. The planner leader is determined by the ownership of the leader key. Builders connect to the planner leader to pull tasks. ---- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Ashwanth	281407f146	chore(ci): fix failing arm builds (#14022)	9 months ago
benclive	33c26f4dd1	feat: Add initial support for a kafka-based ingest path (#13992) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	9 months ago
Shantanu Alshi	25234e8348	fix(detected_labels): Add matchers to get labels from store" (#14012)	9 months ago
Trevor Whitney	ab1caea123	feat: add _extracted suffix to detected fields conflicts (#13993)	9 months ago
benclive	a0c75987a2	fix: Propagate headers/warnings/stats from quantile downstreams (#13881)	9 months ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969) ... **What this PR does / why we need it**: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.*` to `-bloom-build.*`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.*` were removed. See upgrade docs for further information. **Special notes for your reviewer**: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Travis Patterson	3bf7fa9f15	fix: sanatize structured metadata at query time (#13983)	9 months ago
Christian Haudum	8afdfd500c	chore: Rename `HeapIterator` to `MergeEntryIterator` (#13975) ... Stumbled across this iterator interface and was confused about its name. It leaked its implementation details, and additionally that was not even correct any more, since the implementation has changed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Cyril Tovena	7c1a8493b3	perf: Avoid looking up unnecessary TSDB symbols during Volume API (#13960) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	9 months ago
Shantanu Alshi	6f99af6222	fix(detected labels): response when store label values are empty (#13970)	9 months ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958)	9 months ago
Trevor Whitney	b1dc0763d6	feat: ability to ignore cache for volume queries (#13945)	9 months ago
renovate[bot]	246a1dfbe2	fix(deps): update module github.com/ncw/swift to v2 (#13951) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	9 months ago
benclive	115fef49f9	fix: Read full buffer from storage when fetching a range (#13948)	9 months ago
renovate[bot]	d43b2de1b4	fix(deps): update module github.com/fsouza/fake-gcs-server to v1.47.7 (#13935) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	9 months ago