403 Commits (main)

Author	SHA1	Message	Date
Trevor Whitney	03cd8002cf	feat(dataobj/postings,stats): Add merge builders and row decoders (#22229)	1 day ago
Paul Rogers	5c0197c4f3	fix(dataobj): Handle constant predicates in logs predicate ordering (#22160)	2 days ago
Ivan Kalita	3f62bb7207	chore(compactor): Allow overflowed logsobj builders (#22148) ... I'm changing the behavior of logsobj builders to allow appends to overflowed builders. Before this PR: (1) Processor tries to append a record to a builder (2) If a builder is full it doesn't append and returns an error ErrBuilderFull (3) Processor checks the error and flushes the builder (4) Processor tries to append the same record as in the step (1) to the now empty builder This approach works fine but becomes problematic once we introduce multiple TOC windows builders. The issue is that with a single builder we have atomic "try append - flush if error" behavior guarantee. With multiple builders this becomes a problem as a single kafka record might contain entries that belong to different builders. Example: - we have a record R that has a log entry L0 for TOC window W0 AND a log entry L1 for TOC window W1 - we attempt to append L0 to the first builder B0 (for TOC W0) -> success - we attempt to append L1 to the second builder B1 (for TOC W1) -> not appended, err builder full - we flush all the builders (have to flush all of them because otherwise we can lose data as Kafka offset is committed after flushing) - we attempt to append R again -> builder B0 will receive a duplicate of log L0 To simplify the processing we can allow appends to full builders ("one last time"). For the example above we'd allow the append of L1 to B1 even though B1 is already full. This way the dataobjects might become slightly bigger than the target size but this could already be the case given that the existing size estimates are estimates.	3 days ago
Ivan Kalita	378ad03ee6	chore(compactor): EarliestRecordTime per logsobj builder (#22139) ... I'm changing processor / builder such a way that EarliestRecordTime is set per-builder not per processor. This timestamp is used later by dataobj-index-builder to track the e2e indexing latency. The reason to make it per-builder is that when we introduce builders builder-per-time-window approach we'll need to keep separate EarliestRecordTimes for these builders (one per builder). The change is intended to be a no-op (no behavioral changes).	3 days ago
Ivan Kalita	bd6316d9ac	feat(compactor): Reuse logsobj builder metrics (#22138) ... I'm updating logsobj consumer BuilderMetrics so they are registered outside of the builder and passed down to builders through the factory. The reason is – we are going to create builders dynamically in the future (see #21573) and the current approach where builder metrics are registered every time a builder is created is not going to work (the same metrics will be registered multiple times). The change is intended to be a no-op (no behavioral changes), just shuffled metrics registration around.	3 days ago
Joe Elliott	900a82255c	chore: Enable the unused linter (#22078) ... Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	6 days ago
Christian Haudum	257b546d04	chore: Revert "chore: single binary without Kafka (#21318)" (#21979) ... This reverts commit 368583e. The decision was made that we will not ship any components that are built on the new Loki architecture in the upcoming Loki 4.0 release. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 days ago
Bryan Boreham	107dfbde54	chore: Fix more errors reported by go test -race (#22055)	1 week ago
benclive	14947927d6	chore: Remove support for legacy dataobj format (#21909)	1 week ago
Trevor Whitney	29e6d0b9cf	refactor(metastore): Address PR feedback on ReplaceIndexPointers (#21971) ... Co-authored-by: Ivan Kalita <ivan.kalita@grafana.com>	2 weeks ago
Trevor Whitney	10fc249ce8	feat(metastore): Add TableOfContentsWriter.ReplaceIndexPointers method (#21932)	2 weeks ago
benclive	2e3da9a59a	chore: Add better error message on predicate panic (#21935)	3 weeks ago
Trevor Whitney	3f093ef4ca	refactor(dataobj): Move sortMergeIterator to shared pkg/dataobj/sortmerge (#21812) ... Co-authored-by: Bryan Boreham <bjboreham@gmail.com>	3 weeks ago
Robert Fratto	6e9d98616b	chore(engine): Improve log events to observe query lifecycles (#21863)	3 weeks ago
Shantanu Alshi	a2c48847a4	feat(dataobj): Implement sort schema (#21557)	3 weeks ago
Trevor Whitney	3dd8056722	feat(dataobj-compactor): Calculate compaction runs/jobs (#21845)	3 weeks ago
Trevor Whitney	12c419b65d	feat(dataobj-compactor): Add CompactionMerge physical-plan node (#21844)	3 weeks ago
Trevor Whitney	b8668131ab	refactor(metastore): Export tableOfContentsPath, iterTableOfContentsPaths, metastoreWindowSize (#21811)	3 weeks ago
Trevor Whitney	4cd2cf16bd	feat(dataobj-compactor): Add compactor target skeleton, no-op coordinator stub (#21818)	3 weeks ago
Trevor Whitney	620bd56e0a	chore(compaction): Replace postings/stats RowReader with columnar Reader (#21785)	4 weeks ago
Trevor Whitney	96a0574783	feat(dataobj): Reduce postings indexing allocations via observe/aggregate pattern (#21610)	4 weeks ago
Trevor Whitney	deff86de61	feat: On-disk serialization for postings and stats index sections (#21591)	4 weeks ago
George Robinson	c405d54f72	fix: Don't drain in Kafka mode (#21716)	4 weeks ago
George Robinson	61b377db50	fix: Flush reason counter not initialized to 0, breaks PromQL functions (#21772)	4 weeks ago
Dylan Guedes	08820d7fa3	feat: Improve index builder performance by not holding the lock during the whole `Calculate` (#21632)	4 weeks ago
Christian Haudum	aaa2d1137a	chore: Remove unused/dead code (#21724) ... * chore(dataobj): Remove dead code * chore(engine): Remove unused code * chore(ci): Fix PR/commit title for Renovate (#21721) * fix: Incorrect impl of ProduceSync (#21702) * (origin/k304) docs: New usage statistics topic (#21668) * feat(lokitool): Rename `rules lint` to `rules format` (#21671) * chore: single binary without Kafka (#21318)	1 month ago
Shantanu Alshi	368583e778	chore: single binary without Kafka (#21318) ... Signed-off-by: Shantanu Alshi <shantanu.alshi@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Bryan Boreham	9cdc29fb99	fix(columnar): Don't release Arrow records (#21670) ... Multiple goroutines may still be accessing records, so we cannot Release them. Remove from the real code and from an example.	1 month ago
Dylan Guedes	bb55364a78	fix: Fix MatchNotRegexp misbehaving with the new query engine (#21544)	2 months ago
George Robinson	47e1e81f4e	feat: add time sharded estimate (#21463) ... This commit adds a new metric that tracks the number of data objects produced per flush if we build data objects with 12 hour UTC windows.	2 months ago
Salva Corts	2e0a70dfca	feat: clamp pointerscans to index start/end timeranges (#21534)	2 months ago
Trevor Whitney	8553815420	feat: add stats and postings index sections with encoding abstraction (#21442)	2 months ago
Dylan Guedes	bd7530f322	feat: Use `AppendOrdered` strategy to avoid sorting and merging stripes (#21395) ... Signed-off-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	2 months ago
Dan Hopper	4772bfc8f3	feat: Introduce max-age option for index-builder (#21294)	2 months ago
Shantanu Alshi	30549ac587	feat(dataobj): Add sort_schema config and feature flag (#21460)	2 months ago
Robert Fratto	a2736ab590	chore(columnar): add int32 and uint32 numeric type support (#21433)	2 months ago
Dan Hopper	4725e2bb3e	fix(index-builder): don't repeatedly build indexes for stale partitions (#21213)	2 months ago
Salva Corts	bb48b9c247	fix: Fix assignment to entry in nil map in DataobjSectionDescriptor.Merge (#21268)	2 months ago
benclive	e9b4c8831b	chore: Reduce in-use zstd encoders for dataobj building (#21241)	2 months ago
benclive	968dabadd5	chore: Increase batch sizes for pointers reads (#20983)	3 months ago
George Robinson	6cf495f646	chore: rename flushManager to flushCommitter to prepare for future changes (#21135)	3 months ago
George Robinson	8a68101b3b	feat: add builder pool to logsobj package (#21134)	3 months ago
Robert Fratto	fdc807e6df	chore(dataobj): prefetch during open for rowReaderDownloader (#20942) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	b8920125c5	chore(dataobj): add configurable prefetching (#20946) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	b340795f25	chore(dataobj): add missing test for opening legacy object (#20943) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Bryan Boreham	ced1d1b175	perf(dataset): use faster Uvarint implementation (#20824)	3 months ago
benclive	f6bd6b7cb3	chore: Ignore errgroup context in decoder (#20880)	4 months ago
Robert Fratto	f9bdab2fc8	chore(dataobj): restructure dataobjs to store metadata at the front of the file (#20865) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Dylan Guedes	b9af4f5223	fix: Reset lag for index builder when partitions change (#20825)	4 months ago
Robert Fratto	44d6794af2	chore(metastore): fix panic in index sections reader (#20875)	4 months ago