122 Commits (main)

Author	SHA1	Message	Date
Ivan Kalita	3f62bb7207	chore(compactor): Allow overflowed logsobj builders (#22148) ... I'm changing the behavior of logsobj builders to allow appends to overflowed builders. Before this PR: (1) Processor tries to append a record to a builder (2) If a builder is full it doesn't append and returns an error ErrBuilderFull (3) Processor checks the error and flushes the builder (4) Processor tries to append the same record as in the step (1) to the now empty builder This approach works fine but becomes problematic once we introduce multiple TOC windows builders. The issue is that with a single builder we have atomic "try append - flush if error" behavior guarantee. With multiple builders this becomes a problem as a single kafka record might contain entries that belong to different builders. Example: - we have a record R that has a log entry L0 for TOC window W0 AND a log entry L1 for TOC window W1 - we attempt to append L0 to the first builder B0 (for TOC W0) -> success - we attempt to append L1 to the second builder B1 (for TOC W1) -> not appended, err builder full - we flush all the builders (have to flush all of them because otherwise we can lose data as Kafka offset is committed after flushing) - we attempt to append R again -> builder B0 will receive a duplicate of log L0 To simplify the processing we can allow appends to full builders ("one last time"). For the example above we'd allow the append of L1 to B1 even though B1 is already full. This way the dataobjects might become slightly bigger than the target size but this could already be the case given that the existing size estimates are estimates.	3 days ago
Ivan Kalita	378ad03ee6	chore(compactor): EarliestRecordTime per logsobj builder (#22139) ... I'm changing processor / builder such a way that EarliestRecordTime is set per-builder not per processor. This timestamp is used later by dataobj-index-builder to track the e2e indexing latency. The reason to make it per-builder is that when we introduce builders builder-per-time-window approach we'll need to keep separate EarliestRecordTimes for these builders (one per builder). The change is intended to be a no-op (no behavioral changes).	4 days ago
Ivan Kalita	bd6316d9ac	feat(compactor): Reuse logsobj builder metrics (#22138) ... I'm updating logsobj consumer BuilderMetrics so they are registered outside of the builder and passed down to builders through the factory. The reason is – we are going to create builders dynamically in the future (see #21573) and the current approach where builder metrics are registered every time a builder is created is not going to work (the same metrics will be registered multiple times). The change is intended to be a no-op (no behavioral changes), just shuffled metrics registration around.	4 days ago
Joe Elliott	900a82255c	chore: Enable the unused linter (#22078) ... Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	6 days ago
Christian Haudum	257b546d04	chore: Revert "chore: single binary without Kafka (#21318)" (#21979) ... This reverts commit 368583e. The decision was made that we will not ship any components that are built on the new Loki architecture in the upcoming Loki 4.0 release. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 days ago
Trevor Whitney	3f093ef4ca	refactor(dataobj): Move sortMergeIterator to shared pkg/dataobj/sortmerge (#21812) ... Co-authored-by: Bryan Boreham <bjboreham@gmail.com>	3 weeks ago
Shantanu Alshi	a2c48847a4	feat(dataobj): Implement sort schema (#21557)	3 weeks ago
George Robinson	c405d54f72	fix: Don't drain in Kafka mode (#21716)	4 weeks ago
George Robinson	61b377db50	fix: Flush reason counter not initialized to 0, breaks PromQL functions (#21772)	4 weeks ago
Christian Haudum	aaa2d1137a	chore: Remove unused/dead code (#21724) ... * chore(dataobj): Remove dead code * chore(engine): Remove unused code * chore(ci): Fix PR/commit title for Renovate (#21721) * fix: Incorrect impl of ProduceSync (#21702) * (origin/k304) docs: New usage statistics topic (#21668) * feat(lokitool): Rename `rules lint` to `rules format` (#21671) * chore: single binary without Kafka (#21318)	1 month ago
Shantanu Alshi	368583e778	chore: single binary without Kafka (#21318) ... Signed-off-by: Shantanu Alshi <shantanu.alshi@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
George Robinson	47e1e81f4e	feat: add time sharded estimate (#21463) ... This commit adds a new metric that tracks the number of data objects produced per flush if we build data objects with 12 hour UTC windows.	2 months ago
Dylan Guedes	bd7530f322	feat: Use `AppendOrdered` strategy to avoid sorting and merging stripes (#21395) ... Signed-off-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	2 months ago
Shantanu Alshi	30549ac587	feat(dataobj): Add sort_schema config and feature flag (#21460)	2 months ago
George Robinson	6cf495f646	chore: rename flushManager to flushCommitter to prepare for future changes (#21135)	3 months ago
George Robinson	8a68101b3b	feat: add builder pool to logsobj package (#21134)	3 months ago
George Robinson	a6c83aa2e0	chore: remove deprecated metrics (#20844)	4 months ago
Robert Fratto	d85fb0317f	chore(dataobj): Add Open semantics to section readers (#20790) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
George Robinson	93f4bdf796	feat: allow the initial offset to be set in the starting phase (#20770)	4 months ago
Robert Fratto	7e03400309	chore(dataobj): rename dataset.ReaderOptions to dataset.RowReaderOptions (#20756)	4 months ago
Trevor Whitney	83e275775a	chore: bring comparison metrics and goldfish metrics more in line (#20706) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	4 months ago
George Robinson	c169ce3ae9	feat: add flush manager to control data object flush lifecycle (#20680)	4 months ago
George Robinson	4d1c4032cc	chore: small doc fix, missing period end of help (#20681)	4 months ago
George Robinson	72aaea20a6	chore: no lazy struct embed (#20682)	4 months ago
George Robinson	fcdd2adf2a	fix: service failure on stopping kafkav2 consumer (#20672)	4 months ago
George Robinson	7aee83d6d2	chore: use temporary builder to sort data objects (#20664)	4 months ago
George Robinson	18f2c5049e	chore: move metrics into observeRecordErr (#20625)	4 months ago
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629)	4 months ago
George Robinson	14d2285b76	fix: set consumption lag to zero after idle flush timeout (#20620)	4 months ago
George Robinson	0d9df6b055	chore: clean up consumer metrics (#20619)	4 months ago
George Robinson	cb938d0a9d	feat: separate metastore events and offsets from flushes (#20605)	4 months ago
George Robinson	45a7f55d39	fix: revert "feat: reset the consumption lag gauge after 1 minute of idle period" (#20617)	4 months ago
George Robinson	f1d0cdf145	feat: reset the consumption lag gauge after 1 minute of idle period (#20616)	4 months ago
George Robinson	8b86d42507	chore: no need for defer func (#20615)	4 months ago
George Robinson	4b2f46fc37	chore: move builder initialization out of processor.go (#20594)	4 months ago
George Robinson	7010fa8a87	chore: clean up partition processor code (#20593)	4 months ago
George Robinson	a85a9a200c	chore: replace quartz with synctest (#20452)	5 months ago
George Robinson	d5f6026e30	fix: service failed to shutdown errors when context canceled (#20450)	5 months ago
George Robinson	0b0a98de44	chore: use services.BasicService in partition_processor.go (#20448)	5 months ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	5 months ago
George Robinson	27df4ab481	feat: create separate flusher to support parallel flushes (#20446)	5 months ago
George Robinson	6a8b879d8f	feat: add support for cancelation to copy and sort (#20370)	5 months ago
George Robinson	ed4f27ed60	feat: add metric to track flush failures (#20399)	5 months ago
Dylan Guedes	498656bbe7	feat: Add new dataobj builder flush criteria (#20323)	5 months ago
George Robinson	533b1f0ae8	chore: remove unregister metrics (#20385)	5 months ago
George Robinson	3acb3106bb	feat: remove final flush (#20360)	5 months ago
George Robinson	c3e909d538	feat: decouple dataobj consumers from the reader service (#20315)	5 months ago
George Robinson	5a5e90e9f7	feat: add histogram loki_dataobj_consumer_flush_duration_seconds (#20304)	5 months ago
George Robinson	fba0c5d8a6	feat: dataobj-consumer add processed bytes metric (#20303)	5 months ago
George Robinson	bd77a47df9	chore: remove use of namespace/subsystem in builder metrics (#20301)	5 months ago