16 Commits (bf06e45bcfc7745ee26798fdfbbbe31ecfb3bd3f)

Author	SHA1	Message	Date
George Robinson	c405d54f72	fix: Don't drain in Kafka mode (#21716)	1 month ago
Shantanu Alshi	368583e778	chore: single binary without Kafka (#21318) ... Signed-off-by: Shantanu Alshi <shantanu.alshi@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
George Robinson	47e1e81f4e	feat: add time sharded estimate (#21463) ... This commit adds a new metric that tracks the number of data objects produced per flush if we build data objects with 12 hour UTC windows.	2 months ago
George Robinson	6cf495f646	chore: rename flushManager to flushCommitter to prepare for future changes (#21135)	3 months ago
George Robinson	a6c83aa2e0	chore: remove deprecated metrics (#20844)	4 months ago
George Robinson	c169ce3ae9	feat: add flush manager to control data object flush lifecycle (#20680)	4 months ago
George Robinson	fcdd2adf2a	fix: service failure on stopping kafkav2 consumer (#20672)	4 months ago
George Robinson	18f2c5049e	chore: move metrics into observeRecordErr (#20625)	4 months ago
George Robinson	14d2285b76	fix: set consumption lag to zero after idle flush timeout (#20620)	4 months ago
George Robinson	0d9df6b055	chore: clean up consumer metrics (#20619)	4 months ago
George Robinson	cb938d0a9d	feat: separate metastore events and offsets from flushes (#20605)	4 months ago
George Robinson	45a7f55d39	fix: revert "feat: reset the consumption lag gauge after 1 minute of idle period" (#20617)	4 months ago
George Robinson	f1d0cdf145	feat: reset the consumption lag gauge after 1 minute of idle period (#20616)	4 months ago
George Robinson	8b86d42507	chore: no need for defer func (#20615)	4 months ago
George Robinson	4b2f46fc37	chore: move builder initialization out of processor.go (#20594)	4 months ago
George Robinson	7010fa8a87	chore: clean up partition processor code (#20593)	4 months ago
George Robinson	a85a9a200c	chore: replace quartz with synctest (#20452)	5 months ago
George Robinson	d5f6026e30	fix: service failed to shutdown errors when context canceled (#20450)	5 months ago
George Robinson	0b0a98de44	chore: use services.BasicService in partition_processor.go (#20448)	5 months ago
George Robinson	27df4ab481	feat: create separate flusher to support parallel flushes (#20446)	5 months ago
George Robinson	6a8b879d8f	feat: add support for cancelation to copy and sort (#20370)	5 months ago
George Robinson	ed4f27ed60	feat: add metric to track flush failures (#20399)	5 months ago
Dylan Guedes	498656bbe7	feat: Add new dataobj builder flush criteria (#20323)	5 months ago
George Robinson	533b1f0ae8	chore: remove unregister metrics (#20385)	5 months ago
George Robinson	3acb3106bb	feat: remove final flush (#20360)	5 months ago
George Robinson	c3e909d538	feat: decouple dataobj consumers from the reader service (#20315)	5 months ago
George Robinson	5a5e90e9f7	feat: add histogram loki_dataobj_consumer_flush_duration_seconds (#20304)	5 months ago
George Robinson	fba0c5d8a6	feat: dataobj-consumer add processed bytes metric (#20303)	5 months ago
George Robinson	e0e5d78e69	chore: move appends total metric into builder (#20300)	5 months ago
George Robinson	333da739c7	feat: add processed records metric (#20191)	6 months ago
benclive	225a0595ac	chore: Define an E2E ingestion processing time metric (#20054)	6 months ago
Christian Haudum	0b4a7f5a2c	chore(dataobj-consumer): Re-use builder for CopyAndSort() (#19696) ... Re-use logsobj.Builder instance in partition processor when sorting logs object wide after flushing. The metrics for the builder in CopyAndSort are never registered, which means the loki_dataobj_sort_duration_seconds histogram is never exported/scraped. We cannot register them with RegisterMetrics() because the metrics are already registered here, and registering them twice would cause a panic.	7 months ago
benclive	6ee50064ea	chore: Ensure partition is set on partition-processor (#19693)	7 months ago
George Robinson	977fb2ba6f	feat: consume single topics in dataobj-consumer using partition ring (#19186)	8 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	9 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	9 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	10 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928)	10 months ago
George Robinson	20d93e3a8d	chore: rename lastFlush to lastFlushed and move some tests (#18918)	10 months ago
George Robinson	3ed2c4e127	chore: add safety check for estimated size (#18919)	10 months ago
George Robinson	31d5f1a738	fix: reset the last modified timestamp on flush (#18916)	10 months ago
George Robinson	4608048388	chore: remove unused sync.Pool (#18895)	10 months ago
George Robinson	be7ff4b294	chore: add flushAndCommit to make it harder to forget to commit offsets in future (#18851)	10 months ago
George Robinson	0941b90614	fix: dataobj-consumers did not commit offsets on idle flush (#18834)	10 months ago
George Robinson	8895090643	fix: dataobj-consumers commit incorrect offset, which can cause data loss (#18833)	10 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	10 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
George Robinson	53416a65ef	chore: use quartz.Clock to improve tests (#18810)	10 months ago
George Robinson	69a70dda88	chore: use zero value for time in partition_processor.go (#18798)	10 months ago