11 Commits (main)

Author	SHA1	Message	Date
George Robinson	c169ce3ae9	feat: add flush manager to control data object flush lifecycle (#20680)	5 days ago
George Robinson	fcdd2adf2a	fix: service failure on stopping kafkav2 consumer (#20672)	7 days ago
George Robinson	18f2c5049e	chore: move metrics into observeRecordErr (#20625)	1 week ago
George Robinson	14d2285b76	fix: set consumption lag to zero after idle flush timeout (#20620)	2 weeks ago
George Robinson	0d9df6b055	chore: clean up consumer metrics (#20619)	2 weeks ago
George Robinson	cb938d0a9d	feat: separate metastore events and offsets from flushes (#20605)	2 weeks ago
George Robinson	45a7f55d39	fix: revert "feat: reset the consumption lag gauge after 1 minute of idle period" (#20617)	2 weeks ago
George Robinson	f1d0cdf145	feat: reset the consumption lag gauge after 1 minute of idle period (#20616)	2 weeks ago
George Robinson	8b86d42507	chore: no need for defer func (#20615)	2 weeks ago
George Robinson	4b2f46fc37	chore: move builder initialization out of processor.go (#20594)	2 weeks ago
George Robinson	7010fa8a87	chore: clean up partition processor code (#20593)	2 weeks ago
George Robinson	a85a9a200c	chore: replace quartz with synctest (#20452)	3 weeks ago
George Robinson	d5f6026e30	fix: service failed to shutdown errors when context canceled (#20450)	3 weeks ago
George Robinson	0b0a98de44	chore: use services.BasicService in partition_processor.go (#20448)	3 weeks ago
George Robinson	27df4ab481	feat: create separate flusher to support parallel flushes (#20446)	3 weeks ago
George Robinson	6a8b879d8f	feat: add support for cancelation to copy and sort (#20370)	4 weeks ago
George Robinson	ed4f27ed60	feat: add metric to track flush failures (#20399)	4 weeks ago
Dylan Guedes	498656bbe7	feat: Add new dataobj builder flush criteria (#20323)	4 weeks ago
George Robinson	533b1f0ae8	chore: remove unregister metrics (#20385)	1 month ago
George Robinson	3acb3106bb	feat: remove final flush (#20360)	1 month ago
George Robinson	c3e909d538	feat: decouple dataobj consumers from the reader service (#20315)	1 month ago
George Robinson	5a5e90e9f7	feat: add histogram loki_dataobj_consumer_flush_duration_seconds (#20304)	1 month ago
George Robinson	fba0c5d8a6	feat: dataobj-consumer add processed bytes metric (#20303)	1 month ago
George Robinson	e0e5d78e69	chore: move appends total metric into builder (#20300)	1 month ago
George Robinson	333da739c7	feat: add processed records metric (#20191)	2 months ago
benclive	225a0595ac	chore: Define an E2E ingestion processing time metric (#20054)	2 months ago
Christian Haudum	0b4a7f5a2c	chore(dataobj-consumer): Re-use builder for CopyAndSort() (#19696) ... Re-use logsobj.Builder instance in partition processor when sorting logs object wide after flushing. The metrics for the builder in CopyAndSort are never registered, which means the loki_dataobj_sort_duration_seconds histogram is never exported/scraped. We cannot register them with RegisterMetrics() because the metrics are already registered here, and registering them twice would cause a panic.	3 months ago
benclive	6ee50064ea	chore: Ensure partition is set on partition-processor (#19693)	3 months ago
George Robinson	977fb2ba6f	feat: consume single topics in dataobj-consumer using partition ring (#19186)	4 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	5 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	6 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	6 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928)	6 months ago
George Robinson	20d93e3a8d	chore: rename lastFlush to lastFlushed and move some tests (#18918)	6 months ago
George Robinson	3ed2c4e127	chore: add safety check for estimated size (#18919)	6 months ago
George Robinson	31d5f1a738	fix: reset the last modified timestamp on flush (#18916)	6 months ago
George Robinson	4608048388	chore: remove unused sync.Pool (#18895)	6 months ago
George Robinson	be7ff4b294	chore: add flushAndCommit to make it harder to forget to commit offsets in future (#18851)	6 months ago
George Robinson	0941b90614	fix: dataobj-consumers did not commit offsets on idle flush (#18834)	6 months ago
George Robinson	8895090643	fix: dataobj-consumers commit incorrect offset, which can cause data loss (#18833)	6 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	6 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
George Robinson	53416a65ef	chore: use quartz.Clock to improve tests (#18810)	6 months ago
George Robinson	69a70dda88	chore: use zero value for time in partition_processor.go (#18798)	6 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	6 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622)	6 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	7 months ago
Christian Haudum	b1fc685397	chore: Extend instrumentation of dataobject uploader (#18292) ... These changes provide better visibility into the dataobj uploader. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
benclive	e6fe98c7c3	chore: Emit metastore event to kafka after object build (#17842)	9 months ago