51 Commits (bf06e45bcfc7745ee26798fdfbbbe31ecfb3bd3f)

Author	SHA1	Message	Date
Trevor Whitney	620bd56e0a	chore(compaction): Replace postings/stats RowReader with columnar Reader (#21785)	1 month ago
Trevor Whitney	96a0574783	feat(dataobj): Reduce postings indexing allocations via observe/aggregate pattern (#21610)	1 month ago
Trevor Whitney	deff86de61	feat: On-disk serialization for postings and stats index sections (#21591)	1 month ago
Dylan Guedes	08820d7fa3	feat: Improve index builder performance by not holding the lock during the whole `Calculate` (#21632)	1 month ago
Christian Haudum	aaa2d1137a	chore: Remove unused/dead code (#21724) ... * chore(dataobj): Remove dead code * chore(engine): Remove unused code * chore(ci): Fix PR/commit title for Renovate (#21721) * fix: Incorrect impl of ProduceSync (#21702) * (origin/k304) docs: New usage statistics topic (#21668) * feat(lokitool): Rename `rules lint` to `rules format` (#21671) * chore: single binary without Kafka (#21318)	1 month ago
Trevor Whitney	8553815420	feat: add stats and postings index sections with encoding abstraction (#21442)	2 months ago
Dylan Guedes	bd7530f322	feat: Use `AppendOrdered` strategy to avoid sorting and merging stripes (#21395) ... Signed-off-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	2 months ago
Dan Hopper	4772bfc8f3	feat: Introduce max-age option for index-builder (#21294)	2 months ago
Dan Hopper	4725e2bb3e	fix(index-builder): don't repeatedly build indexes for stale partitions (#21213)	2 months ago
Robert Fratto	b8920125c5	chore(dataobj): add configurable prefetching (#20946) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Dylan Guedes	b9af4f5223	fix: Reset lag for index builder when partitions change (#20825)	4 months ago
Robert Fratto	d85fb0317f	chore(dataobj): Add Open semantics to section readers (#20790) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
George Robinson	b5c7ea2dc2	chore: use StartAndAwait functions (#20755)	4 months ago
George Robinson	8bf62a5124	chore: implement idiomatic worker pattern in indexer.go (#20646)	4 months ago
George Robinson	2949f75444	fix: handle lost partitions in index builders (#20744)	4 months ago
benclive	7567393229	chore: Fix flaky indexer test (#20676)	4 months ago
benclive	328f3d45b9	fix(dataobj): Mark events completed after successful commit (#20658)	4 months ago
Joe Elliott	08e5c9d77c	perf: Prealloc slice in index builder when downloading objects (#20647) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	4 months ago
benclive	322a02d35d	chore: Refactor to avoid sharing a download worker (#20640)	4 months ago
benclive	294d574bc9	fix(dataobj): Avoid deadlock in index-builder when cancelling mid-build (#20621)	4 months ago
benclive	9ccf55d0ff	perf(dataobj): Optimize estimate size calculation (#20595)	4 months ago
benclive	225a0595ac	chore: Define an E2E ingestion processing time metric (#20054)	6 months ago
Ivan Kalita	3559d4bd42	chore(metastore): add column reader for pointers section (#19992) ... Adding columnar reader for (log)pointers section. Using it to read the stream section pointers. New reader is slower than existing RowReader due to double-translation of the data (columns->rows->rows vs columns->rows->columns->rows). The idea though is to get rid of the double-translation in the future.	6 months ago
benclive	044ded2e29	chore: Split indexing into distinct steps (#19987)	6 months ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037) ... Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	6 months ago
Periklis Tsirakidis	32dbef99b8	fix(dataobj): Flush into multiple index objects when ErrBuilderFull (#19223)	7 months ago
benclive	bda352b67d	chore: Use flag to set index consumer group (#19965)	7 months ago
benclive	ea2b4d0f84	chore: Use Kafka round-robin balancer for index-builder (#19574)	8 months ago
Christian Haudum	386d4e1653	chore(dataobj): Make logs sort order in dataobjects configurable (#19373) ... This PR introduces a new configuration option `-dataobj-consumer.dataobj-sort-order` which controls the sort order of logs in the logs section of the dataobj when building new objects. There are two available sort options: 1. `timestamp DESC, streamID ASC` (configuration value `timestamp-desc`) 2. `streamID ASC, timestamp DESC` (configuration value `stream-asc`) The new setting is undocumented and may be removed in the future without notice. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	9 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	9 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	9 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	9 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	9 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	9 months ago
benclive	a7f001512a	chore: Fix index generation with filesystem (#19063)	9 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	9 months ago
benclive	28fe97dcce	chore: fix metastore indexes being written with double prefixes (#19018)	9 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	10 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	10 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	10 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622)	10 months ago
benclive	41d93f8a86	fix: Fix empty streamIDs in multi-section index objects (#18556)	11 months ago
benclive	798f77892e	chore: Fix slicelabels merge conflict in calculator.go (#18517)	11 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	11 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490)	11 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	11 months ago