30 Commits (main)

Author	SHA1	Message	Date
benclive	225a0595ac	chore: Define an E2E ingestion processing time metric (#20054)	4 weeks ago
Ivan Kalita	3559d4bd42	chore(metastore): add column reader for pointers section (#19992) ... Adding columnar reader for (log)pointers section. Using it to read the stream section pointers. New reader is slower than existing RowReader due to double-translation of the data (columns->rows->rows vs columns->rows->columns->rows). The idea though is to get rid of the double-translation in the future.	4 weeks ago
benclive	044ded2e29	chore: Split indexing into distinct steps (#19987)	4 weeks ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037) ... Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	4 weeks ago
Periklis Tsirakidis	32dbef99b8	fix(dataobj): Flush into multiple index objects when ErrBuilderFull (#19223)	1 month ago
benclive	bda352b67d	chore: Use flag to set index consumer group (#19965)	1 month ago
benclive	ea2b4d0f84	chore: Use Kafka round-robin balancer for index-builder (#19574)	2 months ago
Christian Haudum	386d4e1653	chore(dataobj): Make logs sort order in dataobjects configurable (#19373) ... This PR introduces a new configuration option `-dataobj-consumer.dataobj-sort-order` which controls the sort order of logs in the logs section of the dataobj when building new objects. There are two available sort options: 1. `timestamp DESC, streamID ASC` (configuration value `timestamp-desc`) 2. `streamID ASC, timestamp DESC` (configuration value `stream-asc`) The new setting is undocumented and may be removed in the future without notice. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	3 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	4 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	4 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	4 months ago
benclive	a7f001512a	chore: Fix index generation with filesystem (#19063)	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	4 months ago
benclive	28fe97dcce	chore: fix metastore indexes being written with double prefixes (#19018)	4 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	4 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	5 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	5 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622)	5 months ago
benclive	41d93f8a86	fix: Fix empty streamIDs in multi-section index objects (#18556)	5 months ago
benclive	798f77892e	chore: Fix slicelabels merge conflict in calculator.go (#18517)	5 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	5 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490)	5 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	6 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	6 months ago