324 Commits (shantanu/dataobj-compactor)

Author	SHA1	Message	Date
Sandeep Sukhani	7f1450d8c5	feat(dataobj-compactor): add a planner for compaction of data objects (#20527)	4 months ago
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629)	4 months ago
George Robinson	14d2285b76	fix: set consumption lag to zero after idle flush timeout (#20620)	4 months ago
benclive	294d574bc9	fix(dataobj): Avoid deadlock in index-builder when cancelling mid-build (#20621)	4 months ago
George Robinson	0d9df6b055	chore: clean up consumer metrics (#20619)	4 months ago
George Robinson	cb938d0a9d	feat: separate metastore events and offsets from flushes (#20605)	4 months ago
Robert Fratto	747d75aa78	chore: make memory, columnar APIs more idiomatic (#20622)	4 months ago
George Robinson	45a7f55d39	fix: revert "feat: reset the consumption lag gauge after 1 minute of idle period" (#20617)	4 months ago
George Robinson	f1d0cdf145	feat: reset the consumption lag gauge after 1 minute of idle period (#20616)	4 months ago
George Robinson	8b86d42507	chore: no need for defer func (#20615)	4 months ago
George Robinson	4b2f46fc37	chore: move builder initialization out of processor.go (#20594)	4 months ago
George Robinson	7010fa8a87	chore: clean up partition processor code (#20593)	4 months ago
benclive	9ccf55d0ff	perf(dataobj): Optimize estimate size calculation (#20595)	4 months ago
benclive	6f72400daa	fix(enginev2): Copy values when storing min/max stats (#20589)	4 months ago
Robert Fratto	08aaaa3754	chore(columnar): add scalars and builders (#20583)	4 months ago
Robert Fratto	3e3a293215	chore(columnar): switch to generic Number type (#20580)	4 months ago
Ivan Kalita	86f87e447c	chore(dataobj): use reader adapter (#20572) ... Continuation of #20525, I'm updating logs, pointers, streams readers to use reader adapter. Also adding binary support to arrowconv.	4 months ago
Robert Fratto	44cfaedf85	chore(dataobj): move columnar.Array translation up to basicReader (#20551)	4 months ago
benclive	82b93d0102	chore: Use columnar reader for IndexSectionReader (#20501)	4 months ago
Ivan Kalita	aab681617b	chore(dataobj): add reader_adapter (#20525) ... Add a temporary translation layer []dataset.Row -> columnar.RecordBatch -> arrow.RecordBatch. - reader_adapter.go is temporary that's why I put it into the old columnar package - I added builders to reader_adapter.go and not to pkg/columnar.... as I'm not sure if we really would need them anywhere except for this adapter. If so I'm happy to move them to pkg/columnar. - I only updated indexpointers reader for now but will add support for more readers in the next PRs	4 months ago
Robert Fratto	b3f0e8797f	chore(dataobj): move columnar.Array translation up to columnReader (#20531)	4 months ago
Robert Fratto	b557429014	chore(dataobj): Use columnar API in pageReader (#20509) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
George Robinson	a85a9a200c	chore: replace quartz with synctest (#20452)	4 months ago
George Robinson	d5f6026e30	fix: service failed to shutdown errors when context canceled (#20450)	4 months ago
yy	0574159216	chore: typos in comments (#20481)	4 months ago
George Robinson	0b0a98de44	chore: use services.BasicService in partition_processor.go (#20448)	4 months ago
Robert Fratto	e0370b13c9	chore(dataobj): reduce pageReader allocations (#20496)	4 months ago
Robert Fratto	d14bcc1c06	chore(columnar): add experimental columnar package for pkg/dataobj (#20478)	4 months ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	4 months ago
George Robinson	27df4ab481	feat: create separate flusher to support parallel flushes (#20446)	4 months ago
Robert Fratto	8fbe1df00a	chore(memory): move subpackages into memory package (#20469)	4 months ago
benclive	d974141db5	perf: Optimize delta decoder (#20451)	5 months ago
Robert Fratto	c89a0bb0b4	chore(dataobj): optimize plain value decoder (#20441) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
George Robinson	6a8b879d8f	feat: add support for cancelation to copy and sort (#20370)	5 months ago
Robert Fratto	8996bdcff3	chore(dataobj): add benchmarks for plain decoder (#20423)	5 months ago
benclive	c513a8cb6f	chore: Add benchmarks for delta decoder (#20419)	5 months ago
Ivan Kalita	84bd529866	chore(dataset): bench for bitmapDecoder (#20410) ... Add benchmark for bitmapDecoder that covers decoding of multiple values (instead of a single value). I also add the following improvements: - avoid loading io.EOF into registers on every value which gives ~8% performance (sec/op, processed bytes/s, million of values/s) improvement - reuse set buffer which makes the memory allocations constant	5 months ago
George Robinson	ed4f27ed60	feat: add metric to track flush failures (#20399)	5 months ago
Dylan Guedes	498656bbe7	feat: Add new dataobj builder flush criteria (#20323)	5 months ago
Trevor Whitney	66fd9d8bde	feat: disambiguate metadata for better scans (#20245) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	5 months ago
George Robinson	533b1f0ae8	chore: remove unregister metrics (#20385)	5 months ago
George Robinson	3acb3106bb	feat: remove final flush (#20360)	5 months ago
George Robinson	c3e909d538	feat: decouple dataobj consumers from the reader service (#20315)	5 months ago
Stas Spiridonov	c2116618bd	chore: Improving memory allocations for new query engine (#20321)	5 months ago
George Robinson	5a5e90e9f7	feat: add histogram loki_dataobj_consumer_flush_duration_seconds (#20304)	5 months ago
George Robinson	fba0c5d8a6	feat: dataobj-consumer add processed bytes metric (#20303)	5 months ago
George Robinson	bd77a47df9	chore: remove use of namespace/subsystem in builder metrics (#20301)	5 months ago
George Robinson	e0e5d78e69	chore: move appends total metric into builder (#20300)	5 months ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	5 months ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	5 months ago