310 Commits (shantanu/dataobj-compactor)

Author	SHA1	Message	Date
Robert Fratto	08aaaa3754	chore(columnar): add scalars and builders (#20583)	6 days ago
Robert Fratto	3e3a293215	chore(columnar): switch to generic Number type (#20580)	6 days ago
Ivan Kalita	86f87e447c	chore(dataobj): use reader adapter (#20572) ... Continuation of #20525, I'm updating logs, pointers, streams readers to use reader adapter. Also adding binary support to arrowconv.	6 days ago
Robert Fratto	44cfaedf85	chore(dataobj): move columnar.Array translation up to basicReader (#20551)	7 days ago
benclive	82b93d0102	chore: Use columnar reader for IndexSectionReader (#20501)	1 week ago
Ivan Kalita	aab681617b	chore(dataobj): add reader_adapter (#20525) ... Add a temporary translation layer []dataset.Row -> columnar.RecordBatch -> arrow.RecordBatch. - reader_adapter.go is temporary that's why I put it into the old columnar package - I added builders to reader_adapter.go and not to pkg/columnar.... as I'm not sure if we really would need them anywhere except for this adapter. If so I'm happy to move them to pkg/columnar. - I only updated indexpointers reader for now but will add support for more readers in the next PRs	1 week ago
Robert Fratto	b3f0e8797f	chore(dataobj): move columnar.Array translation up to columnReader (#20531)	2 weeks ago
Robert Fratto	b557429014	chore(dataobj): Use columnar API in pageReader (#20509) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 weeks ago
George Robinson	a85a9a200c	chore: replace quartz with synctest (#20452)	2 weeks ago
George Robinson	d5f6026e30	fix: service failed to shutdown errors when context canceled (#20450)	2 weeks ago
yy	0574159216	chore: typos in comments (#20481)	2 weeks ago
George Robinson	0b0a98de44	chore: use services.BasicService in partition_processor.go (#20448)	2 weeks ago
Robert Fratto	e0370b13c9	chore(dataobj): reduce pageReader allocations (#20496)	2 weeks ago
Robert Fratto	d14bcc1c06	chore(columnar): add experimental columnar package for pkg/dataobj (#20478)	2 weeks ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	2 weeks ago
George Robinson	27df4ab481	feat: create separate flusher to support parallel flushes (#20446)	2 weeks ago
Robert Fratto	8fbe1df00a	chore(memory): move subpackages into memory package (#20469)	2 weeks ago
benclive	d974141db5	perf: Optimize delta decoder (#20451)	3 weeks ago
Robert Fratto	c89a0bb0b4	chore(dataobj): optimize plain value decoder (#20441) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
George Robinson	6a8b879d8f	feat: add support for cancelation to copy and sort (#20370)	3 weeks ago
Robert Fratto	8996bdcff3	chore(dataobj): add benchmarks for plain decoder (#20423)	3 weeks ago
benclive	c513a8cb6f	chore: Add benchmarks for delta decoder (#20419)	3 weeks ago
Ivan Kalita	84bd529866	chore(dataset): bench for bitmapDecoder (#20410) ... Add benchmark for bitmapDecoder that covers decoding of multiple values (instead of a single value). I also add the following improvements: - avoid loading io.EOF into registers on every value which gives ~8% performance (sec/op, processed bytes/s, million of values/s) improvement - reuse set buffer which makes the memory allocations constant	3 weeks ago
George Robinson	ed4f27ed60	feat: add metric to track flush failures (#20399)	3 weeks ago
Dylan Guedes	498656bbe7	feat: Add new dataobj builder flush criteria (#20323)	3 weeks ago
Trevor Whitney	66fd9d8bde	feat: disambiguate metadata for better scans (#20245) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 weeks ago
George Robinson	533b1f0ae8	chore: remove unregister metrics (#20385)	3 weeks ago
George Robinson	3acb3106bb	feat: remove final flush (#20360)	3 weeks ago
George Robinson	c3e909d538	feat: decouple dataobj consumers from the reader service (#20315)	4 weeks ago
Stas Spiridonov	c2116618bd	chore: Improving memory allocations for new query engine (#20321)	4 weeks ago
George Robinson	5a5e90e9f7	feat: add histogram loki_dataobj_consumer_flush_duration_seconds (#20304)	4 weeks ago
George Robinson	fba0c5d8a6	feat: dataobj-consumer add processed bytes metric (#20303)	4 weeks ago
George Robinson	bd77a47df9	chore: remove use of namespace/subsystem in builder metrics (#20301)	4 weeks ago
George Robinson	e0e5d78e69	chore: move appends total metric into builder (#20300)	4 weeks ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	1 month ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	2 months ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	2 months ago
Ashwanth	7da7f000ff	chore: use projection for indexobj streams reader (#20205)	2 months ago
George Robinson	a03e00a3bc	chore: better comments when end offset is zero (#20196)	2 months ago
George Robinson	333da739c7	feat: add processed records metric (#20191)	2 months ago
George Robinson	c0c27b3596	feat: add downscalePermittedFunc to check downscale is permitted (#20171)	2 months ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	2 months ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095)	2 months ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	2 months ago
Ivan Kalita	827292d8ed	chore(metastore): add columnar indexpointers reader (#20068) ... Similar to https://github.com/grafana/loki/pull/20053 and https://github.com/grafana/loki/pull/19992 I add a columnar reader for index pointers sections.	2 months ago
Ivan Kalita	ee58c333b3	chore(metastore): use column reader for blooms (#20053) ... This PR is a continuation of https://github.com/grafana/loki/pull/19992. I use `pointers.Reader` to refine the sections that match bloom filters. I also fix minor issues that I introduced in the previous PR: - convert the columns only once per column (instead of doing it for every row) - ignore the sections that are of a wrong "type"	2 months ago
benclive	225a0595ac	chore: Define an E2E ingestion processing time metric (#20054)	2 months ago
Ivan Kalita	3559d4bd42	chore(metastore): add column reader for pointers section (#19992) ... Adding columnar reader for (log)pointers section. Using it to read the stream section pointers. New reader is slower than existing RowReader due to double-translation of the data (columns->rows->rows vs columns->rows->columns->rows). The idea though is to get rid of the double-translation in the future.	2 months ago
benclive	044ded2e29	chore: Split indexing into distinct steps (#19987)	2 months ago
benclive	2ce207fbdb	chore: Use shared zstd encoders during dataobj build (#20044)	2 months ago