103 Commits (shantanu/dataobj-compactor)

Author	SHA1	Message	Date
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629)	4 days ago
Robert Fratto	747d75aa78	chore: make memory, columnar APIs more idiomatic (#20622)	4 days ago
benclive	6f72400daa	fix(enginev2): Copy values when storing min/max stats (#20589)	7 days ago
Robert Fratto	3e3a293215	chore(columnar): switch to generic Number type (#20580)	1 week ago
Ivan Kalita	86f87e447c	chore(dataobj): use reader adapter (#20572) ... Continuation of #20525, I'm updating logs, pointers, streams readers to use reader adapter. Also adding binary support to arrowconv.	1 week ago
Robert Fratto	44cfaedf85	chore(dataobj): move columnar.Array translation up to basicReader (#20551)	1 week ago
Ivan Kalita	aab681617b	chore(dataobj): add reader_adapter (#20525) ... Add a temporary translation layer []dataset.Row -> columnar.RecordBatch -> arrow.RecordBatch. - reader_adapter.go is temporary that's why I put it into the old columnar package - I added builders to reader_adapter.go and not to pkg/columnar.... as I'm not sure if we really would need them anywhere except for this adapter. If so I'm happy to move them to pkg/columnar. - I only updated indexpointers reader for now but will add support for more readers in the next PRs	2 weeks ago
Robert Fratto	b3f0e8797f	chore(dataobj): move columnar.Array translation up to columnReader (#20531)	2 weeks ago
Robert Fratto	b557429014	chore(dataobj): Use columnar API in pageReader (#20509) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 weeks ago
Robert Fratto	e0370b13c9	chore(dataobj): reduce pageReader allocations (#20496)	2 weeks ago
Robert Fratto	d14bcc1c06	chore(columnar): add experimental columnar package for pkg/dataobj (#20478)	2 weeks ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	2 weeks ago
Robert Fratto	8fbe1df00a	chore(memory): move subpackages into memory package (#20469)	3 weeks ago
benclive	d974141db5	perf: Optimize delta decoder (#20451)	3 weeks ago
Robert Fratto	c89a0bb0b4	chore(dataobj): optimize plain value decoder (#20441) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	8996bdcff3	chore(dataobj): add benchmarks for plain decoder (#20423)	3 weeks ago
benclive	c513a8cb6f	chore: Add benchmarks for delta decoder (#20419)	3 weeks ago
Ivan Kalita	84bd529866	chore(dataset): bench for bitmapDecoder (#20410) ... Add benchmark for bitmapDecoder that covers decoding of multiple values (instead of a single value). I also add the following improvements: - avoid loading io.EOF into registers on every value which gives ~8% performance (sec/op, processed bytes/s, million of values/s) improvement - reuse set buffer which makes the memory allocations constant	3 weeks ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	2 months ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095)	2 months ago
benclive	2ce207fbdb	chore: Use shared zstd encoders during dataobj build (#20044)	2 months ago
Robert Fratto	806bcd5dd7	chore(dataobj): stop preallocating memory for page builders (#20043)	2 months ago
benclive	a6de577b3a	chore: Fix page skipping with empty predicates (#19602)	3 months ago
Ashwanth	bfd70602dd	chore: correct dataobj reader rows read return value (#19583)	3 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	3 months ago
Robert Fratto	7556192423	chore(dataobj): use single Zstd decoder (#19508)	4 months ago
benclive	2dac5ef8a7	chore: Write empty message values in logs section writer (#19359)	4 months ago
Sophie Waldman	af6983e9de	fix: Add special case handling for comparing empty and nil values (#19348) ... We can have a predicate checking for an empty string, e.g. {"foo" = ""}. If a row has no value for the column "foo", it will return a nil value for that column. Thus, when we go to compare the row with the predicate, we will be comparing an empty value to a nil value. This change adjusts the comparison check so that we correctly handle this special case.	4 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	2ecf07d52c	chore(dataobj): Fix panic in column builder (#19153) ... There is an edge case in the column builder where it panics when the AppendNulls() call tries to append more NULL values than the specified MaxPageRows setting when backfilling rows. In that case the builder would panic because canAppend would always return false even when the previous page was flushed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	515ebea4f2	chore(dataobj): track total download time of page metadata/data (#18896)	6 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Robert Fratto	642d826a00	chore(dataobj): make section tenancy a global construct (#18857) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	6 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694)	6 months ago
George Robinson	916246a2be	feat: add tenant to section metadata (#18843)	6 months ago
Robert Fratto	100e259a8d	chore(dataobj): introduce the concept of section info "extensions" (#18832)	6 months ago
Robert Fratto	61c40fca0c	chore(dataobj): remove support for deprecated fields (breaking change) (#18822)	6 months ago
Robert Fratto	a5c722cbed	chore(dataobj): list pages across all columns in one request (#18678)	6 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	6 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499)	6 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	7 months ago
Periklis Tsirakidis	6eedcaaded	chore(dataobj): Add new index pointers section type (#18441)	7 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	7 months ago
Robert Fratto	26cb50e65a	chore(dataobj): Improve performance of dataset.Value API (#18339) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	7 months ago
Ashwanth	5c02493fe2	chore(dataobj): avoid reading a predicate column more than once (#18324)	7 months ago
Robert Fratto	30ccf5172c	chore(util/arrowtest): export arrowtest package (#18255)	7 months ago