67 Commits (shantanu/dataobj-compactor)

Author	SHA1	Message	Date
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629)	4 days ago
Robert Fratto	747d75aa78	chore: make memory, columnar APIs more idiomatic (#20622)	4 days ago
benclive	6f72400daa	fix(enginev2): Copy values when storing min/max stats (#20589)	7 days ago
Robert Fratto	3e3a293215	chore(columnar): switch to generic Number type (#20580)	1 week ago
Robert Fratto	44cfaedf85	chore(dataobj): move columnar.Array translation up to basicReader (#20551)	1 week ago
Robert Fratto	b3f0e8797f	chore(dataobj): move columnar.Array translation up to columnReader (#20531)	2 weeks ago
Robert Fratto	b557429014	chore(dataobj): Use columnar API in pageReader (#20509) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 weeks ago
Robert Fratto	e0370b13c9	chore(dataobj): reduce pageReader allocations (#20496)	2 weeks ago
Robert Fratto	d14bcc1c06	chore(columnar): add experimental columnar package for pkg/dataobj (#20478)	2 weeks ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	2 weeks ago
Robert Fratto	8fbe1df00a	chore(memory): move subpackages into memory package (#20469)	3 weeks ago
benclive	d974141db5	perf: Optimize delta decoder (#20451)	3 weeks ago
Robert Fratto	c89a0bb0b4	chore(dataobj): optimize plain value decoder (#20441) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	8996bdcff3	chore(dataobj): add benchmarks for plain decoder (#20423)	3 weeks ago
benclive	c513a8cb6f	chore: Add benchmarks for delta decoder (#20419)	3 weeks ago
Ivan Kalita	84bd529866	chore(dataset): bench for bitmapDecoder (#20410) ... Add benchmark for bitmapDecoder that covers decoding of multiple values (instead of a single value). I also add the following improvements: - avoid loading io.EOF into registers on every value which gives ~8% performance (sec/op, processed bytes/s, million of values/s) improvement - reuse set buffer which makes the memory allocations constant	3 weeks ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	2 months ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095)	2 months ago
benclive	2ce207fbdb	chore: Use shared zstd encoders during dataobj build (#20044)	2 months ago
Robert Fratto	806bcd5dd7	chore(dataobj): stop preallocating memory for page builders (#20043)	2 months ago
benclive	a6de577b3a	chore: Fix page skipping with empty predicates (#19602)	3 months ago
Ashwanth	bfd70602dd	chore: correct dataobj reader rows read return value (#19583)	3 months ago
Robert Fratto	7556192423	chore(dataobj): use single Zstd decoder (#19508)	4 months ago
benclive	2dac5ef8a7	chore: Write empty message values in logs section writer (#19359)	4 months ago
Sophie Waldman	af6983e9de	fix: Add special case handling for comparing empty and nil values (#19348) ... We can have a predicate checking for an empty string, e.g. {"foo" = ""}. If a row has no value for the column "foo", it will return a nil value for that column. Thus, when we go to compare the row with the predicate, we will be comparing an empty value to a nil value. This change adjusts the comparison check so that we correctly handle this special case.	4 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	2ecf07d52c	chore(dataobj): Fix panic in column builder (#19153) ... There is an edge case in the column builder where it panics when the AppendNulls() call tries to append more NULL values than the specified MaxPageRows setting when backfilling rows. In that case the builder would panic because canAppend would always return false even when the previous page was flushed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	515ebea4f2	chore(dataobj): track total download time of page metadata/data (#18896)	6 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694)	6 months ago
Robert Fratto	a5c722cbed	chore(dataobj): list pages across all columns in one request (#18678)	6 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	6 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	7 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	7 months ago
Robert Fratto	26cb50e65a	chore(dataobj): Improve performance of dataset.Value API (#18339) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	7 months ago
Ashwanth	5c02493fe2	chore(dataobj): avoid reading a predicate column more than once (#18324)	7 months ago
Robert Fratto	c5ac922e55	chore(dataobj): Use pooled buffers when decoding pages (#17955)	8 months ago
Christian Haudum	48f1d0ebc0	chore(dataobj): Decompress zstd compressed pages upfront (#17841) ... This PR changes the decompression of data object pages, so that they are decompressed fully upfront into a bytes buffer, instead of passing a streaming zstd decompressor (reader) to the page decoder. This this allows to release the zstd decompressor to the pool earlier.Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	3ad1a64bc1	chore(dataobj): fixes a memory aliasing bug in dataobj reader (#17537)	9 months ago
Ashwanth	5961923836	chore(dataobj): apply predicates sequentially (#17283)	9 months ago
Ashwanth	42e2407137	chore(dataobj): backfill sparse rows using EncodeN (#16485)	9 months ago
Ashwanth	7dd4ab4b09	chore(dataobj): correct stats calculation for prepredicate bytes read (#17279)	10 months ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128)	10 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
Ashwanth	42eed6ded5	chore(dataobj): introduce InPredicate to replace long chain of OR operators (#16919)	10 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	11 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747)	11 months ago