21 Commits (7c85dd9e2d2b463d8d362aecefecfb2be02adaaf)

Author	SHA1	Message	Date
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Robert Fratto	bd4331363b	chore: align benchmark results between chunks and dataobjs (#17127)	9 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	10 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747)	10 months ago
Robert Fratto	cd11367148	chore(dataobj): Wire dataset.Reader into dataobj.LogsReader and dataobj.StreamsReader (#16701) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	10 months ago
Robert Fratto	7f0dc01e21	chore(dataobj): introduce more efficient dataset reading (#16429) ... This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Jackson Coelho	290c14a8cb	chore: fix flush process for dataobj (#16393)	11 months ago
Ashwanth	651d410eea	chore(dataobj): add "uncompressed size" column to streams section (#16322)	11 months ago
Owen Diehl	66889ec51d	feat(dataobj): cardinality estimation (#16233)	11 months ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055)	11 months ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	11 months ago
Robert Fratto	5ccc6790cb	chore(dataobj): support computation of min/max values in pages and columns (#16015)	11 months ago
Robert Fratto	1adb1e39d3	chore(dataobj): add initial high-level APIs for reading streams and log records (#15974)	11 months ago
Robert Fratto	5929b05a64	chore(dataobj): deduplicate decoder code across bucket and io.ReadSeeker (#15945)	11 months ago
Robert Fratto	948f5c5f3e	chore(dataobj): Reintroduce sorting of the logs section (#15906)	12 months ago
Robert Fratto	7eb5053d1c	chore(dataobj): improve performance of Streams.EstimatedSize (#15833)	12 months ago
Robert Fratto	093e36af58	chore(dataobj): instrument data objects (#15803)	12 months ago
Robert Fratto	848c31fce9	chore(dataobj): temporarily disable sorting (#15799)	12 months ago
Robert Fratto	127ad5326f	chore(dataobj): support for building and flushing data objects (#15780)	12 months ago
Robert Fratto	11714a71c7	chore(dataobj): add logs tracking and encoding (#15757)	12 months ago
Robert Fratto	d1fd0b9ffc	chore(dataobj): add streams tracking and encoding (#15713)	12 months ago