5 Commits (3e415aca3d904fa6fa3cbedfdc7a2bdfd7fb6d41)

Author	SHA1	Message	Date
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	5 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	6 months ago
Robert Fratto	30ccf5172c	chore(util/arrowtest): export arrowtest package (#18255)	6 months ago
Robert Fratto	06cf52757b	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976) ... This PR introduces columnar reading APIs to both the logs and streams sections. As part of this change, both sections now expose the columns stored in the section, which are then used to define predicates on the readers. The columnar reading APIs emit a sequence of `arrow.Record`s, where each record has no more than the batch size passed to the `Reader.Read` method. Unlike the original row-based readers, the columnar readers: * Allow reading a subset of columns * Have column-based predicates that map literally to the new query engine To be able to represent a column's value for use in predicates, we import [github.com/apache/arrow-go/v18/arrow/scalar](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.3.0/arrow/scalar). Section Reader implementations perform mapping to and from the internal `dataset.Value`. The implementation of `logs.Reader` is almost an identical copy of the `streams.Reader` implementation. I've opted for duplicating the implementation in the short term, since there's no obvious way for how to deduplicate it in a way keeps the package structure easy to understand. We can change this in the future if a clean and simple approach presents itself. This PR does not yet update `DataObjScan` to make use of these new APIs; that's being left out of scope for another PR to handle.	7 months ago