10 Commits (main)

Author	SHA1	Message	Date
Robert Fratto	c12b305b05	chore(memory): permit slicing memory.Bitmap (#20699) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	9d9d8ce14d	chore(engine): introduce experimental expression engine (#20626)	3 weeks ago
Robert Fratto	747d75aa78	chore: make memory, columnar APIs more idiomatic (#20622)	4 weeks ago
Robert Fratto	3e3a293215	chore(columnar): switch to generic Number type (#20580)	4 weeks ago
Ivan Kalita	86f87e447c	chore(dataobj): use reader adapter (#20572) ... Continuation of #20525, I'm updating logs, pointers, streams readers to use reader adapter. Also adding binary support to arrowconv.	4 weeks ago
Ivan Kalita	aab681617b	chore(dataobj): add reader_adapter (#20525) ... Add a temporary translation layer []dataset.Row -> columnar.RecordBatch -> arrow.RecordBatch. - reader_adapter.go is temporary that's why I put it into the old columnar package - I added builders to reader_adapter.go and not to pkg/columnar.... as I'm not sure if we really would need them anywhere except for this adapter. If so I'm happy to move them to pkg/columnar. - I only updated indexpointers reader for now but will add support for more readers in the next PRs	1 month ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	4 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	7 months ago
Robert Fratto	06cf52757b	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976) ... This PR introduces columnar reading APIs to both the logs and streams sections. As part of this change, both sections now expose the columns stored in the section, which are then used to define predicates on the readers. The columnar reading APIs emit a sequence of `arrow.Record`s, where each record has no more than the batch size passed to the `Reader.Read` method. Unlike the original row-based readers, the columnar readers: * Allow reading a subset of columns * Have column-based predicates that map literally to the new query engine To be able to represent a column's value for use in predicates, we import [github.com/apache/arrow-go/v18/arrow/scalar](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.3.0/arrow/scalar). Section Reader implementations perform mapping to and from the internal `dataset.Value`. The implementation of `logs.Reader` is almost an identical copy of the `streams.Reader` implementation. I've opted for duplicating the implementation in the short term, since there's no obvious way for how to deduplicate it in a way keeps the package structure easy to understand. We can change this in the future if a clean and simple approach presents itself. This PR does not yet update `DataObjScan` to make use of these new APIs; that's being left out of scope for another PR to handle.	9 months ago