4 Commits (cd3fd30a46d7c59fea5b124f3f252df960ca2148)

Author	SHA1	Message	Date
Robert Fratto	7e03400309	chore(dataobj): rename dataset.ReaderOptions to dataset.RowReaderOptions (#20756)	1 month ago
Robert Fratto	747d75aa78	chore: make memory, columnar APIs more idiomatic (#20622)	1 month ago
Ivan Kalita	aab681617b	chore(dataobj): add reader_adapter (#20525) ... Add a temporary translation layer []dataset.Row -> columnar.RecordBatch -> arrow.RecordBatch. - reader_adapter.go is temporary that's why I put it into the old columnar package - I added builders to reader_adapter.go and not to pkg/columnar.... as I'm not sure if we really would need them anywhere except for this adapter. If so I'm happy to move them to pkg/columnar. - I only updated indexpointers reader for now but will add support for more readers in the next PRs	2 months ago
Ivan Kalita	827292d8ed	chore(metastore): add columnar indexpointers reader (#20068) ... Similar to https://github.com/grafana/loki/pull/20053 and https://github.com/grafana/loki/pull/19992 I add a columnar reader for index pointers sections.	3 months ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	4 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	5 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	8 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	8 months ago
Robert Fratto	06cf52757b	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976) ... This PR introduces columnar reading APIs to both the logs and streams sections. As part of this change, both sections now expose the columns stored in the section, which are then used to define predicates on the readers. The columnar reading APIs emit a sequence of `arrow.Record`s, where each record has no more than the batch size passed to the `Reader.Read` method. Unlike the original row-based readers, the columnar readers: * Allow reading a subset of columns * Have column-based predicates that map literally to the new query engine To be able to represent a column's value for use in predicates, we import [github.com/apache/arrow-go/v18/arrow/scalar](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.3.0/arrow/scalar). Section Reader implementations perform mapping to and from the internal `dataset.Value`. The implementation of `logs.Reader` is almost an identical copy of the `streams.Reader` implementation. I've opted for duplicating the implementation in the short term, since there's no obvious way for how to deduplicate it in a way keeps the package structure easy to understand. We can change this in the future if a clean and simple approach presents itself. This PR does not yet update `DataObjScan` to make use of these new APIs; that's being left out of scope for another PR to handle.	9 months ago