9 Commits (bf06e45bcfc7745ee26798fdfbbbe31ecfb3bd3f)

Author	SHA1	Message	Date
Robert Fratto	d85fb0317f	chore(dataobj): Add Open semantics to section readers (#20790) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Robert Fratto	7e03400309	chore(dataobj): rename dataset.ReaderOptions to dataset.RowReaderOptions (#20756)	4 months ago
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629)	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499)	11 months ago
benclive	f075da4f94	chore: Use correct TimeRange predicate for indexpointers (#18529)	11 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	11 months ago
Periklis Tsirakidis	6eedcaaded	chore(dataobj): Add new index pointers section type (#18441)	11 months ago