5 Commits (main)

Author	SHA1	Message	Date
Ashwanth	356cd36d8e	chore: close stream section reader (#20107)	3 weeks ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	3 weeks ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	2 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Robert Fratto	f98c932dcb	chore(engine): propagate batch and page cache size down to streamsView (#18744)	5 months ago
Robert Fratto	385d397404	chore(engine/executor): reduce cost of executing DataObjScan plan node (#18679)	5 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	5 months ago