116 Commits (3e415aca3d904fa6fa3cbedfdc7a2bdfd7fb6d41)

Author	SHA1	Message	Date
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Stas Spiridonov	1474ede675	feat(engine): More aggregations for the new query engine (#19350)	3 months ago
Trevor Whitney	aec34ed30a	feat: add json parsing stage to v2 engine (#19240)	3 months ago
Christian Haudum	ed757b50c2	chore(engine): Do not print ID of physical plan nodes (#19254) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241)	3 months ago
Robert Fratto	20e04a4192	chore(engine/logical): push SORT operation towards bottom of plan (#19239)	3 months ago
Christian Haudum	8499840af6	chore(engine): Eliminate no-op Merge/SortMerge nodes during plan optimization (#19238) ... Merge and SortMerge nodes with 1 or less children are effectively no-op nodes and can safely be removed from the plan during the optimization phase. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	35816aa16f	chore(engine): Eliminate unnecessary SortMerge node from physical plan (#19236) ... This PR avoid creating the SortMerge node from the logical.Sort. This node was a no-op and only passed down the sort direction to its children. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Robert Fratto	3700dc5395	chore(engine/logical): fix order/duplicates of SSA form (#19237) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	24e968cfcb	perf(engine): reuse existing array in genericFunction.Evaluate (#19235) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
benclive	4f628c0d8b	chore: Fix time range grouping logic in some cases (#19226)	3 months ago
Trevor Whitney	fc58d21eb3	chore: exclude types package from linting rules. (#19220)	3 months ago
benclive	c23bb7ce4e	chore: Fix Dataobjscan grouping wrt sort order (#19215)	3 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931)	4 months ago
Ashwanth	a5e9964d7d	chore(engine): Retain Null information when filtering rows (#19214)	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	4 months ago
benclive	ad46063601	chore: Empty builder returns non-nil result (#19184)	4 months ago
benclive	b7f14f9554	fix: Stop applying limits pushdown under a filter (#19154)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110) ... This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Sophie Waldman	8a9da8f0e3	chore: Update engine planner to sort by input time range (#18951) ... When convering a [logical.MakeTable] into nodes, keep information on the time range for each location and use the time ranges to sort the nodes created. The processMakeTable method will now always return a root Merge node. Inside the Merge node, overlapping shards will be grouped as DataObjScan nodes inside a SortMerge, while any individual shards without any overlap will be DataObjScan nodes directly under the Merge. Signed-off-by: Sophie Waldman <sophie.waldman@grafana.com>	4 months ago
Ashwanth	d950d4075d	chore(engine): improve performance of matching window calculation (#19013) ... Currently range aggregation linearly scans through all the possible time windows (a 24h query with a 1m step results in 1440 windows) to find the matching ones. This is quite expensive as this matching is done for each row. This PR updates the matching code to use arithmetic which runs in constant time for cases where step >= range and for overlapping windows (step < range) it uses binary search. Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039) ... This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
benclive	84efa59a06	chore: Fix init of engine with metastore (#19064)	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	4 months ago
Robert Fratto	aa66baec97	chore(logqlmodel/stats): print new engine summary in metrics.go (#19052)	4 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Christian Haudum	389fc5ade6	chore(engine): Correctly free record allocations (#19040) ... Fixes panics in the v2 query engine, due to Arrow records being freed at incorrect places. Rather than freeing the previous batch created by the KWayMerge on every Read() call, fully consumed batches by the KWayMerge need to be freed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	b972ccb3f6	chore(planner/physical): simplify replace code (#19003)	4 months ago
Sophie Waldman	3d23e83a8f	chore: Add fixed ordering to edges in plan and only allow for single parent per plan node (#18978) ... When creating a query plan, we now retain the order in which edges were added to the plan. Additionally, no node should ever have more than one parent, so the Plan struct and corresponding functions were simplified accordingly. We want to order inputs in the query plan based on the time range of each input. While it's possible to group the inputs without any change to the Plan struct, there is no guarantee that the inputs will remain in order. To force the ordering to stay the same, we replace non-ordered nodeSet properties with []Node arrays.	4 months ago
Robert Fratto	2326bb0f4b	chore(engine): fix instances of double frees (#18915) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Sophie Waldman	40423f8f04	fix(engine): Don't attempt to release batch reference on read for merge node (#18913) ... Signed-off-by: Sophie Waldman <sophie.waldman@grafana.com>	4 months ago
Ashwanth	827b1f2df4	chore(engine): add support for range metric queries (#18892)	4 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694)	5 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	5 months ago
Robert Fratto	ade03f313c	chore(engine): fix flaky test (#18739)	5 months ago
Robert Fratto	f98c932dcb	chore(engine): propagate batch and page cache size down to streamsView (#18744)	5 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622)	5 months ago
Robert Fratto	385d397404	chore(engine/executor): reduce cost of executing DataObjScan plan node (#18679)	5 months ago
Robert Fratto	e751db8194	chore(engine): fix race condition when closing prefetch wrapper (#18668)	5 months ago
Ashwanth	7c5bce70dc	chore(engine): introduce merge node (#18641)	5 months ago
Robert Fratto	a3380f6480	chore(engine): add tracing instrumentation (#18648)	5 months ago
Robert Fratto	56e7318bec	chore(engine): don't sort metric queries (#18594)	5 months ago
Robert Fratto	ed1b61d253	chore(engine): improve performance of topk pipeline (#18593)	5 months ago
Robert Fratto	e1bc2f9d7f	chore(engine): fix panic when closing uninitialized prefetchWrapper (#18592)	5 months ago
Ashwanth	cc9f9e199f	chore(engine): add predicate columns to projection list for metric queries (#18608)	5 months ago