6 Commits (08e3c4385f7830d7097bc65e22aaa846c7f3dc89)

Author	SHA1	Message	Date
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821)	2 months ago
Trevor Whitney	8b39d4d077	refactor: implment parse as a projection (#19579) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Trevor Whitney	3ce6fa2946	feat: implement `unwrap` as a projection (#19409)	3 months ago
Christian Haudum	290d1b711f	chore(engine): Add support for drop stage (drop projection) (#19533) ... This PR add support for dropping columns from the schema. The drop projection only supports dropping column by name, not by matcher.	3 months ago
Robert Fratto	bd3f3dabe1	chore(engine): introduce ScanSet node (#19524) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Trevor Whitney	aec34ed30a	feat: add json parsing stage to v2 engine (#19240)	4 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931)	4 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110) ... This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499)	6 months ago
Christian Haudum	cea56469b9	chore(engine): Pass context to Pipeline's Read() function (#18459) ... This change unifies how the request context is passed through the execution of pipelines. Instead of storing the context on pipelines, pass them as argument to the `Read()` function. This change alters the interface method of `Pipeline` from `Read() bool` to `Read(context.Context) bool`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Robert Fratto	4841b2abe2	chore(engine): provide basic implementation of DataObjScan (#17568) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	8 months ago
Christian Haudum	1f8e4eebfb	chore(engine): Implement execution pipeline for SortMerge operator (#17406) ... This PR contains an implementation of the k-way merge operation without using a heap, like @rfratto described [here](https://github.com/grafana/loki/pull/17280). The SortMerge is implemented only using slices: * Maintain the following invariant: * For each input pipeline, we store the next record to process. (this already exists as `HeapSortMerge.batches`) * Additionally for each record, track the starting slice offset (which resets to zero whenever a new record is loaded in). * Iteration stops when all input pipelines have been exhausted (no change from how this is now). * To get the next record: * Iterate through each record, looking at the value from their starting slice offset. * Track the top _two_ winners (e.g., the record whose next value is the smallest and the record whose next value is the next smallest). * Find the largest offset in the starting record whose value is still less than the value of the runner-up record from the previous step. * Return the slice of that record using the two offsets, and update the stored offset of the returned record for the next call to `Read`. This approach, like the one with heap, still requires to concatenate (coalesce) the single row records - which is not implemented in this PR yet. On that note, single row records are the worst case scenario with this implementation, not necessarily the regular case. **Update:** After an offline discussion, @owen-d and I agreed on ignoring the worst-case scenario of single-row records for now. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Owen Diehl	cf54d502fc	feat(dataobj, executor): filter node execution (#17327) ... # Add Filter Pipeline Implementation This PR implements a filter pipeline for Loki's engine executor component. The filter operator enables row-level filtering of Arrow record batches based on boolean expressions. This is a fundamental component for query execution. ## Changes - Implement `NewFilterPipeline` for evaluating filter predicates on record batches - Add `filterBatch` utility to efficiently filter Arrow records - Update `executeFilter` in the executor to use the new implementation - Add comprehensive test suite for the filter pipeline: - Basic literal predicates (true/false) - Column reference predicates - Empty batch handling - Multiple batch processing ## Implementation Notes The filter implementation follows the same pattern as the project pipeline, evaluating predicates against each record and creating a new filtered record based on the results. The implementation supports compound predicates with AND logic. This implementation provides a foundation for more advanced filtering operations as the expression evaluator is enhanced to support more complex expressions.	9 months ago
Owen Diehl	395030b130	feat(dataobj executor): project node (#17312)	9 months ago
Christian Haudum	e828c48702	chore(engine): Implement execution pipeline for the limit operator (#17264) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	64c477b1f7	chore(engine): Add framework for query executor (#17260) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago