apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	4841b2abe2	chore(engine): provide basic implementation of DataObjScan (#17568 ) Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376 )	1 year ago
Christian Haudum	3c72560c71	chore(engine): Add basic metrics for next generation execution engine (#17475 ) This PR adds basic metrics for observe the duration of the different phases of the new execution engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447 ) This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	208e519564	chore(engine): Add framework for function evaluation (#17374 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	1f8e4eebfb	chore(engine): Implement execution pipeline for SortMerge operator (#17406 ) This PR contains an implementation of the k-way merge operation without using a heap, like @rfratto described [here](https://github.com/grafana/loki/pull/17280). The SortMerge is implemented only using slices: * Maintain the following invariant: * For each input pipeline, we store the next record to process. (this already exists as `HeapSortMerge.batches`) * Additionally for each record, track the starting slice offset (which resets to zero whenever a new record is loaded in). * Iteration stops when all input pipelines have been exhausted (no change from how this is now). * To get the next record: * Iterate through each record, looking at the value from their starting slice offset. * Track the top _two_ winners (e.g., the record whose next value is the smallest and the record whose next value is the next smallest). * Find the largest offset in the starting record whose value is still less than the value of the runner-up record from the previous step. * Return the slice of that record using the two offsets, and update the stored offset of the returned record for the next call to `Read`. This approach, like the one with heap, still requires to concatenate (coalesce) the single row records - which is not implemented in this PR yet. On that note, single row records are the worst case scenario with this implementation, not necessarily the regular case. Update: After an offline discussion, @owen-d and I agreed on ignoring the worst-case scenario of single-row records for now. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	cf54d502fc	feat(dataobj, executor): filter node execution (#17327 ) # Add Filter Pipeline Implementation This PR implements a filter pipeline for Loki's engine executor component. The filter operator enables row-level filtering of Arrow record batches based on boolean expressions. This is a fundamental component for query execution. ## Changes - Implement `NewFilterPipeline` for evaluating filter predicates on record batches - Add `filterBatch` utility to efficiently filter Arrow records - Update `executeFilter` in the executor to use the new implementation - Add comprehensive test suite for the filter pipeline: - Basic literal predicates (true/false) - Column reference predicates - Empty batch handling - Multiple batch processing ## Implementation Notes The filter implementation follows the same pattern as the project pipeline, evaluating predicates against each record and creating a new filtered record based on the results. The implementation supports compound predicates with AND logic. This implementation provides a foundation for more advanced filtering operations as the expression evaluator is enhanced to support more complex expressions.	1 year ago
Owen Diehl	395030b130	feat(dataobj executor): project node (#17312 )	1 year ago
Owen Diehl	920e3b23ed	chore(dataobj,testing,executor): pipeline equality testing (#17311 )	1 year ago
Christian Haudum	e828c48702	chore(engine): Implement execution pipeline for the limit operator (#17264 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	cff0df63b3	chore(dataobj,executor): CSV arrow batches & pipeline helpers (#17290 )	1 year ago
Christian Haudum	64c477b1f7	chore(engine): Add framework for query executor (#17260 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257 ) The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032 ) This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	ba61aa88de	chore(engine): Integrate metastore into physical query planning (#17016 ) This PR integrates the data object lookup using the metastore into the query planning. To achieve that, I extended the `Metastore` interface with an additional function that not only returns the data object paths, but also the stream IDs of streams that match the given label matchers. ```go type Metastore interface { ... // StreamsIDs returns object store paths and stream IDs for all matching objects for the given matchers between [start,end] StreamIDs(ctx context.Context, start, end time.Time, matchers ...*labels.Matcher) ([]string, [][]int64, error) ... } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6824f4e575	chore(engine): Add printer for Mermaid diagram format (#17000 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	04d2ba3ca8	chore(engine): Add simple plan optimizer (#16965 ) This PR adds a plan optimizer that pushes down limit and filter predicates to the scan node. ### Example Plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── Filter #0xc00039dd10 │ └── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Filter #0xc00039ddd0 │ └── Predicate expr=GT(age.metadata, 21) └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=0 └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=0 ``` Optimized plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=1000 │ ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) │ └── Predicate expr=GT(age.metadata, 21) └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=1000 ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Predicate expr=GT(age.metadata, 21) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	40223d7320	chore(engine): Unify `Literal` and `ColumnRef` across logical and physical plan (#16927 ) Literals and column refs are used by both the logical and physical plan. However, the interfaces they need to implement are partly private (`isValue()` and `isExpression()`). Therefore they are wrapped in their own struct that implement these interfaces. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	138093b900	chore(planner): Convert AST into logical plan (#16916 ) Minimal implementation of the logical planner that converts a parsed LogQL statement in AST form into a logical plan in SSA form. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	0a3230f457	chore(ast): Update WalkFn signature to return bool (#16989 ) The boolean return value indicates whether walking/traversing should be continued with child nodes. This PR does not change any existing traversal behaviour. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	efe15486fe	chore(engine): Add first stage of physical query planning (#16891 ) Planning consists of two stages: 1. Conversion stage: Instructions from the logical plan are converted into Nodes in the physical plan. Most instructions can be translated into a single Node. However, `MakeTable` translates into multiple `DataObjScan` nodes. 2. Pushdown stage: a) Push down the limit of the `Limit` node to the `DataObjScan` nodes. b) Push down filter predicates from the `Filter` nodes to the `DataObjScan` nodes. This PR implements only the first stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	f79906ca75	chore(engine): Add utility function `canExecuteWithNewEngine` (#16892 ) The function determines whether a query can be executed by the new execution engine. In the first step, the new execution engine will only be able to execute limited/filter queries without any transformation stages. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Robert Fratto	cbe97e6c1a	chore(engine): Update logical planner types to be structured as SSA (#16841 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	c51fa39684	chore(dataobj): pass down line filters to logs reader (#16802 )	1 year ago
Christian Haudum	9691f3e5dc	chore: Move `pkg/dataobj/planner` to `pkg/engine/planner` (#16815 ) The package pkg/dataobj/planner/logical was initially created within the package pkg/dataobj because of the dependency on the internal package pkg/dataobj/internal/metadatamd. However, the package should be located inside a separate package without the dependency on the dataobj package, since the engine should be storage agnostic. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago

25 Commits (4841b2abe265f5d945befd90565eb679ddc5206c)