12 Commits (60c76318e66d33a726b41d8542e71663e581c92a)

Author	SHA1	Message	Date
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456) ... **What this PR does / why we need it**: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	64c477b1f7	chore(engine): Add framework for query executor (#17260) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032) ... This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	ba61aa88de	chore(engine): Integrate metastore into physical query planning (#17016) ... This PR integrates the data object lookup using the metastore into the query planning. To achieve that, I extended the `Metastore` interface with an additional function that not only returns the data object paths, but also the stream IDs of streams that match the given label matchers. ```go type Metastore interface { ... // StreamsIDs returns object store paths and stream IDs for all matching objects for the given matchers between [start,end] StreamIDs(ctx context.Context, start, end time.Time, matchers ...*labels.Matcher) ([]string, [][]int64, error) ... } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6824f4e575	chore(engine): Add printer for Mermaid diagram format (#17000) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	04d2ba3ca8	chore(engine): Add simple plan optimizer (#16965) ... This PR adds a plan optimizer that pushes down limit and filter predicates to the scan node. ### Example Plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── Filter #0xc00039dd10 │ └── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Filter #0xc00039ddd0 │ └── Predicate expr=GT(age.metadata, 21) └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=0 └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=0 ``` Optimized plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=1000 │ ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) │ └── Predicate expr=GT(age.metadata, 21) └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=1000 ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Predicate expr=GT(age.metadata, 21) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	40223d7320	chore(engine): Unify `Literal` and `ColumnRef` across logical and physical plan (#16927) ... Literals and column refs are used by both the logical and physical plan. However, the interfaces they need to implement are partly private (`isValue()` and `isExpression()`). Therefore they are wrapped in their own struct that implement these interfaces. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	138093b900	chore(planner): Convert AST into logical plan (#16916) ... Minimal implementation of the logical planner that converts a parsed LogQL statement in AST form into a logical plan in SSA form. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	efe15486fe	chore(engine): Add first stage of physical query planning (#16891) ... Planning consists of two stages: 1. Conversion stage: Instructions from the logical plan are converted into Nodes in the physical plan. Most instructions can be translated into a single Node. However, `MakeTable` translates into multiple `DataObjScan` nodes. 2. Pushdown stage: a) Push down the limit of the `Limit` node to the `DataObjScan` nodes. b) Push down filter predicates from the `Filter` nodes to the `DataObjScan` nodes. **This PR implements only the first stage.** Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Robert Fratto	cbe97e6c1a	chore(engine): Update logical planner types to be structured as SSA (#16841) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	c51fa39684	chore(dataobj): pass down line filters to logs reader (#16802)	1 year ago
Christian Haudum	9691f3e5dc	chore: Move `pkg/dataobj/planner` to `pkg/engine/planner` (#16815) ... The package pkg/dataobj/planner/logical was initially created within the package pkg/dataobj because of the dependency on the internal package pkg/dataobj/internal/metadatamd. However, the package should be located inside a separate package without the dependency on the dataobj package, since the engine should be storage agnostic. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago