10 Commits (60c76318e66d33a726b41d8542e71663e581c92a)

Author	SHA1	Message	Date
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608) ... Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	0ea1097c8f	chore(engine): Use configured storage bucket in execution context (#17583) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456) ... **What this PR does / why we need it**: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376)	8 months ago
Christian Haudum	3c72560c71	chore(engine): Add basic metrics for next generation execution engine (#17475) ... This PR adds basic metrics for observe the duration of the different phases of the new execution engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447) ... This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032) ... This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	0a3230f457	chore(ast): Update WalkFn signature to return bool (#16989) ... The boolean return value indicates whether walking/traversing should be continued with child nodes. This PR does not change any existing traversal behaviour. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	f79906ca75	chore(engine): Add utility function `canExecuteWithNewEngine` (#16892) ... The function determines whether a query can be executed by the new execution engine. In the first step, the new execution engine will only be able to execute limited/filter queries without any transformation stages. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago