6 Commits (main)

Author	SHA1	Message	Date
Christian Haudum	16ccf9be1e	chore(engine): Filter by unknown column (#18131) ... The new engine fails to execute a query when an `arrow.Record` (batch) does not contain the column on which a label filter is used, e.g. `| foo="bar"` where `foo` is neither a label nor a structured metadata. This change also fixed an incorrect conversion of the label filter operator `=`, which was converted to `MATCH_STR`, but requires to be `EQ`. The new engine reflects current behaviour of a label filter `| foo=""` where `foo` is undefined. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 week ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608) ... Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456) ... **What this PR does / why we need it**: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	208e519564	chore(engine): Add framework for function evaluation (#17374) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	1f8e4eebfb	chore(engine): Implement execution pipeline for SortMerge operator (#17406) ... This PR contains an implementation of the k-way merge operation without using a heap, like @rfratto described [here](https://github.com/grafana/loki/pull/17280). The SortMerge is implemented only using slices: * Maintain the following invariant: * For each input pipeline, we store the next record to process. (this already exists as `HeapSortMerge.batches`) * Additionally for each record, track the starting slice offset (which resets to zero whenever a new record is loaded in). * Iteration stops when all input pipelines have been exhausted (no change from how this is now). * To get the next record: * Iterate through each record, looking at the value from their starting slice offset. * Track the top _two_ winners (e.g., the record whose next value is the smallest and the record whose next value is the next smallest). * Find the largest offset in the starting record whose value is still less than the value of the runner-up record from the previous step. * Return the slice of that record using the two offsets, and update the stored offset of the returned record for the next call to `Read`. This approach, like the one with heap, still requires to concatenate (coalesce) the single row records - which is not implemented in this PR yet. On that note, single row records are the worst case scenario with this implementation, not necessarily the regular case. **Update:** After an offline discussion, @owen-d and I agreed on ignoring the worst-case scenario of single-row records for now. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257) ... The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago