apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Stas Spiridonov	bedfb78378	chore: Thor query engine memory improvements, part 2 (#20473 )	2 months ago
Ashwanth	8a6c4f8611	chore(planner): add filter stages to drop lines matching delete requests (#20395 )	2 months ago
Christian Haudum	f066d378a5	chore(engine): Do not embed `types.Literal` interface in logical and physical Literal (#19858 ) This ensures that logical.Literal and physical.LiteralExpr do not implicitly implement the types.Literal interface, which caused a bug fixed with #19853 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750 ) Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	4 months ago
benclive	01cab53447	feat(engine): implement strict and keepEmpty logfmt parsing (#19668 )	4 months ago
Trevor Whitney	8b39d4d077	refactor: implment parse as a projection (#19579 ) Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	73bc30d9a8	perf(engine): Improve regexp expression evaluation (#19644 ) This PR changes the evaluation of regex-match binary expression evaluations so that the regexp on right hand side of the expression is only compiled once for each batch evaluation, rather than for each row. This saves a ton of allocations and improves general performance of queries with a regexp line or label filter.	4 months ago
Christian Haudum	9e76f1d251	chore(engine): Refactor `ColumnVector` (#19549 ) This PR removes the need of the ColumnVector as a result of expression evaluation and instead returns an arrow.Array. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569 )	5 months ago
Trevor Whitney	13429aa1ea	refactor: move allocator into execution Context (#19550 )	5 months ago
Trevor Whitney	3ce6fa2946	feat: implement `unwrap` as a projection (#19409 )	5 months ago
Christian Haudum	849a9214fa	chore: Release column vectors and their arrays correctly (#19496 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	2214a700b3	chore(engine): Implement new semantic column naming (#19415 ) This PR implements the column naming convention introduced in https://github.com/grafana/loki/pull/19396 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Stas Spiridonov	16dab82593	chore: consistent expressionEvaluator.eval result memory ownership (#19438 )	5 months ago
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396 ) In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384 ) This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Christian Haudum	4305a6aa61	chore(engine): Make literals simple types (#17733 ) Representing data types as simple types (string, int64, float64) makes them more efficient to cast without conversion. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	50374f0bb1	chore(engine): column expression evaluation for ambiguous column type look-ups (#18228 )	8 months ago
Christian Haudum	16ccf9be1e	chore(engine): Filter by unknown column (#18131 ) The new engine fails to execute a query when an `arrow.Record` (batch) does not contain the column on which a label filter is used, e.g. `\| foo="bar"` where `foo` is neither a label nor a structured metadata. This change also fixed an incorrect conversion of the label filter operator `=`, which was converted to `MATCH_STR`, but requires to be `EQ`. The new engine reflects current behaviour of a label filter `\| foo=""` where `foo` is undefined. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Ashwanth	1effa30851	chore(engine): implement range aggregation operator (#17997 )	9 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456 ) What this PR does / why we need it: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Christian Haudum	208e519564	chore(engine): Add framework for function evaluation (#17374 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	1f8e4eebfb	chore(engine): Implement execution pipeline for SortMerge operator (#17406 ) This PR contains an implementation of the k-way merge operation without using a heap, like @rfratto described [here](https://github.com/grafana/loki/pull/17280). The SortMerge is implemented only using slices: * Maintain the following invariant: * For each input pipeline, we store the next record to process. (this already exists as `HeapSortMerge.batches`) * Additionally for each record, track the starting slice offset (which resets to zero whenever a new record is loaded in). * Iteration stops when all input pipelines have been exhausted (no change from how this is now). * To get the next record: * Iterate through each record, looking at the value from their starting slice offset. * Track the top _two_ winners (e.g., the record whose next value is the smallest and the record whose next value is the next smallest). * Find the largest offset in the starting record whose value is still less than the value of the runner-up record from the previous step. * Return the slice of that record using the two offsets, and update the stored offset of the returned record for the next call to `Read`. This approach, like the one with heap, still requires to concatenate (coalesce) the single row records - which is not implemented in this PR yet. On that note, single row records are the worst case scenario with this implementation, not necessarily the regular case. Update: After an offline discussion, @owen-d and I agreed on ignoring the worst-case scenario of single-row records for now. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257 ) The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago

16 Commits (080a9300a1daccac562ae05c0d3be45e09a6ca91)