apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ashwanth	baf10907bd	chore: introduces TopK node and replace the usage of SortMerge (#19520 )	3 months ago
Robert Fratto	3a0480698c	chore(engine): add Parallelize hint node (#19521 )	3 months ago
Robert Fratto	cfd07c3683	chore(engine): simplify Pipeline interface (#19518 )	3 months ago
Ashwanth	fa8bd848f6	chore: update buildTable to sort rows using `BuilderOptions.SortOrder` (#19515 )	3 months ago
Christian Haudum	849a9214fa	chore: Release column vectors and their arrays correctly (#19496 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	8b7ef09e69	chore(engine): Cleanup: remove state from pipelines (#19495 ) * Do not pass `state` in `genericPipeline` but instead return `(arrow.Record, error)` in `readFunc` * Remove unused `state` fields from pipelines.	3 months ago
Christian Haudum	3c5f759f24	chore(engine): Add compatibility node for parsed columns (#19485 ) The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the second case. This PR is a follow up on #19470 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ivan Kalita <input.txt@hotmail.com>	3 months ago
Christian Haudum	130a29efe9	chore: Revert "chore: remove prefetching for sortmerge" (#19491 )	3 months ago
Christian Haudum	662e1b93f1	chore(engine): Add `LOGQL_COMPAT` marker to logical plan (#19488 ) The marker operation `LOGQL_COMPAT` indicates that the query is expected to return results that are compatible with the old v1 chunks engine. It is used in the physical planner to add required `ColumnCompat` nodes to the plan that handle columns with colliding names but of different semantic types (label, metadata, parsed).	3 months ago
Ashwanth	26c07fa6bc	chore(planner): fix limits pushdown to work with empty predicates (#19489 )	3 months ago
Ashwanth	0f28e4be9e	chore: remove prefetching for sortmerge (#19486 )	3 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470 ) ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Ashwanth	6a829e3d6f	chore(engine): allow usage of empty groupby in aggregations (#19344 )	3 months ago
Christian Haudum	c280531e05	chore: Fix column name regression (#19468 ) The temporary streamID column from the dataobj scan node caused an error when the expression evaluator tried to parse the FQN, which was not valid any more after the recent change that implemented the semantic naming conventions. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	2214a700b3	chore(engine): Implement new semantic column naming (#19415 ) This PR implements the column naming convention introduced in https://github.com/grafana/loki/pull/19396 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Stas Spiridonov	16dab82593	chore: consistent expressionEvaluator.eval result memory ownership (#19438 )	3 months ago
Robert Fratto	17b1418a92	chore(engine): move DAG logic to common package (#19406 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Ashwanth	c24675f854	chore: adds separate v2 engine config section (#19423 )	3 months ago
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396 ) In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384 ) This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Stas Spiridonov	1474ede675	feat(engine): More aggregations for the new query engine (#19350 )	3 months ago
Trevor Whitney	aec34ed30a	feat: add json parsing stage to v2 engine (#19240 )	4 months ago
Trevor Whitney	fc58d21eb3	chore: exclude types package from linting rules. (#19220 )	4 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931 )	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206 )	4 months ago
Robert Fratto	ed1b61d253	chore(engine): improve performance of topk pipeline (#18593 )	6 months ago
Christian Haudum	4305a6aa61	chore(engine): Make literals simple types (#17733 ) Representing data types as simple types (string, int64, float64) makes them more efficient to cast without conversion. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Robert Fratto	5266865fef	chore(engine): add implementation for performing topk across streams of Arrow records (#18330 ) This commit adds a `engine.topkBatch` type, which can be used by the implementation of execution nodes to perform compute the set of topk rows from a stream of records. This is needed as part of #17976, and more generally for efficiently sorting rows when a limit is also used. Implementing this required quite a few helper utilities in the new `arrowagg` package. The `arrowagg` package provides utilities for aggregating various Arrow types: * `arrowagg.Arrays` to concatenate a stream of `arrow.Array`s. * `arrayagg.Records` to concatenate a stream of `arrow.Record`s, creating a merged schema. * `arrowagg.Mapper` for quickly finding fields in incoming schemas, since arrow-go can return false when comparing two identical schemas/fields. Co-authored-by: Ashwanth <iamashwanth@gmail.com>	6 months ago
Ashwanth	50374f0bb1	chore(engine): column expression evaluation for ambiguous column type look-ups (#18228 )	6 months ago
Ashwanth	646ebe81d8	chore(engine): adds AST conversion and result builders for metric queries (#18166 )	6 months ago
Ashwanth	635896dab9	chore(engine): adds vector aggregation planning and execution nodes (#18100 )	7 months ago
Christian Haudum	3956ea1be7	chore(engine): Map line filter expressions to data object predicates (#18102 ) This PR converts line filter expressions into message predicates for the data object scan, instead of returning an error. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Ashwanth	1effa30851	chore(engine): implement range aggregation operator (#17997 )	7 months ago
Ashwanth	e1206fb84d	feat(planning): adds physical and logical nodes for range aggregation (#17974 ) Signed-off-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	7 months ago
Christian Haudum	cbb5be6d57	feat(engine): Implements predicate mapping for DataObjScan (#17621 ) This is an initial implementation of `buildLogsPredicate(expr physical.Expression) (dataobj.LogsPredicate, error)`. While there have been discussions about bypassing this approach in favor of directly using `dataset.Reader` and `dataset.Predicate`, this PR provides essential supported functionality to unblock progress in the interim. Specifically, it handles mapping common timestamp and metadata equality predicates. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	8 months ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608 ) Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456 ) What this PR does / why we need it: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447 ) This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	208e519564	chore(engine): Add framework for function evaluation (#17374 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257 ) The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032 ) This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	ba61aa88de	chore(engine): Integrate metastore into physical query planning (#17016 ) This PR integrates the data object lookup using the metastore into the query planning. To achieve that, I extended the `Metastore` interface with an additional function that not only returns the data object paths, but also the stream IDs of streams that match the given label matchers. ```go type Metastore interface { ... // StreamsIDs returns object store paths and stream IDs for all matching objects for the given matchers between [start,end] StreamIDs(ctx context.Context, start, end time.Time, matchers ...*labels.Matcher) ([]string, [][]int64, error) ... } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	40223d7320	chore(engine): Unify `Literal` and `ColumnRef` across logical and physical plan (#16927 ) Literals and column refs are used by both the logical and physical plan. However, the interfaces they need to implement are partly private (`isValue()` and `isExpression()`). Therefore they are wrapped in their own struct that implement these interfaces. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	138093b900	chore(planner): Convert AST into logical plan (#16916 ) Minimal implementation of the logical planner that converts a parsed LogQL statement in AST form into a logical plan in SSA form. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	efe15486fe	chore(engine): Add first stage of physical query planning (#16891 ) Planning consists of two stages: 1. Conversion stage: Instructions from the logical plan are converted into Nodes in the physical plan. Most instructions can be translated into a single Node. However, `MakeTable` translates into multiple `DataObjScan` nodes. 2. Pushdown stage: a) Push down the limit of the `Limit` node to the `DataObjScan` nodes. b) Push down filter predicates from the `Filter` nodes to the `DataObjScan` nodes. This PR implements only the first stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago

45 Commits (b95bfabd422a9b458dbbbde3eda33a95ed768a3a)