apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	36bcb5aa2d	chore(dataobj): fix regression in allocations of row readers (#17784 )	8 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762 ) Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	8 months ago
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	cbb5be6d57	feat(engine): Implements predicate mapping for DataObjScan (#17621 ) This is an initial implementation of `buildLogsPredicate(expr physical.Expression) (dataobj.LogsPredicate, error)`. While there have been discussions about bypassing this approach in favor of directly using `dataset.Reader` and `dataset.Predicate`, this PR provides essential supported functionality to unblock progress in the interim. Specifically, it handles mapping common timestamp and metadata equality predicates. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	8 months ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608 ) Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	0ea1097c8f	chore(engine): Use configured storage bucket in execution context (#17583 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	23c4f8d89a	chore(engine): Fix regression introduced with merging PR #17456 (#17578 ) `types.ColumnTypeMetadataKey` was renamed to `types.MetadataKeyColumnType` in https://github.com/grafana/loki/pull/17456 However, https://github.com/grafana/loki/pull/17568 was merged before that, which still used the old name. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456 ) What this PR does / why we need it: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	4841b2abe2	chore(engine): provide basic implementation of DataObjScan (#17568 ) Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	8 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376 )	9 months ago
Christian Haudum	3c72560c71	chore(engine): Add basic metrics for next generation execution engine (#17475 ) This PR adds basic metrics for observe the duration of the different phases of the new execution engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447 ) This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	208e519564	chore(engine): Add framework for function evaluation (#17374 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	1f8e4eebfb	chore(engine): Implement execution pipeline for SortMerge operator (#17406 ) This PR contains an implementation of the k-way merge operation without using a heap, like @rfratto described [here](https://github.com/grafana/loki/pull/17280). The SortMerge is implemented only using slices: * Maintain the following invariant: * For each input pipeline, we store the next record to process. (this already exists as `HeapSortMerge.batches`) * Additionally for each record, track the starting slice offset (which resets to zero whenever a new record is loaded in). * Iteration stops when all input pipelines have been exhausted (no change from how this is now). * To get the next record: * Iterate through each record, looking at the value from their starting slice offset. * Track the top _two_ winners (e.g., the record whose next value is the smallest and the record whose next value is the next smallest). * Find the largest offset in the starting record whose value is still less than the value of the runner-up record from the previous step. * Return the slice of that record using the two offsets, and update the stored offset of the returned record for the next call to `Read`. This approach, like the one with heap, still requires to concatenate (coalesce) the single row records - which is not implemented in this PR yet. On that note, single row records are the worst case scenario with this implementation, not necessarily the regular case. Update: After an offline discussion, @owen-d and I agreed on ignoring the worst-case scenario of single-row records for now. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Owen Diehl	cf54d502fc	feat(dataobj, executor): filter node execution (#17327 ) # Add Filter Pipeline Implementation This PR implements a filter pipeline for Loki's engine executor component. The filter operator enables row-level filtering of Arrow record batches based on boolean expressions. This is a fundamental component for query execution. ## Changes - Implement `NewFilterPipeline` for evaluating filter predicates on record batches - Add `filterBatch` utility to efficiently filter Arrow records - Update `executeFilter` in the executor to use the new implementation - Add comprehensive test suite for the filter pipeline: - Basic literal predicates (true/false) - Column reference predicates - Empty batch handling - Multiple batch processing ## Implementation Notes The filter implementation follows the same pattern as the project pipeline, evaluating predicates against each record and creating a new filtered record based on the results. The implementation supports compound predicates with AND logic. This implementation provides a foundation for more advanced filtering operations as the expression evaluator is enhanced to support more complex expressions.	9 months ago
Owen Diehl	395030b130	feat(dataobj executor): project node (#17312 )	9 months ago
Owen Diehl	920e3b23ed	chore(dataobj,testing,executor): pipeline equality testing (#17311 )	9 months ago
Christian Haudum	e828c48702	chore(engine): Implement execution pipeline for the limit operator (#17264 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Owen Diehl	cff0df63b3	chore(dataobj,executor): CSV arrow batches & pipeline helpers (#17290 )	9 months ago
Christian Haudum	64c477b1f7	chore(engine): Add framework for query executor (#17260 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257 ) The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032 ) This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	ba61aa88de	chore(engine): Integrate metastore into physical query planning (#17016 ) This PR integrates the data object lookup using the metastore into the query planning. To achieve that, I extended the `Metastore` interface with an additional function that not only returns the data object paths, but also the stream IDs of streams that match the given label matchers. ```go type Metastore interface { ... // StreamsIDs returns object store paths and stream IDs for all matching objects for the given matchers between [start,end] StreamIDs(ctx context.Context, start, end time.Time, matchers ...*labels.Matcher) ([]string, [][]int64, error) ... } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	6824f4e575	chore(engine): Add printer for Mermaid diagram format (#17000 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	04d2ba3ca8	chore(engine): Add simple plan optimizer (#16965 ) This PR adds a plan optimizer that pushes down limit and filter predicates to the scan node. ### Example Plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── Filter #0xc00039dd10 │ └── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Filter #0xc00039ddd0 │ └── Predicate expr=GT(age.metadata, 21) └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=0 └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=0 ``` Optimized plan: ``` Limit #0xc0000110f8 offset=0 limit=1000 └── SortMerge #0xc00039de60 column=timestamp.builtin order=ASC ├── DataObjScan #0xc00011b420 location=obj1 stream_ids=(1, 2) projections=() direction=0 limit=1000 │ ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) │ └── Predicate expr=GT(age.metadata, 21) └── DataObjScan #0xc00011b490 location=obj2 stream_ids=(3, 4) projections=() direction=0 limit=1000 ├── Predicate expr=LT(timestamp.builtin, 1742826126000000000) └── Predicate expr=GT(age.metadata, 21) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	40223d7320	chore(engine): Unify `Literal` and `ColumnRef` across logical and physical plan (#16927 ) Literals and column refs are used by both the logical and physical plan. However, the interfaces they need to implement are partly private (`isValue()` and `isExpression()`). Therefore they are wrapped in their own struct that implement these interfaces. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	138093b900	chore(planner): Convert AST into logical plan (#16916 ) Minimal implementation of the logical planner that converts a parsed LogQL statement in AST form into a logical plan in SSA form. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	0a3230f457	chore(ast): Update WalkFn signature to return bool (#16989 ) The boolean return value indicates whether walking/traversing should be continued with child nodes. This PR does not change any existing traversal behaviour. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	efe15486fe	chore(engine): Add first stage of physical query planning (#16891 ) Planning consists of two stages: 1. Conversion stage: Instructions from the logical plan are converted into Nodes in the physical plan. Most instructions can be translated into a single Node. However, `MakeTable` translates into multiple `DataObjScan` nodes. 2. Pushdown stage: a) Push down the limit of the `Limit` node to the `DataObjScan` nodes. b) Push down filter predicates from the `Filter` nodes to the `DataObjScan` nodes. This PR implements only the first stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Christian Haudum	f79906ca75	chore(engine): Add utility function `canExecuteWithNewEngine` (#16892 ) The function determines whether a query can be executed by the new execution engine. In the first step, the new execution engine will only be able to execute limited/filter queries without any transformation stages. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Robert Fratto	cbe97e6c1a	chore(engine): Update logical planner types to be structured as SSA (#16841 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Ashwanth	c51fa39684	chore(dataobj): pass down line filters to logs reader (#16802 )	10 months ago
Christian Haudum	9691f3e5dc	chore: Move `pkg/dataobj/planner` to `pkg/engine/planner` (#16815 ) The package pkg/dataobj/planner/logical was initially created within the package pkg/dataobj because of the dependency on the internal package pkg/dataobj/internal/metadatamd. However, the package should be located inside a separate package without the dependency on the dataobj package, since the engine should be storage agnostic. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago

33 Commits (60c76318e66d33a726b41d8542e71663e581c92a)