apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134 ) Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	5 days ago
Stas Spiridonov	b6b7459435	chore: Aggregation groupings for by() and without() (#19928 )	1 week ago
Robert Fratto	2d54ddc6d9	chore(engine): adjustments to log task short circuiting (#20121 )	3 weeks ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099 )	3 weeks ago
Ashwanth	356cd36d8e	chore: close stream section reader (#20107 )	3 weeks ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095 )	3 weeks ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096 )	3 weeks ago
benclive	99ba51e48a	fix(enginev2): Close pipeline before building query results (#20100 )	3 weeks ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037 ) Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	4 weeks ago
Stas Spiridonov	5b88bcef3a	chore: Short circuiting for new scheduler (#20000 )	1 month ago
Ashwanth	5bb6b54052	chore: compat should also handles metadata collisions (#20005 )	1 month ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821 )	1 month ago
Christian Haudum	f066d378a5	chore(engine): Do not embed `types.Literal` interface in logical and physical Literal (#19858 ) This ensures that logical.Literal and physical.LiteralExpr do not implicitly implement the types.Literal interface, which caused a bug fixed with #19853 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750 ) Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Stas Spiridonov	f2c874e6ab	chore: scheduler tasks are aware of their time range (#19709 )	2 months ago
benclive	36613bd017	fix: Empty keys are returned if desired in v2 engine (#19717 )	2 months ago
benclive	01cab53447	feat(engine): implement strict and keepEmpty logfmt parsing (#19668 )	2 months ago
Ashwanth	b3803113ad	chore(engine): registers functions for logical op (#19704 )	2 months ago
Ashwanth	5d961a4cc3	chore(topk): improvements to reduce alloc bytes and alloc space (#19660 )	2 months ago
Trevor Whitney	dd6b31473e	fix: errors in parse pipeline (#19667 ) Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	2 months ago
Trevor Whitney	8b39d4d077	refactor: implment parse as a projection (#19579 ) Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	73bc30d9a8	perf(engine): Improve regexp expression evaluation (#19644 ) This PR changes the evaluation of regex-match binary expression evaluations so that the regexp on right hand side of the expression is only compiled once for each batch evaluation, rather than for each row. This saves a ton of allocations and improves general performance of queries with a regexp line or label filter.	2 months ago
Robert Fratto	3c99535b0a	chore(engine): provide worker (#19588 ) The new worker package connects to an instance of a scheduler (#19570) for task assignment and execution. A worker spawns a fixed number of threads, each of which execute one task at a time. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
benclive	9a68900bc1	fix: Sanitize logfmt keys in v2 engine (#19621 )	2 months ago
Ashwanth	9a50c8076e	chore: stream injector: propagate orig error (#19616 )	2 months ago
Stas Spiridonov	7eda6749a4	feat(engine): basic math expressions (#19407 )	2 months ago
benclive	dd52234fc5	chore: Use first-wins semantics for duplicate fields in logfmt/json parsers (#19568 )	2 months ago
Christian Haudum	9e76f1d251	chore(engine): Refactor `ColumnVector` (#19549 ) This PR removes the need of the ColumnVector as a result of expression evaluation and instead returns an arrow.Array. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569 )	2 months ago
Trevor Whitney	13429aa1ea	refactor: move allocator into execution Context (#19550 )	2 months ago
Trevor Whitney	3ce6fa2946	feat: implement `unwrap` as a projection (#19409 )	2 months ago
Christian Haudum	290d1b711f	chore(engine): Add support for drop stage (drop projection) (#19533 ) This PR add support for dropping columns from the schema. The drop projection only supports dropping column by name, not by matcher.	2 months ago
Robert Fratto	bd3f3dabe1	chore(engine): introduce ScanSet node (#19524 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ashwanth	baf10907bd	chore: introduces TopK node and replace the usage of SortMerge (#19520 )	2 months ago
Robert Fratto	3a0480698c	chore(engine): add Parallelize hint node (#19521 )	2 months ago
Robert Fratto	cfd07c3683	chore(engine): simplify Pipeline interface (#19518 )	2 months ago
Ashwanth	fa8bd848f6	chore: update buildTable to sort rows using `BuilderOptions.SortOrder` (#19515 )	2 months ago
Christian Haudum	849a9214fa	chore: Release column vectors and their arrays correctly (#19496 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	8b7ef09e69	chore(engine): Cleanup: remove state from pipelines (#19495 ) * Do not pass `state` in `genericPipeline` but instead return `(arrow.Record, error)` in `readFunc` * Remove unused `state` fields from pipelines.	2 months ago
Christian Haudum	3c5f759f24	chore(engine): Add compatibility node for parsed columns (#19485 ) The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the second case. This PR is a follow up on #19470 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ivan Kalita <input.txt@hotmail.com>	2 months ago
Christian Haudum	130a29efe9	chore: Revert "chore: remove prefetching for sortmerge" (#19491 )	2 months ago
Ashwanth	0f28e4be9e	chore: remove prefetching for sortmerge (#19486 )	2 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470 ) ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Ashwanth	6a829e3d6f	chore(engine): allow usage of empty groupby in aggregations (#19344 )	3 months ago
Christian Haudum	c280531e05	chore: Fix column name regression (#19468 ) The temporary streamID column from the dataobj scan node caused an error when the expression evaluator tried to parse the FQN, which was not valid any more after the recent change that implemented the semantic naming conventions. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	2214a700b3	chore(engine): Implement new semantic column naming (#19415 ) This PR implements the column naming convention introduced in https://github.com/grafana/loki/pull/19396 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Stas Spiridonov	16dab82593	chore: consistent expressionEvaluator.eval result memory ownership (#19438 )	3 months ago
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396 ) In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384 ) This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago

49 Commits (main)