145 Commits (v3.6.2)

Author	SHA1	Message	Date
Trevor Whitney	3ce6fa2946	feat: implement `unwrap` as a projection (#19409)	3 months ago
Robert Fratto	1750b34448	chore(engine): fix main build (#19536)	3 months ago
Christian Haudum	290d1b711f	chore(engine): Add support for drop stage (drop projection) (#19533) ... This PR add support for dropping columns from the schema. The drop projection only supports dropping column by name, not by matcher.	3 months ago
Robert Fratto	f7d4614a54	chore(engine): address extra feedback from #19522 review (#19535)	3 months ago
Robert Fratto	3add93a5b8	chore(engine): add parallel pushdown optimization pass (#19522)	3 months ago
Ivan Kalita	263980061a	chore(engine): remove unused logical.Value.Schema() (#19531)	3 months ago
Robert Fratto	bd3f3dabe1	chore(engine): introduce ScanSet node (#19524) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	c1b010567c	chore(engine): print stringified planning durations (#19155) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Ashwanth	baf10907bd	chore: introduces TopK node and replace the usage of SortMerge (#19520)	3 months ago
Robert Fratto	3a0480698c	chore(engine): add Parallelize hint node (#19521)	3 months ago
Robert Fratto	cfd07c3683	chore(engine): simplify Pipeline interface (#19518)	3 months ago
Ashwanth	fa8bd848f6	chore: update buildTable to sort rows using `BuilderOptions.SortOrder` (#19515)	3 months ago
Ashwanth	72cb649623	chore: introduce mw to route requests between v1 and v2 engines (#19452)	3 months ago
Christian Haudum	849a9214fa	chore: Release column vectors and their arrays correctly (#19496) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	8b7ef09e69	chore(engine): Cleanup: remove state from pipelines (#19495) ... * Do not pass `state` in `genericPipeline` but instead return `(arrow.Record, error)` in `readFunc` * Remove unused `state` fields from pipelines.	3 months ago
Christian Haudum	3c5f759f24	chore(engine): Add compatibility node for parsed columns (#19485) ... The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the second case. This PR is a follow up on #19470 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ivan Kalita <input.txt@hotmail.com>	3 months ago
Christian Haudum	130a29efe9	chore: Revert "chore: remove prefetching for sortmerge" (#19491)	3 months ago
Christian Haudum	662e1b93f1	chore(engine): Add `LOGQL_COMPAT` marker to logical plan (#19488) ... The marker operation `LOGQL_COMPAT` indicates that the query is expected to return results that are compatible with the old v1 chunks engine. It is used in the physical planner to add required `ColumnCompat` nodes to the plan that handle columns with colliding names but of different semantic types (label, metadata, parsed).	3 months ago
Ashwanth	26c07fa6bc	chore(planner): fix limits pushdown to work with empty predicates (#19489)	3 months ago
Ashwanth	0f28e4be9e	chore: remove prefetching for sortmerge (#19486)	3 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470) ... ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Ashwanth	6a829e3d6f	chore(engine): allow usage of empty groupby in aggregations (#19344)	3 months ago
Christian Haudum	c280531e05	chore: Fix column name regression (#19468) ... The temporary streamID column from the dataobj scan node caused an error when the expression evaluator tried to parse the FQN, which was not valid any more after the recent change that implemented the semantic naming conventions. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	d5f15705c2	chore: Fix vector results builder (#19467) ... The recent change that implemented the new semantic column naming in PR #19415 introduced a regression in the vector results builder that caused rows that had a NULL value in any column (also the group by columns) to be discarded. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	2214a700b3	chore(engine): Implement new semantic column naming (#19415) ... This PR implements the column naming convention introduced in https://github.com/grafana/loki/pull/19396 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Stas Spiridonov	16dab82593	chore: consistent expressionEvaluator.eval result memory ownership (#19438)	3 months ago
Robert Fratto	17b1418a92	chore(engine): move DAG logic to common package (#19406) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Ashwanth	c24675f854	chore: adds separate v2 engine config section (#19423)	3 months ago
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396) ... In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Stas Spiridonov	1474ede675	feat(engine): More aggregations for the new query engine (#19350)	3 months ago
Trevor Whitney	aec34ed30a	feat: add json parsing stage to v2 engine (#19240)	3 months ago
Christian Haudum	ed757b50c2	chore(engine): Do not print ID of physical plan nodes (#19254) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241)	3 months ago
Robert Fratto	20e04a4192	chore(engine/logical): push SORT operation towards bottom of plan (#19239)	4 months ago
Christian Haudum	8499840af6	chore(engine): Eliminate no-op Merge/SortMerge nodes during plan optimization (#19238) ... Merge and SortMerge nodes with 1 or less children are effectively no-op nodes and can safely be removed from the plan during the optimization phase. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	35816aa16f	chore(engine): Eliminate unnecessary SortMerge node from physical plan (#19236) ... This PR avoid creating the SortMerge node from the logical.Sort. This node was a no-op and only passed down the sort direction to its children. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	3700dc5395	chore(engine/logical): fix order/duplicates of SSA form (#19237) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Robert Fratto	24e968cfcb	perf(engine): reuse existing array in genericFunction.Evaluate (#19235) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
benclive	4f628c0d8b	chore: Fix time range grouping logic in some cases (#19226)	4 months ago
Trevor Whitney	fc58d21eb3	chore: exclude types package from linting rules. (#19220)	4 months ago
benclive	c23bb7ce4e	chore: Fix Dataobjscan grouping wrt sort order (#19215)	4 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931)	4 months ago
Ashwanth	a5e9964d7d	chore(engine): Retain Null information when filtering rows (#19214)	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	4 months ago
benclive	ad46063601	chore: Empty builder returns non-nil result (#19184)	4 months ago
benclive	b7f14f9554	fix: Stop applying limits pushdown under a filter (#19154)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago