34 Commits (bf06e45bcfc7745ee26798fdfbbbe31ecfb3bd3f)

Author	SHA1	Message	Date
Trevor Whitney	88cdd1badd	refactor(dataobj-compactor): Split CompactionMerge into IndexMerge + LogMerge (#21930)	3 weeks ago
Trevor Whitney	12c419b65d	feat(dataobj-compactor): Add CompactionMerge physical-plan node (#21844)	4 weeks ago
Salva Corts	5f06abe88d	perf: short-circuit cache when over max cacheable size (#21277)	2 months ago
Salva Corts	9ed2dd0e7d	feat: Add caching node to dataobjscan node (#21246)	2 months ago
Salva Corts	bd51fcae09	feat(WIP): Task level caching (#21160)	3 months ago
benclive	968dabadd5	chore: Increase batch sizes for pointers reads (#20983)	3 months ago
Salva Corts	a4de7ad732	refactor: Move batching to pipeline wrapper (#21123)	3 months ago
Robert Fratto	b8920125c5	chore(dataobj): add configurable prefetching (#20946) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	00c8211fbf	chore(engine): add concurrency to Pipeline.Open (#20839)	4 months ago
Ashwanth	73961f18a3	chore(xcap): leaner xcap API (#20771)	4 months ago
Robert Fratto	11f885af7a	chore(engine): fix streamsView.Open semantics (#20802)	4 months ago
Robert Fratto	2d63582b15	chore: finish adding Open semantics in engine, metastore (#20798)	4 months ago
Ashwanth	8b7161a9fc	chore(engine): enforce max_query_series (#20557)	5 months ago
Stas Spiridonov	bedfb78378	chore: Thor query engine memory improvements, part 2 (#20473)	5 months ago
Salva Corts	76a21de886	feat: Add Header Propagation and Stream Filtering to V2 Query Engine (#20449)	5 months ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	6 months ago
Stas Spiridonov	b6b7459435	chore: Aggregation groupings for by() and without() (#19928)	6 months ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	6 months ago
Ashwanth	5bb6b54052	chore: compat should also handles metadata collisions (#20005)	7 months ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821)	7 months ago
Stas Spiridonov	f2c874e6ab	chore: scheduler tasks are aware of their time range (#19709)	7 months ago
Trevor Whitney	8b39d4d077	refactor: implment parse as a projection (#19579) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	3c99535b0a	chore(engine): provide worker (#19588) ... The new worker package connects to an instance of a scheduler (#19570) for task assignment and execution. A worker spawns a fixed number of threads, each of which execute one task at a time. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	8 months ago
Trevor Whitney	13429aa1ea	refactor: move allocator into execution Context (#19550)	8 months ago
Trevor Whitney	3ce6fa2946	feat: implement `unwrap` as a projection (#19409)	8 months ago
Christian Haudum	290d1b711f	chore(engine): Add support for drop stage (drop projection) (#19533) ... This PR add support for dropping columns from the schema. The drop projection only supports dropping column by name, not by matcher.	8 months ago
Robert Fratto	bd3f3dabe1	chore(engine): introduce ScanSet node (#19524) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Ashwanth	baf10907bd	chore: introduces TopK node and replace the usage of SortMerge (#19520)	8 months ago
Robert Fratto	3a0480698c	chore(engine): add Parallelize hint node (#19521)	8 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470) ... ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Stas Spiridonov	16dab82593	chore: consistent expressionEvaluator.eval result memory ownership (#19438)	8 months ago
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396) ... In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Stas Spiridonov	1474ede675	feat(engine): More aggregations for the new query engine (#19350)	8 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931)	9 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	9 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039) ... This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	9 months ago
Ashwanth	827b1f2df4	chore(engine): add support for range metric queries (#18892)	10 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694)	10 months ago
Robert Fratto	385d397404	chore(engine/executor): reduce cost of executing DataObjScan plan node (#18679)	10 months ago
Ashwanth	7c5bce70dc	chore(engine): introduce merge node (#18641)	10 months ago
Robert Fratto	a3380f6480	chore(engine): add tracing instrumentation (#18648)	10 months ago
Robert Fratto	56e7318bec	chore(engine): don't sort metric queries (#18594)	11 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	11 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499)	11 months ago
Christian Haudum	cea56469b9	chore(engine): Pass context to Pipeline's Read() function (#18459) ... This change unifies how the request context is passed through the execution of pipelines. Instead of storing the context on pipelines, pass them as argument to the `Read()` function. This change alters the interface method of `Pipeline` from `Read() bool` to `Read(context.Context) bool`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	d9f7d785ff	chore(engine): Sort entries in dataobj sections by timestamp descending (#18367) ... Sorting sections by `timestamp DESC` first and `streamID ASC` second allows for more efficient reading (by section) for BACKWARD log queries. - This change breaks FORWARD log queries. Which is ok for the time being. We will add support later again when columnar reading of the dataobj is introduced. - This change renders existing dataobj in storage incompatible. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Robert Fratto	a0914281b8	chore(engine): change DataObjScan to only cover a single logs section (#18350)	11 months ago