87 Commits (bf06e45bcfc7745ee26798fdfbbbe31ecfb3bd3f)

Author	SHA1	Message	Date
Robert Fratto	6e9d98616b	chore(engine): Improve log events to observe query lifecycles (#21863)	4 weeks ago
Robert Fratto	122f168075	chore(dataset): Add dataset/array, dataset/buffer packages (#21586)	1 month ago
Salva Corts	904c963d17	feat: Resolve cached tasks in parent task (#21558)	1 month ago
Bryan Boreham	9cdc29fb99	fix(columnar): Don't release Arrow records (#21670) ... Multiple goroutines may still be accessing records, so we cannot Release them. Remove from the real code and from an example.	2 months ago
Bryan Boreham	dc6cb78e74	chore(queries): Log User-Agent on query metrics and finished lines (#21663) ... Add user_agent entry on query metrics log lines and on "finished executing" lines. This can help identify the source of particular queries. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 months ago
Salva Corts	24ca0e4a2f	feat: Remove task_pruning_cache_fetch_batch_size in favor of existing memcached batching (#21593)	2 months ago
Salva Corts	f3673747f6	feat: Batch cache fetches in schedulers during task pruning (#21548)	2 months ago
Salva Corts	b68de56092	feat: use query logger on metastore workflow (#21511)	2 months ago
Salva Corts	1eae14bcf5	docs: Rename task results cache config (#21503)	2 months ago
Salva Corts	fb6abe5d08	feat: Avoid executing tasks for which we have cached empty results (#21437)	2 months ago
Salva Corts	c32b57f36b	feat: return caching stats in new engine (#21369)	2 months ago
Salva Corts	5f06abe88d	perf: short-circuit cache when over max cacheable size (#21277)	2 months ago
Salva Corts	9ed2dd0e7d	feat: Add caching node to dataobjscan node (#21246)	2 months ago
Salva Corts	bd51fcae09	feat(WIP): Task level caching (#21160)	2 months ago
Salva Corts	a4de7ad732	refactor: Move batching to pipeline wrapper (#21123)	3 months ago
Assel Meher	a7c75a6007	feat(loki-query-engine): instrument query-engine with runtime/trace (#21000)	3 months ago
Robert Fratto	b8920125c5	chore(dataobj): add configurable prefetching (#20946) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Stas Spiridonov	987079d842	chore(thor): Disable admission lanes for non-log queries (#20947)	3 months ago
Ashwanth	73961f18a3	chore(xcap): leaner xcap API (#20771)	4 months ago
Robert Fratto	6a2c7c79e3	refactor(engine): introduce Pipeline.Open method (#20785)	4 months ago
Ashwanth	9339218b48	chore(xcap): link parents in workflow while merging captures (#20596)	4 months ago
Ashwanth	8b7161a9fc	chore(engine): enforce max_query_series (#20557)	5 months ago
Ivan Kalita	c05498f937	fix(xcap): properly report end (#20505) ... Fixing a typo, currently end is reported incorrectly in traces.	5 months ago
Salva Corts	76a21de886	feat: Add Header Propagation and Stream Filtering to V2 Query Engine (#20449)	5 months ago
Ashwanth	a2791ba98a	feat(deletes): support time range filtering for listing deletes (#20427)	5 months ago
Ashwanth	8a6c4f8611	chore(planner): add filter stages to drop lines matching delete requests (#20395)	5 months ago
Robert Fratto	005321ed5a	chore(engine): add hierarchical fair priority queues to v2 engine scheduler (#20405)	5 months ago
Robert Fratto	c8a49d4d72	chore(engine): add support for simplifying regular expressions (#20297) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	f69fa66a71	chore(engine): log number of tasks in execution plan (workflow) (#20305)	5 months ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	6 months ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	6 months ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	6 months ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189) ... This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	6 months ago
Ashwanth	beb1728870	chore: stop adding plans to span events (#20219)	6 months ago
Ashwanth	635934869d	chore: include execution stats in `finished executing` log line (#20132)	6 months ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	6 months ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	6 months ago
benclive	99ba51e48a	fix(enginev2): Close pipeline before building query results (#20100)	6 months ago
Ashwanth	484299df05	chore(xcap): Adds wrapper around bucket client to track operations (#20094)	6 months ago
benclive	0dea806656	fix: Respect categorizeLabels encoding flag in v2 queries (#20098)	6 months ago
Christian Haudum	85573045a3	chore(engine): Clean up query engine configuration (#20049) ... chore(engine): More coherent naming of config options chore(engine): Remove duplicate engine configuation The configuration options `dataobj_storage_lag` and `dataobj_storage_start` have been registered twice, once under `query_engine.executor` and a second time under `querier`. chore(engine): Move configuration options Move configuration options for the next generation query engine that were previously located under `query_range` into `query_engine`. chore(engine): Re-organize configuration for next generatio query engine Organize the configuration settings in the config.yaml under `query_engine`, rather than under `querier.engine_v2`. Respectively change the CLI argument prefix from `-querier.engine-v2.*` to `-query-engine.*`.	6 months ago
Robert Fratto	892d68f38f	chore(engine): add query tags into new engine logs (#20048) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Ashwanth	25c8827f56	chore(xcap): export captures as traces (#19880)	7 months ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821)	7 months ago
Robert Fratto	03f917a357	refactor(engine): update workflow to register all tasks/streams at once (#19859) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Robert Fratto	0f0846adec	chore(engine): make stream/task debug logging configurable (#19842) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Ashwanth	53230deee4	chore(scheduler): send tracing context to workers (#19839) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ashwanth	5d961a4cc3	chore(topk): improvements to reduce alloc bytes and alloc space (#19660)	7 months ago
Ed Welch	c3f12ef3df	chore: track time spent waiting for data from ingesters (#19648)	7 months ago