6 Commits (080a9300a1daccac562ae05c0d3be45e09a6ca91)

Author	SHA1	Message	Date
Ashwanth	8b7161a9fc	chore(engine): enforce max_query_series (#20557)	3 months ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	4 months ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	4 months ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	4 months ago
Robert Fratto	75e5193a97	chore(engine): expose new scheduler and worker (#19633) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
benclive	b3a2e867cd	chore: Add debug tooling to compare Loki query output (#19587)	6 months ago