25 Commits (main)

Author	SHA1	Message	Date
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	2 weeks ago
Ashwanth	2b97b207e0	chore: stream sink handle retryable errors (#20264)	2 weeks ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	2 weeks ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	2 weeks ago
Stas Spiridonov	377af1aa8b	chore: graceful worker shutdown (#20136)	3 weeks ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189) ... This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	3 weeks ago
Robert Fratto	eaeb7afe88	chore(engine)!: Share worker threads across all scheduler connections (#20229) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	0c9dd8b7b7	chore(engine): update how scheduler tracks readiness (#20225) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	1272bbb384	chore(engine): send structured errors over the wire (#20211) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	2d54ddc6d9	chore(engine): adjustments to log task short circuiting (#20121)	1 month ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	1 month ago
Ashwanth	484299df05	chore(xcap): Adds wrapper around bucket client to track operations (#20094)	1 month ago
Stas Spiridonov	5b88bcef3a	chore: Short circuiting for new scheduler (#20000)	1 month ago
Robert Fratto	9d0d1084e3	chore(engine): use github.com/grafana/dskit/dns for scheduler discovery (#19835) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	1 month ago
Ashwanth	25c8827f56	chore(xcap): export captures as traces (#19880)	1 month ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821)	2 months ago
Robert Fratto	e9600aed4b	chore(engine)!: Make scheduler aware of total compute capacity (#19876)	2 months ago
Ashwanth	53230deee4	chore(scheduler): send tracing context to workers (#19839) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Robert Fratto	785743214e	chore(engine): propagate worker listen address to scheduler (#19730) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Christian Haudum	c2cf38383c	chore: Wire up new scheduler and worker components (#19728) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Robert Fratto	7473aee3a5	chore(engine): add periodic discovery of remote schedulers (#19695) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Robert Fratto	75e5193a97	chore(engine): expose new scheduler and worker (#19633) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Robert Fratto	3c99535b0a	chore(engine): provide worker (#19588) ... The new worker package connects to an instance of a scheduler (#19570) for task assignment and execution. A worker spawns a fixed number of threads, each of which execute one task at a time. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago