124 Commits (main)

Author	SHA1	Message	Date
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	2 weeks ago
Ashwanth	2b97b207e0	chore: stream sink handle retryable errors (#20264)	2 weeks ago
Ashwanth	be9f5fd729	chore(scheduler): reduce lock contention during task assignment (#20254)	2 weeks ago
Stas Spiridonov	d776c10dff	fix: Scheduler memory leak (#20268)	2 weeks ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	3 weeks ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	3 weeks ago
Stas Spiridonov	b6b7459435	chore: Aggregation groupings for by() and without() (#19928)	3 weeks ago
Stas Spiridonov	377af1aa8b	chore: graceful worker shutdown (#20136)	3 weeks ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189) ... This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	3 weeks ago
Robert Fratto	eaeb7afe88	chore(engine)!: Share worker threads across all scheduler connections (#20229) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	0c9dd8b7b7	chore(engine): update how scheduler tracks readiness (#20225) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Robert Fratto	1272bbb384	chore(engine): send structured errors over the wire (#20211) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
benclive	f22e854dea	chore: Dispatch tasks more frequently (#20195)	4 weeks ago
Robert Fratto	2d54ddc6d9	chore(engine): adjustments to log task short circuiting (#20121)	1 month ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	1 month ago
Ashwanth	356cd36d8e	chore: close stream section reader (#20107)	1 month ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095)	1 month ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	1 month ago
benclive	99ba51e48a	fix(enginev2): Close pipeline before building query results (#20100)	1 month ago
Ashwanth	484299df05	chore(xcap): Adds wrapper around bucket client to track operations (#20094)	1 month ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037) ... Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	1 month ago
Ashwanth	6a6681d9b7	chore(scheduler): extract trace info from orig query context (#20041)	1 month ago
Stas Spiridonov	5b88bcef3a	chore: Short circuiting for new scheduler (#20000)	1 month ago
Ashwanth	5bb6b54052	chore: compat should also handles metadata collisions (#20005)	1 month ago
Robert Fratto	9d0d1084e3	chore(engine): use github.com/grafana/dskit/dns for scheduler discovery (#19835) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ashwanth	25c8827f56	chore(xcap): export captures as traces (#19880)	2 months ago
Robert Fratto	86ccc65355	chore(engine): report saturation as load when there are no workers (#19925) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Robert Fratto	e9030beedf	chore(engine): track scheduler load/pressure metrics (#19924)	2 months ago
Robert Fratto	2fce7da611	chore(engine): add base set of new scheduler metrics (#19913) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821)	2 months ago
Sophie Waldman	b9a51f0129	feat: Enable support for max, min, max_over_time, min_over_time for new engine (#19841)	2 months ago
Christian Haudum	97fc8b3a50	chore(metastore): Simplify public API of Metastore (#19886) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Robert Fratto	e9600aed4b	chore(engine)!: Make scheduler aware of total compute capacity (#19876)	2 months ago
Robert Fratto	03f917a357	refactor(engine): update workflow to register all tasks/streams at once (#19859) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Christian Haudum	f066d378a5	chore(engine): Do not embed `types.Literal` interface in logical and physical Literal (#19858) ... This ensures that logical.Literal and physical.LiteralExpr do not implicitly implement the types.Literal interface, which caused a bug fixed with #19853 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	0f35890d4b	chore(engine): Fix conversion from logical to physical Literal in math expressions (#19853) ... Since both logical.Literal and physical.LiteralExpr embed the types.Literal interface, they both automatically also implement the interface. This is problematic when creating a physical LiteralExpr from a logical Literal, rather than from the underlying value. The serialization of the physical plan fails, because the pointer value of the interface is then a logical.Literal, rather than a physical.LiteralExpr	2 months ago
Robert Fratto	0f0846adec	chore(engine): make stream/task debug logging configurable (#19842) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ashwanth	53230deee4	chore(scheduler): send tracing context to workers (#19839) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ivan Kalita	9b001dfd01	fix(engine): unset write and read deadlines for wire listeners (#19828) ... Wire http2 peers continuously drop connections due to http server write timeouts. I'm unsetting them.	2 months ago
Robert Fratto	abaeff7fad	chore(engine): query new schedulers over HTTP from query-frontend (#19744) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Ivan Kalita <ivan.kalita@grafana.com>	2 months ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Stas Spiridonov	f2c874e6ab	chore: scheduler tasks are aware of their time range (#19709)	2 months ago
Robert Fratto	785743214e	chore(engine): propagate worker listen address to scheduler (#19730) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Christian Haudum	c2cf38383c	chore: Wire up new scheduler and worker components (#19728) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
benclive	36613bd017	fix: Empty keys are returned if desired in v2 engine (#19717)	2 months ago
Ivan Kalita	f3798af153	chore(engine): Add HTTP/2 transport for wire protocol (#19575) ... Implements HTTP/2 transport layer with protobufCodec, HTTP2Listener/Conn. - Listener receives an http request and holds it. The underlying connection is now managed by the listener and will be returned to the next .Accept() caller (wrapped into an http2Conn). - http2Conn implements wire.Conn interface and supports reads and writes of frames. It uses protobufCodec to serialize, write, read, deserialize the data. - there's proto definition for frames and messages that is used by protobufCodec	2 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Stas Spiridonov	d02e55c57a	chore: remove limit physical plan node (#19682)	2 months ago
benclive	01cab53447	feat(engine): implement strict and keepEmpty logfmt parsing (#19668)	2 months ago
Ashwanth	b3803113ad	chore(engine): registers functions for logical op (#19704)	2 months ago