152 Commits (shantanu/dataobj-compactor)

Author	SHA1	Message	Date
Stas Spiridonov	e1df6a50f1	chore: Panic on marshaling errors in scheduler wire implementation. (#20588)	4 days ago
Joe Elliott	a4e75ce474	perf: avoid bytes to runes allocs by decoding in place (#20602) ... Signed-off-by: Joe Elliott <number101010@gmail.com>	5 days ago
Ashwanth	9339218b48	chore(xcap): link parents in workflow while merging captures (#20596)	5 days ago
Stas Spiridonov	6280be8b91	chore: Fix protos for remote transport (#20582)	7 days ago
Stas Spiridonov	514a142859	fix(thor): Fix error filtering before range aggregations (#20560)	1 week ago
Stas Spiridonov	efdb1f5a37	chore: Reducing waste on processing empty records (#20562)	1 week ago
Ashwanth	8b7161a9fc	chore(engine): enforce max_query_series (#20557)	2 weeks ago
Ashwanth	b21825ec23	chore: capture aggregation exec duration (#20299)	2 weeks ago
Stas Spiridonov	805b8020df	chore: Thor query engine memory improvements, part 3 (#20530)	2 weeks ago
benclive	787be333c2	chore: Tag worker profiles with fragment root node (#20521)	2 weeks ago
Stas Spiridonov	bedfb78378	chore: Thor query engine memory improvements, part 2 (#20473)	2 weeks ago
Salva Corts	76a21de886	feat: Add Header Propagation and Stream Filtering to V2 Query Engine (#20449)	2 weeks ago
Robert Fratto	6a77385878	chore(engine): add logs for task analysis (#20507) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 weeks ago
yy	0574159216	chore: typos in comments (#20481)	2 weeks ago
Ashwanth	a2791ba98a	feat(deletes): support time range filtering for listing deletes (#20427)	2 weeks ago
Ashwanth	8a6c4f8611	chore(planner): add filter stages to drop lines matching delete requests (#20395)	2 weeks ago
Ivan Kalita	7da73aaa7e	chore(dataobj): optimize bitmap decoder (#20467) ... - Use pkg/memory.Bitmap instead of []Value for pageReader presence bitmap - More efficient decoding for bitmapDecoder - only support 0/1 values (this encoding is used only for presence bitmap).	2 weeks ago
Robert Fratto	6b74130288	chore(engine): fix regex simplification when there is >1 regex (#20474)	2 weeks ago
Robert Fratto	005321ed5a	chore(engine): add hierarchical fair priority queues to v2 engine scheduler (#20405)	3 weeks ago
Stas Spiridonov	168da488db	fix: Out of bounds error fix for gapped window matcher (#20396)	3 weeks ago
Trevor Whitney	66fd9d8bde	feat: disambiguate metadata for better scans (#20245) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 weeks ago
Robert Fratto	3939e6ac7b	chore(engine): bypass regex simplfiication for stream selectors (#20387)	3 weeks ago
Assel Meher	5663f9c26d	feat(engine): add regexp parser support for log queries (#20286)	4 weeks ago
Stas Spiridonov	c2116618bd	chore: Improving memory allocations for new query engine (#20321)	4 weeks ago
Robert Fratto	c8a49d4d72	chore(engine): add support for simplifying regular expressions (#20297) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 weeks ago
Robert Fratto	f69fa66a71	chore(engine): log number of tasks in execution plan (workflow) (#20305)	4 weeks ago
Ashwanth	559c7ecd47	chore(xcap): track task queue & admission duration (#20298)	4 weeks ago
Stas Spiridonov	2203c1d8e8	chore: tweaks for feature parity (#19940)	4 weeks ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	1 month ago
Ashwanth	2b97b207e0	chore: stream sink handle retryable errors (#20264)	2 months ago
Ashwanth	be9f5fd729	chore(scheduler): reduce lock contention during task assignment (#20254)	2 months ago
Stas Spiridonov	d776c10dff	fix: Scheduler memory leak (#20268)	2 months ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	2 months ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	2 months ago
Stas Spiridonov	b6b7459435	chore: Aggregation groupings for by() and without() (#19928)	2 months ago
Stas Spiridonov	377af1aa8b	chore: graceful worker shutdown (#20136)	2 months ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189) ... This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	2 months ago
Robert Fratto	eaeb7afe88	chore(engine)!: Share worker threads across all scheduler connections (#20229) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Robert Fratto	0c9dd8b7b7	chore(engine): update how scheduler tracks readiness (#20225) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Robert Fratto	1272bbb384	chore(engine): send structured errors over the wire (#20211) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
benclive	f22e854dea	chore: Dispatch tasks more frequently (#20195)	2 months ago
Robert Fratto	2d54ddc6d9	chore(engine): adjustments to log task short circuiting (#20121)	2 months ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	2 months ago
Ashwanth	356cd36d8e	chore: close stream section reader (#20107)	2 months ago
Ashwanth	728b308edf	chore: update read stats to use xcap (#20095)	2 months ago
Ashwanth	cbad9d4ccb	chore(xcap): support events, status and improve coverage (#20096)	2 months ago
benclive	99ba51e48a	fix(enginev2): Close pipeline before building query results (#20100)	2 months ago
Ashwanth	484299df05	chore(xcap): Adds wrapper around bucket client to track operations (#20094)	2 months ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037) ... Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	2 months ago
Ashwanth	6a6681d9b7	chore(scheduler): extract trace info from orig query context (#20041)	2 months ago