23 Commits (main)

Author	SHA1	Message	Date
Paul Rogers	ac1e9877e5	chore(deps): Upgrade to moby from docker (#22034)	1 week ago
Salva Corts	bd51fcae09	feat(WIP): Task level caching (#21160)	2 months ago
Jason Nochlin	5102ed0592	feat(bench): add simulated object storage latency to LogQL benchmarks (#20733) ... Co-authored-by: Cursor <cursoragent@cursor.com>	4 months ago
Ashwanth	8b7161a9fc	chore(engine): enforce max_query_series (#20557)	4 months ago
benclive	934256a58b	chore: Set scheduler lookup interval in logql bench tests (#20550)	4 months ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	5 months ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	6 months ago
Christian Haudum	aec0f53536	chore: Run benchmarks locally with remote transport enabled (#19855) ... This PR adds a flag for running the TestStorageEquality test suite using remote transport over loopback interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	785743214e	chore(engine): propagate worker listen address to scheduler (#19730) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Christian Haudum	c2cf38383c	chore: Wire up new scheduler and worker components (#19728) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	4068f9a4a8	chore(logql/bench): make engine logs opt in via flag (#19661)	7 months ago
Robert Fratto	75e5193a97	chore(engine): expose new scheduler and worker (#19633) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	7 months ago
Robert Fratto	bd3f3dabe1	chore(engine): introduce ScanSet node (#19524) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Ashwanth	c24675f854	chore: adds separate v2 engine config section (#19423)	8 months ago
Christian Haudum	1e0d525c4a	chore(bench): Fix logql benchmark test setup (#19207) ... Recent pull request #19198 changed the folder structure of the benchmark dataset so it can also easily be read by a local Loki instance. However, this change broke the tests of the benchmarks itself, due to a wrong storage path, which caused the execution to not return any results. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
benclive	d4f67646b4	chore: Make EngineV2 use indexes in benchmarks (#19066)	9 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	9 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	10 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	11 months ago
Christian Haudum	d9f7d785ff	chore(engine): Sort entries in dataobj sections by timestamp descending (#18367) ... Sorting sections by `timestamp DESC` first and `streamID ASC` second allows for more efficient reading (by section) for BACKWARD log queries. - This change breaks FORWARD log queries. Which is ok for the time being. We will add support later again when columnar reading of the dataobj is introduced. - This change renders existing dataobj in storage incompatible. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ashwanth	5f3300a084	chore(bench_test): correctly integrate new query engine + misc changes (#18172)	11 months ago
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago