4965 Commits (main)

Author	SHA1	Message	Date
Noah Krause	993b3ae65d	feat(canary): Support passing arbitrary set of labels to use for the query (#17008) ... Signed-off-by: Noah Krause <krausenoah@gmail.com> Co-authored-by: Ed Welch <ed@edjusted.com>	6 days ago
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134) ... Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	2 weeks ago
Daniel-Davies	d940433736	[feat] Add config option to override Azure AD endpoint (#20267) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 weeks ago
Assel Meher	6d74050e63	chore: update dskit to include LRU cache for partition ring shuffle sharding (#20270)	2 weeks ago
Ashwanth	2b97b207e0	chore: stream sink handle retryable errors (#20264)	2 weeks ago
Ashwanth	be9f5fd729	chore(scheduler): reduce lock contention during task assignment (#20254)	2 weeks ago
Stas Spiridonov	d776c10dff	fix: Scheduler memory leak (#20268)	3 weeks ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253) ... This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	3 weeks ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234) ... Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	3 weeks ago
Stas Spiridonov	b6b7459435	chore: Aggregation groupings for by() and without() (#19928)	3 weeks ago
Stas Spiridonov	377af1aa8b	chore: graceful worker shutdown (#20136)	3 weeks ago
zethis	0c1561d15e	feat(lokitool): Add regex namespace filtering (#20209)	3 weeks ago
Jake Leahy	c524203a6e	feat(logcli): Allow custom headers to be passed (#20231)	3 weeks ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189) ... This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	3 weeks ago
Ashwanth	7da7f000ff	chore: use projection for indexobj streams reader (#20205)	3 weeks ago
Robert Fratto	eaeb7afe88	chore(engine)!: Share worker threads across all scheduler connections (#20229) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
Trevor Whitney	89396c190f	chore: use tenant instead of service_name for pattern_ingester_metric_samples (#20033)	3 weeks ago
Ed Welch	657d9ac55f	chore: add stream count to metrics.go for queriers (#20215)	3 weeks ago
Ashwanth	e08721487c	chore: compute stats summary in query engine scheduler (#20222)	3 weeks ago
Robert Fratto	0c9dd8b7b7	chore(engine): update how scheduler tracks readiness (#20225) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 weeks ago
benclive	ea88458c03	fix(enginev2): Always compute summary when creating the stats object (#20224)	3 weeks ago
Assel Meher	de092daf0a	feat: Handle state change lock in prepare downscale (#20141)	3 weeks ago
Ashwanth	beb1728870	chore: stop adding plans to span events (#20219)	3 weeks ago
Robert Fratto	c57dc2cac8	chore(querier): fix engine router splitting logic (#20217) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 weeks ago
Robert Fratto	1272bbb384	chore(engine): send structured errors over the wire (#20211) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 weeks ago
Robert Fratto	9dc824bf7f	chore(engine): specify invalid time range (#20207) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 weeks ago
George Robinson	a03e00a3bc	chore: better comments when end offset is zero (#20196)	4 weeks ago
George Robinson	333da739c7	feat: add processed records metric (#20191)	4 weeks ago
benclive	f22e854dea	chore: Dispatch tasks more frequently (#20195)	4 weeks ago
Ashwanth	833b4e1690	chore: revert change to shrink v2 split (#20190)	4 weeks ago
Trevor Whitney	43a3f159b3	fix: persist correct goldfish result comparison in database (#20172)	4 weeks ago
George Robinson	c0c27b3596	feat: add downscalePermittedFunc to check downscale is permitted (#20171)	4 weeks ago
nicolevanderhoeven	7f47dddb3a	ci: Remove Helm 4 verification and fix flaky ruler test (#20178) ... Signed-off-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: JStickler <4106682+JStickler@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	4 weeks ago
Ashwanth	c6d6bfc0b6	chore: xcap summary treat duration observation values as seconds (#20175)	4 weeks ago
Julien Pinsonneau	0c3dd8ce7e	fix: compactor file descriptor leak (#20077)	4 weeks ago
Ashwanth	5bbb50562a	chore: improvements to v2 engine split range (#20174)	4 weeks ago
George Robinson	23ef8eccce	feat: add gauge to track in-flight bytes (#20091)	4 weeks ago
Trevor Whitney	aab9e46bcf	feat: implement query splitting in the query-tee (#20039)	1 month ago
renovate-sh-app[bot]	0aac50bffb	fix(deps): update module github.com/prometheus/prometheus to v0.308.0 (main) (#20131) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 month ago
Trevor Whitney	8156f809e3	chore: add shared vscode launch config (#20103)	1 month ago
Ashwanth	9bd5d6a7d0	chore: revert dropping spans generated by gcs storage clients (#20133)	1 month ago
Ashwanth	635934869d	chore: include execution stats in `finished executing` log line (#20132)	1 month ago
Ashwanth	16df5f2f66	chore: set Dataobj logql stats from xcap (#20126)	1 month ago
Christian Haudum	0046bfb3a4	fix: Do not override S3 region if already specified in configuration chain (#20127) ... The region (and other settings) for the S3 configuration can be specified in various ways. When the region was already set by LoadDefaultConfig it should not be set to InvalidAWSRegion. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Christian Haudum	326c7d1ea0	feat: Client side index gateway shuffle sharding (#20124) ... This PR adds the functionality to add a client-side shuffle sharding for index gateway requests. The per-tenant configuration index_gateway_max_capacity defines a fraction (0.0 to 1.0) of index gateways that become available for each tenant. The client uses the jumphash of the tenant ID to achieve greatest possible consitant hashing without using a server-side, statefule hash ring. This will allow to distribute workloads from different tenants across all index gateway instances and prevent individual tenants from overloading all index gateways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Ed Welch	0ecb4fb745	chore: improve metrics.go stats logging to include bloom filter time as well as fix missing size info in query-frontend (#20093)	1 month ago
Robert Fratto	2d54ddc6d9	chore(engine): adjustments to log task short circuiting (#20121)	1 month ago
Christian Haudum	d3f9532b06	fix: Fix regression in S3 client configuration (#20110) ... ### Summary This PR fixes a regression in how Loki's S3 configuration options are handled to create the S3 client which was introduced with the upgrade of the AWS SDK from v1 to v2 (#19205). With the v1 SDK the `s3` field of the `aws` configuration block was parsed to extract credentials, region, host, and schema. ``` storage_config: aws: s3: s3://<accesskey>:<secret>@<endpoint_or_region>/<bucketnames> ``` After the upgrade, only credentials were extraced, leaving all other parts of the URL as default, which would require explicit overrides. This was a breaking change and not documented in the upgrade notes. We want to keep backwards compatibility, even though the Thanos object client is going to supersed the custom S3/GCS/... clients. ref: #19908 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Ashwanth	54f9723af1	chore(xcap): adds exporter to summarise capture as a structured log line (#20099)	1 month ago
Ashwanth	356cd36d8e	chore: close stream section reader (#20107)	1 month ago