244 Commits (TR-query4)

Author	SHA1	Message	Date
Julie Stickler	fbe69fd3ae	rebasing	1 week ago
Julie Stickler	296536f587	docs: Troubleshooting queries Part 4	1 week ago
J Stickler	e1edee38fb	docs: Troubleshooting queries Part 3 (#20241)	1 week ago
J Stickler	9085453c6e	docs: Troubleshooting queries Part 2 (#20240)	3 weeks ago
Daniel-Davies	d940433736	[feat] Add config option to override Azure AD endpoint (#20267) ... Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	3 weeks ago
J Stickler	bc3c0ca190	docs: Troubleshooting query docs - Part 1 (#20182)	3 weeks ago
Assel Meher	de092daf0a	feat: Handle state change lock in prepare downscale (#20141)	4 weeks ago
renovate-sh-app[bot]	87f3b590d5	fix(deps): update module github.com/aws/aws-sdk-go-v2/service/s3 to v1.93.0 (main) (#20142) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	4 weeks ago
Ashwanth	9bd5d6a7d0	chore: revert dropping spans generated by gcs storage clients (#20133)	1 month ago
Christian Haudum	326c7d1ea0	feat: Client side index gateway shuffle sharding (#20124) ... This PR adds the functionality to add a client-side shuffle sharding for index gateway requests. The per-tenant configuration index_gateway_max_capacity defines a fraction (0.0 to 1.0) of index gateways that become available for each tenant. The client uses the jumphash of the tenant ID to achieve greatest possible consitant hashing without using a server-side, statefule hash ring. This will allow to distribute workloads from different tenants across all index gateway instances and prevent individual tenants from overloading all index gateways. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Francisco Robles Martín	129ebd696d	docs: Fix max_line_size default behavior comment (#20067) ... Signed-off-by: Francisco Robles Martín <f.robles.martin@pm.me> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 month ago
Christian Haudum	8b59b944f2	chore(dataobj): Use common base config for indexobj and logsobj builders (#20037) ... Use the same config struct for both the indexobj builder and the logsobj builder, since they are configured identically, only with different default values.	1 month ago
Christian Haudum	85573045a3	chore(engine): Clean up query engine configuration (#20049) ... chore(engine): More coherent naming of config options chore(engine): Remove duplicate engine configuation The configuration options `dataobj_storage_lag` and `dataobj_storage_start` have been registered twice, once under `query_engine.executor` and a second time under `querier`. chore(engine): Move configuration options Move configuration options for the next generation query engine that were previously located under `query_range` into `query_engine`. chore(engine): Re-organize configuration for next generatio query engine Organize the configuration settings in the config.yaml under `query_engine`, rather than under `querier.engine_v2`. Respectively change the CLI argument prefix from `-querier.engine-v2.*` to `-query-engine.*`.	1 month ago
Ashwanth	c797f10abe	chore: allow dropping spans generated by gcs storage clients (#20040)	1 month ago
Trevor Whitney	b8b9648ae4	docs: document the json pattern tokenizable field limits (#19964)	2 months ago
George Robinson	3a24f5d291	feat: write to dataobj partitions based on segmentation key (#19946)	2 months ago
renovate-sh-app[bot]	aafc5792e2	fix(deps): update module github.com/prometheus/common to v0.67.3 (main) (#19906) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Robert Fratto	0f0846adec	chore(engine): make stream/task debug logging configurable (#19842) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Sandeep Sukhani	856c11dad2	feat: Add support for storing chunk deletion markers in object storage instead of local disk (#19689)	2 months ago
Christian Haudum	1c7d581f4e	chore: Fix storage CLI arguments for Goldfish UI (#19823)	2 months ago
Robert Fratto	abaeff7fad	chore(engine): query new schedulers over HTTP from query-frontend (#19744) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Ivan Kalita <ivan.kalita@grafana.com>	2 months ago
renovate[bot]	0b0faf174b	fix(deps): update module github.com/aws/aws-sdk-go-v2/service/s3 to v1.88.4 (main) (#19341) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
Robert Fratto	377e067725	chore(engine): determine proper advertise address for scheduler (#19731) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Christian Haudum	c2cf38383c	chore: Wire up new scheduler and worker components (#19728) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Trevor Whitney	b2396e1fb8	fix: expose RDS args for goldfish ui (#19724)	2 months ago
Christian Haudum	4b118fb729	chore: Admission lanes for workflow tasks (#19584) ... This PR introduces "admission lanes" for dispatching tasks to the runner (scheduler), that are controlled by token buckets, to avoid overflowing the runner with a large amount of tasks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Robert Fratto	7473aee3a5	chore(engine): add periodic discovery of remote schedulers (#19695) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Salva Corts	a87863ca68	docs: Add docs for config self-serve API (#19656)	2 months ago
Trevor Whitney	e17ae2d98f	feat(goldfish): add endpoints for serving stored results (#19640)	2 months ago
Robert Fratto	75e5193a97	chore(engine): expose new scheduler and worker (#19633) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
George Robinson	d146c9f4ad	feat: add enabled flag for dataobjs (#19613)	2 months ago
George Robinson	977fb2ba6f	feat: consume single topics in dataobj-consumer using partition ring (#19186)	2 months ago
Trevor Whitney	26f8b992ee	fix: correctly register allowed log level fields (#19261)	3 months ago
Paul Rogers	cce4511c74	feat(ingester): Add WAL throttling for partition ingesters (#19538)	3 months ago
Ashwanth	3eb48a4c39	chore: add new config to enable v2 engine router (#19530)	3 months ago
Ashwanth	72cb649623	chore: introduce mw to route requests between v1 and v2 engines (#19452)	3 months ago
Sophie Waldman	f18c62e2d6	chore: Add configurable time range for data object availability. (#19478) ... In ops, data objects are not available until 24h behind the current timestamp, while we hard code the time range at one hour. Make the availability configurable instead. Fixes grafana/loki-private#2001	3 months ago
Trevor Whitney	52b5d95ed1	feat: drilldown config endpoint (#19028)	3 months ago
Trevor Whitney	71e2e01ee2	feat: implement volume-based pattern filtering (#18838)	3 months ago
Ashwanth	c24675f854	chore: adds separate v2 engine config section (#19423)	3 months ago
Trevor Whitney	3a1e5ad5f1	feat!: move the Loki UI to a Grafana Plugin (#19390) ... there is still a ui target, which serves the APIs needed for the plugin, which must be manually assigned to a pod	3 months ago
George Robinson	64bbfe00c2	chore: remove block builder code (#19234)	4 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	4 months ago
Christian Haudum	0c49d26ea0	chore: Assign index gateway requests to server buckets (#19177) ... The idea of this PR is to apply time-based client side sharding of index gateway requests. As of now, when index gateways are run in simple mode, the addresses from the DNS resolver are randomized for each request to distribute request load across index gateway instances evenly. However, we've seen more and more that individual long range queries (e.g. >= 30d) could overload and slow down index gateway by "saturating" mmap reads due to high amount of major page faults (page cache misses). When this is happening, any query hitting the index gateway is affected. Distributing queries based on their end time into different buckets of index gateway servers helps to separate executing recent queries from executing "old" queries. Usually, there is a higher amount of recent queries than there is of old queries. Therefore the subset of index gateways assigned to "recent query" buckets is greater than the subset of index gateways assigned to "old query" buckets. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	4 months ago
George Robinson	a03e86c858	feat: remove tenant topic tee (#19082)	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039) ... This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	4 months ago