204 Commits (3e415aca3d904fa6fa3cbedfdc7a2bdfd7fb6d41)

Author	SHA1	Message	Date
Trevor Whitney	3a1e5ad5f1	feat!: move the Loki UI to a Grafana Plugin (#19390) ... there is still a ui target, which serves the APIs needed for the plugin, which must be manually assigned to a pod	3 months ago
George Robinson	64bbfe00c2	chore: remove block builder code (#19234)	4 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	4 months ago
Christian Haudum	0c49d26ea0	chore: Assign index gateway requests to server buckets (#19177) ... The idea of this PR is to apply time-based client side sharding of index gateway requests. As of now, when index gateways are run in simple mode, the addresses from the DNS resolver are randomized for each request to distribute request load across index gateway instances evenly. However, we've seen more and more that individual long range queries (e.g. >= 30d) could overload and slow down index gateway by "saturating" mmap reads due to high amount of major page faults (page cache misses). When this is happening, any query hitting the index gateway is affected. Distributing queries based on their end time into different buckets of index gateway servers helps to separate executing recent queries from executing "old" queries. Usually, there is a higher amount of recent queries than there is of old queries. Therefore the subset of index gateways assigned to "recent query" buckets is greater than the subset of index gateways assigned to "old query" buckets. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	4 months ago
George Robinson	a03e86c858	feat: remove tenant topic tee (#19082)	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039) ... This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	5 months ago
J Stickler	6a91b6a0cc	docs: prep OTEL shared content (#19025)	5 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928)	5 months ago
Salva Corts	3b2498b896	feat: Default ingestion policy mappings merged with per-tenant mappings (#18926)	5 months ago
Sandeep Sukhani	e9b1d89f70	chore: do not rely on GOMAXPROCS to configure the number of sub-workers to run in compactor worker (#18848)	5 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	5 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	5 months ago
Trevor Whitney	e06fea9538	fix: register goldfish flags (#18728)	5 months ago
Trevor Whitney	4fabb8d632	feat: add traceId to Goldfish UI (#18726)	5 months ago
Jackson Coelho	19233d0461	fix(deps): update dskit dependency (#18711)	5 months ago
Trevor Whitney	2f0dc74309	feat: add traceId support to goldfish (#18681) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	5 months ago
Sandeep Sukhani	9281ab3959	docs(compactor HS): add documentation for horizontally scalable compactor (#18654)	5 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	6 months ago
Trevor Whitney	c690827c6a	fix: limit volume and frequency of persisted patterns (#18362) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Trevor Whitney	73de7b5378	feat: filter tenant limits endpoint by allowlist (#18521) ... Co-authored-by: Claude <noreply@anthropic.com>	6 months ago
Trevor Whitney	462acb52bb	feat: Goldfish UI (#17983) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	6 months ago
Christian Haudum	ce03b7943a	chore(dataobj): Improve default builder target sizes and settings help (#18543) ... Explain which settings are based on compressed and which are based on uncompressed size and why. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Sandeep Sukhani	5bf3f74ab2	feat(compactor HS): tune compactor worker config and add experimental feature warning to the docs (#18539)	6 months ago
Christian Haudum	db6c6d974b	chore: Remove `slicelabels` build tag from Makefiles (#18518) ... Since the codebase now consistently uses the new `stringlabels` labels API after merging #18490 the build tags in the Makefile can safely be removed again. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	6 months ago
Sandeep Sukhani	bd293206cd	feat(compactor HS): Wire things up and add some metrics (#18421)	6 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	6 months ago
Christian Haudum	0ee8e76ba2	feat: Mark truncated log lines with identifier (#18262) ... While truncated JSON log lines are usually easy to identify (because the JSON becomes invalid), it is not always the case with other plain text log lines. This PR adds the possibility to add an identifier as suffix for truncated log lines, such as `[...]` so that truncated log lines are easier to spot and identify when querying them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Ashwanth	8481b34dd5	fix(frontend): allow resolution of v6 addresses. (#18251)	7 months ago
Ed Welch	0990ecdf40	chore: Update dskit (#18127)	7 months ago
George Robinson	c33cd3dd4e	chore: remove unused recheck period (#17972)	7 months ago
Periklis Tsirakidis	40d59969b1	chore(kafka): Cleanup deprecated fields address and client-id (#18030)	7 months ago
Trevor Whitney	888e4012bc	chore: clean up pattern persistence configs (#18017)	7 months ago
Trevor Whitney	fdcc12da4d	feat: query persisted patterns (#17980) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	7 months ago
Trevor Whitney	fd7321c88b	feat: persist patterns as aggregated metrics (#17737) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	7 months ago
Ed Welch	ad0bef31e7	chore: improve logging of push request streams (#17872) ... Signed-off-by: Ed Welch <ed@edjusted.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	7 months ago
George Robinson	6b66122421	chore: do not count active streams in update cond to make this faster (#17933)	7 months ago
Periklis Tsirakidis	60c76318e6	fix(limits): Read the consumer group and topic from the ingest-limits config (#17831)	7 months ago
Periklis Tsirakidis	d6906649bd	fix(limits): Adapt defaults and expose evict interval (#17808)	8 months ago
Jackson Coelho	b7ae8499b8	chore: parallel kafka consumer (#17262) ... Signed-off-by: Jackson Coelho <fcjack@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	8 months ago
George Robinson	376124ed6e	chore: rename bucket_duration to bucket_size (#17759)	8 months ago
George Robinson	942a451043	chore: rename WindowSize to ActiveWindow (#17758)	8 months ago
Periklis Tsirakidis	6ef715fc2c	feat(kafka): Support multiple kafka clients in component (#17757)	8 months ago
Owen Diehl	1228e713f6	chore(tenant-topics): configurable throughput per partition (#17457)	8 months ago