4627 Commits (dfb60dcaf094dc5fc2c74be2b2e3827ed3de1db1)

Author	SHA1	Message	Date
benclive	dfb60dcaf0	fix: Metastore timerange filter (#19171)	4 months ago
benclive	3148a77966	chore: improve metastore Section calls (#19157)	4 months ago
George Robinson	3535ae4ae1	fix: use cooperative active sticky load balancer (#19160)	4 months ago
Trevor Whitney	89509dee9a	feat(goldfish): improve query performance with indexes and paritions (#19023)	4 months ago
Sandeep Sukhani	3618279603	chore: optimize chunk rewrite while filtering out deleted data (#19140)	4 months ago
Periklis Tsirakidis	a082b8a061	chore: Replace per section impl tenant function with generic one (#19139)	4 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	4 months ago
George Robinson	6fa4d0d6d4	chore: should use IsZero() instead of comparison to time.Time{} (#19145)	4 months ago
George Robinson	e321bf946c	chore: remove ConsumeRegex() from partitionring consumer (#19141)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110) ... This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
renovate[bot]	70cff6ae0d	chore(deps): update dependency vite to v7.1.4 (main) (#19116) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	4 months ago
renovate[bot]	ab86bc301a	chore(deps): update dependency @types/react-dom to v19.1.9 (main) (#19114) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	4 months ago
renovate[bot]	c4a14bf996	chore(deps): update dependency @types/node to v22.18.1 (main) (#19112) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	4 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	4 months ago
Sandeep Sukhani	0b2d0b070e	chore: make index updates idempotent from horizontally scalable compactor (#19093)	4 months ago
Salva Corts	af257fd058	feat: per policy overridable stream limits (#18994)	4 months ago
Sophie Waldman	8a9da8f0e3	chore: Update engine planner to sort by input time range (#18951) ... When convering a [logical.MakeTable] into nodes, keep information on the time range for each location and use the time ranges to sort the nodes created. The processMakeTable method will now always return a root Merge node. Inside the Merge node, overlapping shards will be grouped as DataObjScan nodes inside a SortMerge, while any individual shards without any overlap will be DataObjScan nodes directly under the Merge. Signed-off-by: Sophie Waldman <sophie.waldman@grafana.com>	4 months ago
Ashwanth	d950d4075d	chore(engine): improve performance of matching window calculation (#19013) ... Currently range aggregation linearly scans through all the possible time windows (a 24h query with a 1m step results in 1440 windows) to find the matching ones. This is quite expensive as this matching is done for each row. This PR updates the matching code to use arithmetic which runs in constant time for cases where step >= range and for overlapping windows (step < range) it uses binary search. Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
keeghcet	81229effb9	chore: fix inconsistent function name in comment (#19081) ... Signed-off-by: keeghcet <keeghcet@outlook.com>	4 months ago
Charles Korn	8509f1df90	chore(deps): upgrade dskit (#19089)	4 months ago
benclive	5d428b2ac4	chore: Fix panic for queries with unsupported predicate types (#19086)	4 months ago
benclive	653d7ac19e	chore: Fix tenant info in sections when resetting (#19085) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	4 months ago
George Robinson	a03e86c858	feat: remove tenant topic tee (#19082)	4 months ago
Shantanu Alshi	4dfef8fc15	chore(otlp): Add support for zstd and lz4 content-encodings for OTLP logs (#18842)	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	4 months ago
Karsten Jeschkies	adb27ff177	perf: Reduce allocations for JSON and logfmt parser (#18637) ... **What this PR does / why we need it**: The JSON and logfmt parsers suffer from a high number of allocations when the parsed labels have a high cardinality. This change reduces the allocations quite a bit by copying the labels bytes to a byte buffer. Note: The benchmarks ran without a validation step that calls `LabelResult.Labels()` as that method uses a lot of allocation but is used only once in production code.	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039) ... This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	4 months ago
Robert Fratto	32e3b46928	chore(rangeio): add tracing to rangeio.ReadRanges (#19067)	4 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	4 months ago
George Robinson	44aba62243	fix: fix possible data loss in dataobj-consumers (#19060)	4 months ago
George Robinson	cb3b74a45d	feat: add p50, p95 and p99 section sizes per tenant (#19054)	4 months ago
benclive	d4f67646b4	chore: Make EngineV2 use indexes in benchmarks (#19066)	4 months ago
benclive	84efa59a06	chore: Fix init of engine with metastore (#19064)	4 months ago
benclive	a7f001512a	chore: Fix index generation with filesystem (#19063)	4 months ago
mbaykara	2b01a310d8	fix: extended detected level for debug and critical level (#18370)	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	4 months ago
George Robinson	598cab79e5	fix: don't need to open section to read tenant (#19053)	4 months ago
George Robinson	d256df481e	feat: add p50, p95 and p99 sections per tenant to stats (#19044)	4 months ago
Robert Fratto	aa66baec97	chore(logqlmodel/stats): print new engine summary in metrics.go (#19052)	4 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Christian Haudum	389fc5ade6	chore(engine): Correctly free record allocations (#19040) ... Fixes panics in the v2 query engine, due to Arrow records being freed at incorrect places. Rather than freeing the previous batch created by the KWayMerge on every Read() call, fully consumed batches by the KWayMerge need to be freed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
renovate[bot]	1c70e7d75a	chore(deps): update dependency @types/react to v19.1.12 (main) (#19046) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	5 months ago
renovate[bot]	014ec14fc2	fix(deps): update module github.com/stretchr/testify to v1.11.1 (main) (#19042) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	5 months ago
George Robinson	3bab0ff5fe	fix: backoff cannot be canceled via ctx (#19041)	5 months ago
benclive	28fe97dcce	chore: fix metastore indexes being written with double prefixes (#19018)	5 months ago