4707 Commits (3e415aca3d904fa6fa3cbedfdc7a2bdfd7fb6d41)

Author	SHA1	Message	Date
Christian Haudum	ed757b50c2	chore(engine): Do not print ID of physical plan nodes (#19254) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
George Robinson	7fc404c99e	chore: deleted unused code from OffsetManager (#19256)	7 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241)	7 months ago
Christian Haudum	457ec0956d	chore(dataobj-index): Handle special cases in the section lookup of the metastore (#19203) ... There are three edge cases that this PR improves: * No table (toc) paths could be resolved: return early with empty result * No index paths could be resolved: return early with empty result * No relevant sections could be resolved: return with empty result instead of with an error Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
George Robinson	64bbfe00c2	chore: remove block builder code (#19234)	7 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ed Welch	37eddabac7	chore: Revert "perf: Reduce allocations for JSON and logfmt parser (#18637)" (#19244)	8 months ago
Trevor Whitney	cb3f2ad5b6	feat: capture logs drilldown flag in query samples (#19242)	8 months ago
Robert Fratto	20e04a4192	chore(engine/logical): push SORT operation towards bottom of plan (#19239)	8 months ago
Christian Haudum	8499840af6	chore(engine): Eliminate no-op Merge/SortMerge nodes during plan optimization (#19238) ... Merge and SortMerge nodes with 1 or less children are effectively no-op nodes and can safely be removed from the plan during the optimization phase. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	35816aa16f	chore(engine): Eliminate unnecessary SortMerge node from physical plan (#19236) ... This PR avoid creating the SortMerge node from the logical.Sort. This node was a no-op and only passed down the sort direction to its children. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	3700dc5395	chore(engine/logical): fix order/duplicates of SSA form (#19237) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Robert Fratto	24e968cfcb	perf(engine): reuse existing array in genericFunction.Evaluate (#19235) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
benclive	4f628c0d8b	chore: Fix time range grouping logic in some cases (#19226)	8 months ago
Trevor Whitney	fc58d21eb3	chore: exclude types package from linting rules. (#19220)	8 months ago
benclive	c23bb7ce4e	chore: Fix Dataobjscan grouping wrt sort order (#19215)	8 months ago
Trevor Whitney	ccb4549ddd	feat: add logfmt parse support to the v2 query engine (#18931)	8 months ago
Ashwanth	a5e9964d7d	chore(engine): Retain Null information when filtering rows (#19214)	8 months ago
Sandeep Sukhani	8057c5c424	fix: apply global otlp config to tenant config only when it is updated in the overrides (#19213)	8 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	8 months ago
George Robinson	acb8b89a2f	fix: fix context canceled stops reader from cleaning up (#19210)	8 months ago
Christian Haudum	1e0d525c4a	chore(bench): Fix logql benchmark test setup (#19207) ... Recent pull request #19198 changed the folder structure of the benchmark dataset so it can also easily be read by a local Loki instance. However, this change broke the tests of the benchmarks itself, due to a wrong storage path, which caused the execution to not return any results. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	8 months ago
Christian Haudum	7dbeb45d3a	chore(bench): Restructure data directory of logql benchmarks (#19198) ... Reconfigure benchmark storage setup so that the resulting data can be read by a local Loki instance. An example config file for running Loki against the benchmark data directory is included. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	0c49d26ea0	chore: Assign index gateway requests to server buckets (#19177) ... The idea of this PR is to apply time-based client side sharding of index gateway requests. As of now, when index gateways are run in simple mode, the addresses from the DNS resolver are randomized for each request to distribute request load across index gateway instances evenly. However, we've seen more and more that individual long range queries (e.g. >= 30d) could overload and slow down index gateway by "saturating" mmap reads due to high amount of major page faults (page cache misses). When this is happening, any query hitting the index gateway is affected. Distributing queries based on their end time into different buckets of index gateway servers helps to separate executing recent queries from executing "old" queries. Usually, there is a higher amount of recent queries than there is of old queries. Therefore the subset of index gateways assigned to "recent query" buckets is greater than the subset of index gateways assigned to "old query" buckets. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
George Robinson	1ad71a0216	chore: rename ExtractIngesterPartitionID() to make it generic (#19185)	8 months ago
benclive	ad46063601	chore: Empty builder returns non-nil result (#19184)	8 months ago
Christian Haudum	2ecf07d52c	chore(dataobj): Fix panic in column builder (#19153) ... There is an edge case in the column builder where it panics when the AppendNulls() call tries to append more NULL values than the specified MaxPageRows setting when backfilling rows. In that case the builder would panic because canAppend would always return false even when the previous page was flushed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
benclive	b7f14f9554	fix: Stop applying limits pushdown under a filter (#19154)	8 months ago
benclive	941864cec9	chore: Init objects in metastore in parallel (#19175)	8 months ago
benclive	dfb60dcaf0	fix: Metastore timerange filter (#19171)	8 months ago
benclive	3148a77966	chore: improve metastore Section calls (#19157)	8 months ago
George Robinson	3535ae4ae1	fix: use cooperative active sticky load balancer (#19160)	8 months ago
Trevor Whitney	89509dee9a	feat(goldfish): improve query performance with indexes and paritions (#19023)	8 months ago
Sandeep Sukhani	3618279603	chore: optimize chunk rewrite while filtering out deleted data (#19140)	8 months ago
Periklis Tsirakidis	a082b8a061	chore: Replace per section impl tenant function with generic one (#19139)	8 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	8 months ago
George Robinson	6fa4d0d6d4	chore: should use IsZero() instead of comparison to time.Time{} (#19145)	8 months ago
George Robinson	e321bf946c	chore: remove ConsumeRegex() from partitionring consumer (#19141)	8 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	8 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	8 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110) ... This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
renovate[bot]	70cff6ae0d	chore(deps): update dependency vite to v7.1.4 (main) (#19116) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
renovate[bot]	ab86bc301a	chore(deps): update dependency @types/react-dom to v19.1.9 (main) (#19114) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
renovate[bot]	c4a14bf996	chore(deps): update dependency @types/node to v22.18.1 (main) (#19112) ... Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	8 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	8 months ago
Sandeep Sukhani	0b2d0b070e	chore: make index updates idempotent from horizontally scalable compactor (#19093)	8 months ago
Salva Corts	af257fd058	feat: per policy overridable stream limits (#18994)	8 months ago