254 Commits (9d0d1084e39fdfd4a58d701b13ff755f4e5f36eb)

Author	SHA1	Message	Date
Paul Rogers	91d4eb6ce1	feat: Add partition state to consumption lag metric (#19912)	1 month ago
benclive	17285207f1	chore: Reduce buffer size for bloom lookups (#19926)	1 month ago
benclive	9c552a23eb	chore: Add a dataobj readme (#19910)	1 month ago
Christian Haudum	1cc97513d7	chore(metastore): Cleanup and reduce log volume (#19889) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 month ago
Christian Haudum	97fc8b3a50	chore(metastore): Simplify public API of Metastore (#19886) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
renovate-sh-app[bot]	d76b3bf495	fix(deps): update module github.com/apache/arrow-go/v18 to v18.4.1 (main) (#19750) ... Signed-off-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: renovate-sh-app[bot] <219655108+renovate-sh-app[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	2 months ago
George Robinson	a28e755e79	chore: rename the prepare delayed downscale handler to better communicate what it does (#19727)	2 months ago
George Robinson	0af297cd4c	chore: remove loki_dataobj_consumer_processing_delay histogram (#19718)	2 months ago
Christian Haudum	0b4a7f5a2c	chore(dataobj-consumer): Re-use builder for CopyAndSort() (#19696) ... Re-use logsobj.Builder instance in partition processor when sorting logs object wide after flushing. The metrics for the builder in CopyAndSort are never registered, which means the loki_dataobj_sort_duration_seconds histogram is never exported/scraped. We cannot register them with RegisterMetrics() because the metrics are already registered here, and registering them twice would cause a panic.	2 months ago
benclive	6ee50064ea	chore: Ensure partition is set on partition-processor (#19693)	2 months ago
George Robinson	da37290a40	fix: loki_dataobj_sort_duration_seconds never registered (#19679)	2 months ago
Christian Haudum	73bc30d9a8	perf(engine): Improve regexp expression evaluation (#19644) ... This PR changes the evaluation of regex-match binary expression evaluations so that the regexp on right hand side of the expression is only compiled once for each batch evaluation, rather than for each row. This saves a ton of allocations and improves general performance of queries with a regexp line or label filter.	2 months ago
George Robinson	d146c9f4ad	feat: add enabled flag for dataobjs (#19613)	2 months ago
Ashwanth	5a06d44826	chore: table build fill all columns to max rows (#19614)	2 months ago
benclive	a6de577b3a	chore: Fix page skipping with empty predicates (#19602)	2 months ago
Ashwanth	bfd70602dd	chore: correct dataobj reader rows read return value (#19583)	2 months ago
George Robinson	977fb2ba6f	feat: consume single topics in dataobj-consumer using partition ring (#19186)	2 months ago
benclive	ea2b4d0f84	chore: Use Kafka round-robin balancer for index-builder (#19574)	2 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569)	2 months ago
benclive	c31e25762e	chore: Fix metadata bloom check with multiple predicates (#19389)	2 months ago
Ashwanth	fa8bd848f6	chore: update buildTable to sort rows using `BuilderOptions.SortOrder` (#19515)	3 months ago
Robert Fratto	7556192423	chore(dataobj): use single Zstd decoder (#19508)	3 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470) ... ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	b0f5ca7f8c	chore: Dedup log objects during build (#19378)	3 months ago
Christian Haudum	386d4e1653	chore(dataobj): Make logs sort order in dataobjects configurable (#19373) ... This PR introduces a new configuration option `-dataobj-consumer.dataobj-sort-order` which controls the sort order of logs in the logs section of the dataobj when building new objects. There are two available sort options: 1. `timestamp DESC, streamID ASC` (configuration value `timestamp-desc`) 2. `streamID ASC, timestamp DESC` (configuration value `stream-asc`) The new setting is undocumented and may be removed in the future without notice. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	c5a0a3827d	chore: Memory improvements in dataobj page downloads (#19301)	3 months ago
benclive	2dac5ef8a7	chore: Write empty message values in logs section writer (#19359)	3 months ago
Sophie Waldman	af6983e9de	fix: Add special case handling for comparing empty and nil values (#19348) ... We can have a predicate checking for an empty string, e.g. {"foo" = ""}. If a row has no value for the column "foo", it will return a nil value for that column. Thus, when we go to compare the row with the predicate, we will be comparing an empty value to a nil value. This change adjusts the comparison check so that we correctly handle this special case.	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241)	3 months ago
Christian Haudum	457ec0956d	chore(dataobj-index): Handle special cases in the section lookup of the metastore (#19203) ... There are three edge cases that this PR improves: * No table (toc) paths could be resolved: return early with empty result * No index paths could be resolved: return early with empty result * No relevant sections could be resolved: return with empty result instead of with an error Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	4 months ago
Christian Haudum	2ecf07d52c	chore(dataobj): Fix panic in column builder (#19153) ... There is an edge case in the column builder where it panics when the AppendNulls() call tries to append more NULL values than the specified MaxPageRows setting when backfilling rows. In that case the builder would panic because canAppend would always return false even when the previous page was flushed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
benclive	941864cec9	chore: Init objects in metastore in parallel (#19175)	4 months ago
benclive	dfb60dcaf0	fix: Metastore timerange filter (#19171)	4 months ago
benclive	3148a77966	chore: improve metastore Section calls (#19157)	4 months ago
Periklis Tsirakidis	a082b8a061	chore: Replace per section impl tenant function with generic one (#19139)	4 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	4 months ago
benclive	5d428b2ac4	chore: Fix panic for queries with unsupported predicate types (#19086)	4 months ago
benclive	653d7ac19e	chore: Fix tenant info in sections when resetting (#19085) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	4 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	4 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	4 months ago
George Robinson	44aba62243	fix: fix possible data loss in dataobj-consumers (#19060)	4 months ago