234 Commits (v3.6.2)

Author	SHA1	Message	Date
Ashwanth	fa8bd848f6	chore: update buildTable to sort rows using `BuilderOptions.SortOrder` (#19515)	2 months ago
Robert Fratto	7556192423	chore(dataobj): use single Zstd decoder (#19508)	2 months ago
Christian Haudum	c06eb636e5	chore(engine): Add "compatibility node" to physical plan to adhere with naming of "colliding labels" in v1 engine (#19470) ... ### Summary The v1 engine has a mechanism to rename labels in case they have the same name but different origin, such as labels, structured metadata, or parsed fields. 1. In case a log line has a structured metadata key with the same name as the label name of the stream, than the metadata key is suffixed with `_extracted`, such as `service_extracted`, if `service` exists in both `labels` and `metadata`. 2. In case a parser creates a parsed field with the same as the label name of the stream, then the parsed key is suffixed with `_extracted` in the same way as case 1. However, if the field name also collides with a structured metadata key, then the extracted structured metadata is replaced with the extracted parsed field. This PR only implements the first case. As a follow up PR, the second case needs to be implemented as well. Additionally, the newly introduced "compatibility node" should also be made optional with a feature flag and/or per-request. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	b0f5ca7f8c	chore: Dedup log objects during build (#19378)	3 months ago
Christian Haudum	386d4e1653	chore(dataobj): Make logs sort order in dataobjects configurable (#19373) ... This PR introduces a new configuration option `-dataobj-consumer.dataobj-sort-order` which controls the sort order of logs in the logs section of the dataobj when building new objects. There are two available sort options: 1. `timestamp DESC, streamID ASC` (configuration value `timestamp-desc`) 2. `streamID ASC, timestamp DESC` (configuration value `stream-asc`) The new setting is undocumented and may be removed in the future without notice. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
benclive	c5a0a3827d	chore: Memory improvements in dataobj page downloads (#19301)	3 months ago
benclive	2dac5ef8a7	chore: Write empty message values in logs section writer (#19359)	3 months ago
Sophie Waldman	af6983e9de	fix: Add special case handling for comparing empty and nil values (#19348) ... We can have a predicate checking for an empty string, e.g. {"foo" = ""}. If a row has no value for the column "foo", it will return a nil value for that column. Thus, when we go to compare the row with the predicate, we will be comparing an empty value to a nil value. This change adjusts the comparison check so that we correctly handle this special case.	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241)	3 months ago
Christian Haudum	457ec0956d	chore(dataobj-index): Handle special cases in the section lookup of the metastore (#19203) ... There are three edge cases that this PR improves: * No table (toc) paths could be resolved: return early with empty result * No index paths could be resolved: return early with empty result * No relevant sections could be resolved: return with empty result instead of with an error Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231) ... Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Periklis Tsirakidis	ea45fa3f07	chore: Refactor dataobj index builder to handle buffered events from stale partitions (#19189)	3 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206)	3 months ago
Christian Haudum	2ecf07d52c	chore(dataobj): Fix panic in column builder (#19153) ... There is an edge case in the column builder where it panics when the AppendNulls() call tries to append more NULL values than the specified MaxPageRows setting when backfilling rows. In that case the builder would panic because canAppend would always return false even when the previous page was flushed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
benclive	941864cec9	chore: Init objects in metastore in parallel (#19175)	4 months ago
benclive	dfb60dcaf0	fix: Metastore timerange filter (#19171)	4 months ago
benclive	3148a77966	chore: improve metastore Section calls (#19157)	4 months ago
Periklis Tsirakidis	a082b8a061	chore: Replace per section impl tenant function with generic one (#19139)	4 months ago
Periklis Tsirakidis	55ce348a38	chore: Replace processingDelay histogram with a gauge metric (#19151)	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128) ... This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	4 months ago
benclive	5d428b2ac4	chore: Fix panic for queries with unsupported predicate types (#19086)	4 months ago
benclive	653d7ac19e	chore: Fix tenant info in sections when resetting (#19085) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	4 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	4 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	4 months ago
George Robinson	44aba62243	fix: fix possible data loss in dataobj-consumers (#19060)	4 months ago
George Robinson	cb3b74a45d	feat: add p50, p95 and p99 section sizes per tenant (#19054)	4 months ago
benclive	a7f001512a	chore: Fix index generation with filesystem (#19063)	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	4 months ago
George Robinson	598cab79e5	fix: don't need to open section to read tenant (#19053)	4 months ago
George Robinson	d256df481e	feat: add p50, p95 and p99 sections per tenant to stats (#19044)	4 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
George Robinson	3bab0ff5fe	fix: backoff cannot be canceled via ctx (#19041)	4 months ago
benclive	28fe97dcce	chore: fix metastore indexes being written with double prefixes (#19018)	4 months ago
George Robinson	1475762994	feat: print the tenant in the dump and stats sub-commands (#19033)	4 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	4 months ago
George Robinson	d8b22bfef8	fix: deadlock when shutting down dataobj-consumers (#18932)	4 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928)	4 months ago
George Robinson	20d93e3a8d	chore: rename lastFlush to lastFlushed and move some tests (#18918)	4 months ago
George Robinson	3ed2c4e127	chore: add safety check for estimated size (#18919)	4 months ago
George Robinson	058eaa7e0b	chore: improve flush unit tests with better separation (#18917)	4 months ago
George Robinson	31d5f1a738	fix: reset the last modified timestamp on flush (#18916)	4 months ago
Robert Fratto	71dc2d9bce	chore(dataobj): store section regions contiguously within the data object (#18901) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
George Robinson	278374609d	chore: create a factory to create partition processors (#18897)	4 months ago
Robert Fratto	515ebea4f2	chore(dataobj): track total download time of page metadata/data (#18896)	4 months ago
George Robinson	4608048388	chore: remove unused sync.Pool (#18895)	4 months ago