214 Commits (f6091a67d168d3a341a53dd6ab2aeffaea5fe71d)

Author	SHA1	Message	Date
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Periklis Tsirakidis	383032bccd	chore: Remove dataobj querier component (#19132)	5 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104)	5 months ago
benclive	065fc936fd	chore: Fix bugs in the index-builders (#19105)	5 months ago
benclive	5d428b2ac4	chore: Fix panic for queries with unsupported predicate types (#19086)	5 months ago
benclive	653d7ac19e	chore: Fix tenant info in sections when resetting (#19085) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	5 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034)	5 months ago
George Robinson	90ac000ce6	fix: records discarded if one fetch returns error (#19069)	5 months ago
George Robinson	5809c4739b	fix: nil error returned when client closed (#19068)	5 months ago
George Robinson	44aba62243	fix: fix possible data loss in dataobj-consumers (#19060)	5 months ago
George Robinson	cb3b74a45d	feat: add p50, p95 and p99 section sizes per tenant (#19054)	5 months ago
benclive	a7f001512a	chore: Fix index generation with filesystem (#19063)	5 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959)	5 months ago
George Robinson	598cab79e5	fix: don't need to open section to read tenant (#19053)	5 months ago
George Robinson	d256df481e	feat: add p50, p95 and p99 sections per tenant to stats (#19044)	5 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
George Robinson	3bab0ff5fe	fix: backoff cannot be canceled via ctx (#19041)	5 months ago
benclive	28fe97dcce	chore: fix metastore indexes being written with double prefixes (#19018)	5 months ago
George Robinson	1475762994	feat: print the tenant in the dump and stats sub-commands (#19033)	5 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	5 months ago
George Robinson	d8b22bfef8	fix: deadlock when shutting down dataobj-consumers (#18932)	6 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928)	6 months ago
George Robinson	20d93e3a8d	chore: rename lastFlush to lastFlushed and move some tests (#18918)	6 months ago
George Robinson	3ed2c4e127	chore: add safety check for estimated size (#18919)	6 months ago
George Robinson	058eaa7e0b	chore: improve flush unit tests with better separation (#18917)	6 months ago
George Robinson	31d5f1a738	fix: reset the last modified timestamp on flush (#18916)	6 months ago
Robert Fratto	71dc2d9bce	chore(dataobj): store section regions contiguously within the data object (#18901) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
George Robinson	278374609d	chore: create a factory to create partition processors (#18897)	6 months ago
Robert Fratto	515ebea4f2	chore(dataobj): track total download time of page metadata/data (#18896)	6 months ago
George Robinson	4608048388	chore: remove unused sync.Pool (#18895)	6 months ago
George Robinson	05f6ad1eb0	feat: add tenant to section builders (#18864)	6 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
benclive	19d90ef023	chore: Add sorting & tenant info to remaining section types (#18861)	6 months ago
Robert Fratto	642d826a00	chore(dataobj): make section tenancy a global construct (#18857) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: George Robinson <george.robinson@grafana.com>	6 months ago
George Robinson	be7ff4b294	chore: add flushAndCommit to make it harder to forget to commit offsets in future (#18851)	6 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694)	6 months ago
George Robinson	414538ae24	fix: mistake in error, should have said logs section (#18844)	6 months ago
George Robinson	36b2615156	feat: decode tenant when reading streams and logs sections (#18846)	6 months ago
George Robinson	0941b90614	fix: dataobj-consumers did not commit offsets on idle flush (#18834)	6 months ago
George Robinson	916246a2be	feat: add tenant to section metadata (#18843)	6 months ago
Ashwanth	88c8b44f2d	chore(dataobj): read non-projected predicate columns (#18797)	6 months ago
George Robinson	8895090643	fix: dataobj-consumers commit incorrect offset, which can cause data loss (#18833)	6 months ago
Robert Fratto	100e259a8d	chore(dataobj): introduce the concept of section info "extensions" (#18832)	6 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	6 months ago
Robert Fratto	61c40fca0c	chore(dataobj): remove support for deprecated fields (breaking change) (#18822)	6 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
George Robinson	53416a65ef	chore: use quartz.Clock to improve tests (#18810)	6 months ago
Robert Fratto	40057f5ce2	chore(dataobj): permit reading subset of section metadata (#18815) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
George Robinson	69a70dda88	chore: use zero value for time in partition_processor.go (#18798)	6 months ago
George Robinson	c47841024e	chore: move mocks into mock.go (#18799)	6 months ago