Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Robert Fratto f6091a67d1 chore(engine): move to toggleable section prefetching (#19142) ... dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>		4 months ago
..
config	chore: Remove dataobj querier component (#19132)	4 months ago
consumer	chore: Fix tenant info in sections when resetting (#19085)	4 months ago
explorer	chore(dataobj)!: introduce new centralized dataset encoding format (#18871)	5 months ago
index	chore: Fix bugs in the index-builders (#19105)	4 months ago
internal	chore(engine): move to toggleable section prefetching (#19142)	4 months ago
metastore	chore: Read multi-tenant objects in query-engine (#19104)	4 months ago
sections	chore(engine): move to toggleable section prefetching (#19142)	4 months ago
tools	feat: add p50, p95 and p99 section sizes per tenant (#19054)	4 months ago
uploader	feat: build multi-tenant data objects (#19034)	4 months ago
builder.go	chore(dataobj): make section tenancy a global construct (#18857)	5 months ago
builder_test.go	chore(dataobj): make section tenancy a global construct (#18857)	5 months ago
dataobj.go	chore(dataobj): make section tenancy a global construct (#18857)	5 months ago
decoder.go	chore(dataobj): introduce the concept of section info "extensions" (#18832)	5 months ago
decoder_metadata.go	…
encoder.go	chore(dataobj): store section regions contiguously within the data object (#18901)	5 months ago
encoder_test.go	chore(dataobj): introduce the concept of section info "extensions" (#18832)	5 months ago
metrics.go	chore(dataobj): remove support for deprecated fields (breaking change) (#18822)	5 months ago
range_reader.go	…
section.go	chore(dataobj)!: introduce new centralized dataset encoding format (#18871)	5 months ago
section_reader.go	chore(dataobj): introduce the concept of section info "extensions" (#18832)	5 months ago
snapshot.go	chore(dataobj): store section regions contiguously within the data object (#18901)	5 months ago
snapshot_test.go	chore(dataobj): store section regions contiguously within the data object (#18901)	5 months ago