apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	736ccdbf93	chore(dataobj): rename dataset.Reader to RowReader (#20629 )	4 days ago
Christian Haudum	a3017f2b13	chore(dataobj-consumer): Sort logs object-wide (#19231 ) Sorting logs globally (object-wide per tenant) removes overlapping time ranges of sections in the objects. Sections contain logs from multiple streams, and the ingest lag of streams may vary. This means that although logs of individual sections are sorted by timestamp, the overall sorting of logs is not guaranteed. Therefore sections are sorted with a k-way merge (SortMerge), which does over-query data in case the query would reach its result limit early. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142 ) dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	6 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499 )	6 months ago
benclive	f66781d426	fix: Row reader with no stream ID predicates (#18368 )	7 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325 )	7 months ago
Robert Fratto	06cf52757b	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976 ) This PR introduces columnar reading APIs to both the logs and streams sections. As part of this change, both sections now expose the columns stored in the section, which are then used to define predicates on the readers. The columnar reading APIs emit a sequence of `arrow.Record`s, where each record has no more than the batch size passed to the `Reader.Read` method. Unlike the original row-based readers, the columnar readers: * Allow reading a subset of columns * Have column-based predicates that map literally to the new query engine To be able to represent a column's value for use in predicates, we import [github.com/apache/arrow-go/v18/arrow/scalar](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.3.0/arrow/scalar). Section Reader implementations perform mapping to and from the internal `dataset.Value`. The implementation of `logs.Reader` is almost an identical copy of the `streams.Reader` implementation. I've opted for duplicating the implementation in the short term, since there's no obvious way for how to deduplicate it in a way keeps the package structure easy to understand. We can change this in the future if a clean and simple approach presents itself. This PR does not yet update `DataObjScan` to make use of these new APIs; that's being left out of scope for another PR to handle.	8 months ago
Robert Fratto	36bcb5aa2d	chore(dataobj): fix regression in allocations of row readers (#17784 )	9 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762 ) Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	9 months ago
Robert Fratto	7b0d6d7144	chore(dataobj): Migrate to dynamic section types (#17708 )	9 months ago
Robert Fratto	3fbebd3bc4	chore(dataobj): migrate to section-relative offsets (#17704 )	9 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376 )	9 months ago
Ashwanth	5961923836	chore(dataobj): apply predicates sequentially (#17283 )	9 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988 ) Co-authored-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
Ashwanth	42eed6ded5	chore(dataobj): introduce InPredicate to replace long chain of OR operators (#16919 )	10 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833 )	11 months ago
Ashwanth	c51fa39684	chore(dataobj): pass down line filters to logs reader (#16802 )	11 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747 )	11 months ago
Robert Fratto	cd11367148	chore(dataobj): Wire dataset.Reader into dataobj.LogsReader and dataobj.StreamsReader (#16701 ) Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	11 months ago
Robert Fratto	f87dd2ddf9	chore(dataobj): add predicate type for reading logs and streams (#16200 )	12 months ago
Robert Fratto	1adb1e39d3	chore(dataobj): add initial high-level APIs for reading streams and log records (#15974 )	1 year ago

10 Commits (shantanu/dataobj-compactor)