63 Commits (90689775bfd6bba3025f2a24b8a9dde47ea4c41b)

Author	SHA1	Message	Date
Robert Fratto	a5c722cbed	chore(dataobj): list pages across all columns in one request (#18678)	5 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	6 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499)	6 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	6 months ago
Periklis Tsirakidis	6eedcaaded	chore(dataobj): Add new index pointers section type (#18441)	6 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	6 months ago
Robert Fratto	26cb50e65a	chore(dataobj): Improve performance of dataset.Value API (#18339) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	6 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	6 months ago
Ashwanth	5c02493fe2	chore(dataobj): avoid reading a predicate column more than once (#18324)	6 months ago
Robert Fratto	30ccf5172c	chore(util/arrowtest): export arrowtest package (#18255)	6 months ago
benclive	9642af3ac6	chore: Add new pointers section type for dataobj (#18243)	6 months ago
Robert Fratto	06cf52757b	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976) ... This PR introduces columnar reading APIs to both the logs and streams sections. As part of this change, both sections now expose the columns stored in the section, which are then used to define predicates on the readers. The columnar reading APIs emit a sequence of `arrow.Record`s, where each record has no more than the batch size passed to the `Reader.Read` method. Unlike the original row-based readers, the columnar readers: * Allow reading a subset of columns * Have column-based predicates that map literally to the new query engine To be able to represent a column's value for use in predicates, we import [github.com/apache/arrow-go/v18/arrow/scalar](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.3.0/arrow/scalar). Section Reader implementations perform mapping to and from the internal `dataset.Value`. The implementation of `logs.Reader` is almost an identical copy of the `streams.Reader` implementation. I've opted for duplicating the implementation in the short term, since there's no obvious way for how to deduplicate it in a way keeps the package structure easy to understand. We can change this in the future if a clean and simple approach presents itself. This PR does not yet update `DataObjScan` to make use of these new APIs; that's being left out of scope for another PR to handle.	7 months ago
Robert Fratto	c5ac922e55	chore(dataobj): Use pooled buffers when decoding pages (#17955)	7 months ago
Christian Haudum	48f1d0ebc0	chore(dataobj): Decompress zstd compressed pages upfront (#17841) ... This PR changes the decompression of data object pages, so that they are decompressed fully upfront into a bytes buffer, instead of passing a streaming zstd decompressor (reader) to the page decoder. This this allows to release the zstd decompressor to the pool earlier.Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762) ... Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	8 months ago
Robert Fratto	7b0d6d7144	chore(dataobj): Migrate to dynamic section types (#17708)	8 months ago
Robert Fratto	3fbebd3bc4	chore(dataobj): migrate to section-relative offsets (#17704)	8 months ago
Ashwanth	3ad1a64bc1	chore(dataobj): fixes a memory aliasing bug in dataobj reader (#17537)	8 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376)	8 months ago
Ashwanth	5961923836	chore(dataobj): apply predicates sequentially (#17283)	8 months ago
Ashwanth	42e2407137	chore(dataobj): backfill sparse rows using EncodeN (#16485)	8 months ago
Ashwanth	7dd4ab4b09	chore(dataobj): correct stats calculation for prepredicate bytes read (#17279)	9 months ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128)	9 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Robert Fratto	bd4331363b	chore: align benchmark results between chunks and dataobjs (#17127)	9 months ago
Ashwanth	42eed6ded5	chore(dataobj): introduce InPredicate to replace long chain of OR operators (#16919)	9 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	10 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747)	10 months ago
Robert Fratto	cd11367148	chore(dataobj): Wire dataset.Reader into dataobj.LogsReader and dataobj.StreamsReader (#16701) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	10 months ago
Robert Fratto	8c4eb428c3	chore(dataobj): Download pages in 16MB batches (#16689)	10 months ago
Robert Fratto	abf3772366	perf(dataobj): improve parallelized read spead of Zstd (#16632)	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572)	10 months ago
Robert Fratto	7f0dc01e21	chore(dataobj): introduce more efficient dataset reading (#16429) ... This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Christian Haudum	f6fcc1194e	chore(dataobj): Decode values in batches (#16418) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Jackson Coelho	290c14a8cb	chore: fix flush process for dataobj (#16393)	11 months ago
Ashwanth	651d410eea	chore(dataobj): add "uncompressed size" column to streams section (#16322)	11 months ago
Owen Diehl	66889ec51d	feat(dataobj): cardinality estimation (#16233)	11 months ago
Robert Fratto	66b2721cfa	chore(dataobj): do not panic on unrecognized compression type (#16088)	11 months ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055)	11 months ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	11 months ago
Robert Fratto	5ccc6790cb	chore(dataobj): support computation of min/max values in pages and columns (#16015)	11 months ago
Robert Fratto	1adb1e39d3	chore(dataobj): add initial high-level APIs for reading streams and log records (#15974)	12 months ago
Robert Fratto	5929b05a64	chore(dataobj): deduplicate decoder code across bucket and io.ReadSeeker (#15945)	12 months ago
Christian Haudum	ed863fe1be	chore: Fix Loki arm builds (#15936) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Robert Fratto	93cee63f59	chore(dataobj): add compression ratio and final object size distributions (#15848)	12 months ago
Robert Fratto	948f5c5f3e	chore(dataobj): Reintroduce sorting of the logs section (#15906)	12 months ago
Robert Fratto	64665c4595	chore(dataobj): promote metadata types to hold uint64 instead of uint32 (#15850)	12 months ago
Robert Fratto	2a60011b13	chore(dataobj): reduce total allocations while encoding (#15846)	12 months ago
Robert Fratto	7eb5053d1c	chore(dataobj): improve performance of Streams.EstimatedSize (#15833)	12 months ago
Robert Fratto	093e36af58	chore(dataobj): instrument data objects (#15803)	12 months ago