34 Commits (JStickler-patch-1)

Author	SHA1	Message	Date
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	6 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	6 months ago
benclive	1f0edcd4ac	feat: Initial index-builder implementation (#18297)	6 months ago
Robert Fratto	26cb50e65a	chore(dataobj): Improve performance of dataset.Value API (#18339) ... Co-authored-by: Ashwanth <iamashwanth@gmail.com>	6 months ago
benclive	a238816751	perf: Use map for InPredicate when reading dataobj (#18325)	6 months ago
Ashwanth	5c02493fe2	chore(dataobj): avoid reading a predicate column more than once (#18324)	6 months ago
Robert Fratto	c5ac922e55	chore(dataobj): Use pooled buffers when decoding pages (#17955)	7 months ago
Christian Haudum	48f1d0ebc0	chore(dataobj): Decompress zstd compressed pages upfront (#17841) ... This PR changes the decompression of data object pages, so that they are decompressed fully upfront into a bytes buffer, instead of passing a streaming zstd decompressor (reader) to the page decoder. This this allows to release the zstd decompressor to the pool earlier.Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ashwanth	3ad1a64bc1	chore(dataobj): fixes a memory aliasing bug in dataobj reader (#17537)	8 months ago
Ashwanth	5961923836	chore(dataobj): apply predicates sequentially (#17283)	8 months ago
Ashwanth	42e2407137	chore(dataobj): backfill sparse rows using EncodeN (#16485)	8 months ago
Ashwanth	7dd4ab4b09	chore(dataobj): correct stats calculation for prepredicate bytes read (#17279)	9 months ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128)	9 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988) ... Co-authored-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Ashwanth	42eed6ded5	chore(dataobj): introduce InPredicate to replace long chain of OR operators (#16919)	9 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	10 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747)	10 months ago
Robert Fratto	cd11367148	chore(dataobj): Wire dataset.Reader into dataobj.LogsReader and dataobj.StreamsReader (#16701) ... Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	10 months ago
Robert Fratto	abf3772366	perf(dataobj): improve parallelized read spead of Zstd (#16632)	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572)	10 months ago
Robert Fratto	7f0dc01e21	chore(dataobj): introduce more efficient dataset reading (#16429) ... This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Christian Haudum	f6fcc1194e	chore(dataobj): Decode values in batches (#16418) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Owen Diehl	66889ec51d	feat(dataobj): cardinality estimation (#16233)	11 months ago
Robert Fratto	66b2721cfa	chore(dataobj): do not panic on unrecognized compression type (#16088)	11 months ago
Robert Fratto	5ccc6790cb	chore(dataobj): support computation of min/max values in pages and columns (#16015)	11 months ago
Christian Haudum	ed863fe1be	chore: Fix Loki arm builds (#15936) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Robert Fratto	948f5c5f3e	chore(dataobj): Reintroduce sorting of the logs section (#15906)	12 months ago
Robert Fratto	127ad5326f	chore(dataobj): support for building and flushing data objects (#15780)	12 months ago
Robert Fratto	5980396b76	chore(dataobj): fix bug with partial reads of strings (#15767)	12 months ago
Robert Fratto	aec8e968da	chore(dataobj): track non-NULL values in pages/columns (#15758)	12 months ago
Robert Fratto	c581218c4f	chore(dataobj): add Dataset with iteration and sorting support (#15652)	1 year ago
Robert Fratto	f2fc0c26c2	chore(dataobj): column building (#15634) ... This commit adds the ability to accumulate sequences of dataset.Value into a column, which is split up across multiple pages. Each page is broken down into two parts: * A bitmap-encoded sequence of booleans, where 1 indicates a row has a value and 0 indicates the row is NULL, and * the encoded sequence of non-NULL values, whose encoding is determined by the column options. The sequence of non-NULL values is then optionally compressed. This commit also includes initial support for reading these columns, starting with internal-only helper utilities for unit tests.	1 year ago
Robert Fratto	6042d0712e	chore(dataobj): add bitmap encoding (#15629)	1 year ago
Robert Fratto	9a21590238	chore(dataobj): initial commit of value encoding (#15606)	1 year ago