apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762 ) Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	7 months ago
Robert Fratto	7b0d6d7144	chore(dataobj): Migrate to dynamic section types (#17708 )	8 months ago
Robert Fratto	3fbebd3bc4	chore(dataobj): migrate to section-relative offsets (#17704 )	8 months ago
Ashwanth	3ad1a64bc1	chore(dataobj): fixes a memory aliasing bug in dataobj reader (#17537 )	8 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376 )	8 months ago
Ashwanth	5961923836	chore(dataobj): apply predicates sequentially (#17283 )	8 months ago
Ashwanth	42e2407137	chore(dataobj): backfill sparse rows using EncodeN (#16485 )	8 months ago
Ashwanth	7dd4ab4b09	chore(dataobj): correct stats calculation for prepredicate bytes read (#17279 )	9 months ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128 )	9 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988 ) Co-authored-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Robert Fratto	bd4331363b	chore: align benchmark results between chunks and dataobjs (#17127 )	9 months ago
Ashwanth	42eed6ded5	chore(dataobj): introduce InPredicate to replace long chain of OR operators (#16919 )	9 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833 )	9 months ago
benclive	979f7614bc	perf: Optimization pass for reading from dataobjs (#16747 )	10 months ago
Robert Fratto	cd11367148	chore(dataobj): Wire dataset.Reader into dataobj.LogsReader and dataobj.StreamsReader (#16701 ) Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	10 months ago
Robert Fratto	8c4eb428c3	chore(dataobj): Download pages in 16MB batches (#16689 )	10 months ago
Robert Fratto	abf3772366	perf(dataobj): improve parallelized read spead of Zstd (#16632 )	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572 )	10 months ago
Robert Fratto	7f0dc01e21	chore(dataobj): introduce more efficient dataset reading (#16429 ) This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	10 months ago
Christian Haudum	f6fcc1194e	chore(dataobj): Decode values in batches (#16418 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Jackson Coelho	290c14a8cb	chore: fix flush process for dataobj (#16393 )	10 months ago
Ashwanth	651d410eea	chore(dataobj): add "uncompressed size" column to streams section (#16322 )	10 months ago
Owen Diehl	66889ec51d	feat(dataobj): cardinality estimation (#16233 )	11 months ago
Robert Fratto	66b2721cfa	chore(dataobj): do not panic on unrecognized compression type (#16088 )	11 months ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055 )	11 months ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011 ) Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	11 months ago
Robert Fratto	5ccc6790cb	chore(dataobj): support computation of min/max values in pages and columns (#16015 )	11 months ago
Robert Fratto	1adb1e39d3	chore(dataobj): add initial high-level APIs for reading streams and log records (#15974 )	11 months ago
Robert Fratto	5929b05a64	chore(dataobj): deduplicate decoder code across bucket and io.ReadSeeker (#15945 )	11 months ago
Christian Haudum	ed863fe1be	chore: Fix Loki arm builds (#15936 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Robert Fratto	93cee63f59	chore(dataobj): add compression ratio and final object size distributions (#15848 )	11 months ago
Robert Fratto	948f5c5f3e	chore(dataobj): Reintroduce sorting of the logs section (#15906 )	11 months ago
Robert Fratto	64665c4595	chore(dataobj): promote metadata types to hold uint64 instead of uint32 (#15850 )	11 months ago
Robert Fratto	2a60011b13	chore(dataobj): reduce total allocations while encoding (#15846 )	11 months ago
Robert Fratto	7eb5053d1c	chore(dataobj): improve performance of Streams.EstimatedSize (#15833 )	12 months ago
Robert Fratto	093e36af58	chore(dataobj): instrument data objects (#15803 )	12 months ago
Robert Fratto	848c31fce9	chore(dataobj): temporarily disable sorting (#15799 )	12 months ago
Robert Fratto	127ad5326f	chore(dataobj): support for building and flushing data objects (#15780 )	12 months ago
Robert Fratto	83a506954c	chore(dataobj): fix bug where instances of *bytes.Buffer are incorrectly shared (#15776 )	12 months ago
Robert Fratto	5980396b76	chore(dataobj): fix bug with partial reads of strings (#15767 )	12 months ago
Robert Fratto	11714a71c7	chore(dataobj): add logs tracking and encoding (#15757 )	12 months ago
Robert Fratto	aec8e968da	chore(dataobj): track non-NULL values in pages/columns (#15758 )	12 months ago
Robert Fratto	d1fd0b9ffc	chore(dataobj): add streams tracking and encoding (#15713 )	12 months ago
Robert Fratto	bcec63d9c2	chore(dataobj): logs section metadata and encoding/decoding (#15720 )	12 months ago
Robert Fratto	896e138425	chore(dataobj): data object encoding and decoding (#15676 )	12 months ago
Robert Fratto	c581218c4f	chore(dataobj): add Dataset with iteration and sorting support (#15652 )	12 months ago
Robert Fratto	f2fc0c26c2	chore(dataobj): column building (#15634 ) This commit adds the ability to accumulate sequences of dataset.Value into a column, which is split up across multiple pages. Each page is broken down into two parts: * A bitmap-encoded sequence of booleans, where 1 indicates a row has a value and 0 indicates the row is NULL, and * the encoded sequence of non-NULL values, whose encoding is determined by the column options. The sequence of non-NULL values is then optionally compressed. This commit also includes initial support for reading these columns, starting with internal-only helper utilities for unit tests.	12 months ago
Robert Fratto	6042d0712e	chore(dataobj): add bitmap encoding (#15629 )	12 months ago
Robert Fratto	9a21590238	chore(dataobj): initial commit of value encoding (#15606 )	12 months ago

49 Commits (60c76318e66d33a726b41d8542e71663e581c92a)