Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Robert Fratto 7f0dc01e21 chore(dataobj): introduce more efficient dataset reading (#16429) ... This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>		11 months ago
..
dataset	chore(dataobj): introduce more efficient dataset reading (#16429)	11 months ago
encoding	chore(dataobj): introduce more efficient dataset reading (#16429)	11 months ago
metadata	chore(dataobj): add "uncompressed size" column to streams section (#16322)	11 months ago
result	chore(dataobj): column building (#15634)	1 year ago
sections	chore(dataobj): introduce more efficient dataset reading (#16429)	11 months ago
streamio	chore(dataobj): column building (#15634)	1 year ago
util	chore(dataobj): introduce more efficient dataset reading (#16429)	11 months ago