apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Robert Fratto	595715146f	chore(dataobj)!: introduce new centralized dataset encoding format (#18871 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
benclive	e5784d7a82	perf: Reuse Values in dataobj.Reader to reduce allocs (#16988 ) Co-authored-by: Robert Fratto <robertfratto@gmail.com>	10 months ago
Robert Fratto	7f0dc01e21	chore(dataobj): introduce more efficient dataset reading (#16429 ) This commit introduces a new API, dataset.Reader, which scans through values in datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been removed in favour of the new Reader. dataset.Reader is more efficient than dataset.Iter because: - It avoids using iter.Pull; the number of values flowing through coroutines added a significant amount of CPU overhead. - It supports efficient uses of predicates, where non-predicate columns are only loaded after finding matching rows from predicate columns; this avoids loading unnecessary pages. - It adds initial support for bulk page downloading, so that we need fewer roundtrips to object storage when iterating over a dataset. (This is not fully implemented yet; see below). - Early performance benchmarks see a ~3.4x speed improvement over reading 7GB of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s (with dataset.Reader). This commit is truly massive, and to avoid it getting any larger, I'm leaving some follow up work to additional commits: - Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API. - Adding support for downloading pages in batches; dataset.Reader will request multiple pages at once, but our decoder still unwraps that into several GET requests to object storage. Signed-off-by: Robert Fratto <robertfratto@gmail.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	11 months ago

Author

SHA1

Message

Date

Robert Fratto

595715146f

chore(dataobj)!: introduce new centralized dataset encoding format (#18871 )

Signed-off-by: Robert Fratto <robertfratto@gmail.com>

5 months ago

benclive

e5784d7a82

perf: Reuse Values in dataobj.Reader to reduce allocs (#16988 )

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

10 months ago

Robert Fratto

7f0dc01e21

chore(dataobj): introduce more efficient dataset reading (#16429 )

This commit introduces a new API, dataset.Reader, which scans through values in
datasets more efficiently than dataset.Iter. As a result, dataset.Iter has been
removed in favour of the new Reader.

dataset.Reader is more efficient than dataset.Iter because:

- It avoids using iter.Pull; the number of values flowing through coroutines
  added a significant amount of CPU overhead.

- It supports efficient uses of predicates, where non-predicate columns are
  only loaded after finding matching rows from predicate columns; this avoids
  loading unnecessary pages.

- It adds initial support for bulk page downloading, so that we need fewer
  roundtrips to object storage when iterating over a dataset. (This is not
  fully implemented yet; see below).

- Early performance benchmarks see a ~3.4x speed improvement over reading 7GB
  of uncompressed logs, going from 192 MB/s (with dataset.Iter) to 652 MB/s
  (with dataset.Reader).

This commit is truly massive, and to avoid it getting any larger, I'm leaving
some follow up work to additional commits:

- Updating dataobj.LogsReader and dataobj.StreamsReader to use the new API.

- Adding support for downloading pages in batches; dataset.Reader will request
  multiple pages at once, but our decoder still unwraps that into several GET
  requests to object storage.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>

11 months ago

3 Commits (c23bb7ce4e48b321e88ccfc13eb3fe414fef5afb)