apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128 ) This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142 ) dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104 )	4 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110 ) This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Ashwanth	d950d4075d	chore(engine): improve performance of matching window calculation (#19013 ) Currently range aggregation linearly scans through all the possible time windows (a 24h query with a 1m step results in 1440 windows) to find the matching ones. This is quite expensive as this matching is done for each row. This PR updates the matching code to use arithmetic which runs in constant time for cases where step >= range and for overlapping windows (step < range) it uses binary search. Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034 )	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039 ) This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
Christian Haudum	389fc5ade6	chore(engine): Correctly free record allocations (#19040 ) Fixes panics in the v2 query engine, due to Arrow records being freed at incorrect places. Rather than freeing the previous batch created by the KWayMerge on every Read() call, fully consumed batches by the KWayMerge need to be freed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Robert Fratto	2326bb0f4b	chore(engine): fix instances of double frees (#18915 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Sophie Waldman	40423f8f04	fix(engine): Don't attempt to release batch reference on read for merge node (#18913 ) Signed-off-by: Sophie Waldman <sophie.waldman@grafana.com>	5 months ago
Ashwanth	827b1f2df4	chore(engine): add support for range metric queries (#18892 )	5 months ago
Ashwanth	5203a0c9ad	chore(dataobj): add reader stats (#18694 )	5 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Robert Fratto	f98c932dcb	chore(engine): propagate batch and page cache size down to streamsView (#18744 )	5 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622 )	5 months ago
Robert Fratto	385d397404	chore(engine/executor): reduce cost of executing DataObjScan plan node (#18679 )	5 months ago
Robert Fratto	e751db8194	chore(engine): fix race condition when closing prefetch wrapper (#18668 )	5 months ago
Ashwanth	7c5bce70dc	chore(engine): introduce merge node (#18641 )	5 months ago
Robert Fratto	a3380f6480	chore(engine): add tracing instrumentation (#18648 )	6 months ago
Robert Fratto	56e7318bec	chore(engine): don't sort metric queries (#18594 )	6 months ago
Robert Fratto	ed1b61d253	chore(engine): improve performance of topk pipeline (#18593 )	6 months ago
Robert Fratto	e1bc2f9d7f	chore(engine): fix panic when closing uninitialized prefetchWrapper (#18592 )	6 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577 )	6 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499 )	6 months ago
Christian Haudum	6f7dc12eb4	chore(engine): Add support for regex and substr filters (#18501 ) Filters predicates on ambiguous columns (basically any label filter expression) cannot be pushed down to the DataObjScan and need to be evaluated in the Filter node. This PR adds the evaluation functions for MATCH_RE(string,string) and MATCH_SUBSTR(string, string) and their negating equivalent. Under the hood, the PR also replaces the type specific comparison function implementations with a generic one, which now also handle arrow.Timestamp arrays correctly. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490 )	6 months ago
Christian Haudum	cea56469b9	chore(engine): Pass context to Pipeline's Read() function (#18459 ) This change unifies how the request context is passed through the execution of pipelines. Instead of storing the context on pipelines, pass them as argument to the `Read()` function. This change alters the interface method of `Pipeline` from `Read() bool` to `Read(context.Context) bool`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	d446d47cf8	chore(engine): Prefetch inputs of SortMerge operator (#18395 ) This PR changes how inputs of the SortMerge operator are consumed. The inputs are wrapped with a `prefetchWrapper` which consumes the next batch (`arrow.Record`) in the background while the current batch is processed by the parent node. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	d9f7d785ff	chore(engine): Sort entries in dataobj sections by timestamp descending (#18367 ) Sorting sections by `timestamp DESC` first and `streamID ASC` second allows for more efficient reading (by section) for BACKWARD log queries. - This change breaks FORWARD log queries. Which is ok for the time being. We will add support later again when columnar reading of the dataobj is introduced. - This change renders existing dataobj in storage incompatible. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Ashwanth	f1d6c37342	chore(dataobjscan): correctly pass context to readers (#18386 )	6 months ago
Christian Haudum	4305a6aa61	chore(engine): Make literals simple types (#17733 ) Representing data types as simple types (string, int64, float64) makes them more efficient to cast without conversion. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Robert Fratto	a0914281b8	chore(engine): change DataObjScan to only cover a single logs section (#18350 )	6 months ago
Robert Fratto	5266865fef	chore(engine): add implementation for performing topk across streams of Arrow records (#18330 ) This commit adds a `engine.topkBatch` type, which can be used by the implementation of execution nodes to perform compute the set of topk rows from a stream of records. This is needed as part of #17976, and more generally for efficiently sorting rows when a limit is also used. Implementing this required quite a few helper utilities in the new `arrowagg` package. The `arrowagg` package provides utilities for aggregating various Arrow types: * `arrowagg.Arrays` to concatenate a stream of `arrow.Array`s. * `arrayagg.Records` to concatenate a stream of `arrow.Record`s, creating a merged schema. * `arrowagg.Mapper` for quickly finding fields in incoming schemas, since arrow-go can return false when comparing two identical schemas/fields. Co-authored-by: Ashwanth <iamashwanth@gmail.com>	6 months ago
Christian Haudum	4fbe7c1a21	fix(engine): Fix bug in `SortMerge` which caused rows to be skipped due to incorrect sorting (#18334 ) This PR fixes two problems with the sorting of rows of input records: 1) When sorting input indexes (`batchIndexes []int`) based on their current timestamp at index `offset` (`timestamps []int64`), the positional correspondence was not maintained leading to pull rows from wrong records. 2) When calculating how many rows from the current batch could be yield, the comparison of the timestamp with the next highest timestamp was wrong. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Ashwanth	50374f0bb1	chore(engine): column expression evaluation for ambiguous column type look-ups (#18228 )	6 months ago
Ashwanth	646ebe81d8	chore(engine): adds AST conversion and result builders for metric queries (#18166 )	6 months ago
Christian Haudum	b316d52612	chore(engine): Use shard information from query frontend in the new query engine (#17792 ) What this PR does / why we need it: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. Note, this is only an interim solution used for validating and testing. By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. Shard size considerations While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. How does sharding with data objects in this PR work? The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Christian Haudum	16ccf9be1e	chore(engine): Filter by unknown column (#18131 ) The new engine fails to execute a query when an `arrow.Record` (batch) does not contain the column on which a label filter is used, e.g. `\| foo="bar"` where `foo` is neither a label nor a structured metadata. This change also fixed an incorrect conversion of the label filter operator `=`, which was converted to `MATCH_STR`, but requires to be `EQ`. The new engine reflects current behaviour of a label filter `\| foo=""` where `foo` is undefined. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ashwanth	635896dab9	chore(engine): adds vector aggregation planning and execution nodes (#18100 )	7 months ago
Christian Haudum	3956ea1be7	chore(engine): Map line filter expressions to data object predicates (#18102 ) This PR converts line filter expressions into message predicates for the data object scan, instead of returning an error. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Christian Haudum	78342c030a	chore(engine): Correctly propagate search direction to data object scan (#18112 ) The sort order of the physical plan was only used in the SortMerge node, but not on the DataObjScan. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Ashwanth	1effa30851	chore(engine): implement range aggregation operator (#17997 )	7 months ago
Karsten Jeschkies	c2573fea5a	refactor(stringlabels): Support stringlabels in util, engine and logql. (#17998 ) This is a next step of support Prometheus `stringlabels` implementation in Loki. It adds support in the `util`, `engine` and `logql` packages. The tests should compile with build tag `stringlabels`.	7 months ago
Christian Haudum	dd68679fd7	fix(engine): Fix out-of-bounds panic in SortMerge implementation (#17967 ) If p.inputs[i].Read() returns a batch with 0 rows, getting the timestamp for the record would panic, because it would try to access the value of the timestamp column at position 0, which does not exist. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	7 months ago
Robert Fratto	36bcb5aa2d	chore(dataobj): fix regression in allocations of row readers (#17784 )	8 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762 ) Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	8 months ago
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	cbb5be6d57	feat(engine): Implements predicate mapping for DataObjScan (#17621 ) This is an initial implementation of `buildLogsPredicate(expr physical.Expression) (dataobj.LogsPredicate, error)`. While there have been discussions about bypassing this approach in favor of directly using `dataset.Reader` and `dataset.Predicate`, this PR provides essential supported functionality to unblock progress in the interim. Specifically, it handles mapping common timestamp and metadata equality predicates. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	8 months ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608 ) Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	0ea1097c8f	chore(engine): Use configured storage bucket in execution context (#17583 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago

1 2

63 Commits (dfb60dcaf094dc5fc2c74be2b2e3827ed3de1db1)