apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ivan Kalita	08e3c4385f	feat(metastore): shard sections queries over index files (#20134 ) Running metastore queriesin a distributed manner using query engine workers: - Build a distributed metastore plan from GetIndexes and execute it via the v2 scheduler/worker pipeline - Split the request into per-index PointersScan tasks, then fan-in and CollectSections to produce final section descriptors - Add physical plan + protobuf support for Merge/PointersScan, improve tracing/cleanup, and add coverage for planner/workflow/proto roundtrips	4 days ago
Ivan Kalita	9be17c7a42	feat(metastore): metastore DI (#20253 ) This commit fixes a problem that happens when we instantiate 2 metastores in a single process. Even though this can't happen at the moment it will be the case once we introduce distributed metastore queries - when running both schedulers and workers in the same process all of them will need an instance of metastore. Each instance (before this PR) internally creates and registers metastore metrics which causes panic due to exactly the same metrics were already registered by the same registerer. I attempt to fix this problem via creating and registering metastore metrics only once.	1 week ago
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234 ) Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	1 week ago
Ivan Kalita	3a74fe71fe	feat(engine): delegate metastore queries to engine (#20189 ) This is a prerequisite to distributed metastore queries execution. I'm updating the catalog such a way to receive a callback from the caller to call metastore. This is needed to separate the execution of the metastore queries that will be handled by the engine and not by the catalog.	1 week ago
benclive	0dea806656	fix: Respect categorizeLabels encoding flag in v2 queries (#20098 )	3 weeks ago
Ashwanth	25c8827f56	chore(xcap): export captures as traces (#19880 )	1 month ago
Ashwanth	11b1dcb2a3	chore(engine): introduce execution capture (#19821 )	1 month ago
Ashwanth	5d961a4cc3	chore(topk): improvements to reduce alloc bytes and alloc space (#19660 )	2 months ago
Robert Fratto	75e5193a97	chore(engine): expose new scheduler and worker (#19633 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Stas Spiridonov	160dc2c493	chore: removed arrow-go allocators/retain/release (#19569 )	2 months ago
Robert Fratto	c1b010567c	chore(engine): print stringified planning durations (#19155 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	2 months ago
Ashwanth	72cb649623	chore: introduce mw to route requests between v1 and v2 engines (#19452 )	2 months ago
Ashwanth	c24675f854	chore: adds separate v2 engine config section (#19423 )	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384 ) This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241 )	3 months ago
Robert Fratto	f6091a67d1	chore(engine): move to toggleable section prefetching (#19142 ) dataset.readerDownloader was originally introduced in #16429, an attempt to balance peak memory usage of reading a section with read times by downloading a configurable size of pages in advance. In practice, each roundtrip to object storage adds too much of a latency hit, and we've started to set the cache limit high enough to ensure that each reader only needs a single prefetch. Given what we've found, it no longer makes sense to control peak memory usage via the prefetch size. Other options, such as downloading directly to disk, may be explored in the future. In the meantime, this PR removes the ability to specify a cache size. All non-pruned pages will be bulk requested using the range reader (#19067) on the first read call. Pages which have left the potential read window will continue to be eagerly removed for garbage collection. However, we don't want to prefetch when the dataset is entirely in memory, which is the case when the logs section builder is performing k-way merge over in-memory sections. To lower the memory usage of builders, prefetching is configurable. For this initial PR, prefetching is only disabled for the logs section builder; all other reads force prefetching. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Christian Haudum	65de5f95db	chore(engine): Simplify `Pipeline` interface (#19110 ) This PR replaces the two calls to retrieve a value from the pipeline with a single one. To do this, the `Read()` method now returns two arguments: `arrow.Record` and `error`, rather than just the error. If the error is not `nil`, the first argument (arrow.Record) must be ignored by the caller. The `Value()` method is not needed any more and was removed. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Christian Haudum	f50c87fce2	feat(engine): Prefetch inputs of Merge pipeline concurrently (#19039 ) This PR adds prefetching to the inputs of the Merge node. The Merge is a pipeline that takes N inputs and sequentially consumes each one of them. It completely exhausts an input before moving to the next one. However the inputs do not prefetch any data, so calling Read() may wait on i/o (fetching data from object storage, decoding, etc). Prefetching data allows to fetch the next batch which the caller node is processing the current batch. This PR changes the behaviour of the Merge so inputs are prefetched. Since inputs of the Merge pipeline are read sequentially, the current input is always prefetched. The setting `-querier.engine.merge-prefetch-count` controls how many of the next inputs are prefetched concurrently. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	4 months ago
benclive	84efa59a06	chore: Fix init of engine with metastore (#19064 )	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959 )	4 months ago
Robert Fratto	aa66baec97	chore(logqlmodel/stats): print new engine summary in metrics.go (#19052 )	4 months ago
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 months ago
Ashwanth	827b1f2df4	chore(engine): add support for range metric queries (#18892 )	4 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555 )	5 months ago
Robert Fratto	a3380f6480	chore(engine): add tracing instrumentation (#18648 )	5 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577 )	5 months ago
Ashwanth	da4e59f8c6	chore(dataobj): store sort order information in section metadata (#18499 )	5 months ago
Christian Haudum	aa71369f24	chore(engine): Add warning to response when query was executed with new engine (#18534 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	31fcc816e4	chore(engine): Minor logging cleanup in new query engine (#18513 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
benclive	517c317c97	chore: Integrate indexes with the new query engine (#18427 )	5 months ago
Christian Haudum	cea56469b9	chore(engine): Pass context to Pipeline's Read() function (#18459 ) This change unifies how the request context is passed through the execution of pipelines. Instead of storing the context on pipelines, pass them as argument to the `Read()` function. This change alters the interface method of `Pipeline` from `Read() bool` to `Read(context.Context) bool`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	5 months ago
Christian Haudum	d9f7d785ff	chore(engine): Sort entries in dataobj sections by timestamp descending (#18367 ) Sorting sections by `timestamp DESC` first and `streamID ASC` second allows for more efficient reading (by section) for BACKWARD log queries. - This change breaks FORWARD log queries. Which is ok for the time being. We will add support later again when columnar reading of the dataobj is introduced. - This change renders existing dataobj in storage incompatible. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Ashwanth	646ebe81d8	chore(engine): adds AST conversion and result builders for metric queries (#18166 )	6 months ago
Ashwanth	5f3300a084	chore(bench_test): correctly integrate new query engine + misc changes (#18172 )	6 months ago
Christian Haudum	b316d52612	chore(engine): Use shard information from query frontend in the new query engine (#17792 ) What this PR does / why we need it: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. Note, this is only an interim solution used for validating and testing. By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. Shard size considerations While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. How does sharding with data objects in this PR work? The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	6 months ago
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
Christian Haudum	1629181f9c	chore(engine): Unify name and type of builtin columns (#17608 ) Names and types used when creating and collecting records need to match. * `Entry.Line` maps to `builtin.message` of type `arrow.BinaryTypes.String` * `Entry.Timestamp` maps to `builin.timestamp` of type `arrow.FixedWidthTypes.Timestamp_ns` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	0ea1097c8f	chore(engine): Use configured storage bucket in execution context (#17583 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	94b1d2d5ce	chore(engine): Improve type system of the new query engine (#17456 ) What this PR does / why we need it: The original type system for the new engine was introduced before adding Apache Arrow and with that an additional type system. One aspect of the new type system of this PR is that the logical types of the engine are aligned/interchangeable with the arrow logical types. ```go type Type uint8 const ( NULL = Type(arrow.NULL) BOOL = Type(arrow.BOOL) STRING = Type(arrow.STRING) INT64 = Type(arrow.INT64) FLOAT64 = Type(arrow.FLOAT64) ) ``` The actual data types of Loki (`Boolean`, `String`, `Integer`, `Float`, `Timestamp`, `Duration`, and `Bytes`) are implemented as wrapper around a logical type. Different data types can use the same logical type as representation, such as `Integer`, `Timestamp` and `Duration` both use `INT64`. Since Arrow array types are bound to logical types, and one logical type can have multiple Loki data types, it is required to "annotate" Fields of an arrow.Record using its Metadata, similar to how it is done for distinguishing builtin columns from label/metadata/parsed columns. Additionally, the PR contains a mapping between Loki types and Arrow types. The `Literal` has been updated to use the new types. The old system also used UINT64 as representation of the `Timestamp` type, which is incorrect, since both LogQL and the DataObjects allow timestamps before unix epoch. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Ashwanth	009600cb51	chore(dataobj): order predicates based on row selectivity and cost of reading (#17376 )	8 months ago
Christian Haudum	3c72560c71	chore(engine): Add basic metrics for next generation execution engine (#17475 ) This PR adds basic metrics for observe the duration of the different phases of the new execution engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447 ) This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	8 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032 ) This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	0a3230f457	chore(ast): Update WalkFn signature to return bool (#16989 ) The boolean return value indicates whether walking/traversing should be continued with child nodes. This PR does not change any existing traversal behaviour. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	f79906ca75	chore(engine): Add utility function `canExecuteWithNewEngine` (#16892 ) The function determines whether a query can be executed by the new execution engine. In the first step, the new execution engine will only be able to execute limited/filter queries without any transformation stages. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago

9 Commits (main)