Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Christian Haudum b316d52612 chore(engine): Use shard information from query frontend in the new query engine (#17792) ... **What this PR does / why we need it**: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. **Note, this is only an interim solution used for validating and testing.** By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. **Shard size considerations** While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. **How does sharding with data objects in this PR work?** The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>		4 days ago
..
dataobjscan.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	4 days ago
dataobjscan_predicate.go	chore(engine): Map line filter expressions to data object predicates (#18102)	2 weeks ago
dataobjscan_predicate_test.go	chore(engine): Map line filter expressions to data object predicates (#18102)	2 weeks ago
dataobjscan_test.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	4 days ago
executor.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	4 days ago
executor_test.go	chore(engine): provide basic implementation of DataObjScan (#17568)	2 months ago
expressions.go	chore(engine): Filter by unknown column (#18131)	1 week ago
expressions_test.go	chore(engine): Filter by unknown column (#18131)	1 week ago
filter.go	chore(engine): Filter by unknown column (#18131)	1 week ago
filter_test.go	chore(engine): Improve type system of the new query engine (#17456)	2 months ago
functions.go	chore(engine): Add framework for function evaluation (#17374)	2 months ago
limit.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	4 days ago
limit_test.go	chore(dataobj,testing,executor): pipeline equality testing (#17311)	2 months ago
pipeline.go	chore(engine): Add framework for query executor (#17260)	2 months ago
pipeline_test.go	chore(engine): Collect pipeline records into Loki result (#17447)	2 months ago
pipeline_utils.go	chore(engine): Collect pipeline records into Loki result (#17447)	2 months ago
pipeline_utils_test.go	chore(engine): provide basic implementation of DataObjScan (#17568)	2 months ago
project.go	chore(engine): Improve type system of the new query engine (#17456)	2 months ago
project_test.go	chore(engine): Improve type system of the new query engine (#17456)	2 months ago
range_aggregation.go	chore(engine): adds vector aggregation planning and execution nodes (#18100)	1 week ago
range_aggregation_test.go	chore(engine): adds vector aggregation planning and execution nodes (#18100)	1 week ago
sortmerge.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	4 days ago
sortmerge_test.go	chore(engine): Filter by unknown column (#18131)	1 week ago
util_test.go	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627)	2 months ago
vector_aggregate.go	chore(engine): adds vector aggregation planning and execution nodes (#18100)	1 week ago
vector_aggregate_test.go	chore(engine): adds vector aggregation planning and execution nodes (#18100)	1 week ago