Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Christian Haudum b316d52612 chore(engine): Use shard information from query frontend in the new query engine (#17792) ... **What this PR does / why we need it**: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. **Note, this is only an interim solution used for validating and testing.** By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. **Shard size considerations** While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. **How does sharding with data objects in this PR work?** The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>		7 months ago
..
metastore.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	7 months ago
metastore.pb.go	chore: Emit metastore event to kafka after object build (#17842)	8 months ago
metastore.proto	chore: Emit metastore event to kafka after object build (#17842)	8 months ago
metastore_test.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	7 months ago
metrics.go	feat: refactors kafka/client to improve metric collection (#17308)	9 months ago
object.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	7 months ago
object_test.go	chore(engine): Use shard information from query frontend in the new query engine (#17792)	7 months ago
updater.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	8 months ago