Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Christian Haudum b316d52612 chore(engine): Use shard information from query frontend in the new query engine (#17792) ... **What this PR does / why we need it**: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. **Note, this is only an interim solution used for validating and testing.** By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. **Shard size considerations** While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. **How does sharding with data objects in this PR work?** The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>		11 months ago
..
config	refactor(dataobj): Consolidate configuration and add querier support (#16144)	1 year ago
consumer	chore: Emit metastore event to kafka after object build (#17842)	1 year ago
explorer	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
internal	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976)	12 months ago
metastore	chore(engine): Use shard information from query frontend in the new query engine (#17792)	11 months ago
querier	chore(engine): Use shard information from query frontend in the new query engine (#17792)	11 months ago
sections	chore(dataobj): Add columnar reading APIs to logs and streams sections (#17976)	12 months ago
tools	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
uploader	chore(dataobj): Refactor processor & builder to separate concerns (#16055)	1 year ago
builder.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
dataobj.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
decoder.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
decoder_metadata.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
encoder.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
encoder_test.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
metrics.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
range_reader.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
section.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago
section_reader.go	refactor(dataobj): invert dependency between dataobj and sections (#17762)	1 year ago