Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Salva Corts 52a3f16039 Flag categorize labels on streams response (#10419) ... We recently introduced support for ingesting and querying structured metadata in Loki. This adds a new dimension to Loki's labels since now we arguably have three categories of labels: _stream_, _structured metadata_, and _parsed_ labels. Depending on the origin of the labels, they should be used in LogQL expressions differently to achieve optimal performance. _stream_ labels should be added to stream matchers, _structured metadata_ labels should be used in a filter expression before any parsing expression, and _parsed_ labels should be placed after the parser expression extracting them. The Grafana UI has a hard time dealing with this same problem. Before https://github.com/grafana/grafana/pull/73955, the filtering functionality in Grafana was broken since it was not able to distinguish between _stream_ and _structured metadata_ labels. Also, as soon as a parser expression was added to the query, filters added by Grafana would be appended to the end of the query regardless of the label category. The PR above implements a workaround for this problem but needs a better API on Loki's end to mitigate all corner cases. Loki currently returns the following JSON for log queries: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "info", "traceID": "68810cf0c94bfcca" }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n" ], ... }, { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", "level": "debug", "traceID": "a7116cj54c4bjz8s" }, "values": [ [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n" ], ... }, ... ``` As can be seen, there is no way we can distinguish the category of each label. This PR introduces a new flag `X-Loki-Response-Encoding-Flags: categorize-labels` that makes Loki return categorized labels as follows: ```json ... { "stream": { "cluster": "us-central", "container": "query-frontend", "namespace": "loki", }, "values": [ [ "1693996529000222496", "1693996529000222496 aaaaaaaaa.....\n", { "structuredMetadata": { "traceID": "68810cf0c94bfcca" }, "parsed": { "level": "info" } } ], [ "1693996529000222497", "1693996529000222497 bbbbbbbbb.....\n", { "structuredMetadata": { "traceID": "a7116cj54c4bjz8s" }, "parsed": { "level": "debug" } } ], ... }, ... ``` Note that this PR only supports log queries, not metric queries. From a UX perspective, being able to categorize labels in metric queries doesn't have any benefit yet. Having said that, supporting this for metric queries would require some minor refactoring on top of what has been implemented here. If we decide to do that, I think we should do it on a separate PR to avoid making this PR even larger. I also decided to leave out support for Tail queries to avoid making this PR even larger. Once this one gets merged, we can work to support tailing. --- **Note to reviewers** This PR is huge since we need to forward categorized all over the codebase (from parsing logs all the way to marshaling), fortunately, many of the changes come from updating tests and refactoring iterators. Tested out in a dev cell with query `'{stream="stdout"} | label_format new="text"`. - Without the new flag: ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED { "data": { "result": [ { "stream": { "new": "text", "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n" ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n" ], ... ``` - With the new flag ``` $ http http://127.0.0.1:3100/loki/api/v1/query_range\?direction\=BACKWARD\&end\=1693996529322486000\&limit\=30\&query\=%7Bstream%3D%22stdout%22%7D+%7C+label_format+new%3D%22text%22\&start\=1693992929322486000 X-Scope-Orgid:REDACTED X-Loki-Response-Encoding-Flags:categorize-labels { "data": { "encodingFlags": [ "categorize-labels" ], "result": [ { "stream": { "pod": "loki-canary-986bd6f4b-xqmb7", "stream": "stdout" }, "values": [ [ "1693996529000222496", "1693996529000222496 pppppppppppp...\n", { "parsed": { "new": "text" } } ], [ "1693996528499160852", "1693996528499160852 pppppppppppp...\n", { "parsed": { "new": "text" } } ], ... ```		2 years ago
..
analytics	Update dskit to 7b512eb. (#10927)	2 years ago
bloomcompactor	Unify ring managers across components (#10931)	2 years ago
bloomgateway	Fix flaky bloomgateway test case (#11001)	2 years ago
canary	clients,pkg: use http.NewRequestWithContext (#10616)	2 years ago
chunkenc	Flag categorize labels on streams response (#10419)	2 years ago
compactor	Flag categorize labels on streams response (#10419)	2 years ago
configs	Upgrade dskit to use packages migrated from weaveworks/common (#10164)	2 years ago
distributor	ingestion: native otlp ingestion support (#10727)	2 years ago
ingester	Add basic structure of bloom gateways (#10782)	2 years ago
iter	Flag categorize labels on streams response (#10419)	2 years ago
logcli	chore(storage): Restructure shipper packages (#10729)	2 years ago
loghttp	Flag categorize labels on streams response (#10419)	2 years ago
logproto	Adjust bloom gateway request structure (#10926)	2 years ago
logql	Flag categorize labels on streams response (#10419)	2 years ago
logqlanalyzer	fixed logql-analyzer (#8258)	3 years ago
logqlmodel	renamed `non indexed labels` feature to `structured metadata` (#10432)	2 years ago
loki	Flag categorize labels on streams response (#10419)	2 years ago
lokifrontend	Turn frontend Tripperware into a Middleware. (#10688)	2 years ago
push	Flag categorize labels on streams response (#10419)	2 years ago
querier	Flag categorize labels on streams response (#10419)	2 years ago
queue	Introduce worker queue in bloom gateway (#10976)	2 years ago
ruler	ruler: Support `writeNotify` on Loki ruler (#10906)	2 years ago
runtime	Loki: Implement initial phase of `limited_log_push_errors` (#9556)	3 years ago
scheduler	Introduce worker queue in bloom gateway (#10976)	2 years ago
server	Upgrade dskit to use packages migrated from weaveworks/common (#10164)	2 years ago
sizing	fix xss vulnerability (#10559)	2 years ago
storage	Flag categorize labels on streams response (#10419)	2 years ago
tracing	Promtail: Add basic tracing support (#7414)	3 years ago
util	Flag categorize labels on streams response (#10419)	2 years ago
validation	Add basic structure of bloom gateways (#10782)	2 years ago