324 Commits (8eac851962a23d39c33339146e204d257174463c)

Author	SHA1	Message	Date
Danny Kopping	e2ac2d50e2	Changes from linter (#8769) ... **What this PR does / why we need it**: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Callum Styan	5a85f6647e	Add initial implementation of per-query limits (#8727) ... **What this PR does / why we need it**: Sometimes we want to limit the impact of a single query by imposing limits that are stricter than the current tenant limit. E.g. the maximum query length could be seven days but based on the query or an admins decision a query should just have a maximum length of one day. This is where per-request limits come into play. They are passed via the `X-Loki-Query-Limit` header and extracted into the requests context. It is the responsibility of the operator or admin that the header is valid. **Which issue(s) this PR fixes**: Fixes #8762 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Travis Patterson	9ac33e216e	Reimplement Unpack in terms of jsonparser rather than jsoniter (#8739) ... This PR reimplements the functionality of `Unpack` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` name old time/op new time/op delta _Parser/unpack/no_labels_hints-8 900ns ± 4% 309ns ± 3% -65.62% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 1.06µs ±12% 0.32µs ± 2% -69.72% (p=0.000 n=9+10) _Parser/unpack-not_json_line/no_labels_hints-8 37.5ns ± 8% 10.6ns ± 3% -71.70% (p=0.000 n=9+9) _Parser/unpack-not_json_line/labels_hints-8 39.6ns ±11% 10.3ns ± 2% -74.00% (p=0.000 n=10+9) name old alloc/op new alloc/op delta _Parser/unpack/no_labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00B 0.00B ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta _Parser/unpack/no_labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00 0.00 ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00 0.00 ~ (all equal) ```	2 years ago
Travis Patterson	9c6e509a6a	Reimplement JsonExpressionParser in terms of jsonparser (#8734) ... This PR reimplements the functionality of `JSONExpressionParser` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` benchstat json_expression_old.txt json_expression_new.txt name old time/op new time/op delta JsonExpressionParser/json-expression-8 4.15µs ± 4% 0.82µs ± 3% -80.17% (p=0.000 n=10+10) name old alloc/op new alloc/op delta JsonExpressionParser/json-expression-8 1.07kB ± 0% 0.02kB ± 0% -97.76% (p=0.000 n=10+10) name old allocs/op new allocs/op delta JsonExpressionParser/json-expression-8 72.0 ± 0% 2.0 ± 0% -97.22% (p=0.000 n=10+10) ```	2 years ago
Travis Patterson	7f42137584	Use Line Filters in StringLabelFilter (#8659) ... We've done a ton of work to optimize regexes in line filters but nothing in label filters. To avoid duplicating the work or reworking all of our filter types, this PR creates a new label filter that has an optimized line filter. It uses that against the label value to determine if they match. Note on Benchmarks: the input string for this is 1114195 bytes long. The `not` test tells us that our approach is quite a lot slower when the literal pattern isn't present in the input. The size of the input really emphasizes it. Benchmarks: ``` name old time/op new time/op delta LineLabelFilters/foo-8 50.6ns ± 9% 14.3ns ± 1% -71.67% (p=0.000 n=10+8) LineLabelFilters/not-8 15.7ns ± 1% 64165.6ns ± 2% +407766.77% (p=0.000 n=10+10) LineLabelFilters/(foo)-8 58.4ns ± 1% 14.8ns ±10% -74.68% (p=0.000 n=9+9) LineLabelFilters/(foo|ba)-8 64.9ns ± 0% 18.2ns ± 2% -71.90% (p=0.000 n=7+9) LineLabelFilters/(foo|ba|ar)-8 74.8ns ± 7% 21.4ns ± 2% -71.42% (p=0.000 n=10+8) LineLabelFilters/(foo|(ba|ar))-8 65.1ns ± 1% 18.1ns ± 2% -72.22% (p=0.000 n=8+8) LineLabelFilters/foo.*-8 882ns ± 4% 15ns ±11% -98.30% (p=0.000 n=9+10) LineLabelFilters/.*foo-8 11.5ns ± 8% 14.9ns ± 7% +30.39% (p=0.000 n=10+10) LineLabelFilters/.*foo.*-8 5.46µs ± 7% 0.01µs ± 3% -99.74% (p=0.000 n=10+9) LineLabelFilters/(.*)(foo).*-8 6.26µs ± 7% 0.01µs ± 2% -99.77% (p=0.000 n=10+9) LineLabelFilters/(foo.*|.*ba)-8 5.58µs ± 3% 0.02µs ± 8% -99.67% (p=0.000 n=9+10) LineLabelFilters/(foo.*|.*bar.*)-8 7.02µs ± 7% 0.02µs ± 8% -99.73% (p=0.000 n=10+10) LineLabelFilters/.*foo.*|bar-8 5.53µs ± 6% 0.02µs ± 2% -99.67% (p=0.000 n=10+8) LineLabelFilters/.*foo|bar-8 3.73µs ± 6% 0.02µs ± 3% -99.52% (p=0.000 n=10+9) LineLabelFilters/(?:.*foo.*|bar)-8 5.43µs ± 8% 0.02µs ± 8% -99.65% (p=0.000 n=10+10) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 5.71µs ± 3% 0.02µs ±10% -99.68% (p=0.000 n=9+9) LineLabelFilters/.*foo.*|bar|buzz-8 5.34µs ± 2% 0.02µs ± 3% -99.65% (p=0.000 n=10+9) LineLabelFilters/.*foo.*|bar|uzz-8 5.36µs ± 1% 0.02µs ± 3% -99.60% (p=0.000 n=8+8) LineLabelFilters/foo|bar|b|buzz|zz-8 68.8ns ± 8% 22.6ns ± 7% -67.19% (p=0.000 n=10+10) LineLabelFilters/f|foo|foobar-8 77.1ns ± 1% 18.6ns ± 9% -75.90% (p=0.000 n=8+10) LineLabelFilters/f.*|foobar.*|.*buzz-8 5.64µs ± 7% 0.02µs ±10% -99.60% (p=0.000 n=10+10) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 6.29µs ± 8% 0.02µs ±10% -99.65% (p=0.000 n=10+10) LineLabelFilters/.*-8 963ns ± 8% 2ns ±14% -99.77% (p=0.000 n=10+10) LineLabelFilters/.*|.*-8 4.46µs ± 8% 0.00µs ± 8% -99.95% (p=0.000 n=10+10) LineLabelFilters/.*||||-8 2.99µs ± 1% 0.00µs ±16% -99.93% (p=0.000 n=8+10) LineLabelFilters/#00-8 43.5ns ± 8% 2.1ns ± 1% -95.25% (p=0.000 n=10+8) LineLabelFilters/(?i)foo-8 71.7ns ± 6% 13.1ns ± 1% -81.73% (p=0.000 n=10+8) LineLabelFilters/(?i)界-8 16.5ns ± 8% 150.2ns ± 8% +809.55% (p=0.000 n=10+10) LineLabelFilters/(?i)ïB-8 23.8ns ± 1% 14748979.8ns ± 7% +62074730.81% (p=0.000 n=8+10) LineLabelFilters/(?:)foo|fatal|exception-8 185ns ± 2% 21ns ± 2% -88.38% (p=0.000 n=9+8) LineLabelFilters/(?i)foo|fatal|exception-8 78.4ns ± 3% 20.2ns ± 2% -74.23% (p=0.000 n=9+9) LineLabelFilters/(?i)f|foo|foobar-8 94.3ns ±16% 15.4ns ± 9% -83.66% (p=0.000 n=10+10) LineLabelFilters/(?i)f|fatal|e.*-8 65.7ns ± 5% 18.2ns ± 3% -72.24% (p=0.000 n=9+8) LineLabelFilters/(?i).*foo.*-8 6.04µs ± 8% 0.01µs ± 2% -99.78% (p=0.000 n=10+10) name old alloc/op new alloc/op delta LineLabelFilters/foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/not-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*|.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*||||-8 0.00B 0.00B ~ (all equal) LineLabelFilters/#00-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)界-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta LineLabelFilters/foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/not-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00 0.00 ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*|.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*||||-8 0.00 0.00 ~ (all equal) LineLabelFilters/#00-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)界-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00 0.00 ~ (all equal) ```	2 years ago
Callum Styan	9a2a038f43	Allow passing of context to query related limits functions (#8689) ... In this PR we're allowing for passing of a `context.Context` via the Limits interfaces (some of which are new, to clean up hardcoding/embedding of `validation.Overrides`) This is based on work/ideas by @jeschkies . Fixes #8694 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Travis Patterson	c42a0ba868	Optimize .+ regex line filter (#8646) ... Interpolated variables from Grafana dashboards may result in the regex matcher `.+`. Filters with `+` are already being rewritten to be non greedy but there is a further optimization to make replace the regex altogether with a filter that just matchers `len(line) > 0`	2 years ago
Christian Haudum	d4a6b159fa	chore: Cleanup label extraction expression (#8596) ... The aim of this is to remove redundancy from the parser grammar and make the expression name more descriptive. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Michel Hollands	4f8d3240f6	Use 0.28.1 build image and update go and alpine versions (#8583) ... **What this PR does / why we need it**: - Use the 0.28.1 build image - Update Go version to 1.20.1 - Use Alpine 3.16.4 - Fix linter issues due to updated govet in 1.20.1 - Modify images to not create `/etc/nsswitch.conf` file anymore (available by default on Alpine 3.16.4) - Remove impossible test cases for `TestMemChunk_IteratorBounds`(see https://github.com/grafana/loki/pull/8591#discussion_r1115020111 for context) Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	2 years ago
Bayan Taani	68a9fd6c0a	Add logfmt selective label extraction (#6675) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR introduces extracting labels from a log line in `logfmt`, with an extra option to rename them. For example, this query: ``` {app=”foo”} | logfmt msg,status ``` will extract the labels `msg` and `status` from the following logfmt line: ``` level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..." ``` With the following results: ``` msg="final error sending batch" status=”400” ``` -------------- Another possible scenario with label renaming: ``` {app=”foo”} | logfmt message="msg", status ``` That produces the following results: ``` message="final error sending batch" status=”400” ``` **Which issue(s) this PR fixes**: Fixes #3355 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
李国忠	b78ed9d106	[RFR] fix vector() function produces wrong timestamp on instant query (#8370)	2 years ago
Tristan Morgan	2e11ff00eb	Replace deprecated inet.af/netaddr (#8560) ... This removes the deprecated `inet.af/netaddr` that imports `go4.org/unsafe/assume-no-moving-gc` and panic when it's not updated. Fixes #8399 and replaces #8495	2 years ago
Garrett	433d5bf913	fix panics when cloning a special query (#8531) ... Signed-off-by: garrettlish <garrett.li.sh@gmail.com>	2 years ago
Owen Diehl	7871ddd12b	More precise sharding logic for `on|ignoring` (#8461)	2 years ago
Christian Haudum	96d5227532	Fix parsing of vector expression (#8448) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
李国忠	322783e3d8	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	2 years ago
Ed Welch	35510ba4eb	Loki: only log "executing query" once per query in the frontend (#8337) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Kaviraj Kanagaraj	807aa459d4	logql: Support `urlencode` and `urldecode` template functions (#8271) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Dylan Guedes	797d4bce45	Loki: Log hashed queries emitted by `metrics.go` (#8244) ... - What this PR does / why we need it: * Adds a new `query_hash` label pair to queries logged by `metrics.go` and our query engine. As the name suggests, `query_hash` is just the hashed version of the query being logged. * Especially useful when grouping results/behavior of different queries. Ex, you can easily find specific query patterns with `sum by(query_hash) (...)` .	2 years ago
Ed Welch	c8760f31e2	Loki: Remove validation of matchers when performing label values lookup. (#8224) ... the curren validation fails on != style matchers when they are the only matchers present, requiring at least one `=` matcher. However for looking up label values in TSDB, by design it uses a `desired_label != ""` style query, as such it should be allowed to also include these as matchers. Additional matchers on this Label Value queries typically only exist with LBAC headers adding additional labels. Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	a7ea9c1422	Revert "fix(logql): remove __error_details__ label on drop __error__ or other way around" (#8200) ... Reverts grafana/loki#8150 I was talking with @adityacs about reverting this PR and found the following: Looking at why tests don’t keep `__error_details__` when `__error__` is dropped and I found this line of code: ```go // LabelsResult returns the LabelsResult from the builder. // No grouping is applied and the cache is used when possible. func (b *LabelsBuilder) LabelsResult() LabelsResult { // unchanged path. if len(b.del) == 0 && len(b.add) == 0 && b.err == "" { return b.currentResult } return b.toResult(b.labels()) }``` https://github.com/grafana/loki/blob/main/pkg/logql/log/labels.go#L314 We only add err details when err isn’t empty anyway, so I guess we’ll revert for simplicity but just keep the same functionality	2 years ago
Aditya C S	cf23194d05	fix(logql): remove __error_details__ label on drop __error__ or other way around (#8150)	2 years ago
Danny Kopping	b9a9cdf2f7	Fixing test regression (#8130) ... Regression introduced by #7723 which was merged around the same time Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Cyril Tovena	2861c0036f	Improve the JSON parser performance. (#7723) ... Sorry I couldn't help myself and worked a bit on the JSON parser by using another library that allows way less allocations. On top of this, the parser code is also much simpler to understand. I realize that we could possibly also intern/cache (limited obviously) extracted JSON value per key. I would do it per key because otherwise, a high cardinality key can take over the intern/cache which will make it worthless. Happy to walk someone through the idea, I think this would be a big jump too. Would be great to run and test this in our dev environment using some JSON queries before releasing it. ``` ❯ benchstat before.txt after.txt name old time/op new time/op delta _Parser/json/no_labels_hints-10 2.53µs ± 0% 2.16µs ± 2% -14.59% (p=0.008 n=5+5) _Parser/json/labels_hints-10 1.93µs ± 0% 1.47µs ± 0% -23.96% (p=0.008 n=5+5) name old alloc/op new alloc/op delta _Parser/json/no_labels_hints-10 656B ± 0% 280B ± 0% -57.32% (p=0.008 n=5+5) _Parser/json/labels_hints-10 512B ± 0% 176B ± 0% -65.62% (p=0.008 n=5+5) name old allocs/op new allocs/op delta _Parser/json/no_labels_hints-10 46.0 ± 0% 18.0 ± 0% -60.87% (p=0.008 n=5+5) _Parser/json/labels_hints-10 39.0 ± 0% 12.0 ± 0% -69.23% (p=0.008 n=5+5) ```	2 years ago
Aditya C S	8df5803d9a	feat(logql): Support drop labels in logql pipeline (#7975) ... This PR introduces `drop` stage in logql pipeline. Fixes #7870, Fixes #7368	2 years ago
Aditya C S	a08a7324f4	fix(logql): fix caseinsensitive search in filter (#8037) ... **What this PR does / why we need it**: Fix caseinsensitive search in filter **Which issue(s) this PR fixes**: Fixes #7904 #7837 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
李国忠	b4440911e0	[new feature] logql:support sort and sort_desc (#7989) ... **What this PR does / why we need it**: sort() sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order. sort_desc() Same as sort, but sorts in descending order. ex: `sort(rate(({app=~"foo|bar"} |~".+bar")[1m])) without (app) ` prometheud doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#sort **Which issue(s) this PR fixes**: Fixes #7933 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Michel Hollands	59eff5ca58	Add count to template functions (#7951) ... **What this PR does / why we need it**: Add a `count` template function to count occurrences of a substring. For example if `label="abc abd abc"` ``` {{ .label | count "abc" }} ``` would return 2. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Christian Haudum	089ec1b05f	Fix various linter errors ... These changes have been generated using `make lint`. ```console $ golangci-lint --version golangci-lint has version 1.50.0 built from 704109c6 on 2022-10-04T10:25:07Z ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	a4f306399a	Add store & cache download statistics (#7982)	2 years ago
Travis Patterson	209b281593	Fix min and max aggregations when using instant queries. (#7957) ... Instant queries use the new `streamRangeVectorIterator` which has a new set of `aggregators`. Before, aggregations were based on the first point in the range to be aggregated. With `streamRangeVectorIterator`, points are instead treated one at a time. The aggregations were simply checking whether the value was less than the min or greater than the max without any initialization. This PR fixes these aggregations by initializing `min` or `max` to `math.Nan` so the first point is always considered.	2 years ago
Michel Hollands	5fe9c964db	Fix validation for pattern and rexexp parsers to they don't panic in query-frontend (#7926) ... Running the query `topk(10,sum by(namespace)(count_over_time({application="nginx", site!="eu-west-1-dev"} |= "/artifactory/" != "api" != "binarystore" | regexp `` [1d])))` causes a panic. This PR adds validation for regexp and pattern parsers at parse time. If they don't have a (valid) parameter an error will be returned. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	2 years ago
Kaviraj Kanagaraj	cbd6ec15ce	feat(logql): Supporting prettifying LogQL expressions (#7906) ... **What this PR does / why we need it**: Changes 1. Added `Pretty()` method to `Expr` interface. So that every expression would know how to render themselfs as pretty :) 2. Implemented `Pretty()` for every single LogQL expressions 3. Exposed `/api/v1/format_query` endpoint that takes `query` argument and returns formatted version. 4. Integerated into `logcli`. echo `'<query>' | logcli fmt` would format the query. Why? 1. Readability, helpful in debugging and **more importantly, to understand the execution flow** example **before** ``` quantile_over_time(0.99,{container="ingress-nginx",service="hosted-grafana"}| json| unwrap response_latency_seconds| __error__=""[1m]) by (cluster) ``` ``` sum(rate({job="loki", namespace="loki", cluster="loki-dev-us"} |= "err" [5m])) + sum(rate({job="loki-dev", namespace="loki", cluster="loki-dev-eu"}|logfmt | level != "info" [5m])) / sum(rate({job="loki-prod", namespace="loki", cluster="loki-prod-us"} |logfmt | level="error"[5m])) ``` ``` label_replace(rate({job="api-server",service="a:c"}|= "err" [5m]), "foo", "$1", "service", "(.*):.*") ``` **after** ``` quantile_over_time( 0.99, {container="ingress-nginx", service="hosted-grafana"} | json | unwrap response_latency_seconds | __error__="" [1m] ) by (cluster) ``` ``` sum( rate( {job="loki", namespace="loki", cluster="loki-dev-us"} |= "err" [5m] ) ) + sum( rate( {job="loki-dev", namespace="loki", cluster="loki-dev-eu"} | logfmt | level!="info" [5m] ) ) / sum( rate( {job="loki-prod", namespace="loki", cluster="loki-prod-us"} | logfmt | level="error" [5m] ) ) ``` ``` label_replace( rate({job="api-server", service="a:c"} |= "err"[5m]), "foo", "$1", "service", "(.*):.*" ) ``` You can find more examples in the `prettier_test.go` Future plans * Integerate into LogQL analyzer * Integrate into Grafana UI. **Which issue(s) this PR fixes**: Fixes # NA **Special notes for your reviewer**: This whole idea was inspired from last [PromCon lighting talk](https://youtu.be/pjkWzDVxWk4?t=24469) **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Simon <simon@swine.de>	2 years ago
Susana Ferreira	f93b91bfb5	Add configuration documentation generation tool (#7916) ... **What this PR does / why we need it**: Add a tool to generate configuration flags documentation based on the flags properties defined on registration on the code. This tool is based on the [Mimir doc generation tool](https://github.com/grafana/mimir/tree/main/tools/doc-generator) and adapted according to Loki configuration specifications. Prior to this PR, the configuration flags documentation was dispersed across two sources: * [_index.md](5550cd65ec/docs/sources/configuration/_index.md) * configuration flags registration in the code This meant that there was no single source of truth. In this PR, the previous `_index.md` file is replaced with the new file generated by the tool. The next step includes adding a CI step that validates if the _index.md file was generated according to the flags settings. This will be done in a follow-up PR. **NOTE:** this is not a documentation update PR. Apart from some minor typo fixes, the documentation changes on the code, were copied from the `_index.md` file. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki-private/issues/83 **Special notes for your reviewer**: Files: * [docs/sources/configuration/index.template](5550cd65ec/docs/sources/configuration/index.template): template used to generate the final configuration file * [/docs/sources/configuration/_index.md](c32e5d0acb/docs/sources/configuration/_index.md): file generated by tool * `loki/pkg` directory files updated with up-to-date documentation from `_index.md` file * [tools/doc-generator](5550cd65ec/tools/doc-generator) directory with documentation generation tool. **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Mohamed-Amine Bouqsimi	03a34db8eb	Fix alertmanager per tenant override config behavior (#7805) ... **Context:** #7359 implement multi tenancy for the alertmanager config. However, the behavior of the per-tenant alertmanager config is erroneous, as it tries to merge the global alertmanager config with the per-tenant config. This does not work because there is no way of knowing if certain values were set by the user or if they were undefined and have the default Golang zero value (typically booleans). This PR changes the behavior of the alertmanager config to override all the configuration instead of trying to merge the global config with the per-tenant config. It also applies default values when appropriate.	3 years ago
Christian Simon	2ff6f13c8b	Better error message for empty selector queries (#6936) ... This improves the error when an empty label selector is given: Before: ```sh-session # Before $ logcli query "{}" Error response from server: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER # After $ logcli query "{}" Error response from server: parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will ``` I fear that might have wider implications, so it would be great to get a lot of eyes on this. In theory I think we should remove the restriction on querying all log streams, but this should maybe discussed in a follow up PR 🙂 Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Sandeep Sukhani	89d81020ce	fix lint issues from PR 7804 (#7814) ... **What this PR does / why we need it**: I had enabled auto-merge in PR #7804, but somehow it still merged the PR without all the checks passing. This PR fixes the failing lint and tests.	3 years ago
李国忠	6d05ade6a8	[performance] metrics query: range vector support streaming agg when no overlap (#7380) ... metrics query: range vector support streaming agg when no overlap	3 years ago
Danny Kopping	a63ad06509	Querier/Ruler: query blocker (#7785) ... Block malicious or expensive queries using a per-tenant runtime configuration.	3 years ago
Dylan Guedes	ad2260aec2	Loki: Fix multitenant querying (#7708) ... **What this PR does / why we need it**: We recently broke multitenant querying because of recent changes to how timeouts working across Loki. This PR fixes this by: - Adapt the timeout wrapper to work with multitenant queries. It will take the shortest timeout across all given tenants - Adapt the query engine timeout assigning to work with multitenant queries. It will take the shortest timeout between all the tenants - Adapt query sharding to use smallest max query parallelism across given tenants - Add a functional test to ensure multitenant is behaving as expected Signed-off-by: DylanGuedes <djmgguedes@gmail.com> Signed-off-by: Mehmet Burak Devecí <mhmtbrkdvc@gmail.com> **Which issue(s) this PR fixes**: https://github.com/grafana/loki/issues/7696 **Special notes for your reviewer**: The regression was probably introduced by https://github.com/grafana/loki/pull/7555	3 years ago
Dylan Guedes	8e38ebb2b7	Loki: Implement timeouts migration (#7555) ... **What this PR does / why we need it**: Improve the experience of migrating to our new timeouts behavior, which is the unifying of the engine:timeout and querier:query_timeout into a new configuration named limits_config:query_timeout. This PR makes sure users won't be surprised by their timeouts being shorter, but at the same time, it makes it easier to migrate to the new configuration. Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602) ... Implement `decolorize` filter that get rids of ANSI color codes in the log message. **Which issue(s) this PR fixes**: Fixes #7601	3 years ago
Owen Diehl	fd3a6dfaf1	Adds extra docs to flehs out ParseLabels implementation (#7562) ... Adding some extra comments for archaeological purposes	3 years ago
Ed Welch	23cc448881	Loki: log when a query starts in the logql engine (#7469) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: There are cases where a query cannot finish (OOM of a querier/ruler) and most of our logs are emitted upon completion of a query. This can help us find queries which are OOM crashing a component. Note: It only covers queries executed by the engine, so only instant and range queries. Another PR will be necessary to cover labels/series endpoints. However this approach seemed the best to be able to get this information from ruler executed queries. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Travis Patterson	c1bccac141	Results cache fix improvements (#7444) ... Move middleware to someplace more sensible and incorporate feedback missed in the review for this.	3 years ago
Travis Patterson	f2297d3d2a	Fix result cache misses on sharded queries (#7429) ... Headers weren't being propagated from query responses on sharded queries causing result cache misses. This PR: - Adds headers to `logproto.Result` so we can save response headers as they come back from queries - Metric-query results are turned into numbers well away from where responses are sent to the frontend: save all headers in the context so they can be pulled out when the final result is calculated - Propagate headers in the final responses	3 years ago
Owen Diehl	eaa0919e53	Use labelbuilder in syntax.ParseLabels to remove empty label values (#7355) ... There's a bug in TSDB where empty label values alter the series fingerprints used in chunk addresses, but are stripped out and create different fingerprints in our TSDB based index. I plan to open a PR to Prometheus, but I also prefer normalizing labels here by removing empty label values. According to [Prometheus](https://prometheus.io/docs/concepts/data_model/), > A label with an empty label value is considered equivalent to a label that does not exist. Our bug comes from the Prometheus model `LabelBuilder` stripping out empty label values, [here](https://github.com/grafana/loki/blob/main/vendor/github.com/prometheus/prometheus/model/labels/labels.go#L419-L423). This means that `Labels.NewBuilder({job="foo",bar=""}) => {job="foo"}`. The [labels.Hash()](https://github.com/grafana/loki/blob/main/vendor/github.com/prometheus/prometheus/model/labels/labels.go#L136-L182) function does not skip empty label values, meaning it's easy to end up with different fingerprints for effectively the same series (an empty label value is supposed to be equivalent to a missing label in prometheus, but they generate different label hashes).	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340) ... I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Dylan Guedes	f9f1eddfc5	Loki: Improve query timeouts behavior (#7230) ... **What this PR does / why we need it**: Improves the different query-related timeouts behavior by fixing a bug and having better defaults. The changes are: - Fix the timeout middleware by creating a new request that will use the new WithDeadline context - `engine:timeout` was being ignored. This PR improves such scenario by using **the query timeout as a fallback** instead of the default `5min` for all cases	3 years ago