115 Commits (7cbfdb5d6d5f6bb7fd6cc97d3f63cfb2997bfed6)

Author	SHA1	Message	Date
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Salva Corts	61d407b4bb	Add keep stage to LogQL (#9693) ... **What this PR does / why we need it**: This PR adds a new `keep` stage to LogQL. It allows filtering specific labels and helps to avoid hitting the _max streams per query_ limit by keeping only specific labels in the label set. It works the opposite of [drop stage](https://grafana.com/docs/loki/latest/logql/log_queries/#drop-labels-expression). **Special notes for your reviewer**: We do not drop [special error labels](edc6b0bff7/pkg/logqlmodel/error.go (L18-L20)) added by Loki at query time. Users will have to explicitly drop them by using the drop stage. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Michel Hollands	79485cd30d	Add handling of timezones in unixToTime tests (#9707) ... **What this PR does / why we need it**: The unit tests for unixToTime do not take the current timezone into account. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Michel Hollands	22779e13c5	Fix date template function with epoch times (#8886) ... **What this PR does / why we need it**: Adds new toUnixEpoch... functions to convert from a string with a Unix/Epoch time to an integer that can be used in the existing `toDate` function. Note that these are the opposites of some of the functions introduced in https://github.com/grafana/loki/pull/8774. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/8624. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
Gregor Zeitlinger	b1917a6178	add "alignLeft" and "alignRight" functions (#9672) ... Fixes https://github.com/grafana/loki/issues/9667	2 years ago
Travis Patterson	fd0efe0799	Regex line/label filters return original regex when `String` is called (#9584) ... To optimize regex line and label filters, Loki rewrites them to be non-greedy. This can cause extra characters to appear in the regex when `String()` is called. In addition to confusing logging, this can also cause internal panics when a previously-parsable query becomes too long. This PR stores the original regex in the label/line filter and uses that in calls to `String`. This PR also removes regex optimizations from label selectors. Making the optimization at parse time caused the original regex to be lost.	2 years ago
Bryan Boreham	ee74fbdf3a	logql/log tests: use abstractions over labels (#9532) ... **What this PR does / why we need it**: Call functions like `FromStrings`, so we don't have to know how the data structure is laid out. `FromStrings` sorts labels, so we don't have to call `Sort()`. **Special notes for your reviewer**: This PR only touches tests. Upstream Prometheus is using a different implementation of `labels.Labels`, so I plan to create a series of PRs which make Loki compatible with that. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - NA For Helm chart changes bump the Helm chart version. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Nick Moore	2f87acd284	Fix case insensitive label filter (#9404) ... **What this PR does / why we need it**: When comparing a case insensitive regular expression to a label value, Loki was only matching values that were in upper case. This was caused by compiling a case insensitive regular expression (such as `(?i)hello`), which produced an upper case `prefix` that was then used in `containsLower` as the `substr` value - but that function was written in such a way that only allowed matches of exact case or where the `line` value was upper case and the `substr` value was lower case. This PR added a some additional test cases to demonstrate/understand the issue, and adds two additional checks to the `containsLower` function to enable it to work correctly when either input is in either case. **Which issue(s) this PR fixes**: Fixes #9294 **Special notes for your reviewer**: Given the importance of this code path, I used `BenchmarkLineLabelFilters` to validate that the performance of this code was not affected. Overall, [the differences appeared negligible](https://docs.google.com/spreadsheets/d/1BRKkCzO0i11AObC6ADguDNa7cfG93hB7JLtqYWmnfW0/edit?usp=sharing) on my device (Intel(R) Core(TM) i7-12700H). **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Travis Patterson	48ad068ce1	fix json expression parser (#9410) ... When the value parsed by the `JsonExpressionParser` is an escaped json string, the result still contains escape character and may not be valid json. This causes subsequent json parsing steps to fail. For example: `{app="foo"} | json message | line_format "{{.message}}" | json` fails when `message` remains escaped. This PR unescapes the results of extracted `JsonExpressions` so they can be parsed by subsequent json parsers	2 years ago
李国忠	38b298c3fe	LogQL: Introduce `distinct` (#8662) ... **What this PR does / why we need it**: Introduces `distinct` to LogQL. Usage: `{job="varlogs"} | distinct filename` It is similar to `distinct` in SQL and especially useful for sampling. Similar syntax exists in other log query languages. **Example** For the following log lines: ```json {"event": "access", "id": "1", "time": "2023-02-28 15:12:11"} {"event": "access", "id": "1", "time": "2023-02-28 15:13:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:14:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:15:11"} ``` The query below: ```logql {app="order"} | json | distinct id ``` Will return: ```json {"event": "access", `"id": "1",` "time": "2023-02-28 15:13:11"} {"event": "access", `"id": "2", `"time": "2023-02-28 15:15:11"} ``` **Example with multiple labels** ```logql {app="order"} | json | distinct id,time ``` **Which issue(s) this PR fixes**: Fixes #8649 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	2 years ago
Travis Patterson	7735b1e98e	Sanitize label keys in the unpack parser (#9121) ... This PR makes the `unpack` parser sanitize label keys like all the other parsers. When invalid label names are included, it causes the user to get `could not write JSON response` errors when the response is serialized to send back to the user.	2 years ago
Ed Welch	edc6b0bff7	Loki: Add a limit for the [range] value on range queries (#8343) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: Loki does not currently split queries by time to a value smaller than what's in the [range] of a range query. Example ``` sum(rate({job="foo"}[2d])) ``` Imagine now this query being executed over a longer window of a few days with a step of something like 30m. Every step evaluation would query the last [2d] of data. There are use cases where this is desired, specifically if you force the step to match the value in the range, however what is more common is someone accidentally uses `[$__range]` in here instead of `[$__interval]` within Grafana and then sets the query time selector to a large value like 7 days. This PR adds a limit which will fail queries that set the [range] value higher than the configured limit. It's disabled by default. In the future it may be possible for Loki to perform splits within the [range] and remove the need for this limit, but until then this can be an important safeguard in clusters with a lot of data. **Which issue(s) this PR fixes**: Fixes #8746 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	2 years ago
Travis Patterson	163fd9d8af	Short circuit parsing when label matchers are present (#8890) ... This PR makes parsers aware of any downstream label-matcher stages at parse time. As labels are parsed, if one has a matcher, the matcher is checked at parse time. If the label does not match it's matcher, parsing is halted on that log line. **ex 1:** consider the log: `foo=1 bar=2 baz=3` And the query `{} | logfmt | bar=3` When `bar` is parsed it is immediately checked against it's matcher. The match fails so we the parser never spends time parsing the rest of the line. **ex 2:** consider the log: `foo=1 baz=3 bletch=4` And the query `{} | logfmt | bar=3` `bar` is never seen in the log so the whole line is parsed. **Benchmarks:** ``` │ parsers__old_2.txt │ parsers__new_3.txt │ │ sec/op │ sec/op vs base │ _Parser/json/inline_stages-8 3413.5n ± 5% 766.4n ± 4% -77.55% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 101.5n ± 6% 103.1n ± 8% ~ (p=0.645 n=10) _Parser/unpack/inline_stages-8 383.8n ± 4% 388.0n ± 9% ~ (p=0.954 n=10) _Parser/unpack-not_json_line/inline_stages-8 13.30n ± 2% 13.11n ± 1% ~ (p=0.247 n=10) _Parser/logfmt/inline_stages-8 2105.5n ± 16% 727.7n ± 4% -65.44% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 4.220µ ± 4% 4.175µ ± 4% ~ (p=0.739 n=10) _Parser/regex_status_digits/inline_stages-8 319.8n ± 5% 326.4n ± 8% ~ (p=0.481 n=10) _Parser/pattern/inline_stages-8 185.2n ± 7% 154.2n ± 3% -16.74% (p=0.000 n=10) │ parsers__old_2.txt │ parsers__new_3.txt │ │ B/op │ B/op vs base │ _Parser/json/inline_stages-8 280.00 ± 0% 64.00 ± 0% -77.14% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=10) _Parser/unpack/inline_stages-8 80.00 ± 0% 80.00 ± 0% ~ (p=1.000 n=10) _Parser/unpack-not_json_line/inline_stages-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) _Parser/logfmt/inline_stages-8 336.00 ± 0% 74.00 ± 0% -77.98% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 193.0 ± 1% 192.0 ± 1% ~ (p=0.656 n=10) _Parser/regex_status_digits/inline_stages-8 51.00 ± 0% 51.00 ± 0% ~ (p=1.000 n=10) _Parser/pattern/inline_stages-8 35.000 ± 0% 3.000 ± 0% -91.43% (p=0.000 n=10) │ parsers__old_2.txt │ parsers__new_3.txt │ │ allocs/op │ allocs/op vs base │ _Parser/json/inline_stages-8 18.000 ± 0% 4.000 ± 0% -77.78% (p=0.000 n=10) _Parser/jsonParser-not_json_line/inline_stages-8 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) _Parser/unpack/inline_stages-8 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) _Parser/unpack-not_json_line/inline_stages-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) _Parser/logfmt/inline_stages-8 16.000 ± 0% 6.000 ± 0% -62.50% (p=0.000 n=10) _Parser/regex_greedy/inline_stages-8 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) _Parser/regex_status_digits/inline_stages-8 2.000 ± 0% 2.000 ± 0% ~ (p=1.000 n=10) _Parser/pattern/inline_stages-8 2.000 ± 0% 1.000 ± 0% -50.00% (p=0.000 n=10) ``` --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Karsten Jeschkies	94725e7908	Define `RequiredLabels` query limit. (#8851) ... **What this PR does / why we need it**: Some end-users can impose great workload on a cluster by selecting too many streams in their queries. We should be able to limit them. Therefore we introduce a new limit `RequiredLabelMatchers` which list label names that must be included in the stream selectors. The implementation follows the same approach as for max query limit. **Which issue(s) this PR fixes**: Fixes #8745 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Travis Patterson	9d20bed26a	Short circuit parsers (#8724) ... In many cases we only need a few labels from a line. Because most of our parsers parse lines incrementally, we can stop parsing a line after we have all the labels we want from it. This pr uses `ParserHints` to keep track of the number of extracted labels. It also provides a way for parsers to know when they should stop parsing. Notes: - parsers had inconsistent ordering between the `ShouldExtract` call and adding the `_extracted` label to duplicates. This PR makes appending `_extracted` always happen before `ShouldExtract` to keep counts of what is extracted compared to expected labels consistent. Next Steps: - When the user specifies a query with a grouping containing the `_extracted` label but there is no duplicate between the passed line and labels, short circuiting will not work. I'll address this in a follow up PR. Benchmarks: To try and show a balanced view of what this buys us, this pr picks a label our of the middle of a line. In a best case this might be much better. In the worst case, we have to parse the whole line. ``` benchstat short_circuit_old.txt short_circuit_new.txt name old time/op new time/op delta KeyExtraction/json-8 456ns ± 3% 256ns ± 2% -43.84% (p=0.000 n=9+10) KeyExtraction/logfmt-8 347ns ± 4% 171ns ± 2% -50.86% (p=0.000 n=10+10) KeyExtraction/logfmt-expression-8 552ns ± 2% 368ns ± 2% -33.23% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KeyExtraction/json-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 16.0B ± 0% 16.0B ± 0% ~ (all equal) name old allocs/op new allocs/op delta KeyExtraction/json-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 2.00 ± 0% 2.00 ± 0% ~ (all equal) ``` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Travis Patterson	5a08a6bcb9	Label selector optimizations (#8763) ... This PR introduces the following label selector optimizations: - any selector that is `=~<literal>` is converted to an `Equals/NotEquals` matcher - Use the greedy->non-greedy regex improvements that exist in line and label filters - Never run `.*` label matchers. Prometheus regex matchers anchor the match to the beginning and end of the line. That means matchers of the form `=~<literal_value>` are just testing equality. Benchmarks -- this is just a quick bench to demonstrate equality matchers are faster than their equivalent regex matchers: ``` BenchmarkMatcherTypes/regex-8 27694834 53.74 ns/op 0 B/op 0 allocs/op BenchmarkMatcherTypes/equals-8 627045850 2.153 ns/op 0 B/op 0 allocs/op ```	2 years ago
Ashwanth	7d2eeddfb8	LogQL: Fix panic when using `unpack` with empty line (#8853) ... **What this PR does / why we need it**: Fixes panic in unpack parser when processing empty lines ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:583 github.com/grafana/loki/pkg/logql/log.(*UnpackParser).unpack(...) panic: runtime error: index out of range [0] with length 0 ``` Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Ashwanth	d08b87f799	json parser: fix panic when processing empty line (#8835) ... **What this PR does / why we need it**: Fixes panic when an empty line is processed by `JSONExpressionParser` ``` /src/enterprise-logs/vendor/github.com/grafana/loki/pkg/logql/log/parser.go:537 github.com/grafana/loki/pkg/logql/log.isValidJSONStart(...) ... panic: runtime error: index out of range [0] with length 0 ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Travis Patterson	37c0b3dec2	Allow math on errors (#8823) ... Currently metric expressions will fail if they contain the parser error label `__error__`. There are cases where you may want to count errors. For example, `sum(count_over_time({foo="bar"} | json | __error__= "JSONParserErr" [$__interval]))` to see how many of your logs are unparsable JSON. This pr uses `ParseHint` to determine whether the `__error__` label was requested in the query and, if it was, do math on it like any other query. Parse hints are only included in VectorAggregations so the label used to signal whether or not to keep errors is never seen by a user. All other operations remain unaffected	2 years ago
Travis Patterson	83ba93c867	fix to-string for noop label filters (#8799) ... Label Filter optimizations omit `.*` label filters from the execution pipeline but intruduced a bug where those stages were also ommitted from calls to `String` on the expression. This results in unparsable queries and panics when queries are cloned via `query.String()` This PR makes `noopLabelFilter` embed matchers and inherit their `String` method just like the `StringLabelFilters` do	2 years ago
Ed Welch	0352b75f0d	Loki: Add new logql template functions `bytes`, `duration`, `unixEpochMillis`, `unixEpochNanos`, `toDateInZone`, `b64Enc`, and `b64Dec` (#8774) ... **What this PR does / why we need it**: This adds several new template functions to the logql to be used in `line_format` and `label_format` - bytes - duration (and duration_seconds) - unixEpochMillis - unixEpochNanos - toDateInZone - b64Enc and b64Dec `bytes` and `duration` adds parity with the `unwrap` pipeline stage in logql, allowing you to convert human readable versions of bytes and times into their respective primitives of bytes and seconds. This makes it possible to do math and comparisons of these extracted values in `line_format` and `label_format` `unixEpochMillis` and `unixEpochNanos` extends the existing `unixEpoch` function which can only generate seconds level precision to allow having millisecond and nanosecond level precision. `toDateInZone` allows for specifying a specific timezone when parsing a date. `b64Enc` and `b64Dec` allow base64 encoding and decoding values. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: @JStickler I made a lot of changes to the template_functions.md doc page to try and create some more consistency as well as remove some notices that really aren't relevant anymore: * Tried to make sure every function had a `Signature` and `Examples` block * Removed a lot of `NOTE` about functions being introduced in a specific version * Removed documentation for some deprecated string functions (they still exist in code but I removed the docs for them, their exist better replacements for all of them) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Danny Kopping	e2ac2d50e2	Changes from linter (#8769) ... **What this PR does / why we need it**: We had a few outstanding linter issues, and running `make lint` fixes them all automatically. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Travis Patterson	9ac33e216e	Reimplement Unpack in terms of jsonparser rather than jsoniter (#8739) ... This PR reimplements the functionality of `Unpack` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` name old time/op new time/op delta _Parser/unpack/no_labels_hints-8 900ns ± 4% 309ns ± 3% -65.62% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 1.06µs ±12% 0.32µs ± 2% -69.72% (p=0.000 n=9+10) _Parser/unpack-not_json_line/no_labels_hints-8 37.5ns ± 8% 10.6ns ± 3% -71.70% (p=0.000 n=9+9) _Parser/unpack-not_json_line/labels_hints-8 39.6ns ±11% 10.3ns ± 2% -74.00% (p=0.000 n=10+9) name old alloc/op new alloc/op delta _Parser/unpack/no_labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 536B ± 0% 80B ± 0% -85.07% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00B 0.00B ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta _Parser/unpack/no_labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack/labels_hints-8 17.0 ± 0% 4.0 ± 0% -76.47% (p=0.000 n=10+10) _Parser/unpack-not_json_line/no_labels_hints-8 0.00 0.00 ~ (all equal) _Parser/unpack-not_json_line/labels_hints-8 0.00 0.00 ~ (all equal) ```	2 years ago
Travis Patterson	9c6e509a6a	Reimplement JsonExpressionParser in terms of jsonparser (#8734) ... This PR reimplements the functionality of `JSONExpressionParser` using the `jsonparser` library rather than `jsoniter` Benchmarks: ``` benchstat json_expression_old.txt json_expression_new.txt name old time/op new time/op delta JsonExpressionParser/json-expression-8 4.15µs ± 4% 0.82µs ± 3% -80.17% (p=0.000 n=10+10) name old alloc/op new alloc/op delta JsonExpressionParser/json-expression-8 1.07kB ± 0% 0.02kB ± 0% -97.76% (p=0.000 n=10+10) name old allocs/op new allocs/op delta JsonExpressionParser/json-expression-8 72.0 ± 0% 2.0 ± 0% -97.22% (p=0.000 n=10+10) ```	2 years ago
Travis Patterson	7f42137584	Use Line Filters in StringLabelFilter (#8659) ... We've done a ton of work to optimize regexes in line filters but nothing in label filters. To avoid duplicating the work or reworking all of our filter types, this PR creates a new label filter that has an optimized line filter. It uses that against the label value to determine if they match. Note on Benchmarks: the input string for this is 1114195 bytes long. The `not` test tells us that our approach is quite a lot slower when the literal pattern isn't present in the input. The size of the input really emphasizes it. Benchmarks: ``` name old time/op new time/op delta LineLabelFilters/foo-8 50.6ns ± 9% 14.3ns ± 1% -71.67% (p=0.000 n=10+8) LineLabelFilters/not-8 15.7ns ± 1% 64165.6ns ± 2% +407766.77% (p=0.000 n=10+10) LineLabelFilters/(foo)-8 58.4ns ± 1% 14.8ns ±10% -74.68% (p=0.000 n=9+9) LineLabelFilters/(foo|ba)-8 64.9ns ± 0% 18.2ns ± 2% -71.90% (p=0.000 n=7+9) LineLabelFilters/(foo|ba|ar)-8 74.8ns ± 7% 21.4ns ± 2% -71.42% (p=0.000 n=10+8) LineLabelFilters/(foo|(ba|ar))-8 65.1ns ± 1% 18.1ns ± 2% -72.22% (p=0.000 n=8+8) LineLabelFilters/foo.*-8 882ns ± 4% 15ns ±11% -98.30% (p=0.000 n=9+10) LineLabelFilters/.*foo-8 11.5ns ± 8% 14.9ns ± 7% +30.39% (p=0.000 n=10+10) LineLabelFilters/.*foo.*-8 5.46µs ± 7% 0.01µs ± 3% -99.74% (p=0.000 n=10+9) LineLabelFilters/(.*)(foo).*-8 6.26µs ± 7% 0.01µs ± 2% -99.77% (p=0.000 n=10+9) LineLabelFilters/(foo.*|.*ba)-8 5.58µs ± 3% 0.02µs ± 8% -99.67% (p=0.000 n=9+10) LineLabelFilters/(foo.*|.*bar.*)-8 7.02µs ± 7% 0.02µs ± 8% -99.73% (p=0.000 n=10+10) LineLabelFilters/.*foo.*|bar-8 5.53µs ± 6% 0.02µs ± 2% -99.67% (p=0.000 n=10+8) LineLabelFilters/.*foo|bar-8 3.73µs ± 6% 0.02µs ± 3% -99.52% (p=0.000 n=10+9) LineLabelFilters/(?:.*foo.*|bar)-8 5.43µs ± 8% 0.02µs ± 8% -99.65% (p=0.000 n=10+10) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 5.71µs ± 3% 0.02µs ±10% -99.68% (p=0.000 n=9+9) LineLabelFilters/.*foo.*|bar|buzz-8 5.34µs ± 2% 0.02µs ± 3% -99.65% (p=0.000 n=10+9) LineLabelFilters/.*foo.*|bar|uzz-8 5.36µs ± 1% 0.02µs ± 3% -99.60% (p=0.000 n=8+8) LineLabelFilters/foo|bar|b|buzz|zz-8 68.8ns ± 8% 22.6ns ± 7% -67.19% (p=0.000 n=10+10) LineLabelFilters/f|foo|foobar-8 77.1ns ± 1% 18.6ns ± 9% -75.90% (p=0.000 n=8+10) LineLabelFilters/f.*|foobar.*|.*buzz-8 5.64µs ± 7% 0.02µs ±10% -99.60% (p=0.000 n=10+10) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 6.29µs ± 8% 0.02µs ±10% -99.65% (p=0.000 n=10+10) LineLabelFilters/.*-8 963ns ± 8% 2ns ±14% -99.77% (p=0.000 n=10+10) LineLabelFilters/.*|.*-8 4.46µs ± 8% 0.00µs ± 8% -99.95% (p=0.000 n=10+10) LineLabelFilters/.*||||-8 2.99µs ± 1% 0.00µs ±16% -99.93% (p=0.000 n=8+10) LineLabelFilters/#00-8 43.5ns ± 8% 2.1ns ± 1% -95.25% (p=0.000 n=10+8) LineLabelFilters/(?i)foo-8 71.7ns ± 6% 13.1ns ± 1% -81.73% (p=0.000 n=10+8) LineLabelFilters/(?i)界-8 16.5ns ± 8% 150.2ns ± 8% +809.55% (p=0.000 n=10+10) LineLabelFilters/(?i)ïB-8 23.8ns ± 1% 14748979.8ns ± 7% +62074730.81% (p=0.000 n=8+10) LineLabelFilters/(?:)foo|fatal|exception-8 185ns ± 2% 21ns ± 2% -88.38% (p=0.000 n=9+8) LineLabelFilters/(?i)foo|fatal|exception-8 78.4ns ± 3% 20.2ns ± 2% -74.23% (p=0.000 n=9+9) LineLabelFilters/(?i)f|foo|foobar-8 94.3ns ±16% 15.4ns ± 9% -83.66% (p=0.000 n=10+10) LineLabelFilters/(?i)f|fatal|e.*-8 65.7ns ± 5% 18.2ns ± 3% -72.24% (p=0.000 n=9+8) LineLabelFilters/(?i).*foo.*-8 6.04µs ± 8% 0.01µs ± 2% -99.78% (p=0.000 n=10+10) name old alloc/op new alloc/op delta LineLabelFilters/foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/not-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*|.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/.*||||-8 0.00B 0.00B ~ (all equal) LineLabelFilters/#00-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)界-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00B 0.00B ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta LineLabelFilters/foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/not-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|ba|ar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo|(ba|ar))-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(.*)(foo).*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*ba)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(foo.*|.*bar.*)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo|bar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?P<foo>.*foo.*|bar)-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*foo.*|bar|uzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/foo|bar|b|buzz|zz-8 0.00 0.00 ~ (all equal) LineLabelFilters/f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/f.*|foobar.*|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/((f.*)|foobar.*)|.*buzz-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*|.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/.*||||-8 0.00 0.00 ~ (all equal) LineLabelFilters/#00-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)界-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)ïB-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?:)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)foo|fatal|exception-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|foo|foobar-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i)f|fatal|e.*-8 0.00 0.00 ~ (all equal) LineLabelFilters/(?i).*foo.*-8 0.00 0.00 ~ (all equal) ```	2 years ago
Travis Patterson	c42a0ba868	Optimize .+ regex line filter (#8646) ... Interpolated variables from Grafana dashboards may result in the regex matcher `.+`. Filters with `+` are already being rewritten to be non greedy but there is a further optimization to make replace the regex altogether with a filter that just matchers `len(line) > 0`	2 years ago
Christian Haudum	d4a6b159fa	chore: Cleanup label extraction expression (#8596) ... The aim of this is to remove redundancy from the parser grammar and make the expression name more descriptive. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Bayan Taani	68a9fd6c0a	Add logfmt selective label extraction (#6675) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR introduces extracting labels from a log line in `logfmt`, with an extra option to rename them. For example, this query: ``` {app=”foo”} | logfmt msg,status ``` will extract the labels `msg` and `status` from the following logfmt line: ``` level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..." ``` With the following results: ``` msg="final error sending batch" status=”400” ``` -------------- Another possible scenario with label renaming: ``` {app=”foo”} | logfmt message="msg", status ``` That produces the following results: ``` message="final error sending batch" status=”400” ``` **Which issue(s) this PR fixes**: Fixes #3355 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Tristan Morgan	2e11ff00eb	Replace deprecated inet.af/netaddr (#8560) ... This removes the deprecated `inet.af/netaddr` that imports `go4.org/unsafe/assume-no-moving-gc` and panic when it's not updated. Fixes #8399 and replaces #8495	2 years ago
李国忠	322783e3d8	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	2 years ago
Kaviraj Kanagaraj	807aa459d4	logql: Support `urlencode` and `urldecode` template functions (#8271) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Ed Welch	c8760f31e2	Loki: Remove validation of matchers when performing label values lookup. (#8224) ... the curren validation fails on != style matchers when they are the only matchers present, requiring at least one `=` matcher. However for looking up label values in TSDB, by design it uses a `desired_label != ""` style query, as such it should be allowed to also include these as matchers. Additional matchers on this Label Value queries typically only exist with LBAC headers adding additional labels. Signed-off-by: Edward Welch <edward.welch@grafana.com> **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com> Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Owen Diehl	a7ea9c1422	Revert "fix(logql): remove __error_details__ label on drop __error__ or other way around" (#8200) ... Reverts grafana/loki#8150 I was talking with @adityacs about reverting this PR and found the following: Looking at why tests don’t keep `__error_details__` when `__error__` is dropped and I found this line of code: ```go // LabelsResult returns the LabelsResult from the builder. // No grouping is applied and the cache is used when possible. func (b *LabelsBuilder) LabelsResult() LabelsResult { // unchanged path. if len(b.del) == 0 && len(b.add) == 0 && b.err == "" { return b.currentResult } return b.toResult(b.labels()) }``` https://github.com/grafana/loki/blob/main/pkg/logql/log/labels.go#L314 We only add err details when err isn’t empty anyway, so I guess we’ll revert for simplicity but just keep the same functionality	2 years ago
Aditya C S	cf23194d05	fix(logql): remove __error_details__ label on drop __error__ or other way around (#8150)	2 years ago
Danny Kopping	b9a9cdf2f7	Fixing test regression (#8130) ... Regression introduced by #7723 which was merged around the same time Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Cyril Tovena	2861c0036f	Improve the JSON parser performance. (#7723) ... Sorry I couldn't help myself and worked a bit on the JSON parser by using another library that allows way less allocations. On top of this, the parser code is also much simpler to understand. I realize that we could possibly also intern/cache (limited obviously) extracted JSON value per key. I would do it per key because otherwise, a high cardinality key can take over the intern/cache which will make it worthless. Happy to walk someone through the idea, I think this would be a big jump too. Would be great to run and test this in our dev environment using some JSON queries before releasing it. ``` ❯ benchstat before.txt after.txt name old time/op new time/op delta _Parser/json/no_labels_hints-10 2.53µs ± 0% 2.16µs ± 2% -14.59% (p=0.008 n=5+5) _Parser/json/labels_hints-10 1.93µs ± 0% 1.47µs ± 0% -23.96% (p=0.008 n=5+5) name old alloc/op new alloc/op delta _Parser/json/no_labels_hints-10 656B ± 0% 280B ± 0% -57.32% (p=0.008 n=5+5) _Parser/json/labels_hints-10 512B ± 0% 176B ± 0% -65.62% (p=0.008 n=5+5) name old allocs/op new allocs/op delta _Parser/json/no_labels_hints-10 46.0 ± 0% 18.0 ± 0% -60.87% (p=0.008 n=5+5) _Parser/json/labels_hints-10 39.0 ± 0% 12.0 ± 0% -69.23% (p=0.008 n=5+5) ```	2 years ago
Aditya C S	8df5803d9a	feat(logql): Support drop labels in logql pipeline (#7975) ... This PR introduces `drop` stage in logql pipeline. Fixes #7870, Fixes #7368	2 years ago
Aditya C S	a08a7324f4	fix(logql): fix caseinsensitive search in filter (#8037) ... **What this PR does / why we need it**: Fix caseinsensitive search in filter **Which issue(s) this PR fixes**: Fixes #7904 #7837 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	2 years ago
Michel Hollands	59eff5ca58	Add count to template functions (#7951) ... **What this PR does / why we need it**: Add a `count` template function to count occurrences of a substring. For example if `label="abc abd abc"` ``` {{ .label | count "abc" }} ``` would return 2. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602) ... Implement `decolorize` filter that get rids of ANSI color codes in the log message. **Which issue(s) this PR fixes**: Fixes #7601	3 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340) ... I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Vladyslav Diachenko	615941234a	Logql Analyzer (#6171) ... * created logql-debugger page and API endpoint that returns information for each stage of the query * added docker-compose setup for local testing * refactored analyzer and added unit tests * removed docs page to release it in a separate PR when back-end API is published * changes structure of results object in response to return origin log line needed for case when only stream selector is sent in query * merged the latest main and resolved conflicts * added endpoint for readiness probe and configured CI * fixed drone.yml discrepancy * fixed method signature * fixed path to Dockerfile * added clean step	3 years ago
Ed Welch	6d54505485	Loki: add `__line__` and `__timestamp__` to label_format (#6983) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: We have the `__line__` and `__timestamp__` functions currently available in the `line_format` stage but not in the `label_format` stage. This PR fixes that. Specifically `__timestamp__` can be valuable in a label because you can do things like ``` label_format hour_of_day=`{{__timestamp__ | date "15"}}` ``` To be able to conditionaly select log lines based on the hour of the day they were created, this can be useful for conditional alerting as an example. **Which issue(s) this PR fixes**: Fixes #6402 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Christian Simon	ee2fe6514d	Propagate error detail properly in all cases (#6890) ... The error handling was missing appending the error detail properly in all cases. Fixes #6265	3 years ago
Kaviraj Kanagaraj	7b89f20378	fix(logql): Make `LabelSampleExtractor` work as expected when label doesn't exist in logline (#6766) ... * fix(logql): Make `StringLabelfilter` work as expected Consider Log lines ``` msg=hello subqueries=5 msg=hello subqueries=0 msg=hello ``` If you run following logql ``` {foo="bar"}|logfmt|subqueries!=0 ``` it returns ``` msg=hello subqueries=5 ``` But if you run following logql ``` {foo="bar"}|logfmt|subqueries!="0" # NOTE: "0" as string. ``` it returns ``` msg=hello subqueries=5 msg=hello ``` NOTE: it also returns log lines that doesn't contain `subqueries` label in the first place. This cause subtle error if used in metric queries using that `label` in unwrap. e.g: following logql on above log lines returns ``` sum_over_time({foo="bar"}|logfmt|subqueries!="0"|unwrap subqueries[1m]) ``` Returns `pipeline error: 'SampleExtractionErr'`. Because, the lines without `subqueries` labels are not skipped during `ProcessLine` and during metric extraction, our extractor sets this `SampleExtractionerror`. Fixes: https://github.com/grafana/loki/issues/6713 * Add it on changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more tests and fix the edge case with equality to empty strings Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * handle this edge case during metric extraction. Not in label filter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove commented code Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Ed Welch	1a75bb8e22	Loki: Return an __error_details__ label for any line which incurs a __error__ while being processed (#6543) ... * adds an __error_details__ label which includes any details available about a logql error. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint, changed a test a little Signed-off-by: Edward Welch <edward.welch@grafana.com> * reset error details in Reset() function Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago