apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626 ) What this PR does / why we need it: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Susana Ferreira	35465d0297	Fix instant query summary split stats (#9773 ) What this PR does / why we need it: Fix instant query summary statistic's `splits` corresponding to the number of subqueries a query is split into based on `split_queries_by_interval`. * Update rangemapper with a statistics structure to include the number of split queries a query is mapped into. * In the `split_by_range` middleware once the mapped query is returned update the middleware statistics with the number of split queries. This value will then be merged with the statistics of the Loki response. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Susana Ferreira	a894c3ccf0	Bugfix: Fix logic of offset operator in instant query splitting (#9763 ) What this PR does / why we need it: Fix the logic of the `offset` operator for downstream queries on instant query splitting of (range) vector aggregation expressions already containing an offset. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Michel Hollands	478510a1bb	Fix Grouping String function and usage so it prints "without ()" (#9515 ) What this PR does / why we need it: The changes rangemapper made to LogQL expressions created invalid LogQL. It added a `without` clause instead of `without ()`. This PR changes the String function of the Grouping struct. This is then used in the rangemapper. This caused the following panic: ``` panic: error cloning query: count without (rate({namespace="apps"}[15s])): parse error at line 1, col 16: syntax error: unexpected RATE, expecting IDENTIFIER or ) ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	2 years ago
李国忠	b4440911e0	[new feature] logql:support sort and sort_desc (#7989 ) What this PR does / why we need it: sort() sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order. sort_desc() Same as sort, but sorts in descending order. ex: `sort(rate(({app=~"foo\|bar"} \|~".+bar")[1m])) without (app) ` prometheud doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#sort Which issue(s) this PR fixes: Fixes #7933 Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Danny Kopping	c3e92b876c	Restoring lint checks that were previously disabled (#7340 ) I removed these checks in https://github.com/grafana/loki/pull/7337 to not make https://github.com/grafana/loki/pull/7243 overly complex to review. Refactored code to fix valid lint issues, ignored the irrelevant/dangerous ones. I also removed `unused` and `unparam` as these produce low-quality results that are largely useless.	3 years ago
Susana Ferreira	10e9695b21	Fix rangemapper with VectorExpr support (#7288 ) What this PR does / why we need it: The new expression type VectorExpr introduced recently in [PR](https://github.com/grafana/loki/pull/7007), is not taken into account in the query mapping in the AST mapper rangemapper. The AST mapper shardmapper was updated in [PR](https://github.com/grafana/loki/pull/7045). Special notes for your reviewer: Checklist - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	72d8627a55	log query which causes panic in logql.clone (#6934 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Susana Ferreira	79b2d01970	Add label_replace and literal expressions to instant query split by range (#6515 ) * Add label_replace logic to instant query split by range * Add label_replace and literal expressions support * Support binary operations with literal expressions * Remove Literal expression as splittable * Fix lint * Address comments	3 years ago
Christian Haudum	798677ae7f	Fix panic in instant query splitting when using unwrapped rate (#6348 ) * Fix panic in instant query splitting when using unwrapped rate The range aggregation `rate()` supports both log ranges and unwrapped ranges, e.g. `rate({app="foo"} [$__interval])` and `rate({app="foo"} \| unwrap bar [$__interval])` Since `rate()` was split into multiple `count_over_time()` over total duration, but `count_over_time()` does not support `unwrap`, unwrapped rate queries caused panics. This fix changes the splitting of `rate({app="foo"} \| unwrap bar [$__interval]` into multiple `sum_over_time()` over total duration. Fixes #6344 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Integrate review feedback Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>	3 years ago
Callum Styan	97ee1b93f3	[logql] don't try to execute a modified function that is not allowed by the AST (#6346 ) * Panic with an actual error if clone (which calls range aggregation expr's validate) errors, instead of swallowing the error and then attempting to execute a nil function, which panics anyways. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Call the validation function on a modified expression before attempting to execute it. If the validation fails we should return the original expression. Signed-off-by: Callum Styan <callumstyan@gmail.com>	3 years ago
Susana Ferreira	0dfd2cbbcc	Fix label extraction pushdown on instant queries (#6215 ) * Add downstream label extraction tests * Fix label extraction downstream pushdown	3 years ago
Christian Haudum	ee4ab1b394	Fix certain binary expression queries by optimizing through push down (#6132 ) * Add test cases that fail Instant queries of the following type fail and return an `unimplemented` error: ``` sum(count_over_time({foo="bar"} \| logfmt \| duration > 2s [3s])) / sum(count_over_time({foo="bar"} [3s])) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix certain binary expressions for instant queries If either the left hand side or the right hand side of a binary expression is a noop, we need to return the original expression so the whole expression is a noop as well, and thus not executed using the downstream engine. Otherwise, a binary expression that has a noop on either side, results in an `unimplemented` error when executed using the downstream engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Optimize instant vector aggregations with log extraction stage This change optimizes queries that use a vector aggregation without grouping around a range aggregation with a label extraction stage such as `json` or `logfmt`. Since the vector aggregation can be pushed down to the downstream query, the downstream query does not create a massive amount of streams, even though it contains a generic label extraction stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Susana Ferreira	6b6521f791	Add metrics to range mapper (#6030 ) * Refactor ShardingMetrics to MapperMetrics * Add mapper metrics to range mapper	3 years ago
Susana Ferreira	ea59e4bdc4	Add changelog entry for split by range of instant queries (#5966 )	3 years ago
Christian Haudum	99c5b35895	Add more function comments to the rangemapper implementation (#5861 ) * Add more function comments to the rangemapper implementation Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>	3 years ago
Susana Ferreira	0bce8d95a2	Split by range of instant queries (#5662 ) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `\| json` or `\| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago

17 Commits (7cbfdb5d6d5f6bb7fd6cc97d3f63cfb2997bfed6)