35 Commits (6631c0f01d193fdb97e6875cd5a63d2b18b30fdc)

Author	SHA1	Message	Date
Sven Grossmann	9981e9e40d	fix(orFilters): fix multiple or filters would get wrong filtertype (#13169)	2 years ago
benclive	6df81db978	feat: Support negative numbers in LogQL (#13091)	2 years ago
Kaviraj Kanagaraj	e9b6ce9de4	fix(logql): Linefilter behaviour with chained `or` (#12463) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Anton Kolesnikov	36c703dae8	feat: add pattern match line filter (#12398)	2 years ago
Anton Kolesnikov	cc941fe42a	chore: refactor line filter MatchType (#12388)	2 years ago
Poyzan	6d66ea382c	[Fix] OR statements not being evaluated as part of nested line filters (#11735) ... **What this PR does / why we need it**: When nester line filters were being evaluated, the `OR` statements in right expressions were omitted and only `LineFilters` were returned. Resulting in only first value being returned. It can be reproduced in stringer unit test failing with, ``` Error Trace: /Users/poyzannur/workspace/loki/pkg/logql/syntax/ast_test.go:535 Error: Not equal: expected: "{app=\"foo\"} |= \"foo\" or \"bar\" |= \"baz\" or \"bal\"" actual : "{app=\"foo\"} |= \"foo\" or \"bar\" |= \"baz\"" ``` We now return the `OR` expression as part of nested line filters. Thanks a million to @ashwanthgoli for help with debugging, and extra unit test. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/support-escalations/issues/9042 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492) ... In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Sven Grossmann	1496eb4752	Fix `or` filter not being stringified correctly (#11337) ... **What this PR does / why we need it**: This PR adds a missing `String()` call for the `or` statement in line filters. It also fixes a bug where `or` statements were not applied in conjunction with an IPLineFilter expression. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Kaviraj Kanagaraj	9366343e18	Bug(LogQL): Fix mismatch results on `scalar` and `vector` binary ops (#10997) ... **What this PR does / why we need it**: [According to LogQL doc,](https://grafana.com/docs/loki/latest/query/#comparison-operators) binary operators between `scalar` and `vector` should behave as follows (either it's `scalar` op `vector` or `vector` op `scalar`) ``` Between a vector and a scalar, these operators are applied to the value of every data sample in the vector ``` PromQL (where LogQL inspired from) [also works that way](https://prometheus.io/docs/prometheus/latest/querying/operators/). Currently LogQL violates this behaviour. It returns whatever on the `left` (even if it's scalar). Look into the attached issue for more details. Changes: 1. Fixed `MergeBinOps` by making it aware when to return left or right (depending on which is from vector originally) 2. Add tests in both `engine_test.go` and `evaluator_test.go` to lock this behaviour 3. Added appropriate comments to the types and functions. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/10741 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Travis Patterson	60edfe0006	Implement "or" statement in line filters (#8962) ... It might be nice to not force a user to use `|~` when they need to express `or` in LogQL. Example: `{app="foo"} |= "foo" or "bar"` Edit: A ton of this is generated code or refactors. I've highlighed the parts most interesting to a reviewer --------- Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Kaviraj Kanagaraj	c030217e1b	Revert "LogQL: Introduce `distinct` (#8662)" (#10356) ... **What this PR does / why we need it**: This reverts commit 38b298c3fe. We are removing this PR #8662 because [known issue tracked here](https://github.com/grafana/loki/issues/9594) and some issues we facing internally with query splitting and sharding work(hard to parallelize, merging two `distinct` sets can have duplicates, hard to make it work with metric queries, etc). We decided best to remove it before make it to public release. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Owen Diehl	9097f1ff42	Sharding optimizations (#10101) ... A few bugfixes and more sharding optimizations * fix bug on `<aggr> by|without ()` groupings which removed the grouping while downstreaming * shardable implementations for `max+min`, operation specific merge strategies which enable many more types of sharded requests, even when label-reduction is performed at edge.	2 years ago
Travis Patterson	6dc722bc9f	correctly handle unpack when reordering stages (#9936) ... The optimization to reorder stages doesn't take into account the `unpack` stage. When line filters come after unpack stages, they shouldn't be moved.	2 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Bryan Boreham	0a2f58891c	logql tests: use abstractions over Labels (#9533) ... **What this PR does / why we need it**: Call functions like `FromStrings`, so we don't have to know how the data structure is laid out. **Special notes for your reviewer**: This PR only touches tests. Upstream Prometheus is using a different implementation of `labels.Labels`, so I plan to create a series of PRs which make Loki compatible with that. #9532 was the first. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - NA For Helm chart changes bump the Helm chart version. --------- Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Michel Hollands	478510a1bb	Fix Grouping String function and usage so it prints "without ()" (#9515) ... **What this PR does / why we need it**: The changes rangemapper made to LogQL expressions created invalid LogQL. It added a `without` clause instead of `without ()`. This PR changes the String function of the Grouping struct. This is then used in the rangemapper. This caused the following panic: ``` panic: error cloning query: count without (rate({namespace="apps"}[15s])): parse error at line 1, col 16: syntax error: unexpected RATE, expecting IDENTIFIER or ) ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com>	3 years ago
李国忠	38b298c3fe	LogQL: Introduce `distinct` (#8662) ... **What this PR does / why we need it**: Introduces `distinct` to LogQL. Usage: `{job="varlogs"} | distinct filename` It is similar to `distinct` in SQL and especially useful for sampling. Similar syntax exists in other log query languages. **Example** For the following log lines: ```json {"event": "access", "id": "1", "time": "2023-02-28 15:12:11"} {"event": "access", "id": "1", "time": "2023-02-28 15:13:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:14:11"} {"event": "access", "id": "2", "time": "2023-02-28 15:15:11"} ``` The query below: ```logql {app="order"} | json | distinct id ``` Will return: ```json {"event": "access", `"id": "1",` "time": "2023-02-28 15:13:11"} {"event": "access", `"id": "2", `"time": "2023-02-28 15:15:11"} ``` **Example with multiple labels** ```logql {app="order"} | json | distinct id,time ``` **Which issue(s) this PR fixes**: Fixes #8649 --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago
Karsten Jeschkies	7109c56bdc	Use un-escaped regexp literal for string matching. (#9252) ... **What this PR does / why we need it**: The Regexp simplification would use the original string value instead of the escaped literal. E.g. ``` | logfmt | level =~ `foo\.bar` ``` would match the label `level` against `foo\.bar` and not `foo.bar`. That is why the un-escaped literalt should be used in the equal and not-equal matchers. **Which issue(s) this PR fixes**: Fixes #9111 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	3 years ago
Travis Patterson	c4261b19fd	Fix parsing of long queries (#9134) ... Loki uses a `scanner.Scanner` to consume the characters in a query. `lex.go` relies heavily on the assumption a scanner can be copied to perform speculative parsing. The problem is that sometimes speculative parsing overruns the scanners internal buffer of `1024` bytes and incurs an additional read to the underlying reader. At that point, all subsequent scanners are out of sync with their underlying readers and the query may fail to be parsed. In the case of #9132, there is a function call at line `1024`. `lex.go` tries to understand this is a function call by speculatively parsing and incurs an underlying read which consumers the rest of the reader. When the lexer tries to scan the next character with the original scanner, the reader is already consumed even though only 1024 bytes have been scanned The scanner has an internal buffer size of `1024 bytes`. This PR copies `scanner.Scanner` from the `text/scanner` package and increases the internal buffer to `maxInputSize` which guarantees that no calls to `Scan` will incur reads beyond the first one. Ideally we wouldn't pass Scanners around in such a way but the cost of that is to rethink much of Loki's lexing logic. Given we know a max input size and we already have that in memory, this solution should suffice. Fixes https://github.com/grafana/loki/issues/9132	3 years ago
Travis Patterson	ee045312a9	Automatically Reorder Pipeline Filters (#8914) ... This PR moves Line Filters to be as early in a log pipeline as possible. For example: `{app="foo"} | logfmt |="some stuff"` becomes `{app="foo"} |="some stuff" | logfmt` Any LineFilter after a `LineFormat` stage will be moved to directly after the nearest `LineFormat` stage benchmarks: ``` goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/logql/syntax cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz │ reorder_old.txt │ reorder_new.txt │ │ sec/op │ sec/op vs base │ ReorderedPipeline-8 2104.5n ± 3% 173.9n ± 2% -91.74% (p=0.000 n=10) │ reorder_old.txt │ reorder_new.txt │ │ B/op │ B/op vs base │ ReorderedPipeline-8 336.0 ± 0% 0.0 ± 0% -100.00% (p=0.000 n=10) │ reorder_old.txt │ reorder_new.txt │ │ allocs/op │ allocs/op vs base │ ReorderedPipeline-8 16.00 ± 0% 0.00 ± 0% -100.00% (p=0.000 n=10) ```	3 years ago
Bayan Taani	68a9fd6c0a	Add logfmt selective label extraction (#6675) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: This PR introduces extracting labels from a log line in `logfmt`, with an extra option to rename them. For example, this query: ``` {app=”foo”} | logfmt msg,status ``` will extract the labels `msg` and `status` from the following logfmt line: ``` level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..." ``` With the following results: ``` msg="final error sending batch" status=”400” ``` -------------- Another possible scenario with label renaming: ``` {app=”foo”} | logfmt message="msg", status ``` That produces the following results: ``` message="final error sending batch" status=”400” ``` **Which issue(s) this PR fixes**: Fixes #3355 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Garrett	433d5bf913	fix panics when cloning a special query (#8531) ... Signed-off-by: garrettlish <garrett.li.sh@gmail.com>	3 years ago
李国忠	322783e3d8	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	3 years ago
李国忠	b4440911e0	[new feature] logql:support sort and sort_desc (#7989) ... **What this PR does / why we need it**: sort() sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order. sort_desc() Same as sort, but sorts in descending order. ex: `sort(rate(({app=~"foo|bar"} |~".+bar")[1m])) without (app) ` prometheud doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#sort **Which issue(s) this PR fixes**: Fixes #7933 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Michel Hollands	5fe9c964db	Fix validation for pattern and rexexp parsers to they don't panic in query-frontend (#7926) ... Running the query `topk(10,sum by(namespace)(count_over_time({application="nginx", site!="eu-west-1-dev"} |= "/artifactory/" != "api" != "binarystore" | regexp `` [1d])))` causes a panic. This PR adds validation for regexp and pattern parsers at parse time. If they don't have a (valid) parameter an error will be returned. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
李国忠	3296c6dbe8	[new feature] logql engine support exec `vector(0)` grama (#7023) ... **What this PR does / why we need it**: logql engine support exec `vector(0)` grama. new PR of :https://github.com/grafana/loki/pull/7007 **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: test passed. test1: vectorExpr only  test2: binExpr and vectorExpr  **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	6f4671a572	[new feature] syntax: support "vector" grama (#7007) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: Implement vector() function. The "vector function" feature is divided into multiple PRs to avoid excessively large PRs resulting in too long review time. **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: The test for syntax has passed  <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Susana Ferreira	3f3f4eda1d	Fix topk and bottomk operations with int <= 0 (#6937) ... * Fix topk and bottomk operations with int <= 0 * Fix VectorAggregationExpr String method * Update rangemapper_test expression * Add tests * Update CHANGELOG.md	3 years ago
Owen Diehl	d6f50ca5e1	[TSDB] Use index sampling to determine shard factor (#6396) ... * use index sampling to determine shard factor * logging/tracing + better parallelism planning * queryrange downstreamer now checks max_query_parallelism * lint * handles interval, offset in tsdb planning and adds logging * fixes ns->ms confusion in index stats proto * handle zero shard value without panics * shardmapper will downstream a single unsharded query * uses concat expr with no shards to avoid query parsing errors * better logging * fixes wrong Size() method call and rounds to nearest KB when calculating chunk size * humanize bytes in log line * only adds defaultLookback to index sampling when interval is zero * removes comment * more logging * adjust through correctly * adds query length for index queries	4 years ago
Owen Diehl	9218e4654b	TSDB/query-planning-groundwork (#6367) ... * begins building queryrange compatible types for index sampling * adds MatcherGroups() method to SampleExpr * MatcherGroups test + ShardResolver interface * index stats codec wiring * millisecond precision on index stats codec * lint * cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil	4 years ago
Owen Diehl	93de7a7061	renames skip -> ok in ProcessString (#6064) ... * renames skip -> ok in ProcessString Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * streampipeline matches convention Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Travis Patterson	5939d5e9d8	Query filtering in the ingester and storage (#5629) ... * Add timestamp to pipeline and extractor interfaces * implement filter extractor and pipeline * implement storage filtering * implement ingester filtering * test cleanup * Refactor pipeline setup to util * linter * enable filtering based on mode * review feedback * review comments * review feedback * no need to include label filters in the test pipeline * review feedpack	4 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	4 years ago
Cyril Tovena	a411bb9420	Disable sharding of count/avg when labels are mutated (#5474)	4 years ago
Sas Swart	c80244a3e4	Parse duration expressions in accordance with promql (#5275) ... * Add support for Prometheus like duration units in logql * Remove redundant type conversion * deduplicate call to sb.String Add a comment to explain why both promql and time's ParseDuration methods are used * extract duration parsing to separate function. * Add tests to ensure promql duration units are supported * Sort duration rune comment and list of runes in ascending order of length.	4 years ago
Marco Pracucci	c53457feb9	Upgrade Cortex, Prometheus and Thanos (#4830) ... * Upgrade Cortex, Prometheus and Thanos Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed imported ordering Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed data type used for series ref when recovering from checkpoint/WAL Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Thanos to fix arm build Signed-off-by: Marco Pracucci <marco@pracucci.com> * Run go mod tidy Signed-off-by: Marco Pracucci <marco@pracucci.com>	4 years ago
Karsten Jeschkies	c19a6c5bbc	Join contain filters into one. (#4528) ... * Join contain filters into one. * Update pkg/logql/log/filter.go Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> * Test Filter method. * Test old and new filter consturctors. * Remove comment. * Define more benchmark cases. * Take care of empty contain filters. * Satisfy linter. * USe different method to filter. * Simplify all filter concat. * Format code. * Push regex filters to the end. * Remove unused test. * Add a few more tests. * Make filters private again. * Fix test failure. * Only accumulate successive contain filters. * Reverse accumulation. * Satisfy linter. * Remove old version. Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	4 years ago
lingpeng0314	76e2a14029	add group_{left,right} to LogQL (#4510) ... * Add group_left/group_right support * Update docs/sources/logql/_index.md * Minor change for a test case * removes CardManyToMany * removes now-unused IsSetOperator helper Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Garrett	a046d79261	make LogQL syntax scope from private to public (#4446) ... * make LogQL syntax scope from private to public The follow-up for https://github.com/grafana/loki/pull/3991, it makes using LogQL syntax outside possible, it also makes LogRange implements Expr interface. * make constructor also public * Revert "make constructor also public" This reverts commit ab9d0a942c.	4 years ago
Kaviraj	bfb24e6a4c	Ip matcher for LogQL (#3986) ... * IP matcher and filter via netaddr.IP package Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Adding IP filter into label filter gramer 1. ast tests 2. parse tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Label filters all test cases passes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Rename ipFilter -> ipLabelFilter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * More tests Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove testlogs.txt Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more cases to the benchmark Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * PR remarks 1. support NEQ 2. reduce allocation. 3. Minor tweaks Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make IP filter work as line filter 1. Add necessary interface implementation 2. More tests for parser and ast Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Make linter happy Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Bug with filter matcher Its wierd bug. Not handing string() method breaks the parser :( Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Split Line and Label filter to different struct. It got out of hands to handle both in same struct. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add basic doc for IP matcher Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add docs feedback from Karen * Label filter add patter error report * PR remarks 1. Indendation fix 2. Remove string() interface for ip label filte * Linter happy * PR remaks * Fix ip label matcher string() method if pattern is invalid * Fix label filter tests * Have local copy of `pattern` in label filter to use in string() * Rebase conflicts * ast.go fixes	4 years ago
Periklis Tsirakidis	cb337c3b71	Add LogQL AST walker (#3991) ... * Make expressions public scoped * Add walkable implementation * Use walkable in logql/optimize	4 years ago
Cyril Tovena	59bb6d3fba	LogQL: Pattern Parser (#3837) ... * The beginning of a fun story. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Working on adding ragel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adding AST parsing with Yacc and Ragel. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Got a pattern parser working. Reworking ast to works with bytes and not runes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Setup tests and the matches algorithm. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * moar tests case. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some validation for the pattern expression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Hooking to LogQL + performance boost. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Adds documentation Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve bound check. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Removes generated files from being linted. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Review feedback Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Danny Kopping <dannykopping@gmail.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/logql/_index.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Docs suggestions Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	5 years ago
Cyril Tovena	4d1da2edf3	first_over_time and last_over_time (#3050) ... * Wip Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * [POC] last_over_time and first_over_time. This is an attempt to allow users to select the first or last discrete sample values from a range vector. This way we can create gauge, the only drawback being the log need to exists at least in every range. May be we could transfer the last or first value to the next range, but it will still be a problem for small ranges. Still wondering how useful this is compare to min/max_over_time. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago
Sandeep Sukhani	b152bc2cf3	check for stream selectors to have atleast one equality matcher (#3216) ... * check for stream selectors to have atleast one equality matcher * move equality matchers check to logql	5 years ago
Garrett	f70c7ea0bc	Implement offset modifier for range vector aggregation in LogQL (#3455) ... * Implement offset modifier for range vector aggregation in LogQL implementation for https://github.com/grafana/loki/issues/2785, the offset modifier allows changing the time offset for range vectors in a query to support e.g. selective timeshift in Grafana Signed-off-by: garrettlish <garrett.li.sh@gmail.com> * add docs for offset modifier * refine offset expr * add offset unit test in engine.go * remove duplicated offset shift * add offset test in range vector test Co-authored-by: garrettlish <garrettlish@163.com>	5 years ago
Cyril Tovena	7886f35b8f	Introduce a unpack parser. (#3420) ... * Introduce a unpack parser. To go with the new pack stage in promtail https://github.com/grafana/loki/pull/3401 this PR introduces a unpack parser that will automatically unpack all labels packed into the log line and set back the original log line. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add documentation. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * tweak promtail docs a little now that we have unpack in Loki Co-authored-by: Edward Welch <edward.welch@grafana.com>	5 years ago
Danny Kopping	feb7fb470b	LogQL: Simple JSON expressions (#3280) ... * New approach, still rough Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding benchmark Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Minor refactoring Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Further appeasing the linter Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding more tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding documentation Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Docs fixup Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing unnecessary condition Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding extra tests from suggestion in review Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding JSONParseErr Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding test to cover invalid JSON line Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding equivalent benchmarks for JSON and JSONExpression parsing Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding suffix if label would be overridden Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Reparenting jsonexpr directory to more appropriate location Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Setting empty label on non-matching expression, to retain parity with label_format Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding statement about returned complex JSON types Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Added check for valid label name Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Making json expressions shardable Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	5 years ago
Cyril Tovena	1a36874e1c	Ensure we have parenthetis for bin ops. (#3251) ... This ensure we correctly keep the order of execution when we stringify a bin operation. This has recently introduced a bug because we now use stringify to deep clone a LogQL expression. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>	5 years ago