apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Alfredo	8f6d31c34e	Docs: update template function (#9037 ) What this PR does / why we need it: - Add `eq` function - Add example for nested if and `AND`/`OR` logic Which issue(s) this PR fixes: Fixes #<issue number> N/A Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Alfredo	f3a49f4734	Update contains and hasPrefix/hasSuffix with the correct examples (#8929 ) What this PR does / why we need it: WHAT: - Update examples for `contains` and `hasSuffix` - Correct order should be `functionName arg1 arg2 argN`, eg. `contains s src` WHY: - Current example for `contains` and `hasPrefix` / `hasSuffix` has a wrong order that will cause `TemplateFormatErr`. - Examples I'm referring to `{{ if .err hasSuffix "Timeout" }} timeout {{end}}` Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Travis Patterson	9d20bed26a	Short circuit parsers (#8724 ) In many cases we only need a few labels from a line. Because most of our parsers parse lines incrementally, we can stop parsing a line after we have all the labels we want from it. This pr uses `ParserHints` to keep track of the number of extracted labels. It also provides a way for parsers to know when they should stop parsing. Notes: - parsers had inconsistent ordering between the `ShouldExtract` call and adding the `_extracted` label to duplicates. This PR makes appending `_extracted` always happen before `ShouldExtract` to keep counts of what is extracted compared to expected labels consistent. Next Steps: - When the user specifies a query with a grouping containing the `_extracted` label but there is no duplicate between the passed line and labels, short circuiting will not work. I'll address this in a follow up PR. Benchmarks: To try and show a balanced view of what this buys us, this pr picks a label our of the middle of a line. In a best case this might be much better. In the worst case, we have to parse the whole line. ``` benchstat short_circuit_old.txt short_circuit_new.txt name old time/op new time/op delta KeyExtraction/json-8 456ns ± 3% 256ns ± 2% -43.84% (p=0.000 n=9+10) KeyExtraction/logfmt-8 347ns ± 4% 171ns ± 2% -50.86% (p=0.000 n=10+10) KeyExtraction/logfmt-expression-8 552ns ± 2% 368ns ± 2% -33.23% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KeyExtraction/json-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 16.0B ± 0% 16.0B ± 0% ~ (all equal) name old allocs/op new allocs/op delta KeyExtraction/json-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 2.00 ± 0% 2.00 ± 0% ~ (all equal) ``` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Ed Welch	0352b75f0d	Loki: Add new logql template functions `bytes`, `duration`, `unixEpochMillis`, `unixEpochNanos`, `toDateInZone`, `b64Enc`, and `b64Dec` (#8774 ) What this PR does / why we need it: This adds several new template functions to the logql to be used in `line_format` and `label_format` - bytes - duration (and duration_seconds) - unixEpochMillis - unixEpochNanos - toDateInZone - b64Enc and b64Dec `bytes` and `duration` adds parity with the `unwrap` pipeline stage in logql, allowing you to convert human readable versions of bytes and times into their respective primitives of bytes and seconds. This makes it possible to do math and comparisons of these extracted values in `line_format` and `label_format` `unixEpochMillis` and `unixEpochNanos` extends the existing `unixEpoch` function which can only generate seconds level precision to allow having millisecond and nanosecond level precision. `toDateInZone` allows for specifying a specific timezone when parsing a date. `b64Enc` and `b64Dec` allow base64 encoding and decoding values. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: @JStickler I made a lot of changes to the template_functions.md doc page to try and create some more consistency as well as remove some notices that really aren't relevant anymore: * Tried to make sure every function had a `Signature` and `Examples` block * Removed a lot of `NOTE` about functions being introduced in a specific version * Removed documentation for some deprecated string functions (they still exist in code but I removed the docs for them, their exist better replacements for all of them) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Bayan Taani	68a9fd6c0a	Add logfmt selective label extraction (#6675 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: This PR introduces extracting labels from a log line in `logfmt`, with an extra option to rename them. For example, this query: ``` {app=”foo”} \| logfmt msg,status ``` will extract the labels `msg` and `status` from the following logfmt line: ``` level=error ts=2021-02-12T19:18:10.037940878Z caller=client.go:294 component=client host=observability-loki-gateway msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-02-12 19:18:08.917452 +0000 UTC ignored, reason: 'entry out of order' for stream..." ``` With the following results: ``` msg="final error sending batch" status=”400” ``` -------------- Another possible scenario with label renaming: ``` {app=”foo”} \| logfmt message="msg", status ``` That produces the following results: ``` message="final error sending batch" status=”400” ``` Which issue(s) this PR fixes: Fixes #3355 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [x] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Vladyslav Diachenko	4e4359e67c	Fixed XSS on LogQL Analyzer page (#8521 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	3 years ago
J Stickler	06a6c7c9fc	Temporarily unpublish Analyzer page. (#8503 )	3 years ago
Kaviraj Kanagaraj	807aa459d4	logql: Support `urlencode` and `urldecode` template functions (#8271 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	3 years ago
Travis Patterson	1c954166b8	Fix documentation linter errors (#8229 )	3 years ago
Owen Diehl	a7ea9c1422	Revert "fix(logql): remove __error_details__ label on drop __error__ or other way around" (#8200 ) Reverts grafana/loki#8150 I was talking with @adityacs about reverting this PR and found the following: Looking at why tests don’t keep `__error_details__` when `__error__` is dropped and I found this line of code: ```go // LabelsResult returns the LabelsResult from the builder. // No grouping is applied and the cache is used when possible. func (b *LabelsBuilder) LabelsResult() LabelsResult { // unchanged path. if len(b.del) == 0 && len(b.add) == 0 && b.err == "" { return b.currentResult } return b.toResult(b.labels()) }``` https://github.com/grafana/loki/blob/main/pkg/logql/log/labels.go#L314 We only add err details when err isn’t empty anyway, so I guess we’ll revert for simplicity but just keep the same functionality	3 years ago
Aditya C S	cf23194d05	fix(logql): remove __error_details__ label on drop __error__ or other way around (#8150 )	3 years ago
Aditya C S	8df5803d9a	feat(logql): Support drop labels in logql pipeline (#7975 ) This PR introduces `drop` stage in logql pipeline. Fixes #7870, Fixes #7368	3 years ago
李国忠	b4440911e0	[new feature] logql:support sort and sort_desc (#7989 ) What this PR does / why we need it: sort() sort(v instant-vector) returns vector elements sorted by their sample values, in ascending order. sort_desc() Same as sort, but sorts in descending order. ex: `sort(rate(({app=~"foo\|bar"} \|~".+bar")[1m])) without (app) ` prometheud doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#sort Which issue(s) this PR fixes: Fixes #7933 Special notes for your reviewer: Checklist - [x] Reviewed the `CONTRIBUTING.md` guide - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Susana Ferreira	0f139e2cb3	Update log queries regex documentation (#8002 ) What this PR does / why we need it: Update log queries regex documentation with a clarification on the difference between the newline character in: * label matching operator: `.` does not match newlines line filter expressions: `.` matches newlines Checklist* - [x] Documentation updated	3 years ago
Michel Hollands	59eff5ca58	Add count to template functions (#7951 ) What this PR does / why we need it: Add a `count` template function to count occurrences of a substring. For example if `label="abc abd abc"` ``` {{ .label \| count "abc" }} ``` would return 2. Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Vladyslav Diachenko	c6b55e7512	[docs] removed static URL to LogQL Analyzer API (#7773 ) added logic to determine logql-analyzer version dynamically based on the value selected in docs version dropdown. Adding this logic to the docs, allows us to skip the step on release when we manually update the link to the new version of LogQL Analyzer	3 years ago
Wade Fitzpatrick	8a5c8f0a3d	Add closing code block backticks (#7772 )	3 years ago
Ursula Kallio	83d0f1c05a	Swap direction in "The label identifier is always on the right side of the operation." sentence (#7631 )	3 years ago
Max Vorobev	c5911d5342	Loki: implement decolorize filter (#7602 ) Implement `decolorize` filter that get rids of ANSI color codes in the log message. Which issue(s) this PR fixes: Fixes #7601	3 years ago
Vladyslav Diachenko	57c0c2cde2	[docs: logql-analyzer] changed url for main branch (#7378 ) pointed unreleased docs(`next`) to logql-analyzer that is built from main branch	3 years ago
Christian Haudum	c34078719c	[docs] Fix docs for `rate_counter()` `rate_counter` behaves like `rate` and therefore does not support grouping. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
Yahya	6bb73ef97e	Docs: remove redundant section in Template functions (#6458 ) Pipelines are already defined in L18-19	3 years ago
Brian Evans	e7acb5356b	Docs: Fix template function examples (#7148 ) Some examples only had a single open brace, but require two. I added the missing braces. What this PR does / why we need it: Improves documentation, helps users who are reading docs about template functions.	3 years ago
李国忠	93a5a71e62	[doc] logql: logql engine support exec vector(0) grama (#7044 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: logql engine support exec vector(0) grama. new PR of :https://github.com/grafana/loki/pull/7023 Which issue(s) this PR fixes: Fixes #6946 Special notes for your reviewer: preview ![image](https://user-images.githubusercontent.com/9583245/188355186-9a74d1ce-f062-45e8-8516-52c89383eeec.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Co-authored-by: Danny Kopping <dannykopping@gmail.com>	3 years ago
Ed Welch	6d54505485	Loki: add `__line__` and `__timestamp__` to label_format (#6983 ) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> What this PR does / why we need it: We have the `__line__` and `__timestamp__` functions currently available in the `line_format` stage but not in the `label_format` stage. This PR fixes that. Specifically `__timestamp__` can be valuable in a label because you can do things like ``` label_format hour_of_day=`{{__timestamp__ \| date "15"}}` ``` To be able to conditionaly select log lines based on the hour of the day they were created, this can be useful for conditional alerting as an example. Which issue(s) this PR fixes: Fixes #6402 Special notes for your reviewer: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> Checklist - [x] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Karen Miller	a02ce69a98	Docs: add minimal docs explaining how to use the LogQL Analyzer (#6900 )	3 years ago
Vladyslav Diachenko	b494713106	fixed typo in API host (#6892 )	3 years ago
Vladyslav Diachenko	bec014dd86	Logql analyzer page cherry picks (#6889 ) * added Logql Analyzer page into loki docs * changed template to support changes in API and supported the case when query is run against empty line * added more json examples, changed some labels and changes css for line numbers * fixed last comments from Karen and pointed to production API	3 years ago
Karen Miller	8330ff2c61	Docs: Revise location of some LogQL examples; improve prose (#6751 ) * Docs: Revise location of some LogQL examples; improve prose * Update docs/sources/logql/query_examples.md Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> * Update docs/sources/logql/metric_queries.md Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * Revise per review comments. Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Andrii Dembitskyi	ab2d50a2b2	Typo in 'trimPrefix' function signature (#6733 )	3 years ago
Ed Welch	33761194d9	Loki: Add the ability to use the log line timestamp in label and line formatter (#6517 ) * Change Stage interface Process method to include the timestamp of the line Signed-off-by: Edward Welch <edward.welch@grafana.com> * allow using log line timestamp in formatter Signed-off-by: Edward Welch <edward.welch@grafana.com> * update formatting in test Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Christian Haudum	a56a88bd35	Add unwrapped aggregation `rate_counter()` (#6412 ) `rate_counter()` takes an unwrapped range as input, but unlike `rate()` it treats the values extracted from the log lines as "counter metric" like in Prometheus' `rate()` function. This is a replacement for the reverted change of #5013 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Christian Haudum	b315ed0ddf	Sum values in unwrapped rate aggregation instead of treating them as counter (#6361 ) * Revert unwrapped rate aggregation to previous implementation This PR reverts the implementation done in #5013 to the original implementation that sums the extracted values from the log lines instead of treating them like a Prometheus counter metric. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Move changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove unused/dead code Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Clean changelog Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Bryan Boreham	dc185f6ac2	Fix wording about pattern parser (#6264 ) The last sentence was saying the opposite of what it meant. Also fix 'start' to match singluar 'expression'.	4 years ago
Richard Sartor	eedfcd5170	Fix the signature for substr (#6236 )	4 years ago
Christian Haudum	d5c337c4d3	docs: Fix broken link from unpack parser to pack stage (#6207 ) and make hyperlinks more obvious Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Aditya C S	c6af640798	feat: support simple selection of labels in json expr parser (#6151 )	4 years ago
Justin Burnham	97c7d7ca3c	LogQL: add the `default` sprig template function to logql label/line formatter (#6163 ) * add the `default` sprig template function to logql label/line formatter With the `default` sprig template function, we will be able to make it easier to output a default value if the source string is otherwise empty. Useful for json fields that can be missing, like HTTP headers in a log line that aren't required as in the following: ``` {job="access_log"} \| json \| line_format `{{.http_request_headers_x_forwarded_for \| default "-"}}` ``` Previously this could only be done (that I found) with the much more verbose ``` {{regexReplaceAllLiteral "^$" .http_request_headers_x_forwarded_for "-"}} ``` * update changelog entry * add pr number * enhance docs * Update docs/sources/logql/template_functions.md Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	4 years ago
psytester	40b2373398	corrected copy&paste error in list of labels (#6146 ) response_size was twice mentioned	4 years ago
Karen Miller	2264b73311	Docs: remove Cortex references (#6079 ) * Docs: remove Cortex references * Bug fix: (after PR 6015) Cortex to Loki update	4 years ago
Alexandre de Verteuil	b08ddc5f93	Minor fix in docs/sources/logql/log_queries.md (#5868 )	4 years ago
Febrian Setianto	8d7dbc3e13	Docs: Fix broken link to "Matching IP addresses" (#5550 )	4 years ago
Ivana Huckova	2eebdff2e2	Update metric_queries.md (#5525 ) Provided query is incorrect as it has extra `)`. This PR fixes it.	4 years ago
Ivana Huckova	7320cb27ad	Update _index.md (#5458 )	4 years ago
Kaviraj	e5c514359b	Fix broken `template_variable` doc link (#4960 )	4 years ago
Cyril Tovena	21e61315df	LogQL: add __line__ function to \| line_format template (#4879 ) * LogQL: add __line__ function to \| line_format template Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * add changelog entry Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Clone the function map to avoid a race between multiple expr Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Cyril Tovena	1253d5c1b7	Update _index.md (#4893 )	4 years ago
Karen Miller	35ebe967ea	Docs: fix broken links (#4770 ) * Docs: fix broken links * Docs: fix another link name	4 years ago
Ed Welch	3fef08ca1a	2.4 release notes, changelog, upgrade guide (#4658 ) * update changelog, release notes, and upgrade guide for 2.4 * tweak release notes a little * make path relative for kafka config * update upgrade guide * more docs fix the cache size not applied to results cache. * fix prefix name in ring_config	4 years ago
Garrett	1c7df620c4	add date time sprig template functions in logql label/line formatter (#4603 ) * add date time sprig template functions in logql label/line formatter With date time sprig template functions, we will be able to filter test jobs which create time is 1 day before as the following: ``` {job="test"} \| label_format nowEpoch=`{{(unixEpoch now)}}`,createDateEpoch=`{{unixEpoch (toDate "2006-01-02" .createDate)}}` \| label_format dateTimeDiff="{{sub .nowEpoch .createDateEpoch}}" \| dateTimeDiff > 86400 ``` * add documents for date time sprig template functions * resolve comments * add golang datetime layout doc reference * update CHANGELOG.md accordingly to include this PR	4 years ago

1 2

93 Commits (ec516facaf9e3f855855df0971d96e65f5faf022)