Add function label_replace. (#3047)

You might be wondering why do we need label_replace when we already have label_format.
Well for two reasons:

1. It's nice to have something similar to Prometheus, so users building metric queries will feel right at home.
2. Interestingly, they are quite different in how they execute. label_format executes for every log line, however label_replace execute for every sample. Since you aggregate log lines into sample then label_replace is way more efficient at scale.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
pull/3053/head^2
Cyril Tovena 5 years ago committed by GitHub
parent 4e7a123c77
commit 3f0800dc44
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 6
      docs/sources/logql/_index.md
  2. 369
      docs/sources/logql/functions.md
  3. 373
      docs/sources/logql/template_functions.md
  4. 58
      pkg/logql/ast.go
  5. 14
      pkg/logql/ast_test.go
  6. 69
      pkg/logql/engine_test.go
  7. 50
      pkg/logql/evaluator.go
  8. 11
      pkg/logql/expr.y
  9. 925
      pkg/logql/expr.y.go
  10. 39
      pkg/logql/lex.go
  11. 162
      pkg/logql/parser_test.go

@ -295,7 +295,7 @@ Will extract and rewrite the log line to only contains the query and the duratio
You can use double quoted string for the template or backticks `` `{{.label_name}}` `` to avoid the need to escape special characters.
See [functions](functions/) to learn about available functions in the template format.
See [template functions](template_functions/) to learn about available functions in the template format.
#### Labels Format Expression
@ -514,6 +514,10 @@ Get the rate of HTTP GET of /home requests from NGINX logs by region:
avg(rate(({job="nginx"} |= "GET" | json | path="/home")[10s])) by (region)
```
### Functions
Loki supports several functions to operate on data. These are described in detail in the expression language [functions](functions/) page.
### Binary Operators
#### Arithmetic Binary Operators

@ -1,370 +1,15 @@
---
title: Template functions
title: Functions
---
The [text template](https://golang.org/pkg/text/template) format used in `| line_format` and `| label_format` support the usage of functions.
# Functions
All labels are added as variables in the template engine. They can be referenced using they label name prefixed by a `.`(e.g `.label_name`). For example the following template will output the value of the path label:
## label_replace()
```template
{{ .path }}
```
You can take advantage of [pipeline](https://golang.org/pkg/text/template/#hdr-Pipelines) to join together multiple functions.
In a chained pipeline, the result of each command is passed as the last argument of the following command.
Example:
```template
{{ .path | replace " " "_" | trunc 5 | upper }}
```
## ToLower and ToUpper
This function converts the entire string to lowercase or uppercase.
Signatures:
- `ToLower(string) string`
- `ToUpper(string) string`
Examples:
```template
"{{.request_method | ToLower}}"
"{{.request_method | ToUpper}}"
`{{ToUpper "This is a string" | ToLower}}`
```
> **Note:** In Loki 2.1 you can also use respectively [`lower`](#lower) and [`upper`](#upper) shortcut, e.g `{{.request_method | lower }}`.
## Replace string
> **Note:** In Loki 2.1 [`replace`](#replace) (as opposed to `Replace`) is available with a different signature but easier to chain within pipeline.
Use this function to perform a simple string replacement.
Signature:
`Replace(s, old, new string, n int) string`
It takes four arguments:
- `s` source string
- `old` string to replace
- `new` string to replace with
- `n` the maximun amount of replacement (-1 for all)
Example:
```template
`{{ Replace "This is a string" " " "-" -1 }}`
```
The results in `This-is-a-string`.
## Trim, TrimLeft, TrimRight, and TrimSpace
> **Note:** In Loki 2.1 [trim](#trim), [trimAll](#trimAll), [trimSuffix](#trimSuffix) and [trimPrefix](trimPrefix) have been added with a different signature for better pipeline chaining.
`Trim` returns a slice of the string s with all leading and
trailing Unicode code points contained in cutset removed.
Signature: `Trim(value, cutset string) string`
`TrimLeft` and `TrimRight` are the same as `Trim` except that it trims only leading and trailing characters respectively.
```template
`{{ Trim .query ",. " }}`
`{{ TrimLeft .uri ":" }}`
`{{ TrimRight .path "/" }}`
```
`TrimSpace` TrimSpace returns string s with all leading
and trailing white space removed, as defined by Unicode.
Signature: `TrimSpace(value string) string`
```template
{{ TrimSpace .latency }}
```
`TrimPrefix` and `TrimSuffix` will trim respectively the prefix or suffix supplied.
Signature:
- `TrimPrefix(value string, prefix string) string`
- `TrimSuffix(value string, suffix string) string`
```template
{{ TrimPrefix .path "/" }}
```
## regexReplaceAll and regexReplaceAllLiteral
`regexReplaceAll` returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement. Inside string replacement, $ signs are interpreted as in Expand, so for instance $1 represents the text of the first sub-match. See the golang [Regexp.replaceAll documentation](https://golang.org/pkg/regexp/#Regexp.ReplaceAll) for more examples.
```template
`{{ regexReplaceAllLiteral "(a*)bc" .some_label "${1}a" }}`
```
`regexReplaceAllLiteral` function returns a copy of the input string and replaces matches of the Regexp with the replacement string replacement. The replacement string is substituted directly, without using Expand.
```template
`{{ regexReplaceAllLiteral "(ts=)" .timestamp "timestamp=" }}`
```
You can combine multiple functions using pipe. For example, to strip out spaces and make the request method in capital, you would write the following template: `{{ .request_method | TrimSpace | ToUpper }}`.
## lower
> Added in Loki 2.1
Use this function to convert to lower case.
Signature:
`lower(string) string`
Examples:
```template
"{{ .request_method | lower }}"
`{{ lower "HELLO"}}`
```
The last example will return `hello`.
## upper
> Added in Loki 2.1
Use this function to convert to upper case.
Signature:
`upper(string) string`
Examples:
```template
"{{ .request_method | upper }}"
`{{ upper "hello"}}`
```
This results in `HELLO`.
## title
> **Note:** Added in Loki 2.1.
Convert to title case.
Signature:
`title(string) string`
Examples:
```template
"{{.request_method | title}}"
`{{ title "hello world"}}`
```
The last example will return `Hello World`.
## trunc
> **Note:** Added in Loki 2.1.
Truncate a string and add no suffix.
Signature:
`trunc(count int,value string) string`
Examples:
```template
"{{ .path | trunc 2 }}"
`{{ trunc 5 "hello world"}}` // output: hello
`{{ trunc -5 "hello world"}}` // output: world
```
## substr
> **Note:** Added in Loki 2.1.
Get a substring from a string.
Signature:
`trunc(start int,end int,value string) string`
If start is < 0, this calls value[:end].
If start is >= 0 and end < 0 or end bigger than s length, this calls value[start:]
Otherwise, this calls value[start, end].
Examples:
```template
"{{ .path | substr 2 5 }}"
`{{ substr 0 5 "hello world"}}` // output: hello
`{{ substr 6 11 "hello world"}}` // output: world
```
## replace
> **Note:** Added in Loki 2.1.
This function performs simple string replacement.
Signature: `replace(old string, new string, src string) string`
It takes three arguments:
- `old` string to replace
- `new` string to replace with
- `src` source string
Examples:
```template
{{ .cluster | replace "-cluster" "" }}
{{ replace "hello" "world" "hello world" }}
```
The last example will return `world world`.
## trim
> **Note:** Added in Loki 2.1.
The trim function removes space from either side of a string.
Signature: `trim(string) string`
Examples:
```template
{{ .ip | trim }}
{{ trim " hello " }} // output: hello
```
## trimAll
> **Note:** Added in Loki 2.1.
Use this function to remove given characters from the front or back of a string.
Signature: `trimAll(chars string,src string) string`
Examples:
```template
{{ .path | trimAll "/" }}
{{ trimAll "$" "$5.00" }} // output: 5.00
```
## trimSuffix
> **Note:** Added in Loki 2.1.
Use this function to trim just the suffix from a string.
Signature: `trimSuffix(suffix string, src string) string`
Examples:
```template
{{ .path | trimSuffix "/" }}
{{ trimSuffix "-" "hello-" }} // output: hello
```
## trimPrefix
> **Note:** Added in Loki 2.1.
Use this function to trim just the prefix from a string.
Signature: `trimPrefix(suffix string, src string) string`
Examples:
```template
{{ .path | trimPrefix "/" }}
{{ trimPrefix "-" "-hello" }} // output: hello
```
## indent
> **Note:** Added in Loki 2.1.
The indent function indents every line in a given string to the specified indent width. This is useful when aligning multi-line strings.
Signature: `indent(spaces int,src string) string`
```template
{{ indent 4 .query }}
```
This indents each line contained in the `.query` by four (4) spaces.
## nindent
> **Note:** Added in Loki 2.1.
The nindent function is the same as the indent function, but prepends a new line to the beginning of the string.
Signature: `nindent(spaces int,src string) string`
```template
{{ nindent 4 .query }}
```
This will indent every line of text by 4 space characters and add a new line to the beginning.
## repeat
> **Note:** Added in Loki 2.1.
Use this function to repeat a string multiple times.
Signature: `repeat(c int,value string) string`
```template
{{ repeat 3 "hello" }} // output: hellohellohello
```
## contains
> **Note:** Added in Loki 2.1.
Use this function to test to see if one string is contained inside of another.
Signature: `contains(s string, src string) bool`
Examples:
```template
{{ if .err contains "ErrTimeout" }} timeout {{end}}
{{ if contains "he" "hello" }} yes {{end}}
```
## hasPrefix and hasSuffix
> **Note:** Added in Loki 2.1.
The `hasPrefix` and `hasSuffix` functions test whether a string has a given prefix or suffix.
Signatures:
- `hasPrefix(prefix string, src string) bool`
- `hasSuffix(suffix string, src string) bool`
For each timeseries in `v`, `label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string)` matches the regular expression `regex` against the label `src_label`. If it matches, then the timeseries is returned with the label `dst_label` replaced by the expansion of `replacement`. `$1` is replaced with the first matching subgroup, `$2` with the second etc. If the regular expression doesn't match then the timeseries is returned unchanged.
Examples:
This example will return a vector with each time series having a `foo` label with the value `a` added to it:
```template
{{ if .err hasSuffix "Timeout" }} timeout {{end}}
{{ if hasPrefix "he" "hello" }} yes {{end}}
```logql
label_replace(rate({job="api-server",service="a:c"} |= "err" [1m]), "foo", "$1", "service", "(.*):.*")
```

@ -0,0 +1,373 @@
---
title: Template functions
---
# Template functions
The [text template](https://golang.org/pkg/text/template) format used in `| line_format` and `| label_format` support the usage of functions.
All labels are added as variables in the template engine. They can be referenced using they label name prefixed by a `.`(e.g `.label_name`). For example the following template will output the value of the path label:
```template
{{ .path }}
```
You can take advantage of [pipeline](https://golang.org/pkg/text/template/#hdr-Pipelines) to join together multiple functions.
In a chained pipeline, the result of each command is passed as the last argument of the following command.
Example:
```template
{{ .path | replace " " "_" | trunc 5 | upper }}
```
## ToLower and ToUpper
This function converts the entire string to lowercase or uppercase.
Signatures:
- `ToLower(string) string`
- `ToUpper(string) string`
Examples:
```template
"{{.request_method | ToLower}}"
"{{.request_method | ToUpper}}"
`{{ToUpper "This is a string" | ToLower}}`
```
> **Note:** In Loki 2.1 you can also use respectively [`lower`](#lower) and [`upper`](#upper) shortcut, e.g `{{.request_method | lower }}`.
## Replace string
> **Note:** In Loki 2.1 [`replace`](#replace) (as opposed to `Replace`) is available with a different signature but easier to chain within pipeline.
Use this function to perform a simple string replacement.
Signature:
`Replace(s, old, new string, n int) string`
It takes four arguments:
- `s` source string
- `old` string to replace
- `new` string to replace with
- `n` the maximun amount of replacement (-1 for all)
Example:
```template
`{{ Replace "This is a string" " " "-" -1 }}`
```
The results in `This-is-a-string`.
## Trim, TrimLeft, TrimRight, and TrimSpace
> **Note:** In Loki 2.1 [trim](#trim), [trimAll](#trimAll), [trimSuffix](#trimSuffix) and [trimPrefix](trimPrefix) have been added with a different signature for better pipeline chaining.
`Trim` returns a slice of the string s with all leading and
trailing Unicode code points contained in cutset removed.
Signature: `Trim(value, cutset string) string`
`TrimLeft` and `TrimRight` are the same as `Trim` except that it trims only leading and trailing characters respectively.
```template
`{{ Trim .query ",. " }}`
`{{ TrimLeft .uri ":" }}`
`{{ TrimRight .path "/" }}`
```
`TrimSpace` TrimSpace returns string s with all leading
and trailing white space removed, as defined by Unicode.
Signature: `TrimSpace(value string) string`
```template
{{ TrimSpace .latency }}
```
`TrimPrefix` and `TrimSuffix` will trim respectively the prefix or suffix supplied.
Signature:
- `TrimPrefix(value string, prefix string) string`
- `TrimSuffix(value string, suffix string) string`
```template
{{ TrimPrefix .path "/" }}
```
## regexReplaceAll and regexReplaceAllLiteral
`regexReplaceAll` returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement. Inside string replacement, $ signs are interpreted as in Expand, so for instance $1 represents the text of the first sub-match. See the golang [Regexp.replaceAll documentation](https://golang.org/pkg/regexp/#Regexp.ReplaceAll) for more examples.
```template
`{{ regexReplaceAllLiteral "(a*)bc" .some_label "${1}a" }}`
```
`regexReplaceAllLiteral` function returns a copy of the input string and replaces matches of the Regexp with the replacement string replacement. The replacement string is substituted directly, without using Expand.
```template
`{{ regexReplaceAllLiteral "(ts=)" .timestamp "timestamp=" }}`
```
You can combine multiple functions using pipe. For example, to strip out spaces and make the request method in capital, you would write the following template: `{{ .request_method | TrimSpace | ToUpper }}`.
## lower
> Added in Loki 2.1
Use this function to convert to lower case.
Signature:
`lower(string) string`
Examples:
```template
"{{ .request_method | lower }}"
`{{ lower "HELLO"}}`
```
The last example will return `hello`.
## upper
> Added in Loki 2.1
Use this function to convert to upper case.
Signature:
`upper(string) string`
Examples:
```template
"{{ .request_method | upper }}"
`{{ upper "hello"}}`
```
This results in `HELLO`.
## title
> **Note:** Added in Loki 2.1.
Convert to title case.
Signature:
`title(string) string`
Examples:
```template
"{{.request_method | title}}"
`{{ title "hello world"}}`
```
The last example will return `Hello World`.
## trunc
> **Note:** Added in Loki 2.1.
Truncate a string and add no suffix.
Signature:
`trunc(count int,value string) string`
Examples:
```template
"{{ .path | trunc 2 }}"
`{{ trunc 5 "hello world"}}` // output: hello
`{{ trunc -5 "hello world"}}` // output: world
```
## substr
> **Note:** Added in Loki 2.1.
Get a substring from a string.
Signature:
`trunc(start int,end int,value string) string`
If start is < 0, this calls value[:end].
If start is >= 0 and end < 0 or end bigger than s length, this calls value[start:]
Otherwise, this calls value[start, end].
Examples:
```template
"{{ .path | substr 2 5 }}"
`{{ substr 0 5 "hello world"}}` // output: hello
`{{ substr 6 11 "hello world"}}` // output: world
```
## replace
> **Note:** Added in Loki 2.1.
This function performs simple string replacement.
Signature: `replace(old string, new string, src string) string`
It takes three arguments:
- `old` string to replace
- `new` string to replace with
- `src` source string
Examples:
```template
{{ .cluster | replace "-cluster" "" }}
{{ replace "hello" "world" "hello world" }}
```
The last example will return `world world`.
## trim
> **Note:** Added in Loki 2.1.
The trim function removes space from either side of a string.
Signature: `trim(string) string`
Examples:
```template
{{ .ip | trim }}
{{ trim " hello " }} // output: hello
```
## trimAll
> **Note:** Added in Loki 2.1.
Use this function to remove given characters from the front or back of a string.
Signature: `trimAll(chars string,src string) string`
Examples:
```template
{{ .path | trimAll "/" }}
{{ trimAll "$" "$5.00" }} // output: 5.00
```
## trimSuffix
> **Note:** Added in Loki 2.1.
Use this function to trim just the suffix from a string.
Signature: `trimSuffix(suffix string, src string) string`
Examples:
```template
{{ .path | trimSuffix "/" }}
{{ trimSuffix "-" "hello-" }} // output: hello
```
## trimPrefix
> **Note:** Added in Loki 2.1.
Use this function to trim just the prefix from a string.
Signature: `trimPrefix(suffix string, src string) string`
Examples:
```template
{{ .path | trimPrefix "/" }}
{{ trimPrefix "-" "-hello" }} // output: hello
```
## indent
> **Note:** Added in Loki 2.1.
The indent function indents every line in a given string to the specified indent width. This is useful when aligning multi-line strings.
Signature: `indent(spaces int,src string) string`
```template
{{ indent 4 .query }}
```
This indents each line contained in the `.query` by four (4) spaces.
## nindent
> **Note:** Added in Loki 2.1.
The nindent function is the same as the indent function, but prepends a new line to the beginning of the string.
Signature: `nindent(spaces int,src string) string`
```template
{{ nindent 4 .query }}
```
This will indent every line of text by 4 space characters and add a new line to the beginning.
## repeat
> **Note:** Added in Loki 2.1.
Use this function to repeat a string multiple times.
Signature: `repeat(c int,value string) string`
```template
{{ repeat 3 "hello" }} // output: hellohellohello
```
## contains
> **Note:** Added in Loki 2.1.
Use this function to test to see if one string is contained inside of another.
Signature: `contains(s string, src string) bool`
Examples:
```template
{{ if .err contains "ErrTimeout" }} timeout {{end}}
{{ if contains "he" "hello" }} yes {{end}}
```
## hasPrefix and hasSuffix
> **Note:** Added in Loki 2.1.
The `hasPrefix` and `hasSuffix` functions test whether a string has a given prefix or suffix.
Signatures:
- `hasPrefix(prefix string, src string) bool`
- `hasSuffix(suffix string, src string) bool`
Examples:
```template
{{ if .err hasSuffix "Timeout" }} timeout {{end}}
{{ if hasPrefix "he" "hello" }} yes {{end}}
```

@ -3,6 +3,7 @@ package logql
import (
"context"
"fmt"
"regexp"
"strconv"
"strings"
"time"
@ -524,6 +525,8 @@ const (
OpConvBytes = "bytes"
OpConvDuration = "duration"
OpConvDurationSeconds = "duration_seconds"
OpLabelReplace = "label_replace"
)
func IsComparisonOperator(op string) bool {
@ -893,3 +896,58 @@ func formatOperation(op string, grouping *grouping, params ...string) string {
sb.WriteString(")")
return sb.String()
}
type labelReplaceExpr struct {
left SampleExpr
dst string
replacement string
src string
regex string
re *regexp.Regexp
implicit
}
func mustNewLabelReplaceExpr(left SampleExpr, dst, replacement, src, regex string) *labelReplaceExpr {
re, err := regexp.Compile("^(?:" + regex + ")$")
if err != nil {
panic(newParseError(fmt.Sprintf("invalid regex in label_replace: %s", err.Error()), 0, 0))
}
return &labelReplaceExpr{
left: left,
dst: dst,
replacement: replacement,
src: src,
re: re,
regex: regex,
}
}
func (e *labelReplaceExpr) Selector() LogSelectorExpr {
return e.left.Selector()
}
func (e *labelReplaceExpr) Extractor() (SampleExtractor, error) {
return e.left.Extractor()
}
func (e *labelReplaceExpr) Operations() []string {
return e.left.Operations()
}
func (e *labelReplaceExpr) String() string {
var sb strings.Builder
sb.WriteString(OpLabelReplace)
sb.WriteString("(")
sb.WriteString(e.left.String())
sb.WriteString(",")
sb.WriteString(strconv.Quote(e.dst))
sb.WriteString(",")
sb.WriteString(strconv.Quote(e.replacement))
sb.WriteString(",")
sb.WriteString(strconv.Quote(e.src))
sb.WriteString(",")
sb.WriteString(strconv.Quote(e.regex))
sb.WriteString(")")
return sb.String()
}

@ -108,6 +108,20 @@ func Test_SampleExpr_String(t *testing.T) {
/
count_over_time({namespace="tns"} | logfmt | label_format foo=bar[5m])
)`,
`label_replace(
sum by (job) (
sum_over_time(
{namespace="tns"} |= "level=error" | json | avg=5 and bar<25ms | unwrap duration(latency) | __error__!~".*" [5m]
)
/
count_over_time({namespace="tns"} | logfmt | label_format foo=bar[5m])
),
"foo",
"$1",
"service",
"(.*):.*"
)
`,
} {
t.Run(tc, func(t *testing.T) {
expr, err := ParseExpr(tc)

@ -221,6 +221,41 @@ func TestEngine_LogsInstantQuery(t *testing.T) {
},
},
},
{
`label_replace(
sum(count_over_time({app=~"foo|bar"} |~".+bar" [1m])) by (namespace,app),
"new",
"$1",
"app",
"f(.*)"
)`, time.Unix(60, 0), logproto.FORWARD, 100,
[][]logproto.Series{
{
newSeries(testSize, factor(10, identity), `{app="foo", namespace="a"}`),
newSeries(testSize, factor(10, identity), `{app="bar", namespace="b"}`),
},
},
[]SelectSampleParams{
{&logproto.SampleQueryRequest{Start: time.Unix(0, 0), End: time.Unix(60, 0), Selector: `sum by (namespace,app) (count_over_time({app=~"foo|bar"} |~".+bar" [1m])) `}},
},
promql.Vector{
promql.Sample{
Point: promql.Point{T: 60 * 1000, V: 6},
Metric: labels.Labels{
labels.Label{Name: "app", Value: "bar"},
labels.Label{Name: "namespace", Value: "b"},
},
},
promql.Sample{
Point: promql.Point{T: 60 * 1000, V: 6},
Metric: labels.Labels{
labels.Label{Name: "app", Value: "foo"},
labels.Label{Name: "namespace", Value: "a"},
labels.Label{Name: "new", Value: "oo"},
},
},
},
},
{
`count(count_over_time({app=~"foo|bar"} |~".+bar" [1m])) without (app)`, time.Unix(60, 0), logproto.FORWARD, 100,
[][]logproto.Series{
@ -1295,6 +1330,40 @@ func TestEngine_RangeQuery(t *testing.T) {
},
},
},
{
`label_replace(
avg by (app) (
sum by (app) (rate({app=~"foo|bar"} |~".+bar" [1m])) +
sum by (app) (rate({app=~"foo|bar"} |~".+bar" [1m])) /
sum by (app) (rate({app=~"foo|bar"} |~".+bar" [1m]))
) * 2,
"new",
"$1",
"app",
"f(.*)"
)
`,
time.Unix(60, 0), time.Unix(180, 0), 30 * time.Second, 0, logproto.FORWARD, 100,
[][]logproto.Series{
{
newSeries(testSize, factor(5, identity), `{app="foo"}`),
newSeries(testSize, factor(5, identity), `{app="bar"}`),
},
},
[]SelectSampleParams{
{&logproto.SampleQueryRequest{Start: time.Unix(0, 0), End: time.Unix(180, 0), Selector: `sum by (app) (rate({app=~"foo|bar"} |~".+bar" [1m]))`}},
},
promql.Matrix{
promql.Series{
Metric: labels.Labels{{Name: "app", Value: "bar"}},
Points: []promql.Point{{T: 60 * 1000, V: 2.4}, {T: 90 * 1000, V: 2.4}, {T: 120 * 1000, V: 2.4}, {T: 150 * 1000, V: 2.4}, {T: 180 * 1000, V: 2.4}},
},
promql.Series{
Metric: labels.Labels{{Name: "app", Value: "foo"}, {Name: "new", Value: "oo"}},
Points: []promql.Point{{T: 60 * 1000, V: 2.4}, {T: 90 * 1000, V: 2.4}, {T: 120 * 1000, V: 2.4}, {T: 150 * 1000, V: 2.4}, {T: 180 * 1000, V: 2.4}},
},
},
},
{
` sum (
sum by (app) (rate({app=~"foo|bar"} |~".+bar" [1m])) +

@ -206,6 +206,8 @@ func (ev *DefaultEvaluator) StepEvaluator(
return rangeAggEvaluator(iter.NewPeekingSampleIterator(it), e, q)
case *binOpExpr:
return binOpStepEvaluator(ctx, nextEv, e, q)
case *labelReplaceExpr:
return labelReplaceEvaluator(ctx, nextEv, e, q)
default:
return nil, EvaluatorUnsupportedType(e, ev)
}
@ -898,3 +900,51 @@ func literalStepEvaluator(
eval.Error,
)
}
func labelReplaceEvaluator(
ctx context.Context,
ev SampleEvaluator,
expr *labelReplaceExpr,
q Params,
) (StepEvaluator, error) {
nextEvaluator, err := ev.StepEvaluator(ctx, ev, expr.left, q)
if err != nil {
return nil, err
}
buf := make([]byte, 0, 1024)
var labelCache map[uint64]labels.Labels
return newStepEvaluator(func() (bool, int64, promql.Vector) {
next, ts, vec := nextEvaluator.Next()
if !next {
return false, 0, promql.Vector{}
}
if labelCache == nil {
labelCache = make(map[uint64]labels.Labels, len(vec))
}
var hash uint64
for i, s := range vec {
hash, buf = s.Metric.HashWithoutLabels(buf)
if labels, ok := labelCache[hash]; ok {
vec[i].Metric = labels
continue
}
src := s.Metric.Get(expr.src)
indexes := expr.re.FindStringSubmatchIndex(src)
if indexes == nil {
// If there is no match, no replacement should take place.
labelCache[hash] = s.Metric
continue
}
res := expr.re.ExpandString([]byte{}, expr.replacement, src, indexes)
lb := labels.NewBuilder(s.Metric).Del(expr.dst)
if len(res) > 0 {
lb.Set(expr.dst, string(res))
}
outLbs := lb.Labels()
labelCache[hash] = outLbs
vec[i].Metric = outLbs
}
return next, ts, vec
}, nextEvaluator.Close, nextEvaluator.Error)
}

@ -26,6 +26,7 @@ import (
MetricExpr SampleExpr
VectorOp string
BinOpExpr SampleExpr
LabelReplaceExpr SampleExpr
binOp string
bytes uint64
str string
@ -67,6 +68,7 @@ import (
%type <VectorOp> vectorOp
%type <BinOpExpr> binOpExpr
%type <LiteralExpr> literalExpr
%type <LabelReplaceExpr> labelReplaceExpr
%type <BinOpModifier> binOpModifier
%type <LabelParser> labelParser
%type <PipelineExpr> pipelineExpr
@ -81,7 +83,7 @@ import (
%type <LabelFormat> labelFormat
%type <LabelsFormat> labelsFormat
%type <UnwrapExpr> unwrapExpr
%type <UnitFilter> unitFilter
%type <UnitFilter> unitFilter
%token <bytes> BYTES
%token <str> IDENTIFIER STRING NUMBER
@ -90,6 +92,7 @@ import (
OPEN_PARENTHESIS CLOSE_PARENTHESIS BY WITHOUT COUNT_OVER_TIME RATE SUM AVG MAX MIN COUNT STDDEV STDVAR BOTTOMK TOPK
BYTES_OVER_TIME BYTES_RATE BOOL JSON REGEXP LOGFMT PIPE LINE_FMT LABEL_FMT UNWRAP AVG_OVER_TIME SUM_OVER_TIME MIN_OVER_TIME
MAX_OVER_TIME STDVAR_OVER_TIME STDDEV_OVER_TIME QUANTILE_OVER_TIME BYTES_CONV DURATION_CONV DURATION_SECONDS_CONV
LABEL_REPLACE
// Operators are listed with increasing precedence.
%left <binOp> OR
@ -113,6 +116,7 @@ metricExpr:
| vectorAggregationExpr { $$ = $1 }
| binOpExpr { $$ = $1 }
| literalExpr { $$ = $1 }
| labelReplaceExpr { $$ = $1 }
| OPEN_PARENTHESIS metricExpr CLOSE_PARENTHESIS { $$ = $2 }
;
@ -168,6 +172,11 @@ vectorAggregationExpr:
| vectorOp OPEN_PARENTHESIS NUMBER COMMA metricExpr CLOSE_PARENTHESIS grouping { $$ = mustNewVectorAggregationExpr($5, $1, $7, &$3) }
;
labelReplaceExpr:
LABEL_REPLACE OPEN_PARENTHESIS metricExpr COMMA STRING COMMA STRING COMMA STRING COMMA STRING CLOSE_PARENTHESIS
{ $$ = mustNewLabelReplaceExpr($3, $5, $7, $9, $11)}
;
filter:
PIPE_MATCH { $$ = labels.MatchRegexp }
| PIPE_EXACT { $$ = labels.MatchEqual }

File diff suppressed because it is too large Load Diff

@ -12,25 +12,26 @@ import (
)
var tokens = map[string]int{
",": COMMA,
".": DOT,
"{": OPEN_BRACE,
"}": CLOSE_BRACE,
"=": EQ,
OpTypeNEQ: NEQ,
"=~": RE,
"!~": NRE,
"|=": PIPE_EXACT,
"|~": PIPE_MATCH,
OpPipe: PIPE,
OpUnwrap: UNWRAP,
"(": OPEN_PARENTHESIS,
")": CLOSE_PARENTHESIS,
"by": BY,
"without": WITHOUT,
"bool": BOOL,
"[": OPEN_BRACKET,
"]": CLOSE_BRACKET,
",": COMMA,
".": DOT,
"{": OPEN_BRACE,
"}": CLOSE_BRACE,
"=": EQ,
OpTypeNEQ: NEQ,
"=~": RE,
"!~": NRE,
"|=": PIPE_EXACT,
"|~": PIPE_MATCH,
OpPipe: PIPE,
OpUnwrap: UNWRAP,
"(": OPEN_PARENTHESIS,
")": CLOSE_PARENTHESIS,
"by": BY,
"without": WITHOUT,
"bool": BOOL,
"[": OPEN_BRACKET,
"]": CLOSE_BRACKET,
OpLabelReplace: LABEL_REPLACE,
// binops
OpTypeOr: OR,

@ -185,6 +185,32 @@ func TestParse(t *testing.T) {
groups: []string{"bar", "foo"},
}, nil),
},
{
in: `avg(
label_replace(
count_over_time({ foo !~ "bar" }[5h]),
"bar",
"$1$2",
"foo",
"(.*).(.*)"
)
) by (bar,foo)`,
exp: mustNewVectorAggregationExpr(
mustNewLabelReplaceExpr(
&rangeAggregationExpr{
left: &logRange{
left: &matchersExpr{matchers: []*labels.Matcher{mustNewMatcher(labels.MatchNotRegexp, "foo", "bar")}},
interval: 5 * time.Hour,
},
operation: "count_over_time",
},
"bar", "$1$2", "foo", "(.*).(.*)",
),
"avg", &grouping{
without: false,
groups: []string{"bar", "foo"},
}, nil),
},
{
in: `avg(count_over_time({ foo !~ "bar" }[5h])) by ()`,
exp: mustNewVectorAggregationExpr(&rangeAggregationExpr{
@ -283,6 +309,22 @@ func TestParse(t *testing.T) {
col: 22,
},
},
{
in: `label_replace(rate({ foo !~ "bar" }[5m]),"")`,
err: ParseError{
msg: `syntax error: unexpected ), expecting ,`,
line: 1,
col: 44,
},
},
{
in: `label_replace(rate({ foo !~ "bar" }[5m]),"foo","$1","bar","^^^^x43\\q")`,
err: ParseError{
msg: "invalid regex in label_replace: error parsing regexp: invalid escape sequence: `\\q`",
line: 0,
col: 0,
},
},
{
in: `rate({ foo !~ "bar" }[5)`,
err: ParseError{
@ -406,6 +448,39 @@ func TestParse(t *testing.T) {
interval: 5 * time.Minute,
}, OpRangeTypeBytes, nil, nil),
},
{
in: `
label_replace(
bytes_over_time(({foo="bar"} |= "baz" |~ "blip" != "flip" !~ "flap")[5m]),
"buzz",
"$2",
"bar",
"(.*):(.*)"
)
`,
exp: mustNewLabelReplaceExpr(
newRangeAggregationExpr(
&logRange{
left: newPipelineExpr(
newMatcherExpr([]*labels.Matcher{mustNewMatcher(labels.MatchEqual, "foo", "bar")}),
MultiStageExpr{
newLineFilterExpr(
newLineFilterExpr(
newLineFilterExpr(
newLineFilterExpr(nil, labels.MatchEqual, "baz"),
labels.MatchRegexp, "blip"),
labels.MatchNotEqual, "flip"),
labels.MatchNotRegexp, "flap"),
},
),
interval: 5 * time.Minute,
}, OpRangeTypeBytes, nil, nil),
"buzz",
"$2",
"bar",
"(.*):(.*)",
),
},
{
in: `sum(count_over_time(({foo="bar"} |= "baz" |~ "blip" != "flip" !~ "flap")[5m])) by (foo)`,
exp: mustNewVectorAggregationExpr(newRangeAggregationExpr(
@ -1760,6 +1835,93 @@ func TestParse(t *testing.T) {
),
),
},
{
in: `
label_replace(
sum by (foo,bar) (
quantile_over_time(0.99998,{app="foo"} |= "bar" | json | latency >= 250ms or ( status_code < 500 and status_code > 200)
| line_format "blip{{ .foo }}blop {{.status_code}}" | label_format foo=bar,status_code="buzz{{.bar}}" | unwrap foo [5m]
) by (namespace,instance)
)
+
avg(
avg_over_time({app="foo"} |= "bar" | json | latency >= 250ms or ( status_code < 500 and status_code > 200)
| line_format "blip{{ .foo }}blop {{.status_code}}" | label_format foo=bar,status_code="buzz{{.bar}}" | unwrap foo [5m]
) by (namespace,instance)
) by (foo,bar),
"foo",
"$1",
"svc",
"(.*)"
)`,
exp: mustNewLabelReplaceExpr(
mustNewBinOpExpr(OpTypeAdd, BinOpOptions{ReturnBool: false},
mustNewVectorAggregationExpr(
newRangeAggregationExpr(
newLogRange(&pipelineExpr{
left: newMatcherExpr([]*labels.Matcher{{Type: labels.MatchEqual, Name: "app", Value: "foo"}}),
pipeline: MultiStageExpr{
newLineFilterExpr(nil, labels.MatchEqual, "bar"),
newLabelParserExpr(OpParserTypeJSON, ""),
&labelFilterExpr{
LabelFilterer: log.NewOrLabelFilter(
log.NewDurationLabelFilter(log.LabelFilterGreaterThanOrEqual, "latency", 250*time.Millisecond),
log.NewAndLabelFilter(
log.NewNumericLabelFilter(log.LabelFilterLesserThan, "status_code", 500.0),
log.NewNumericLabelFilter(log.LabelFilterGreaterThan, "status_code", 200.0),
),
),
},
newLineFmtExpr("blip{{ .foo }}blop {{.status_code}}"),
newLabelFmtExpr([]log.LabelFmt{
log.NewRenameLabelFmt("foo", "bar"),
log.NewTemplateLabelFmt("status_code", "buzz{{.bar}}"),
}),
},
},
5*time.Minute,
newUnwrapExpr("foo", "")),
OpRangeTypeQuantile, &grouping{without: false, groups: []string{"namespace", "instance"}}, NewStringLabelFilter("0.99998"),
),
OpTypeSum,
&grouping{groups: []string{"foo", "bar"}},
nil,
),
mustNewVectorAggregationExpr(
newRangeAggregationExpr(
newLogRange(&pipelineExpr{
left: newMatcherExpr([]*labels.Matcher{{Type: labels.MatchEqual, Name: "app", Value: "foo"}}),
pipeline: MultiStageExpr{
newLineFilterExpr(nil, labels.MatchEqual, "bar"),
newLabelParserExpr(OpParserTypeJSON, ""),
&labelFilterExpr{
LabelFilterer: log.NewOrLabelFilter(
log.NewDurationLabelFilter(log.LabelFilterGreaterThanOrEqual, "latency", 250*time.Millisecond),
log.NewAndLabelFilter(
log.NewNumericLabelFilter(log.LabelFilterLesserThan, "status_code", 500.0),
log.NewNumericLabelFilter(log.LabelFilterGreaterThan, "status_code", 200.0),
),
),
},
newLineFmtExpr("blip{{ .foo }}blop {{.status_code}}"),
newLabelFmtExpr([]log.LabelFmt{
log.NewRenameLabelFmt("foo", "bar"),
log.NewTemplateLabelFmt("status_code", "buzz{{.bar}}"),
}),
},
},
5*time.Minute,
newUnwrapExpr("foo", "")),
OpRangeTypeAvg, &grouping{without: false, groups: []string{"namespace", "instance"}}, nil,
),
OpTypeAvg,
&grouping{groups: []string{"foo", "bar"}},
nil,
),
),
"foo", "$1", "svc", "(.*)",
),
},
{
// ensure binary ops with two literals are reduced recursively
in: `1 + 1 + 1`,

Loading…
Cancel
Save