@ -144,17 +144,19 @@ A regex stage will take the provided regex and set the named groups as data in t
```yaml
- regex:
expression: ①
source: ②
```
① `expression` is **required** and needs to be a [golang RE2 regex string](https://github.com/google/re2/wiki/Syntax). Every capture group `(re)` will be set into the `extracted` map, every capture group **must be named:**`(?P<name>re)`, the name will be used as the key in the map.
② `source` is optional and contains the name of key in the `extracted` map containing the data to parse. If omitted, the regex stage will parse the log `entry`.
These map entries can then be used by other pipeline stages such as [timestamp](#timestamp) and/or [output](#output)
### json
A json stage will take the provided [JMESPath expressions](http://jmespath.org/) and set the key/value data in the `extracted` map.
@ -180,16 +206,18 @@ A json stage will take the provided [JMESPath expressions](http://jmespath.org/)
- json:
expressions: ①
key: expression ②
source: ③
```
① `expressions` is a required yaml object containing key/value pairs of JMESPath expressions
① `expressions` is a required yaml object containing key/value pairs of JMESPath expressions
② `key: expression` where `key` will be the key in the `extracted` map, and the value will be the evaluated JMESPath expression.
③ `source` is optional and contains the name of key in the `extracted` map containing the json to parse. If omitted, the json stage will parse the log `entry`.
This stage uses the Go JSON unmarshaller, which means non string types like numbers or booleans will be unmarshalled into those types. The `extracted` map will accept non-string values and this stage will keep primitive types as they are unmarshalled (e.g. bool or float64). Downstream stages will need to perform correct type conversion of these values as necessary.
If the value is a complex type, for example a JSON object, it will be marshalled back to JSON before being put in the `extracted` map.
##### Example:
##### Example (without source):
```yaml
- json:
@ -212,6 +240,36 @@ Would create the following `extracted` map:
```
[Example in unit test](../../pkg/logentry/stages/json_test.go)
A template stage lets you manipulate the values in the `extracted` data map using [Go's template package](https://golang.org/pkg/text/template/). This can be useful if you want to manipulate data extracted by regex or json stages before setting label values. Maybe to replace all spaces with underscores or make everything lowercase, or append some values to the extracted data.
// If a source key is provided, the json stage should process it
// from the exctracted map, otherwise should fallback to the entry
input:=entry
ifj.cfg.Source!=""{
if_,ok:=extracted[j.cfg.Source];!ok{
level.Debug(j.logger).Log("msg","source does not exist in the set of extracted values","source",j.cfg.Source)
return
}
value,err:=getString(extracted[j.cfg.Source])
iferr!=nil{
level.Debug(j.logger).Log("msg","failed to convert source value to string","source",j.cfg.Source,"err",err,"type",reflect.TypeOf(extracted[j.cfg.Source]).String())
return
}
input=&value
}
ifinput==nil{
level.Debug(j.logger).Log("msg","cannot parse a nil entry")
// If a source key is provided, the regex stage should process it
// from the exctracted map, otherwise should fallback to the entry
input:=entry
ifr.cfg.Source!=""{
if_,ok:=extracted[r.cfg.Source];!ok{
level.Debug(r.logger).Log("msg","source does not exist in the set of extracted values","source",r.cfg.Source)
return
}
value,err:=getString(extracted[r.cfg.Source])
iferr!=nil{
level.Debug(r.logger).Log("msg","failed to convert source value to string","source",r.cfg.Source,"err",err,"type",reflect.TypeOf(extracted[r.cfg.Source]).String())
return
}
input=&value
}
ifinput==nil{
level.Debug(r.logger).Log("msg","cannot parse a nil entry")
return
}
match:=r.expression.FindStringSubmatch(*entry)
match:=r.expression.FindStringSubmatch(*input)
ifmatch==nil{
level.Debug(r.logger).Log("msg","regex did not match")
Marco Pracucci
7 years ago
committed by