@ -92,7 +92,7 @@ Refer to [Templates Introduction](ref:intro-to-templates) for a more detailed ex
Both types of templates are written in the Go templating system. However, it's important to understand that variables and functions used in notification templates are different from those used in annotation and label templates.
1. **Template annotations and labels**: These templates add extra information to individual alert instances. Template variables like [`$labels`](ref:labels) and [`$values`](ref:values) represent alert query data of the individual alert instance.
1. **Template annotations and labels**: These templates add extra information to individual alert instances. Template variables like [`$labels`](ref:reference-labels) and [`$values`](ref:reference-values) represent alert query data of the individual alert instance.
1. **Template notifications**: Notification templates format the notification content for a group of alerts. Variables like [`.Alerts`](ref:notification-data-reference) include all firing and resolved alerts in the notification.
## Template annotations
@ -111,11 +111,11 @@ CPU usage has exceeded 80% for the last 5 minutes.
However, if you want to display dynamic query values in annotations, you need to use template code. Common use cases include:
- Displaying the query value or threshold that triggered the alert.
- Highlighting label information that identifies the alert, such as environment, region, or priority.
- Displaying the query value that triggered the alert.
- Highlighting label information that identifies the alert, such as the environment, instance, or region.
- Providing specific instructions based on query values.
- Customizing runbook links depending on query or label values.
- Including contact information based on alert labels.
- Customizing runbook links depending on query labels.
- Including contact information based on query labels.
For instance, you can template the previous example to display the specific instance and CPU value that triggered the alert.
@ -115,7 +115,9 @@ CPU usage has exceeded 80% (81.2345) for the last 5 minutes.
### Include labels for extra details
To provide additional context, you can include labels from the query. For instance, access the [`$labels`](ref:reference-labels) variable to display a label that informs about the affected instance:
To provide additional context, you can include labels from the query using the [`$labels`](ref:reference-labels) variable.
For instance, the previous case could also include the affected `instance`.
```
CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes.
@ -125,14 +127,14 @@ CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) fo
CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes.
```
Annotations can also be used to provide a summary of key alert labels, such as the environment and alert severity. For instance, you can display a summary of the alert with important labels like:
You can incorporate any labels returned by the query into the template. For instance, the following template includes information about the environment and region where the alert occurred.
```
Alert triggered in {{ $labels.environment }} with severity {{ $labels.severity }}
Alert triggered in {{ $labels.environment }} on {{ $labels.region }} region.
```
```template_output
Alert triggered in production with severity critical.
| [$labels](#labels) | Contains all labels from the query. |
| [$labels](#labels) | Contains all labels from the query, only query labels. |
| [$values](#values) | Contains the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs. |
| [$value](#value) | A string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. |
### $labels
The `$labels` variable contains all labels from the query.
The `$labels` variable contains all labels from the query. It excludes [user-configured and reserved labels](ref:label-types), containing only query labels.
{{<figuresrc="/media/docs/alerting/query-labels-and_value.png"max-width="1200px"caption="An alert rule displaying labels and value from a query">}}
{{<figuresrc="/media/docs/alerting/query-labels-and-values.png"max-width="1200px"caption="An alert rule displaying labels and value from a query.">}}
For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as:
| `Labels` | [KV](#kv) | The labels for this alert. |
| `Labels` | [KV](#kv) | The labels for this alert. It includes all [types of labels](ref:label-types). |
| `Annotations` | [KV](#kv) | The annotations for this alert. |
| `StartsAt` | [Time](#time) | The time the alert fired |
| `EndsAt` | [Time](#time) | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. |
@ -87,11 +87,13 @@ Labels that you manually configure in the alert rule to identify the generated a
Additionally, you can use a [template](ref:templates) to customize the label value and generate dynamic values from query data.
**Data source query labels**
**Query labels**
For example, if you are monitoring temperature readings and each time series for these readings has a `sensor_id`, and a `location` label, an alert instance might have the labels `{sensor_id="1",location="NY"}`, while another alert instance might have `{sensor_id="2",location="WA"}`.
Query labels are labels returned by the data source query.
Data source query labels labels are also used to generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data.
{{<figuresrc="/media/docs/alerting/query-labels-and-values.png"max-width="1200px"caption="An alert rule query returning labels from the query.">}}
Query labels can generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data. In this example, the `instance` label generates an alert instance for each server.
Pepe Cano
1 year ago
committed by