@ -85,7 +85,11 @@ The rules are stored within the data source. In a distributed architecture, they
We recommend using [Grafana-managed alert rules](ref:configure-grafana-managed-rules) whenever possible and opting for data source-managed alert rules when scaling your alerting setup is necessary.
> Rules from a Prometheus data source appear in the **Data source-managed** section of the **Alert rules** page when [Manage alerts via Alerting UI](ref:shared-configure-prometheus-data-source-alerting) is enabled.
>
> However, Grafana can only create and edit data source-managed rules for Mimir and Loki, not for a Prometheus instance.
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can [provision alerting resources](ref:shared-provision-alerting-resources) using options such as configuration files, Terraform, or the Alerting API.
Use [annotations](ref:shared-annotations) to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or [templates](ref:shared-alert-rule-template) to display dynamic data from queries.
Grafana provides several optional annotations.
1. Optional: Add a summary.
Short summary of what happened and why.
1. Optional: Add a description.
Description of what the alert rule does.
1. Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
1. Optional: Add a custom annotation.
Add any additional information that could help address the alert.
1. Optional: **Link dashboard and panel**.
[Link the alert rule to a panel](ref:shared-link-alert-rules-to-panels) to facilitate alert investigation.
@ -145,7 +145,13 @@ Verify that the data sources you plan to query in the alert rule are [compatible
Only users with **Edit** permissions for the folder storing the rules can edit or delete Grafana-managed alert rules. Only admins can restore deleted Grafana-managed alert rules.
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can [provision alerting resources](ref:shared-provision-alerting-resources) using options such as configuration files, Terraform, or the Alerting API.
Use [annotations](ref:shared-annotations) to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or [templates](ref:shared-alert-rule-template) to display dynamic data from queries.
Grafana provides several optional annotations.
1. Optional: Add a summary.
Short summary of what happened and why.
1. Optional: Add a description.
Description of what the alert rule does.
1. Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
1. Optional: Add a custom annotation.
Add any additional information that could help address the alert.
1. Optional: **Link dashboard and panel**.
[Link the alert rule to a panel](ref:shared-link-alert-rules-to-panels) to facilitate alert investigation.
@ -67,7 +67,18 @@ Silences stop notifications from being created for a specified time window but d
Silences are assigned to a [specific Alertmanager](ref:alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
[Mute timings](ref:shared-mute-timings) and [silences](ref:shared-silences) are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.
The following table highlights the key differences between mute timings and silences.
@ -81,9 +92,60 @@ To add a silence, complete the following steps.
1. Optionally, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field.
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_ to determine which alerts the silence applies to.
Use [labels](ref:shared-alert-labels) and label matchers to link alert rules to [notification policies](ref:shared-notification-policies) and [silences](ref:shared-silences). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.
- The **Label** field is the name of the label to match. It must exactly match the label name.
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
- The **Operator** field is the operator to match against the label value. The available operators are:
| `=` | Select labels that are exactly equal to the value. |
| `!=` | Select labels that are not equal to the value. |
| `=~` | Select labels that regex-match the value. |
| `!~` | Select labels that do not regex-match the value. |
{{% admonition type="note" %}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
{{% /admonition %}}
**Label matching example**
If you define the following set of labels for your alert:
`{ foo=bar, baz=qux, id=12 }`
then:
- A label matcher defined as `foo=bar` matches this alert rule.
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
**Exclude labels**
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.
@ -47,7 +47,18 @@ Use mute timings to temporarily pause notifications for a specific recurring per
Mute timings are assigned to a [specific Alertmanager](ref:alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
[Mute timings](ref:shared-mute-timings) and [silences](ref:shared-silences) are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.
The following table highlights the key differences between mute timings and silences.
> Rules from a Prometheus data source appear in the **Data source-managed** section of the **Alert rules** page when [Manage alerts via Alerting UI](ref:shared-configure-prometheus-data-source-alerting) is enabled.
>
> However, Grafana can only create and edit data source-managed rules for Mimir and Loki, not for a Prometheus instance.
@ -71,7 +71,58 @@ Notification policies are _not_ a list, but rather are structured according to a
Each policy consists of a set of label matchers (0 or more) that specify which alerts they are or aren't interested in handling. A matching policy refers to a notification policy with label matchers that match the alert instance’s labels.
Use [labels](ref:shared-alert-labels) and label matchers to link alert rules to [notification policies](ref:shared-notification-policies) and [silences](ref:shared-silences). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.
- The **Label** field is the name of the label to match. It must exactly match the label name.
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
- The **Operator** field is the operator to match against the label value. The available operators are:
| `=` | Select labels that are exactly equal to the value. |
| `!=` | Select labels that are not equal to the value. |
| `=~` | Select labels that regex-match the value. |
| `!~` | Select labels that do not regex-match the value. |
{{% admonition type="note" %}}
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
{{% /admonition %}}
**Label matching example**
If you define the following set of labels for your alert:
`{ foo=bar, baz=qux, id=12 }`
then:
- A label matcher defined as `foo=bar` matches this alert rule.
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
**Exclude labels**
You can also write label matchers to exclude labels.
Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.
{{<figuresrc="/media/docs/alerting/notification-routing.png"max-width="750px"caption="Matching alert instances with notification policies"alt="Example of a notification policy tree">}}
Pepe Cano
2 months ago
committed by