An alert rule consists of one or more queries and expressions that select the data you want to measure. It also contains a condition, which is the threshold that an alert rule must meet or exceed in order to fire.
[Alert rules](ref:alert-rules) are the central component of your alerting system.
Create, manage, view, and adjust alert rules to alert on your metrics data or log entries from multiple data sources — no matter where your data is stored.
An alert rule consists of one or more queries and expressions that select the data you want to measure. It contains a condition to trigger the alert, an evaluation period that determines how often the rule is evaluated, and additional options to manage alert events and their notifications.
The main parts of alert rule creation are:
Grafana supports two types of alert rules:
1. Select your data source
1. Query your data
1. Normalize your data
1. Set your threshold
1. Grafana-managed alert rules: These can query multiple data sources.
**Query, expressions, and alert condition**
1. Data source-managed alert rules: These can only query Prometheus-based data sources and support horizontal scaling.
What are you monitoring? How are you measuring it?
We recommend using Grafana-managed alert rules whenever possible, and opting for data source-managed alert rules when horizontal scaling is required. Refer to the [comparison table of alert rule types](ref:alert-types-comparison-table) for a more detailed overview.
{{<admonitiontype="note">}}
Expressions can only be used for Grafana-managed alert rules.
{{</admonition>}}
Both types of alert rules can be configured in Grafana using the **+ New alert rule** flow. For step-by-step instructions, refer to:
Data source-managed alert rules can only query Prometheus-based data sources, such as Prometheus, Grafana Mimir, or Grafana Loki. They are one of the two [alert rule types](ref:alert-rules) supported in Grafana.
Data source-managed alert rules are stored within the data source. In a distributed architecture, they can scale horizontally to provide high-availability.
We recommend using [Grafana-managed alert rules](ref:configure-grafana-managed-rules) whenever possible and opting for data source-managed alert rules when scaling your alerting setup is necessary.
To create or edit data source-managed alert rules, follow these instructions.
## Before you begin
Verify that you have write permission to the Prometheus, Mimir, or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.
### Enable the Ruler API
For more information, refer to the [Mimir Ruler API](/docs/mimir/latest/references/http-api/#ruler) or [Loki Ruler API](/docs/loki/latest/api/#ruler).
- **Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
### Permissions
Alert rules for Prometheus, Mimir, or Loki instances can be edited or deleted by users with **Editor** or **Admin** roles.
If you do not want to manage alert rules for a particular data source, go to its settings and clear the **Manage alerts via Alerting UI** checkbox.
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
{{% admonition type="note" %}}
By default, new alert rules are Grafana-managed. To switch to **Data source-managed**, follow these instructions.
{{% /admonition %}}
1. Select a Prometheus-based data source from the drop-down list.
You can also click **Open advanced data source picker** to find more options.
1. Enter a PromQL or LogQL query, including the alert condition.
1. In the **Rule type** option, select **Data source-managed**.
1. Click **Preview alerts**.
## Set alert evaluation behavior
Use [alert rule evaluation](ref:alert-rule-evaluation) to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
1. Select a namespace or click **+ New namespace**.
1. Select an evaluation group or click **+ New evaluation group**.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated sequentially over the same time interval.
1. Enter a pending period.
The [pending period](ref:pending-period) is the period in which an alert rule can be in breach of the condition until it fires.
Once a condition is met, the alert goes into the **Pending** state. If the condition remains active for the duration specified, the alert transitions to the **Firing** state, else it reverts to the **Normal** state.
## Configure labels and notifications
Add [labels](ref:alert-rule-labels) to your alert rules to set which [notification policy](ref:notification-policies) should handle your firing alert instances.
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.
1. Add labels if you want to change the way your notifications are routed.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Grafana-managed rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of our supported data sources. In addition to supporting multiple data sources, you can also add expressions to transform your data and set alert conditions. Using images in alert notifications is also supported. This is the only type of rule that allows alerting from multiple data sources in a single rule definition.
Multiple alert instances can be created as a result of one alert rule (also known as a multi-dimensional alerting).
Grafana-managed rules can query data from multiple data sources in a single alert rule. They are the most flexible [alert rule type](ref:alert-rules). You can also add expressions to transform your data, set alert conditions, and images in alert notifications.
{{% admonition type="note" %}}
For Grafana Cloud Free Forever, you can create up to 100 free Grafana-managed alert rules with each alert rule having a maximum of 1000 alert instances.
In Grafana Cloud, the number of Grafana-managed alert rules you can create depends on your Grafana Cloud plan.
For all paid tiers (Cloud Pro and Advanced), there is a soft limit of 2000 alert rules and unlimited alert instances. To increase the limit, open a support ticket from the [Cloud portal](https://grafana.com/docs/grafana-cloud/account-management/support/).
- Free Forever plan: You can create up to 100 free alert rules, with each alert rule having a maximum of 1000 alert instances.
- All paid plans (Pro and Advanced): They have a soft limit of 2000 alert rules and support unlimited alert instances. To increase the limit, open a support ticket from the [Cloud portal](/docs/grafana-cloud/account-management/support/).
{{% /admonition %}}
Grafana-managed alert rules can only be edited or deleted by users with Edit permissions for the folder storing the rules.
If you delete an alert resource created in the UI, you can no longer retrieve it.
To make a backup of your configuration and to be able to restore deleted alerting resources, create your alerting resources using file provisioning, Terraform, or the Alerting API.
To create or edit Grafana-managed alert rules, follow the instructions below. For a practical example, check out our [tutorial on getting started with Grafana alerting](http://grafana.com/tutorials/alerting-get-started/).
## Before you begin
If you are using Grafana OSS:
Verify that the data sources you plan to query in the alert rule are [compatible with Grafana-managed alert rules](ref:compatible-data-sources) and are properly configured.
1. Configure your [data sources](ref:data-sources).
2. Check which [data sources](ref:compatible-data-sources) are compatible with and supported by Grafana Alerting.
### Permissions
If you are using Grafana OSS, Enterprise, or Cloud:
Only users with **Edit** permissions for the folder storing the rules can edit or delete Grafana-managed alert rules.
You can use default or advanced options for Grafana-managed alert rule creation. The default options streamline rule creation with a cleaner header and a single query and condition. For more complex rules, use advanced options to add multiple queries and expressions.
Default and advanced options are enabled by default for Grafana Cloud users and this feature is being rolled out progressively.
For OSS users,enable the `alertingQueryAndExpressionsStepMode` feature toggle.
{{% admonition type="note" %}}
Once you have created an alert rule using one of the options, the system defaults to this option for the next alert rule you create.
### Default vs Advanced options
You can toggle between the two options. However, if you want to switch from advanced options to the default, it may be that your query and expressions cannot be converted. In this case, a warning message checks whether you want to continue to reset to default settings.
{{% /admonition %}}
## Steps
To create a Grafana-managed alert rule, use the in-product alert creation flow and follow these steps.
You can use default or advanced options for Grafana-managed alert rule creation. The default options streamline rule creation with a cleaner header and a single query and condition. For more complex rules, use advanced options to add multiple queries and expressions.
To get started quickly, refer to our [tutorial on getting started with Grafana alerting](http://grafana.com/tutorials/alerting-get-started/).
You can toggle between the two options. Once you have created an alert rule, the system defaults to your previous choice for the next alert rule.
## Set alert rule name
Switching from advanced to default may result in queries and expressions that cannot be converted. In this case, a warning message asks if you want to continue to reset to default settings.
Default and advanced options are enabled by default for Grafana Cloud users and this feature is being rolled out progressively. OSS users can enable them via the [`alertingQueryAndExpressionsStepMode` feature toggle](/setup-grafana/configure-grafana/feature-toggles/).
This name is displayed in the alert rule list. It is also the `alertname` label for every alert instance that is created from this rule.
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
You can toggle between **Default** and **Advanced** options. If the [Default vs. Advanced feature](#default-vs-advanced-options) is not enabled in your Grafana instance, follow the **Advanced options** instructions.
{{<collapsetitle="Default options">}}
1. Add a query.
1. Add an alert condition.
1. Add a [query](ref:alert-rule-query).
1. Add an [alert condition](ref:alert-condition).
The **When** input includes the reducer function and the last input is the threshold.
@ -152,15 +173,11 @@ Define a query to get the data you want to measure and a condition that needs to
1. Select a data source.
1. From the **Options** dropdown, specify a [time range](ref:time-units-and-relative-ranges).
{{% admonition type="note" %}}
Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`.
It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
{{% /admonition %}}
Note that Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`. It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
1. Add a query.
To add multiple [queries](ref:add-a-query), click **Add query**.
To add multiple [queries](ref:alert-rule-query), click **Add query**.
All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click **Switch to data source-managed alert rule**.
@ -168,9 +185,7 @@ It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:
a. For each expression, select either **Classic condition** to create a single alert rule, or choose from the **Math**, **Reduce**, and **Resample** options to generate separate alert for each series.
{{% admonition type="note" %}}
When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.
{{% /admonition %}}
b. Click **Preview** to verify that the expression is successful.
@ -178,14 +193,14 @@ It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:
You can only add one recovery threshold in a query and it must be the alert condition.
1. Click **Set as alert condition** on the query or expression you want to set as your alert condition.
1. Click **Set as alert condition** on the query or expression you want to set as your [alert condition](ref:alert-condition).
{{</collapse>}}
## Add folders and labels
## Set folder and labels
Organize your alert rule with a folder and set of labels.
In the **Labels** section, you can optionally choose whether to add labels to organize your alert rules, make searching easier, as well as set which notification policy should handle your firing alert instance.
In the **Labels** section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to [alert rule labels](ref:alert-rule-labels).
1. Select a folder or click **+ New folder**.
@ -193,9 +208,9 @@ In the **Labels** section, you can optionally choose whether to add labels to or
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
## Set alert evaluation behavior
## Configure alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
Use [alert rule evaluation](ref:alert-rule-evaluation) to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
@ -205,7 +220,7 @@ To do this, you need to make sure that your alert rule is in the right evaluatio
All rules within the same group are evaluated concurrently over the same time interval.
1. Enter a pending period.
1. Enter a [pending period](ref:pending-period).
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
@ -213,15 +228,20 @@ To do this, you need to make sure that your alert rule is in the right evaluatio
1. Turn on pause alert notifications, if required.
{{<admonitiontype="note">}}
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts.
Pausing stops alert rule evaluation and doesn't create any alert instances.
This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.
{{</admonition>}}
This is different to [mute timings](ref:mute-timings), which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.
1. In **Configure no data and error handling**, you can define the alerting behavior and alerting state for two scenarios:
- When the evaluation returns **No data** or all values are null.
- When the evaluation returns **Error** or timeout.
### Configure no data and error handling
1. In **Configure no data and error handling**, configure alerting behavior in the absence of data.
Use the guidelines in [No data and error handling](#configure-no-data-and-error-handling).
For more details, refer to [alert instance states](ref:alert-instance-state) and [modify the no data and error state](ref:modify-the-no-data-and-error-state).
## Configure notifications
@ -233,7 +253,7 @@ Complete the following steps to set up notifications.
**Select contact point**
1. Choose this option to select an existing contact point.
1. Choose this option to select an existing [contact point](ref:contact-points).
All notifications for this alert rule are sent to this contact point automatically and notification policies are not used.
@ -244,7 +264,7 @@ Complete the following steps to set up notifications.
**Use notification policy**
1. Choose this option to use the notification policy tree to direct your notifications.
1. Choose this option to use the [notification policy tree](ref:notification-policies) to direct your notifications.
{{<admonitiontype="note">}}
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.
@ -258,53 +278,4 @@ Complete the following steps to set up notifications.
1. Click **See details** to view alert routing details and an email preview.
## Configure notification message
Add more context on the alert in your alert notification message. For more information, see [annotations](ref:annotation-label).
Annotations add metadata to provide more information on the alert in your alert notification message. For example, add a **Summary** annotation to tell you which value caused the alert to fire or which server it happened on.
1. Optional: Add a summary.
Short summary of what happened and why.
1. Optional: Add a description.
Description of what the alert rule does.
1. Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
1. Optional: Add a custom annotation
1. Optional: **Link dashboard and panel**.
[Link the alert rule to a panel](ref:link-alert-rules-to-panels) to facilitate alert investigation.
1. Click **Save rule**.
## Configure no data and error handling
In **Configure no data and error handling**, you can define the alerting behavior when the evaluation returns no data or an error.
For details about alert states, refer to [lifecycle of alert instances](ref:alert-instance-state).
You can configure the alert instance state when its evaluation returns no data:
| No Data | The default option. Sets alert instance state to `No data`. <br/> The alert rule also creates a new alert instance `DatasourceNoData` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
| Alerting | Sets the alert instance state to `Pending` and then transitions to `Alerting` once the [pending period](ref:pending-period) ends. If you sent the pending period to 0, the alert instance state is immediately set to `Alerting`. |
| Normal | Sets alert instance state to `Normal`. |
| Keep Last State | Maintains the alert instance in its last state. Useful for mitigating temporary issues, refer to [Keep last state](ref:keep-last-state). |
You can also configure the alert instance state when its evaluation returns an error or timeout.
| Error | The default option. Sets alert instance state to `Error`. <br/> The alert rule also creates a new alert instance `DatasourceError` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
| Alerting | Sets alert instance state to `Alerting`. It transitions from `Pending` to `Alerting` after the [pending period](ref:pending-period) has finished. |
| Normal | Sets alert instance state to `Normal`. |
| Keep Last State | Maintains the alert instance in its last state. Useful for mitigating temporary issues, refer to [Keep last state](ref:keep-last-state). |
When you configure the No data or Error behavior to `Alerting` or `Normal`, Grafana will attempt to keep a stable set of fields under notification `Values`. If your query returns no data or an error, Grafana re-uses the latest known set of fields in `Values`, but will use `-1` in place of the measured value.
Create data source-managed alert rules for Grafana Mimir or Grafana Loki data sources, which have been configured to support rule creation.
To configure your Grafana Mimir or Loki data source for alert rule creation, enable either the Loki Ruler API or the Mimir Ruler API.
For more information, refer to [Loki Ruler API](/docs/loki/<GRAFANA_VERSION>/api/#ruler) or [Mimir Ruler API](/docs/mimir/<GRAFANA_VERSION>/references/http-api/#ruler).
**Note**:
Alert rules for a Grafana Mimir or Loki instance can be edited or deleted by users with Editor or Admin roles.
If you delete an alerting resource created in the UI, you can no longer retrieve it.
To make a backup of your configuration and to be able to restore deleted alerting resources, create your alerting resources using file provisioning, Terraform, or the Alerting API.
## Before you begin
- Verify that you have write permission to the Mimir or Loki data source. Otherwise, you cannot create or update Grafana Mimir or Loki-managed alert rules.
- Enable the Mimir or Loki Ruler API.
- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
- **Grafana Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
Watch this video to learn more about how to create a Mimir-managed alert rule: {{<vimeo720001865>}}
{{% admonition type="note" %}}
If you do not want to manage alert rules for a particular Loki or Mimir data source, go to its settings and clear the **Manage alerts via Alerting UI** checkbox.
{{% /admonition %}}
To create a data source-managed alert rule, use the in-product alert creation flow and follow these steps to help you.
This name is displayed in the alert rule list. It is also the `alertname` label for every alert instance that is created from this rule.
## Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
**Note**:
All alert rules are managed by Grafana by default. To switch to a data source-managed alert rule, click **Switch to data source-managed alert rule**.
1. Select a data source from the drop-down list.
You can also click **Open advanced data source picker** to see more options, including adding a data source (Admins only).
1. Enter a PromQL or LogQL query.
1. Click **Preview alerts**.
## Set alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
1. Select a namespace or click **+ New namespace**.
1. Select an evaluation group or click **+ New evaluation group**.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated sequentially over the same time interval.
1. Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
Once a condition is met, the alert goes into the **Pending** state. If the condition remains active for the duration specified, the alert transitions to the **Firing** state, else it reverts to the **Normal** state.
## Configure notifications
Add labels to your alert rules to set which notification policy should handle your firing alert instances.
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.
1. Add labels if you want to change the way your notifications are routed.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
## Add annotations
Add [annotations](ref:annotation-label). to provide more context on the alert in your alert notifications.
Annotations add metadata to provide more information on the alert in your alert notifications. For example, add a **Summary** annotation to tell you which value caused the alert to fire or which server it happened on.
1. Optional: Add a summary.
Short summary of what happened and why.
1. Optional: Add a description.
Description of what the alert rule does.
1. Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
1. Optional: Add a custom annotation
1. Optional: **Link dashboard and panel**.
[Link the alert rule to a panel](ref:link-alert-rules-to-panels) to facilitate alert investigation.
@ -41,7 +41,7 @@ Note that in data source-managed groups, the alert rules and recording rules wit
- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
- **Grafana Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
- **Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
@ -72,6 +72,8 @@ In [Configure no data and error handling](ref:no-data-and-error-handling), you c
{{<figuresrc="/media/docs/alerting/alert-rule-configure-no-data-and-error.png"alt="A screenshot of the `Configure no data and error handling` option in Grafana Alerting."max-width="500px">}}
To reduce the number of **No Data** or **Error** state alerts received, try the following.
1. Use the **Keep last state** option. For more information, refer to the section below. This option allows the alert to retain its last known state when there is no data available, rather than switching to a **No Data** state.
@ -81,6 +83,8 @@ To reduce the number of **No Data** or **Error** state alerts received, try the
1. Change the default [evaluation time out](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#evaluation_timeout). The default is set at 30 seconds. To increase the default evaluation timeout, open a support ticket from the [Cloud Portal](https://grafana.com/docs/grafana-cloud/account-management/support/#grafana-cloud-support-options). Note that this should be a last resort, because it may affect the performance of all alert rules and cause missed evaluations if the timeout is too long.
Note that when you configure the **No data** or **Error** behavior to `Alerting` or `Normal`, Grafana attempts to keep a stable set of fields under notification `Values`. If your query returns no data or an error, Grafana re-uses the latest known set of fields in `Values`, but will use `-1` in place of the measured value.
#### Keep last state
The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.
@ -84,9 +84,9 @@ Find the public data sources supporting Alerting in the [Grafana Plugins directo
## Data source-managed alert rules
Data source-managed alert rules can query Prometheus-based data sources, such as Grafana Mimir or Grafana Loki. Alert rules are stored within the data source when the Ruler API is enabled (e.g., [Mimir Ruler API](/docs/mimir/<GRAFANA_VERSION>/references/http-api/#ruler) or [Loki Ruler API](/docs/loki/<GRAFANA_VERSION>/api/#ruler)).
Data source-managed alert rules can only query Prometheus-based data sources, such as Prometheus, Grafana Mimir, or Grafana Loki.
In this setup, the distributed architecture can provide high-availability and fault tolerance.
Alert rules are stored within the data source. In this distributed architecture, the separation of components can provide high-availability and fault tolerance, enabling the scaling of your alerting setup.
@ -95,25 +95,27 @@ In this setup, the distributed architecture can provide high-availability and fa
1. Alert rules are evaluated by the Alert Rule Evaluation Engine.
1. Firing and resolved alert instances are forwarded to [handle their notifications](ref:notifications).
## Recording rules
## Comparison between alert rule types
A recording rule pre-compute frequently used or computationally expensive queries, and saves the results as a new time series metric.
We recommend using Grafana-managed alert rules whenever possible, and opting for data source-managed alert rules when you need to scale your alerting setup.
The new metric can then be used in alert rules and dashboards to optimize their queries.
The table below compares Grafana-managed and data source-managed alert rules.
Similar to alert rules, recording rules are evaluated periodically. For more details, refer to [Create recording rules](ref:create-recording-rules).
| Create alert rules<wbr/> that query [data sources supporting Alerting](#supported-data-sources) | Yes | No. Only query Prometheus-based data sources. |
| Mix and match data sources | Yes | No |
| Add [expressions](ref:expression-queries) to transform<wbr/> your data and set [alert conditions](ref:alert-condition) | Yes | No |
| Use [images in alert notifications](ref:notification-images) | Yes | No |
| Support for [recording rules](#recording-rules) | Yes | Yes |
| Organization | Organize and manage access with folders | Use namespaces |
| Alert rule evaluation and delivery | Alert evaluation is done in Grafana, while delivery can be handled by Grafana or an external Alertmanager. | Alert rule evaluation and alert delivery are distributed. |
| Scaling | Alert rules are stored in the Grafana database, which may experience transient errors. It only scales vertically. | Alert rules are stored within the data source and allow for horizontal scaling. |
## Comparison between alert rule types
## Recording rules
Similar to alert rules, recording rules are evaluated periodically. A recording rule pre-computes frequently used or computationally expensive queries, and saves the results as a new time series metric.
The new recording metric can then be used in alert rules and dashboards to optimize their queries.
When choosing which alert rule type to use, consider the following comparison between Grafana-managed and data source-managed alert rules.
| Create alert rules<wbr/> that query [data sources supporting Alerting](#supported-data-sources) | Yes | No. Only query Prometheus-based data sources. |
| Mix and match data sources | Yes | No |
| Add [expressions](ref:expression-queries) to transform<wbr/> your data and set [alert conditions](ref:alert-condition) | Yes | No |
| Use [images in alert notifications](ref:notification-images) | Yes | No |
| Support for [recording rules](#recording-rules) | Yes | Yes |
| Organization | Organize and manage access with folders | Use namespaces |
| Scaling | More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically. | Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data. |
| Alert rule evaluation and delivery | Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both. | Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure. |
For more details, refer to [Create recording rules](ref:create-recording-rules).
@ -386,7 +386,7 @@ To reset the notification policy tree to the default and unlock it for editing i
## Data source-managed resources
The Alerting Provisioning HTTP API can only be used to manage Grafana-managed alert resources. To manage resources related to [data source-managed alerts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-mimir-loki-managed-rule/), consider the following tools:
The Alerting Provisioning HTTP API can only be used to manage Grafana-managed alert resources. To manage resources related to [data source-managed alerts](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-data-source-managed-rule/), consider the following tools:
- [mimirtool](https://grafana.com/docs/mimir/<GRAFANA_VERSION>/manage/tools/mimirtool/): to interact with the Mimir alertmanager and ruler configuration.
- [cortex-tools](https://github.com/grafana/cortex-tools#cortextool): to interact with the Cortex alertmanager and ruler configuration.
Use [annotations](ref:shared-annotations) to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or [templates](ref:shared-alert-rule-template) to display dynamic data from queries.
Grafana provides several optional annotations.
1. Optional: Add a summary.
Short summary of what happened and why.
1. Optional: Add a description.
Description of what the alert rule does.
1. Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
1. Optional: Add a custom annotation.
Add any additional information that could help address the alert.
1. Optional: **Link dashboard and panel**.
[Link the alert rule to a panel](ref:shared-link-alert-rules-to-panels) to facilitate alert investigation.
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can [provision alerting resources](ref:shared-provision-alerting-resources) using options such as configuration files, Terraform, or the Alerting API.
| NoData | No Data | The default option for **No Data** events.<br/>Sets alert instance state to `No data`. <br/> The alert rule also creates a new alert instance `DatasourceNoData` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
| Error | Error | The default option for **Error** events.<br/>Sets alert instance state to `Error`. <br/> The alert rule also creates a new alert instance `DatasourceError` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
| NoData/Error | Alerting | Sets the alert instance state to `Pending` and then transitions to `Alerting` once the pending period ends. If you sent the pending period to 0, the alert instance state is immediately set to `Alerting`. |
| NoData/Error | Normal | Sets alert instance state to `Normal`. |
| NoData/Error | Keep Last State | Maintains the alert instance in its last state. Useful for mitigating temporary issues. |
Pepe Cano
6 months ago
committed by