@ -68,6 +68,24 @@ Alert instances will be routed for [notifications](ref:notifications) when they
{{<figuresrc="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png"alt="A diagram of the alert instance states and when to route their notifications."max-width="750px">}}
### Stale alert instances (MissingSeries)
The `No Data` state occurs when the alert rule query runs successfully but returns no data points at all.
An alert instance is considered stale if the query returns data but its dimension or series has disappeared for two evaluation intervals. In this case, the alert instance transitions to the **Normal (MissingSeries)** state as resolved, and is then evicted.
1. The alert rule runs and returns data for some label sets.
1. An alert instance that previously existed is now missing.
1. Grafana keeps the previous state of the alert instance for two evaluation intervals.
1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
1. The alert instance is removed from the UI.
### `No Data` and `Error` alerts
When an alert rule evaluation results in a `No Data` or `Error` state, Grafana Alerting immediately creates a new alert instance —skipping the pending period—with the following additional labels:
@ -82,12 +100,6 @@ You can manage these alerts like regular ones by using their labels to apply act
If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.
### Lifecycle of stale alert instances
An alert instance is considered stale if its dimension or series has disappeared from the query results entirely for two evaluation intervals.
Stale alert instances that are in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**. Once transitioned, these resolved alert instances are routed for notifications like other resolved alerts.
## Modify the `No Data` or `Error` state
These states are supported only for Grafana-managed alert rules.
@ -121,7 +133,7 @@ However, in situations where strict monitoring is critical, relying solely on th
Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:
- Stale alert instances in the `Alerting` state transition to the `Normal` state when the series disappear.
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Alerting` state transition to the `Normal` state when the series disappear.
- If "no data" handling is configured to transition to a state other than `No Data`.
- If "error" handling is configured to transition to a state other than `Error`.
- If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state.
@ -130,7 +142,7 @@ In these situations, the evaluation state may differ from the alert state, and i
The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:
- Stale alert instances in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
- If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively.
- If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.