From 740c47c33a98b789078ab9ab574bee6f652f9902 Mon Sep 17 00:00:00 2001 From: Pepe Cano <825430+ppcano@users.noreply.github.com> Date: Thu, 27 Mar 2025 22:14:28 +0100 Subject: [PATCH] docs(alerting): stale alert instances updates (#103014) * docs(alerting): stale alert instances updates * correct vale error --- .../alert-rule-evaluation/state-and-health.md | 28 +++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md b/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md index 79b137c1a7d..471d04ad777 100644 --- a/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md +++ b/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md @@ -68,6 +68,24 @@ Alert instances will be routed for [notifications](ref:notifications) when they {{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}} +### Stale alert instances (MissingSeries) + +The `No Data` state occurs when the alert rule query runs successfully but returns no data points at all. + +An alert instance is considered stale if the query returns data but its dimension or series has disappeared for two evaluation intervals. In this case, the alert instance transitions to the **Normal (MissingSeries)** state as resolved, and is then evicted. + +1. The alert rule runs and returns data for some label sets. + +1. An alert instance that previously existed is now missing. + +1. Grafana keeps the previous state of the alert instance for two evaluation intervals. + +1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation. + +1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts. + +1. The alert instance is removed from the UI. + ### `No Data` and `Error` alerts When an alert rule evaluation results in a `No Data` or `Error` state, Grafana Alerting immediately creates a new alert instance —skipping the pending period—with the following additional labels: @@ -82,12 +100,6 @@ You can manage these alerts like regular ones by using their labels to apply act If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved. -### Lifecycle of stale alert instances - -An alert instance is considered stale if its dimension or series has disappeared from the query results entirely for two evaluation intervals. - -Stale alert instances that are in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**. Once transitioned, these resolved alert instances are routed for notifications like other resolved alerts. - ## Modify the `No Data` or `Error` state These states are supported only for Grafana-managed alert rules. @@ -121,7 +133,7 @@ However, in situations where strict monitoring is critical, relying solely on th Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example: -- Stale alert instances in the `Alerting` state transition to the `Normal` state when the series disappear. +- [Stale alert instances](#stale-alert-instances-missingseries) in the `Alerting` state transition to the `Normal` state when the series disappear. - If "no data" handling is configured to transition to a state other than `No Data`. - If "error" handling is configured to transition to a state other than `Error`. - If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state. @@ -130,7 +142,7 @@ In these situations, the evaluation state may differ from the alert state, and i The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example: -- Stale alert instances in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**. +- [Stale alert instances](#stale-alert-instances-missingseries) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**. - If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively. - If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.