The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
grafana/docs/sources/alerting/fundamentals/alert-rule-evaluation/_index.md

113 lines
5.4 KiB

---
aliases:
- ../fundamentals/alert-rules/rule-evaluation/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/rule-evaluation/
canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/
description: Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state
keywords:
- grafana
- alerting
- evaluation
labels:
products:
- cloud
- enterprise
- oss
title: Alert rule evaluation
weight: 108
refs:
alerts-state-health:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/
---
# Alert rule evaluation
The criteria determining when an alert rule fires are based on two settings:
- [Evaluation group](#evaluation-group): how frequently the alert rule is evaluated.
- [Pending period](#pending-period): how long the condition must be met to start firing.
{{< figure src="/media/docs/alerting/alert-rule-evaluation.png" max-width="750px" alt="Set the evaluation behavior of the alert rule in Grafana." caption="Set alert rule evaluation" >}}
## Evaluation group
Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.
Each evaluation group contains an **evaluation interval** that determines how frequently the alert rule is checked. For instance, the evaluation may occur every `10s`, `30s`, `1m`, `10m`, etc.
**Evaluation strategies**
Alert rules in different groups can be evaluated simultaneously.
- **Grafana-managed** alert rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.
- **Data-source managed** alert rules within the same group are evaluated sequentially, one after the other—this is necessary to ensure that recording rules are evaluated before alert rules.
## Pending period
You can set a pending period to prevent unnecessary alerts from temporary issues.
The pending period specifies how long the condition must be met before firing, ensuring the condition is consistently met over a consecutive period.
You can also set the pending period to zero to skip it and have the alert fire immediately once the condition is met.
## Condition operator
There are several condition operators available.
- **and**: Two conditions before and after must be true for the overall condition to be true.
- **or**: If one of conditions before and after are true, the overall condition is true.
- **logic-or**: If the condition before logic-or is true, the overall condition is immediately true, without evaluating subsequent conditions.
Here are some examples of operators.
- `TRUE and TRUE or FALSE and FALSE` evaluate to `FALSE`, because last two conditions return `FALSE`.
- `TRUE and TRUE logic-or FALSE and FALSE` evaluate to `TRUE`, because the preceding condition returns `TRUE`.
## Evaluation example
Keep in mind:
- One alert rule can generate multiple alert instances - one for each time series produced by the alert rule's query.
- Alert instances from the same alert rule may be in different states. For instance, only one observed machine might start firing.
- Only **Alerting** and **Resolved** alert instances are routed to manage their notifications.
{{< figure src="/media/docs/alerting/alert-rule-evaluation-overview-statediagram-v2.png" alt="A diagram of the alert instance states and when to route their notifications." max-width="750px" >}}
<!--
Remove ///
stateDiagram-v2
direction LR
Normal --///> Pending
note right of Normal
Route "Resolved" alert instances
for notifications
end note
Pending --///> Alerting
Alerting --///> Normal: Resolved
note right of Alerting
Route "Alerting" alert instances
for notifications
end note
-->
Consider an alert rule with an **evaluation interval** set at every 30 seconds and a **pending period** of 90 seconds. The evaluation occurs as follows:
| Time | Condition | Alert instance state | Pending counter |
| ------------------------- | --------- | --------------------- | --------------- |
| 00:30 (first evaluation) | Not met | Normal | - |
| 01:00 (second evaluation) | Breached | Pending | 0s |
| 01:30 (third evaluation) | Breached | Pending | 30s |
| 02:00 (fourth evaluation) | Breached | Pending | 60s |
| 02:30 (fifth evaluation) | Breached | Alerting<sup>\*</sup> | 90s |
An alert instance is resolved when it transitions from the `Firing` to the `Normal` state. For instance, in the previous example:
| Time | Condition | Alert instance state | Pending counter |
| -------------------------- | --------- | ----------------------------- | --------------- |
| 03:00 (sixth evaluation) | Not met | Normal <sup>Resolved \*</sup> | 120s |
| 03:30 (seventh evaluation) | Not met | Normal | 150s |
To learn more about the state changes of alert rules and alert instances, refer to [State and health of alert rules](ref:alerts-state-health).