docs(alerting): updates Missing series evaluations to resolve option (#106319)

docs(alerting): correct detailed process of the missingSeriesToEval option
pull/106354/head
Pepe Cano 7 months ago committed by GitHub
parent 33298550b4
commit be8d153dc3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
  1. 21
      docs/sources/alerting/best-practices/missing-data.md
  2. 12
      docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md

@ -88,8 +88,7 @@ The following tables illustrate both scenarios using the previous example, with
| :---- | :--------- | :--------- | :--------------------------- |
| 00:00 | 1.5s 🟢 | 1s 🟢 | ✅ No Alert |
| 01:00 | No Data ⚠ | No Data ⚠ | ⚠ No Alert (Silent Failure) |
| 02:00 | No Data ⚠ | No Data ⚠ | ⚠ No Alert (Silent Failure) |
| 03:00 | 1.4s 🟢 | 1s 🟢 | ✅ No Alert |
| 02:00 | 1.4s 🟢 | 1s 🟢 | ✅ No Alert |
**MissingSeries Scenario:** Only a specific series (`region2`) disappears:
@ -97,8 +96,7 @@ The following tables illustrate both scenarios using the previous example, with
| :---- | :------ | :---------------- | :--------------------------- |
| 00:00 | 1.5s 🟢 | 1s 🟢 | ✅ No Alert |
| 01:00 | 1.6s 🟢 | Missing Series ⚠ | ⚠ No Alert (Silent Failure) |
| 02:00 | 1.6s 🟢 | Missing Series ⚠ | ⚠ No Alert (Silent Failure) |
| 03:00 | 1.4s 🟢 | 1s 🟢 | ✅ No Alert |
| 02:00 | 1.4s 🟢 | 1s 🟢 | ✅ No Alert |
In both cases, something broke silently.
@ -162,14 +160,13 @@ Grafana marks missing series as [**stale**](ref:stale-alert-instances) after two
If an alert instance becomes stale, you’ll find it in the [alert history](ref:alert-history) as `Normal (Missing Series)` before it disappears. This table shows the eviction process from the previous example:
| Time | region1 | region2 | Alert triggered |
| :---- | :-------------------- | :--------------------------------- | :--------------------------------------------------------------------------------------------- |
| 00:00 | 1.5s 🟢 | 1s 🟢 | 🟢🟢 No Alerts |
| 01:00 | 3s 🔴 <br> `Alerting` | 3s 🔴 <br> `Alerting` | 🔴🔴 Alert instances triggered for both regions |
| 02:00 | 1.6s 🟢 | MissingSeries ⚠ <br> `Alerting` | 🟢🔴 Region2 missing, state maintained. |
| 03:00 | 1.6s 🟢 | MissingSeries ⚠ `Alerting` | 🟢🔴Region2 missing, state maintained. |
| 04:00 | 1.4s 🟢 | — | 🟢 🟢 `region2` Normal (Missing Series), resolved, and instance evicted; 📩 Notification sent. |
| 05:00 | 1.4s 🟢 | — | 🟢 No Alerts |
| Time | region1 | region2 | Alert triggered |
| :---- | :-------------------- | :------------------------------------ | :----------------------------------------------------------------------- |
| 00:00 | 1.5s 🟢 | 1s 🟢 | 🟢🟢 No Alerts |
| 01:00 | 3s 🔴 <br> `Alerting` | 3s 🔴 <br> `Alerting` | 🔴🔴 Alert instances triggered for both regions |
| 02:00 | 1.6s 🟢 | `(MissingSeries)` <br> `Alerting` | 🟢🔴 Region2 missing, state maintained. |
| 03:00 | 1.4s 🟢 | `(MissingSeries)` <br> `Normal` | 🟢🟢 `region2` was resolved, 📩 notification sent, and instance evicted. |
| 04:00 | 1.4s 🟢 | — | 🟢 No Alerts. `region2` was evicted. |
###

@ -39,10 +39,10 @@ This is different from the [**No Data** state](ref:no-data-state), which occurs
A stale alert instance transitions to the **Normal (MissingSeries)** state as **Resolved**, and is then evicted:
| Eval. Interval | 1 | 2 | 3 | 4 | 5 |
| :--------------- | :-- | :-------------- | :-------------- | :-- | :-- |
| Alert instance A | ✔ | ✔ | ✔ | ✔ | ✔ |
| Alert instance B | ✔ | `MissingSeries` | `MissingSeries` | 📩 | |
| Eval. Interval | 1 | 2 | 3 | 4 |
| :--------------- | :-- | :-------------- | :--------------------------------------- | :-- |
| Alert instance A | ✔ | ✔ | ✔ | ✔ |
| Alert instance B | ✔ | `MissingSeries` | `Normal(MissingSeries)` 📩<sup>\*</sup> | |
{{< admonition type="note" >}}
@ -60,9 +60,9 @@ The process for handling stale alert instances is as follows:
1. Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in [Missing series evaluations to resolve](#configure-missing-series-evaluations-to-resolve).
1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
1. If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
1. The alert instance is removed from the UI.

Loading…
Cancel
Save