From ff4d7d35c607c22e26338108048d4e49c872b001 Mon Sep 17 00:00:00 2001 From: Pepe Cano <825430+ppcano@users.noreply.github.com> Date: Thu, 26 Jun 2025 21:06:34 +0200 Subject: [PATCH] docs(alerting): add new guidelines to the missing data alerting guide (#107232) --- docs/sources/alerting/best-practices/missing-data.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/sources/alerting/best-practices/missing-data.md b/docs/sources/alerting/best-practices/missing-data.md index fff5bb83422..047b12b00ec 100644 --- a/docs/sources/alerting/best-practices/missing-data.md +++ b/docs/sources/alerting/best-practices/missing-data.md @@ -201,6 +201,9 @@ Grafana Alerting handles distinct scenarios automatically. Here’s how to think - Use Grafana’s _No Data_ handling options to define what happens when a query returns nothing. - When _NoData_ is not an issue, consider rewriting the query to always return data — for example, in Prometheus, use `your_metric_query OR on() vector(0)` to return `0` when `your_metric_query` returns nothing. - Use `absent()` or `absent_over_time()` in Prometheus for fine-grained detection when a metric or label disappears entirely. +- If data is frequently missing due to scrape delays, use techniques to account for data delays: + - Adjust the **Time Range** query option in Grafana to evaluate slightly behind real time (e.g., set **To** to `now-1m`) to account for late data points. + - In Prometheus, you can use `last_over_time(metric_name[10m])` to pick the most recent sample within a given window. - Don’t alert on every instance by default. In dynamic environments, it’s better to aggregate and alert on symptoms — unless a missing individual instance directly impacts users. - If you’re getting too much noise from disappearing data, consider adjusting alerts, using `Keep Last State`, or routing those alerts differently. - For connectivity issues involving alert query failures, see the sibling guide: [Handling connectivity errors in Grafana Alerting](ref:connectivity-errors-guide).