From 579e412d19f3d7e845b192ef907d300a7086e987 Mon Sep 17 00:00:00 2001 From: Andres Martinez Gotor Date: Fri, 18 Jul 2025 12:02:13 +0200 Subject: [PATCH] Advisor: Document how to alert based on checks (#108176) --- .../administration/grafana-advisor/_index.md | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/docs/sources/administration/grafana-advisor/_index.md b/docs/sources/administration/grafana-advisor/_index.md index 28caf9b1953..4284432b73b 100644 --- a/docs/sources/administration/grafana-advisor/_index.md +++ b/docs/sources/administration/grafana-advisor/_index.md @@ -106,3 +106,100 @@ To resolve issues flagged by Grafana Advisor and maintain system reliability, fo - **Immediate Action:** Address "Action needed" items promptly to ensure system reliability - **Systematic Review:** After fixing flagged issues, use the "Refresh" button to confirm all checks pass - **Proactive Updates:** Address plugin update recommendations under "Investigation needed" even if they haven't caused failures yet + +## How to create an alert based on Advisor results + +This guide walks you through creating a Grafana alert that monitors Advisor check results and triggers when failures are detected. + +### Step 1: Create a service account and token + +1. Navigate to **Administration → Users and access → Service accounts** in your Grafana instance +2. Click **Add service account** +3. Provide a name (for example, "advisor-alert-service-account") +4. Set the role to **Admin** to ensure proper permissions +5. Click **Create** +6. In the service account details, click **Add service account token** +7. Provide a token name and set an appropriate expiration date +8. Click **Generate token** + +> **Important**: Copy the token value immediately and store it securely - you won't be able to see it again + +### Step 2: Set up the Grafana Infinity data source + +> **Important**: Use Infinity plugin >=v3.3.0 for the JQ parser used later. + +1. Go to **Connections → Add new connection** +2. Search for "Infinity" +3. If not installed, click **Install**. Wait for the plugin to be installed. +4. From the plugin page, click **Add new data source**. +5. Configure the data source: + - **Name**: Give it a descriptive name (e.g., "Advisor API") + - **Setup Authentication**: In the **Auth type**, select **Bearer Token**. In the **Auth details** section, paste the service account token from Step 1 and in the **Allowed hosts** section, write your Grafana app URL and click the "Add" button (e.g., `https://your-grafana-host.com`). +6. Click **Save & test** to verify the connection + +### Step 3: Create the alert rule + +Now you have everything you need to create an alert based on Advisor results. + +1. Navigate to **Alerting → Alert rules** +2. Click **New alert rule** +3. Provide a rule name (e.g., "Advisor Failures Alert") + +#### Configure the query + +1. **Data source**: Select the Infinity data source created in Step 3 +2. Configure the query settings: + - **Type**: JSON + - **Parser**: JQ + - **Source**: URL + - **Format**: Table + - **Method**: GET + - **URL**: Get this from the Advisor interface: + - Visit the Advisor in your Grafana instance + - Open browser Developer Tools (F12) → Network tab + - Look for a request ending with `/checks` + - Copy the full URL (format: `https:///apis/advisor.grafana.app/v0alpha1/namespaces//checks`) + +#### Configure parsing options + +**Rows/Root** (paste this JQ expression): + +```jq +.items | map({ + type: .metadata.labels["advisor.grafana.app/type"], + creationTimestamp: .metadata.creationTimestamp, + failuresCount: (.status.report.failures | length) +}) | group_by(.type) | map(sort_by(.creationTimestamp) | last) +``` + +This JQ query processes Grafana Advisor check data to get the most recent result for each check type. It transforms each check into a simplified object with type, timestamp, and failure count. +The result is a clean array showing the current state of each check type (data source, plugin, configuration, etc.) with their failure counts, perfect for alerting when any type has failures > 0. + +**Columns** (add these three columns): + +- **Selector**: `creationTimestamp`, **Format**: Time +- **Selector**: `failuresCount`, **Format**: Number +- **Selector**: `type`, **Format**: String + +#### Optional: Filter by check type + +If you want to alert only for specific check types: + +1. In the **Computed columns, Filter, Group by** section +2. Add a **Filter**: `type == "license"` (replace "license" with your desired check type) + +#### Set alert condition + +- **Alert condition**: Select "WHEN Last OF QUERY Is above 0" +- This will trigger when any check type has failures. +- Click on "Preview alert rule condition" to see the result of the query. + +#### Complete alert configuration + +Select your preferred evaluation (e.g. every 24 hours) and notification settings. + +### Step 4: Save the alert rule + +Click **Save** and check the alert is being triggered. + +Your alert is now configured to monitor Advisor results and notify you when failures are detected!