The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

prometheus hacktoberfest metrics monitoring alerting grafana go influxdb mysql postgres analytics data-visualization dashboard business-intelligence elasticsearch

3.1 KiB

Raw Blame History

description	labels	menuTitle	title	weight
Detect and revoke leaked Grafana service account tokens	[{products [enterprise oss]}]	Configure secret scanning	Configure Grafana secret scanning and notifications	1000

Configure Grafana secret scanning and notifications

With Grafana, you can use the GitHub Secret Scanning service to determine if your service account tokens have been leaked on GitHub.

When GitHub Secret Scanning detects a Grafana secret, its hash is stored in Grafana Labs' secret scanning service.

Grafana instances, whether on-premises or on the cloud, can use this service to verify if a token generated by the instance has been made public. This verification is done by comparing the token's hash with the exposed token's hash.

If the service detects a leaked token, it immediately revokes it, making it useless, and logs the event.

{{% admonition type="note" %}} If the revoke option is disabled, the service only sends a notification to the configured webhook URL and logs the event. The token is not automatically revoked. {{% /admonition %}}

You can also configure the service to send an outgoing webhook notification to a webhook URL.

The notification includes a JSON payload that contains the following data:

{
  "alert_uid": "c9ce50a1-d66b-45e4-9b5d-175766cfc026",
  "link_to_upstream_details": <URL to token leak>,
  "message": "Token of type grafana_service_account_token with name
sa-the-toucans has been publicly exposed in <URL to token leak>.
Grafana has revoked this token",
  "state": "alerting",
  "title": "SecretScan Alert: Grafana Token leaked"
}

{{% admonition type="note" %}} Secret scanning is disabled by default. Outgoing connections are made once you enable it. {{% /admonition %}}

Before you begin

Ensure all your API keys have been migrated to service accounts. For more information about service account migration, refer to Migrate API keys to Grafana service accounts.

Configure secret scanning

Open the Grafana configuration file.
In the [secretscan] section, update the following parameters:

[secretscan]
# Enable secretscan feature
enabled = true

# Whether to revoke the token if a leak is detected or just send a notification
revoke = true

Save the configuration file and restart Grafana.

Configure outgoing webhook notifications

Create an oncall integration of the type Webhook and set up alerts. To learn how to create a Grafana OnCall integration, refer to Inbound Webhook integrations for Grafana OnCall.
Copy the webhook URL of the new integration.
Open the Grafana configuration file.
In the [secretscan] section, update the following parameters, replacing the URL with the webhook URL you copied in step 2.

[secretscan]
# URL to send a webhook payload in oncall format
oncall_url = https://example.url/integrations/v1/webhook/3a359nib9eweAd9lAAAETVdOx/