--- aliases: - ../../provision-alerting-resources/file-provisioning/ canonical: https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/ description: Create and manage resources using file provisioning keywords: - grafana - alerting - alerting resources - file provisioning - provisioning labels: products: - enterprise - oss menuTitle: Use configuration files to provision title: Use configuration files to provision alerting resources weight: 100 --- # Use configuration files to provision alerting resources Manage your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones. Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type. Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning. For a complete guide about how Grafana provisions resources, refer to the [Provision Grafana][provisioning] documentation. {{< admonition type="note" >}} - You cannot edit provisioned resources from files in Grafana. You can only change the resource properties by changing the provisioning file and restarting Grafana or carrying out a hot reload. This prevents changes being made to the resource that would be overwritten if a file is provisioned again or a hot reload is carried out. - Importing takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Admin API](/docs/grafana//developers/http_api/admin#reload-provisioning-configurations). - Importing an existing alerting resource results in a conflict. First, when present, remove the resources you plan to import. {{< /admonition >}} ## Import alert rules Create or delete alert rules in your Grafana instance(s). 1. Create alert rules in Grafana. 1. [Export][alerting_export] and download a provisioning file for your alert rules. 1. Copy the contents into a YAML or JSON configuration file in the `provisioning/alerting` directory. Example configuration files can be found below. 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). Here is an example of a configuration file for creating alert rules. ```yaml # config file version apiVersion: 1 # List of rule groups to import or update groups: # organization ID, default = 1 - orgId: 1 # name of the rule group name: my_rule_group # name of the folder the rule group will be stored in folder: my_first_folder # interval that the rule group should evaluated at interval: 60s # list of rules that are part of the rule group rules: # unique identifier for the rule. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed. - uid: my_id_1 # title of the rule that will be displayed in the UI title: my_first_rule # which query should be used for the condition condition: A # list of query objects that should be executed on each # evaluation - should be obtained through the API data: - refId: A datasourceUid: '__expr__' model: conditions: - evaluator: params: - 3 type: gt operator: type: and query: params: - A reducer: type: last type: query datasource: type: __expr__ uid: '__expr__' expression: 1==0 intervalMs: 1000 maxDataPoints: 43200 refId: A type: math # UID of a dashboard that the alert rule should be linked to dashboardUid: my_dashboard # ID of the panel that the alert rule should be linked to panelId: 123 # the state the alert rule will have when no data is returned # possible values: "NoData", "Alerting", "OK", default = NoData noDataState: Alerting # the state the alert rule will have when the query execution # failed - possible values: "Error", "Alerting", "OK" # default = Alerting execErrState: Alerting # for how long should the alert fire before alerting for: 60s # > a map of strings to pass around any data annotations: some_key: some_value # a map of strings that can be used to filter and # route alerts labels: team: sre_team_1 ``` Here is an example of a configuration file for deleting alert rules. ```yaml # config file version apiVersion: 1 # List of alert rule UIDs that should be deleted deleteRules: # organization ID, default = 1 - orgId: 1 # unique identifier for the rule uid: my_id_1 ``` ## Import contact points Create or delete contact points in your Grafana instance(s). 1. Create a contact point in Grafana. 1. [Export][alerting_export] and download a provisioning file for your contact point. 1. Copy the contents into a YAML or JSON configuration file in the `provisioning/alerting` directory. Example configuration files can be found below. 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). Here is an example of a configuration file for creating contact points. ```yaml # config file version apiVersion: 1 # List of contact points to import or update contactPoints: # organization ID, default = 1 - orgId: 1 # name of the contact point name: cp_1 receivers: # unique identifier for the receiver. Should not exceed 40 symbols. Only letters, numbers, - (hyphen), and _ (underscore) allowed. - uid: first_uid # type of the receiver type: prometheus-alertmanager # Disable the additional [Incident Resolved] follow-up alert, default = false disableResolveMessage: false # settings for the specific receiver type settings: url: http://test:9000 ``` Here is an example of a configuration file for deleting contact points. ```yaml # config file version apiVersion: 1 # List of receivers that should be deleted deleteContactPoints: # organization ID, default = 1 - orgId: 1 # unique identifier for the receiver uid: first_uid ``` ### Settings Here are some examples of settings you can use for the different contact point integrations. {{< collapse title="Alertmanager" >}} #### Alertmanager ```yaml type: prometheus-alertmanager settings: # url: http://localhost:9093 # basicAuthUser: abc # basicAuthPassword: abc123 ``` {{< /collapse >}} {{< collapse title="DingDing" >}} #### DingDing ```yaml type: dingding settings: # url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx # options: link, actionCard msgType: link # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Discord" >}} #### Discord ```yaml type: discord settings: # url: https://discord/webhook # avatar_url: https://my_avatar # use_discord_username: Grafana # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="E-Mail" >}} #### E-Mail ```yaml type: email settings: # addresses: me@example.com;you@example.com # singleEmail: false # message: my optional message to include # subject: | {{ template "default.title" . }} ``` {{< /collapse >}} {{< collapse title="Google Chat" >}} #### Google Chat ```yaml type: googlechat settings: # url: https://google/webhook # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Kafka" >}} #### Kafka ```yaml type: kafka settings: # kafkaRestProxy: http://localhost:8082 # kafkaTopic: topic1 ``` {{< /collapse >}} {{< collapse title="LINE" >}} #### LINE ```yaml type: line settings: # token: xxx ``` {{< /collapse >}} {{< collapse title="Microsoft Teams" >}} #### Microsoft Teams ```yaml type: teams settings: # url: https://ms_teams_url # title: | {{ template "default.title" . }} # sectiontitle: '' # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="OpsGenie" >}} #### OpsGenie ```yaml type: opsgenie settings: # apiKey: xxx # apiUrl: https://api.opsgenie.com/v2/alerts # message: | {{ template "default.title" . }} # description: some descriptive description # autoClose: false # overridePriority: false # options: tags, details, both sendTagsAs: both ``` {{< /collapse >}} {{< collapse title="PagerDuty" >}} #### PagerDuty ```yaml type: pagerduty settings: # the 32-character Events API key https://support.pagerduty.com/docs/api-access-keys#events-api-keys integrationKey: XXX # options: critical, error, warning, info severity: critical # class: ping failure # component: Grafana # group: app-stack # summary: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Pushover" >}} #### Pushover ```yaml type: pushover settings: # apiToken: XXX # userKey: user1,user2 # device: device1,device2 # options (high to low): 2,1,0,-1,-2 priority: '2' # retry: '30' # expire: '120' # the number of seconds before a message expires and is deleted automatically. Examples: 10s, 5m30s, 8h. ttl: # sound: siren # okSound: magic # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Slack" >}} #### Slack ```yaml type: slack settings: # recipient: alerting-dev # token: xxx # username: grafana_bot # icon_emoji: heart # icon_url: https://icon_url # mentionUsers: user_1,user_2 # mentionGroups: group_1,group_2 # options: here, channel mentionChannel: here # Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary url: https://some_webhook_url # endpointUrl: https://custom_url/api/chat.postMessage # title: | {{ template "slack.default.title" . }} text: | {{ template "slack.default.text" . }} ``` {{< /collapse >}} {{< collapse title="Sensu Go" >}} #### Sensu Go ```yaml type: sensugo settings: # url: http://sensu-api.local:8080 # apikey: xxx # entity: default # check: default # handler: some_handler # namespace: default # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Telegram" >}} #### Telegram ```yaml type: telegram settings: # bottoken: xxx # chatid: some_chat_id # message: | {{ template "default.message" . }} ``` {{< /collapse >}} {{< collapse title="Threema Gateway" >}} #### Threema Gateway ```yaml type: threema settings: # api_secret: xxx # gateway_id: A5K94S9 # recipient_id: A9R4KL4S ``` {{< /collapse >}} {{< collapse title="VictorOps" >}} #### VictorOps ```yaml type: victorops settings: # url: XXX # options: CRITICAL, WARNING messageType: CRITICAL ``` {{< /collapse >}} {{< collapse title="Webhook" >}} #### Webhook ```yaml type: webhook settings: # url: https://endpoint_url # options: POST, PUT httpMethod: POST # username: abc # password: abc123 # authorization_scheme: Bearer # authorization_credentials: abc123 # maxAlerts: '10' ``` {{< /collapse >}} {{< collapse title="WeCom" >}} #### WeCom ```yaml type: wecom settings: # url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx # message: | {{ template "default.message" . }} # title: | {{ template "default.title" . }} ``` {{< /collapse >}} ## Import notification policies Create or reset the notification policy tree in your Grafana instance(s). In Grafana, the entire notification policy tree is considered a single, large resource. Add new specific policies as sub-policies under the root policy. Since specific policies may depend on each other, you cannot provision subsets of the policy tree; the entire tree must be defined in a single place. {{% admonition type="warning" %}} Since the policy tree is a single resource, provisioning it will overwrite a policy tree created through any other means. {{< /admonition >}} 1. Create a notification policy in Grafana. 1. [Export][alerting_export] and download a provisioning file for your notification policy. 1. Copy the contents into a YAML or JSON configuration file in the `provisioning/alerting` directory. Example configuration files can be found below. 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). Here is an example of a configuration file for creating notification policies. ```yaml # config file version apiVersion: 1 # List of notification policies policies: # organization ID, default = 1 - orgId: 1 # name of the contact point that should be used for this route receiver: grafana-default-email # The labels by which incoming alerts are grouped together. For example, # multiple alerts coming in for cluster=A and alertname=LatencyHigh would # be batched into a single group. # # To aggregate by all possible labels use the special value '...' as # the sole label name, for example: # group_by: ['...'] # This effectively disables aggregation entirely, passing through all # alerts as-is. This is unlikely to be what you want, unless you have # a very low alert volume or your upstream notification system performs # its own grouping. group_by: ['...'] # a list of prometheus-like matchers that an alert rule has to fulfill to match the node (allowed chars # [a-zA-Z_:]) matchers: - alertname = Watchdog - service_id_X = serviceX - severity =~ "warning|critical" # a list of grafana-like matchers that an alert rule has to fulfill to match the node object_matchers: - ['alertname', '=', 'CPUUsage'] - ['service_id-X', '=', 'serviceX'] - ['severity', '=~', 'warning|critical'] # Times when the route should be muted. These must match the name of a # mute time interval. # Additionally, the root node cannot have any mute times. # When a route is muted it will not send any notifications, but # otherwise acts normally (including ending the route-matching process # if the `continue` option is not set) mute_time_intervals: - abc # How long to initially wait to send a notification for a group # of alerts. Allows to collect more initial alerts for the same group. # (Usually ~0s to few minutes), default = 30s group_wait: 30s # How long to wait before sending a notification about new alerts that # are added to a group of alerts for which an initial notification has # already been sent. (Usually ~5m or more), default = 5m group_interval: 5m # How long to wait before sending a notification again if it has already # been sent successfully for an alert. (Usually ~3h or more), default = 4h repeat_interval: 4h # Zero or more child policies. The schema is the same as the root policy. # routes: # # Another recursively nested policy... # - receiver: another-receiver # matchers: # - ... # ... ``` Here is an example of a configuration file for resetting the policy tree back to its default value: ```yaml # config file version apiVersion: 1 # List of orgIds that should be reset to the default policy resetPolicies: - 1 ``` ## Import templates Create or delete templates in your Grafana instance(s). 1. Create a YAML or JSON configuration file. Example configuration files can be found below. 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). Here is an example of a configuration file for creating templates. ```yaml # config file version apiVersion: 1 # List of templates to import or update templates: # organization ID, default = 1 - orgId: 1 # name of the template, must be unique name: my_first_template # content of the template template: Alerting with a custom text template ``` Here is an example of a configuration file for deleting templates. ```yaml # config file version apiVersion: 1 # List of alert rule UIDs that should be deleted deleteTemplates: # organization ID, default = 1 - orgId: 1 # name of the template, must be unique name: my_first_template ``` ## Import mute timings Create or delete mute timings in your Grafana instance(s). 1. Create a YAML or JSON configuration file. Example configuration files can be found below. 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). Here is an example of a configuration file for creating mute timings. ```yaml # config file version apiVersion: 1 # List of mute time intervals to import or update muteTimes: # organization ID, default = 1 - orgId: 1 # name of the mute time interval, must be unique name: mti_1 # time intervals that should trigger the muting # refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0 time_intervals: - times: - start_time: '06:00' end_time: '23:59' location: 'UTC' weekdays: ['monday:wednesday', 'saturday', 'sunday'] months: ['1:3', 'may:august', 'december'] years: ['2020:2022', '2030'] days_of_month: ['1:5', '-3:-1'] ``` Here is an example of a configuration file for deleting mute timings. ```yaml # config file version apiVersion: 1 # List of mute time intervals that should be deleted deleteMuteTimes: # organization ID, default = 1 - orgId: 1 # name of the mute time interval, must be unique name: mti_1 ``` ## File provisioning using Kubernetes If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps. 1. Create one or more configuration maps as follows. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: grafana-alerting data: provisioning.yaml: | templates: - name: my_first_template template: the content for my template ``` 1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: grafana spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: name: grafana labels: app: grafana spec: containers: - name: grafana image: grafana/grafana:latest ports: - name: grafana containerPort: 3000 volumeMounts: - mountPath: /etc/grafana/provisioning/alerting name: grafana-alerting readOnly: false volumes: - name: grafana-alerting configMap: defaultMode: 420 name: grafana-alerting ``` This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment. Grafana still requires a database for normal operation, you do not need to persist the contents of the database between restarts if all objects are provisioned using files. **Useful Links:** [Grafana provisioning][provisioning] {{% docs/reference %}} [alerting_export]: "/docs/grafana/ -> /docs/grafana//alerting/set-up/provision-alerting-resources/export-alerting-resources" [alerting_export]: "/docs/grafana-cloud/ -> /docs/grafana//alerting/set-up/provision-alerting-resources/export-alerting-resources" [provisioning]: "/docs/grafana/ -> /docs/grafana//administration/provisioning" [provisioning]: "/docs/grafana-cloud/ -> /docs/grafana//administration/provisioning" {{% /docs/reference %}}