diff --git a/docs/sources/alerting/set-up/provision-alerting-resources/file-provisioning/index.md b/docs/sources/alerting/set-up/provision-alerting-resources/file-provisioning/index.md new file mode 100644 index 00000000000..635b10c989d --- /dev/null +++ b/docs/sources/alerting/set-up/provision-alerting-resources/file-provisioning/index.md @@ -0,0 +1,699 @@ +--- +aliases: + - /docs/grafana/latest/alerting/provision-alerting-resources/file-provisioning + - /docs/grafana/latest/alerting/provision-alerting-resources/file-provisioning +description: Create and manage resources using file provisioning +keywords: + - grafana + - alerting + - alerting resources + - file provisioning +title: Create and manage alerting resources using file provisioning +weight: 100 +--- + +## Create and manage alerting resources using file provisioning + +Provision your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones. + +Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type. + +Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning. + +**Note:** + +Provisioning takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations). + +### Provision alert rules + +Create or delete alert rules in your Grafana instance(s). + +1. Create an alert rule in Grafana. +1. Use the [Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations) to extract the alert rule. +1. Copy the contents into a YAML or JSON configuration file. + + Example configuration files can be found below. + +1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). +1. Delete the alert rule in Grafana. + + **Note:** + + If you do not delete the alert rule, it will clash with the provisioned alert rule once uploaded. + +Here is an example of a configuration file for creating alert rules. + +```yaml +# config file version +apiVersion: 1 + +# List of rule groups to import or update +groups: + # organization ID, default = 1 + - orgId: 1 + # name of the rule group + name: my_rule_group + # name of the folder the rule group will be stored in + folder: my_first_folder + # interval that the rule group should evaluated at + interval: 60s + # list of rules that are part of the rule group + rules: + # unique identifier for the rule + - uid: my_id_1 + # title of the rule that will be displayed in the UI + title: my_first_rule + # which query should be used for the condition + condition: A + # list of query objects that should be executed on each + # evaluation - should be obtained trough the API + data: + - refId: A + datasourceUid: '-100' + model: + conditions: + - evaluator: + params: + - 3 + type: gt + operator: + type: and + query: + params: + - A + reducer: + type: last + type: query + datasource: + type: __expr__ + uid: '-100' + expression: 1==0 + intervalMs: 1000 + maxDataPoints: 43200 + refId: A + type: math + # UID of a dashboard that the alert rule should be linked to + dashboardUid: my_dashboard + # ID of the panel that the alert rule should be linked to + panelId: 123 + # the state the alert rule will have when no data is returned + # possible values: "NoData", "Alerting", "OK", default = NoData + noDataState: Alerting + # the state the alert rule will have when the query execution + # failed - possible values: "Error", "Alerting", "OK" + # default = Alerting + # for how long should the alert fire before alerting + for: 60s + # > a map of strings to pass around any data + annotations: + some_key: some_value + # a map of strings that can be used to filter and + # route alerts + labels: + team: sre_team_1 +``` + +Here is an example of a configuration file for deleting alert rules. + +```yaml +# config file version +apiVersion: 1 + +# List of alert rule UIDs that should be deleted +deleteRules: + # organization ID, default = 1 + - orgId: 1 + # unique identifier for the rule + uid: my_id_1 +``` + +### Provision contact points + +Create or delete contact points in your Grafana instance(s). + +1. Create a YAML or JSON configuration file. + + Example configuration files can be found below. + +1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). + +Here is an example of a configuration file for creating contact points. + +```yaml +# config file version +apiVersion: 1 + +# List of contact points to import or update +contactPoints: + # organization ID, default = 1 + - orgId: 1 + # name of the contact point + name: cp_1 + receivers: + # unique identifier for the receiver + - uid: first_uid + # type of the receiver + type: prometheus-alertmanager + # settings for the specific receiver type + settings: + url: http://test:9000 +``` + +Here is an example of a configuration file for deleting contact points. + +```yaml +# config file version +apiVersion: 1 + +# List of receivers that should be deleted +deleteContactPoints: + # organization ID, default = 1 + - orgId: 1 + # unique identifier for the receiver + uid: first_uid +``` + +#### Settings + +Here are some examples of settings you can use for the different +contact point types. + +##### Alertmanager + +```yaml +type: prometheus-alertmanager +settings: + # + url: http://localhost:9093 + # + basicAuthUser: abc + # + basicAuthPassword: abc123 +``` + +##### DingDing + +```yaml +type: dingding +settings: + # + url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx + # options: link, actionCard + msgType: link + # + message: | + {{ template "default.message" . }} +``` + +##### Discord + +```yaml +type: discord +settings: + # + url: https://discord/webhook + # + avatar_url: https://my_avatar + # + use_discord_username: Grafana + # + message: | + {{ template "default.message" . }} +``` + +##### E-Mail + +```yaml +type: email +settings: + # + addresses: me@example.com;you@example.com + # + singleEmail: false + # + message: my optional message to include + # + subject: | + {{ template "default.title" . }} +``` + +##### Google Hangouts Chat + +```yaml +type: googlechat +settings: + # + url: https://google/webhook + # + message: | + {{ template "default.message" . }} +``` + +##### Kafka + +```yaml +type: kafka +settings: + # + kafkaRestProxy: http://localhost:8082 + # + kafkaTopic: topic1 +``` + +##### LINE + +```yaml +type: line +settings: + # + token: xxx +``` + +##### Microsoft Teams + +```yaml +type: teams +settings: + # + url: https://ms_teams_url + # + title: | + {{ template "default.title" . }} + # + sectiontitle: '' + # + message: | + {{ template "default.message" . }} +``` + +##### OpsGenie + +```yaml +type: opsgenie +settings: + # + apiKey: xxx + # + apiUrl: https://api.opsgenie.com/v2/alerts + # + message: | + {{ template "default.title" . }} + # + description: some descriptive description + # + autoClose: false + # + overridePriority: false + # options: tags, details, both + sendTagsAs: both +``` + +##### PagerDuty + +```yaml +type: pagerduty +settings: + # + integrationKey: XXX + # options: critical, error, warning, info + severity: critical + # + class: ping failure + # + component: Grafana + # + group: app-stack + # + summary: | + {{ template "default.message" . }} +``` + +##### Pushover + +```yaml +type: pushover +settings: + # + apiToken: XXX + # + userKey: user1,user2 + # + device: device1,device2 + # options (high to low): 2,1,0,-1,-2 + priority: '2' + # + retry: '30' + # + expire: '120' + # + sound: siren + # + okSound: magic + # + message: | + {{ template "default.message" . }} +``` + +##### Slack + +```yaml +type: slack +settings: + # + recipient: alerting-dev + # + token: xxx + # + username: grafana_bot + # + icon_emoji: heart + # + icon_url: https://icon_url + # + mentionUsers: user_1,user_2 + # + mentionGroups: group_1,group_2 + # options: here, channel + mentionChannel: here + # Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary + url: https://some_webhook_url + # + endpointUrl: https://custom_url/api/chat.postMessage + # + title: | + {{ template "slack.default.title" . }} + text: | + {{ template "slack.default.text" . }} +``` + +##### Sensu Go + +```yaml +type: sensugo +settings: + # + url: http://sensu-api.local:8080 + # + apikey: xxx + # + entity: default + # + check: default + # + handler: some_handler + # + namespace: default + # + message: | + {{ template "default.message" . }} +``` + +##### Telegram + +```yaml +type: telegram +settings: + # + bottoken: xxx + # + chatid: some_chat_id + # + message: | + {{ template "default.message" . }} +``` + +##### Threema Gateway + +```yaml +type: threema +settings: + # + api_secret: xxx + # + gateway_id: A5K94S9 + # + recipient_id: A9R4KL4S +``` + +##### VictorOps + +```yaml +type: victorops +settings: + # + url: XXX + # options: CRITICAL, WARNING + messageType: CRITICAL +``` + +##### Webhook + +```yaml +type: webhook +settings: + # + url: https://endpoint_url + # options: POST, PUT + httpMethod: POST + # + username: abc + # + password: abc123 + # + authorization_scheme: Bearer + # + authorization_credentials: abc123 + # + maxAlerts: '10' +``` + +##### WeCom + +```yaml +type: wecom +settings: + # + url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx + # + message: | + {{ template "default.message" . }} + # + title: | + {{ template "default.title" . }} +``` + +### Provision notification policies + +Create or reset notification policies in your Grafana instance(s). + +1. Create a YAML or JSON configuration file. + + Example configuration files can be found below. + +2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). + +Here is an example of a configuration file for creating notification policiies. + +```yaml +# config file version +apiVersion: 1 + +# List of notification policies +policies: + # organization ID, default = 1 + - orgId: 1 + # name of the contact point that should be used for this route + receiver: grafana-default-email + # The labels by which incoming alerts are grouped together. For example, + # multiple alerts coming in for cluster=A and alertname=LatencyHigh would + # be batched into a single group. + # + # To aggregate by all possible labels use the special value '...' as + # the sole label name, for example: + # group_by: ['...'] + # This effectively disables aggregation entirely, passing through all + # alerts as-is. This is unlikely to be what you want, unless you have + # a very low alert volume or your upstream notification system performs + # its own grouping. + group_by: ['...'] + # a list of matchers that an alert has to fulfill to match the node + matchers: + - alertname = Watchdog + - severity =~ "warning|critical" + # Times when the route should be muted. These must match the name of a + # mute time interval. + # Additionally, the root node cannot have any mute times. + # When a route is muted it will not send any notifications, but + # otherwise acts normally (including ending the route-matching process + # if the `continue` option is not set) + mute_time_intervals: + - abc + # How long to initially wait to send a notification for a group + # of alerts. Allows to collect more initial alerts for the same group. + # (Usually ~0s to few minutes), default = 30s + group_wait: 30s + # How long to wait before sending a notification about new alerts that + # are added to a group of alerts for which an initial notification has + # already been sent. (Usually ~5m or more), default = 5m + group_internval: 5m + # How long to wait before sending a notification again if it has already + # been sent successfully for an alert. (Usually ~3h or more), default = 4h + repeat_interval: 4h + # Zero or more child routes + # routes: + # ... +``` + +Here is an example of a configuration file for resetting notification policies. + +```yaml +# config file version +apiVersion: 1 + +# List of orgIds that should be reset to the default policy +resetPolicies: + - 1 +``` + +### Provision templates + +Create or delete templates in your Grafana instance(s). + +1. Create a YAML or JSON configuration file. + + Example configuration files can be found below. + +2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). + +Here is an example of a configuration file for creating templates. + +```yaml +# config file version +apiVersion: 1 + +# List of templates to import or update +templates: + # organization ID, default = 1 + - orgID: 1 + # name of the template, must be unique + name: my_first_template + # content of the the template + template: Alerting with a custom text template +``` + +Here is an example of a configuration file for deleting templates. + +```yaml +# config file version +apiVersion: 1 + +# List of alert rule UIDs that should be deleted +deleteTemplates: + # organization ID, default = 1 + - orgId: 1 + # name of the template, must be unique + name: my_first_template +``` + +### Provision mute timings + +Create or delete mute timings in your Grafana instance(s). + +1. Create a YAML or JSON configuration file. + + Example configuration files can be found below. + +1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). + +Here is an example of a configuration file for creating mute timings. + +```yaml +# config file version +apiVersion: 1 + +# List of mute time intervals to import or update +muteTimes: + # organization ID, default = 1 + - orgId: 1 + # name of the mute time interval, must be unique + name: mti_1 + # time intervals that should trigger the muting + # refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0 + time_intervals: + - times: + - start_time: '06:00' + end_time: '23:59' + weekdays: ['monday:wednesday', 'saturday', 'sunday'] + months: ['1:3', 'may:august', 'december'] + years: ['2020:2022', '2030'] + days_of_month: ['1:5', '-3:-1'] +``` + +Here is an example of a configuration file for deleting mute timings. + +```yaml +# config file version +apiVersion: 1 + +# List of mute time intervals that should be deleted +deleteMuteTimes: + # organization ID, default = 1 + - orgId: 1 + # name of the mute time interval, must be unique + name: mti_1 +``` + +### File provisioning using Kubernetes + +If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps. + +1. Create one or more configuration maps as follows. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-alerting +data: + provisioning.yaml: | + templates: + - name: my_first_template + template: the content for my template +``` + +2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: grafana +spec: + replicas: 1 + selector: + matchLabels: + app: grafana + template: + metadata: + name: grafana + labels: + app: grafana + spec: + containers: + - name: grafana + image: grafana/grafana:latest + ports: + - name: grafana + containerPort: 3000 + volumeMounts: + - mountPath: /etc/grafana/provisioning/alerting + name: grafana-alerting + readOnly: false + volumes: + - name: grafana-alerting + configMap: + defaultMode: 420 + name: grafana-alerting +``` + +This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment. diff --git a/docs/sources/alerting/set-up/provision-alerting-resources/index.md b/docs/sources/alerting/set-up/provision-alerting-resources/index.md index 5584197a359..151e93f4ef4 100644 --- a/docs/sources/alerting/set-up/provision-alerting-resources/index.md +++ b/docs/sources/alerting/set-up/provision-alerting-resources/index.md @@ -38,689 +38,3 @@ Currently, provisioning for Grafana Alerting supports alert rules, contact point [Grafana Cloud provisioning](https://grafana.com/docs/grafana-cloud/infrastructure-as-code/terraform/) [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/alerting_provisioning) - -## Create and manage alerting resources using file provisioning - -Provision your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones. - -Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type. - -Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning. - -**Note:** - -Provisioning takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations). - -### Provision alert rules - -Create or delete alert rules in your Grafana instance(s). - -1. Create an alert rule in Grafana. -1. Use the [Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations) to extract the alert rule. -1. Copy the contents into a YAML or JSON configuration file. - - Example configuration files can be found below. - -1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). -1. Delete the alert rule in Grafana. - - **Note:** - - If you do not delete the alert rule, it will clash with the provisioned alert rule once uploaded. - -Here is an example of a configuration file for creating alert rules. - -```yaml -# config file version -apiVersion: 1 - -# List of rule groups to import or update -groups: - # organization ID, default = 1 - - orgId: 1 - # name of the rule group - name: my_rule_group - # name of the folder the rule group will be stored in - folder: my_first_folder - # interval that the rule group should evaluated at - interval: 60s - # list of rules that are part of the rule group - rules: - # unique identifier for the rule - - uid: my_id_1 - # title of the rule that will be displayed in the UI - title: my_first_rule - # which query should be used for the condition - condition: A - # list of query objects that should be executed on each - # evaluation - should be obtained trough the API - data: - - refId: A - datasourceUid: '-100' - model: - conditions: - - evaluator: - params: - - 3 - type: gt - operator: - type: and - query: - params: - - A - reducer: - type: last - type: query - datasource: - type: __expr__ - uid: '-100' - expression: 1==0 - intervalMs: 1000 - maxDataPoints: 43200 - refId: A - type: math - # UID of a dashboard that the alert rule should be linked to - dashboardUid: my_dashboard - # ID of the panel that the alert rule should be linked to - panelId: 123 - # the state the alert rule will have when no data is returned - # possible values: "NoData", "Alerting", "OK", default = NoData - noDataState: Alerting - # the state the alert rule will have when the query execution - # failed - possible values: "Error", "Alerting", "OK" - # default = Alerting - # for how long should the alert fire before alerting - for: 60s - # > a map of strings to pass around any data - annotations: - some_key: some_value - # a map of strings that can be used to filter and - # route alerts - labels: - team: sre_team_1 -``` - -Here is an example of a configuration file for deleting alert rules. - -```yaml -# config file version -apiVersion: 1 - -# List of alert rule UIDs that should be deleted -deleteRules: - # organization ID, default = 1 - - orgId: 1 - # unique identifier for the rule - uid: my_id_1 -``` - -### Provision contact points - -Create or delete contact points in your Grafana instance(s). - -1. Create a YAML or JSON configuration file. - - Example configuration files can be found below. - -1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). - -Here is an example of a configuration file for creating contact points. - -```yaml -# config file version -apiVersion: 1 - -# List of contact points to import or update -contactPoints: - # organization ID, default = 1 - - orgId: 1 - # name of the contact point - name: cp_1 - receivers: - # unique identifier for the receiver - - uid: first_uid - # type of the receiver - type: prometheus-alertmanager - # settings for the specific receiver type - settings: - url: http://test:9000 -``` - -Here is an example of a configuration file for deleting contact points. - -```yaml -# config file version -apiVersion: 1 - -# List of receivers that should be deleted -deleteContactPoints: - # organization ID, default = 1 - - orgId: 1 - # unique identifier for the receiver - uid: first_uid -``` - -#### Settings - -Here are some examples of settings you can use for the different -contact point types. - -##### Alertmanager - -```yaml -type: prometheus-alertmanager -settings: - # - url: http://localhost:9093 - # - basicAuthUser: abc - # - basicAuthPassword: abc123 -``` - -##### DingDing - -```yaml -type: dingding -settings: - # - url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx - # options: link, actionCard - msgType: link - # - message: | - {{ template "default.message" . }} -``` - -##### Discord - -```yaml -type: discord -settings: - # - url: https://discord/webhook - # - avatar_url: https://my_avatar - # - use_discord_username: Grafana - # - message: | - {{ template "default.message" . }} -``` - -##### E-Mail - -```yaml -type: email -settings: - # - addresses: me@example.com;you@example.com - # - singleEmail: false - # - message: my optional message to include - # - subject: | - {{ template "default.title" . }} -``` - -##### Google Hangouts Chat - -```yaml -type: googlechat -settings: - # - url: https://google/webhook - # - message: | - {{ template "default.message" . }} -``` - -##### Kafka - -```yaml -type: kafka -settings: - # - kafkaRestProxy: http://localhost:8082 - # - kafkaTopic: topic1 -``` - -##### LINE - -```yaml -type: line -settings: - # - token: xxx -``` - -##### Microsoft Teams - -```yaml -type: teams -settings: - # - url: https://ms_teams_url - # - title: | - {{ template "default.title" . }} - # - sectiontitle: '' - # - message: | - {{ template "default.message" . }} -``` - -##### OpsGenie - -```yaml -type: opsgenie -settings: - # - apiKey: xxx - # - apiUrl: https://api.opsgenie.com/v2/alerts - # - message: | - {{ template "default.title" . }} - # - description: some descriptive description - # - autoClose: false - # - overridePriority: false - # options: tags, details, both - sendTagsAs: both -``` - -##### PagerDuty - -```yaml -type: pagerduty -settings: - # - integrationKey: XXX - # options: critical, error, warning, info - severity: critical - # - class: ping failure - # - component: Grafana - # - group: app-stack - # - summary: | - {{ template "default.message" . }} -``` - -##### Pushover - -```yaml -type: pushover -settings: - # - apiToken: XXX - # - userKey: user1,user2 - # - device: device1,device2 - # options (high to low): 2,1,0,-1,-2 - priority: '2' - # - retry: '30' - # - expire: '120' - # - sound: siren - # - okSound: magic - # - message: | - {{ template "default.message" . }} -``` - -##### Slack - -```yaml -type: slack -settings: - # - recipient: alerting-dev - # - token: xxx - # - username: grafana_bot - # - icon_emoji: heart - # - icon_url: https://icon_url - # - mentionUsers: user_1,user_2 - # - mentionGroups: group_1,group_2 - # options: here, channel - mentionChannel: here - # Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary - url: https://some_webhook_url - # - endpointUrl: https://custom_url/api/chat.postMessage - # - title: | - {{ template "slack.default.title" . }} - text: | - {{ template "slack.default.text" . }} -``` - -##### Sensu Go - -```yaml -type: sensugo -settings: - # - url: http://sensu-api.local:8080 - # - apikey: xxx - # - entity: default - # - check: default - # - handler: some_handler - # - namespace: default - # - message: | - {{ template "default.message" . }} -``` - -##### Telegram - -```yaml -type: telegram -settings: - # - bottoken: xxx - # - chatid: some_chat_id - # - message: | - {{ template "default.message" . }} -``` - -##### Threema Gateway - -```yaml -type: threema -settings: - # - api_secret: xxx - # - gateway_id: A5K94S9 - # - recipient_id: A9R4KL4S -``` - -##### VictorOps - -```yaml -type: victorops -settings: - # - url: XXX - # options: CRITICAL, WARNING - messageType: CRITICAL -``` - -##### Webhook - -```yaml -type: webhook -settings: - # - url: https://endpoint_url - # options: POST, PUT - httpMethod: POST - # - username: abc - # - password: abc123 - # - authorization_scheme: Bearer - # - authorization_credentials: abc123 - # - maxAlerts: '10' -``` - -##### WeCom - -```yaml -type: wecom -settings: - # - url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx - # - message: | - {{ template "default.message" . }} - # - title: | - {{ template "default.title" . }} -``` - -### Provision notification policies - -Create or reset notification policies in your Grafana instance(s). - -1. Create a YAML or JSON configuration file. - - Example configuration files can be found below. - -2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). - -Here is an example of a configuration file for creating notification policiies. - -```yaml -# config file version -apiVersion: 1 - -# List of notification policies -policies: - # organization ID, default = 1 - - orgId: 1 - # name of the contact point that should be used for this route - receiver: grafana-default-email - # The labels by which incoming alerts are grouped together. For example, - # multiple alerts coming in for cluster=A and alertname=LatencyHigh would - # be batched into a single group. - # - # To aggregate by all possible labels use the special value '...' as - # the sole label name, for example: - # group_by: ['...'] - # This effectively disables aggregation entirely, passing through all - # alerts as-is. This is unlikely to be what you want, unless you have - # a very low alert volume or your upstream notification system performs - # its own grouping. - group_by: ['...'] - # a list of matchers that an alert has to fulfill to match the node - matchers: - - alertname = Watchdog - - severity =~ "warning|critical" - # Times when the route should be muted. These must match the name of a - # mute time interval. - # Additionally, the root node cannot have any mute times. - # When a route is muted it will not send any notifications, but - # otherwise acts normally (including ending the route-matching process - # if the `continue` option is not set) - mute_time_intervals: - - abc - # How long to initially wait to send a notification for a group - # of alerts. Allows to collect more initial alerts for the same group. - # (Usually ~0s to few minutes), default = 30s - group_wait: 30s - # How long to wait before sending a notification about new alerts that - # are added to a group of alerts for which an initial notification has - # already been sent. (Usually ~5m or more), default = 5m - group_internval: 5m - # How long to wait before sending a notification again if it has already - # been sent successfully for an alert. (Usually ~3h or more), default = 4h - repeat_interval: 4h - # Zero or more child routes - # routes: - # ... -``` - -Here is an example of a configuration file for resetting notification policies. - -```yaml -# config file version -apiVersion: 1 - -# List of orgIds that should be reset to the default policy -resetPolicies: - - 1 -``` - -### Provision templates - -Create or delete templates in your Grafana instance(s). - -1. Create a YAML or JSON configuration file. - - Example configuration files can be found below. - -2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). - -Here is an example of a configuration file for creating templates. - -```yaml -# config file version -apiVersion: 1 - -# List of templates to import or update -templates: - # organization ID, default = 1 - - orgID: 1 - # name of the template, must be unique - name: my_first_template - # content of the the template - template: Alerting with a custom text template -``` - -Here is an example of a configuration file for deleting templates. - -```yaml -# config file version -apiVersion: 1 - -# List of alert rule UIDs that should be deleted -deleteTemplates: - # organization ID, default = 1 - - orgId: 1 - # name of the template, must be unique - name: my_first_template -``` - -### Provision mute timings - -Create or delete mute timings in your Grafana instance(s). - -1. Create a YAML or JSON configuration file. - - Example configuration files can be found below. - -1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). - -Here is an example of a configuration file for creating mute timings. - -```yaml -# config file version -apiVersion: 1 - -# List of mute time intervals to import or update -muteTimes: - # organization ID, default = 1 - - orgId: 1 - # name of the mute time interval, must be unique - name: mti_1 - # time intervals that should trigger the muting - # refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0 - time_intervals: - - times: - - start_time: '06:00' - end_time: '23:59' - weekdays: ['monday:wednesday', 'saturday', 'sunday'] - months: ['1:3', 'may:august', 'december'] - years: ['2020:2022', '2030'] - days_of_month: ['1:5', '-3:-1'] -``` - -Here is an example of a configuration file for deleting mute timings. - -```yaml -# config file version -apiVersion: 1 - -# List of mute time intervals that should be deleted -deleteMuteTimes: - # organization ID, default = 1 - - orgId: 1 - # name of the mute time interval, must be unique - name: mti_1 -``` - -### File provisioning using Kubernetes - -If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps. - -1. Create one or more configuration maps as follows. - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: grafana-alerting -data: - provisioning.yaml: | - templates: - - name: my_first_template - template: the content for my template -``` - -2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s). - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: grafana -spec: - replicas: 1 - selector: - matchLabels: - app: grafana - template: - metadata: - name: grafana - labels: - app: grafana - spec: - containers: - - name: grafana - image: grafana/grafana:latest - ports: - - name: grafana - containerPort: 3000 - volumeMounts: - - mountPath: /etc/grafana/provisioning/alerting - name: grafana-alerting - readOnly: false - volumes: - - name: grafana-alerting - configMap: - defaultMode: 420 - name: grafana-alerting -``` - -This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment. diff --git a/docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md b/docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md new file mode 100644 index 00000000000..60aecf5341e --- /dev/null +++ b/docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md @@ -0,0 +1,316 @@ +--- +aliases: + - /docs/grafana/latest/alerting/provision-alerting-resources/terraform-provisioning +description: Create and manage alerting resources using Terraform +keywords: + - grafana + - alerting + - alerting resources + - provisioning + - Terraform +title: Create and manage alerting resources using Terraform +weight: 200 +--- + +# Create and manage alerting resources using Terraform + +Use Terraform’s Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code. + +For more information on managing your alerting resources using Terraform, refer to the [Grafana Provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) documentation. + +Complete the following tasks to create and manage your alerting resources using Terraform. + +1. Create an API key for provisioning. +1. Configure the Terraform provider. +1. Provision your alerting resources. + +## Before you begin + +- Ensure you have the grafana/grafana [Terraform provider](https://registry.terraform.io/providers/grafana/grafana/1.28.0) 1.27.0 or higher. + +- Ensure you are using Grafana 9.1 or higher. + +## Create an API key for provisioning + +You can [create a normal Grafana API key](https://grafana.com/docs/grafana/latest/administration/api-keys/) to authenticate Terraform with Grafana. Most existing tooling using API keys should automatically work with the new Grafana Alerting support. + +There are also dedicated RBAC roles for alerting provisioning. This lets you easily authenticate as a [service account](https://grafana.com/docs/grafana/latest/administration/service-accounts/) with the minimum permissions needed to provision your Alerting infrastructure. + +To create an API key for provisioning, complete the following steps. + +1. Create a new service account for your CI pipeline. +1. Assign the role “Access the alert rules Provisioning API.” +1. Create a new service account token. +1. Name and save the token for use in Terraform. + +Alternatively, you can use Terraform authentication: [basic auth](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication). + +## Configure the Terraform provider + +Grafana Alerting support is included as part of the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). + +The following is an example you can use to configure the Terraform provider. + +```terraform +terraform { + required_providers { + grafana = { + source = "grafana/grafana" + version = ">= 1.28.2" + } + } +} + +provider "grafana" { + url = + auth = +} +``` + +## Provision contact points and templates + +Contact points connect an alerting stack to the outside world. They tell Grafana how to connect to your external systems and where to deliver notifications. There are over fifteen different integrations to choose from. + +To provision contact points and templates, complete the following steps. + +1. Copy this code block into a .tf file on your local machine. + +This example uses a contact point that sends alert notifications to Slack. + +```terraform +resource "grafana_contact_point" "my_slack_contact_point" { + name = "Send to My Slack Channel" + + slack { + url = + text = <