IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-25 18:55:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

Docs: adds terraform provisioning docs (#55660)

Browse Source 
* Docs: adds set up section and alertmanager

* deletes info that got copied to new file
This commit is contained in:
brendamuir 2022-09-28 09:18:46 +01:00 committed by GitHub
parent 445fed0875
commit b7f356df31
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 1043 additions and 686 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

699
docs/sources/alerting/set-up/provision-alerting-resources/file-provisioning/index.md Normal file
View File

@ -0,0 +1,699 @@
---
aliases:
- /docs/grafana/latest/alerting/provision-alerting-resources/file-provisioning
- /docs/grafana/latest/alerting/provision-alerting-resources/file-provisioning
description: Create and manage resources using file provisioning
keywords:
- grafana
- alerting
- alerting resources
- file provisioning
title: Create and manage alerting resources using file provisioning
weight: 100
---
## Create and manage alerting resources using file provisioning
Provision your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones.
Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type.
Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.
**Note:**
Provisioning takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations).
### Provision alert rules
Create or delete alert rules in your Grafana instance(s).
1. Create an alert rule in Grafana.
1. Use the [Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations) to extract the alert rule.
1. Copy the contents into a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
1. Delete the alert rule in Grafana.
**Note:**
If you do not delete the alert rule, it will clash with the provisioned alert rule once uploaded.
Here is an example of a configuration file for creating alert rules.
```yaml
# config file version
apiVersion: 1
# List of rule groups to import or update
groups:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the rule group
name: my_rule_group
# <string, required> name of the folder the rule group will be stored in
folder: my_first_folder
# <duration, required> interval that the rule group should evaluated at
interval: 60s
# <list, required> list of rules that are part of the rule group
rules:
# <string, required> unique identifier for the rule
- uid: my_id_1
# <string, required> title of the rule that will be displayed in the UI
title: my_first_rule
# <string, required> which query should be used for the condition
condition: A
# <list, required> list of query objects that should be executed on each
# evaluation - should be obtained trough the API
data:
- refId: A
datasourceUid: '-100'
model:
conditions:
- evaluator:
params:
- 3
type: gt
operator:
type: and
query:
params:
- A
reducer:
type: last
type: query
datasource:
type: __expr__
uid: '-100'
expression: 1==0
intervalMs: 1000
maxDataPoints: 43200
refId: A
type: math
# <string> UID of a dashboard that the alert rule should be linked to
dashboardUid: my_dashboard
# <int> ID of the panel that the alert rule should be linked to
panelId: 123
# <string> the state the alert rule will have when no data is returned
# possible values: "NoData", "Alerting", "OK", default = NoData
noDataState: Alerting
# <string> the state the alert rule will have when the query execution
# failed - possible values: "Error", "Alerting", "OK"
# default = Alerting
# <duration, required> for how long should the alert fire before alerting
for: 60s
# <map<string, string>> a map of strings to pass around any data
annotations:
some_key: some_value
# <map<string, string> a map of strings that can be used to filter and
# route alerts
labels:
team: sre_team_1
```
Here is an example of a configuration file for deleting alert rules.
```yaml
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteRules:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the rule
uid: my_id_1
```
### Provision contact points
Create or delete contact points in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating contact points.
```yaml
# config file version
apiVersion: 1
# List of contact points to import or update
contactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the contact point
name: cp_1
receivers:
# <string, required> unique identifier for the receiver
- uid: first_uid
# <string, required> type of the receiver
type: prometheus-alertmanager
# <object, required> settings for the specific receiver type
settings:
url: http://test:9000
```
Here is an example of a configuration file for deleting contact points.
```yaml
# config file version
apiVersion: 1
# List of receivers that should be deleted
deleteContactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the receiver
uid: first_uid
```
#### Settings
Here are some examples of settings you can use for the different
contact point types.
##### Alertmanager
```yaml
type: prometheus-alertmanager
settings:
# <string, required>
url: http://localhost:9093
# <string>
basicAuthUser: abc
# <string>
basicAuthPassword: abc123
```
##### DingDing
```yaml
type: dingding
settings:
# <string, required>
url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx
# <string> options: link, actionCard
msgType: link
# <string>
message: |
{{ template "default.message" . }}
```
##### Discord
```yaml
type: discord
settings:
# <string, required>
url: https://discord/webhook
# <string>
avatar_url: https://my_avatar
# <string>
use_discord_username: Grafana
# <string>
message: |
{{ template "default.message" . }}
```
##### E-Mail
```yaml
type: email
settings:
# <string, required>
addresses: me@example.com;you@example.com
# <bool>
singleEmail: false
# <string>
message: my optional message to include
# <string>
subject: |
{{ template "default.title" . }}
```
##### Google Hangouts Chat
```yaml
type: googlechat
settings:
# <string, required>
url: https://google/webhook
# <string>
message: |
{{ template "default.message" . }}
```
##### Kafka
```yaml
type: kafka
settings:
# <string, required>
kafkaRestProxy: http://localhost:8082
# <string, required>
kafkaTopic: topic1
```
##### LINE
```yaml
type: line
settings:
# <string, required>
token: xxx
```
##### Microsoft Teams
```yaml
type: teams
settings:
# <string, required>
url: https://ms_teams_url
# <string>
title: |
{{ template "default.title" . }}
# <string>
sectiontitle: ''
# <string>
message: |
{{ template "default.message" . }}
```
##### OpsGenie
```yaml
type: opsgenie
settings:
# <string, required>
apiKey: xxx
# <string, required>
apiUrl: https://api.opsgenie.com/v2/alerts
# <string>
message: |
{{ template "default.title" . }}
# <string>
description: some descriptive description
# <bool>
autoClose: false
# <bool>
overridePriority: false
# <string> options: tags, details, both
sendTagsAs: both
```
##### PagerDuty
```yaml
type: pagerduty
settings:
# <string, required>
integrationKey: XXX
# <string> options: critical, error, warning, info
severity: critical
# <string>
class: ping failure
# <string>
component: Grafana
# <string>
group: app-stack
# <string>
summary: |
{{ template "default.message" . }}
```
##### Pushover
```yaml
type: pushover
settings:
# <string, required>
apiToken: XXX
# <string, required>
userKey: user1,user2
# <string>
device: device1,device2
# <string> options (high to low): 2,1,0,-1,-2
priority: '2'
# <string>
retry: '30'
# <string>
expire: '120'
# <string>
sound: siren
# <string>
okSound: magic
# <string>
message: |
{{ template "default.message" . }}
```
##### Slack
```yaml
type: slack
settings:
# <string, required>
recipient: alerting-dev
# <string, required>
token: xxx
# <string>
username: grafana_bot
# <string>
icon_emoji: heart
# <string>
icon_url: https://icon_url
# <string>
mentionUsers: user_1,user_2
# <string>
mentionGroups: group_1,group_2
# <string> options: here, channel
mentionChannel: here
# <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
url: https://some_webhook_url
# <string>
endpointUrl: https://custom_url/api/chat.postMessage
# <string>
title: |
{{ template "slack.default.title" . }}
text: |
{{ template "slack.default.text" . }}
```
##### Sensu Go
```yaml
type: sensugo
settings:
# <string, required>
url: http://sensu-api.local:8080
# <string, required>
apikey: xxx
# <string>
entity: default
# <string>
check: default
# <string>
handler: some_handler
# <string>
namespace: default
# <string>
message: |
{{ template "default.message" . }}
```
##### Telegram
```yaml
type: telegram
settings:
# <string, required>
bottoken: xxx
# <string, required>
chatid: some_chat_id
# <string>
message: |
{{ template "default.message" . }}
```
##### Threema Gateway
```yaml
type: threema
settings:
# <string, required>
api_secret: xxx
# <string, required>
gateway_id: A5K94S9
# <string, required>
recipient_id: A9R4KL4S
```
##### VictorOps
```yaml
type: victorops
settings:
# <string, required>
url: XXX
# <string> options: CRITICAL, WARNING
messageType: CRITICAL
```
##### Webhook
```yaml
type: webhook
settings:
# <string, required>
url: https://endpoint_url
# <string> options: POST, PUT
httpMethod: POST
# <string>
username: abc
# <string>
password: abc123
# <string>
authorization_scheme: Bearer
# <string>
authorization_credentials: abc123
# <string>
maxAlerts: '10'
```
##### WeCom
```yaml
type: wecom
settings:
# <string, required>
url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx
# <string>
message: |
{{ template "default.message" . }}
# <string>
title: |
{{ template "default.title" . }}
```
### Provision notification policies
Create or reset notification policies in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating notification policiies.
```yaml
# config file version
apiVersion: 1
# List of notification policies
policies:
# <int> organization ID, default = 1
- orgId: 1
# <string> name of the contact point that should be used for this route
receiver: grafana-default-email
# <list> The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
#
# To aggregate by all possible labels use the special value '...' as
# the sole label name, for example:
# group_by: ['...']
# This effectively disables aggregation entirely, passing through all
# alerts as-is. This is unlikely to be what you want, unless you have
# a very low alert volume or your upstream notification system performs
# its own grouping.
group_by: ['...']
# <list> a list of matchers that an alert has to fulfill to match the node
matchers:
- alertname = Watchdog
- severity =~ "warning|critical"
# <list> Times when the route should be muted. These must match the name of a
# mute time interval.
# Additionally, the root node cannot have any mute times.
# When a route is muted it will not send any notifications, but
# otherwise acts normally (including ending the route-matching process
# if the `continue` option is not set)
mute_time_intervals:
- abc
# <duration> How long to initially wait to send a notification for a group
# of alerts. Allows to collect more initial alerts for the same group.
# (Usually ~0s to few minutes), default = 30s
group_wait: 30s
# <duration> How long to wait before sending a notification about new alerts that
# are added to a group of alerts for which an initial notification has
# already been sent. (Usually ~5m or more), default = 5m
group_internval: 5m
# <duration> How long to wait before sending a notification again if it has already
# been sent successfully for an alert. (Usually ~3h or more), default = 4h
repeat_interval: 4h
# <list> Zero or more child routes
# routes:
# ...
```
Here is an example of a configuration file for resetting notification policies.
```yaml
# config file version
apiVersion: 1
# List of orgIds that should be reset to the default policy
resetPolicies:
- 1
```
### Provision templates
Create or delete templates in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating templates.
```yaml
# config file version
apiVersion: 1
# List of templates to import or update
templates:
# <int> organization ID, default = 1
- orgID: 1
# <string, required> name of the template, must be unique
name: my_first_template
# <string, required> content of the the template
template: Alerting with a custom text template
```
Here is an example of a configuration file for deleting templates.
```yaml
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteTemplates:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the template, must be unique
name: my_first_template
```
### Provision mute timings
Create or delete mute timings in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating mute timings.
```yaml
# config file version
apiVersion: 1
# List of mute time intervals to import or update
muteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
# <list> time intervals that should trigger the muting
# refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0
time_intervals:
- times:
- start_time: '06:00'
end_time: '23:59'
weekdays: ['monday:wednesday', 'saturday', 'sunday']
months: ['1:3', 'may:august', 'december']
years: ['2020:2022', '2030']
days_of_month: ['1:5', '-3:-1']
```
Here is an example of a configuration file for deleting mute timings.
```yaml
# config file version
apiVersion: 1
# List of mute time intervals that should be deleted
deleteMuteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
```
### File provisioning using Kubernetes
If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps.
1. Create one or more configuration maps as follows.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerting
data:
provisioning.yaml: |
templates:
- name: my_first_template
template: the content for my template
```
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- name: grafana
containerPort: 3000
volumeMounts:
- mountPath: /etc/grafana/provisioning/alerting
name: grafana-alerting
readOnly: false
volumes:
- name: grafana-alerting
configMap:
defaultMode: 420
name: grafana-alerting
```
This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment.

686
docs/sources/alerting/set-up/provision-alerting-resources/index.md
View File

@ -38,689 +38,3 @@ Currently, provisioning for Grafana Alerting supports alert rules, contact point
[Grafana Cloud provisioning](https://grafana.com/docs/grafana-cloud/infrastructure-as-code/terraform/) [Grafana Cloud provisioning](https://grafana.com/docs/grafana-cloud/infrastructure-as-code/terraform/)
[Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/alerting_provisioning) [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/alerting_provisioning)
## Create and manage alerting resources using file provisioning
Provision your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones.
Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type.
Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.
**Note:**
Provisioning takes place during the initial set up of your Grafana system, but you can re-run it at any time using the [Grafana Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations).
### Provision alert rules
Create or delete alert rules in your Grafana instance(s).
1. Create an alert rule in Grafana.
1. Use the [Alerting provisioning API](https://grafana.com/docs/grafana/latest/developers/http_api/admin/#reload-provisioning-configurations) to extract the alert rule.
1. Copy the contents into a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
1. Delete the alert rule in Grafana.
**Note:**
If you do not delete the alert rule, it will clash with the provisioned alert rule once uploaded.
Here is an example of a configuration file for creating alert rules.
```yaml
# config file version
apiVersion: 1
# List of rule groups to import or update
groups:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the rule group
name: my_rule_group
# <string, required> name of the folder the rule group will be stored in
folder: my_first_folder
# <duration, required> interval that the rule group should evaluated at
interval: 60s
# <list, required> list of rules that are part of the rule group
rules:
# <string, required> unique identifier for the rule
- uid: my_id_1
# <string, required> title of the rule that will be displayed in the UI
title: my_first_rule
# <string, required> which query should be used for the condition
condition: A
# <list, required> list of query objects that should be executed on each
# evaluation - should be obtained trough the API
data:
- refId: A
datasourceUid: '-100'
model:
conditions:
- evaluator:
params:
- 3
type: gt
operator:
type: and
query:
params:
- A
reducer:
type: last
type: query
datasource:
type: __expr__
uid: '-100'
expression: 1==0
intervalMs: 1000
maxDataPoints: 43200
refId: A
type: math
# <string> UID of a dashboard that the alert rule should be linked to
dashboardUid: my_dashboard
# <int> ID of the panel that the alert rule should be linked to
panelId: 123
# <string> the state the alert rule will have when no data is returned
# possible values: "NoData", "Alerting", "OK", default = NoData
noDataState: Alerting
# <string> the state the alert rule will have when the query execution
# failed - possible values: "Error", "Alerting", "OK"
# default = Alerting
# <duration, required> for how long should the alert fire before alerting
for: 60s
# <map<string, string>> a map of strings to pass around any data
annotations:
some_key: some_value
# <map<string, string> a map of strings that can be used to filter and
# route alerts
labels:
team: sre_team_1
```
Here is an example of a configuration file for deleting alert rules.
```yaml
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteRules:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the rule
uid: my_id_1
```
### Provision contact points
Create or delete contact points in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating contact points.
```yaml
# config file version
apiVersion: 1
# List of contact points to import or update
contactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the contact point
name: cp_1
receivers:
# <string, required> unique identifier for the receiver
- uid: first_uid
# <string, required> type of the receiver
type: prometheus-alertmanager
# <object, required> settings for the specific receiver type
settings:
url: http://test:9000
```
Here is an example of a configuration file for deleting contact points.
```yaml
# config file version
apiVersion: 1
# List of receivers that should be deleted
deleteContactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the receiver
uid: first_uid
```
#### Settings
Here are some examples of settings you can use for the different
contact point types.
##### Alertmanager
```yaml
type: prometheus-alertmanager
settings:
# <string, required>
url: http://localhost:9093
# <string>
basicAuthUser: abc
# <string>
basicAuthPassword: abc123
```
##### DingDing
```yaml
type: dingding
settings:
# <string, required>
url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx
# <string> options: link, actionCard
msgType: link
# <string>
message: |
{{ template "default.message" . }}
```
##### Discord
```yaml
type: discord
settings:
# <string, required>
url: https://discord/webhook
# <string>
avatar_url: https://my_avatar
# <string>
use_discord_username: Grafana
# <string>
message: |
{{ template "default.message" . }}
```
##### E-Mail
```yaml
type: email
settings:
# <string, required>
addresses: me@example.com;you@example.com
# <bool>
singleEmail: false
# <string>
message: my optional message to include
# <string>
subject: |
{{ template "default.title" . }}
```
##### Google Hangouts Chat
```yaml
type: googlechat
settings:
# <string, required>
url: https://google/webhook
# <string>
message: |
{{ template "default.message" . }}
```
##### Kafka
```yaml
type: kafka
settings:
# <string, required>
kafkaRestProxy: http://localhost:8082
# <string, required>
kafkaTopic: topic1
```
##### LINE
```yaml
type: line
settings:
# <string, required>
token: xxx
```
##### Microsoft Teams
```yaml
type: teams
settings:
# <string, required>
url: https://ms_teams_url
# <string>
title: |
{{ template "default.title" . }}
# <string>
sectiontitle: ''
# <string>
message: |
{{ template "default.message" . }}
```
##### OpsGenie
```yaml
type: opsgenie
settings:
# <string, required>
apiKey: xxx
# <string, required>
apiUrl: https://api.opsgenie.com/v2/alerts
# <string>
message: |
{{ template "default.title" . }}
# <string>
description: some descriptive description
# <bool>
autoClose: false
# <bool>
overridePriority: false
# <string> options: tags, details, both
sendTagsAs: both
```
##### PagerDuty
```yaml
type: pagerduty
settings:
# <string, required>
integrationKey: XXX
# <string> options: critical, error, warning, info
severity: critical
# <string>
class: ping failure
# <string>
component: Grafana
# <string>
group: app-stack
# <string>
summary: |
{{ template "default.message" . }}
```
##### Pushover
```yaml
type: pushover
settings:
# <string, required>
apiToken: XXX
# <string, required>
userKey: user1,user2
# <string>
device: device1,device2
# <string> options (high to low): 2,1,0,-1,-2
priority: '2'
# <string>
retry: '30'
# <string>
expire: '120'
# <string>
sound: siren
# <string>
okSound: magic
# <string>
message: |
{{ template "default.message" . }}
```
##### Slack
```yaml
type: slack
settings:
# <string, required>
recipient: alerting-dev
# <string, required>
token: xxx
# <string>
username: grafana_bot
# <string>
icon_emoji: heart
# <string>
icon_url: https://icon_url
# <string>
mentionUsers: user_1,user_2
# <string>
mentionGroups: group_1,group_2
# <string> options: here, channel
mentionChannel: here
# <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
url: https://some_webhook_url
# <string>
endpointUrl: https://custom_url/api/chat.postMessage
# <string>
title: |
{{ template "slack.default.title" . }}
text: |
{{ template "slack.default.text" . }}
```
##### Sensu Go
```yaml
type: sensugo
settings:
# <string, required>
url: http://sensu-api.local:8080
# <string, required>
apikey: xxx
# <string>
entity: default
# <string>
check: default
# <string>
handler: some_handler
# <string>
namespace: default
# <string>
message: |
{{ template "default.message" . }}
```
##### Telegram
```yaml
type: telegram
settings:
# <string, required>
bottoken: xxx
# <string, required>
chatid: some_chat_id
# <string>
message: |
{{ template "default.message" . }}
```
##### Threema Gateway
```yaml
type: threema
settings:
# <string, required>
api_secret: xxx
# <string, required>
gateway_id: A5K94S9
# <string, required>
recipient_id: A9R4KL4S
```
##### VictorOps
```yaml
type: victorops
settings:
# <string, required>
url: XXX
# <string> options: CRITICAL, WARNING
messageType: CRITICAL
```
##### Webhook
```yaml
type: webhook
settings:
# <string, required>
url: https://endpoint_url
# <string> options: POST, PUT
httpMethod: POST
# <string>
username: abc
# <string>
password: abc123
# <string>
authorization_scheme: Bearer
# <string>
authorization_credentials: abc123
# <string>
maxAlerts: '10'
```
##### WeCom
```yaml
type: wecom
settings:
# <string, required>
url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx
# <string>
message: |
{{ template "default.message" . }}
# <string>
title: |
{{ template "default.title" . }}
```
### Provision notification policies
Create or reset notification policies in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating notification policiies.
```yaml
# config file version
apiVersion: 1
# List of notification policies
policies:
# <int> organization ID, default = 1
- orgId: 1
# <string> name of the contact point that should be used for this route
receiver: grafana-default-email
# <list> The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
#
# To aggregate by all possible labels use the special value '...' as
# the sole label name, for example:
# group_by: ['...']
# This effectively disables aggregation entirely, passing through all
# alerts as-is. This is unlikely to be what you want, unless you have
# a very low alert volume or your upstream notification system performs
# its own grouping.
group_by: ['...']
# <list> a list of matchers that an alert has to fulfill to match the node
matchers:
- alertname = Watchdog
- severity =~ "warning|critical"
# <list> Times when the route should be muted. These must match the name of a
# mute time interval.
# Additionally, the root node cannot have any mute times.
# When a route is muted it will not send any notifications, but
# otherwise acts normally (including ending the route-matching process
# if the `continue` option is not set)
mute_time_intervals:
- abc
# <duration> How long to initially wait to send a notification for a group
# of alerts. Allows to collect more initial alerts for the same group.
# (Usually ~0s to few minutes), default = 30s
group_wait: 30s
# <duration> How long to wait before sending a notification about new alerts that
# are added to a group of alerts for which an initial notification has
# already been sent. (Usually ~5m or more), default = 5m
group_internval: 5m
# <duration> How long to wait before sending a notification again if it has already
# been sent successfully for an alert. (Usually ~3h or more), default = 4h
repeat_interval: 4h
# <list> Zero or more child routes
# routes:
# ...
```
Here is an example of a configuration file for resetting notification policies.
```yaml
# config file version
apiVersion: 1
# List of orgIds that should be reset to the default policy
resetPolicies:
- 1
```
### Provision templates
Create or delete templates in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating templates.
```yaml
# config file version
apiVersion: 1
# List of templates to import or update
templates:
# <int> organization ID, default = 1
- orgID: 1
# <string, required> name of the template, must be unique
name: my_first_template
# <string, required> content of the the template
template: Alerting with a custom text template
```
Here is an example of a configuration file for deleting templates.
```yaml
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteTemplates:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the template, must be unique
name: my_first_template
```
### Provision mute timings
Create or delete mute timings in your Grafana instance(s).
1. Create a YAML or JSON configuration file.
Example configuration files can be found below.
1. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating mute timings.
```yaml
# config file version
apiVersion: 1
# List of mute time intervals to import or update
muteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
# <list> time intervals that should trigger the muting
# refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0
time_intervals:
- times:
- start_time: '06:00'
end_time: '23:59'
weekdays: ['monday:wednesday', 'saturday', 'sunday']
months: ['1:3', 'may:august', 'december']
years: ['2020:2022', '2030']
days_of_month: ['1:5', '-3:-1']
```
Here is an example of a configuration file for deleting mute timings.
```yaml
# config file version
apiVersion: 1
# List of mute time intervals that should be deleted
deleteMuteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
```
### File provisioning using Kubernetes
If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps.
1. Create one or more configuration maps as follows.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerting
data:
provisioning.yaml: |
templates:
- name: my_first_template
template: the content for my template
```
2. Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- name: grafana
containerPort: 3000
volumeMounts:
- mountPath: /etc/grafana/provisioning/alerting
name: grafana-alerting
readOnly: false
volumes:
- name: grafana-alerting
configMap:
defaultMode: 420
name: grafana-alerting
```
This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment.

316
docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md Normal file
View File

@ -0,0 +1,316 @@
---
aliases:
- /docs/grafana/latest/alerting/provision-alerting-resources/terraform-provisioning
description: Create and manage alerting resources using Terraform
keywords:
- grafana
- alerting
- alerting resources
- provisioning
- Terraform
title: Create and manage alerting resources using Terraform
weight: 200
---
# Create and manage alerting resources using Terraform
Use Terraform’s Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code.
For more information on managing your alerting resources using Terraform, refer to the [Grafana Provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) documentation.
Complete the following tasks to create and manage your alerting resources using Terraform.
1. Create an API key for provisioning.
1. Configure the Terraform provider.
1. Provision your alerting resources.
## Before you begin
- Ensure you have the grafana/grafana [Terraform provider](https://registry.terraform.io/providers/grafana/grafana/1.28.0) 1.27.0 or higher.
- Ensure you are using Grafana 9.1 or higher.
## Create an API key for provisioning
You can [create a normal Grafana API key](https://grafana.com/docs/grafana/latest/administration/api-keys/) to authenticate Terraform with Grafana. Most existing tooling using API keys should automatically work with the new Grafana Alerting support.
There are also dedicated RBAC roles for alerting provisioning. This lets you easily authenticate as a [service account](https://grafana.com/docs/grafana/latest/administration/service-accounts/) with the minimum permissions needed to provision your Alerting infrastructure.
To create an API key for provisioning, complete the following steps.
1. Create a new service account for your CI pipeline.
1. Assign the role “Access the alert rules Provisioning API.”
1. Create a new service account token.
1. Name and save the token for use in Terraform.
Alternatively, you can use Terraform authentication: [basic auth](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).
## Configure the Terraform provider
Grafana Alerting support is included as part of the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
The following is an example you can use to configure the Terraform provider.
```terraform
terraform {
required_providers {
grafana = {
source = "grafana/grafana"
version = ">= 1.28.2"
}
}
}
provider "grafana" {
url = <YOUR_GRAFANA_URL>
auth = <YOUR_GRAFANA_API_KEY>
}
```
## Provision contact points and templates
Contact points connect an alerting stack to the outside world. They tell Grafana how to connect to your external systems and where to deliver notifications. There are over fifteen different integrations to choose from.
To provision contact points and templates, complete the following steps.
1. Copy this code block into a .tf file on your local machine.
This example uses a contact point that sends alert notifications to Slack.
```terraform
resource "grafana_contact_point" "my_slack_contact_point" {
name = "Send to My Slack Channel"
slack {
url = <YOUR_SLACK_WEBHOOK_URL>
text = <<EOT
{{ len .Alerts.Firing }} alerts are firing!
Alert summaries:
{{ range .Alerts.Firing }}
{{ template "Alert Instance Template" . }}
{{ end }}
EOT
}
}
```
1. Enter text for your notification in the text field.
The `text` field supports [Go-style templating](https://pkg.go.dev/text/template). This enables you to manage your Grafana Alerting message templates directly in Terraform.
1. Run the command ‘terraform apply’.
1. Go to the Grafana UI and check the details of your contact point.
**Note:**
You cannot edit resources provisioned from Terraform from the UI. This ensures that your alerting stack always stays in sync with your code.
1. Click **Test** to verify that the contact point works correctly.
**Note:**
You can re-use the same templates across many contact points. In the example above, a shared template ie embedded using the statement `{{ template “Alert Instance Template” . }}`
This fragment can then be managed separately in Terraform:
```terraform
resource "grafana_message_template" "my_alert_template" {
name = "Alert Instance Template"
template = <<EOT
{{ define "Alert Instance Template" }}
Firing: {{ .Labels.alertname }}
Silence: {{ .SilenceURL }}
{{ end }}
EOT
}
```
## Provision notification policies and routing
Notification policies tell Grafana how to route alert instances, as opposed to where. They connect firing alerts to your previously defined contact points using a system of labels and matchers.
To provision notification policies and routing, complete the following steps.
1. Copy this code block into a .tf file on your local machine.
In this example, the alerts are grouped by `alertname`, which means that any notifications coming from alerts which share the same name, are grouped into the same Slack message.
resource "grafana_notification_policy" "my_policy" {
group_by = ["alertname"]
contact_point = grafana_contact_point.my_slack_contact_point.name
group_wait = "45s"
group_interval = "6m"
repeat_interval = "3h"
policy {
matcher {
label = "a"
match = "="
value = "b"
}
group_by = ["..."]
contact_point = grafana_contact_point.a_different_contact_point.name
mute_timings = [grafana_mute_timing.my_mute_timing.name]
policy {
matcher {
label = "sublabel"
match = "="
value = "subvalue"
}
contact_point = grafana_contact_point.a_third_contact_point.name
group_by = ["..."]
}
}
}
1. In the mute_timings field, link a mute timing to your notification policy.
1. Run the command ‘terraform apply’.
1. Go to the Grafana UI and check the details of your notification policy.
**Note:**
You cannot edit resources provisioned from Terraform from the UI. This ensures that your alerting stack always stays in sync with your code.
1. Click **Test** to verify that the notification point is working correctly.
## Provision mute timings
Mute timings provide the ability to mute alert notifications for defined time periods.
To provision mute timings, complete the following steps.
1. Copy this code block into a .tf file on your local machine.
In this example, alert notifications are muted on weekends.
resource "grafana_mute_timing" "my_mute_timing" {
name = "My Mute Timing"
intervals {
times {
start = "04:56"
end = "14:17"
}
weekdays = ["saturday", "sunday", "tuesday:thursday"]
months = ["january:march", "12"]
years = ["2025:2027"]
}
}
1. Run the command ‘terraform apply’.
1. Go to the Grafana UI and check the details of your mute timing.
**Note:**
You cannot edit resources provisioned from Terraform from the UI. This ensures that your alerting stack always stays in sync with your code.
1. Click **Test** to verify that the mute timing is working correctly.
## Provision alert rules
[Alert rules](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/) enable you to alert against any Grafana data source. This can be a data source that you already have configured, or you can [define your data sources in Terraform](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source) alongside your alert rules.
To provision alert rules, complete the following steps.
1. Create a data source to query and a folder to store your rules in.
In this example, the [TestData](https://grafana.com/docs/grafana/latest/datasources/testdata/) data source is used.
```terraform
resource "grafana_data_source" "testdata_datasource" {
name = "TestData"
type = "testdata"
}
resource "grafana_folder" "rule_folder" {
title = "My Rule Folder"
}
```
1. Define an alert rule.
For more information on alert rules, refer to [how to create Grafana-managed alerts](https://grafana.com/blog/2022/08/01/grafana-alerting-video-how-to-create-alerts-in-grafana-9/).
1. Group your alert rules.
In this example, the `grafana_rule_group` resource group is used.
```terraform
resource "grafana_rule_group" "my_rule_group" {
name = "My Alert Rules"
folder_uid = grafana_folder.rule_folder.uid
interval_seconds = 60
org_id = 1
rule {
name = "My Random Walk Alert"
condition = "C"
for = "0s"
// Query the datasource.
data {
ref_id = "A"
relative_time_range {
from = 600
to = 0
}
datasource_uid = grafana_data_source.testdata_datasource.uid
// `model` is a JSON blob that sends datasource-specific data.
// It's different for every datasource. The alert's query is defined here.
model = jsonencode({
intervalMs = 1000
maxDataPoints = 43200
refId = "A"
})
}
// The query was configured to obtain data from the last 60 seconds. Let's alert on the average value of that series using a Reduce stage.
data {
datasource_uid = "-100"
// You can also create a rule in the UI, then GET that rule to obtain the JSON.
// This can be helpful when using more complex reduce expressions.
model = <<EOT
{"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"last"},"type":"avg"}],"datasource":{"name":"Expression","type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"last","refId":"B","type":"reduce"}
EOT
ref_id = "B"
relative_time_range {
from = 0
to = 0
}
}
// Now, let's use a math expression as our threshold.
// We want to alert when the value of stage "B" above exceeds 70.
data {
datasource_uid = "-100"
ref_id = "C"
relative_time_range {
from = 0
to = 0
}
model = jsonencode({
expression = "$B > 70"
type = "math"
refId = "C"
})
}
}
}
```
1. Go to the Grafana UI and check your alert rule.
You can see whether or not the alert rule is firing. You can also see a visualization of each of the alert rule’s query stages
When the alert fires, Grafana routes a notification through the policy you defined.
For example, if you chose Slack as a contact point, Grafana’s embedded [Alertmanager](https://github.com/prometheus/alertmanager) automatically posts a message to Slack.

28
docs/sources/alerting/set-up/provision-alerting-resources/view-provisioned-resources/index.md Normal file
View File

@ -0,0 +1,28 @@
---
aliases:
- /docs/grafana/latest/alerting/provision-alerting-resources/view-provisioned-resources
description: View provisioned resources in Grafana
keywords:
- grafana
- alerting
- alerting resources
- provisioning
title: View provisioned resources in Grafana
weight: 300
---
# View provisioned alerting resources in Grafana
Verify that your alerting resources were created in Grafana.
To view your provisioned resources in Grafana, complete the following steps.
1. Open your Grafana instance.
1. Navigate to Alerting.
1. Click an alerting resource folder, for example, Alert rules.
Provisioned resources are labeled **Provisioned**, so that it is clear that they were not created manually.
**Note:**
You cannot edit provisioned resources from Grafana. You can only change the resource properties by changing the provisioning file and restarting Grafana or carrying out a hot reload. This prevents changes being made to the resource that would be overwritten if a file is provisioned again or a hot reload is carried out.