Alerting docs: update the Terraform Provision guide (#83773)

* Alerting docs: update the Terraform Provision guide

* Fix incorrect links

* Update docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/provision-alerting-resources/terraform-provisioning/index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

---------

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
This commit is contained in:
Pepe Cano 2024-03-04 12:10:37 +01:00 committed by GitHub
parent 07e26226b7
commit 111df1bba0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -23,267 +23,103 @@ weight: 200
Use Terraforms Grafana Provider to manage your alerting resources and provision them into your Grafana system. Terraform provider support for Grafana Alerting makes it easy to create, manage, and maintain your entire Grafana Alerting stack as code.
Refer to [Grafana Terraform Provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) documentation for more examples and information on Terraform Alerting schemas.
This guide outlines the steps and references to provision alerting resources with Terraform. For a practical demo, you can clone and try this [example using Grafana OSS and Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/terraform).
Complete the following tasks to create and manage your alerting resources using Terraform.
1. Create an API key for provisioning.
1. Configure the Terraform provider.
1. Define your alerting resources in Terraform. [Export alerting resources][alerting_export] in Terraform format, or implement the [Terraform Alerting schemas](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
To create and manage your alerting resources using Terraform, you have to complete the following tasks.
1. Create an API key to configure the Terraform provider.
1. Create your alerting resources in Terraform format by
- [exporting configured alerting resources][alerting_export]
- or writing the [Terraform Alerting schemas](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
> By default, you cannot edit provisioned resources. Enable [`disable_provenance` in the Terraform resource](#enable-editing-resources-in-the-grafana-ui) to allow changes in the Grafana UI.
1. Run `terraform apply` to provision your alerting resources.
{{< admonition type="note" >}}
Before you begin, you should have available a Grafana instance and [Terraform installed](https://www.terraform.io/downloads) on your machine.
- By default, you cannot edit resources provisioned from Terraform from the UI. This ensures that your alerting stack always stays in sync with your code. To change the default behaviour, refer to [Edit provisioned resources in the Grafana UI](#edit-provisioned-resources-in-the-grafana-ui).
## Create an API key and configure the Terraform provider
- Before you begin, ensure you have the [Grafana Terraform Provider](https://registry.terraform.io/providers/grafana/grafana/) 1.27.0 or higher, and are using Grafana 9.1 or higher.
{{< /admonition >}}
## Create an API key for provisioning
You can create a [service account token][service-accounts] to authenticate Terraform with Grafana. Most existing tooling using API keys should automatically work with the new Grafana Alerting support.
There are also dedicated RBAC roles for alerting provisioning. This lets you easily authenticate as a service account with the minimum permissions needed to provision your Alerting infrastructure.
To create an API key for provisioning, complete the following steps.
You can create a [service account token][service-accounts] to authenticate Terraform with Grafana. To create an API key for provisioning alerting resources, complete the following steps.
1. Create a new service account.
1. Assign the role or permission to access the [Alerting provisioning API][alerting_http_provisioning].
1. Create a new service account token.
1. Name and save the token for use in Terraform.
Alternatively, you can use basic authentication. To view all the supported authentication formats, see [here](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).
You can now move to the working directory for your Terraform configurations, and create a file named `main.tf` like:
## Configure the Terraform provider
Grafana Alerting support is included as part of the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
The following is an example you can use to configure the Terraform provider.
```HCL
```main.tf
terraform {
required_providers {
grafana = {
source = "grafana/grafana"
version = ">= 1.28.2"
version = ">= 2.9.0"
}
}
}
provider "grafana" {
url = <YOUR_GRAFANA_URL>
auth = <YOUR_GRAFANA_API_KEY>
url = <grafana-url>
auth = <api-key>
}
```
## Import contact points and templates
Replace the following values:
Contact points connect an alerting stack to the outside world. They tell Grafana how to connect to your external systems and where to deliver notifications.
- `<grafana-url>` with the URL of the Grafana instance.
- `<api-key>` with the API token previously created.
To provision contact points and templates, refer to the [grafana_contact_point schema](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/contact_point) and [grafana_message_template schema](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/message_template), and complete the following steps.
This Terraform configuration installs the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) and authenticates against your Grafana instance using an API token. For other authentication alternatives including basic authentication, refer to the [`auth` option documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication).
1. Copy this code block into a `.tf` file on your local machine.
For Grafana Cloud, refer to the [instructions to manage a Grafana Cloud stack with Terraform][provision-cloud-with-terraform]. For role-based access control, refer to [Provisioning RBAC with Terraform][rbac-terraform-provisioning] and the [alerting provisioning roles (`fixed:alerting.provisioning.*`)][rbac-role-definitions].
This example creates a contact point that sends alert notifications to Slack.
## Create Terraform configurations for alerting resources
```HCL
resource "grafana_contact_point" "my_slack_contact_point" {
name = "Send to My Slack Channel"
[Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) enables you to manage the following alerting resources.
slack {
url = <YOUR_SLACK_WEBHOOK_URL>
text = <<EOT
{{ len .Alerts.Firing }} alerts are firing!
| Alerting resource | Terraform resource |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| [Alert rules][alerting-rules] | [grafana_rule_group](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/rule_group) |
| [Contact points][contact-points] | [grafana_contact_point](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/contact_point) |
| [Notification templates][notification-template] | [grafana_message_template](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/message_template) |
| [Notification policy tree][notification-policy] | [grafana_notification_policy](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/notification_policy) |
| [Mute timings][mute-timings] | [grafana_mute_timing](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing) |
Alert summaries:
{{ range .Alerts.Firing }}
{{ template "Alert Instance Template" . }}
{{ end }}
EOT
}
}
```
In this section, we'll create Terraform configurations for each alerting resource and demonstrate how to link them together.
You can create multiple external integrations in a single contact point. Notifications routed to this contact point will be sent to all integrations. This example shows multiple integrations in the same Terraform resource.
### Add alert rules
```
resource "grafana_contact_point" "my_multi_contact_point" {
name = "Send to Many Places"
[Alert rules][alerting-rules] enable you to receive alerts by querying any backend Grafana data sources.
slack {
url = "webhook1"
...
}
slack {
url = "webhook2"
...
}
teams {
...
}
email {
...
}
}
```
1. Enter text for your notification in the text field.
The `text` field supports [Go-style templating](https://pkg.go.dev/text/template). This enables you to manage your Grafana Alerting notification templates directly in Terraform.
1. Run the command `terraform apply`.
1. Go to the Grafana UI and check the details of your contact point.
1. Click **Test** to verify that the contact point works correctly.
### Reuse templates
You can reuse the same templates across many contact points. In the example above, a shared template ie embedded using the statement `{{ template “Alert Instance Template” . }}`
This fragment can then be managed separately in Terraform:
```HCL
resource "grafana_message_template" "my_alert_template" {
name = "Alert Instance Template"
template = <<EOT
{{ define "Alert Instance Template" }}
Firing: {{ .Labels.alertname }}
Silence: {{ .SilenceURL }}
{{ end }}
EOT
}
```
## Import notification policies and routing
Notification policies tell Grafana how to route alert instances to your contact points. They connect firing alerts to your previously defined contact points using a system of labels and matchers.
To provision notification policies and routing, refer to the [grafana_notification_policy schema](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/notification_policy), and complete the following steps.
{{% admonition type="warning" %}}
Since the policy tree is a single resource, provisioning the `grafana_notification_policy` resource will overwrite a policy tree created through any other means.
{{< /admonition >}}
1. Copy this code block into a `.tf` file on your local machine.
In this example, the alerts are grouped by `alertname`, which means that any notifications coming from alerts which share the same name, are grouped into the same Slack message. You can provide any set of label keys here, or you can use the special label `"..."` to route by all label keys, sending each alert in a separate notification.
If you want to route specific notifications differently, you can add sub-policies. Sub-policies allow you to apply routing to different alerts based on label matching. In this example, we apply a mute timing to all alerts with the label a=b.
```HCL
resource "grafana_notification_policy" "my_policy" {
group_by = ["alertname"]
contact_point = grafana_contact_point.my_slack_contact_point.name
group_wait = "45s"
group_interval = "6m"
repeat_interval = "3h"
policy {
matcher {
label = "a"
match = "="
value = "b"
}
group_by = ["..."]
contact_point = grafana_contact_point.a_different_contact_point.name
mute_timings = [grafana_mute_timing.my_mute_timing.name]
policy {
matcher {
label = "sublabel"
match = "="
value = "subvalue"
}
contact_point = grafana_contact_point.a_third_contact_point.name
group_by = ["..."]
}
}
}
```
1. In the mute_timings field, link a mute timing to your notification policy.
1. Run the command `terraform apply`.
1. Go to the Grafana UI and check the details of your notification policy.
1. Click **Test** to verify that the notification point is working correctly.
## Import mute timings
Mute timings provide the ability to mute alert notifications for defined time periods.
To provision mute timings, refer to the [grafana_mute_timing schema](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing), and complete the following steps.
1. Copy this code block into a `.tf` file on your local machine.
In this example, alert notifications are muted on weekends.
```HCL
resource "grafana_mute_timing" "my_mute_timing" {
name = "My Mute Timing"
intervals {
times {
start = "04:56"
end = "14:17"
}
weekdays = ["saturday", "sunday", "tuesday:thursday"]
months = ["january:march", "12"]
years = ["2025:2027"]
}
}
```
1. Run the command `terraform apply`.
1. Go to the Grafana UI and check the details of your mute timing.
1. Reference your newly created mute timing in a notification policy using the `mute_timings` field.
This will apply your mute timing to some or all of your notifications.
1. Click **Test** to verify that the mute timing is working correctly.
## Import alert rules
[Alert rules][alerting-rules] enable you to alert against any Grafana data source. This can be a data source that you already have configured, or you can [define your data sources in Terraform](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source) alongside your alert rules.
To provision alert rules, refer to the [grafana_rule_group schema](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/rule_group), and complete the following steps.
1. Create a data source to query and a folder to store your rules in.
1. First, create a data source to query and a folder to store your rules in.
In this example, the [TestData][testdata] data source is used.
Alerts can be defined against any backend datasource in Grafana.
```HCL
resource "grafana_data_source" "testdata_datasource" {
```terraform
resource "grafana_data_source" "<terraform_data_source_name>" {
name = "TestData"
type = "testdata"
}
resource "grafana_folder" "rule_folder" {
resource "grafana_folder" "<terraform_folder_name>" {
title = "My Rule Folder"
}
```
1. Define an alert rule.
Replace the following field values:
For more information on alert rules, refer to [how to create Grafana-managed alerts](/blog/2022/08/01/grafana-alerting-video-how-to-create-alerts-in-grafana-9/).
- `<terraform_data_source_name>` with the terraform name of the data source.
- `<terraform_folder_name>` with the terraform name of the folder.
1. Create a rule group containing one or more rules.
1. Create or find an alert rule you want to import in Grafana.
In this example, the `grafana_rule_group` resource group is used.
1. [Export][alerting_export] the alert rule group in Terraform format. This exports the alert rule group as [`grafana_rule_group` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/rule_group).
```HCL
resource "grafana_rule_group" "my_rule_group" {
You can edit the exported resource, or alternatively, consider creating the resource from scratch.
```terraform
resource "grafana_rule_group" "<terraform_rule_group_name>" {
name = "My Alert Rules"
folder_uid = grafana_folder.rule_folder.uid
folder_uid = grafana_folder.<terraform_folder_name>.uid
interval_seconds = 60
org_id = 1
@ -299,7 +135,7 @@ To provision alert rules, refer to the [grafana_rule_group schema](https://regis
from = 600
to = 0
}
datasource_uid = grafana_data_source.testdata_datasource.uid
datasource_uid = grafana_data_source.<terraform_data_source_name>.uid
// `model` is a JSON blob that sends datasource-specific data.
// It's different for every datasource. The alert's query is defined here.
model = jsonencode({
@ -343,42 +179,232 @@ To provision alert rules, refer to the [grafana_rule_group schema](https://regis
}
```
1. Run the command `terraform apply`.
1. Go to the Grafana UI and check your alert rule.
Replace the following field values:
You can see whether or not the alert rule is firing. You can also see a visualization of each of the alert rules query stages
- `<terraform_rule_group_name>` with the name of the alert rule group.
When the alert fires, Grafana routes a notification through the policy you defined.
Note that the distinct Grafana resources are connected through `uid` values in their Terraform configurations. The `uid` value will be randomly generated when provisioning.
For example, if you chose Slack as a contact point, Grafanas embedded [Alertmanager](https://github.com/prometheus/alertmanager) automatically posts a message to Slack.
To link the alert rule group with its respective data source and folder in this example, replace the following field values:
## Edit provisioned resources in the Grafana UI
- `<terraform_data_source_name>` with the terraform name of the previously defined data source.
- `<terraform_folder_name>` with the terraform name of the the previously defined folder.
By default, you cannot edit resources provisioned via Terraform in Grafana. To enable editing these resources in the Grafana UI, use the `disable_provenance` attribute on alerting resources:
1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).
```HCL
provider "grafana" {
url = "http://grafana.example.com/"
auth = var.grafana_auth
}
### Add contact points
[Contact points][contact-points] are the receivers of alert notifications.
1. Create or find the contact points you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
1. [Export][alerting_export] the contact point in Terraform format. This exports the contact point as [`grafana_contact_point` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/contact_point)—edit it if necessary.
1. In this example, notifications are muted on weekends.
```terraform
resource "grafana_contact_point" "<terraform_contact_point_name>" {
name = "My contact point email"
email {
addresses = ["<email_address>"]
}
}
```
Replace the following field values:
- `<terraform_contact_point_name>` with the terraform name of the contact point. It will be used to reference the contact point in other Terraform resources.
- `<email_address>` with the email to receive alert notifications.
1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).
### Add and enable templates
[Notification templates][notification-template] allow customization of alert notifications across multiple contact points.
1. Create or find the notification template you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
1. [Export][alerting_export] the template as [`grafana_message_template` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/message_template).
This example is a simple demo template defined as `custom_email.message`.
```terraform
resource "grafana_message_template" "<terraform_message_template_name>" {
name = "custom_email.message"
template = <<EOT
{{ define "custom_email.message" }}
Lorem ipsum - Custom alert!
{{ end }}
EOT
}
```
1. In the previous contact point, enable the template by setting the `email.message` property as follows.
```terraform
resource "grafana_contact_point" "<terraform_contact_point_name>" {
name = "My contact point email"
email {
addresses = ["<email_address>"]
message = "{{ template \"custom_email.message\" .}}"
}
}
```
1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).
### Add mute timings
[Mute timings][mute-timings] pause alert notifications during predetermined intervals.
1. Create or find the mute timings you want to import in Grafana. Alternatively, consider writing the resource in code as demonstrated in the example below.
1. [Export][alerting_export] the mute timing in Terraform format. This exports the mute timing as [`grafana_mute_timing` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing)—edit it if necessary.
1. This example turns off notifications on weekends.
```terraform
resource "grafana_mute_timing" "<terraform_mute_timing_name>" {
name = "No weekends"
intervals {
weekdays = ["saturday", "sunday"]
}
}
```
Replace the following field values:
- `<terraform_mute_timing_name>` with the name of the Terraform resource. It will be used to reference the mute timing in the Terraform notification policy tree.
1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).
### Add the notification policy tree
[Notification policies][notification-policy] defines how to route alert instances to your contact points.
{{% admonition type="warning" %}}
Since the policy tree is a single resource, provisioning the `grafana_notification_policy` resource will overwrite a policy tree created through any other means.
{{< /admonition >}}
1. Find the default notification policy tree. Alternatively, consider writing the resource in code as demonstrated in the example below.
1. [Export][alerting_export] the notification policy tree in Terraform format. This exports it as [`grafana_notification_policy` Terraform resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/notification_policy)—edit it if necessary.
```terraform
resource "grafana_notification_policy" "my_policy_tree" {
contact_point = grafana_contact_point.<terraform_contact_point_name>.name
...
policy {
contact_point = grafana_contact_point.<terraform_contact_point_name>.name
matcher {...}
mute_timings = [grafana_mute_timing.<terraform_mute_timing_name>.name]
}
}
```
To configure the mute timing and contact point previously created in the notification policy tree, replace the following field values:
- `<terraform_data_source_name>` with the terraform name of the previously defined contact point.
- `<terraform_folder_name>` with the terraform name of the the previously defined mute timing.
1. Continue to add more Grafana resources or [use the Terraform CLI for provisioning](#provision-grafana-resources-with-terraform).
### Enable editing resources in the Grafana UI
By default, you cannot edit resources provisioned via Terraform in Grafana. This ensures that your alerting stack always stays in sync with your Terraform code.
To make provisioned resources editable in the Grafana UI, enable the `disable_provenance` attribute on alerting resources.
```terraform
resource "grafana_contact_point" "my_contact_point" {
name = "My Contact Point"
resource "grafana_mute_timing" "mute_all" {
name = "mute all"
disable_provenance = true
intervals {}
}
resource "grafana_message_template" "my_template" {
name = "My Reusable Template"
template = "{{define \"My Reusable Template\" }}\n template content\n{{ end }}"
disable_provenance = true
}
...
```
Note that `disable_provenance` is not supported for [grafana_mute_timing](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/mute_timing).
## Provision Grafana resources with Terraform
To create the previous alerting resources in Grafana with the Terraform CLI, complete the following steps.
1. Initialize the working directory containing the Terraform configuration files.
```shell
terraform init
```
This command initializes the Terraform directory, installing the Grafana Terraform provider configured in the `main.tf` file.
1. Apply the Terraform configuration files to provision the resources.
```shell
terraform apply
```
Before applying any changes to Grafana, Terraform displays the execution plan and requests your approval.
```shell
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
```
Once you have confirmed to proceed with the changes, Terraform will create the provisioned resources in Grafana!
```shell
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
```
You can now access Grafana to verify the creation of the distinct resources.
## More examples
- [Grafana Terraform Provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs)
- [Creating and managing a Grafana Cloud stack using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/terraform-cloud-stack)
For more examples on the concept of this guide:
- Try the demo [provisioning alerting resources in Grafana OSS using Terraform and Docker Compose](https://github.com/grafana/provisioning-alerting-examples/tree/main/terraform).
- Review all the available options and examples of the Terraform Alerting schemas in the [Grafana Terraform Provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
- Review the [tutorial to manage a Grafana Cloud stack using Terraform][provision-cloud-with-terraform].
{{% docs/reference %}}
[alerting-rules]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules"
[alerting-rules]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules"
[contact-points]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points"
[contact-points]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points"
[mute-timings]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings"
[mute-timings]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings"
[notification-policy]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy"
[notification-policy]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-notification-policy"
[notification-template]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications"
[notification-template]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications"
[alerting_export]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/provision-alerting-resources/export-alerting-resources"
[alerting_export]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/set-up/provision-alerting-resources/export-alerting-resources"
@ -389,4 +415,11 @@ resource "grafana_mute_timing" "mute_all" {
[testdata]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/datasources/testdata"
[testdata]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/connect-externally-hosted/data-sources/testdata"
[provision-cloud-with-terraform]: "/docs/ -> /docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/terraform-cloud-stack"
[rbac-role-definitions]: "/docs/ -> /docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-fixed-basic-role-definitions"
[rbac-terraform-provisioning]: "/docs/ -> /docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-terraform-provisioning"
{{% /docs/reference %}}