mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Alerting docs: update Configure notifications (#97862)
* Update `Configure notifications` * wip notification policies * Finish Create Notification Policy * Minor update to Contact point intro * Fix `How label matching works` links and remove links from contact point intro * Silences and Mute timings * Reorder + MuteTimings + Silences * Complete Create silence * fix mispell * Update docs/sources/alerting/configure-notifications/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/configure-notifications/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/configure-notifications/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/configure-notifications/_index.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * fix prettier error --------- Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
This commit is contained in:
Pepe Cano
GitHub
parent
ed6f41cd2c
commit
b8c4126fbc
@ -14,29 +14,75 @@ menuTitle: Configure notifications
|
||||
title: Configure notifications
|
||||
weight: 125
|
||||
refs:
|
||||
contact-points:
|
||||
intro-notifications:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/
|
||||
notification-policies:
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/
|
||||
configure-contact-points:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
templates-page:
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/
|
||||
configure-notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/templates/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/templates/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-notification-policy/
|
||||
configure-templates:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/
|
||||
configure-silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
configure-mute-timings:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings/
|
||||
---
|
||||
|
||||
# Configure notifications
|
||||
|
||||
Choose how, when, and where to send your alert notifications.
|
||||
Configuring how, when, and where to send alert notifications is an essential part of your alerting system.
|
||||
|
||||
As a first step, define your [contact points](ref:contact-points) where to send your alert notifications to. A contact point is a set of one or more integrations that are used to deliver notifications.
|
||||
By default, Grafana Alerting provides default notification messages with relevant alert information, so you don't need to configure messages initially. In the alert rule, you need to configure how to forward alerts:
|
||||
|
||||
Next, create a [notification policy](ref:notification-policies) which is a set of rules for where, when and how your alerts are routed to contact points. In a notification policy, you define where to send your alert notifications by choosing one of the contact points you created.
|
||||
1. Directly to a contact point.
|
||||
2. To a contact point via notification policies (more flexible).
|
||||
|
||||
Optionally, you can add [notification templates](ref:templates-page) to contact points for reuse and consistent messaging in your notifications.
|
||||
{{< figure src="/media/docs/alerting/alerting-configure-notifications-v2.png" max-width="750px" alt="In the alert rule, you can configure alert forwarding directly to a contact point or through notification policies" >}}
|
||||
|
||||
Notification setup is essential for an effective alerting system to scale across multiple teams and services. For a quick overview about the various components involved in handling notifications, refer to the [introduction about notifications](ref:intro-notifications).
|
||||
|
||||
The topics in this section include step-by-step instructions for:
|
||||
|
||||
- [Configuring contact points](ref:configure-contact-points) to specify where to receive alert notifications.
|
||||
- [Configuring notification policies](ref:configure-notification-policies) to determine how alerts are routed to contact points.
|
||||
- [Templating notifications](ref:configure-templates) to customize notification messages.
|
||||
- [Configuring silences](ref:configure-silences) or [mute timings](ref:configure-mute-timings) to stop notifications.
|
||||
|
||||
## Alertmanager architecture
|
||||
|
||||
Grafana Alerting is based on the Prometheus Alerting model, whose architecture decouples rule evaluation from notification handling.
|
||||
|
||||
- The alert rule evaluator, either Grafana or the data source, evaluates alert rules and triggers alerts.
|
||||
- The alert notification manager, known as the **Alertmanager**, receives alerts and manages their notifications.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-alertmanager-architecture.png" max-width="750px" alt="A diagram with the alert generator and alert manager architecture" >}}
|
||||
|
||||
In Grafana, you can use different types of alert rules and configure multiple Alertmanagers.
|
||||
|
||||
By default, Grafana uses its built-in Alertmanager, and Grafana Cloud instances include an additional Alertmanager.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-choose-alertmanager.png" max-width="750px" alt="A screenshot choosing an Alertmanager in the notification policies UI" >}}
|
||||
|
||||
When having multiple Alertmanagers, note that each Alertmanager manages its own independent notification resources, such as contact points, templates, policies, silences, mute timings, and active notifications.
|
||||
|
||||
These notification resources cannot be shared across different Alertmanagers.
|
||||
|
||||
Use the **Choose Alertmanager** dropdown to select the Alertmanager you want to configure.
|
||||
|
||||
@ -20,26 +20,67 @@ labels:
|
||||
title: Configure notification policies
|
||||
weight: 420
|
||||
refs:
|
||||
notification-policies:
|
||||
alertmanager-architecture:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/#alertmanager-architecture
|
||||
intro-notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
configure-mute-timings:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings/
|
||||
configure-contact-points:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/
|
||||
policy-routing:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#routing
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/#routing
|
||||
policy-inheritance:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/#inheritance
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/#inheritance
|
||||
policy-grouping:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#group-notifications
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/#group-notifications
|
||||
policy-timing-options:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/#timing-options
|
||||
---
|
||||
|
||||
# Configure notification policies
|
||||
|
||||
Notification policies determine how alerts are routed to contact points.
|
||||
|
||||
Policies have a tree structure and each policy can have one or more child policies. Each policy, except for the default policy, can also match specific alert labels.
|
||||
Policies have a tree structure. Each policy can have one or more child policies and a set of label matchers.
|
||||
|
||||
Each alert is evaluated by the default policy and subsequently by each child policy.
|
||||
Each alert (or alert instance) is evaluated by the default policy and subsequently by each child policy. Alerts are routed to the appropriate notification policy by matching alert labels with the policy's label matchers. For further details, refer to [label matching and routing in notification policies](ref:intro-notification-policies).
|
||||
|
||||
If the **Continue matching subsequent sibling nodes** option is enabled for a child policy, evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the child policies. A default policy governs any alert that does not match a child policy.
|
||||
{{< figure src="/media/docs/alerting//get-started-notification-policy-tree-combo.png" max-width="750px" alt="A diagram displaying how the notification policy tree routes alerts" >}}
|
||||
|
||||
You can configure Grafana-managed notification policies as well as notification policies for an external Alertmanager data source.
|
||||
When an alert instance is assigned to a notification policy, the notification policy is responsible for:
|
||||
|
||||
For more information on notification policies, refer to [fundamentals of Notification Policies](ref:notification-policies).
|
||||
- [Grouping similar alerts](ref:policy-grouping) to minimize the number of notifications.
|
||||
- Controlling when notifications are sent using the [timing options](ref:policy-timing-options).
|
||||
- Determining the [contact points](ref:configure-contact-points) that receive the alert notification.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
The default notification policy and its child policies are assigned to a [specific Alertmanager](ref:alertmanager-architecture), and they cannot use contact points or mute timings from other Alertmanagers.
|
||||
{{% /admonition %}}
|
||||
|
||||
## Edit the default notification policy
|
||||
|
||||
@ -47,12 +88,12 @@ For more information on notification policies, refer to [fundamentals of Notific
|
||||
1. Click **Notification policies**.
|
||||
1. From the **Choose Alertmanager** dropdown, select an external Alertmanager. By default, the **Grafana Alertmanager** is selected.
|
||||
1. In the Default policy section, click **...** -> **Edit**.
|
||||
1. In **Default contact point**, update the contact point for where to send notifications when alert rules do not match any specific policy.
|
||||
1. In **Group by**, choose labels to group alerts by. If multiple alerts are matched for this policy, then they are grouped by these labels. A notification is sent per group. If the field is empty (default), then all notifications are sent in a single group. Use a special label `...` to group alerts by all labels (which effectively disables grouping).
|
||||
1. In **Timing options**, select from the following options:
|
||||
- **Group wait** Time to wait to buffer alerts of the same group before sending an initial notification. Default is 30 seconds.
|
||||
- **Group interval** Minimum time interval between two notifications for a group. Default is 5 minutes.
|
||||
- **Repeat interval** Minimum time interval for re-sending a notification if no new alerts were added to the group. Default is 4 hours.
|
||||
1. In **Default contact point**, update the [contact point](ref:configure-contact-points) for where to send notifications when alert rules do not match any specific policy.
|
||||
1. In **Group by**, choose labels to group alerts. If multiple alerts match this policy, they are grouped by the selected labels, and notifications are sent per group. For more details on using this option, refer to [Group notifications](ref:policy-grouping).
|
||||
1. In **Timing options**, set the [timing options](ref:policy-timing-options) to configure when notifications are sent.
|
||||
- **Group wait**: the time to wait before sending the first notification for a new group of alerts. Default is 30 seconds.
|
||||
- **Group interval**: the time to wait before sending a notification about changes in the alert group. Default is 5 minutes.
|
||||
- **Repeat interval**: the time to wait before sending a notification if the group has not changed since the last notification. Default is 4 hours.
|
||||
1. Click **Save** to save your changes.
|
||||
|
||||
## Add a child policy
|
||||
@ -62,15 +103,18 @@ You can create a child policy under a default policy or under an existing child
|
||||
If you want to choose where to position your policy, see the section on **Add a sibling policy**.
|
||||
|
||||
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
|
||||
2. Click **Notification policies**.
|
||||
3. From the **Choose Alertmanager** dropdown, select an Alertmanager. By default, the **Grafana Alertmanager** is selected.
|
||||
4. Click **+New child policy** from the default policy.
|
||||
5. In the Matching labels section, add one or more rules for matching alert labels.
|
||||
6. In the **Contact point** dropdown, select the contact point to send notification to if alert matches only this specific policy and not any of the child policies.
|
||||
7. Optionally, enable **Continue matching subsequent sibling nodes** to continue matching sibling policies even after the alert matched the current policy. When this option is enabled, you can get more than one notification for one alert.
|
||||
8. Optionally, enable **Override grouping** to specify the same grouping as the default policy. If this option is not enabled, the default policy grouping is used.
|
||||
9. Optionally, enable **Override general timings** to override the timing options configured in the group notification policy.
|
||||
10. Click **Save policy** to save your changes.
|
||||
1. Click **Notification policies**.
|
||||
1. From the **Choose Alertmanager** dropdown, select an Alertmanager. By default, the **Grafana Alertmanager** is selected.
|
||||
1. Click **+New child policy** from the default policy or an existing child policy.
|
||||
1. In the Matching labels section of the child policy, add one or more matching label rules to narrow down a specific case from the parent policy.
|
||||
|
||||
For instance, a child policy of the default policy handles `team=security` alerts, or a child policy handles only the `severity=critical` alerts of a parent policy.
|
||||
|
||||
1. In the **Contact point** dropdown, select the [contact point](ref:configure-contact-points) to send notifications. If left empty, the contact point of the parent policy is [inherited](ref:policy-inheritance).
|
||||
1. Optionally, enable **Continue matching subsequent sibling nodes** to continue matching sibling policies even after the alert matched the current policy. If enabled, multiple policies can handle the same alert.
|
||||
1. Optionally, enable **Override grouping** to set different [grouping](ref:policy-grouping) than the parent policy. If disabled, the grouping of the parent policy is [inherited](ref:policy-inheritance).
|
||||
1. Optionally, enable **Override general timings** to set different [timing options](ref:policy-timing-options) than the parent policy. If disabled, the timing options of the parent policy are [inherited](ref:policy-inheritance).
|
||||
1. Click **Save policy** to save your changes.
|
||||
|
||||
## Add a sibling policy
|
||||
|
||||
@ -79,9 +123,11 @@ If you want to choose where to position your policy, see the section on **Add a
|
||||
1. Find the child policy you want to create a sibling for.
|
||||
1. Click **Add new policy** -> **New sibling above** or **New sibling below**.
|
||||
|
||||
Notification policies are evaluated from top to bottom, so it is key to be able to choose which notification policy receives alerts first. Adding sibling notification policies instead of always inserting a child policy as well as being able to choose where to insert is key to managing where your alerts go.
|
||||
It's important to determine which policy receives the alert first and to set the correct order of sibling and child policies.
|
||||
|
||||
1. Click **Save policy** to save your changes.
|
||||
Policies are evaluated from top to bottom. If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. For more details, refer to [notification policies routing](ref:policy-routing).
|
||||
|
||||
1. Follow the instructions from step 5 onward in [adding a child policy](#add-a-child-policy).
|
||||
|
||||
## Search for policies
|
||||
|
||||
@ -98,7 +144,7 @@ It is important to note that all matched policies are **exact** matches. Grafana
|
||||
|
||||
## Mute timings
|
||||
|
||||
Mute timings are not inherited from a parent notification policy. They have to be configured in full on each level.
|
||||
Mute timings are not inherited from a parent notification policy, and they have to be configured on each level. For instructions, refer to [Configure mute timings](ref:configure-mute-timings).
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
@ -22,32 +22,47 @@ labels:
|
||||
title: Configure silences
|
||||
weight: 440
|
||||
refs:
|
||||
alert-labels:
|
||||
configure-alertmanager:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager/
|
||||
silence-url:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference/#alert
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#alert
|
||||
shared-alert-labels:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/
|
||||
notification-policies:
|
||||
shared-notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
silences:
|
||||
shared-silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
shared-mute-timings:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings/
|
||||
---
|
||||
|
||||
# Configure silences
|
||||
|
||||
Silences stop notifications from getting created and last for only a specified window of time.
|
||||
Silences stop notifications from getting created and last for only a specified window of time. Use them to temporarily prevent alert notifications, such as during incident response or a maintenance window.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Silences are assigned to a [specific Alertmanager](ref:alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
|
||||
{{< /admonition >}}
|
||||
|
||||
- Inhibition rules are not supported in the Grafana Alertmanager.
|
||||
- The preview of silenced alerts only applies to alerts in firing state.
|
||||
{{< /admonition >}}
|
||||
{{< docs/shared lookup="alerts/mute-timings-vs-silences.md" source="grafana" version="<GRAFANA_VERSION>" >}}
|
||||
|
||||
## Add silences
|
||||
|
||||
@ -59,10 +74,12 @@ To add a silence, complete the following steps.
|
||||
1. Click **Create silence** to open the Create silence page.
|
||||
1. In **Silence start and end**, select the start and end date to indicate when the silence should go into effect and expire.
|
||||
1. Optionally, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field.
|
||||
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_. Matchers determine which rules the silence will apply to. Any matching alerts (in firing state) will show in the **Silenced alert instances** field
|
||||
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_ to determine which alerts the silence applies to.
|
||||
|
||||
{{< docs/shared lookup="alerts/how_label_matching_works.md" source="grafana" version="<GRAFANA_VERSION>" >}}
|
||||
|
||||
Any matching alerts (in the firing state only) will show under **Affected alert rule instances**.
|
||||
|
||||
1. In **Comment**, add details about the silence.
|
||||
1. Click **Submit**.
|
||||
|
||||
@ -75,14 +92,6 @@ To edit a silence, complete the following steps.
|
||||
1. Find the silence you want to edit, then click **Edit** (pen icon).
|
||||
1. Make the desired changes, then click **Submit** to save your changes.
|
||||
|
||||
## Create a URL to link to a silence form
|
||||
|
||||
When linking to a silence form, provide the default matching labels and comment via `matcher` and `comment` query parameters. The `matcher` parameter should be in the following format `[label][operator][value]` where the `operator` parameter can be one of the following: `=` (equals, not regular expression), `!=` (not equals, not regular expression), `=~` (equals, regular expression), `!~` (not equals, regular expression).
|
||||
The URL can contain many query parameters with the key `matcher`.
|
||||
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
|
||||
|
||||
To link to a new silence page for an external Alertmanager, add a `alertmanager` query parameter with the Alertmanager data source name.
|
||||
|
||||
## Remove silences
|
||||
|
||||
To remove a silence, complete the following steps.
|
||||
@ -95,13 +104,30 @@ To remove a silence, complete the following steps.
|
||||
|
||||
## Rule-specific silences
|
||||
|
||||
Rule-specific silences are silences that apply only to a specific alert rule.
|
||||
They're created when you silence an alert rule directly using the **Silence notifications** action in the UI.
|
||||
Rule-specific silences are silences that apply only to a specific alert rule. They're created when you silence an alert rule directly using the **Silence notifications** action in the UI.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
As opposed to general silences, rule-specific silence access is tied directly to the alert rule they act on. They can be created manually by including the specific label matcher: `__alert_rule_uid__=<alert rule UID>`.
|
||||
{{< /admonition >}}
|
||||
|
||||
## Useful links
|
||||
## URL link to a silence form
|
||||
|
||||
[Aggregation operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators)
|
||||
Default notification messages often include a link to silence alerts.
|
||||
|
||||
In custom notification templates, you can use [`.Alert.SilenceURL`](ref:silence-url) to redirect users to the UI where they can silence the given alert.
|
||||
|
||||
If [`.Alert.SilenceURL`](ref:silence-url) doesn’t fit your specific use case, you can also create a custom silence link for your custom templates.
|
||||
|
||||
{{< collapse title="Create a custom silence link" >}}
|
||||
|
||||
When linking to a silence form, provide the default matching labels and comment via `matcher` and `comment` query parameters. The `matcher` parameter should be in the following format `[label][operator][value]` where the `operator` parameter can be one of the following: `=` (equals, not regular expression), `!=` (not equals, not regular expression), `=~` (equals, regular expression), `!~` (not equals, regular expression).
|
||||
The URL can contain many query parameters with the key `matcher`.
|
||||
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
|
||||
|
||||
To link to a new silence page for an external Alertmanager, add a `alertmanager` query parameter with the Alertmanager data source name.
|
||||
|
||||
{{< /collapse >}}
|
||||
|
||||
## Inhibition rules
|
||||
|
||||
Inhibition rules are supported in the Prometheus Alertmanager. You can [configure a Prometheus Alertmanager](ref:configure-alertmanager) to handle the notification of alerts and suppress notifications via inhibition rules.
|
||||
|
||||
Inhibition rules are not currently supported in the Grafana Alertmanager. For tracking the progress of this feature request, follow [this GitHub issue](https://github.com/grafana/grafana/issues/68822).
|
||||
|
||||
@ -81,16 +81,21 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/configure-teams/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/configure-teams/
|
||||
external-alertmanager:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager/
|
||||
mqtt:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/configure-mqtt/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/configure-mqtt/
|
||||
alertmanager-architecture:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/#alertmanager-architecture
|
||||
external-alertmanager:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager/
|
||||
manage-notification-templates:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/manage-notification-templates/
|
||||
@ -113,6 +118,10 @@ On the **Contact Points** tab, you can:
|
||||
- Export individual contact points or all contact points in JSON, YAML, or Terraform format.
|
||||
- Delete contact points. Note that you cannot delete contact points that are in use by a notification policy. To proceed, either delete the notification policy or update it to use another contact point.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
Contact points are assigned to a [specific Alertmanager](ref:alertmanager-architecture) and cannot be used by notification policies in other Alertmanagers.
|
||||
{{% /admonition %}}
|
||||
|
||||
## Add a contact point
|
||||
|
||||
Complete the following steps to add a contact point.
|
||||
@ -150,6 +159,16 @@ Testing a contact point is only available for Grafana Alertmanager. Complete the
|
||||
1. Choose whether to send a predefined test notification or choose custom to add your own custom annotations and labels to include in the notification.
|
||||
1. Click **Send test notification** to fire the alert.
|
||||
|
||||
## Customize notification messages
|
||||
|
||||
In contact points, you can also customize notification messages. For example, when setting up an email contact point integration, click **Message** or **Subject** to modify it.
|
||||
|
||||
By default, notification messages include common alert details, which are usually sufficient for most cases.
|
||||
|
||||
If necessary, you can customize the content and format of notification messages. You can create a custom notification template, which can then be applied to one or more contact points.
|
||||
|
||||
On the **Notification templates** tab, you can view, edit, copy or delete notification templates. Refer to [manage notification templates](ref:manage-notification-templates) for instructions on selecting or creating a template for a contact point.
|
||||
|
||||
## List of supported integrations
|
||||
|
||||
Each contact point integration has its own configuration options and setup process. In most cases, this involves providing an API key or a Webhook URL.
|
||||
@ -182,13 +201,3 @@ The following table lists the contact point integrations supported by Grafana.
|
||||
| WeCom | `wecom` |
|
||||
|
||||
Some of these integrations are not compatible with [external Alertmanagers](ref:external-alertmanager). For the list of Prometheus Alertmanager integrations, refer to the [Prometheus Alertmanager receiver settings](https://prometheus.io/docs/alerting/latest/configuration/#receiver-integration-settings).
|
||||
|
||||
## Customize notification messages
|
||||
|
||||
In contact points, you can also customize notification messages. For example, when setting up an email contact point integration, click **Message** or **Subject** to modify it.
|
||||
|
||||
By default, notification messages include common alert details, which are usually sufficient for most cases.
|
||||
|
||||
If necessary, you can customize the content and format of notification messages. You can create a custom notification template, which can then be applied to one or more contact points.
|
||||
|
||||
On the **Notification templates** tab, you can view, edit, copy or delete notification templates. Refer to [manage notification templates](ref:manage-notification-templates) for instructions on selecting or creating a template for a contact point.
|
||||
|
||||
@ -18,31 +18,34 @@ labels:
|
||||
- enterprise
|
||||
- oss
|
||||
title: Configure mute timings
|
||||
weight: 450
|
||||
weight: 430
|
||||
refs:
|
||||
external-alertmanager:
|
||||
alertmanager-architecture:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/set-up/configure-alertmanager/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/#alertmanager-architecture
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/#alertmanager-architecture
|
||||
shared-silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
shared-mute-timings:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/mute-timings/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/mute-timings/
|
||||
---
|
||||
|
||||
# Configure mute timings
|
||||
|
||||
A mute timing is a recurring interval of time when no new notifications for a policy are generated or sent. Use them to prevent alerts from firing a specific and reoccurring period, for example, a regular maintenance period.
|
||||
A mute timing is a recurring interval of time when no new notifications for a policy are generated or sent. Use them to prevent alerts from firing a specific and reoccurring period, for example, a regular maintenance period or weekends.
|
||||
|
||||
Similar to silences, mute timings do not prevent alert rules from being evaluated, nor do they stop alert instances from being shown in the user interface. They only prevent notifications from being created.
|
||||
{{< admonition type="note" >}}
|
||||
Mute timings are assigned to a [specific Alertmanager](ref:alertmanager-architecture) and only suppress notifications for alerts managed by that Alertmanager.
|
||||
{{< /admonition >}}
|
||||
|
||||
You can configure Grafana managed mute timings as well as mute timings for an [external Alertmanager](ref:external-alertmanager).
|
||||
|
||||
## Mute timings vs silences
|
||||
|
||||
The following table highlights the key differences between mute timings and silences.
|
||||
|
||||
| Mute timing | Silence |
|
||||
| -------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| Uses time interval definitions that can reoccur | Has a fixed start and end time |
|
||||
| Is created and then added to notification policies | Uses labels to match against an alert to determine whether to silence or not |
|
||||
{{< docs/shared lookup="alerts/mute-timings-vs-silences.md" source="grafana" version="<GRAFANA_VERSION>" >}}
|
||||
|
||||
## Add mute timings
|
||||
|
||||
@ -65,6 +68,8 @@ The following table highlights the key differences between mute timings and sile
|
||||
|
||||
A time interval is a specific duration during which alerts are suppressed. The duration typically consists of a specific time range and the days of the week, month, or year.
|
||||
|
||||
A mute timing can contain multiple time intervals.
|
||||
|
||||
Supported time interval options are:
|
||||
|
||||
- Time range: The time inclusive of the start and exclusive of the end time (in UTC if no location has been selected, otherwise local time).
|
||||
@ -76,9 +81,13 @@ Supported time interval options are:
|
||||
|
||||
All fields are lists; to match the field, at least one list element must be satisfied. Fields also support ranges using `:` (e.g., `monday:thursday`).
|
||||
|
||||
If a field is left blank, any moment of time matches the field. For an instant of time to match a complete time interval, all fields must match. A mute timing can contain multiple time intervals.
|
||||
If a field is left blank, any moment of time matches the field. For an instant of time to match a complete time interval, all fields must match.
|
||||
|
||||
If you want to specify an exact duration, specify all the options. For example, if you wanted to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC your time interval specification would be:
|
||||
If you want to specify an exact duration, specify all the options.
|
||||
|
||||
**Example**
|
||||
|
||||
If you wanted to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC your time interval specification would be:
|
||||
|
||||
- Time range:
|
||||
- Start time: `12:00`
|
||||
|
||||
@ -14,7 +14,7 @@ labels:
|
||||
- enterprise
|
||||
- oss
|
||||
title: Template notifications
|
||||
weight: 430
|
||||
weight: 450
|
||||
refs:
|
||||
template-annotations-and-labels:
|
||||
- pattern: /docs/grafana/
|
||||
|
||||
@ -21,19 +21,48 @@ labels:
|
||||
title: Contact points
|
||||
weight: 112
|
||||
refs:
|
||||
contact-point-integrations:
|
||||
configure-contact-points:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points#list-of-supported-integrations
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points#list-of-supported-integrations
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points
|
||||
---
|
||||
|
||||
# Contact points
|
||||
|
||||
Contact points contain the configuration for sending alert notifications. You can assign a contact point either in the alert rule or notification policy options.
|
||||
|
||||
A contact point is a [list of integrations](ref:contact-point-integrations), each of which sends a notification to a particular email, webhook, or service such as Slack, Pagerduty, or Grafana OnCall. Each contact point also defines the notification message to be sent, which can use the predefined message, a custom message, or notification templates.
|
||||
A contact point includes one or more contact point integrations for sending alert notifications, such as:
|
||||
|
||||
Contact points can have multiple integrations of the same kind, or a combination of integrations of different kinds. For example, a contact point could contain a Pagerduty integration; an email and Slack integration; or a Pagerduty integration, a Slack integration, and two email integrations.
|
||||
{{< column-list >}}
|
||||
|
||||
You can also configure a contact point with no integrations; in which case no notifications are sent.
|
||||
- Alertmanager
|
||||
- Amazon SNS
|
||||
- Cisco Webex Teams
|
||||
- DingDing
|
||||
- Discord
|
||||
- Email
|
||||
- Google Chat
|
||||
- Grafana Oncall
|
||||
- Kafka REST Proxy
|
||||
- Line
|
||||
- Microsoft Teams
|
||||
- MQTT
|
||||
- Opsgenie
|
||||
- Pagerduty
|
||||
- Pushover
|
||||
- Sensu Go
|
||||
- Slack
|
||||
- Telegram
|
||||
- Threema Gateway
|
||||
- VictorOps
|
||||
- Webhook
|
||||
- WeCom
|
||||
|
||||
{{< /column-list >}}
|
||||
|
||||
For example, a contact point could contain a Pagerduty integration; an email and Slack integration; or a Pagerduty integration, a Slack integration, and two email integrations. You can also configure a contact point with no integrations; in which case no notifications are sent.
|
||||
|
||||
Each contact point integration can also define the notification message to be sent, which can use the predefined message, a custom message, or notification templates.
|
||||
|
||||
For a complete list of supported integrations and more details about contact points, refer to [Configure contact points](ref:configure-contact-points).
|
||||
|
||||
@ -18,6 +18,21 @@ labels:
|
||||
title: Notification policies
|
||||
weight: 113
|
||||
refs:
|
||||
shared-alert-labels:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/
|
||||
shared-notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
shared-silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
contact-points:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/contact-points/
|
||||
|
||||
@ -8,7 +8,7 @@ title: 'How label matching works'
|
||||
|
||||
{{< collapse title="How label matching works" >}}
|
||||
|
||||
Use [labels](ref:alert-labels) and label matchers to link alert rules to [notification policies](ref:notification-policies) and [silences](ref:silences). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
|
||||
Use [labels](ref:shared-alert-labels) and label matchers to link alert rules to [notification policies](ref:shared-notification-policies) and [silences](ref:shared-silences). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
|
||||
|
||||
A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.
|
||||
|
||||
|
||||
17
docs/sources/shared/alerts/mute-timings-vs-silences.md
Normal file
17
docs/sources/shared/alerts/mute-timings-vs-silences.md
Normal file
@ -0,0 +1,17 @@
|
||||
---
|
||||
labels:
|
||||
products:
|
||||
- oss
|
||||
title: 'Mute timings vs silences'
|
||||
---
|
||||
|
||||
## Mute timings vs silences
|
||||
|
||||
[Mute timings](ref:shared-mute-timings) and [silences](ref:shared-silences) are distinct methods to suppress notifications. They do not prevent alert rules from being evaluated or stop alert instances from appearing in the user interface; they only prevent notifications from being created.
|
||||
|
||||
The following table highlights the key differences between mute timings and silences.
|
||||
|
||||
| | Mute timing | Silence |
|
||||
| ---------- | ----------------------------------------------------------- | ---------------------------------------------------------------- |
|
||||
| **Setup** | Created and then added to notification policies | Matches alerts using labels to determine whether to silence them |
|
||||
| **Period** | Uses time interval definitions that can repeat periodically | Has a fixed start and end time |
|
||||
Loading…
Reference in New Issue
Block a user