mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Alerting docs: Update Intro - alert rule labels and annotations (#88230)
* Update `Labels & Annotations` * Complete labels and annotations * Update docs/sources/alerting/fundamentals/alert-rules/annotation-label.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/fundamentals/alert-rules/annotation-label.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/fundamentals/alert-rules/annotation-label.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> --------- Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
This commit is contained in:
Pepe Cano
GitHub
parent
01d5744433
commit
3d7fc8c927
@ -21,6 +21,22 @@ labels:
|
||||
- oss
|
||||
title: Configure silences
|
||||
weight: 440
|
||||
refs:
|
||||
alert-labels:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/
|
||||
notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
---
|
||||
|
||||
# Configure silences
|
||||
@ -44,6 +60,9 @@ To add a silence, complete the following steps.
|
||||
1. In **Silence start and end**, select the start and end date to indicate when the silence should go into effect and expire.
|
||||
1. Optionally, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field.
|
||||
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_. Matchers determine which rules the silence will apply to. Any matching alerts (in firing state) will show in the **Silenced alert instances** field
|
||||
|
||||
{{< docs/shared lookup="alerts/how_label_matching_works.md" source="grafana" version="<GRAFANA_VERSION>" >}}
|
||||
|
||||
1. In **Comment**, add details about the silence.
|
||||
1. Click **Submit**.
|
||||
|
||||
|
||||
@ -21,101 +21,98 @@ labels:
|
||||
title: Labels and annotations
|
||||
weight: 105
|
||||
refs:
|
||||
variables-label-annotation:
|
||||
alert-instances:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templating-labels-annotations/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals#alert-instances
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templating-labels-annotations/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals#alert-instances
|
||||
create-alerts-from-panel:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule#create-alerts-from-panels
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-grafana-managed-rule#create-alerts-from-panels
|
||||
templates:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/templates/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/templates/
|
||||
alert-rule-evaluation:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/
|
||||
silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
---
|
||||
|
||||
# Labels and annotations
|
||||
|
||||
Labels and annotations contain information about an alert. Labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.
|
||||
Labels and annotations add additional information about an alert using key/value pairs:
|
||||
|
||||
When creating alert rules, you can also template labels and annotations to optimize and customize your alerts.
|
||||
- [Labels](#labels) are used to differentiate an alert from all other alerts.
|
||||
- [Annotations](#annotations) are used to provide extra detail on an existing alert.
|
||||
|
||||
## Labels
|
||||
|
||||
**Labels** are unique identifiers of an alert. You can use them for searching, silencing, and routing notifications.
|
||||
**Labels** are unique identifiers of an [alert instance](ref:alert-instances). You can use them for searching, silencing, and routing notifications.
|
||||
|
||||
Examples of labels are `server=server1` or `team=backend`. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.
|
||||
|
||||
For example, an alert rule might have the label set `{alertname="High CPU usage",server="server1"}` while another alert rule might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alert rules because although their `alertname` labels are the same, their `server` labels are different.
|
||||
For example, an alert instance might have the label set `{alertname="High CPU usage",server="server1"}` while another alert instance might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alert instances because although their `alertname` labels are the same, their `server` labels are different.
|
||||
|
||||
{{< figure src="/static/img/docs/alerting/unified/multi-dimensional-alert.png" >}}
|
||||
|
||||
Labels are a fundamental component of alerting:
|
||||
|
||||
- The complete set of labels for an alert is what uniquely identifies an alert within Grafana alerts.
|
||||
- The complete set of labels for an alert is what uniquely identifies an alert instance.
|
||||
- The alerting UI shows labels for every alert instance generated during evaluation of that rule.
|
||||
- Contact points can access labels to send notification messages that contain specific alert information.
|
||||
- The Alertmanager uses labels to match alerts for silences and alert groups in notification policies.
|
||||
- [Notification policies](ref:notification-policies) and [silences](ref:silences) use labels to match alert instances and route them to contact points or stop their notifications.
|
||||
- Contact points can include information from labels in notification messages.
|
||||
|
||||
Note that two alert rules cannot have the same labels. If two alert rules have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the alerts will be discarded.
|
||||
|
||||
### How label matching works
|
||||
|
||||
Use labels and label matchers to link alert rules to notification policies and silences. This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
|
||||
|
||||
A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.
|
||||
|
||||
- The **Label** field is the name of the label to match. It must exactly match the label name.
|
||||
|
||||
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
|
||||
|
||||
- The **Operator** field is the operator to match against the label value. The available operators are:
|
||||
|
||||
| Operator | Description |
|
||||
| -------- | -------------------------------------------------- |
|
||||
| `=` | Select labels that are exactly equal to the value. |
|
||||
| `!=` | Select labels that are not equal to the value. |
|
||||
| `=~` | Select labels that regex-match the value. |
|
||||
| `!~` | Select labels that do not regex-match the value. |
|
||||
|
||||
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
|
||||
|
||||
{{< collapse title="Label matching example" >}}
|
||||
|
||||
If you define the following set of labels for your alert:
|
||||
|
||||
`{ foo=bar, baz=qux, id=12 }`
|
||||
|
||||
then:
|
||||
|
||||
- A label matcher defined as `foo=bar` matches this alert rule.
|
||||
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
|
||||
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
|
||||
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
|
||||
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
|
||||
|
||||
**Exclude labels**
|
||||
|
||||
You can also write label matchers to exclude labels.
|
||||
|
||||
Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.
|
||||
|
||||
| Label | Operator | Value |
|
||||
| ------ | -------- | ----- |
|
||||
| `team` | `=` | `""` |
|
||||
| `team` | `!~` | `.+` |
|
||||
| `team` | `=~` | `^$` |
|
||||
|
||||
{{< /collapse >}}
|
||||
|
||||
## Label types
|
||||
### Label types
|
||||
|
||||
An alert's label set can contain three types of labels:
|
||||
|
||||
- Data source query labels. For example, if you are monitoring temperature readings and each time series for these readings has a sensor_id, and a location label. These labels are used to provide additional context or dimensions to the metric data, helping to differentiate between different time series.
|
||||
**User-configured labels**
|
||||
|
||||
- Labels that are automatically added by Grafana (i.e. alertname and grafana_folder). These are Grafana reserved labels.
|
||||
Labels that you manually configure in the alert rule to identify the generated alert instances or group them.
|
||||
|
||||
- Labels that you define yourself to help filter data in your alert rules.
|
||||
You can also template labels. For example in your alert rule, you could add a label that uses templating to create more dynamic and customizable alerting. E.g. `environment` `=` `{{ your text/template }}`.
|
||||
You can also use a [template](ref:templates) to customize the label value and generate dynamic values when the rule is evaluated.
|
||||
|
||||
**Data source query labels**
|
||||
|
||||
For example, if you are monitoring temperature readings and each time series for these readings has a `sensor_id`, and a `location` label, an alert instance might have the labels `{sensor_id="1",location="NY"}`, while another alert instance might have `{sensor_id="2",location="WA"}`.
|
||||
|
||||
Data source query labels labels are also used to generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data.
|
||||
|
||||
**Reserved labels**
|
||||
|
||||
Reserved labels are automatically added by Grafana:
|
||||
|
||||
- `alert_name`: the name of the alert rule.
|
||||
- `grafana_folder`: the title of the folder containing the alert.
|
||||
|
||||
Labels prefixed with `grafana_` are reserved by Grafana for special use. To stop Grafana Alerting from adding a reserved label, you can disable it via the `disabled_labels` option in [unified_alerting.reserved_labels](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana#unified_alertingreserved_labels) configuration.
|
||||
|
||||
{{<admonition type="note">}}
|
||||
Ensure the label set for an alert does not have two or more labels with the same name. If a label has the same name as a label from the data source then it will replace that label. However, should a label have the same name as a reserved label then the label will be omitted from the alert.
|
||||
{{</admonition>}}
|
||||
|
||||
{{< collapse title="Key format" >}}
|
||||
Two alert rules cannot produce alert instances with the same labels. If two alert rules have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the generated alert instances will be discarded.
|
||||
|
||||
Ensure the label set for an alert does not have two or more labels with the same name.
|
||||
|
||||
- If a configured label has the same name as a data source query label, it replaces the data source label.
|
||||
- If a configured label has the same name as a reserved label, it is omitted.
|
||||
{{</admonition>}}
|
||||
|
||||
{{< collapse title="Label key format" >}}
|
||||
|
||||
Grafana's built-in Alertmanager supports both Unicode label keys and values. If you are using an external Prometheus Alertmanager, label keys must be compatible with their [data model](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
||||
This means that label keys must only contain **ASCII letters**, **numbers**, as well as **underscores** and match the regex `[a-zA-Z_][a-zA-Z0-9_]*`.
|
||||
@ -131,19 +128,15 @@ If multiple label keys are sanitized to the same value, the duplicates will have
|
||||
|
||||
{{< /collapse >}}
|
||||
|
||||
### Reserved labels
|
||||
|
||||
Reserved labels can be used in the same way as manually configured labels. The current list of available reserved labels are:
|
||||
|
||||
| Label | Description |
|
||||
| -------------- | ----------------------------------------- |
|
||||
| alert_name | The name of the alert rule. |
|
||||
| grafana_folder | Title of the folder containing the alert. |
|
||||
|
||||
Labels prefixed with `grafana_` are reserved by Grafana for special use. To stop Grafana Alerting from adding a reserved label, you can disable it via the `disabled_labels` option in [unified_alerting.reserved_labels](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana#unified_alertingreserved_labels) configuration.
|
||||
|
||||
## Annotations
|
||||
|
||||
Both labels and annotations have the same structure: a set of named values; however their intended uses are different. The purpose of annotations is to add additional information to existing alerts.
|
||||
The purpose of annotations is to add additional information to alert instances, such as extra details for notification messages.
|
||||
|
||||
There are a number of suggested annotations in Grafana such as `description`, `summary`, `runbook_url`, `dashboardUId` and `panelId`. Like labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired.
|
||||
Grafana provides several optional annotations that you can edit for use in notification messages and within Grafana:
|
||||
|
||||
- `summary`: A short summary of what the alert has detected and why.
|
||||
- `description`: A detailed description of what happened and what the alert does.
|
||||
- `runbook_url`: The runbook page to guide operators managing a potential incident.
|
||||
- `dashboardUId` and `panelId`: Link the alert to a dashboard and panel. These are automatically set when [creating an alert from panels](ref:create-alerts-from-panel).
|
||||
|
||||
Like labels, annotations can use a [template](ref:templates) to customize the label value and generate dynamic values when the rule is evaluated.
|
||||
|
||||
@ -18,11 +18,21 @@ labels:
|
||||
title: Notification policies
|
||||
weight: 113
|
||||
refs:
|
||||
labels-and-label-matchers:
|
||||
alert-labels:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#how-label-matching-works
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#how-label-matching-works
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/
|
||||
notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
---
|
||||
|
||||
# Notification policies
|
||||
@ -37,7 +47,7 @@ Notification policies are _not_ a list, but rather are structured according to a
|
||||
|
||||
Each policy consists of a set of label matchers (0 or more) that specify which labels they are or aren't interested in handling.
|
||||
|
||||
For more information on label matching, refer to [how label matching works](ref:labels-and-label-matchers).
|
||||
{{< docs/shared lookup="alerts/how_label_matching_works.md" source="grafana" version="<GRAFANA_VERSION>" >}}
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
If you haven't configured any label matchers for your notification policy, your notification policy matches _all_ alert instances. This may prevent child policies from being evaluated unless you have enabled **Continue matching siblings** on the notification policy.
|
||||
|
||||
56
docs/sources/shared/alerts/how_label_matching_works.md
Normal file
56
docs/sources/shared/alerts/how_label_matching_works.md
Normal file
@ -0,0 +1,56 @@
|
||||
---
|
||||
labels:
|
||||
products:
|
||||
- enterprise
|
||||
- oss
|
||||
title: 'How label matching works'
|
||||
---
|
||||
|
||||
{{< collapse title="How label matching works" >}}
|
||||
|
||||
Use [labels](ref:alert-labels) and label matchers to link alert rules to [notification policies](ref:notification-policies) and [silences](ref:silences). This allows for a flexible way to manage your alert instances, specify which policy should handle them, and which alerts to silence.
|
||||
|
||||
A label matchers consists of 3 distinct parts, the **label**, the **value** and the **operator**.
|
||||
|
||||
- The **Label** field is the name of the label to match. It must exactly match the label name.
|
||||
|
||||
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Operator** value.
|
||||
|
||||
- The **Operator** field is the operator to match against the label value. The available operators are:
|
||||
|
||||
| Operator | Description |
|
||||
| -------- | -------------------------------------------------- |
|
||||
| `=` | Select labels that are exactly equal to the value. |
|
||||
| `!=` | Select labels that are not equal to the value. |
|
||||
| `=~` | Select labels that regex-match the value. |
|
||||
| `!~` | Select labels that do not regex-match the value. |
|
||||
|
||||
If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
|
||||
|
||||
**Label matching example**
|
||||
|
||||
If you define the following set of labels for your alert:
|
||||
|
||||
`{ foo=bar, baz=qux, id=12 }`
|
||||
|
||||
then:
|
||||
|
||||
- A label matcher defined as `foo=bar` matches this alert rule.
|
||||
- A label matcher defined as `foo!=bar` does _not_ match this alert rule.
|
||||
- A label matcher defined as `id=~[0-9]+` matches this alert rule.
|
||||
- A label matcher defined as `baz!~[0-9]+` matches this alert rule.
|
||||
- Two label matchers defined as `foo=bar` and `id=~[0-9]+` match this alert rule.
|
||||
|
||||
**Exclude labels**
|
||||
|
||||
You can also write label matchers to exclude labels.
|
||||
|
||||
Here is an example that shows how to exclude the label `Team`. You can choose between any of the values below to exclude labels.
|
||||
|
||||
| Label | Operator | Value |
|
||||
| ------ | -------- | ----- |
|
||||
| `team` | `=` | `""` |
|
||||
| `team` | `!~` | `.+` |
|
||||
| `team` | `=~` | `^$` |
|
||||
|
||||
{{< /collapse >}}
|
||||
Loading…
Reference in New Issue
Block a user