mirror of
https://github.com/grafana/grafana.git
synced 2026-07-30 00:08:10 -05:00
Docs: Improve Labels and annotations docs (#59325)
This commit makes a number of changes to the docs for Labels and
annotations.
1. It changes the order in which Labels and annotations are mentioned
from Annotations and labels to Labels and annotations as this is
the order shown in the UI when creating and editing alert rules.
It is also the order in the Prometheus documentation.
2. It changes most of the documentation explaining what labels and
annotations are and the differences between them. It also adds
some paragraphs on Custom Labels.
This commit is contained in:
George Robinson
@@ -10,18 +10,34 @@ keywords:
|
||||
- guide
|
||||
- rules
|
||||
- create
|
||||
title: Annotations and labels for alerting rules
|
||||
title: Labels and annotations
|
||||
weight: 401
|
||||
---
|
||||
|
||||
# Annotations and labels for alerting rules
|
||||
# Labels and annotations
|
||||
|
||||
Annotations and labels are key value pairs associated with alerts originating from the alerting rule, datasource response, and as a result of alerting rule evaluation. They can be used in alert notifications directly or in templates and template functions to create notification content dynamically.
|
||||
Labels and annotations contain information about an alert. Both labels and annotations have the same structure: a set of named values; however their intended uses are different. An example of label, or the equivalent annotation, might be `alertname="test"`.
|
||||
|
||||
## Annotations
|
||||
The main difference between a label and an annotation is that labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.
|
||||
|
||||
Annotations are key-value pairs that provide additional meta-information about an alert. You can use the following annotations: `description`, `summary`, `runbook_url`, `alertId`, `dashboardUid`, and `panelId`. For example, a description, a summary, and a runbook URL. These are displayed in rule and alert details in the UI and can be used in contact point message templates.
|
||||
For example, consider two high CPU alerts: one for `server1` and another for `server2`. In such an example we might have a label called `server` where the first alert has the label `server="server1"` and the second alert has the label `server="server2"`. However, we might also want to add a description to each alert such as `"The CPU usage for server1 is above 75%."`, where `server1` and `75%` are replaced with the name and CPU usage of the server (please refer to the documentation on [templating labels and annotations]({{< relref "./variables-label-annotation" >}}) for how to do this). This kind of description would be more suitable as an annotation.
|
||||
|
||||
## Labels
|
||||
|
||||
Labels are key-value pairs that contain information about, and are used to uniquely identify an alert. The label set for an alert is generated and added to throughout the alerting evaluation and notification process.
|
||||
Labels contain information that identifies an alert. An example of a label might be `server=server1`. Each alert can have more than one label, and the complete set of labels for an alert is called its label set. It is this label set that identifies the alert.
|
||||
|
||||
For example, an alert might have the label set `{alertname="High CPU usage",server="server1"}` while another alert might have the label set `{alertname="High CPU usage",server="server2"}`. These are two separate alerts because although their `alertname` labels are the same, their `server` labels are different.
|
||||
|
||||
The label set for an alert is a combination of the labels from the datasource, custom labels from the alert rule, and a number of reserved labels such as `alertname`.
|
||||
|
||||
### Custom Labels
|
||||
|
||||
Custom labels are additional labels from the alert rule. Like annotations, custom labels must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. Documentation on how to template custom labels can be found [here]({{< relref "./variables-label-annotation" >}}).
|
||||
|
||||
When using custom labels with templates it is important to make sure that the label value does not change between consecutive evaluations of the alert rule as this will end up creating large numbers of distinct alerts. However, it is OK for the template to produce different label values for different alerts. For example, do not put the value of the query in a custom label as this will end up creating a new set of alerts each time the value changes. Instead use annotations.
|
||||
|
||||
It is also important to make sure that the label set for an alert does not have two or more labels with the same name. If a custom label has the same name as a label from the datasource then it will replace that label. However, should a custom label have the same name as a reserved label then the custom label will be omitted from the alert.
|
||||
|
||||
## Annotations
|
||||
|
||||
Annotations are named pairs that add additional information to existing alerts. There are a number of suggested annotations in Grafana such as `description`, `summary`, `runbook_url`, `dashboardUId` and `panelId`. Like custom labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. If an annotation contains template code, the template is evaluated once when the alert is fired. It is not re-evaluated, even when the alert is resolved. Documentation on how to template annotations can be found [here]({{< relref "./variables-label-annotation" >}}).
|
||||
|
||||
@@ -1,27 +1,23 @@
|
||||
---
|
||||
aliases:
|
||||
- /docs/grafana/latest/alerting/fundamentals/annotation-label/variables-label-annotation/
|
||||
description: Learn about labels and label matchers in alerting
|
||||
description: Learn about templating of labels and annotations
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
- fundamentals
|
||||
title: How to template annotations and labels
|
||||
title: Templating labels and annotations
|
||||
weight: 117
|
||||
---
|
||||
|
||||
# How to template annotations and labels
|
||||
# Templating labels and annotations
|
||||
|
||||
In Grafana it is possible to template annotations and labels just like you would in Prometheus. Those who have used
|
||||
Prometheus before should be familiar with the `$labels` variable which holds the label key/value pairs of the alert
|
||||
instance and the `$value` variable which holds the evaluated value of the alert instance.
|
||||
In Grafana it is possible to template labels and annotations just like you would in Prometheus. Those who have used Prometheus before should be familiar with the `$labels` variable which holds the label key/value pairs of the alert instance and the `$value` variable which holds the evaluated value of the alert instance.
|
||||
|
||||
In Grafana it is possible to use the same variables from Prometheus to template annotations and labels, even if your
|
||||
alert does not use a Prometheus datasource.
|
||||
In Grafana it is possible to use the same variables from Prometheus to template labels and annotations, even if your alert does not use a Prometheus datasource.
|
||||
|
||||
For example, let's suppose we want to create an alert in Grafana that tells us when one of our instances is down for
|
||||
more than 5 minutes. Like in Prometheus, we can add a summary annotation to show the instance which is down:
|
||||
For example, let's suppose we want to create an alert in Grafana that tells us when one of our instances is down for more than 5 minutes. Like in Prometheus, we can add a summary annotation to show the instance which is down:
|
||||
|
||||
```
|
||||
Instance {{ $labels.instance }} has been down for more than 5 minutes
|
||||
@@ -43,17 +39,13 @@ of the condition at the time the alert fired. For example:
|
||||
|
||||
## Alert rules with two or more queries or expressions
|
||||
|
||||
In the case where an alert rule has two or more queries, or uses reduce and math expressions, it is possible to template
|
||||
the reduced result of each query and expression with the `$values` variable. This variable holds the labels and value of
|
||||
each reduced query, and the results of any math expressions. However, it does not hold the samples for each query.
|
||||
In the case where an alert rule has two or more queries, or uses reduce and math expressions, it is possible to template the reduced result of each query and expression with the `$values` variable. This variable holds the labels and value of each reduced query, and the results of any math expressions. However, it does not hold the samples for each query.
|
||||
|
||||
For example, suppose you have the following alert rule:
|
||||
|
||||
{{< figure src="/static/img/docs/alerting/unified/grafana-alerting-histogram-quantile.png" class="docs-image--no-shadow" caption="An alert rule that uses histogram_quantile to compute 95th percentile" >}}
|
||||
|
||||
Should this rule create an alert instance `$values` will hold the result of the reduce expression `B` and the math
|
||||
expression `C`. It will not hold the results returned by query `A` because query `A` does not return a single value
|
||||
but rather a series of values over time.
|
||||
Should this rule create an alert instance `$values` will hold the result of the reduce expression `B` and the math expression `C`. It will not hold the results returned by query `A` because query `A` does not return a single value but rather a series of values over time.
|
||||
|
||||
If we were to write a summary annotation such as:
|
||||
|
||||
@@ -61,15 +53,13 @@ If we were to write a summary annotation such as:
|
||||
{{ $labels.instance }} has a 95th percentile request latency above 1s: {{ $value }})
|
||||
```
|
||||
|
||||
We would find that because the condition of the alert, the math expression `C` must be a boolean comparison, it must
|
||||
return either a `0` or a `1`. What we want instead is the 95th percentile from the reduce expression `B`:
|
||||
We would find that because the condition of the alert, the math expression `C` must be a boolean comparison, it must return either a `0` or a `1`. What we want instead is the 95th percentile from the reduce expression `B`:
|
||||
|
||||
```
|
||||
{{ $labels.instance }} has a 95th percentile request latency above 1s: {{ $values.B }})
|
||||
```
|
||||
|
||||
We can also show the labels of `B`, however since this alert rule has just one query the labels of `B` are equivalent to
|
||||
`$labels`:
|
||||
We can also show the labels of `B`, however since this alert rule has just one query the labels of `B` are equivalent to `$labels`:
|
||||
|
||||
```
|
||||
{{ $values.B.Labels.instance }} has a 95th percentile request latency above 1s: {{ $values.B }})
|
||||
@@ -78,8 +68,7 @@ We can also show the labels of `B`, however since this alert rule has just one q
|
||||
### No data and execution errors or timeouts
|
||||
|
||||
Should query `A` return no data then the reduce expression `B` will also return no data. This means that
|
||||
`{{ $values.B }}` will be nil. To ensure that annotations and labels can still be templated even when a query returns
|
||||
no data, we can use an if statement to check for `$values.B`:
|
||||
`{{ $values.B }}` will be nil. To ensure that labels and annotations can still be templated even when a query returns no data, we can use an if statement to check for `$values.B`:
|
||||
|
||||
```
|
||||
{{ if $values.B }}{{ $labels.instance }} has a 95th percentile request latency above 1s: {{ $values.B }}){{ end }}
|
||||
@@ -87,12 +76,11 @@ no data, we can use an if statement to check for `$values.B`:
|
||||
|
||||
## Classic conditions
|
||||
|
||||
If the rule uses a classic condition instead of a reduce and math expression, then `$values` contains the combination
|
||||
of the `refID` and position of the condition. For example, `{{ $values.A0 }}` and `{{ $values.A1 }}`.
|
||||
If the rule uses a classic condition instead of a reduce and math expression, then `$values` contains the combination of the `refID` and position of the condition. For example, `{{ $values.A0 }}` and `{{ $values.A1 }}`.
|
||||
|
||||
## Variables
|
||||
|
||||
The following template variables are available when expanding annotations and labels.
|
||||
The following template variables are available when expanding labels and annotations.
|
||||
|
||||
| Name | Description |
|
||||
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
|
||||
Reference in New Issue
Block a user