From 0cc67bb93d6383d0969fc040e22e0f320c67a57e Mon Sep 17 00:00:00 2001 From: antonio <45235678+tonypowa@users.noreply.github.com> Date: Tue, 18 Jun 2024 09:22:23 +0200 Subject: [PATCH] alerting: notes on label matchers (#89269) * alerting: notes on label matchers * routing clarification * Update docs/sources/alerting/alerting-rules/templating-labels-annotations.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> * Update docs/sources/alerting/fundamentals/notifications/templates.md Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> --------- Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com> --- .../alerting-rules/templating-labels-annotations.md | 4 ++++ .../fundamentals/notifications/notification-policies.md | 6 ++++++ .../alerting/fundamentals/notifications/templates.md | 4 ++++ docs/sources/shared/alerts/how_label_matching_works.md | 2 ++ 4 files changed, 16 insertions(+) diff --git a/docs/sources/alerting/alerting-rules/templating-labels-annotations.md b/docs/sources/alerting/alerting-rules/templating-labels-annotations.md index f8773bce011..321d8bfd720 100644 --- a/docs/sources/alerting/alerting-rules/templating-labels-annotations.md +++ b/docs/sources/alerting/alerting-rules/templating-labels-annotations.md @@ -32,6 +32,10 @@ All templates should be written in [text/template](https://pkg.go.dev/text/templ Each template is evaluated whenever the alert rule is evaluated, and is evaluated for every alert separately. For example, if your alert rule has a templated summary annotation, and the alert rule has 10 firing alerts, then the template will be executed 10 times, once for each alert. You should try to avoid doing expensive computations in your templates as much as possible. +{{% admonition type="caution" %}} +Extra whitespace in label templates can break matches with notification policies. +{{% /admonition %}} + ## Examples The following examples attempt to show the most common use-cases we have seen for templates. You can use these examples verbatim, or adapt them as necessary for your use case. For more information on how to write text/template refer see [the beginner's guide to alert notification templates in Grafana](https://grafana.com/blog/2023/04/05/grafana-alerting-a-beginners-guide-to-templating-alert-notifications/). diff --git a/docs/sources/alerting/fundamentals/notifications/notification-policies.md b/docs/sources/alerting/fundamentals/notifications/notification-policies.md index ab1200a2f68..e9ca6e4ff03 100644 --- a/docs/sources/alerting/fundamentals/notifications/notification-policies.md +++ b/docs/sources/alerting/fundamentals/notifications/notification-policies.md @@ -72,6 +72,8 @@ By default, once a matching policy is found, the system does not continue to loo The default notification policy matches all alert instances. It always handles alert instances if there are no child policies or if none of the child policies match the alert instance's labels—this prevents any alerts from being missed. +If alerts use multiple labels, these labels must also be present in a notification policy to match and route notifications to a specific contact point. + {{% /admonition %}} {{< collapse title="Routing example" >}} @@ -84,6 +86,10 @@ The `team=security` policy is not a match and **Continue matching siblings** was **Disk Usage – 80%** has both a `team` and `severity` label, and matches a child policy of the operations team. +{{% admonition type="note" %}} +When an alert matches both a parent policy and a child policy (like it does in this case), the routing follows the child policy (`severity`) as it provides a more specific match. +{{% /admonition %}} + **Unauthorized log entry** has a `team` label but does not match the first policy (`team=operations`) since the values are not the same, so it will continue searching and match the `team=security` policy. It does not have any child policies, so the additional `severity=high` label is ignored. {{< /collapse >}} diff --git a/docs/sources/alerting/fundamentals/notifications/templates.md b/docs/sources/alerting/fundamentals/notifications/templates.md index 0866e7dc3a3..c0461d62b2c 100644 --- a/docs/sources/alerting/fundamentals/notifications/templates.md +++ b/docs/sources/alerting/fundamentals/notifications/templates.md @@ -106,6 +106,10 @@ Here are some commonly used built-in [variables](ref:variables-label-annotation) CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345 +{{% admonition type="caution" %}} +Extra whitespace in label templates can break matches with notification policies. +{{% /admonition %}} + ### Template annotations Both labels and annotations have the same structure: a set of named values; however their intended uses are different. The purpose of annotations is to add additional information to existing alerts. diff --git a/docs/sources/shared/alerts/how_label_matching_works.md b/docs/sources/shared/alerts/how_label_matching_works.md index b7409a97fde..49eae1bfcd0 100644 --- a/docs/sources/shared/alerts/how_label_matching_works.md +++ b/docs/sources/shared/alerts/how_label_matching_works.md @@ -25,7 +25,9 @@ A label matchers consists of 3 distinct parts, the **label**, the **value** and | `=~` | Select labels that regex-match the value. | | `!~` | Select labels that do not regex-match the value. | +{{% admonition type="note" %}} If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy. +{{% /admonition %}} **Label matching example**