alerting: notes on label matchers (#89269)

* alerting: notes on label matchers

* routing clarification

* Update docs/sources/alerting/alerting-rules/templating-labels-annotations.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/fundamentals/notifications/templates.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

---------

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

docs/sources/alerting/alerting-rules/templating-labels-annotations.md

@@ -32,6 +32,10 @@ All templates should be written in [text/template](https://pkg.go.dev/text/templ
 
 Each template is evaluated whenever the alert rule is evaluated, and is evaluated for every alert separately. For example, if your alert rule has a templated summary annotation, and the alert rule has 10 firing alerts, then the template will be executed 10 times, once for each alert. You should try to avoid doing expensive computations in your templates as much as possible.
 
+{{% admonition type="caution" %}}
+Extra whitespace in label templates can break matches with notification policies.
+{{% /admonition %}}
+
 ## Examples
 
 The following examples attempt to show the most common use-cases we have seen for templates. You can use these examples verbatim, or adapt them as necessary for your use case. For more information on how to write text/template refer see [the beginner's guide to alert notification templates in Grafana](https://grafana.com/blog/2023/04/05/grafana-alerting-a-beginners-guide-to-templating-alert-notifications/).

docs/sources/alerting/fundamentals/notifications/notification-policies.md

@@ -72,6 +72,8 @@ By default, once a matching policy is found, the system does not continue to loo
 
 The default notification policy matches all alert instances. It always handles alert instances if there are no child policies or if none of the child policies match the alert instance's labels—this prevents any alerts from being missed.
 
+If alerts use multiple labels, these labels must also be present in a notification policy to match and route notifications to a specific contact point.
+
 {{% /admonition %}}
 
 {{< collapse title="Routing example" >}}
@@ -84,6 +86,10 @@ The `team=security` policy is not a match and **Continue matching siblings** was
 
 **Disk Usage – 80%** has both a `team` and `severity` label, and matches a child policy of the operations team.
 
+{{% admonition type="note" %}}
+When an alert matches both a parent policy and a child policy (like it does in this case), the routing follows the child policy (`severity`) as it provides a more specific match.
+{{% /admonition %}}
+
 **Unauthorized log entry** has a `team` label but does not match the first policy (`team=operations`) since the values are not the same, so it will continue searching and match the `team=security` policy. It does not have any child policies, so the additional `severity=high` label is ignored.
 
 {{< /collapse >}}

docs/sources/alerting/fundamentals/notifications/templates.md

@@ -106,6 +106,10 @@ Here are some commonly used built-in [variables](ref:variables-label-annotation)
 
         CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345
 
+{{% admonition type="caution" %}}
+Extra whitespace in label templates can break matches with notification policies.
+{{% /admonition %}}
+
 ### Template annotations
 
 Both labels and annotations have the same structure: a set of named values; however their intended uses are different. The purpose of annotations is to add additional information to existing alerts.

docs/sources/shared/alerts/how_label_matching_works.md

@@ -25,7 +25,9 @@ A label matchers consists of 3 distinct parts, the **label**, the **value** and
   | `=~`     | Select labels that regex-match the value.          |
   | `!~`     | Select labels that do not regex-match the value.   |
 
+{{% admonition type="note" %}}
 If you are using multiple label matchers, they are combined using the AND logical operator. This means that all matchers must match in order to link a rule to a policy.
+{{% /admonition %}}
 
 **Label matching example**