grafana/pkg/services/ngalert
George Robinson c8ccc4649c
Alerting: Support UTF-8 (#81512)
This pull request updates our fork of Alertmanager to commit 65bdab0, which is based on commit 5658f8c in Prometheus Alertmanager.

It applies the changes from grafana/alerting#155 which removes the overrides for validation of alerts, labels and silences that we had put in place to allow alerts and silences to work for non-Prometheus datasources. However, as this is now supported in Alertmanager with the UTF-8 work, we can use the new upstream functions and remove these overrides.

The compat package is a package in Alertmanager that takes care of backwards compatibility when parsing matchers, validating alerts, labels and silences. It has three modes: classic mode, UTF-8 strict mode, fallback mode. These modes are controlled via compat.InitFromFlags. Grafana initializes the compat package without any feature flags, which is the equivalent of fallback mode. Classic and UTF-8 strict mode are used in Mimir.

While Grafana Managed Alerts have no need for fallback mode, Grafana can still be used as an interface to manage the configurations of Mimir Alertmanagers and view configurations of Prometheus Alertmanager, and those installations might not have migrated or being running on older versions. Such installations behave as if in classic mode, and Grafana must be able to parse their configurations to interact with them for some period of time. As such, Grafana uses fallback mode until we are ready to drop support for outdated installations of Mimir and the Prometheus Alertmanager.
2024-02-06 08:33:47 +00:00
..
accesscontrol Alerting: Fix fine-grained rule access control to use 403 for authorization error (#79239) 2023-12-07 13:43:58 -05:00
api Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
backtesting feat(alerting): add state persister interface (#80384) 2024-01-17 13:33:13 +01:00
client Alerting: Add metrics to the remote Alertmanager struct (#79835) 2024-01-10 11:18:24 +01:00
eval Alerting/Annotations: Add annotation backend for Loki alert state history (#78156) 2024-01-10 18:42:35 -05:00
image Chore: Remove public vars in setting package (#81018) 2024-01-23 12:36:22 +01:00
metrics Alerting: Support UTF-8 (#81512) 2024-02-06 08:33:47 +00:00
migration User: Add uid colum to user table (#81615) 2024-02-01 18:14:10 -08:00
models Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
notifier Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
provisioning Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
remote Alerting: Retry readiness check to the remote Alertmanager on 5xx status code responses (#81174) 2024-01-24 21:39:06 +01:00
schedule QueryField: Handle autocomplete better (#81484) 2024-01-31 10:01:20 +00:00
sender Alerting: Refactor provisioning tests/fakes (#81205) 2024-01-24 17:15:55 -05:00
state Alerting: Add a feature flag to periodically save states (#80987) 2024-01-23 17:03:30 +01:00
store Alerting: Fix scheduler to group folders by the unique key (orgID and UID) (#81303) 2024-01-30 17:14:11 -05:00
tests Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
testutil Folders: Show dashboards and folders with directly assigned permissions in "Shared" folder (#78465) 2023-12-05 16:13:31 +01:00
accesscontrol.go Alerting: Receivers API (read only endpoints) (#81751) 2024-02-05 20:12:15 +02:00
CHANGELOG.md Update Alerting changelog (#56684) 2022-10-11 10:55:18 +00:00
limits_test.go Alerting: Decouple quota configuration logic from API interfaces and add tests (#78930) 2023-12-01 10:47:19 -06:00
limits.go Alerting: Guided legacy alerting upgrade dry-run (#80071) 2024-01-05 18:19:12 -05:00
ngalert_test.go Remove folderid service test (#80433) 2024-01-12 16:43:39 +01:00
ngalert.go Alerting: Support UTF-8 (#81512) 2024-02-06 08:33:47 +00:00
README.md Alerting: Document not supporting inhibition rules (#61313) 2023-01-13 15:06:06 +01:00

Next generation alerting (ngalert) in Grafana 8

Ngalert (Next generation alert) is the next generation of alerting in Grafana 8.

Overview

The ngalert package can be found in pkg/services/ngalert and has the following sub-packages:

- api
- eval
- logging
- metrics
- models
- notifier
- schedule
- sender
- state
- store
- tests

Scheduling and evaluation of alert rules

The scheduling of alert rules happens in the schedule package. This package is responsible for managing the evaluation of alert rules including checking for new alert rules and stopping the evaluation of deleted alert rules.

The scheduler runs at a fixed interval, called its heartbeat, in which it does a number of tasks:

  1. Fetch the alert rules for all organizations (excluding disabled)
  2. Start a goroutine (if this is a new alert rule or the scheduler has just started) to evaluate the alert rule
  3. Send an *evaluation event to the goroutine for each alert rule if its interval has elapsed
  4. Stop the goroutines for all alert rules that have been deleted since the last heartbeat

The function that evaluates each alert rule is called ruleRoutine. It waits for an *evaluation event (sent each interval seconds elapsed and is configurable per alert rule) and then evaluates the alert rule. To ensure that the scheduler is evaluating the latest version of the alert rule it compares its local version of the alert rule with that in the *evaluation event, fetching the latest version of the alert rule from the database if the version numbers mismatch. It then invokes the Evaluator which evaluates any queries, classic conditions or expressions in alert rule and passes the results of this evaluation to the State Manager. An evaluation can return no results in the case of NoData or Error, a single result in the case of classic conditions, or more than one result if the alert rule is multi-dimensional (i.e. one result per label set). In the case of multi-dimensional alert rules the results from an evaluation should never contain more than one per label set.

The State Manager is responsible for determining the current state of the alert rule (normal, pending, firing, etc) by comparing each evaluation result to the previous evaluations of the same label set in the state cache. Given a label set, it updates the state cache with the new current state, the evaluation time of the current evaluation and appends the current evaluation to the slice of previous evaluations. If the alert changes state (i.e. pending to firing) then it also creates an annotation to mark it on the dashboard and panel for this alert rule.

You might have noticed that so far we have avoided using the word "Alert" and instead talked about evaluation results and the current state of an alert rule. The reason for that is at this time in the evaluation of an alert rule the State Manager does not know about alerts, it just knows for each label set the state of an alert rule, the current evaluation and previous evaluations.

Notification of alerts

When an evaluation transitions the state of an alert rule for a given label set from pending to firing or from firing to normal the scheduler creates an alert instance and passes it to Alertmanager. In the case where a label set is transitioning from pending to firing the state of the alert instance is "Firing" and when transitioning from firing to normal the state of the alert instance is "Normal".

Which Alertmanager?

In ngalert it is possible to send alerts to the internal Alertmanager, an external Alertmanager, or both.

The internal Alertmanager is called MultiOrgAlertmanager and creates an Alertmanager for each organization in Grafana to preserve isolation between organizations in Grafana. The MultiOrgAlertmanager receives alerts from the scheduler and then forwards the alert to the correct Alertmanager for the organization.

When Grafana is configured to send alerts to an external Alertmanager it does so via the sender which creates an abstraction over notification of alerts and discovery of external Alertmanagers in Prometheus. The sender receives alerts via the SendAlerts function and then passes them to Prometheus.

How does Alertmanager turn alerts into notifications?

Alertmanager receives alerts via the PutAlerts function. Each alert is validated and its annotations and labels are normalized, then the alerts are put in an in-memory structure. The dispatcher iterates over the alerts and matches it to a route in the configuration as explained here.

The alert is then matched to an alert group depending on the configuration in the route. The alert is then sent through a number of stages including silencing and inhibition (which is currently not supported) and at last the receiver which can include wait, de-duplication, retry.

What are notification channels?

Notification channels receive alerts and turn them into notifications and is often the last callback in the receiver after wait, de-duplication and retry.