IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-01-15 19:22:34 -06:00
Code Issues Packages Projects Releases Wiki Activity

Alerting: Feedback on docs (#77068)

Browse Source
This commit is contained in:
George Robinson 2023-10-24 16:53:26 +01:00 committed by GitHub
parent 3899620234
commit 2c2d8bcc74
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/sources/alerting/fundamentals/high-availability/_index.md
View File

@ -25,13 +25,11 @@ Grafana Alerting uses the Prometheus model of separating the evaluation of alert
{{< figure src="/static/img/docs/alerting/unified/high-availability-ua.png" class="docs-image--no-shadow" max-width= "750px" caption="High availability" >}}
When running multiple instances of Grafana, the operational mode of the alert generator does not change. This means that all alert rules are evaluated on all instances of Grafana. You can think of the evaluation of alert rules as being duplicated. However, this is how Grafana Alerting makes sure that as long as at least one Grafana instance is working, alert rules will still be evaluated and notifications for alerts will still be sent. You will see this duplication in state history, and is a good way to tell if you are using high availability.
When running multiple instances of Grafana, all alert rules are evaluated on all instances. You can think of the evaluation of alert rules as being duplicated. This is how Grafana Alerting makes sure that as long as at least one Grafana instance is working, alert rules will still be evaluated and notifications for alerts will still be sent. You will see this duplication in state history, and is a good way to tell if you are using high availability.
While the alert generator evaluates all alert rules on all instances, Grafana makes a best-effort attempt to avoid sending duplicate notifications. Alertmanager chooses availability over consistency which means that in certain situations notifications can be duplicated or appear out-of-order. Alertmanager takes the opinion that duplicate or out-of-order notifications are better than no notifications, and so it uses a gossip protocol to share information about notifications between Grafana instances instead of a more-consistent but less-available protocol such as two-phase commit, or distributed-consensus protocols such as Raft or Paxos.
While the alert generator evaluates all alert rules on all instances, the alert receiver makes a best-effort attempt to avoid sending duplicate notifications. Alertmanager chooses availability over consistency, which may result in occasional duplicated or out-of-order notifications. It takes the opinion that duplicate or out-of-order notifications are better than no notifications.
The Alertmanager also gossips silences, which means a silence created on one Grafana instance is replicated to all other Grafana instances.
Both notifications and silences are persisted to the database periodically, and during graceful shut down.
The Alertmanager uses a gossip protocol to share information about notifications between Grafana instances. It also gossips silences, which means a silence created on one Grafana instance is replicated to all other Grafana instances. Both notifications and silences are persisted to the database periodically, and during graceful shut down.
It is important to make sure that gossiping is configured and tested. You can find the documentation on how to do that [here][configure-high-availability].