grafana/pkg/services/ngalert/state
gotjosh dd502f22eb
Alerting: Fix alert flapping in the internal alertmanager (#38648)
* Alerting: Fix alert flapping in the alertmanager

fixes a bug that caused Alerts that are evaluated at low intervals (sub 1 minute), to flap in the Alertmanager.
Mostly due to a combination of `EndsAt` and resend delay.

The Alertmanager uses `EndsAt` as a heuristic to know whenever it should resolve a firing alert, in the case that it hasn't heard
back from the alert generation system.

Because grafana sent the alert with an `EndsAt` which is equal to the `For` of the alert itself,
and we had a hard-coded 1 minute re-send delay (only applicable to firing alerts) this meant that a firing alert would resolve in the Alertmanager before we re-notify that it still firing.

This commit, increases the `EndsAt` by 3x the the resend delay or alert interval (depending on which one is higher). The resendDelay has been decreased to 30 seconds.
2021-09-02 16:22:59 +01:00
..
cache_test.go Expand the value string in annotations and labels of alerts (#37051) 2021-07-22 15:20:44 +01:00
cache.go Alerting: automatically remove stale alerting states (#36767) 2021-07-26 18:12:04 +02:00
manager_test.go Alerting: Fix alert flapping in the internal alertmanager (#38648) 2021-09-02 16:22:59 +01:00
manager.go Alerting: Fix alert flapping in the internal alertmanager (#38648) 2021-09-02 16:22:59 +01:00
state_test.go Alerting: Fix alert flapping in the internal alertmanager (#38648) 2021-09-02 16:22:59 +01:00
state.go Alerting: Fix alert flapping in the internal alertmanager (#38648) 2021-09-02 16:22:59 +01:00