mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Merge pull request #12145 from grafana/alerting_reminder
Alerting notification reminder
This commit is contained in:
commit
1ceca5d8ec
@ -16,12 +16,11 @@ weight = 2
|
||||
|
||||
When an alert changes state, it sends out notifications. Each alert rule can have
|
||||
multiple notifications. In order to add a notification to an alert rule you first need
|
||||
to add and configure a `notification` channel (can be email, PagerDuty or other integration). This is done from the Notification Channels page.
|
||||
to add and configure a `notification` channel (can be email, PagerDuty or other integration).
|
||||
This is done from the Notification Channels page.
|
||||
|
||||
## Notification Channel Setup
|
||||
|
||||
{{< imgbox max-width="30%" img="/img/docs/v50/alerts_notifications_menu.png" caption="Alerting Notification Channels" >}}
|
||||
|
||||
On the Notification Channels page hit the `New Channel` button to go the page where you
|
||||
can configure and setup a new Notification Channel.
|
||||
|
||||
@ -30,7 +29,31 @@ sure it's setup correctly.
|
||||
|
||||
### Send on all alerts
|
||||
|
||||
When checked, this option will nofity for all alert rules - existing and new.
|
||||
When checked, this option will notify for all alert rules - existing and new.
|
||||
|
||||
### Send reminders
|
||||
|
||||
> Only available in Grafana v5.3 and above.
|
||||
|
||||
{{< docs-imagebox max-width="600px" img="/img/docs/v53/alerting_notification_reminders.png" class="docs-image--right" caption="Alerting notification reminders setup" >}}
|
||||
|
||||
When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders
|
||||
should be sent using number of seconds (s), minutes (m) or hours (h), for example `30s`, `3m`, `5m` or `1h` etc.
|
||||
|
||||
**Important:** Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured [alert rule evaluation interval](/alerting/rules/#name-evaluation-interval).
|
||||
|
||||
These examples show how often and when reminders are sent for a triggered alert.
|
||||
|
||||
Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
|
||||
---------- | ----------- | -----------
|
||||
`30s` | `15s` | ~30 seconds
|
||||
`1m` | `5m` | ~5 minutes
|
||||
`5m` | `15m` | ~15 minutes
|
||||
`6m` | `20m` | ~24 minutes
|
||||
`1h` | `15m` | ~1 hour
|
||||
`1h` | `2h` | ~2 hours
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
## Supported Notification Types
|
||||
|
||||
@ -132,23 +155,23 @@ Once these two properties are set, you can send the alerts to Kafka for further
|
||||
|
||||
### All supported notifiers
|
||||
|
||||
Name | Type |Support images
|
||||
-----|------------ | ------
|
||||
Slack | `slack` | yes
|
||||
Pagerduty | `pagerduty` | yes
|
||||
Email | `email` | yes
|
||||
Webhook | `webhook` | link
|
||||
Kafka | `kafka` | no
|
||||
Hipchat | `hipchat` | yes
|
||||
VictorOps | `victorops` | yes
|
||||
Sensu | `sensu` | yes
|
||||
OpsGenie | `opsgenie` | yes
|
||||
Threema | `threema` | yes
|
||||
Pushover | `pushover` | no
|
||||
Telegram | `telegram` | no
|
||||
Line | `line` | no
|
||||
Prometheus Alertmanager | `prometheus-alertmanager` | no
|
||||
Microsoft Teams | `teams` | yes
|
||||
Name | Type |Support images | Support reminders
|
||||
-----|------------ | ------ | ------ |
|
||||
Slack | `slack` | yes | yes
|
||||
Pagerduty | `pagerduty` | yes | yes
|
||||
Email | `email` | yes | yes
|
||||
Webhook | `webhook` | link | yes
|
||||
Kafka | `kafka` | no | yes
|
||||
Hipchat | `hipchat` | yes | yes
|
||||
VictorOps | `victorops` | yes | yes
|
||||
Sensu | `sensu` | yes | yes
|
||||
OpsGenie | `opsgenie` | yes | yes
|
||||
Threema | `threema` | yes | yes
|
||||
Pushover | `pushover` | no | yes
|
||||
Telegram | `telegram` | no | yes
|
||||
Line | `line` | no | yes
|
||||
Microsoft Teams | `teams` | yes | yes
|
||||
Prometheus Alertmanager | `prometheus-alertmanager` | no | no
|
||||
|
||||
|
||||
|
||||
|
@ -88,6 +88,11 @@ So as you can see from the above scenario Grafana will not send out notification
|
||||
to fire if the rule already is in state `Alerting`. To improve support for queries that return multiple series
|
||||
we plan to track state **per series** in a future release.
|
||||
|
||||
> Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications
|
||||
> when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will
|
||||
> be included in the reminder notification. Depending on what notification channel you're using you may be able to take advantage
|
||||
> of this feature for identifying new/existing series causing alert to fire. [Read more about notification reminders here](/alerting/notifications/#send-reminders).
|
||||
|
||||
### No Data / Null values
|
||||
|
||||
Below your conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.
|
||||
|
@ -50,6 +50,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
@ -86,6 +87,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"id": 1,
|
||||
"dashboardId": 1,
|
||||
@ -146,6 +148,7 @@ JSON Body Schema:
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"alertId": 1,
|
||||
"state": "Paused",
|
||||
@ -177,6 +180,7 @@ JSON Body Schema:
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"state": "Paused",
|
||||
"message": "alert paused",
|
||||
@ -204,14 +208,21 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"id": 1,
|
||||
"name": "Team A",
|
||||
"type": "email",
|
||||
"isDefault": true,
|
||||
"created": "2017-01-01 12:45",
|
||||
"updated": "2017-01-01 12:45"
|
||||
}
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"name": "Team A",
|
||||
"type": "email",
|
||||
"isDefault": false,
|
||||
"sendReminder": false,
|
||||
"settings": {
|
||||
"addresses": "carl@grafana.com;dev@grafana.com"
|
||||
},
|
||||
"created": "2018-04-23T14:44:09+02:00",
|
||||
"updated": "2018-08-20T15:47:49+02:00"
|
||||
}
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
## Create alert notification
|
||||
@ -232,6 +243,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
"name": "new alert notification", //Required
|
||||
"type": "email", //Required
|
||||
"isDefault": false,
|
||||
"sendReminder": false,
|
||||
"settings": {
|
||||
"addresses": "carl@grafana.com;dev@grafana.com"
|
||||
}
|
||||
@ -243,14 +255,18 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"id": 1,
|
||||
"name": "new alert notification",
|
||||
"type": "email",
|
||||
"isDefault": false,
|
||||
"settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
|
||||
"created": "2017-01-01 12:34",
|
||||
"updated": "2017-01-01 12:34"
|
||||
"sendReminder": false,
|
||||
"settings": {
|
||||
"addresses": "carl@grafana.com;dev@grafana.com"
|
||||
},
|
||||
"created": "2018-04-23T14:44:09+02:00",
|
||||
"updated": "2018-08-20T15:47:49+02:00"
|
||||
}
|
||||
```
|
||||
|
||||
@ -271,6 +287,8 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
"name": "new alert notification", //Required
|
||||
"type": "email", //Required
|
||||
"isDefault": false,
|
||||
"sendReminder": true,
|
||||
"frequency": "15m",
|
||||
"settings": {
|
||||
"addresses: "carl@grafana.com;dev@grafana.com"
|
||||
}
|
||||
@ -282,12 +300,17 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"id": 1,
|
||||
"name": "new alert notification",
|
||||
"type": "email",
|
||||
"isDefault": false,
|
||||
"settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
|
||||
"sendReminder": true,
|
||||
"frequency": "15m",
|
||||
"settings": {
|
||||
"addresses": "carl@grafana.com;dev@grafana.com"
|
||||
},
|
||||
"created": "2017-01-01 12:34",
|
||||
"updated": "2017-01-01 12:34"
|
||||
}
|
||||
@ -311,6 +334,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
|
||||
```http
|
||||
HTTP/1.1 200
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"message": "Notification deleted"
|
||||
}
|
||||
|
@ -192,14 +192,7 @@ func GetAlertNotifications(c *m.ReqContext) Response {
|
||||
result := make([]*dtos.AlertNotification, 0)
|
||||
|
||||
for _, notification := range query.Result {
|
||||
result = append(result, &dtos.AlertNotification{
|
||||
Id: notification.Id,
|
||||
Name: notification.Name,
|
||||
Type: notification.Type,
|
||||
IsDefault: notification.IsDefault,
|
||||
Created: notification.Created,
|
||||
Updated: notification.Updated,
|
||||
})
|
||||
result = append(result, dtos.NewAlertNotification(notification))
|
||||
}
|
||||
|
||||
return JSON(200, result)
|
||||
@ -215,7 +208,7 @@ func GetAlertNotificationByID(c *m.ReqContext) Response {
|
||||
return Error(500, "Failed to get alert notifications", err)
|
||||
}
|
||||
|
||||
return JSON(200, query.Result)
|
||||
return JSON(200, dtos.NewAlertNotification(query.Result))
|
||||
}
|
||||
|
||||
func CreateAlertNotification(c *m.ReqContext, cmd m.CreateAlertNotificationCommand) Response {
|
||||
@ -225,7 +218,7 @@ func CreateAlertNotification(c *m.ReqContext, cmd m.CreateAlertNotificationComma
|
||||
return Error(500, "Failed to create alert notification", err)
|
||||
}
|
||||
|
||||
return JSON(200, cmd.Result)
|
||||
return JSON(200, dtos.NewAlertNotification(cmd.Result))
|
||||
}
|
||||
|
||||
func UpdateAlertNotification(c *m.ReqContext, cmd m.UpdateAlertNotificationCommand) Response {
|
||||
@ -235,7 +228,7 @@ func UpdateAlertNotification(c *m.ReqContext, cmd m.UpdateAlertNotificationComma
|
||||
return Error(500, "Failed to update alert notification", err)
|
||||
}
|
||||
|
||||
return JSON(200, cmd.Result)
|
||||
return JSON(200, dtos.NewAlertNotification(cmd.Result))
|
||||
}
|
||||
|
||||
func DeleteAlertNotification(c *m.ReqContext) Response {
|
||||
|
@ -1,35 +1,76 @@
|
||||
package dtos
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/components/null"
|
||||
"github.com/grafana/grafana/pkg/components/simplejson"
|
||||
m "github.com/grafana/grafana/pkg/models"
|
||||
"github.com/grafana/grafana/pkg/models"
|
||||
)
|
||||
|
||||
type AlertRule struct {
|
||||
Id int64 `json:"id"`
|
||||
DashboardId int64 `json:"dashboardId"`
|
||||
PanelId int64 `json:"panelId"`
|
||||
Name string `json:"name"`
|
||||
Message string `json:"message"`
|
||||
State m.AlertStateType `json:"state"`
|
||||
NewStateDate time.Time `json:"newStateDate"`
|
||||
EvalDate time.Time `json:"evalDate"`
|
||||
EvalData *simplejson.Json `json:"evalData"`
|
||||
ExecutionError string `json:"executionError"`
|
||||
Url string `json:"url"`
|
||||
CanEdit bool `json:"canEdit"`
|
||||
Id int64 `json:"id"`
|
||||
DashboardId int64 `json:"dashboardId"`
|
||||
PanelId int64 `json:"panelId"`
|
||||
Name string `json:"name"`
|
||||
Message string `json:"message"`
|
||||
State models.AlertStateType `json:"state"`
|
||||
NewStateDate time.Time `json:"newStateDate"`
|
||||
EvalDate time.Time `json:"evalDate"`
|
||||
EvalData *simplejson.Json `json:"evalData"`
|
||||
ExecutionError string `json:"executionError"`
|
||||
Url string `json:"url"`
|
||||
CanEdit bool `json:"canEdit"`
|
||||
}
|
||||
|
||||
func formatShort(interval time.Duration) string {
|
||||
var result string
|
||||
|
||||
hours := interval / time.Hour
|
||||
if hours > 0 {
|
||||
result += fmt.Sprintf("%dh", hours)
|
||||
}
|
||||
|
||||
remaining := interval - (hours * time.Hour)
|
||||
mins := remaining / time.Minute
|
||||
if mins > 0 {
|
||||
result += fmt.Sprintf("%dm", mins)
|
||||
}
|
||||
|
||||
remaining = remaining - (mins * time.Minute)
|
||||
seconds := remaining / time.Second
|
||||
if seconds > 0 {
|
||||
result += fmt.Sprintf("%ds", seconds)
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
func NewAlertNotification(notification *models.AlertNotification) *AlertNotification {
|
||||
return &AlertNotification{
|
||||
Id: notification.Id,
|
||||
Name: notification.Name,
|
||||
Type: notification.Type,
|
||||
IsDefault: notification.IsDefault,
|
||||
Created: notification.Created,
|
||||
Updated: notification.Updated,
|
||||
Frequency: formatShort(notification.Frequency),
|
||||
SendReminder: notification.SendReminder,
|
||||
Settings: notification.Settings,
|
||||
}
|
||||
}
|
||||
|
||||
type AlertNotification struct {
|
||||
Id int64 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Created time.Time `json:"created"`
|
||||
Updated time.Time `json:"updated"`
|
||||
Id int64 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
SendReminder bool `json:"sendReminder"`
|
||||
Frequency string `json:"frequency"`
|
||||
Created time.Time `json:"created"`
|
||||
Updated time.Time `json:"updated"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
}
|
||||
|
||||
type AlertTestCommand struct {
|
||||
@ -39,7 +80,7 @@ type AlertTestCommand struct {
|
||||
|
||||
type AlertTestResult struct {
|
||||
Firing bool `json:"firing"`
|
||||
State m.AlertStateType `json:"state"`
|
||||
State models.AlertStateType `json:"state"`
|
||||
ConditionEvals string `json:"conditionEvals"`
|
||||
TimeMs string `json:"timeMs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
@ -59,9 +100,11 @@ type EvalMatch struct {
|
||||
}
|
||||
|
||||
type NotificationTestCommand struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
SendReminder bool `json:"sendReminder"`
|
||||
Frequency string `json:"frequency"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
}
|
||||
|
||||
type PauseAlertCommand struct {
|
||||
|
35
pkg/api/dtos/alerting_test.go
Normal file
35
pkg/api/dtos/alerting_test.go
Normal file
@ -0,0 +1,35 @@
|
||||
package dtos
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFormatShort(t *testing.T) {
|
||||
tcs := []struct {
|
||||
interval time.Duration
|
||||
expected string
|
||||
}{
|
||||
{interval: time.Hour, expected: "1h"},
|
||||
{interval: time.Hour + time.Minute, expected: "1h1m"},
|
||||
{interval: (time.Hour * 10) + time.Minute, expected: "10h1m"},
|
||||
{interval: (time.Hour * 10) + (time.Minute * 10) + time.Second, expected: "10h10m1s"},
|
||||
{interval: time.Minute * 10, expected: "10m"},
|
||||
}
|
||||
|
||||
for _, tc := range tcs {
|
||||
got := formatShort(tc.interval)
|
||||
if got != tc.expected {
|
||||
t.Errorf("expected %s got %s interval: %v", tc.expected, got, tc.interval)
|
||||
}
|
||||
|
||||
parsed, err := time.ParseDuration(tc.expected)
|
||||
if err != nil {
|
||||
t.Fatalf("could not parse expected duration")
|
||||
}
|
||||
|
||||
if parsed != tc.interval {
|
||||
t.Errorf("expectes the parsed duration to equal the interval. Got %v expected: %v", parsed, tc.interval)
|
||||
}
|
||||
}
|
||||
}
|
@ -1,38 +1,50 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/components/simplejson"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrNotificationFrequencyNotFound = errors.New("Notification frequency not specified")
|
||||
ErrJournalingNotFound = errors.New("alert notification journaling not found")
|
||||
)
|
||||
|
||||
type AlertNotification struct {
|
||||
Id int64 `json:"id"`
|
||||
OrgId int64 `json:"-"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
Created time.Time `json:"created"`
|
||||
Updated time.Time `json:"updated"`
|
||||
Id int64 `json:"id"`
|
||||
OrgId int64 `json:"-"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
SendReminder bool `json:"sendReminder"`
|
||||
Frequency time.Duration `json:"frequency"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
Created time.Time `json:"created"`
|
||||
Updated time.Time `json:"updated"`
|
||||
}
|
||||
|
||||
type CreateAlertNotificationCommand struct {
|
||||
Name string `json:"name" binding:"Required"`
|
||||
Type string `json:"type" binding:"Required"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
Name string `json:"name" binding:"Required"`
|
||||
Type string `json:"type" binding:"Required"`
|
||||
SendReminder bool `json:"sendReminder"`
|
||||
Frequency string `json:"frequency"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings"`
|
||||
|
||||
OrgId int64 `json:"-"`
|
||||
Result *AlertNotification
|
||||
}
|
||||
|
||||
type UpdateAlertNotificationCommand struct {
|
||||
Id int64 `json:"id" binding:"Required"`
|
||||
Name string `json:"name" binding:"Required"`
|
||||
Type string `json:"type" binding:"Required"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings" binding:"Required"`
|
||||
Id int64 `json:"id" binding:"Required"`
|
||||
Name string `json:"name" binding:"Required"`
|
||||
Type string `json:"type" binding:"Required"`
|
||||
SendReminder bool `json:"sendReminder"`
|
||||
Frequency string `json:"frequency"`
|
||||
IsDefault bool `json:"isDefault"`
|
||||
Settings *simplejson.Json `json:"settings" binding:"Required"`
|
||||
|
||||
OrgId int64 `json:"-"`
|
||||
Result *AlertNotification
|
||||
@ -63,3 +75,34 @@ type GetAllAlertNotificationsQuery struct {
|
||||
|
||||
Result []*AlertNotification
|
||||
}
|
||||
|
||||
type AlertNotificationJournal struct {
|
||||
Id int64
|
||||
OrgId int64
|
||||
AlertId int64
|
||||
NotifierId int64
|
||||
SentAt int64
|
||||
Success bool
|
||||
}
|
||||
|
||||
type RecordNotificationJournalCommand struct {
|
||||
OrgId int64
|
||||
AlertId int64
|
||||
NotifierId int64
|
||||
SentAt int64
|
||||
Success bool
|
||||
}
|
||||
|
||||
type GetLatestNotificationQuery struct {
|
||||
OrgId int64
|
||||
AlertId int64
|
||||
NotifierId int64
|
||||
|
||||
Result *AlertNotificationJournal
|
||||
}
|
||||
|
||||
type CleanNotificationJournalCommand struct {
|
||||
OrgId int64
|
||||
AlertId int64
|
||||
NotifierId int64
|
||||
}
|
||||
|
@ -1,6 +1,9 @@
|
||||
package alerting
|
||||
|
||||
import "time"
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
type EvalHandler interface {
|
||||
Eval(evalContext *EvalContext)
|
||||
@ -15,10 +18,14 @@ type Notifier interface {
|
||||
Notify(evalContext *EvalContext) error
|
||||
GetType() string
|
||||
NeedsImage() bool
|
||||
ShouldNotify(evalContext *EvalContext) bool
|
||||
|
||||
// ShouldNotify checks this evaluation should send an alert notification
|
||||
ShouldNotify(ctx context.Context, evalContext *EvalContext) bool
|
||||
|
||||
GetNotifierId() int64
|
||||
GetIsDefault() bool
|
||||
GetSendReminder() bool
|
||||
GetFrequency() time.Duration
|
||||
}
|
||||
|
||||
type NotifierSlice []Notifier
|
||||
|
@ -1,10 +1,10 @@
|
||||
package alerting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"golang.org/x/sync/errgroup"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/bus"
|
||||
"github.com/grafana/grafana/pkg/components/imguploader"
|
||||
@ -58,17 +58,47 @@ func (n *notificationService) SendIfNeeded(context *EvalContext) error {
|
||||
return n.sendNotifications(context, notifiers)
|
||||
}
|
||||
|
||||
func (n *notificationService) sendNotifications(context *EvalContext, notifiers []Notifier) error {
|
||||
g, _ := errgroup.WithContext(context.Ctx)
|
||||
|
||||
func (n *notificationService) sendNotifications(evalContext *EvalContext, notifiers []Notifier) error {
|
||||
for _, notifier := range notifiers {
|
||||
not := notifier //avoid updating scope variable in go routine
|
||||
n.log.Debug("Sending notification", "type", not.GetType(), "id", not.GetNotifierId(), "isDefault", not.GetIsDefault())
|
||||
metrics.M_Alerting_Notification_Sent.WithLabelValues(not.GetType()).Inc()
|
||||
g.Go(func() error { return not.Notify(context) })
|
||||
not := notifier
|
||||
|
||||
err := bus.InTransaction(evalContext.Ctx, func(ctx context.Context) error {
|
||||
n.log.Debug("trying to send notification", "id", not.GetNotifierId())
|
||||
|
||||
// Verify that we can send the notification again
|
||||
// but this time within the same transaction.
|
||||
if !evalContext.IsTestRun && !not.ShouldNotify(context.Background(), evalContext) {
|
||||
return nil
|
||||
}
|
||||
|
||||
n.log.Debug("Sending notification", "type", not.GetType(), "id", not.GetNotifierId(), "isDefault", not.GetIsDefault())
|
||||
metrics.M_Alerting_Notification_Sent.WithLabelValues(not.GetType()).Inc()
|
||||
|
||||
//send notification
|
||||
success := not.Notify(evalContext) == nil
|
||||
|
||||
if evalContext.IsTestRun {
|
||||
return nil
|
||||
}
|
||||
|
||||
//write result to db.
|
||||
cmd := &m.RecordNotificationJournalCommand{
|
||||
OrgId: evalContext.Rule.OrgId,
|
||||
AlertId: evalContext.Rule.Id,
|
||||
NotifierId: not.GetNotifierId(),
|
||||
SentAt: time.Now().Unix(),
|
||||
Success: success,
|
||||
}
|
||||
|
||||
return bus.DispatchCtx(ctx, cmd)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
n.log.Error("failed to send notification", "id", not.GetNotifierId())
|
||||
}
|
||||
}
|
||||
|
||||
return g.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *notificationService) uploadImage(context *EvalContext) (err error) {
|
||||
@ -110,7 +140,7 @@ func (n *notificationService) uploadImage(context *EvalContext) (err error) {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []int64, context *EvalContext) (NotifierSlice, error) {
|
||||
func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []int64, evalContext *EvalContext) (NotifierSlice, error) {
|
||||
query := &m.GetAlertNotificationsToSendQuery{OrgId: orgId, Ids: notificationIds}
|
||||
|
||||
if err := bus.Dispatch(query); err != nil {
|
||||
@ -123,7 +153,8 @@ func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if not.ShouldNotify(context) {
|
||||
|
||||
if not.ShouldNotify(evalContext.Ctx, evalContext) {
|
||||
result = append(result, not)
|
||||
}
|
||||
}
|
||||
|
@ -1,6 +1,7 @@
|
||||
package notifiers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/bus"
|
||||
@ -33,7 +34,7 @@ func NewAlertmanagerNotifier(model *m.AlertNotification) (alerting.Notifier, err
|
||||
}
|
||||
|
||||
return &AlertmanagerNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
log: log.New("alerting.notifier.prometheus-alertmanager"),
|
||||
}, nil
|
||||
@ -45,7 +46,7 @@ type AlertmanagerNotifier struct {
|
||||
log log.Logger
|
||||
}
|
||||
|
||||
func (this *AlertmanagerNotifier) ShouldNotify(evalContext *alerting.EvalContext) bool {
|
||||
func (this *AlertmanagerNotifier) ShouldNotify(ctx context.Context, evalContext *alerting.EvalContext) bool {
|
||||
this.log.Debug("Should notify", "ruleId", evalContext.Rule.Id, "state", evalContext.Rule.State, "previousState", evalContext.PrevAlertState)
|
||||
|
||||
// Do not notify when we become OK for the first time.
|
||||
|
@ -1,50 +1,94 @@
|
||||
package notifiers
|
||||
|
||||
import (
|
||||
"github.com/grafana/grafana/pkg/components/simplejson"
|
||||
m "github.com/grafana/grafana/pkg/models"
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/bus"
|
||||
"github.com/grafana/grafana/pkg/log"
|
||||
"github.com/grafana/grafana/pkg/models"
|
||||
|
||||
"github.com/grafana/grafana/pkg/services/alerting"
|
||||
)
|
||||
|
||||
type NotifierBase struct {
|
||||
Name string
|
||||
Type string
|
||||
Id int64
|
||||
IsDeault bool
|
||||
UploadImage bool
|
||||
Name string
|
||||
Type string
|
||||
Id int64
|
||||
IsDeault bool
|
||||
UploadImage bool
|
||||
SendReminder bool
|
||||
Frequency time.Duration
|
||||
|
||||
log log.Logger
|
||||
}
|
||||
|
||||
func NewNotifierBase(id int64, isDefault bool, name, notifierType string, model *simplejson.Json) NotifierBase {
|
||||
func NewNotifierBase(model *models.AlertNotification) NotifierBase {
|
||||
uploadImage := true
|
||||
value, exist := model.CheckGet("uploadImage")
|
||||
value, exist := model.Settings.CheckGet("uploadImage")
|
||||
if exist {
|
||||
uploadImage = value.MustBool()
|
||||
}
|
||||
|
||||
return NotifierBase{
|
||||
Id: id,
|
||||
Name: name,
|
||||
IsDeault: isDefault,
|
||||
Type: notifierType,
|
||||
UploadImage: uploadImage,
|
||||
Id: model.Id,
|
||||
Name: model.Name,
|
||||
IsDeault: model.IsDefault,
|
||||
Type: model.Type,
|
||||
UploadImage: uploadImage,
|
||||
SendReminder: model.SendReminder,
|
||||
Frequency: model.Frequency,
|
||||
log: log.New("alerting.notifier." + model.Name),
|
||||
}
|
||||
}
|
||||
|
||||
func defaultShouldNotify(context *alerting.EvalContext) bool {
|
||||
func defaultShouldNotify(context *alerting.EvalContext, sendReminder bool, frequency time.Duration, lastNotify time.Time) bool {
|
||||
// Only notify on state change.
|
||||
if context.PrevAlertState == context.Rule.State {
|
||||
if context.PrevAlertState == context.Rule.State && !sendReminder {
|
||||
return false
|
||||
}
|
||||
|
||||
// Do not notify if interval has not elapsed
|
||||
if sendReminder && !lastNotify.IsZero() && lastNotify.Add(frequency).After(time.Now()) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Do not notify if alert state if OK or pending even on repeated notify
|
||||
if sendReminder && (context.Rule.State == models.AlertStateOK || context.Rule.State == models.AlertStatePending) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Do not notify when we become OK for the first time.
|
||||
if (context.PrevAlertState == m.AlertStatePending) && (context.Rule.State == m.AlertStateOK) {
|
||||
if (context.PrevAlertState == models.AlertStatePending) && (context.Rule.State == models.AlertStateOK) {
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
func (n *NotifierBase) ShouldNotify(context *alerting.EvalContext) bool {
|
||||
return defaultShouldNotify(context)
|
||||
// ShouldNotify checks this evaluation should send an alert notification
|
||||
func (n *NotifierBase) ShouldNotify(ctx context.Context, c *alerting.EvalContext) bool {
|
||||
cmd := &models.GetLatestNotificationQuery{
|
||||
OrgId: c.Rule.OrgId,
|
||||
AlertId: c.Rule.Id,
|
||||
NotifierId: n.Id,
|
||||
}
|
||||
|
||||
err := bus.DispatchCtx(ctx, cmd)
|
||||
if err == models.ErrJournalingNotFound {
|
||||
return true
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
n.log.Error("Could not determine last time alert notifier fired", "Alert name", c.Rule.Name, "Error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
if !cmd.Result.Success {
|
||||
return true
|
||||
}
|
||||
|
||||
return defaultShouldNotify(c, n.SendReminder, n.Frequency, time.Unix(cmd.Result.SentAt, 0))
|
||||
}
|
||||
|
||||
func (n *NotifierBase) GetType() string {
|
||||
@ -62,3 +106,11 @@ func (n *NotifierBase) GetNotifierId() int64 {
|
||||
func (n *NotifierBase) GetIsDefault() bool {
|
||||
return n.IsDeault
|
||||
}
|
||||
|
||||
func (n *NotifierBase) GetSendReminder() bool {
|
||||
return n.SendReminder
|
||||
}
|
||||
|
||||
func (n *NotifierBase) GetFrequency() time.Duration {
|
||||
return n.Frequency
|
||||
}
|
||||
|
@ -2,7 +2,11 @@ package notifiers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/bus"
|
||||
|
||||
"github.com/grafana/grafana/pkg/components/simplejson"
|
||||
m "github.com/grafana/grafana/pkg/models"
|
||||
@ -10,47 +14,129 @@ import (
|
||||
. "github.com/smartystreets/goconvey/convey"
|
||||
)
|
||||
|
||||
func TestBaseNotifier(t *testing.T) {
|
||||
Convey("Base notifier tests", t, func() {
|
||||
Convey("default constructor for notifiers", func() {
|
||||
bJson := simplejson.New()
|
||||
func TestShouldSendAlertNotification(t *testing.T) {
|
||||
tcs := []struct {
|
||||
name string
|
||||
prevState m.AlertStateType
|
||||
newState m.AlertStateType
|
||||
expected bool
|
||||
sendReminder bool
|
||||
}{
|
||||
{
|
||||
name: "pending -> ok should not trigger an notification",
|
||||
newState: m.AlertStatePending,
|
||||
prevState: m.AlertStateOK,
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "ok -> alerting should trigger an notification",
|
||||
newState: m.AlertStateOK,
|
||||
prevState: m.AlertStateAlerting,
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "ok -> pending should not trigger an notification",
|
||||
newState: m.AlertStateOK,
|
||||
prevState: m.AlertStatePending,
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "ok -> ok should not trigger an notification",
|
||||
newState: m.AlertStateOK,
|
||||
prevState: m.AlertStateOK,
|
||||
expected: false,
|
||||
sendReminder: false,
|
||||
},
|
||||
{
|
||||
name: "ok -> alerting should not trigger an notification",
|
||||
newState: m.AlertStateOK,
|
||||
prevState: m.AlertStateAlerting,
|
||||
expected: true,
|
||||
sendReminder: true,
|
||||
},
|
||||
{
|
||||
name: "ok -> ok with reminder should not trigger an notification",
|
||||
newState: m.AlertStateOK,
|
||||
prevState: m.AlertStateOK,
|
||||
expected: false,
|
||||
sendReminder: true,
|
||||
},
|
||||
}
|
||||
|
||||
Convey("can parse false value", func() {
|
||||
bJson.Set("uploadImage", false)
|
||||
|
||||
base := NewNotifierBase(1, false, "name", "email", bJson)
|
||||
So(base.UploadImage, ShouldBeFalse)
|
||||
})
|
||||
|
||||
Convey("can parse true value", func() {
|
||||
bJson.Set("uploadImage", true)
|
||||
|
||||
base := NewNotifierBase(1, false, "name", "email", bJson)
|
||||
So(base.UploadImage, ShouldBeTrue)
|
||||
})
|
||||
|
||||
Convey("default value should be true for backwards compatibility", func() {
|
||||
base := NewNotifierBase(1, false, "name", "email", bJson)
|
||||
So(base.UploadImage, ShouldBeTrue)
|
||||
})
|
||||
for _, tc := range tcs {
|
||||
evalContext := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
|
||||
State: tc.newState,
|
||||
})
|
||||
|
||||
Convey("should notify", func() {
|
||||
Convey("pending -> ok", func() {
|
||||
context := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
|
||||
State: m.AlertStatePending,
|
||||
})
|
||||
context.Rule.State = m.AlertStateOK
|
||||
So(defaultShouldNotify(context), ShouldBeFalse)
|
||||
evalContext.Rule.State = tc.prevState
|
||||
if defaultShouldNotify(evalContext, true, 0, time.Now()) != tc.expected {
|
||||
t.Errorf("failed %s. expected %+v to return %v", tc.name, tc, tc.expected)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShouldNotifyWhenNoJournalingIsFound(t *testing.T) {
|
||||
Convey("base notifier", t, func() {
|
||||
bus.ClearBusHandlers()
|
||||
|
||||
notifier := NewNotifierBase(&m.AlertNotification{
|
||||
Id: 1,
|
||||
Name: "name",
|
||||
Type: "email",
|
||||
Settings: simplejson.New(),
|
||||
})
|
||||
evalContext := alerting.NewEvalContext(context.TODO(), &alerting.Rule{})
|
||||
|
||||
Convey("should notify if no journaling is found", func() {
|
||||
bus.AddHandlerCtx("", func(ctx context.Context, q *m.GetLatestNotificationQuery) error {
|
||||
return m.ErrJournalingNotFound
|
||||
})
|
||||
|
||||
Convey("ok -> alerting", func() {
|
||||
context := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
|
||||
State: m.AlertStateOK,
|
||||
})
|
||||
context.Rule.State = m.AlertStateAlerting
|
||||
So(defaultShouldNotify(context), ShouldBeTrue)
|
||||
if !notifier.ShouldNotify(context.Background(), evalContext) {
|
||||
t.Errorf("should send notifications when ErrJournalingNotFound is returned")
|
||||
}
|
||||
})
|
||||
|
||||
Convey("should not notify query returns error", func() {
|
||||
bus.AddHandlerCtx("", func(ctx context.Context, q *m.GetLatestNotificationQuery) error {
|
||||
return errors.New("some kind of error unknown error")
|
||||
})
|
||||
|
||||
if notifier.ShouldNotify(context.Background(), evalContext) {
|
||||
t.Errorf("should not send notifications when query returns error")
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func TestBaseNotifier(t *testing.T) {
|
||||
Convey("default constructor for notifiers", t, func() {
|
||||
bJson := simplejson.New()
|
||||
|
||||
model := &m.AlertNotification{
|
||||
Id: 1,
|
||||
Name: "name",
|
||||
Type: "email",
|
||||
Settings: bJson,
|
||||
}
|
||||
|
||||
Convey("can parse false value", func() {
|
||||
bJson.Set("uploadImage", false)
|
||||
|
||||
base := NewNotifierBase(model)
|
||||
So(base.UploadImage, ShouldBeFalse)
|
||||
})
|
||||
|
||||
Convey("can parse true value", func() {
|
||||
bJson.Set("uploadImage", true)
|
||||
|
||||
base := NewNotifierBase(model)
|
||||
So(base.UploadImage, ShouldBeTrue)
|
||||
})
|
||||
|
||||
Convey("default value should be true for backwards compatibility", func() {
|
||||
base := NewNotifierBase(model)
|
||||
So(base.UploadImage, ShouldBeTrue)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
@ -32,7 +32,7 @@ func NewDingDingNotifier(model *m.AlertNotification) (alerting.Notifier, error)
|
||||
}
|
||||
|
||||
return &DingDingNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
log: log.New("alerting.notifier.dingding"),
|
||||
}, nil
|
||||
|
@ -39,7 +39,7 @@ func NewDiscordNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &DiscordNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
WebhookURL: url,
|
||||
log: log.New("alerting.notifier.discord"),
|
||||
}, nil
|
||||
|
@ -52,7 +52,7 @@ func NewEmailNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
})
|
||||
|
||||
return &EmailNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Addresses: addresses,
|
||||
log: log.New("alerting.notifier.email"),
|
||||
}, nil
|
||||
|
@ -59,7 +59,7 @@ func NewHipChatNotifier(model *models.AlertNotification) (alerting.Notifier, err
|
||||
roomId := model.Settings.Get("roomid").MustString()
|
||||
|
||||
return &HipChatNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
ApiKey: apikey,
|
||||
RoomId: roomId,
|
||||
|
@ -43,7 +43,7 @@ func NewKafkaNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &KafkaNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Endpoint: endpoint,
|
||||
Topic: topic,
|
||||
log: log.New("alerting.notifier.kafka"),
|
||||
|
@ -39,7 +39,7 @@ func NewLINENotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &LineNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Token: token,
|
||||
log: log.New("alerting.notifier.line"),
|
||||
}, nil
|
||||
|
@ -56,7 +56,7 @@ func NewOpsGenieNotifier(model *m.AlertNotification) (alerting.Notifier, error)
|
||||
}
|
||||
|
||||
return &OpsGenieNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
ApiKey: apiKey,
|
||||
ApiUrl: apiUrl,
|
||||
AutoClose: autoClose,
|
||||
|
@ -51,7 +51,7 @@ func NewPagerdutyNotifier(model *m.AlertNotification) (alerting.Notifier, error)
|
||||
}
|
||||
|
||||
return &PagerdutyNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Key: key,
|
||||
AutoResolve: autoResolve,
|
||||
log: log.New("alerting.notifier.pagerduty"),
|
||||
|
@ -99,7 +99,7 @@ func NewPushoverNotifier(model *m.AlertNotification) (alerting.Notifier, error)
|
||||
return nil, alerting.ValidationError{Reason: "API token not given"}
|
||||
}
|
||||
return &PushoverNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
UserKey: userKey,
|
||||
ApiToken: apiToken,
|
||||
Priority: priority,
|
||||
|
@ -51,7 +51,7 @@ func NewSensuNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &SensuNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
User: model.Settings.Get("username").MustString(),
|
||||
Source: model.Settings.Get("source").MustString(),
|
||||
|
@ -78,7 +78,7 @@ func NewSlackNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
uploadImage := model.Settings.Get("uploadImage").MustBool(true)
|
||||
|
||||
return &SlackNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
Recipient: recipient,
|
||||
Mention: mention,
|
||||
|
@ -33,7 +33,7 @@ func NewTeamsNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &TeamsNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
log: log.New("alerting.notifier.teams"),
|
||||
}, nil
|
||||
|
@ -78,7 +78,7 @@ func NewTelegramNotifier(model *m.AlertNotification) (alerting.Notifier, error)
|
||||
}
|
||||
|
||||
return &TelegramNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
BotToken: botToken,
|
||||
ChatID: chatId,
|
||||
UploadImage: uploadImage,
|
||||
|
@ -106,7 +106,7 @@ func NewThreemaNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &ThreemaNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
GatewayID: gatewayID,
|
||||
RecipientID: recipientID,
|
||||
APISecret: apiSecret,
|
||||
|
@ -51,7 +51,7 @@ func NewVictoropsNotifier(model *models.AlertNotification) (alerting.Notifier, e
|
||||
}
|
||||
|
||||
return &VictoropsNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
URL: url,
|
||||
AutoResolve: autoResolve,
|
||||
log: log.New("alerting.notifier.victorops"),
|
||||
|
@ -47,7 +47,7 @@ func NewWebHookNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
|
||||
}
|
||||
|
||||
return &WebhookNotifier{
|
||||
NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
|
||||
NotifierBase: NewNotifierBase(model),
|
||||
Url: url,
|
||||
User: model.Settings.Get("username").MustString(),
|
||||
Password: model.Settings.Get("password").MustString(),
|
||||
|
@ -88,6 +88,18 @@ func (handler *DefaultResultHandler) Handle(evalContext *EvalContext) error {
|
||||
}
|
||||
}
|
||||
|
||||
if evalContext.Rule.State == m.AlertStateOK && evalContext.PrevAlertState != m.AlertStateOK {
|
||||
for _, notifierId := range evalContext.Rule.Notifications {
|
||||
cmd := &m.CleanNotificationJournalCommand{
|
||||
AlertId: evalContext.Rule.Id,
|
||||
NotifierId: notifierId,
|
||||
OrgId: evalContext.Rule.OrgId,
|
||||
}
|
||||
if err := bus.DispatchCtx(evalContext.Ctx, cmd); err != nil {
|
||||
handler.log.Error("Failed to clean up old notification records", "notifier", notifierId, "alert", evalContext.Rule.Id, "Error", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
handler.notifier.SendIfNeeded(evalContext)
|
||||
|
||||
return nil
|
||||
|
@ -2,6 +2,7 @@ package sqlstore
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
@ -17,6 +18,9 @@ func init() {
|
||||
bus.AddHandler("sql", DeleteAlertNotification)
|
||||
bus.AddHandler("sql", GetAlertNotificationsToSend)
|
||||
bus.AddHandler("sql", GetAllAlertNotifications)
|
||||
bus.AddHandlerCtx("sql", RecordNotificationJournal)
|
||||
bus.AddHandlerCtx("sql", GetLatestNotification)
|
||||
bus.AddHandlerCtx("sql", CleanNotificationJournal)
|
||||
}
|
||||
|
||||
func DeleteAlertNotification(cmd *m.DeleteAlertNotificationCommand) error {
|
||||
@ -53,7 +57,9 @@ func GetAlertNotificationsToSend(query *m.GetAlertNotificationsToSendQuery) erro
|
||||
alert_notification.created,
|
||||
alert_notification.updated,
|
||||
alert_notification.settings,
|
||||
alert_notification.is_default
|
||||
alert_notification.is_default,
|
||||
alert_notification.send_reminder,
|
||||
alert_notification.frequency
|
||||
FROM alert_notification
|
||||
`)
|
||||
|
||||
@ -91,7 +97,9 @@ func getAlertNotificationInternal(query *m.GetAlertNotificationsQuery, sess *DBS
|
||||
alert_notification.created,
|
||||
alert_notification.updated,
|
||||
alert_notification.settings,
|
||||
alert_notification.is_default
|
||||
alert_notification.is_default,
|
||||
alert_notification.send_reminder,
|
||||
alert_notification.frequency
|
||||
FROM alert_notification
|
||||
`)
|
||||
|
||||
@ -137,17 +145,31 @@ func CreateAlertNotificationCommand(cmd *m.CreateAlertNotificationCommand) error
|
||||
return fmt.Errorf("Alert notification name %s already exists", cmd.Name)
|
||||
}
|
||||
|
||||
alertNotification := &m.AlertNotification{
|
||||
OrgId: cmd.OrgId,
|
||||
Name: cmd.Name,
|
||||
Type: cmd.Type,
|
||||
Settings: cmd.Settings,
|
||||
Created: time.Now(),
|
||||
Updated: time.Now(),
|
||||
IsDefault: cmd.IsDefault,
|
||||
var frequency time.Duration
|
||||
if cmd.SendReminder {
|
||||
if cmd.Frequency == "" {
|
||||
return m.ErrNotificationFrequencyNotFound
|
||||
}
|
||||
|
||||
frequency, err = time.ParseDuration(cmd.Frequency)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if _, err = sess.Insert(alertNotification); err != nil {
|
||||
alertNotification := &m.AlertNotification{
|
||||
OrgId: cmd.OrgId,
|
||||
Name: cmd.Name,
|
||||
Type: cmd.Type,
|
||||
Settings: cmd.Settings,
|
||||
SendReminder: cmd.SendReminder,
|
||||
Frequency: frequency,
|
||||
Created: time.Now(),
|
||||
Updated: time.Now(),
|
||||
IsDefault: cmd.IsDefault,
|
||||
}
|
||||
|
||||
if _, err = sess.MustCols("send_reminder").Insert(alertNotification); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@ -179,16 +201,77 @@ func UpdateAlertNotification(cmd *m.UpdateAlertNotificationCommand) error {
|
||||
current.Name = cmd.Name
|
||||
current.Type = cmd.Type
|
||||
current.IsDefault = cmd.IsDefault
|
||||
current.SendReminder = cmd.SendReminder
|
||||
|
||||
sess.UseBool("is_default")
|
||||
if current.SendReminder {
|
||||
if cmd.Frequency == "" {
|
||||
return m.ErrNotificationFrequencyNotFound
|
||||
}
|
||||
|
||||
frequency, err := time.ParseDuration(cmd.Frequency)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
current.Frequency = frequency
|
||||
}
|
||||
|
||||
sess.UseBool("is_default", "send_reminder")
|
||||
|
||||
if affected, err := sess.ID(cmd.Id).Update(current); err != nil {
|
||||
return err
|
||||
} else if affected == 0 {
|
||||
return fmt.Errorf("Could not find alert notification")
|
||||
return fmt.Errorf("Could not update alert notification")
|
||||
}
|
||||
|
||||
cmd.Result = ¤t
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func RecordNotificationJournal(ctx context.Context, cmd *m.RecordNotificationJournalCommand) error {
|
||||
return inTransactionCtx(ctx, func(sess *DBSession) error {
|
||||
journalEntry := &m.AlertNotificationJournal{
|
||||
OrgId: cmd.OrgId,
|
||||
AlertId: cmd.AlertId,
|
||||
NotifierId: cmd.NotifierId,
|
||||
SentAt: cmd.SentAt,
|
||||
Success: cmd.Success,
|
||||
}
|
||||
|
||||
if _, err := sess.Insert(journalEntry); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func GetLatestNotification(ctx context.Context, cmd *m.GetLatestNotificationQuery) error {
|
||||
return inTransactionCtx(ctx, func(sess *DBSession) error {
|
||||
nj := &m.AlertNotificationJournal{}
|
||||
|
||||
_, err := sess.Desc("alert_notification_journal.sent_at").
|
||||
Limit(1).
|
||||
Where("alert_notification_journal.org_id = ? AND alert_notification_journal.alert_id = ? AND alert_notification_journal.notifier_id = ?", cmd.OrgId, cmd.AlertId, cmd.NotifierId).Get(nj)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if nj.AlertId == 0 && nj.Id == 0 && nj.NotifierId == 0 && nj.OrgId == 0 {
|
||||
return m.ErrJournalingNotFound
|
||||
}
|
||||
|
||||
cmd.Result = nj
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func CleanNotificationJournal(ctx context.Context, cmd *m.CleanNotificationJournalCommand) error {
|
||||
return inTransactionCtx(ctx, func(sess *DBSession) error {
|
||||
sql := "DELETE FROM alert_notification_journal WHERE alert_notification_journal.org_id = ? AND alert_notification_journal.alert_id = ? AND alert_notification_journal.notifier_id = ?"
|
||||
_, err := sess.Exec(sql, cmd.OrgId, cmd.AlertId, cmd.NotifierId)
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
@ -1,7 +1,9 @@
|
||||
package sqlstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/grafana/grafana/pkg/components/simplejson"
|
||||
m "github.com/grafana/grafana/pkg/models"
|
||||
@ -11,7 +13,48 @@ import (
|
||||
func TestAlertNotificationSQLAccess(t *testing.T) {
|
||||
Convey("Testing Alert notification sql access", t, func() {
|
||||
InitTestDB(t)
|
||||
var err error
|
||||
|
||||
Convey("Alert notification journal", func() {
|
||||
var alertId int64 = 5
|
||||
var orgId int64 = 5
|
||||
var notifierId int64 = 5
|
||||
|
||||
Convey("Getting last journal should raise error if no one exists", func() {
|
||||
query := &m.GetLatestNotificationQuery{AlertId: alertId, OrgId: orgId, NotifierId: notifierId}
|
||||
err := GetLatestNotification(context.Background(), query)
|
||||
So(err, ShouldEqual, m.ErrJournalingNotFound)
|
||||
|
||||
Convey("shoulbe be able to record two journaling events", func() {
|
||||
createCmd := &m.RecordNotificationJournalCommand{AlertId: alertId, NotifierId: notifierId, OrgId: orgId, Success: true, SentAt: 1}
|
||||
|
||||
err := RecordNotificationJournal(context.Background(), createCmd)
|
||||
So(err, ShouldBeNil)
|
||||
|
||||
createCmd.SentAt += 1000 //increase epoch
|
||||
|
||||
err = RecordNotificationJournal(context.Background(), createCmd)
|
||||
So(err, ShouldBeNil)
|
||||
|
||||
Convey("get last journaling event", func() {
|
||||
err := GetLatestNotification(context.Background(), query)
|
||||
So(err, ShouldBeNil)
|
||||
So(query.Result.SentAt, ShouldEqual, 1001)
|
||||
|
||||
Convey("be able to clear all journaling for an notifier", func() {
|
||||
cmd := &m.CleanNotificationJournalCommand{AlertId: alertId, NotifierId: notifierId, OrgId: orgId}
|
||||
err := CleanNotificationJournal(context.Background(), cmd)
|
||||
So(err, ShouldBeNil)
|
||||
|
||||
Convey("querying for last junaling should raise error", func() {
|
||||
query := &m.GetLatestNotificationQuery{AlertId: alertId, OrgId: orgId, NotifierId: notifierId}
|
||||
err := GetLatestNotification(context.Background(), query)
|
||||
So(err, ShouldEqual, m.ErrJournalingNotFound)
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Convey("Alert notifications should be empty", func() {
|
||||
cmd := &m.GetAlertNotificationsQuery{
|
||||
@ -24,19 +67,75 @@ func TestAlertNotificationSQLAccess(t *testing.T) {
|
||||
So(cmd.Result, ShouldBeNil)
|
||||
})
|
||||
|
||||
Convey("Can save Alert Notification", func() {
|
||||
Convey("Cannot save alert notifier with send reminder = true", func() {
|
||||
cmd := &m.CreateAlertNotificationCommand{
|
||||
Name: "ops",
|
||||
Type: "email",
|
||||
OrgId: 1,
|
||||
Settings: simplejson.New(),
|
||||
Name: "ops",
|
||||
Type: "email",
|
||||
OrgId: 1,
|
||||
SendReminder: true,
|
||||
Settings: simplejson.New(),
|
||||
}
|
||||
|
||||
err = CreateAlertNotificationCommand(cmd)
|
||||
Convey("and missing frequency", func() {
|
||||
err := CreateAlertNotificationCommand(cmd)
|
||||
So(err, ShouldEqual, m.ErrNotificationFrequencyNotFound)
|
||||
})
|
||||
|
||||
Convey("invalid frequency", func() {
|
||||
cmd.Frequency = "invalid duration"
|
||||
|
||||
err := CreateAlertNotificationCommand(cmd)
|
||||
So(err.Error(), ShouldEqual, "time: invalid duration invalid duration")
|
||||
})
|
||||
})
|
||||
|
||||
Convey("Cannot update alert notifier with send reminder = false", func() {
|
||||
cmd := &m.CreateAlertNotificationCommand{
|
||||
Name: "ops update",
|
||||
Type: "email",
|
||||
OrgId: 1,
|
||||
SendReminder: false,
|
||||
Settings: simplejson.New(),
|
||||
}
|
||||
|
||||
err := CreateAlertNotificationCommand(cmd)
|
||||
So(err, ShouldBeNil)
|
||||
|
||||
updateCmd := &m.UpdateAlertNotificationCommand{
|
||||
Id: cmd.Result.Id,
|
||||
SendReminder: true,
|
||||
}
|
||||
|
||||
Convey("and missing frequency", func() {
|
||||
err := UpdateAlertNotification(updateCmd)
|
||||
So(err, ShouldEqual, m.ErrNotificationFrequencyNotFound)
|
||||
})
|
||||
|
||||
Convey("invalid frequency", func() {
|
||||
updateCmd.Frequency = "invalid duration"
|
||||
|
||||
err := UpdateAlertNotification(updateCmd)
|
||||
So(err, ShouldNotBeNil)
|
||||
So(err.Error(), ShouldEqual, "time: invalid duration invalid duration")
|
||||
})
|
||||
})
|
||||
|
||||
Convey("Can save Alert Notification", func() {
|
||||
cmd := &m.CreateAlertNotificationCommand{
|
||||
Name: "ops",
|
||||
Type: "email",
|
||||
OrgId: 1,
|
||||
SendReminder: true,
|
||||
Frequency: "10s",
|
||||
Settings: simplejson.New(),
|
||||
}
|
||||
|
||||
err := CreateAlertNotificationCommand(cmd)
|
||||
So(err, ShouldBeNil)
|
||||
So(cmd.Result.Id, ShouldNotEqual, 0)
|
||||
So(cmd.Result.OrgId, ShouldNotEqual, 0)
|
||||
So(cmd.Result.Type, ShouldEqual, "email")
|
||||
So(cmd.Result.Frequency, ShouldEqual, 10*time.Second)
|
||||
|
||||
Convey("Cannot save Alert Notification with the same name", func() {
|
||||
err = CreateAlertNotificationCommand(cmd)
|
||||
@ -45,25 +144,42 @@ func TestAlertNotificationSQLAccess(t *testing.T) {
|
||||
|
||||
Convey("Can update alert notification", func() {
|
||||
newCmd := &m.UpdateAlertNotificationCommand{
|
||||
Name: "NewName",
|
||||
Type: "webhook",
|
||||
OrgId: cmd.Result.OrgId,
|
||||
Settings: simplejson.New(),
|
||||
Id: cmd.Result.Id,
|
||||
Name: "NewName",
|
||||
Type: "webhook",
|
||||
OrgId: cmd.Result.OrgId,
|
||||
SendReminder: true,
|
||||
Frequency: "60s",
|
||||
Settings: simplejson.New(),
|
||||
Id: cmd.Result.Id,
|
||||
}
|
||||
err := UpdateAlertNotification(newCmd)
|
||||
So(err, ShouldBeNil)
|
||||
So(newCmd.Result.Name, ShouldEqual, "NewName")
|
||||
So(newCmd.Result.Frequency, ShouldEqual, 60*time.Second)
|
||||
})
|
||||
|
||||
Convey("Can update alert notification to disable sending of reminders", func() {
|
||||
newCmd := &m.UpdateAlertNotificationCommand{
|
||||
Name: "NewName",
|
||||
Type: "webhook",
|
||||
OrgId: cmd.Result.OrgId,
|
||||
SendReminder: false,
|
||||
Settings: simplejson.New(),
|
||||
Id: cmd.Result.Id,
|
||||
}
|
||||
err := UpdateAlertNotification(newCmd)
|
||||
So(err, ShouldBeNil)
|
||||
So(newCmd.Result.SendReminder, ShouldBeFalse)
|
||||
})
|
||||
})
|
||||
|
||||
Convey("Can search using an array of ids", func() {
|
||||
cmd1 := m.CreateAlertNotificationCommand{Name: "nagios", Type: "webhook", OrgId: 1, Settings: simplejson.New()}
|
||||
cmd2 := m.CreateAlertNotificationCommand{Name: "slack", Type: "webhook", OrgId: 1, Settings: simplejson.New()}
|
||||
cmd3 := m.CreateAlertNotificationCommand{Name: "ops2", Type: "email", OrgId: 1, Settings: simplejson.New()}
|
||||
cmd4 := m.CreateAlertNotificationCommand{IsDefault: true, Name: "default", Type: "email", OrgId: 1, Settings: simplejson.New()}
|
||||
cmd1 := m.CreateAlertNotificationCommand{Name: "nagios", Type: "webhook", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
|
||||
cmd2 := m.CreateAlertNotificationCommand{Name: "slack", Type: "webhook", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
|
||||
cmd3 := m.CreateAlertNotificationCommand{Name: "ops2", Type: "email", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
|
||||
cmd4 := m.CreateAlertNotificationCommand{IsDefault: true, Name: "default", Type: "email", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
|
||||
|
||||
otherOrg := m.CreateAlertNotificationCommand{Name: "default", Type: "email", OrgId: 2, Settings: simplejson.New()}
|
||||
otherOrg := m.CreateAlertNotificationCommand{Name: "default", Type: "email", OrgId: 2, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
|
||||
|
||||
So(CreateAlertNotificationCommand(&cmd1), ShouldBeNil)
|
||||
So(CreateAlertNotificationCommand(&cmd2), ShouldBeNil)
|
||||
|
@ -65,6 +65,13 @@ func addAlertMigrations(mg *Migrator) {
|
||||
mg.AddMigration("Add column is_default", NewAddColumnMigration(alert_notification, &Column{
|
||||
Name: "is_default", Type: DB_Bool, Nullable: false, Default: "0",
|
||||
}))
|
||||
mg.AddMigration("Add column frequency", NewAddColumnMigration(alert_notification, &Column{
|
||||
Name: "frequency", Type: DB_BigInt, Nullable: true,
|
||||
}))
|
||||
mg.AddMigration("Add column send_reminder", NewAddColumnMigration(alert_notification, &Column{
|
||||
Name: "send_reminder", Type: DB_Bool, Nullable: true, Default: "0",
|
||||
}))
|
||||
|
||||
mg.AddMigration("add index alert_notification org_id & name", NewAddIndexMigration(alert_notification, alert_notification.Indices[0]))
|
||||
|
||||
mg.AddMigration("Update alert table charset", NewTableCharsetMigration("alert", []*Column{
|
||||
@ -82,4 +89,22 @@ func addAlertMigrations(mg *Migrator) {
|
||||
{Name: "type", Type: DB_NVarchar, Length: 255, Nullable: false},
|
||||
{Name: "settings", Type: DB_Text, Nullable: false},
|
||||
}))
|
||||
|
||||
notification_journal := Table{
|
||||
Name: "alert_notification_journal",
|
||||
Columns: []*Column{
|
||||
{Name: "id", Type: DB_BigInt, IsPrimaryKey: true, IsAutoIncrement: true},
|
||||
{Name: "org_id", Type: DB_BigInt, Nullable: false},
|
||||
{Name: "alert_id", Type: DB_BigInt, Nullable: false},
|
||||
{Name: "notifier_id", Type: DB_BigInt, Nullable: false},
|
||||
{Name: "sent_at", Type: DB_BigInt, Nullable: false},
|
||||
{Name: "success", Type: DB_Bool, Nullable: false},
|
||||
},
|
||||
Indices: []*Index{
|
||||
{Cols: []string{"org_id", "alert_id", "notifier_id"}, Type: IndexType},
|
||||
},
|
||||
}
|
||||
|
||||
mg.AddMigration("create notification_journal table v1", NewAddTableMigration(notification_journal))
|
||||
mg.AddMigration("add index notification_journal org_id & alert_id & notifier_id", NewAddIndexMigration(notification_journal, notification_journal.Indices[0]))
|
||||
}
|
||||
|
@ -11,6 +11,8 @@ export class AlertNotificationEditCtrl {
|
||||
model: any;
|
||||
defaults: any = {
|
||||
type: 'email',
|
||||
sendReminder: false,
|
||||
frequency: '15m',
|
||||
settings: {
|
||||
httpMethod: 'POST',
|
||||
autoResolve: true,
|
||||
@ -18,12 +20,17 @@ export class AlertNotificationEditCtrl {
|
||||
},
|
||||
isDefault: false,
|
||||
};
|
||||
getFrequencySuggestion: any;
|
||||
|
||||
/** @ngInject */
|
||||
constructor(private $routeParams, private backendSrv, private $location, private $templateCache, navModelSrv) {
|
||||
this.navModel = navModelSrv.getNav('alerting', 'channels', 0);
|
||||
this.isNew = !this.$routeParams.id;
|
||||
|
||||
this.getFrequencySuggestion = () => {
|
||||
return ['1m', '5m', '10m', '15m', '30m', '1h'];
|
||||
};
|
||||
|
||||
this.backendSrv
|
||||
.get(`/api/alert-notifiers`)
|
||||
.then(notifiers => {
|
||||
@ -102,6 +109,7 @@ export class AlertNotificationEditCtrl {
|
||||
const payload = {
|
||||
name: this.model.name,
|
||||
type: this.model.type,
|
||||
frequency: this.model.frequency,
|
||||
settings: this.model.settings,
|
||||
};
|
||||
|
||||
|
@ -32,6 +32,29 @@
|
||||
checked="ctrl.model.settings.uploadImage"
|
||||
tooltip="Captures an image and include it in the notification">
|
||||
</gf-form-switch>
|
||||
<gf-form-switch
|
||||
class="gf-form"
|
||||
label="Send reminders"
|
||||
label-class="width-12"
|
||||
checked="ctrl.model.sendReminder"
|
||||
tooltip="Send additional notifications for triggered alerts">
|
||||
</gf-form-switch>
|
||||
<div class="gf-form-inline">
|
||||
<div class="gf-form" ng-if="ctrl.model.sendReminder">
|
||||
<span class="gf-form-label width-12">Send reminder every
|
||||
<info-popover mode="right-normal" position="top center">
|
||||
Specify how often reminders should be sent, e.g. every 30s, 1m, 10m, 30m or 1h etc.
|
||||
</info-popover>
|
||||
</span>
|
||||
<input type="text" placeholder="Select or specify custom" class="gf-form-input width-15" ng-model="ctrl.model.frequency"
|
||||
bs-typeahead="ctrl.getFrequencySuggestion" data-min-length=0 ng-required="ctrl.model.sendReminder">
|
||||
</div>
|
||||
</div>
|
||||
<div class="gf-form">
|
||||
<span class="alert alert-info width-30" ng-if="ctrl.model.sendReminder">
|
||||
Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="gf-form-group" ng-include src="ctrl.notifierTemplateId">
|
||||
|
Loading…
Reference in New Issue
Block a user