Merge pull request #12145 from grafana/alerting_reminder

Alerting notification reminder

docs/sources/alerting/notifications.md

@@ -16,12 +16,11 @@ weight = 2
 
 When an alert changes state, it sends out notifications. Each alert rule can have
 multiple notifications. In order to add a notification to an alert rule you first need
-to add and configure a `notification` channel (can be email, PagerDuty or other integration). This is done from the Notification Channels page.
+to add and configure a `notification` channel (can be email, PagerDuty or other integration).
+This is done from the Notification Channels page.
 
 ## Notification Channel Setup
 
-{{< imgbox max-width="30%" img="/img/docs/v50/alerts_notifications_menu.png" caption="Alerting Notification Channels" >}}
-
 On the Notification Channels page hit the `New Channel` button to go the page where you
 can configure and setup a new Notification Channel.
 
@@ -30,7 +29,31 @@ sure it's setup correctly.
 
 ### Send on all alerts
 
-When checked, this option will nofity for all alert rules - existing and new.
+When checked, this option will notify for all alert rules - existing and new.
+
+### Send reminders
+
+> Only available in Grafana v5.3 and above.
+
+{{< docs-imagebox max-width="600px" img="/img/docs/v53/alerting_notification_reminders.png" class="docs-image--right" caption="Alerting notification reminders setup" >}}
+
+When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders
+should be sent using number of seconds (s), minutes (m) or hours (h), for example `30s`, `3m`, `5m` or `1h` etc.
+
+**Important:** Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured [alert rule evaluation interval](/alerting/rules/#name-evaluation-interval).
+
+These examples show how often and when reminders are sent for a triggered alert.
+
+Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
+---------- | ----------- | -----------
+`30s` | `15s` | ~30 seconds
+`1m` | `5m` | ~5 minutes
+`5m` | `15m` | ~15 minutes
+`6m` | `20m` | ~24 minutes
+`1h` | `15m` | ~1 hour
+`1h` | `2h` | ~2 hours
+
+<div class="clearfix"></div>
 
 ## Supported Notification Types
 
@@ -132,23 +155,23 @@ Once these two properties are set, you can send the alerts to Kafka for further
 
 ### All supported notifiers
 
-Name | Type |Support images
------|------------ | ------
-Slack | `slack` | yes
-Pagerduty | `pagerduty` | yes
-Email | `email` | yes
-Webhook | `webhook` | link
-Kafka | `kafka` | no
-Hipchat | `hipchat` | yes
-VictorOps | `victorops` | yes
-Sensu | `sensu` | yes
-OpsGenie | `opsgenie` | yes
-Threema | `threema` | yes
-Pushover | `pushover` | no
-Telegram | `telegram` | no
-Line | `line` | no
-Prometheus Alertmanager | `prometheus-alertmanager` | no
-Microsoft Teams | `teams` | yes
+Name | Type |Support images | Support reminders
+-----|------------ | ------ | ------ |
+Slack | `slack` | yes | yes
+Pagerduty | `pagerduty` | yes | yes
+Email | `email` | yes | yes
+Webhook | `webhook` | link | yes
+Kafka | `kafka` | no | yes
+Hipchat | `hipchat` | yes | yes
+VictorOps | `victorops` | yes | yes
+Sensu | `sensu` | yes | yes
+OpsGenie | `opsgenie` | yes | yes
+Threema | `threema` | yes | yes
+Pushover | `pushover` | no | yes
+Telegram | `telegram` | no | yes
+Line | `line` | no | yes
+Microsoft Teams | `teams` | yes | yes
+Prometheus Alertmanager | `prometheus-alertmanager` | no | no
 
 
 

docs/sources/alerting/rules.md

@@ -88,6 +88,11 @@ So as you can see from the above scenario Grafana will not send out notification
 to fire if the rule already is in state `Alerting`. To improve support for queries that return multiple series
 we plan to track state **per series** in a future release.
 
+> Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications
+> when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will
+> be included in the reminder notification. Depending on what notification channel you're using you may be able to take advantage
+> of this feature for identifying new/existing series causing alert to fire. [Read more about notification reminders here](/alerting/notifications/#send-reminders).
+
 ### No Data / Null values
 
 Below your conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.

docs/sources/http_api/alerting.md

@@ -50,6 +50,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 [
   {
     "id": 1,
@@ -86,6 +87,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "id": 1,
   "dashboardId": 1,
@@ -146,6 +148,7 @@ JSON Body Schema:
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "alertId": 1,
   "state":   "Paused",
@@ -177,6 +180,7 @@ JSON Body Schema:
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "state":   "Paused",
   "message": "alert paused",
@@ -204,14 +208,21 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 HTTP/1.1 200
 Content-Type: application/json
 
-{
-  "id": 1,
-  "name": "Team A",
-  "type": "email",
-  "isDefault": true,
-  "created": "2017-01-01 12:45",
-  "updated": "2017-01-01 12:45"
-}
+[
+  {
+    "id": 1,
+    "name": "Team A",
+    "type": "email",
+    "isDefault": false,
+    "sendReminder": false,
+    "settings": {
+      "addresses": "carl@grafana.com;dev@grafana.com"
+    },
+    "created": "2018-04-23T14:44:09+02:00",
+    "updated": "2018-08-20T15:47:49+02:00"
+  }
+]
+
 ```
 
 ## Create alert notification
@@ -232,6 +243,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
   "name": "new alert notification",  //Required
   "type":  "email", //Required
   "isDefault": false,
+  "sendReminder": false,
   "settings": {
     "addresses": "carl@grafana.com;dev@grafana.com"
   }
@@ -243,14 +255,18 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "id": 1,
   "name": "new alert notification",
   "type": "email",
   "isDefault": false,
-  "settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
-  "created": "2017-01-01 12:34",
-  "updated": "2017-01-01 12:34"
+  "sendReminder": false,
+  "settings": {
+    "addresses": "carl@grafana.com;dev@grafana.com"
+  },
+  "created": "2018-04-23T14:44:09+02:00",
+  "updated": "2018-08-20T15:47:49+02:00"
 }
 ```
 
@@ -271,6 +287,8 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
   "name": "new alert notification",  //Required
   "type":  "email", //Required
   "isDefault": false,
+  "sendReminder": true,
+  "frequency": "15m",
   "settings": {
     "addresses: "carl@grafana.com;dev@grafana.com"
   }
@@ -282,12 +300,17 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "id": 1,
   "name": "new alert notification",
   "type": "email",
   "isDefault": false,
-  "settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
+  "sendReminder": true,
+  "frequency": "15m",
+  "settings": {
+    "addresses": "carl@grafana.com;dev@grafana.com"
+  },
   "created": "2017-01-01 12:34",
   "updated": "2017-01-01 12:34"
 }
@@ -311,6 +334,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 ```http
 HTTP/1.1 200
 Content-Type: application/json
+
 {
   "message": "Notification deleted"
 }

pkg/api/alerting.go

@@ -192,14 +192,7 @@ func GetAlertNotifications(c *m.ReqContext) Response {
 	result := make([]*dtos.AlertNotification, 0)
 
 	for _, notification := range query.Result {
-		result = append(result, &dtos.AlertNotification{
-			Id:        notification.Id,
-			Name:      notification.Name,
-			Type:      notification.Type,
-			IsDefault: notification.IsDefault,
-			Created:   notification.Created,
-			Updated:   notification.Updated,
-		})
+		result = append(result, dtos.NewAlertNotification(notification))
 	}
 
 	return JSON(200, result)
@@ -215,7 +208,7 @@ func GetAlertNotificationByID(c *m.ReqContext) Response {
 		return Error(500, "Failed to get alert notifications", err)
 	}
 
-	return JSON(200, query.Result)
+	return JSON(200, dtos.NewAlertNotification(query.Result))
 }
 
 func CreateAlertNotification(c *m.ReqContext, cmd m.CreateAlertNotificationCommand) Response {
@@ -225,7 +218,7 @@ func CreateAlertNotification(c *m.ReqContext, cmd m.CreateAlertNotificationComma
 		return Error(500, "Failed to create alert notification", err)
 	}
 
-	return JSON(200, cmd.Result)
+	return JSON(200, dtos.NewAlertNotification(cmd.Result))
 }
 
 func UpdateAlertNotification(c *m.ReqContext, cmd m.UpdateAlertNotificationCommand) Response {
@@ -235,7 +228,7 @@ func UpdateAlertNotification(c *m.ReqContext, cmd m.UpdateAlertNotificationComma
 		return Error(500, "Failed to update alert notification", err)
 	}
 
-	return JSON(200, cmd.Result)
+	return JSON(200, dtos.NewAlertNotification(cmd.Result))
 }
 
 func DeleteAlertNotification(c *m.ReqContext) Response {

pkg/api/dtos/alerting.go

@@ -1,35 +1,76 @@
 package dtos
 
 import (
+	"fmt"
 	"time"
 
 	"github.com/grafana/grafana/pkg/components/null"
 	"github.com/grafana/grafana/pkg/components/simplejson"
-	m "github.com/grafana/grafana/pkg/models"
+	"github.com/grafana/grafana/pkg/models"
 )
 
 type AlertRule struct {
-	Id             int64            `json:"id"`
-	DashboardId    int64            `json:"dashboardId"`
-	PanelId        int64            `json:"panelId"`
-	Name           string           `json:"name"`
-	Message        string           `json:"message"`
-	State          m.AlertStateType `json:"state"`
-	NewStateDate   time.Time        `json:"newStateDate"`
-	EvalDate       time.Time        `json:"evalDate"`
-	EvalData       *simplejson.Json `json:"evalData"`
-	ExecutionError string           `json:"executionError"`
-	Url            string           `json:"url"`
-	CanEdit        bool             `json:"canEdit"`
+	Id             int64                 `json:"id"`
+	DashboardId    int64                 `json:"dashboardId"`
+	PanelId        int64                 `json:"panelId"`
+	Name           string                `json:"name"`
+	Message        string                `json:"message"`
+	State          models.AlertStateType `json:"state"`
+	NewStateDate   time.Time             `json:"newStateDate"`
+	EvalDate       time.Time             `json:"evalDate"`
+	EvalData       *simplejson.Json      `json:"evalData"`
+	ExecutionError string                `json:"executionError"`
+	Url            string                `json:"url"`
+	CanEdit        bool                  `json:"canEdit"`
+}
+
+func formatShort(interval time.Duration) string {
+	var result string
+
+	hours := interval / time.Hour
+	if hours > 0 {
+		result += fmt.Sprintf("%dh", hours)
+	}
+
+	remaining := interval - (hours * time.Hour)
+	mins := remaining / time.Minute
+	if mins > 0 {
+		result += fmt.Sprintf("%dm", mins)
+	}
+
+	remaining = remaining - (mins * time.Minute)
+	seconds := remaining / time.Second
+	if seconds > 0 {
+		result += fmt.Sprintf("%ds", seconds)
+	}
+
+	return result
+}
+
+func NewAlertNotification(notification *models.AlertNotification) *AlertNotification {
+	return &AlertNotification{
+		Id:           notification.Id,
+		Name:         notification.Name,
+		Type:         notification.Type,
+		IsDefault:    notification.IsDefault,
+		Created:      notification.Created,
+		Updated:      notification.Updated,
+		Frequency:    formatShort(notification.Frequency),
+		SendReminder: notification.SendReminder,
+		Settings:     notification.Settings,
+	}
 }
 
 type AlertNotification struct {
-	Id        int64     `json:"id"`
-	Name      string    `json:"name"`
-	Type      string    `json:"type"`
-	IsDefault bool      `json:"isDefault"`
-	Created   time.Time `json:"created"`
-	Updated   time.Time `json:"updated"`
+	Id           int64            `json:"id"`
+	Name         string           `json:"name"`
+	Type         string           `json:"type"`
+	IsDefault    bool             `json:"isDefault"`
+	SendReminder bool             `json:"sendReminder"`
+	Frequency    string           `json:"frequency"`
+	Created      time.Time        `json:"created"`
+	Updated      time.Time        `json:"updated"`
+	Settings     *simplejson.Json `json:"settings"`
 }
 
 type AlertTestCommand struct {
@@ -39,7 +80,7 @@ type AlertTestCommand struct {
 
 type AlertTestResult struct {
 	Firing         bool                  `json:"firing"`
-	State          m.AlertStateType      `json:"state"`
+	State          models.AlertStateType `json:"state"`
 	ConditionEvals string                `json:"conditionEvals"`
 	TimeMs         string                `json:"timeMs"`
 	Error          string                `json:"error,omitempty"`
@@ -59,9 +100,11 @@ type EvalMatch struct {
 }
 
 type NotificationTestCommand struct {
-	Name     string           `json:"name"`
-	Type     string           `json:"type"`
-	Settings *simplejson.Json `json:"settings"`
+	Name         string           `json:"name"`
+	Type         string           `json:"type"`
+	SendReminder bool             `json:"sendReminder"`
+	Frequency    string           `json:"frequency"`
+	Settings     *simplejson.Json `json:"settings"`
 }
 
 type PauseAlertCommand struct {

pkg/api/dtos/alerting_test.go

@@ -0,0 +1,35 @@
+package dtos
+
+import (
+	"testing"
+	"time"
+)
+
+func TestFormatShort(t *testing.T) {
+	tcs := []struct {
+		interval time.Duration
+		expected string
+	}{
+		{interval: time.Hour, expected: "1h"},
+		{interval: time.Hour + time.Minute, expected: "1h1m"},
+		{interval: (time.Hour * 10) + time.Minute, expected: "10h1m"},
+		{interval: (time.Hour * 10) + (time.Minute * 10) + time.Second, expected: "10h10m1s"},
+		{interval: time.Minute * 10, expected: "10m"},
+	}
+
+	for _, tc := range tcs {
+		got := formatShort(tc.interval)
+		if got != tc.expected {
+			t.Errorf("expected %s got %s interval: %v", tc.expected, got, tc.interval)
+		}
+
+		parsed, err := time.ParseDuration(tc.expected)
+		if err != nil {
+			t.Fatalf("could not parse expected duration")
+		}
+
+		if parsed != tc.interval {
+			t.Errorf("expectes the parsed duration to equal the interval. Got %v expected: %v", parsed, tc.interval)
+		}
+	}
+}

pkg/models/alert_notifications.go

@@ -1,38 +1,50 @@
 package models
 
 import (
+	"errors"
 	"time"
 
 	"github.com/grafana/grafana/pkg/components/simplejson"
 )
 
+var (
+	ErrNotificationFrequencyNotFound = errors.New("Notification frequency not specified")
+	ErrJournalingNotFound            = errors.New("alert notification journaling not found")
+)
+
 type AlertNotification struct {
-	Id        int64            `json:"id"`
-	OrgId     int64            `json:"-"`
-	Name      string           `json:"name"`
-	Type      string           `json:"type"`
-	IsDefault bool             `json:"isDefault"`
-	Settings  *simplejson.Json `json:"settings"`
-	Created   time.Time        `json:"created"`
-	Updated   time.Time        `json:"updated"`
+	Id           int64            `json:"id"`
+	OrgId        int64            `json:"-"`
+	Name         string           `json:"name"`
+	Type         string           `json:"type"`
+	SendReminder bool             `json:"sendReminder"`
+	Frequency    time.Duration    `json:"frequency"`
+	IsDefault    bool             `json:"isDefault"`
+	Settings     *simplejson.Json `json:"settings"`
+	Created      time.Time        `json:"created"`
+	Updated      time.Time        `json:"updated"`
 }
 
 type CreateAlertNotificationCommand struct {
-	Name      string           `json:"name"  binding:"Required"`
-	Type      string           `json:"type"  binding:"Required"`
-	IsDefault bool             `json:"isDefault"`
-	Settings  *simplejson.Json `json:"settings"`
+	Name         string           `json:"name"  binding:"Required"`
+	Type         string           `json:"type"  binding:"Required"`
+	SendReminder bool             `json:"sendReminder"`
+	Frequency    string           `json:"frequency"`
+	IsDefault    bool             `json:"isDefault"`
+	Settings     *simplejson.Json `json:"settings"`
 
 	OrgId  int64 `json:"-"`
 	Result *AlertNotification
 }
 
 type UpdateAlertNotificationCommand struct {
-	Id        int64            `json:"id"  binding:"Required"`
-	Name      string           `json:"name"  binding:"Required"`
-	Type      string           `json:"type"  binding:"Required"`
-	IsDefault bool             `json:"isDefault"`
-	Settings  *simplejson.Json `json:"settings"  binding:"Required"`
+	Id           int64            `json:"id"  binding:"Required"`
+	Name         string           `json:"name"  binding:"Required"`
+	Type         string           `json:"type"  binding:"Required"`
+	SendReminder bool             `json:"sendReminder"`
+	Frequency    string           `json:"frequency"`
+	IsDefault    bool             `json:"isDefault"`
+	Settings     *simplejson.Json `json:"settings"  binding:"Required"`
 
 	OrgId  int64 `json:"-"`
 	Result *AlertNotification
@@ -63,3 +75,34 @@ type GetAllAlertNotificationsQuery struct {
 
 	Result []*AlertNotification
 }
+
+type AlertNotificationJournal struct {
+	Id         int64
+	OrgId      int64
+	AlertId    int64
+	NotifierId int64
+	SentAt     int64
+	Success    bool
+}
+
+type RecordNotificationJournalCommand struct {
+	OrgId      int64
+	AlertId    int64
+	NotifierId int64
+	SentAt     int64
+	Success    bool
+}
+
+type GetLatestNotificationQuery struct {
+	OrgId      int64
+	AlertId    int64
+	NotifierId int64
+
+	Result *AlertNotificationJournal
+}
+
+type CleanNotificationJournalCommand struct {
+	OrgId      int64
+	AlertId    int64
+	NotifierId int64
+}

pkg/services/alerting/interfaces.go

@@ -1,6 +1,9 @@
 package alerting
 
-import "time"
+import (
+	"context"
+	"time"
+)
 
 type EvalHandler interface {
 	Eval(evalContext *EvalContext)
@@ -15,10 +18,14 @@ type Notifier interface {
 	Notify(evalContext *EvalContext) error
 	GetType() string
 	NeedsImage() bool
-	ShouldNotify(evalContext *EvalContext) bool
+
+	// ShouldNotify checks this evaluation should send an alert notification
+	ShouldNotify(ctx context.Context, evalContext *EvalContext) bool
 
 	GetNotifierId() int64
 	GetIsDefault() bool
+	GetSendReminder() bool
+	GetFrequency() time.Duration
 }
 
 type NotifierSlice []Notifier

pkg/services/alerting/notifier.go

@@ -1,10 +1,10 @@
 package alerting
 
 import (
+	"context"
 	"errors"
 	"fmt"
-
-	"golang.org/x/sync/errgroup"
+	"time"
 
 	"github.com/grafana/grafana/pkg/bus"
 	"github.com/grafana/grafana/pkg/components/imguploader"
@@ -58,17 +58,47 @@ func (n *notificationService) SendIfNeeded(context *EvalContext) error {
 	return n.sendNotifications(context, notifiers)
 }
 
-func (n *notificationService) sendNotifications(context *EvalContext, notifiers []Notifier) error {
-	g, _ := errgroup.WithContext(context.Ctx)
-
+func (n *notificationService) sendNotifications(evalContext *EvalContext, notifiers []Notifier) error {
 	for _, notifier := range notifiers {
-		not := notifier //avoid updating scope variable in go routine
-		n.log.Debug("Sending notification", "type", not.GetType(), "id", not.GetNotifierId(), "isDefault", not.GetIsDefault())
-		metrics.M_Alerting_Notification_Sent.WithLabelValues(not.GetType()).Inc()
-		g.Go(func() error { return not.Notify(context) })
+		not := notifier
+
+		err := bus.InTransaction(evalContext.Ctx, func(ctx context.Context) error {
+			n.log.Debug("trying to send notification", "id", not.GetNotifierId())
+
+			// Verify that we can send the notification again
+			// but this time within the same transaction.
+			if !evalContext.IsTestRun && !not.ShouldNotify(context.Background(), evalContext) {
+				return nil
+			}
+
+			n.log.Debug("Sending notification", "type", not.GetType(), "id", not.GetNotifierId(), "isDefault", not.GetIsDefault())
+			metrics.M_Alerting_Notification_Sent.WithLabelValues(not.GetType()).Inc()
+
+			//send notification
+			success := not.Notify(evalContext) == nil
+
+			if evalContext.IsTestRun {
+				return nil
+			}
+
+			//write result to db.
+			cmd := &m.RecordNotificationJournalCommand{
+				OrgId:      evalContext.Rule.OrgId,
+				AlertId:    evalContext.Rule.Id,
+				NotifierId: not.GetNotifierId(),
+				SentAt:     time.Now().Unix(),
+				Success:    success,
+			}
+
+			return bus.DispatchCtx(ctx, cmd)
+		})
+
+		if err != nil {
+			n.log.Error("failed to send notification", "id", not.GetNotifierId())
+		}
 	}
 
-	return g.Wait()
+	return nil
 }
 
 func (n *notificationService) uploadImage(context *EvalContext) (err error) {
@@ -110,7 +140,7 @@ func (n *notificationService) uploadImage(context *EvalContext) (err error) {
 	return nil
 }
 
-func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []int64, context *EvalContext) (NotifierSlice, error) {
+func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []int64, evalContext *EvalContext) (NotifierSlice, error) {
 	query := &m.GetAlertNotificationsToSendQuery{OrgId: orgId, Ids: notificationIds}
 
 	if err := bus.Dispatch(query); err != nil {
@@ -123,7 +153,8 @@ func (n *notificationService) getNeededNotifiers(orgId int64, notificationIds []
 		if err != nil {
 			return nil, err
 		}
-		if not.ShouldNotify(context) {
+
+		if not.ShouldNotify(evalContext.Ctx, evalContext) {
 			result = append(result, not)
 		}
 	}

pkg/services/alerting/notifiers/alertmanager.go

@@ -1,6 +1,7 @@
 package notifiers
 
 import (
+	"context"
 	"time"
 
 	"github.com/grafana/grafana/pkg/bus"
@@ -33,7 +34,7 @@ func NewAlertmanagerNotifier(model *m.AlertNotification) (alerting.Notifier, err
 	}
 
 	return &AlertmanagerNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		log:          log.New("alerting.notifier.prometheus-alertmanager"),
 	}, nil
@@ -45,7 +46,7 @@ type AlertmanagerNotifier struct {
 	log log.Logger
 }
 
-func (this *AlertmanagerNotifier) ShouldNotify(evalContext *alerting.EvalContext) bool {
+func (this *AlertmanagerNotifier) ShouldNotify(ctx context.Context, evalContext *alerting.EvalContext) bool {
 	this.log.Debug("Should notify", "ruleId", evalContext.Rule.Id, "state", evalContext.Rule.State, "previousState", evalContext.PrevAlertState)
 
 	// Do not notify when we become OK for the first time.

pkg/services/alerting/notifiers/base.go

@@ -1,50 +1,94 @@
 package notifiers
 
 import (
-	"github.com/grafana/grafana/pkg/components/simplejson"
-	m "github.com/grafana/grafana/pkg/models"
+	"context"
+	"time"
+
+	"github.com/grafana/grafana/pkg/bus"
+	"github.com/grafana/grafana/pkg/log"
+	"github.com/grafana/grafana/pkg/models"
 
 	"github.com/grafana/grafana/pkg/services/alerting"
 )
 
 type NotifierBase struct {
-	Name        string
-	Type        string
-	Id          int64
-	IsDeault    bool
-	UploadImage bool
+	Name         string
+	Type         string
+	Id           int64
+	IsDeault     bool
+	UploadImage  bool
+	SendReminder bool
+	Frequency    time.Duration
+
+	log log.Logger
 }
 
-func NewNotifierBase(id int64, isDefault bool, name, notifierType string, model *simplejson.Json) NotifierBase {
+func NewNotifierBase(model *models.AlertNotification) NotifierBase {
 	uploadImage := true
-	value, exist := model.CheckGet("uploadImage")
+	value, exist := model.Settings.CheckGet("uploadImage")
 	if exist {
 		uploadImage = value.MustBool()
 	}
 
 	return NotifierBase{
-		Id:          id,
-		Name:        name,
-		IsDeault:    isDefault,
-		Type:        notifierType,
-		UploadImage: uploadImage,
+		Id:           model.Id,
+		Name:         model.Name,
+		IsDeault:     model.IsDefault,
+		Type:         model.Type,
+		UploadImage:  uploadImage,
+		SendReminder: model.SendReminder,
+		Frequency:    model.Frequency,
+		log:          log.New("alerting.notifier." + model.Name),
 	}
 }
 
-func defaultShouldNotify(context *alerting.EvalContext) bool {
+func defaultShouldNotify(context *alerting.EvalContext, sendReminder bool, frequency time.Duration, lastNotify time.Time) bool {
 	// Only notify on state change.
-	if context.PrevAlertState == context.Rule.State {
+	if context.PrevAlertState == context.Rule.State && !sendReminder {
 		return false
 	}
+
+	// Do not notify if interval has not elapsed
+	if sendReminder && !lastNotify.IsZero() && lastNotify.Add(frequency).After(time.Now()) {
+		return false
+	}
+
+	// Do not notify if alert state if OK or pending even on repeated notify
+	if sendReminder && (context.Rule.State == models.AlertStateOK || context.Rule.State == models.AlertStatePending) {
+		return false
+	}
+
 	// Do not notify when we become OK for the first time.
-	if (context.PrevAlertState == m.AlertStatePending) && (context.Rule.State == m.AlertStateOK) {
+	if (context.PrevAlertState == models.AlertStatePending) && (context.Rule.State == models.AlertStateOK) {
 		return false
 	}
+
 	return true
 }
 
-func (n *NotifierBase) ShouldNotify(context *alerting.EvalContext) bool {
-	return defaultShouldNotify(context)
+// ShouldNotify checks this evaluation should send an alert notification
+func (n *NotifierBase) ShouldNotify(ctx context.Context, c *alerting.EvalContext) bool {
+	cmd := &models.GetLatestNotificationQuery{
+		OrgId:      c.Rule.OrgId,
+		AlertId:    c.Rule.Id,
+		NotifierId: n.Id,
+	}
+
+	err := bus.DispatchCtx(ctx, cmd)
+	if err == models.ErrJournalingNotFound {
+		return true
+	}
+
+	if err != nil {
+		n.log.Error("Could not determine last time alert notifier fired", "Alert name", c.Rule.Name, "Error", err)
+		return false
+	}
+
+	if !cmd.Result.Success {
+		return true
+	}
+
+	return defaultShouldNotify(c, n.SendReminder, n.Frequency, time.Unix(cmd.Result.SentAt, 0))
 }
 
 func (n *NotifierBase) GetType() string {
@@ -62,3 +106,11 @@ func (n *NotifierBase) GetNotifierId() int64 {
 func (n *NotifierBase) GetIsDefault() bool {
 	return n.IsDeault
 }
+
+func (n *NotifierBase) GetSendReminder() bool {
+	return n.SendReminder
+}
+
+func (n *NotifierBase) GetFrequency() time.Duration {
+	return n.Frequency
+}

pkg/services/alerting/notifiers/base_test.go

@@ -2,7 +2,11 @@ package notifiers
 
 import (
 	"context"
+	"errors"
 	"testing"
+	"time"
+
+	"github.com/grafana/grafana/pkg/bus"
 
 	"github.com/grafana/grafana/pkg/components/simplejson"
 	m "github.com/grafana/grafana/pkg/models"
@@ -10,47 +14,129 @@ import (
 	. "github.com/smartystreets/goconvey/convey"
 )
 
-func TestBaseNotifier(t *testing.T) {
-	Convey("Base notifier tests", t, func() {
-		Convey("default constructor for notifiers", func() {
-			bJson := simplejson.New()
+func TestShouldSendAlertNotification(t *testing.T) {
+	tcs := []struct {
+		name         string
+		prevState    m.AlertStateType
+		newState     m.AlertStateType
+		expected     bool
+		sendReminder bool
+	}{
+		{
+			name:      "pending -> ok should not trigger an notification",
+			newState:  m.AlertStatePending,
+			prevState: m.AlertStateOK,
+			expected:  false,
+		},
+		{
+			name:      "ok -> alerting should trigger an notification",
+			newState:  m.AlertStateOK,
+			prevState: m.AlertStateAlerting,
+			expected:  true,
+		},
+		{
+			name:      "ok -> pending should not trigger an notification",
+			newState:  m.AlertStateOK,
+			prevState: m.AlertStatePending,
+			expected:  false,
+		},
+		{
+			name:         "ok -> ok should not trigger an notification",
+			newState:     m.AlertStateOK,
+			prevState:    m.AlertStateOK,
+			expected:     false,
+			sendReminder: false,
+		},
+		{
+			name:         "ok -> alerting should not trigger an notification",
+			newState:     m.AlertStateOK,
+			prevState:    m.AlertStateAlerting,
+			expected:     true,
+			sendReminder: true,
+		},
+		{
+			name:         "ok -> ok with reminder should not trigger an notification",
+			newState:     m.AlertStateOK,
+			prevState:    m.AlertStateOK,
+			expected:     false,
+			sendReminder: true,
+		},
+	}
 
-			Convey("can parse false value", func() {
-				bJson.Set("uploadImage", false)
-
-				base := NewNotifierBase(1, false, "name", "email", bJson)
-				So(base.UploadImage, ShouldBeFalse)
-			})
-
-			Convey("can parse true value", func() {
-				bJson.Set("uploadImage", true)
-
-				base := NewNotifierBase(1, false, "name", "email", bJson)
-				So(base.UploadImage, ShouldBeTrue)
-			})
-
-			Convey("default value should be true for backwards compatibility", func() {
-				base := NewNotifierBase(1, false, "name", "email", bJson)
-				So(base.UploadImage, ShouldBeTrue)
-			})
+	for _, tc := range tcs {
+		evalContext := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
+			State: tc.newState,
 		})
 
-		Convey("should notify", func() {
-			Convey("pending -> ok", func() {
-				context := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
-					State: m.AlertStatePending,
-				})
-				context.Rule.State = m.AlertStateOK
-				So(defaultShouldNotify(context), ShouldBeFalse)
+		evalContext.Rule.State = tc.prevState
+		if defaultShouldNotify(evalContext, true, 0, time.Now()) != tc.expected {
+			t.Errorf("failed %s. expected %+v to return %v", tc.name, tc, tc.expected)
+		}
+	}
+}
+
+func TestShouldNotifyWhenNoJournalingIsFound(t *testing.T) {
+	Convey("base notifier", t, func() {
+		bus.ClearBusHandlers()
+
+		notifier := NewNotifierBase(&m.AlertNotification{
+			Id:       1,
+			Name:     "name",
+			Type:     "email",
+			Settings: simplejson.New(),
+		})
+		evalContext := alerting.NewEvalContext(context.TODO(), &alerting.Rule{})
+
+		Convey("should notify if no journaling is found", func() {
+			bus.AddHandlerCtx("", func(ctx context.Context, q *m.GetLatestNotificationQuery) error {
+				return m.ErrJournalingNotFound
 			})
 
-			Convey("ok -> alerting", func() {
-				context := alerting.NewEvalContext(context.TODO(), &alerting.Rule{
-					State: m.AlertStateOK,
-				})
-				context.Rule.State = m.AlertStateAlerting
-				So(defaultShouldNotify(context), ShouldBeTrue)
+			if !notifier.ShouldNotify(context.Background(), evalContext) {
+				t.Errorf("should send notifications when ErrJournalingNotFound is returned")
+			}
+		})
+
+		Convey("should not notify query returns error", func() {
+			bus.AddHandlerCtx("", func(ctx context.Context, q *m.GetLatestNotificationQuery) error {
+				return errors.New("some kind of error unknown error")
 			})
+
+			if notifier.ShouldNotify(context.Background(), evalContext) {
+				t.Errorf("should not send notifications when query returns error")
+			}
+		})
+	})
+}
+
+func TestBaseNotifier(t *testing.T) {
+	Convey("default constructor for notifiers", t, func() {
+		bJson := simplejson.New()
+
+		model := &m.AlertNotification{
+			Id:       1,
+			Name:     "name",
+			Type:     "email",
+			Settings: bJson,
+		}
+
+		Convey("can parse false value", func() {
+			bJson.Set("uploadImage", false)
+
+			base := NewNotifierBase(model)
+			So(base.UploadImage, ShouldBeFalse)
+		})
+
+		Convey("can parse true value", func() {
+			bJson.Set("uploadImage", true)
+
+			base := NewNotifierBase(model)
+			So(base.UploadImage, ShouldBeTrue)
+		})
+
+		Convey("default value should be true for backwards compatibility", func() {
+			base := NewNotifierBase(model)
+			So(base.UploadImage, ShouldBeTrue)
 		})
 	})
 }

pkg/services/alerting/notifiers/dingding.go

@@ -32,7 +32,7 @@ func NewDingDingNotifier(model *m.AlertNotification) (alerting.Notifier, error)
 	}
 
 	return &DingDingNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		log:          log.New("alerting.notifier.dingding"),
 	}, nil

pkg/services/alerting/notifiers/discord.go

@@ -39,7 +39,7 @@ func NewDiscordNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &DiscordNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		WebhookURL:   url,
 		log:          log.New("alerting.notifier.discord"),
 	}, nil

pkg/services/alerting/notifiers/email.go

@@ -52,7 +52,7 @@ func NewEmailNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	})
 
 	return &EmailNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Addresses:    addresses,
 		log:          log.New("alerting.notifier.email"),
 	}, nil

pkg/services/alerting/notifiers/hipchat.go

@@ -59,7 +59,7 @@ func NewHipChatNotifier(model *models.AlertNotification) (alerting.Notifier, err
 	roomId := model.Settings.Get("roomid").MustString()
 
 	return &HipChatNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		ApiKey:       apikey,
 		RoomId:       roomId,

pkg/services/alerting/notifiers/kafka.go

@@ -43,7 +43,7 @@ func NewKafkaNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &KafkaNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Endpoint:     endpoint,
 		Topic:        topic,
 		log:          log.New("alerting.notifier.kafka"),

pkg/services/alerting/notifiers/line.go

@@ -39,7 +39,7 @@ func NewLINENotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &LineNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Token:        token,
 		log:          log.New("alerting.notifier.line"),
 	}, nil

pkg/services/alerting/notifiers/opsgenie.go

@@ -56,7 +56,7 @@ func NewOpsGenieNotifier(model *m.AlertNotification) (alerting.Notifier, error)
 	}
 
 	return &OpsGenieNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		ApiKey:       apiKey,
 		ApiUrl:       apiUrl,
 		AutoClose:    autoClose,

pkg/services/alerting/notifiers/pagerduty.go

@@ -51,7 +51,7 @@ func NewPagerdutyNotifier(model *m.AlertNotification) (alerting.Notifier, error)
 	}
 
 	return &PagerdutyNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Key:          key,
 		AutoResolve:  autoResolve,
 		log:          log.New("alerting.notifier.pagerduty"),

pkg/services/alerting/notifiers/pushover.go

@@ -99,7 +99,7 @@ func NewPushoverNotifier(model *m.AlertNotification) (alerting.Notifier, error)
 		return nil, alerting.ValidationError{Reason: "API token not given"}
 	}
 	return &PushoverNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		UserKey:      userKey,
 		ApiToken:     apiToken,
 		Priority:     priority,

pkg/services/alerting/notifiers/sensu.go

@@ -51,7 +51,7 @@ func NewSensuNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &SensuNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		User:         model.Settings.Get("username").MustString(),
 		Source:       model.Settings.Get("source").MustString(),

pkg/services/alerting/notifiers/slack.go

@@ -78,7 +78,7 @@ func NewSlackNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	uploadImage := model.Settings.Get("uploadImage").MustBool(true)
 
 	return &SlackNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		Recipient:    recipient,
 		Mention:      mention,

pkg/services/alerting/notifiers/teams.go

@@ -33,7 +33,7 @@ func NewTeamsNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &TeamsNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		log:          log.New("alerting.notifier.teams"),
 	}, nil

pkg/services/alerting/notifiers/telegram.go

@@ -78,7 +78,7 @@ func NewTelegramNotifier(model *m.AlertNotification) (alerting.Notifier, error)
 	}
 
 	return &TelegramNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		BotToken:     botToken,
 		ChatID:       chatId,
 		UploadImage:  uploadImage,

pkg/services/alerting/notifiers/threema.go

@@ -106,7 +106,7 @@ func NewThreemaNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &ThreemaNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		GatewayID:    gatewayID,
 		RecipientID:  recipientID,
 		APISecret:    apiSecret,

pkg/services/alerting/notifiers/victorops.go

@@ -51,7 +51,7 @@ func NewVictoropsNotifier(model *models.AlertNotification) (alerting.Notifier, e
 	}
 
 	return &VictoropsNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		URL:          url,
 		AutoResolve:  autoResolve,
 		log:          log.New("alerting.notifier.victorops"),

pkg/services/alerting/notifiers/webhook.go

@@ -47,7 +47,7 @@ func NewWebHookNotifier(model *m.AlertNotification) (alerting.Notifier, error) {
 	}
 
 	return &WebhookNotifier{
-		NotifierBase: NewNotifierBase(model.Id, model.IsDefault, model.Name, model.Type, model.Settings),
+		NotifierBase: NewNotifierBase(model),
 		Url:          url,
 		User:         model.Settings.Get("username").MustString(),
 		Password:     model.Settings.Get("password").MustString(),

pkg/services/alerting/result_handler.go

@@ -88,6 +88,18 @@ func (handler *DefaultResultHandler) Handle(evalContext *EvalContext) error {
 		}
 	}
 
+	if evalContext.Rule.State == m.AlertStateOK && evalContext.PrevAlertState != m.AlertStateOK {
+		for _, notifierId := range evalContext.Rule.Notifications {
+			cmd := &m.CleanNotificationJournalCommand{
+				AlertId:    evalContext.Rule.Id,
+				NotifierId: notifierId,
+				OrgId:      evalContext.Rule.OrgId,
+			}
+			if err := bus.DispatchCtx(evalContext.Ctx, cmd); err != nil {
+				handler.log.Error("Failed to clean up old notification records", "notifier", notifierId, "alert", evalContext.Rule.Id, "Error", err)
+			}
+		}
+	}
 	handler.notifier.SendIfNeeded(evalContext)
 
 	return nil

pkg/services/sqlstore/alert_notification.go

@@ -2,6 +2,7 @@ package sqlstore
 
 import (
 	"bytes"
+	"context"
 	"fmt"
 	"strings"
 	"time"
@@ -17,6 +18,9 @@ func init() {
 	bus.AddHandler("sql", DeleteAlertNotification)
 	bus.AddHandler("sql", GetAlertNotificationsToSend)
 	bus.AddHandler("sql", GetAllAlertNotifications)
+	bus.AddHandlerCtx("sql", RecordNotificationJournal)
+	bus.AddHandlerCtx("sql", GetLatestNotification)
+	bus.AddHandlerCtx("sql", CleanNotificationJournal)
 }
 
 func DeleteAlertNotification(cmd *m.DeleteAlertNotificationCommand) error {
@@ -53,7 +57,9 @@ func GetAlertNotificationsToSend(query *m.GetAlertNotificationsToSendQuery) erro
 										alert_notification.created,
 										alert_notification.updated,
 										alert_notification.settings,
-										alert_notification.is_default
+										alert_notification.is_default,
+										alert_notification.send_reminder,
+										alert_notification.frequency
 										FROM alert_notification
 	  							`)
 
@@ -91,7 +97,9 @@ func getAlertNotificationInternal(query *m.GetAlertNotificationsQuery, sess *DBS
 										alert_notification.created,
 										alert_notification.updated,
 										alert_notification.settings,
-										alert_notification.is_default
+										alert_notification.is_default,
+										alert_notification.send_reminder,
+										alert_notification.frequency
 										FROM alert_notification
 	  							`)
 
@@ -137,17 +145,31 @@ func CreateAlertNotificationCommand(cmd *m.CreateAlertNotificationCommand) error
 			return fmt.Errorf("Alert notification name %s already exists", cmd.Name)
 		}
 
-		alertNotification := &m.AlertNotification{
-			OrgId:     cmd.OrgId,
-			Name:      cmd.Name,
-			Type:      cmd.Type,
-			Settings:  cmd.Settings,
-			Created:   time.Now(),
-			Updated:   time.Now(),
-			IsDefault: cmd.IsDefault,
+		var frequency time.Duration
+		if cmd.SendReminder {
+			if cmd.Frequency == "" {
+				return m.ErrNotificationFrequencyNotFound
+			}
+
+			frequency, err = time.ParseDuration(cmd.Frequency)
+			if err != nil {
+				return err
+			}
 		}
 
-		if _, err = sess.Insert(alertNotification); err != nil {
+		alertNotification := &m.AlertNotification{
+			OrgId:        cmd.OrgId,
+			Name:         cmd.Name,
+			Type:         cmd.Type,
+			Settings:     cmd.Settings,
+			SendReminder: cmd.SendReminder,
+			Frequency:    frequency,
+			Created:      time.Now(),
+			Updated:      time.Now(),
+			IsDefault:    cmd.IsDefault,
+		}
+
+		if _, err = sess.MustCols("send_reminder").Insert(alertNotification); err != nil {
 			return err
 		}
 
@@ -179,16 +201,77 @@ func UpdateAlertNotification(cmd *m.UpdateAlertNotificationCommand) error {
 		current.Name = cmd.Name
 		current.Type = cmd.Type
 		current.IsDefault = cmd.IsDefault
+		current.SendReminder = cmd.SendReminder
 
-		sess.UseBool("is_default")
+		if current.SendReminder {
+			if cmd.Frequency == "" {
+				return m.ErrNotificationFrequencyNotFound
+			}
+
+			frequency, err := time.ParseDuration(cmd.Frequency)
+			if err != nil {
+				return err
+			}
+
+			current.Frequency = frequency
+		}
+
+		sess.UseBool("is_default", "send_reminder")
 
 		if affected, err := sess.ID(cmd.Id).Update(current); err != nil {
 			return err
 		} else if affected == 0 {
-			return fmt.Errorf("Could not find alert notification")
+			return fmt.Errorf("Could not update alert notification")
 		}
 
 		cmd.Result = &current
 		return nil
 	})
 }
+
+func RecordNotificationJournal(ctx context.Context, cmd *m.RecordNotificationJournalCommand) error {
+	return inTransactionCtx(ctx, func(sess *DBSession) error {
+		journalEntry := &m.AlertNotificationJournal{
+			OrgId:      cmd.OrgId,
+			AlertId:    cmd.AlertId,
+			NotifierId: cmd.NotifierId,
+			SentAt:     cmd.SentAt,
+			Success:    cmd.Success,
+		}
+
+		if _, err := sess.Insert(journalEntry); err != nil {
+			return err
+		}
+
+		return nil
+	})
+}
+
+func GetLatestNotification(ctx context.Context, cmd *m.GetLatestNotificationQuery) error {
+	return inTransactionCtx(ctx, func(sess *DBSession) error {
+		nj := &m.AlertNotificationJournal{}
+
+		_, err := sess.Desc("alert_notification_journal.sent_at").
+			Limit(1).
+			Where("alert_notification_journal.org_id = ? AND alert_notification_journal.alert_id = ? AND alert_notification_journal.notifier_id = ?", cmd.OrgId, cmd.AlertId, cmd.NotifierId).Get(nj)
+
+		if err != nil {
+			return err
+		}
+
+		if nj.AlertId == 0 && nj.Id == 0 && nj.NotifierId == 0 && nj.OrgId == 0 {
+			return m.ErrJournalingNotFound
+		}
+
+		cmd.Result = nj
+		return nil
+	})
+}
+
+func CleanNotificationJournal(ctx context.Context, cmd *m.CleanNotificationJournalCommand) error {
+	return inTransactionCtx(ctx, func(sess *DBSession) error {
+		sql := "DELETE FROM alert_notification_journal WHERE alert_notification_journal.org_id = ? AND alert_notification_journal.alert_id = ? AND alert_notification_journal.notifier_id = ?"
+		_, err := sess.Exec(sql, cmd.OrgId, cmd.AlertId, cmd.NotifierId)
+		return err
+	})
+}

pkg/services/sqlstore/alert_notification_test.go

@@ -1,7 +1,9 @@
 package sqlstore
 
 import (
+	"context"
 	"testing"
+	"time"
 
 	"github.com/grafana/grafana/pkg/components/simplejson"
 	m "github.com/grafana/grafana/pkg/models"
@@ -11,7 +13,48 @@ import (
 func TestAlertNotificationSQLAccess(t *testing.T) {
 	Convey("Testing Alert notification sql access", t, func() {
 		InitTestDB(t)
-		var err error
+
+		Convey("Alert notification journal", func() {
+			var alertId int64 = 5
+			var orgId int64 = 5
+			var notifierId int64 = 5
+
+			Convey("Getting last journal should raise error if no one exists", func() {
+				query := &m.GetLatestNotificationQuery{AlertId: alertId, OrgId: orgId, NotifierId: notifierId}
+				err := GetLatestNotification(context.Background(), query)
+				So(err, ShouldEqual, m.ErrJournalingNotFound)
+
+				Convey("shoulbe be able to record two journaling events", func() {
+					createCmd := &m.RecordNotificationJournalCommand{AlertId: alertId, NotifierId: notifierId, OrgId: orgId, Success: true, SentAt: 1}
+
+					err := RecordNotificationJournal(context.Background(), createCmd)
+					So(err, ShouldBeNil)
+
+					createCmd.SentAt += 1000 //increase epoch
+
+					err = RecordNotificationJournal(context.Background(), createCmd)
+					So(err, ShouldBeNil)
+
+					Convey("get last journaling event", func() {
+						err := GetLatestNotification(context.Background(), query)
+						So(err, ShouldBeNil)
+						So(query.Result.SentAt, ShouldEqual, 1001)
+
+						Convey("be able to clear all journaling for an notifier", func() {
+							cmd := &m.CleanNotificationJournalCommand{AlertId: alertId, NotifierId: notifierId, OrgId: orgId}
+							err := CleanNotificationJournal(context.Background(), cmd)
+							So(err, ShouldBeNil)
+
+							Convey("querying for last junaling should raise error", func() {
+								query := &m.GetLatestNotificationQuery{AlertId: alertId, OrgId: orgId, NotifierId: notifierId}
+								err := GetLatestNotification(context.Background(), query)
+								So(err, ShouldEqual, m.ErrJournalingNotFound)
+							})
+						})
+					})
+				})
+			})
+		})
 
 		Convey("Alert notifications should be empty", func() {
 			cmd := &m.GetAlertNotificationsQuery{
@@ -24,19 +67,75 @@ func TestAlertNotificationSQLAccess(t *testing.T) {
 			So(cmd.Result, ShouldBeNil)
 		})
 
-		Convey("Can save Alert Notification", func() {
+		Convey("Cannot save alert notifier with send reminder = true", func() {
 			cmd := &m.CreateAlertNotificationCommand{
-				Name:     "ops",
-				Type:     "email",
-				OrgId:    1,
-				Settings: simplejson.New(),
+				Name:         "ops",
+				Type:         "email",
+				OrgId:        1,
+				SendReminder: true,
+				Settings:     simplejson.New(),
 			}
 
-			err = CreateAlertNotificationCommand(cmd)
+			Convey("and missing frequency", func() {
+				err := CreateAlertNotificationCommand(cmd)
+				So(err, ShouldEqual, m.ErrNotificationFrequencyNotFound)
+			})
+
+			Convey("invalid frequency", func() {
+				cmd.Frequency = "invalid duration"
+
+				err := CreateAlertNotificationCommand(cmd)
+				So(err.Error(), ShouldEqual, "time: invalid duration invalid duration")
+			})
+		})
+
+		Convey("Cannot update alert notifier with send reminder = false", func() {
+			cmd := &m.CreateAlertNotificationCommand{
+				Name:         "ops update",
+				Type:         "email",
+				OrgId:        1,
+				SendReminder: false,
+				Settings:     simplejson.New(),
+			}
+
+			err := CreateAlertNotificationCommand(cmd)
+			So(err, ShouldBeNil)
+
+			updateCmd := &m.UpdateAlertNotificationCommand{
+				Id:           cmd.Result.Id,
+				SendReminder: true,
+			}
+
+			Convey("and missing frequency", func() {
+				err := UpdateAlertNotification(updateCmd)
+				So(err, ShouldEqual, m.ErrNotificationFrequencyNotFound)
+			})
+
+			Convey("invalid frequency", func() {
+				updateCmd.Frequency = "invalid duration"
+
+				err := UpdateAlertNotification(updateCmd)
+				So(err, ShouldNotBeNil)
+				So(err.Error(), ShouldEqual, "time: invalid duration invalid duration")
+			})
+		})
+
+		Convey("Can save Alert Notification", func() {
+			cmd := &m.CreateAlertNotificationCommand{
+				Name:         "ops",
+				Type:         "email",
+				OrgId:        1,
+				SendReminder: true,
+				Frequency:    "10s",
+				Settings:     simplejson.New(),
+			}
+
+			err := CreateAlertNotificationCommand(cmd)
 			So(err, ShouldBeNil)
 			So(cmd.Result.Id, ShouldNotEqual, 0)
 			So(cmd.Result.OrgId, ShouldNotEqual, 0)
 			So(cmd.Result.Type, ShouldEqual, "email")
+			So(cmd.Result.Frequency, ShouldEqual, 10*time.Second)
 
 			Convey("Cannot save Alert Notification with the same name", func() {
 				err = CreateAlertNotificationCommand(cmd)
@@ -45,25 +144,42 @@ func TestAlertNotificationSQLAccess(t *testing.T) {
 
 			Convey("Can update alert notification", func() {
 				newCmd := &m.UpdateAlertNotificationCommand{
-					Name:     "NewName",
-					Type:     "webhook",
-					OrgId:    cmd.Result.OrgId,
-					Settings: simplejson.New(),
-					Id:       cmd.Result.Id,
+					Name:         "NewName",
+					Type:         "webhook",
+					OrgId:        cmd.Result.OrgId,
+					SendReminder: true,
+					Frequency:    "60s",
+					Settings:     simplejson.New(),
+					Id:           cmd.Result.Id,
 				}
 				err := UpdateAlertNotification(newCmd)
 				So(err, ShouldBeNil)
 				So(newCmd.Result.Name, ShouldEqual, "NewName")
+				So(newCmd.Result.Frequency, ShouldEqual, 60*time.Second)
+			})
+
+			Convey("Can update alert notification to disable sending of reminders", func() {
+				newCmd := &m.UpdateAlertNotificationCommand{
+					Name:         "NewName",
+					Type:         "webhook",
+					OrgId:        cmd.Result.OrgId,
+					SendReminder: false,
+					Settings:     simplejson.New(),
+					Id:           cmd.Result.Id,
+				}
+				err := UpdateAlertNotification(newCmd)
+				So(err, ShouldBeNil)
+				So(newCmd.Result.SendReminder, ShouldBeFalse)
 			})
 		})
 
 		Convey("Can search using an array of ids", func() {
-			cmd1 := m.CreateAlertNotificationCommand{Name: "nagios", Type: "webhook", OrgId: 1, Settings: simplejson.New()}
-			cmd2 := m.CreateAlertNotificationCommand{Name: "slack", Type: "webhook", OrgId: 1, Settings: simplejson.New()}
-			cmd3 := m.CreateAlertNotificationCommand{Name: "ops2", Type: "email", OrgId: 1, Settings: simplejson.New()}
-			cmd4 := m.CreateAlertNotificationCommand{IsDefault: true, Name: "default", Type: "email", OrgId: 1, Settings: simplejson.New()}
+			cmd1 := m.CreateAlertNotificationCommand{Name: "nagios", Type: "webhook", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
+			cmd2 := m.CreateAlertNotificationCommand{Name: "slack", Type: "webhook", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
+			cmd3 := m.CreateAlertNotificationCommand{Name: "ops2", Type: "email", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
+			cmd4 := m.CreateAlertNotificationCommand{IsDefault: true, Name: "default", Type: "email", OrgId: 1, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
 
-			otherOrg := m.CreateAlertNotificationCommand{Name: "default", Type: "email", OrgId: 2, Settings: simplejson.New()}
+			otherOrg := m.CreateAlertNotificationCommand{Name: "default", Type: "email", OrgId: 2, SendReminder: true, Frequency: "10s", Settings: simplejson.New()}
 
 			So(CreateAlertNotificationCommand(&cmd1), ShouldBeNil)
 			So(CreateAlertNotificationCommand(&cmd2), ShouldBeNil)

pkg/services/sqlstore/migrations/alert_mig.go

@@ -65,6 +65,13 @@ func addAlertMigrations(mg *Migrator) {
 	mg.AddMigration("Add column is_default", NewAddColumnMigration(alert_notification, &Column{
 		Name: "is_default", Type: DB_Bool, Nullable: false, Default: "0",
 	}))
+	mg.AddMigration("Add column frequency", NewAddColumnMigration(alert_notification, &Column{
+		Name: "frequency", Type: DB_BigInt, Nullable: true,
+	}))
+	mg.AddMigration("Add column send_reminder", NewAddColumnMigration(alert_notification, &Column{
+		Name: "send_reminder", Type: DB_Bool, Nullable: true, Default: "0",
+	}))
+
 	mg.AddMigration("add index alert_notification org_id & name", NewAddIndexMigration(alert_notification, alert_notification.Indices[0]))
 
 	mg.AddMigration("Update alert table charset", NewTableCharsetMigration("alert", []*Column{
@@ -82,4 +89,22 @@ func addAlertMigrations(mg *Migrator) {
 		{Name: "type", Type: DB_NVarchar, Length: 255, Nullable: false},
 		{Name: "settings", Type: DB_Text, Nullable: false},
 	}))
+
+	notification_journal := Table{
+		Name: "alert_notification_journal",
+		Columns: []*Column{
+			{Name: "id", Type: DB_BigInt, IsPrimaryKey: true, IsAutoIncrement: true},
+			{Name: "org_id", Type: DB_BigInt, Nullable: false},
+			{Name: "alert_id", Type: DB_BigInt, Nullable: false},
+			{Name: "notifier_id", Type: DB_BigInt, Nullable: false},
+			{Name: "sent_at", Type: DB_BigInt, Nullable: false},
+			{Name: "success", Type: DB_Bool, Nullable: false},
+		},
+		Indices: []*Index{
+			{Cols: []string{"org_id", "alert_id", "notifier_id"}, Type: IndexType},
+		},
+	}
+
+	mg.AddMigration("create notification_journal table v1", NewAddTableMigration(notification_journal))
+	mg.AddMigration("add index notification_journal org_id & alert_id & notifier_id", NewAddIndexMigration(notification_journal, notification_journal.Indices[0]))
 }

public/app/features/alerting/notification_edit_ctrl.ts

@@ -11,6 +11,8 @@ export class AlertNotificationEditCtrl {
   model: any;
   defaults: any = {
     type: 'email',
+    sendReminder: false,
+    frequency: '15m',
     settings: {
       httpMethod: 'POST',
       autoResolve: true,
@@ -18,12 +20,17 @@ export class AlertNotificationEditCtrl {
     },
     isDefault: false,
   };
+  getFrequencySuggestion: any;
 
   /** @ngInject */
   constructor(private $routeParams, private backendSrv, private $location, private $templateCache, navModelSrv) {
     this.navModel = navModelSrv.getNav('alerting', 'channels', 0);
     this.isNew = !this.$routeParams.id;
 
+    this.getFrequencySuggestion = () => {
+      return ['1m', '5m', '10m', '15m', '30m', '1h'];
+    };
+
     this.backendSrv
       .get(`/api/alert-notifiers`)
       .then(notifiers => {
@@ -102,6 +109,7 @@ export class AlertNotificationEditCtrl {
     const payload = {
       name: this.model.name,
       type: this.model.type,
+      frequency: this.model.frequency,
       settings: this.model.settings,
     };
 

public/app/features/alerting/partials/notification_edit.html

@@ -32,6 +32,29 @@
           checked="ctrl.model.settings.uploadImage"
           tooltip="Captures an image and include it in the notification">
       </gf-form-switch>
+      <gf-form-switch
+          class="gf-form"
+          label="Send reminders"
+          label-class="width-12"
+          checked="ctrl.model.sendReminder"
+          tooltip="Send additional notifications for triggered alerts">
+      </gf-form-switch>
+      <div class="gf-form-inline">
+        <div class="gf-form" ng-if="ctrl.model.sendReminder">
+          <span class="gf-form-label width-12">Send reminder every
+            <info-popover mode="right-normal" position="top center">
+              Specify how often reminders should be sent, e.g. every 30s, 1m, 10m, 30m or 1h etc.
+            </info-popover>
+          </span>
+          <input type="text" placeholder="Select or specify custom" class="gf-form-input width-15" ng-model="ctrl.model.frequency"
+            bs-typeahead="ctrl.getFrequencySuggestion" data-min-length=0 ng-required="ctrl.model.sendReminder">
+        </div>
+      </div>
+      <div class="gf-form">
+          <span class="alert alert-info width-30" ng-if="ctrl.model.sendReminder">
+            Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.
+          </span>
+        </div>
     </div>
 
     <div class="gf-form-group" ng-include src="ctrl.notifierTemplateId">