mirror of
https://github.com/grafana/grafana.git
synced 2024-11-23 09:26:43 -06:00
docs(): alerting docs, #6490
This commit is contained in:
Torkel Ödegaard
parent
8713dd8221
commit
fd09f17cd3
@ -12,70 +12,109 @@ weight = 2
|
||||
|
||||
# Alert Notifications
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alert_notifications_menu.png" caption="Alerting notifications" >}}
|
||||
|
||||
> Alerting is only available in Grafana v4.0 and above.
|
||||
|
||||
When an alert changes state it sends out notifications. Each alert rule can have
|
||||
multiple notifications. But in order to add a notification to an alert rule you first need
|
||||
to add and configure a `notification` object. This is done from the Alerting/Notifications page.
|
||||
|
||||
When an alert changes state it sends a notification. One alert can be associated with multiple notifications. You can also configure notifications to be sent for all alerts within Grafana to make sure you won’t miss to configure notifications for an alert.
|
||||
## Notification Setup
|
||||
|
||||
On the notifications list page hit the `New Notification` button to go the the page where you
|
||||
can configure and setup a new notification.
|
||||
|
||||
You can find the alert notification page in the main menu under alerting.
|
||||
You you specify name and type, and type specific options. You can also test the notification to make
|
||||
sure it's working and setup correctly.
|
||||
|
||||
<!-- You can reach this page from the Alerting submenu or Alert List page header. -->
|
||||
<!-- When you configure a notification you can have it be a global notifiations, meaning -->
|
||||
<!-- it will be sent for all alerts within Grafana. This is useful to make sure you won’t miss to configure -->
|
||||
<!-- notifications for an alert. You can find the alert notification page in the main menu under alerting. -->
|
||||
<!-- -->
|
||||
|
||||
## Add a notifications to an Alert
|
||||
You can add and remove notifications from an alert by going to the `Notifications` sub menu in the alerting tab.
|
||||
<!-- ## Add a notifications to an Alert -->
|
||||
<!-- You can add and remove notifications from an alert by going to the `Notifications` sub menu in the alerting tab. -->
|
||||
<!-- -->
|
||||
<!-- -->
|
||||
<!-- <img class="no-shadow" src="/img/docs/v4/alerttab_notifications_submenu.png"> -->
|
||||
<!-- -->
|
||||
<!-- -->
|
||||
<!-- Click the `+` button to add a new notification and the `x` to remove. Notifications with a blue backgrounds are enabled by default for all alerts and cannot be modified from this view. -->
|
||||
<!-- -->
|
||||
<!-- -->
|
||||
<!-- <img class="no-shadow" src="/img/docs/v4/add_remove_notifications.png"> -->
|
||||
<!-- -->
|
||||
|
||||
### Send on all alerts
|
||||
|
||||
<img class="no-shadow" src="/img/docs/v4/alerttab_notifications_submenu.png">
|
||||
This option will make this notification used for all alert rules, existing and new.
|
||||
|
||||
## Supported notification types
|
||||
|
||||
Click the `+` button to add a new notification and the `x` to remove. Notifications with a blue backgrounds are enabled by default for all alerts and cannot be modified from this view.
|
||||
Grafana ships with a set of notification types. More will be added in future releases.
|
||||
|
||||
|
||||
<img class="no-shadow" src="/img/docs/v4/add_remove_notifications.png">
|
||||
|
||||
|
||||
## Default notifications
|
||||
|
||||
|
||||
On the notifications list page (`/alerting/notifications`) you can see all notifiers that have been enabled by default. To make an notification enabled by default
|
||||
you can check the `Send on all alerts` checkbox on the notification edit page.
|
||||
|
||||
|
||||
## Testing
|
||||
You can test an notification on the edit page by clicking the `Send Test` button. Grafana will then send test data based on the info on the edit page. This is a simple way of making sure that notifications are working as intended.
|
||||
|
||||
## Supported notifiers
|
||||
### Email
|
||||
|
||||
|
||||
To enable email notification you have to setup [SMTP settings](/installation/configuration/#smtp) in the Grafana config.
|
||||
Email notification will upload an image of the alert graph to an external image destination if available or fallback on attaching the image in the email.
|
||||
|
||||
To enable email notification you have to setup [SMTP settings](/installation/configuration/#smtp)
|
||||
in the Grafana config. Email notification will upload an image of the alert graph to an
|
||||
external image destination if available or fallback on attaching the image in the email.
|
||||
|
||||
### Webhook
|
||||
The webhook notification is a simple way to send information about an state change over HTTP to a custom endpoint.
|
||||
Using this notification you could integrated Grafana into any system you choose, by yourself.
|
||||
|
||||
The webhook notification is a simple way to send information about an state change over HTTP to a custom endpoint.
|
||||
Using this notification you could integrated Grafana into any system you choose, by yourself.
|
||||
|
||||
Example json schema:
|
||||
```json
|
||||
{
|
||||
"title": "My alert",
|
||||
"ruleId": 1,
|
||||
"ruleName": "Load peaking!",
|
||||
"ruleUrl": "http://url.to.grafana/db/dashboard/my_dashboard?panelId=2",
|
||||
"state": "Alerting",
|
||||
"imageUrl": "http://s3.image.url",
|
||||
"evalMatches": [
|
||||
{
|
||||
"metric": "requests",
|
||||
"tags": {},
|
||||
"value": 122
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Slack
|
||||
To set up slack you need to configure an incoming webhook url at slack. You can follow their guide for how to do that https://api.slack.com/incoming-webhooks
|
||||
If you want to include screenshots of the firing alerts in the slack messages you have to configure the [external image destination](/alerting/notifications/#graph-screenshots-in-notifiations) in Grafana.
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/slack_notification.png" caption="Alerting Slack Notification" >}}
|
||||
|
||||
To set up slack you need to configure an incoming webhook url at slack. You can follow their guide for how
|
||||
to do that https://api.slack.com/incoming-webhooks If you want to include screenshots of the firing alerts
|
||||
in the slack messages you have to configure the [external image destination](#external-image-store) in Grafana.
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
Recipient | allows you to override the slack recipient.
|
||||
Mention | make it possible to include a mention in the slack notification sent by Grafana. Ex @here or @channel
|
||||
|
||||
### Pagerduty
|
||||
To set up pagerduty, all you have to do is to provide an api key.
|
||||
### PagerDuty
|
||||
|
||||
To set up PagerDuty, all you have to do is to provide an api key.
|
||||
|
||||
> Our pagerduty integration only support trigger events at the moment. You have to resolve them by yourself.
|
||||
|
||||
|
||||
## Enable screenshot of alerting graphs
|
||||
Some notifiers requires you to configure the [external image uploader](/installation/configuration/#external-image-storage) to be able to upload screenshots of the alerts. This is not a requirement for any notifier within Grafana but we strongly encourage you to use it since the alert notifications give the recipient very valuable information.
|
||||
# Enable images in notifications {#external-image-store}
|
||||
|
||||
Grafan can render the panel associated with the alert rule and include that in the notification. Some types
|
||||
of notifications require that this image be publicly accessable (Slack for example). In order to support
|
||||
images in notifications like Slack Grafana can upload the image to an image store. It currently supports
|
||||
Amazon S3 for this and Webdav. So to set that up you need to configure the
|
||||
[external image uploader](/installation/configuration/#external-image-storage) in your grafana-server ini
|
||||
config file.
|
||||
|
||||
This is not an optional requirement, you can get slack and email notifications without setting this up.
|
||||
|
||||
|
||||
|
||||
|
||||
@ -17,18 +17,23 @@ weight = 1
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/drag_handles_gif.gif" caption="Alerting overview" >}}
|
||||
|
||||
Alerting in Grafana allows you to attach rules to your graph panels. When you save the dashboard
|
||||
Alerting in Grafana allows you to attach rules to your dashboard panels. When you save the dashboard
|
||||
Grafana will extract the alert rules into a separate alert rule storage and schedule them for evaluation.
|
||||
|
||||
In the alert tab of the graph panel you can configure how often the alert rule should be evaluated
|
||||
and the conditions that need to be met for the alert to change state and trigger its
|
||||
[notifications]({{< relref "/notifications.md" >}}).
|
||||
[notifications]({{< relref "notifications.md" >}}).
|
||||
|
||||
## Execution
|
||||
|
||||
The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
|
||||
of core Grafana. We have not implemented clustering for this scheduler yet, so if you run multiple
|
||||
instances of grafana-server you have to make sure [execute_alerts]({{< relref "/installation/configuration.md#alerting" >}})
|
||||
of core Grafana. Only some data soures are supported right now. They include `Graphite`, `Prometheus`,
|
||||
`InfluxDB` and `OpenTSDB`.
|
||||
|
||||
### Clustering
|
||||
|
||||
We have not implemented clustering yet. So if you run multiple instances of grafana-server
|
||||
you have to make sure [execute_alerts]({{< relref "/installation/configuration.md#alerting" >}})
|
||||
is true on only one instance or otherwise you will get duplicated notifications.
|
||||
|
||||
<div class="clearfix"></div>
|
||||
@ -38,8 +43,7 @@ is true on only one instance or otherwise you will get duplicated notifications.
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
|
||||
|
||||
Currently only the graph panel supports alert rules but this will be added to the **Singlestat** and **Table**
|
||||
panels as well in a future release. The rule config allows you to specify a name, how often the rule
|
||||
should be evaluated and a series of conditions that all need to be true for the alert to fire.
|
||||
panels as well in a future release.
|
||||
|
||||
### Name & Evaluation interval
|
||||
|
||||
@ -95,6 +99,14 @@ Keep Last State | Keep the current alert rule state, what ever it is.
|
||||
If you an unreliable time series store that where queries sometime timeout or fail randomly you can set this option
|
||||
t `Keep Last State` to basically ignore them.
|
||||
|
||||
## Notifications
|
||||
|
||||
In alert tab you can also specify alert rule notifications along with a detailed messsage about the alert rule.
|
||||
The message can contain anything, information about how you might solve the issue, link to runbook etc.
|
||||
|
||||
The actual notifications are configured and shared between multiple alerts. Read the
|
||||
[Notifications]({{< relref "notifications.md" >}}) guide for how to configure and setup notifications.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alert_test_rule.png" caption="Test Rule" >}}
|
||||
|
||||
@ -14,9 +14,9 @@ type ResultLogEntry struct {
|
||||
}
|
||||
|
||||
type EvalMatch struct {
|
||||
Value float64
|
||||
Metric string
|
||||
Tags map[string]string
|
||||
Value float64 `json:"value"`
|
||||
Metric string `json:"metric"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
}
|
||||
|
||||
type Level struct {
|
||||
|
||||
@ -51,11 +51,11 @@ func (this *WebhookNotifier) Notify(evalContext *alerting.EvalContext) error {
|
||||
|
||||
ruleUrl, err := evalContext.GetRuleUrl()
|
||||
if err == nil {
|
||||
bodyJSON.Set("rule_url", ruleUrl)
|
||||
bodyJSON.Set("ruleUrl", ruleUrl)
|
||||
}
|
||||
|
||||
if evalContext.ImagePublicUrl != "" {
|
||||
bodyJSON.Set("image_url", evalContext.ImagePublicUrl)
|
||||
bodyJSON.Set("imageUrl", evalContext.ImagePublicUrl)
|
||||
}
|
||||
|
||||
body, _ := bodyJSON.MarshalJSON()
|
||||
|
||||
@ -8,6 +8,11 @@
|
||||
<i class="fa fa-info-circle"></i>
|
||||
How to add an alert
|
||||
</a>
|
||||
|
||||
<a class="btn btn-inverse" href="alerting/notifications" >
|
||||
<i class="fa fa-cog"></i>
|
||||
Configure notifications
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="gf-form-group">
|
||||
|
||||
Loading…
Reference in New Issue
Block a user