mirror of
https://github.com/grafana/grafana.git
synced 2024-11-25 18:30:41 -06:00
docs(): added what new in v4 article and initial draf of alerting docs, #6490
This commit is contained in:
Torkel Ödegaard
parent
8ca18b511f
commit
b658f38036
@ -1,19 +0,0 @@
|
||||
+++
|
||||
title = "Configuring Alert Rules"
|
||||
description = "Configuring Alert Rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
name = "Alert Rules"
|
||||
parent = "alerting"
|
||||
weight = 1
|
||||
+++
|
||||
|
||||
# Alert Rules
|
||||
|
||||
> Alerting is only available in Grafana v4.0 and above.
|
||||
|
||||
|
||||
hk
|
||||
|
||||
|
||||
131
docs/sources/alerting/rules.md
Normal file
131
docs/sources/alerting/rules.md
Normal file
@ -0,0 +1,131 @@
|
||||
+++
|
||||
title = "Alerting Engine Rules Guide"
|
||||
description = "Configuring Alert Rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
name = "Engine & Rules"
|
||||
parent = "alerting"
|
||||
weight = 1
|
||||
+++
|
||||
|
||||
# Alerting Engine & Rules Guide
|
||||
|
||||
> Alerting is only available in Grafana v4.0 and above.
|
||||
|
||||
## Introduction
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/drag_handles_gif.gif" caption="Alerting overview" >}}
|
||||
|
||||
Alerting in Grafana allows you to attach rules to your graph panels. When you save the dashboard
|
||||
Grafana will extract the alert rules into a seperate alert rule storage and schedule them for evaluation.
|
||||
|
||||
In the alert tab of the graph panel you can configure how often the alert rule should be evaluated
|
||||
and the conditions that need to be met for the alert to change state and trigger its
|
||||
[notifications]({{< relref "/notifications.md" >}}).
|
||||
|
||||
## Execution
|
||||
|
||||
The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
|
||||
of core Grafana. We have not implemented clustering for this scheduler yet, so if you run multiple
|
||||
instances of grafana-server you have to make sure [execute_alerts]({{< relref "/installation/configuration.md#alerting" >}})
|
||||
is true on only one instance or otherwise you will get duplicated notifications.
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
## Rule Config
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
|
||||
|
||||
Currently only the graph panel supports alert rules but this will be added to the **Singlestat** and **Table**
|
||||
panels as well in a future release. The rule config allows you to specify a name, how often the rule
|
||||
should be evaluated and a series of conditions that all need to be true for the alert to fire.
|
||||
|
||||
### Name & Evaluation interval
|
||||
|
||||
Here you can specify the name of the alert rule and how often the scheduler should evaluate the alert rule.
|
||||
|
||||
### Conditions
|
||||
|
||||
Currently the only condition type that exists is a `Query` condition that allows you to
|
||||
specify a query letter, time range and an aggregation function. The letter refers to
|
||||
a query you already have added in the **Metrics** tab. The result from the query and the aggregation function is
|
||||
a single value that is then used in the threshold check. The query used in an alert rule cannot
|
||||
contain any template variables. Currently we only support `AND` operator between conditions.
|
||||
|
||||
We plan to add other condition types in the future, like `Other Alert`, where you can include the state
|
||||
of another alert in your conditions, and `Time Of Day`.
|
||||
|
||||
#### Multiple Series
|
||||
|
||||
If a query returns multiple series then the aggregation function and threshold check will be evaluated for each series.
|
||||
What Grafana does not do currently is track alert rule state **per series**. This has implications that is exemplified
|
||||
in the scenario below.
|
||||
|
||||
- Alert condition with query that returns 2 series: **server1** and **server2**
|
||||
- **server1** series cause the alert rule to fire and switch to state `Alerting`
|
||||
- Notifications are sent out with message: _load peaking (server1)_
|
||||
- In a subsequence evaluation of the same alert rule the **server2** series also cause the alert rule to fire
|
||||
- No new notifications are sent as the alert rule is already in state `Alerting`.
|
||||
|
||||
So as you can see from the above scenario Grafana will not send out notifications when other series cause the alert
|
||||
to fire if the rule already is in state ´Alerting`. To improve support for queries that return multiple series
|
||||
we plan to track state **per series** in a future release.
|
||||
|
||||
### No Data / Null values
|
||||
|
||||
Below you condition you can configure how the rule evaluation engine should handle queries that return no data or only null valued
|
||||
data.
|
||||
|
||||
No Data Option | Description
|
||||
------------ | -------------
|
||||
NoData | Set alert rule state to `NoData`
|
||||
Alerting | Set alert rule state to `Alerting`
|
||||
Keep Last State | Keep the current alert rule state, what ever it is.
|
||||
|
||||
### Execution errors or timeouts
|
||||
|
||||
The last option is how to handle execution or timeout errors.
|
||||
|
||||
Error or timeout option | Description
|
||||
------------ | -------------
|
||||
Alerting | Set alert rule state to `Alerting`
|
||||
Keep Last State | Keep the current alert rule state, what ever it is.
|
||||
|
||||
If you an unreliable time series store that where queries sometime timesout or fail randomly you can set this option
|
||||
t `Keep Last State` to basically ignore them.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alert_test_rule.png" caption="Test Rule" >}}
|
||||
|
||||
First level of troubleshooting you can do is hit the **Test Rule** button. You will get result back that you can expand
|
||||
to the point where you can see the raw data that was returned form your query.
|
||||
|
||||
Further troubleshooting can also be done by inspecting the grafana-server log. If it's not an error or for some reason
|
||||
the log does not say anything you can enable debug logging for some relevant components. This is done
|
||||
in grafana's ini config file.
|
||||
|
||||
Example showing loggers that could be relevant when troubleshooting alerting.
|
||||
|
||||
```ini
|
||||
[log]
|
||||
filters = alerting.scheduler:debug \
|
||||
alerting.engine:debug \
|
||||
alerting.resultHandler:debug \
|
||||
alerting.evalHandler:debug \
|
||||
alerting.evalContext:debug \
|
||||
alerting.extractor:debug \
|
||||
alerting.notifier:debug \
|
||||
alerting.notifier.slack:debug \
|
||||
alerting.notifier.pagerduty:debug \
|
||||
alerting.notifier.email:debug \
|
||||
alerting.notifier.webhook:debug \
|
||||
tsdb.graphite:debug \
|
||||
tsdb.prometheus:debug \
|
||||
tsdb.opentsdb:debug \
|
||||
tsdb.influxdb:debug \
|
||||
```
|
||||
|
||||
If you want to see raw query sent to Graphite and raw response in log you also have to set grafana.ini option `app_mode` to
|
||||
`development`.
|
||||
180
docs/sources/guides/whats-new-in-v4.md
Normal file
180
docs/sources/guides/whats-new-in-v4.md
Normal file
@ -0,0 +1,180 @@
|
||||
+++
|
||||
title = "What's New in Grafana v4.0"
|
||||
description = "Feature & improvement highlights for Grafana v4.0"
|
||||
keywords = ["grafana", "new", "documentation", "4.0"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
name = "Version 4.0"
|
||||
identifier = "v4.0"
|
||||
parent = "whatsnew"
|
||||
weight = -1
|
||||
+++
|
||||
|
||||
# What's New in Grafana v4.0
|
||||
|
||||
As usual this release contains a ton of minor new features, fixes and improved UX. But on top of the usual new goodies
|
||||
is a core new feature: Alerting! Read on below for a detailed description of what's new in v4.0.
|
||||
|
||||
## Alerting
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/drag_handles_gif.gif" caption="Alerting overview" >}}
|
||||
|
||||
Alerting is a really revolutionary feature for Grafana. It transforms Grafana from a
|
||||
visualization tool into a truly mission critical monitoring tool. The alert rules are very easy to
|
||||
configure using your existing graph panels and threshold levels can be set simply by dragging handles to
|
||||
the right side of the graph. The rules will continually be evaluated by grafana-server and
|
||||
notifications will be sent out when the rule conditions are met.
|
||||
|
||||
This feature has been worked on for over a year with many iterations and rewrites
|
||||
just to make sure the foundations are really solid. We are really proud to finally release it!
|
||||
Since the alerting execution is processed in the backend all data source plugins are not supported.
|
||||
Right now Graphite, Prometheus, InfluxDB and OpenTSDB are supported. Elasticsearch is being worked
|
||||
on but will be not ready for v4 release.
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
### Rules
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
|
||||
|
||||
The rule config allows you to specify a name, how often the rule should be evaluated and a series
|
||||
of conditions that all need to be true for the alert to fire.
|
||||
|
||||
Currently the only condition type that exists is a `Query` condition that allows you to
|
||||
specify a query letter, time range and an aggregation function. The letter refers to
|
||||
a query you already have added in the **Metrics** tab. The result from the
|
||||
query and the aggregation function is a single value that is then used in the threshold check.
|
||||
|
||||
We plan to add other condition types in the future, like `Other Alert`, where you can include the state
|
||||
of another alert in your conditions, and `Time Of Day`.
|
||||
|
||||
### Notifications
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/slack_notification.png" caption="Alerting Slack Notification" >}}
|
||||
|
||||
Alerting would not be very useful if there was no way to send notifications when rules trigger and change state. You
|
||||
can setup notifications of different types. We currently have `Slack`, `PagerDuty`, `Email` and `Webhook` with more in the
|
||||
pipe that will be added during beta period. The notifications can then be added to your alert rules.
|
||||
If you have configured an external image store in the grafana.ini config file (s3 and webdav options available)
|
||||
you can get very rich notifications with an image of the graph and the metric
|
||||
values all included in the notification.
|
||||
|
||||
### Annotations
|
||||
|
||||
Alert state changes are recorded in a new annotation store that is built into Grafana. This store
|
||||
currently only supports storing annotations in Grafana's own internal database (mysql, postgres or sqlite).
|
||||
The Grafana annotation storage is currently only used for alert state changes but we hope to add the ability for users
|
||||
to add graph comments in the form of annotations directly from within Grafana in a future release.
|
||||
|
||||
### Alert List Panel
|
||||
|
||||
{{< imgbox max-width="30%" img="/img/docs/v4/alert_list_panel.png" caption="Alert List Panel" >}}
|
||||
|
||||
This new panel allows you to show alert rules or a history of alert rule state changes. You can filter based on states your
|
||||
interested in. Very useful panel for overview style dashboards.
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
## Ad-hoc filter variable
|
||||
|
||||
{{< imgbox max-width="30%" img="/img/docs/v4/adhoc_filters.gif" caption="Ad-hoc filters variable" >}}
|
||||
|
||||
This is a new and very different type of template variable. It will allow you to create new key/value filters on the fly.
|
||||
With autocomplete for both key and values. The filter condition will be automatically applied to all
|
||||
queries that use that data source. This feature opens up more exploratory dashboards. In the gif animation to the right
|
||||
you have a dashboard for Elasticsearch log data. It uses one query variable that allow you to quickly change how the data
|
||||
is grouped, and an interval variable for controlling the granularity of the time buckets. What was missing
|
||||
was a way to dynamically apply filters to the log query. With the `Ad-Hoc Filters` variable you can
|
||||
dynamically add filters to any log property!
|
||||
|
||||
## UX Improvements
|
||||
|
||||
We always try to bring some UX/UI refinements & polish in every release.
|
||||
|
||||
### TV-mode & Kiosk mode
|
||||
|
||||
|
||||
<div class="row">
|
||||
<div class="medium-6 columns">
|
||||
<p>
|
||||
Grafana is so often used on wall mounted TVs that we figured a clean TV mode would be
|
||||
really nice. In TV mode the top navbar, row & panel controls will all fade to transparent.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
This happens automatically after one minute of user inactivity but can also be toggled manually
|
||||
with the <code>d v</code> sequence shortcut. Any mouse movement or keyboard action will
|
||||
restore navbar & controls.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Another feature is the kiosk mode. This can be enabled with <code>d k</code>
|
||||
shortcut or by adding <code>&kiosk</code> to the URL when you load a dashboard.
|
||||
In kiosk mode the navbar is completely hidden/removed from view.
|
||||
</p>
|
||||
</div>
|
||||
<div class="medium-6 columns">
|
||||
{{< lightboxhelper max-width="100%" img="/img/docs/v4/tvmode.png" caption="TV mode" >}}
|
||||
<video width="320" height="240" controls>
|
||||
<source src="/assets/videos/tvmode.mp4" type="video/mp4">
|
||||
Your browser does not support the video tag.
|
||||
</video>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
### New row menu & add panel experience
|
||||
|
||||
{{< imgbox max-width="50%" img="/img/docs/v4/add_panel.gif" caption="Add Panel flow" >}}
|
||||
|
||||
We spent a lot of time improving the dashboard building experience. Trying to make it both
|
||||
more efficient and easier for beginners. After many good but not great experiments
|
||||
with a `build mode` we eventually decided to just improve the green row menu and
|
||||
continue work on a `build mode` for a future release.
|
||||
|
||||
The new row menu automatically slides out when you mouse over the edge of the row. You no longer need
|
||||
to hover over the small green icon and the click it to expand the row menu.
|
||||
|
||||
There is some minor improvements to drag and drop behaviour. Now when dragging a panel from one row
|
||||
to another you will insert the panel and Grafana will automatically make room for it.
|
||||
When you drag a panel within a row you will simply reorder the panels.
|
||||
|
||||
If you look at the animation to the right you can see that you can drag and drop a new panel. This is not
|
||||
required, you can also just click the panel type and it will be inserted at the end of the row
|
||||
automatically. Dragging a new panel has an advantage in that you can insert a new panel where ever you want
|
||||
not just at the end of the row.
|
||||
|
||||
We plan to further improve dashboard building in the future with a more rich grid & layout system.
|
||||
|
||||
### Keyboard shortcuts
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v4/shortcuts.png" caption="Shortcuts" >}}
|
||||
|
||||
Grafana v4 introduces a number of really powerful keyboard shortcuts. You can now focus a panel
|
||||
by hovering over it with your mouse. With a panel focused you can simple hit `e` to toggle panel
|
||||
edit mode, or `v` to toggle fullscreen mode. `p r` removes the panel. `p s` opens share
|
||||
modal.
|
||||
|
||||
Some nice navigation shortcuts are:
|
||||
|
||||
- `g h` for go to home dashboard
|
||||
- `s s` open search with starred pre-selected
|
||||
- `s t` open search in tags list view
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
## Upgrade & Breaking changes
|
||||
|
||||
There are no breaking changes. Old dashboards and features should work the same. Grafana-server will automatically upgrade it's db
|
||||
schema on restart. It's advisable to do a backup of Grafana's database before updating.
|
||||
|
||||
If your are using plugins make sure to update your plugins as some might not work perfectly v4.
|
||||
|
||||
You can update plugins using grafana-cli
|
||||
|
||||
grafana-cli plugins update-all
|
||||
|
||||
## Changelog
|
||||
|
||||
Checkout the [CHANGELOG.md](https://github.com/grafana/grafana/blob/master/CHANGELOG.md) file for a complete list
|
||||
of new features, changes, and bug fixes.
|
||||
|
||||
@ -601,7 +601,15 @@ secret key. ex AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
|
||||
|
||||
### url
|
||||
Url to where Grafana will send PUT request with images
|
||||
|
||||
### username
|
||||
basic auth username
|
||||
|
||||
### password
|
||||
basic auth password
|
||||
|
||||
## [alerting]
|
||||
|
||||
### execute_alerts = true
|
||||
|
||||
Makes it possible to turn off alert rule execution.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user