Docs: Set up alert state history (#71123)

Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>
This commit is contained in:
Gilles De Mey 2023-07-06 17:03:32 +02:00 committed by GitHub
parent 420b19e0e4
commit 6af12b1953
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 78 additions and 23 deletions

View File

@ -79,34 +79,13 @@ Use the State history view to get insight into how your alert instances behave o
### Configure the state history view
**Note:** This applies to Open Source only. There is no configuration required if you are using Grafana Cloud.
To enable the state history view, complete the following steps.
1. Ensure you have a Loki instance running to save your history to.
1. Configure the following settings in your Grafana configuration:
a. Enable the Loki backend and Loki remote URL.
b. Enable the three feature toggles for alert state history.
**Example:**
```
[unified_alerting.state_history]
enabled = true
backend = loki
loki_remote_url = http://localhost:3100
[feature_toggles]
enable = alertStateHistoryLokiSecondary, alertStateHistoryLokiPrimary, alertStateHistoryLokiOnly
```
To enable the state history view, see [Configuring alert state history]({{< relref "../set-up/configure-alert-state-history/index.md" >}}).
### View state history
To use the State history view, complete the following steps.
1. Navigate to **Alerts&IRM** -> **Alerting** -> **Alert rules**.
1. Navigate to **Alerts & IRM** -> **Alerting** -> **Alert rules**.
1. Click an alert rule.
1. Select **Show state history**.

View File

@ -0,0 +1,76 @@
---
title: Configure Alert State History
description: Configure Alert State History
keywords:
- grafana
- alerting
- set up
- configure
- alert state history
labels:
products:
- enterprise
- oss
weight: 600
---
# Configure Alert State History
{{% admonition type="note" %}}
This applies to Open Source only. There is no configuration required if you are using Grafana Cloud.
{{% /admonition %}}
Starting with Grafana 10, Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki instance.
This allows you to explore the behavior of your alert rules in the Grafana explore view and levels up the existing state history modal with a powerful new visualisation.
<!-- image here, maybe the one from the blog? -->
## Configuring Loki
To set up alert state history, make sure to have a Loki instance Grafana can write data to. The default settings might need some tweaking as the state history modal might query up to 30 days of data.
The following change to the default configuration should work for most instances, but we recommend looking at the full Loki configuration settings and adjust according to your needs.
As this might impact the performances of an existing Loki instance, we recommend using a separate Loki instance for the alert state history.
```yaml
limits_config:
split_queries_by_interval: '24h'
max_query_parallelism: 32
```
## Configuring Grafana
We need some additional configuration in the Grafana configuration file to have it working with the alert state history.
The example below instructs Grafana to write alert state history to a local Loki instance:
```toml
[unified_alerting.state_history]
enabled = true
backend = "loki"
loki_remote_url = "http://localhost:3100"
[feature_toggles]
enable = alertStateHistoryLokiSecondary, alertStateHistoryLokiPrimary, alertStateHistoryLokiOnly
```
<!-- TODO can we add some more info here about the feature flags and the various different supported setups with Loki as Primary / Secondary, etc? -->
## Adding the Loki data source
See our instructions on [adding a data source](/docs/grafana/latest/administration/data-source-management/).
## Querying the history
If everything is set up correctly you can use the Grafana Explore view to start querying the Loki data source.
A simple litmus test to see if data is being written correctly into the Loki instance is the following query:
```logQL
{ from="state-history" } | json
```