mirror of
https://github.com/grafana/grafana.git
synced 2024-11-26 02:40:26 -06:00
Alerting docs: Update Recording Rules (#95435)
* WIP - Restructure recording rule pages * Add more `Recording rules` use cases * Update `Intro > Alert rules`. Minor style changes and updates to the recording rules section * Minor verb change * Switch `aliases` and fix broken link * Bold for clarity * Complete `Create recording rules in Grafana Alerting` * Complete data source-managed recording rules * Final update
This commit is contained in:
Pepe Cano
GitHub
parent
2f40fd6741
commit
9b91be739d
@ -14,7 +14,13 @@ labels:
|
||||
- enterprise
|
||||
- oss
|
||||
title: Create alert rules from panels
|
||||
weight: 400
|
||||
weight: 300
|
||||
refs:
|
||||
time-series-visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/time-series/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/visualizations/time-series/
|
||||
---
|
||||
|
||||
## Create alert rules from panels
|
||||
@ -30,7 +36,7 @@ The New alert rule form opens where you can configure and create your alert rule
|
||||
{{% admonition type="note" %}}
|
||||
Changes to the panel aren't reflected on the linked alert rules. If you change a query, you have to update it in both the panel and the alert rule.
|
||||
|
||||
Alert rules are only supported in [time series](ref:time-series) visualizations.
|
||||
Alert rules are only supported in [time series](ref:time-series-visualizations) visualizations.
|
||||
{{% /admonition %}}
|
||||
|
||||
{{< docs/play title="visualizations with linked alerts in Grafana" url="https://play.grafana.org/d/000000074/" >}}
|
||||
|
||||
@ -1,188 +0,0 @@
|
||||
---
|
||||
aliases:
|
||||
- ../unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/
|
||||
- ../unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
description: Create recording rules for an external Grafana Mimir or Loki instance
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
- rules
|
||||
- recording rules
|
||||
- configure
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Configure recording rules
|
||||
weight: 300
|
||||
refs:
|
||||
configure-grafana:
|
||||
- pattern: /docs/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/
|
||||
annotation-label:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/
|
||||
---
|
||||
|
||||
# Configure recording rules
|
||||
|
||||
Recording rules calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
|
||||
|
||||
Grafana-managed recording rules offer the same Prometheus-like semantics, but allow you to query any Grafana data source supported by alerting. In addition to the above, you can use this to import and map data from other data sources into Prometheus.
|
||||
|
||||
For more information on recording rules in Prometheus, refer to [Defining recording rules in Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/).
|
||||
|
||||
## Configure data source-managed recording rules
|
||||
|
||||
Configure data source-managed recording rules.
|
||||
|
||||
### Before you begin
|
||||
|
||||
- Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.
|
||||
|
||||
- For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
|
||||
|
||||
- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
|
||||
|
||||
- **Grafana Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
|
||||
|
||||
To configure data-source managed recording rules, complete the following steps.
|
||||
|
||||
1. Click **Alerts & IRM** -> **Alerting** ->
|
||||
**Alert rules**.
|
||||
1. Scroll to the **Data source-managed section** and click **+New recording rule**.
|
||||
|
||||
#### Enter recording rule name
|
||||
|
||||
The recording rule name must be a Prometheus metric name and contain no whitespace.
|
||||
|
||||
#### Define recording rule
|
||||
|
||||
Select your data source and enter a query. The queries used in data source-managed recording rules always run as instant queries.
|
||||
|
||||
#### Add namespace and group
|
||||
|
||||
1. From the **Namespace** dropdown, select an existing rule namespace or add a new one.
|
||||
|
||||
Namespaces can contain one or more rule groups and only have an organizational purpose.
|
||||
|
||||
1. From the **Group** dropdown, select an existing group within the selected namespace or add a new one.
|
||||
|
||||
Newly created rules are appended to the end of the group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.
|
||||
|
||||
#### Add labels
|
||||
|
||||
1. Add custom labels selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
|
||||
|
||||
1. Click **Save rule** to save the rule or **Save rule and exit** to save the rule and go back to the Alerting page.
|
||||
|
||||
## Configure Grafana-managed recording rules
|
||||
|
||||
Configure Grafana-managed recording rules.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
In Grafana OSS and Enterprise, you can create Grafana-managed recording rules if you enable the `grafanaManagedRecordingRules` feature flag.
|
||||
For Grafana Cloud, it is enabled by default.
|
||||
|
||||
For more information on enabling feature toggles, refer to [Configure feature toggles](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles).
|
||||
{{< /admonition >}}
|
||||
|
||||
### Before you begin
|
||||
|
||||
- Enable the `grafanaManagedRecordingRules` [feature flag](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/).
|
||||
|
||||
Grafana does not contain an embedded time-series database. You must bring your own Prometheus-compatible database to store series generated by Grafana-managed recording rules.
|
||||
|
||||
First, enable the feature by setting `enabled = true` in the `[recording_rules]` section of the Grafana config .ini. Then, provide the URL of your Prometheus-compatible remote-write endpoint in the `url` field, along with optional credentials or headers.
|
||||
|
||||
```
|
||||
[recording_rules]
|
||||
enabled = true
|
||||
url = http://my-example-prometheus.local:9090/api/prom/push
|
||||
basic_auth_username = my-user
|
||||
basic_auth_password = my-pass
|
||||
|
||||
[recording_rules.custom_headers]
|
||||
X-My-Header = MyValue
|
||||
```
|
||||
|
||||
You must provide a URL if `enabled` is set to `true`.
|
||||
|
||||
To configure Grafana-managed recording rules, complete the following steps.
|
||||
|
||||
1. Click **Alerts & IRM** -> **Alerting** ->
|
||||
**Alert rules**.
|
||||
1. Scroll to the **Grafana-managed section** and click **+New recording rule**.
|
||||
|
||||
#### Enter a recording rule and metric name
|
||||
|
||||
Enter a names to identify your recording rule and metric. The metric name must be a Prometheus metric name and contain no whitespace.
|
||||
|
||||
For more information, refer to [Metrics and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
||||
|
||||
#### Define recording rule
|
||||
|
||||
Define a query to get the data you want to measure and set the recording rule output.
|
||||
|
||||
1. Select a data source.
|
||||
1. From the **Options** dropdown, specify a time range.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`.
|
||||
|
||||
It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
|
||||
{{< /admonition >}}
|
||||
|
||||
1. Add a query.
|
||||
|
||||
To add multiple queries, click **Add query**.
|
||||
|
||||
2. Add one or more [expressions].
|
||||
|
||||
a. For each expression, select either **Classic condition** to create a single recording rule, or choose from the **Math**, **Reduce**, and **Resample** options.
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.
|
||||
{{% /admonition %}}
|
||||
|
||||
b. Click **Preview** to verify that the expression is successful.
|
||||
|
||||
3. Click **Set as recording rule output** on the query or expression you want to set as your rule output.
|
||||
|
||||
#### Set evaluation behavior
|
||||
|
||||
Use recording rule evaluation to determine how frequently a recording rule should be evaluated.
|
||||
|
||||
To do this, you need to make sure that your recording rule is in the right evaluation group with an interval that works best for your use case.
|
||||
|
||||
1. Select a folder or click **+ New folder**.
|
||||
1. Select an evaluation group or click **+ New evaluation group**.
|
||||
|
||||
If you are creating a new evaluation group, specify the interval for the group.
|
||||
|
||||
All rules within the same group are evaluated concurrently over the same time interval. Every recording rule in a group uses the same evaluation time, meaning that all queries from the same group are always aligned with each other.
|
||||
|
||||
1. Turn on pause recording rule evaluation, if required.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
You can pause recording rule evaluation.
|
||||
{{< /admonition >}}
|
||||
|
||||
#### Add labels
|
||||
|
||||
1. Add custom labels selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
|
||||
|
||||
1. Click **Save rule** to save the rule or **Save rule and exit** to save the rule and go back to the Alerting page.
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
[min_interval](ref:configure-grafana) sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules are adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.
|
||||
|
||||
This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.
|
||||
|
||||
This setting applies to both Grafana-managed alert and recording rules.
|
||||
@ -1,8 +1,8 @@
|
||||
---
|
||||
aliases:
|
||||
- ../unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/
|
||||
- ../unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
- ../unified-alerting/alerting-rules/create-mimir-loki-managed-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-rule/
|
||||
- ../unified-alerting/alerting-rules/edit-cortex-loki-namespace-group/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/edit-cortex-loki-namespace-group/
|
||||
- ../unified-alerting/alerting-rules/edit-mimir-loki-namespace-group/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/edit-mimir-loki-namespace-group/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-mimir-loki-managed-rule/
|
||||
description: Configure data source-managed alert rules alert for an external Grafana Mimir or Loki instance
|
||||
keywords:
|
||||
|
||||
@ -0,0 +1,54 @@
|
||||
---
|
||||
aliases:
|
||||
- ../fundamentals/alert-rules/recording-rules/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/recording-rules/
|
||||
- ../unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/
|
||||
- ../unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
- ../alerting-rules/create-mimir-loki-managed-recording-rule/ # /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/
|
||||
description: Recording rules allow you to pre-compute frequently needed or computationally expensive expressions and save the results as a new set of time series. Querying precomputed results is faster and can reduce system load.
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
- rules
|
||||
- recording rules
|
||||
- configure
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Create recording rules
|
||||
weight: 400
|
||||
refs:
|
||||
grafana-managed-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/
|
||||
data-source-managed-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-data-source-managed-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/create-data-source-managed-recording-rules/
|
||||
---
|
||||
|
||||
# Configure recording rules
|
||||
|
||||
Recording rules allows you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.
|
||||
|
||||
For instance, you can create a recording rule generating a new metric, `error_9001_count`, which counts occurrences of a specific log error within one minute. Then, query the `error_9001_count` metric in dashboards and alert rules.
|
||||
|
||||
Recording rules can be helpful in various scenarios, such as:
|
||||
|
||||
- **Faster queries** are needed: Performing heavy aggregations or querying large data sets is quicker with precomputed results than real-time queries.
|
||||
- **Reducing system load:** Precomputing specific queries in advance can reduce system overload caused by multiple simultaneous queries.
|
||||
- **Simplifying complex aggregations:** Create a new metric from complex aggregations to facilitate alert and dashboard setup.
|
||||
- **Reusing queries across alerts:** Improve efficiency by reusing the same query across similar alert rules and dashboards.
|
||||
|
||||
The evaluation group of the recording rule determines how often the metric is pre-computed.
|
||||
|
||||
Similar to alert rules, Grafana supports two types of recording rules:
|
||||
|
||||
1. [Grafana-managed recording rules](ref:grafana-managed-recording-rules), which can query any Grafana data source supported by alerting.
|
||||
2. [Data source-managed recording rules](ref:data-source-managed-recording-rules), which can query Prometheus-based data sources like Mimir or Loki.
|
||||
@ -0,0 +1,79 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/create-data-source-managed-recording-rules/
|
||||
description: Recording rules allow you to pre-compute expensive queries in advance and save the results as a new set of time series. Data source-managed recording rules can create a recording rule for Prometheus-based data sources like Mimir or Loki.
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
- rules
|
||||
- recording rules
|
||||
- configure
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Create data source-managed recording rules
|
||||
weight: 402
|
||||
refs:
|
||||
create-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/
|
||||
---
|
||||
|
||||
# Create data source-managed recording rules
|
||||
|
||||
[Recording rules](ref:create-recording-rules) allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.
|
||||
|
||||
Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.
|
||||
|
||||
Data source-managed recording rules can query Prometheus-based data sources like Mimir or Loki. For more information on recording rules in Prometheus, refer to [Defining recording rules in Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/).
|
||||
|
||||
Note that in data source-managed groups, the alert rules and recording rules within the same evaluation group are evaluated sequentially. This is useful to ensure that a recording rule is evaluated before any other alert rule queries the pre-computed metric.
|
||||
|
||||
## Before you begin
|
||||
|
||||
- Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.
|
||||
|
||||
- For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
|
||||
|
||||
- **Loki** - The `local` rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
|
||||
|
||||
- **Grafana Mimir** - use the `/prometheus` prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the [Query API](/docs/mimir/latest/operators-guide/reference-http-api/#querier--query-frontend) and [Ruler API](/docs/mimir/latest/operators-guide/reference-http-api/#ruler) are under the same URL. You cannot provide a separate URL for the Ruler API.
|
||||
|
||||
## Add new recording rule
|
||||
|
||||
To create a new data source-managed recording rule:
|
||||
|
||||
1. Click **Alerts & IRM** -> **Alerting** -> **Alert rules**.
|
||||
1. Scroll to the **Data source-managed section** and click **+New recording rule**.
|
||||
|
||||
## Enter recording rule name
|
||||
|
||||
The recording rule name must be a Prometheus metric name and contain no whitespace.
|
||||
|
||||
## Define recording rule
|
||||
|
||||
Select your data source and enter a query. The queries used in data source-managed recording rules always run as instant queries.
|
||||
|
||||
## Add namespace and group
|
||||
|
||||
1. From the **Namespace** dropdown, select an existing rule namespace or add a new one.
|
||||
|
||||
Namespaces can contain one or more rule groups and only have an organizational purpose.
|
||||
|
||||
1. From the **Group** dropdown, select an existing group within the selected namespace or add a new one.
|
||||
|
||||
Newly created rules are appended to the end of the group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.
|
||||
|
||||
## Add labels
|
||||
|
||||
Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.
|
||||
|
||||
## Query the new metric in dashboards or alert rules
|
||||
|
||||
Click **Save rule** or **Save rule and exit** to save the rule.
|
||||
|
||||
Once saved, the new recording metric is available for use in dashboards and alert rules.
|
||||
@ -0,0 +1,151 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/
|
||||
description: Recording rules allow you to pre-compute expensive queries in advance and save the results as a new set of time series. Grafana-managed recording rules can create a recording rule for any data source supported by alerting.
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
- rules
|
||||
- recording rules
|
||||
- configure
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Create Grafana-managed recording rules
|
||||
weight: 401
|
||||
refs:
|
||||
expressions:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries
|
||||
create-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/
|
||||
alerting-data-sources:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/#supported-data-sources
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/#supported-data-sources
|
||||
configure-grafana-min-interval:
|
||||
- pattern: /docs/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#min_interval
|
||||
configure-feature-toggles:
|
||||
- pattern: /docs/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/feature-toggles/
|
||||
---
|
||||
|
||||
# Create Grafana-managed recording rules
|
||||
|
||||
[Recording rules](ref:create-recording-rules) allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.
|
||||
|
||||
Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.
|
||||
|
||||
Grafana does not contain an embedded time-series database to store recording rule results. You must bring your own Prometheus-compatible database to store the series generated by recording rules.
|
||||
|
||||
Grafana-managed recording rules offer the same Prometheus-like semantics but allow you to query [data sources supported by alerting](ref:alerting-data-sources). Additionally, you can use recording rules to import and map data from other data sources into Prometheus.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
|
||||
Grafana-managed recording rules are enabled by default in Grafana Cloud.
|
||||
|
||||
In Grafana OSS and Enterprise, you must enable them by following the [Before you begin](#before-you-begin) instructions.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
To configure Grafana-managed recording rules, complete the following steps.
|
||||
|
||||
## Before you begin
|
||||
|
||||
This section only applies to Grafana OSS and Grafana Enterprise.
|
||||
|
||||
First, enable the `grafanaManagedRecordingRules` [feature flag](ref:configure-feature-toggles).
|
||||
|
||||
Then, enable the feature by setting `enabled = true` in the `[recording_rules]` section of the Grafana config .ini. Provide the URL of your Prometheus-compatible remote-write endpoint in the `url` field, along with optional credentials or headers.
|
||||
|
||||
```
|
||||
[recording_rules]
|
||||
enabled = true
|
||||
url = http://my-example-prometheus.local:9090/api/prom/push
|
||||
basic_auth_username = my-user
|
||||
basic_auth_password = my-pass
|
||||
|
||||
[recording_rules.custom_headers]
|
||||
X-My-Header = MyValue
|
||||
```
|
||||
|
||||
## Add new recording rule
|
||||
|
||||
To create a new Grafana-managed recording rule:
|
||||
|
||||
1. Click **Alerts & IRM** -> **Alerting** ->
|
||||
**Alert rules**.
|
||||
1. Scroll to the **Grafana-managed section** and click **+New recording rule**.
|
||||
|
||||
1. Enter the names to identify your recording rule and metric.
|
||||
|
||||
The metric name must be a Prometheus metric name and contain no whitespace. For details, refer to [Prometheus metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
|
||||
|
||||
## Define recording rule
|
||||
|
||||
Define a query to get the data you want to measure and set the recording rule output.
|
||||
|
||||
1. Select a data source.
|
||||
1. From the **Options** dropdown, specify a time range.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Grafana Alerting only supports fixed relative time ranges, for example, `now-24hr: now`.
|
||||
|
||||
It does not support absolute time ranges: `2021-12-02 00:00:00 to 2021-12-05 23:59:592` or semi-relative time ranges: `now/d to: now`.
|
||||
{{< /admonition >}}
|
||||
|
||||
1. Add a query.
|
||||
|
||||
To add multiple queries, click **Add query**.
|
||||
|
||||
1. Add one or more [expressions](ref:expressions).
|
||||
|
||||
a. For each expression, select either **Classic condition** to create a single recording rule, or choose from the **Math**, **Reduce**, and **Resample** options.
|
||||
|
||||
When using Prometheus, you can use an instant vector and built-in functions, so you don't need to add additional expressions.
|
||||
|
||||
b. Click **Preview** to verify that the expression is successful.
|
||||
|
||||
1. Click **Set as recording rule output** on the query or expression you want to set as your rule output.
|
||||
|
||||
## Set evaluation behavior
|
||||
|
||||
Use recording rule evaluation to determine how frequently a recording rule should be evaluated.
|
||||
|
||||
To do this, you need to make sure that your recording rule is in the right evaluation group with an interval that works best for your use case.
|
||||
|
||||
1. Select a folder or click **+ New folder**.
|
||||
1. Select an evaluation group or click **+ New evaluation group**.
|
||||
|
||||
If you are creating a new evaluation group, specify the interval for the group.
|
||||
|
||||
All rules within the same group are evaluated concurrently over the same time interval. Every recording rule in a group uses the same evaluation time, meaning that all queries from the same group are always aligned with each other.
|
||||
|
||||
1. Before or after creating a recording rule, you have the option to **Pause evaluation** if necessary.
|
||||
|
||||
### Advanced configuration
|
||||
|
||||
[min_interval](ref:configure-grafana-min-interval) sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules are adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.
|
||||
|
||||
This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.
|
||||
|
||||
This setting applies to both Grafana-managed alert and recording rules.
|
||||
|
||||
## Add labels
|
||||
|
||||
Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.
|
||||
|
||||
## Query the new metric in dashboards or alert rules
|
||||
|
||||
Click **Save rule** or **Save rule and exit** to save the rule.
|
||||
|
||||
Once saved, the new recording metric is available for use in dashboards and alert rules.
|
||||
@ -43,7 +43,7 @@ Alert rules in different groups can be evaluated simultaneously.
|
||||
|
||||
- **Grafana-managed** alert rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.
|
||||
|
||||
- **Data-source managed** alert rules within the same group are evaluated sequentially, one after the other—this is necessary to ensure that recording rules are evaluated before alert rules.
|
||||
- **Data-source managed** alert rules within the same group are evaluated sequentially, one after the other—this is useful to ensure that recording rules are evaluated before alert rules.
|
||||
|
||||
## Pending period
|
||||
|
||||
|
||||
@ -3,10 +3,7 @@ aliases:
|
||||
- ../fundamentals/data-source-alerting/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/data-source-alerting/
|
||||
- ../fundamentals/alert-rules/alert-instances/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/alert-instances/
|
||||
- ../fundamentals/alert-rules/organising-alerts/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/organising-alerts/
|
||||
- ../fundamentals/alert-rules/recording-rules/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/recording-rules/
|
||||
- ../fundamentals/alert-rules/alert-rule-types/ # /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/alert-rule-types/
|
||||
- ../unified-alerting/alerting-rules/edit-cortex-loki-namespace-group/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/edit-cortex-loki-namespace-group/
|
||||
- ../unified-alerting/alerting-rules/edit-mimir-loki-namespace-group/ # /docs/grafana/<GRAFANA_VERSION>/alerting/unified-alerting/alerting-rules/edit-mimir-loki-namespace-group/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/
|
||||
description: Learn about alert rules
|
||||
keywords:
|
||||
@ -26,9 +23,6 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/queries-conditions/
|
||||
recorded-queries:
|
||||
- pattern: /docs/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
notification-images:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/images-in-notifications/
|
||||
@ -41,14 +35,19 @@ refs:
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/
|
||||
create-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/
|
||||
expression-queries:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries
|
||||
alert-condition:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#alert-condition
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/queries-conditions/#alert-condition
|
||||
alert-rule-evaluation:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/rule-evaluation/
|
||||
@ -69,54 +68,52 @@ Grafana supports two different alert rule types: Grafana-managed alert rules and
|
||||
|
||||
## Grafana-managed alert rules
|
||||
|
||||
Grafana-managed alert rules are the most flexible alert rule type. They allow you to create alert rules that can act on data from any of the [supported data sources](#supported-data-sources), and use multiple data sources in a single alert rule. You can add [expressions to transform your data](ref:expression-queries) and set custom alert conditions. Using [images in alert notifications](ref:notification-images). is also supported.
|
||||
Grafana-managed alert rules are the most flexible alert rule type. They allow you to create alert rules that can act on data from any of the [supported data sources](#supported-data-sources), and use multiple data sources in a single alert rule.
|
||||
|
||||
{{< figure src="/media/docs/alerting/grafana-managed-alerting-architecture.png" max-width="750px" caption="How Grafana-managed alerting works by default" >}}
|
||||
|
||||
1. Alert rules are created within Grafana based on one or more data sources.
|
||||
1. Alert rules are created within Grafana and query one or more data sources.
|
||||
1. Alert rules are evaluated by the Alert Rule Evaluation Engine from within Grafana.
|
||||
1. Firing and resolved alert instances are forwarded to [handle their notifications](ref:notifications).
|
||||
|
||||
### Supported data sources
|
||||
|
||||
Grafana-managed alert rules can query backend data sources if Grafana Alerting is enabled by specifying `{"backend": true, "alerting": true}` in the [plugin.json](https://grafana.com/developers/plugin-tools/reference/plugin-json).
|
||||
Grafana-managed alert rules can query backend data sources if Grafana Alerting is enabled by specifying `{"backend": true, "alerting": true}` in the `plugin.json` file.
|
||||
|
||||
Find the public data sources supporting Alerting in the [Grafana Plugins directory](/grafana/plugins/data-source-plugins/?features=alerting).
|
||||
|
||||
## Data source-managed alert rules
|
||||
|
||||
Data source-managed alert rules can be used for Grafana Mimir or Grafana Loki data sources which have been configured to support rule creation.
|
||||
Data source-managed alert rules can query Prometheus-based data sources, such as Grafana Mimir or Grafana Loki. Alert rules are stored within the data source when the Ruler API is enabled (e.g., [Mimir Ruler API](/docs/mimir/<GRAFANA_VERSION>/references/http-api/#ruler) or [Loki Ruler API](/docs/loki/<GRAFANA_VERSION>/api/#ruler)).
|
||||
|
||||
They can improve query performance via [recording rules](#recording-rules) and ensure high-availability and fault tolerance when implementing a distributed architecture.
|
||||
|
||||
They are only supported for Grafana Mimir or Grafana Loki data sources with the Ruler API enabled. For more information, refer to the [Loki Ruler API](/docs/loki/<GRAFANA_VERSION>/api/#ruler) or [Mimir Ruler API](/docs/mimir/<GRAFANA_VERSION>/references/http-api/#ruler).
|
||||
In this setup, the distributed architecture can provide high-availability and fault tolerance.
|
||||
|
||||
{{< figure src="/media/docs/alerting/mimir-managed-alerting-architecture-v2.png" max-width="750px" caption="Mimir-managed alerting architecture" >}}
|
||||
|
||||
1. Alert rules are created and stored within the data source itself.
|
||||
1. Alert rules can only query Prometheus-based data. It can use either queries or [recording rules](#recording-rules).
|
||||
1. Alert rules can only query Prometheus-based data.
|
||||
1. Alert rules are evaluated by the Alert Rule Evaluation Engine.
|
||||
1. Firing and resolved alert instances are forwarded to [handle their notifications](ref:notifications).
|
||||
|
||||
## Recording rules
|
||||
|
||||
A recording rule allows you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series. This is useful if you want to run alerts on aggregated data or if you have dashboards that query computationally expensive expressions repeatedly.
|
||||
A recording rule pre-compute frequently used or computationally expensive queries, and saves the results as a new time series metric.
|
||||
|
||||
Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh. For more information, refer to [Create recording rules](ref:create-recording-rules).
|
||||
The new metric can then be used in alert rules and dashboards to optimize their queries.
|
||||
|
||||
Alternatively, Grafana Enterprise and Grafana Cloud offer [recorded queries](ref:recorded-queries) that can be executed against any data source.
|
||||
Similar to alert rules, recording rules are evaluated periodically. For more details, refer to [Create recording rules](ref:create-recording-rules).
|
||||
|
||||
## Comparison between alert rule types
|
||||
|
||||
When choosing which alert rule type to use, consider the following comparison between Grafana-managed and data source-managed alert rules.
|
||||
|
||||
| <div style="width:200px">Feature</div> | <div style="width:200px">Grafana-managed alert rule</div> | <div style="width:200px">Data source-managed alert rule |
|
||||
| ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Create alert rules<wbr /> based on data from any of the supported data sources | Yes | No. You can only create alert rules that are based on Prometheus-based data. |
|
||||
| Mix and match data sources | Yes | No |
|
||||
| Includes support for recording rules | Yes. Only for Grafana OSS users with the `grafanaManagedRecordingRules` feature flag enabled. | Yes |
|
||||
| Add expressions to transform<wbr /> your data and set alert conditions | Yes | No |
|
||||
| Use images in alert notifications | Yes | No |
|
||||
| Organization | Organize and manage access with folders | Use namespaces |
|
||||
| Scaling | More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically. | Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data. |
|
||||
| Alert rule evaluation and delivery | Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both. | Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure. |
|
||||
| <div style="width:200px">Feature</div> | <div style="width:200px">Grafana-managed alert rule</div> | <div style="width:200px">Data source-managed alert rule |
|
||||
| ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Create alert rules<wbr /> that query [data sources supporting Alerting](#supported-data-sources) | Yes | No. Only query Prometheus-based data sources. |
|
||||
| Mix and match data sources | Yes | No |
|
||||
| Add [expressions](ref:expression-queries) to transform<wbr /> your data and set [alert conditions](ref:alert-condition) | Yes | No |
|
||||
| Use [images in alert notifications](ref:notification-images) | Yes | No |
|
||||
| Support for [recording rules](#recording-rules) | Yes | Yes |
|
||||
| Organization | Organize and manage access with folders | Use namespaces |
|
||||
| Scaling | More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically. | Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data. |
|
||||
| Alert rule evaluation and delivery | Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both. | Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure. |
|
||||
|
||||
Loading…
Reference in New Issue
Block a user