mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Alerting: Remove ngalert feature toggle and introduce two new settings for enabling Grafana 8 alerts and disabling them for specific organisations (#38746)
* Remove `ngalert` feature toggle * Update frontend Remove all references of ngalert feature toggle * Update docs * Disable unified alerting for specific orgs * Add backend tests * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Disabled unified alerting by default * Ensure backward compatibility with old ngalert feature toggle * Apply suggestions from code review Co-authored-by: gotjosh <josue@grafana.com>
This commit is contained in:
Sofia Papagiannaki
committed by
GitHub
GitHub
parent
2dedbcd3c3
commit
012d4f0905
@@ -1113,6 +1113,16 @@ Sets a global limit on number of alert rules that can be created. Default is -1
|
||||
|
||||
For more information about the Grafana 8 alerts, refer to [Unified Alerting]({{< relref "../alerting/unified-alerting/_index.md" >}}).
|
||||
|
||||
### enabled
|
||||
|
||||
Enable the Unified Alerting sub-system and interface. When enabled we'll migrate all of your alert rules and notification channels to the new system. New alert rules will be created and your notification channels will be converted into an Alertmanager configuration. Previous data is preserved to enable backwards compatibility but new data is removed. The default value is `false`.
|
||||
|
||||
Alerting Rules migrated from dashboards and panels will include a link back via the `annotations`.
|
||||
|
||||
### disabled_orgs
|
||||
|
||||
Comma-separated list of organization IDs for which to disable Grafana 8 Unified Alerting.
|
||||
|
||||
### admin_config_poll_interval
|
||||
|
||||
Specify the frequency of polling for admin config changes. The default value is `60s`.
|
||||
@@ -1189,7 +1199,7 @@ For more information about the Alerting feature in Grafana, refer to [Alerts ove
|
||||
|
||||
### enabled
|
||||
|
||||
Set to `false` to disable alerting engine and hide Alerting in the Grafana UI. Default is `true`.
|
||||
Set to `false` to [enable Grafana 8 alerting]({{<relref "#unified_alerting">}}) and to disable legacy alerting engine. Default is `true`.
|
||||
|
||||
### execute_alerts
|
||||
|
||||
@@ -1735,7 +1745,7 @@ For more information about Grafana Enterprise, refer to [Grafana Enterprise]({{<
|
||||
|
||||
### enable
|
||||
|
||||
Keys of alpha features to enable, separated by space. Available alpha features are: `ngalert`
|
||||
Keys of alpha features to enable, separated by space.
|
||||
|
||||
## [date_formats]
|
||||
|
||||
|
||||
@@ -7,9 +7,9 @@ weight = 110
|
||||
|
||||
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
|
||||
|
||||
Grafana 8.0 has new and improved alerts. The new alerting system is an [opt-in]({{< relref "./unified-alerting/opt-in.md" >}}) feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible data sources in one UI and API.
|
||||
Grafana 8.0 has new and improved alerts that centralizes alerting information for Grafana managed alerts as well as alerts from Prometheus-compatible data sources into one user interface and API.
|
||||
|
||||
> **Note:** Out of the box, Grafana still supports old [legacy dashboard alerts]({{< relref "./old-alerting/_index.md" >}}). We encourage you to create issues in the Grafana GitHub repository for bugs found while testing Grafana 8 alerts.
|
||||
> **Note:** Grafana 8 alerts is an [opt-in]({{< relref "./unified-alerting/opt-in.md" >}}) feature. Out of the box, Grafana still supports old [legacy dashboard alerts]({{< relref "./old-alerting/_index.md" >}}). We encourage you to create issues in the Grafana GitHub repository for bugs found while testing Grafana 8 alerts.
|
||||
|
||||
Alerts have four main components:
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
+++
|
||||
title = "What's New with Grafana 8 Alerts"
|
||||
title = "What's New with Grafana 8 alerts"
|
||||
description = "What's New with Grafana 8 Alerts"
|
||||
keywords = ["grafana", "alerting", "guide"]
|
||||
weight = 112
|
||||
+++
|
||||
|
||||
# What's New with Grafana 8 Alerts
|
||||
# What's New with Grafana 8 alerts
|
||||
|
||||
The Alerts released with Grafana 8.0 are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible datasources in one UI and API. You are able to create and edit alerting rules for Grafana managed alerts, Cortex alerts, and Loki alerts as well as see alerting information from prometheus-compatible datasources in a single, searchable view.
|
||||
The alerts released with Grafana 8.0 centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible datasources in one UI and API. You can create and edit alerting rules for Grafana managed alerts, Cortex alerts, and Loki alerts as well as see alerting information from prometheus-compatible datasources in a single, searchable view.
|
||||
|
||||
## Multi-dimensional alerting
|
||||
|
||||
|
||||
@@ -6,9 +6,9 @@ weight = 113
|
||||
|
||||
# Overview of Grafana 8 alerts
|
||||
|
||||
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
|
||||
Grafana 8.0 has a new and improved alerting sub-system that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible data sources into one user interface and API.
|
||||
|
||||
> **Note:** Grafana 8 alerts (beta) is an [opt-in]({{< relref"./opt-in.md" >}}) feature. Out of the box, Grafana still supports old [legacy dashboard alerts]({{< relref "../old-alerting/_index.md" >}}). We encourage you to create issues in the Grafana GitHub repository for bugs found while testing this new feature.
|
||||
> **Note:** Grafana 8 alerts is an [opt-in]({{< relref "../unified-alerting/opt-in.md" >}}) feature. Out of the box, Grafana still supports old [legacy dashboard alerts]({{< relref "./old-alerting/_index.md" >}}). We encourage you to create issues in the Grafana GitHub repository for bugs found while testing Grafana 8 alerts.
|
||||
|
||||
Grafana 8 alerts have four main components:
|
||||
|
||||
@@ -45,7 +45,7 @@ Alerting rules can only query backend data sources with alerting enabled:
|
||||
|
||||
## Metrics from the alerting engine
|
||||
|
||||
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
|
||||
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}). See also, [View alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}}).
|
||||
|
||||
| Metric Name | Type | Description |
|
||||
| ------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
|
||||
@@ -57,4 +57,6 @@ The alerting engine publishes some internal metrics about itself. You can read m
|
||||
| `grafana_alerting_rule_evaluation_duration` | summary | The duration for a rule to execute |
|
||||
| `grafana_alerting_rule_group_rules` | gauge | The number of rules |
|
||||
|
||||
- [View alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
|
||||
## Limitation
|
||||
|
||||
Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
|
||||
|
||||
@@ -4,7 +4,7 @@ aliases = ["/docs/grafana/latest/alerting/rules/"]
|
||||
weight = 130
|
||||
+++
|
||||
|
||||
# Create and manage alerting Rules
|
||||
# Create and manage alerting rules
|
||||
|
||||
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
|
||||
|
||||
|
||||
@@ -4,33 +4,53 @@ description = "Enable Grafana 8 Alerts"
|
||||
weight = 128
|
||||
+++
|
||||
|
||||
# Enable Grafana 8 Alerts
|
||||
# Opt-in to Grafana 8 alerts
|
||||
|
||||
Setting the `ngalert` feature toggle enables the new Grafana 8 alerting system.
|
||||
This topic describes how to enable Grafana 8 alerts as well as the rules and restrictions that govern the migration of existing dashboard alerts to this new alerting system. You can also [disable Grafana 8 alerts]({{< relref "./opt-in.md#disable-grafana-8-alerts" >}}) if needed.
|
||||
|
||||
> **Note:** We recommend that you backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
|
||||
Before you begin, we recommend that you backup Grafana's database. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
|
||||
|
||||
At startup, when [the feature toggle is enabled]({{< relref "../../administration/configuration.md">}}#feature_toggles), the legacy Grafana dashboard alerting is disabled and existing dashboard alerts are migrated into a format that is compatible with the Grafana 8 alerting system. You can view these migrated rules, alongside any new alerts you create after the migration, from the Alerting page of your Grafana instance.
|
||||
## Enable Grafana 8 alerts
|
||||
|
||||
> **Note - v8.2 or earlier:** Since the new system stores the notification log and silences on disk, we require the use of persistent disks for using Grafana 8 alerts. Otherwise, the silences and notification log will get lost on a restart, and you might get unwanted or duplicate notifications.
|
||||
To enable Grafana 8 alerts:
|
||||
|
||||
> **Note - v8.3+**: We have removed the need of persistent disk. The notification log and silences are now stored in the database. If you used the file-based approach, we'll read those files and eventually (every 15 minutes) persist them to the database.
|
||||
1. Go to your custom configuration file located in $WORKING_DIR/conf/custom.ini.
|
||||
1. In the [unified alerts]({{< relref "../../administration/configuration.md#unified_alerting" >}}) section, set the `enabled` property to `true`.
|
||||
1. Next, in the [alerting]({{< relref "../../administration/configuration.md#alerting" >}}) section of the configuration file, update the configuration for the legacy dashboard alerts by setting the `enabled` property to `false`.
|
||||
1. Restart Grafana for the configuration changes to take effect.
|
||||
|
||||
Read and write access to dashboard alerts in Grafana versions 7 and earlier were governed by the dashboard and folder permissions under which the alerts were stored. In Grafana 8, alerts are stored in folders and inherit the permissions of those folders. During the migration, dashboard alert permissions are matched to the new rules permissions as follows:
|
||||
> **Note:** Before Grafana v8.2, to enable or disable Grafana 8 alerts, users configured the `ngalert` feature toggle. This toggle option is no longer available.
|
||||
|
||||
Moreover, before v8.2, notification logs and silences were stored on a disk. If you did not use persistent disks, any configured silences and logs would get lost on a restart, resulting in unwanted or duplicate notifications.
|
||||
|
||||
As of Grafana 8.2, we no longer require the use of a persistent disk. Instead, the notification logs and silences are stored regularly (every 15 minutes), and a clean shutdown to the database. If you used the file-based approach, Grafana will read the existing file and persisting it eventually.
|
||||
|
||||
## Migrating legacy alerts to Grafana 8 alerting system
|
||||
|
||||
When Grafana 8 alerting is enabled, existing legacy dashboard alerts migrate in a format compatible with the Grafana 8 alerting system. In the Alerting page of your Grafana instance, you can view the migrated alerts alongside new alerts.
|
||||
|
||||
Read and write access to legacy dashboard alerts was governed by the dashboard and folder permissions storing them. In Grafana 8, alerts inherit the permissions of the folders they are stored in. During migration, legacy dashboard alert permissions are matched to the new rules permissions as follows:
|
||||
|
||||
- If alert's dashboard has permissions, it will create a folder named like `Migrated {"dashboardUid": "UID", "panelId": 1, "alertId": 1}` to match permissions of the dashboard (including the inherited permissions from the folder).
|
||||
- If there are no dashboard permissions and the dashboard is under a folder, then the rule is linked to this folder and inherits its permissions.
|
||||
- If there are no dashboard permissions and the dashboard is under the General folder, then the rule is linked to the `General Alerting` folder and the rule inherits the default permissions.
|
||||
- If there are no dashboard permissions and the dashboard is under the General folder, then the rule is linked to the `General Alerting` folder, and the rule inherits the default permissions.
|
||||
|
||||
During beta, Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
|
||||
Notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the `autogen-unlinked-channel-recv` route.
|
||||
|
||||
Also notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the `autogen-unlinked-channel-recv` route.
|
||||
Since `Hipchat` and `Sensu` notification channels are no longer supported, legacy alerts associated with these channels are not automatically migrated to Grafana 8 alerting. Assign the legacy alerts to a supported notification channel so that you continue to receive notifications for those alerts.
|
||||
Silences (expiring after one year) are created for all paused dashboard alerts.
|
||||
|
||||
Since `Hipchat` and `Sensu` are discontinued, they are not migrated to the new alerting. If you have dashboard alerts associated with those types of channels and you want to migrate to the new alerting, make sure you assign another supported notification channel, so that you continue to receive notifications for those alerts.
|
||||
Finally, silences (expiring after one year) are created for all paused dashboard alerts.
|
||||
### Limitation
|
||||
|
||||
## Disabling Grafana 8 Alerting after migration
|
||||
Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
|
||||
|
||||
To disable Grafana 8 Alerting, remove or disable the `ngalert` feature toggle. Dashboard alerts will be re-enabled and any alerts created during or after the migration are deleted.
|
||||
## Disable Grafana 8 alerts
|
||||
|
||||
> **Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
|
||||
To disable Grafana 8 alerts and enable legacy dashboard alerts:
|
||||
|
||||
1. Go to your custom configuration file located in $WORKING_DIR/conf/custom.ini.
|
||||
1. In the [unified alerts]({{< relref "../../administration/configuration.md#unified_alerting" >}}) section, set the `enabled` property to `false`.
|
||||
1. Next, in the [alerting]({{< relref "../../administration/configuration.md#alerting" >}}) section of the configuration file, update the configuration for the legacy dashboard alerts by setting the `enabled` property to `true`.
|
||||
1. Restart Grafana for the configuration changes to take effect.
|
||||
|
||||
> **Note:** If you choose to migrate from Grafana 8 alerts to legacy dashboard alerts, you will lose any new alerts that you created in the Grafana 8 alerting system.
|
||||
|
||||
Reference in New Issue
Block a user