mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
docs>alerting:tutorial update (#95402)
* updated steps * update before you begin * updated title * update title 2 * elaborated on 'Sample logs' section * updates to improve visibility * updated links * format * updated link * links and steps * updated call-to-action boxes * updated box * title * format * steps * format * Made docs live in the now * all pretty, no pity
This commit is contained in:
antonio
GitHub
parent
fe74dfcb77
commit
0ef08ca737
@ -11,18 +11,18 @@ labels:
|
||||
- cloud
|
||||
tags:
|
||||
- beginner
|
||||
title: Get started with Grafana Alerting - Part 1
|
||||
title: Get started with Grafana Alerting - Part 1 of 2
|
||||
weight: 50
|
||||
killercoda:
|
||||
title: Get started with Grafana Alerting - Part 1
|
||||
description: Get started with Grafana Alerting by creating your first alert in just a few minutes. Learn how to set up an alert, send alert notifications to a public webhook, and generate sample data to observe your alert in action.
|
||||
title: Get started with Grafana Alerting - Part 1 of 2
|
||||
description: Get started with Grafana Alerting by creating your first alert rule, sending notifications to a webhook, and generating data to test it live — Part 1.
|
||||
backend:
|
||||
imageid: ubuntu
|
||||
---
|
||||
|
||||
<!-- INTERACTIVE page intro.md START -->
|
||||
|
||||
# Get Started with Grafana Alerting - Part 1
|
||||
# Get started with Grafana Alerting - Part 1 of 2
|
||||
|
||||
In this guide, we walk you through the process of setting up your first alert in just a few minutes. You'll witness your alert in action with real-time data, as well as sending alert notifications.
|
||||
|
||||
@ -36,7 +36,7 @@ In this tutorial you will:
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
|
||||
Before you dive in, remember that you can [explore advanced topics like alert instances and notification routing](http://grafana.com/tutorials/alerting-get-started-pt2/) in the second part of this guide.
|
||||
Once you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in [Part 2 Alert instances and notification routing](http://www.grafana.com/tutorials/alerting-get-started-pt2/).
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
@ -44,7 +44,7 @@ Before you dive in, remember that you can [explore advanced topics like alert in
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
> Before you dive in, remember that you can [explore advanced topics like alert instances and notification routing](http://grafana.com/tutorials/alerting-get-started-pt2/) in the second part of this guide.
|
||||
> Once you have completed Part 1, don’t forget to explore the advanced but essential alerting topics in [Part 2 Alert instances and notification routing](http://www.grafana.com/tutorials/alerting-get-started-pt2/).
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
@ -52,36 +52,28 @@ Before you dive in, remember that you can [explore advanced topics like alert in
|
||||
<!-- INTERACTIVE page step1.md START -->
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
## Set up the Grafana stack
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
## Before you begin
|
||||
|
||||
There are different ways you can follow along with this tutorial.
|
||||
|
||||
### Grafana Cloud
|
||||
- **Grafana Cloud**
|
||||
|
||||
As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://grafana.com/auth/sign-up/create-user).
|
||||
- As a Grafana Cloud user, you don't have to install anything. [Create your free account](http://www.grafana.com/auth/sign-up/create-user).
|
||||
|
||||
Continue to [Create a contact point](#create-a-contact-point).
|
||||
Continue to [Create a contact point](#create-a-contact-point).
|
||||
|
||||
### Interactive learning environment
|
||||
- **Interactive learning environment**
|
||||
|
||||
Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started/).
|
||||
- Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-get-started/). It's a fully configured environment with all the dependencies already installed.
|
||||
|
||||
It's a fully configured environment with all the dependencies already installed.
|
||||
- **Grafana OSS**
|
||||
|
||||
### Grafana OSS
|
||||
- If you opt to run a Grafana stack locally, ensure you have the following applications installed:
|
||||
|
||||
If you opt to run a Grafana stack locally, ensure you have the following applications installed:
|
||||
- [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
|
||||
- [Git](https://git-scm.com/)
|
||||
|
||||
- [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
|
||||
- [Git](https://git-scm.com/)
|
||||
|
||||
#### Set up the Grafana stack (OSS users)
|
||||
### Set up the Grafana stack (OSS users)
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
@ -154,7 +146,7 @@ To demonstrate the observation of data using the Grafana stack, download and run
|
||||
|
||||
Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.
|
||||
|
||||
In this step, we set up a new [contact point](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). This contact point will use the _webhooks_ integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We will use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.
|
||||
In this step, we set up a new contact point. This contact point uses the [webhook integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.
|
||||
|
||||
1. In your browser, **sign in** to your Grafana Cloud account.
|
||||
|
||||
@ -168,12 +160,12 @@ Your webhook endpoint is now waiting for the first request.
|
||||
Next, let's configure a contact point in Grafana's Alerting UI to send notifications to our webhook endpoint.
|
||||
|
||||
1. Return to Grafana. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Click **+ Add contact point**.
|
||||
1. Click **+ Create contact point**.
|
||||
1. In **Name**, write **Webhook**.
|
||||
1. In **Integration**, choose **Webhook**.
|
||||
1. In **URL**, paste the endpoint to your webhook endpoint.
|
||||
1. Click **Test**, and then click **Send test notification** to send a test alert to your webhook endpoint.
|
||||
1. Navigate back to [Webhook.site](https://webhook.site/). On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.
|
||||
1. Navigate back to _Webhook.site_. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}
|
||||
|
||||
@ -191,20 +183,21 @@ Next, we establish an [alert rule](https://grafana.com/docs/grafana/latest/alert
|
||||
|
||||
1. In Grafana, **navigate to Alerting** > **Alert rules**. Click on **New alert rule**.
|
||||
|
||||
1. Enter alert rule name for your alert rule. Make it short and descriptive as this will appear in your alert notification. For instance, **database-metrics**
|
||||
1. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, **database-metrics**
|
||||
|
||||
### Define query and alert condition
|
||||
|
||||
In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).
|
||||
|
||||
1. Select the **Prometheus** data source from the drop-down menu.
|
||||
1. In the Query editor, switch to **Code** mode by clicking the button on the right.
|
||||
1. Enter the following query:
|
||||
|
||||
```promql
|
||||
vector(1)
|
||||
```
|
||||
|
||||
In Prometheus, `vector(1)` is a special type of PromQL query that generates a constant vector. This is useful in testing and query manipulation, where you might need a constant value for calculations or comparisons. This query will allow you to create an alert rule that will be always firing.
|
||||
In Prometheus, `vector(1)` is a special type of PromQL query that generates a constant vector. This is useful in testing and query manipulation, where you might need a constant value for calculations or comparisons. This query allows you to create an alert rule that is always firing.
|
||||
|
||||
1. In the **Alert condition** section:
|
||||
|
||||
@ -226,9 +219,9 @@ The [alert rule evaluation](https://grafana.com/docs/grafana/latest/alerting/fun
|
||||
|
||||
To set up the evaluation:
|
||||
|
||||
1. In **Folder**, click **+ New folder** and enter a name. For example: _metric-alerts_. This folder will contain our alerts.
|
||||
1. In the **Evaluation group**, repeat the above step to create a new evaluation group. We will name it _1m-evaluation_.
|
||||
1. Choose an **Evaluation interval** (how often the alert will be evaluated).
|
||||
1. In **Folder**, click **+ New folder** and enter a name. For example: _metric-alerts_. This folder contains our alerts.
|
||||
1. In the **Evaluation group**, repeat the above step to create a new evaluation group. Name it _1m-evaluation_.
|
||||
1. Choose an **Evaluation interval** (how often the alert are evaluated).
|
||||
For example, every `1m` (1 minute).
|
||||
1. Set the pending period to, `0s` (zero seconds), so the alert rule fires the moment the condition is met.
|
||||
|
||||
@ -273,13 +266,13 @@ By incrementing the threshold, the condition is no longer met, and after the eva
|
||||
|
||||
<!-- INTERACTIVE page finish.md START -->
|
||||
|
||||
## Learn more
|
||||
## Learn more in [Grafana Alerting Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/)
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
|
||||
Advance your skills by exploring [alert instances and notification routing](http://grafana.com/tutorials/alerting-get-started-pt2/) in Part 2 of your learning journey.
|
||||
In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
@ -287,7 +280,7 @@ Advance your skills by exploring [alert instances and notification routing](http
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
Advance your skills by exploring [alert instances and notification routing](http://grafana.com/tutorials/alerting-get-started-pt2/) in Part 2 of your learning journey.
|
||||
In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
|
||||
@ -1,7 +1,5 @@
|
||||
---
|
||||
Feedback Link: https://github.com/grafana/tutorials/issues/new
|
||||
authors:
|
||||
- melori_arellano
|
||||
categories:
|
||||
- alerting
|
||||
description: Create alerts with Logs
|
||||
@ -15,10 +13,10 @@ labels:
|
||||
- alerting
|
||||
tags:
|
||||
- advanced
|
||||
title: How to create alerts with log data
|
||||
title: How to create alert rules with log data
|
||||
weight: 70
|
||||
killercoda:
|
||||
title: How to create alerts with log data
|
||||
title: How to create alert rules with log data
|
||||
description: Learn how to use Loki with Grafana Alerting to keep track of what’s happening in your environment with real log data.
|
||||
preprocessing:
|
||||
substitutions:
|
||||
@ -43,14 +41,14 @@ In this tutorial, you'll:
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
Check out our [advanced alerting tutorial](https://grafana.com/tutorials/alerting-get-started-pt2/) to explore advanced topics such as alert instances and notification routing.
|
||||
In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
{{< /admonition >}}
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
> Check out our [advanced alerting tutorial](https://grafana.com/tutorials/alerting-get-started-pt2/) to explore advanced topics such as alert instances and notification routing.
|
||||
> In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
@ -62,26 +60,24 @@ Check out our [advanced alerting tutorial](https://grafana.com/tutorials/alertin
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
### Grafana Cloud users
|
||||
There are different ways you can follow along with this tutorial.
|
||||
|
||||
As a Grafana Cloud user, you don't have to install anything.
|
||||
- **Grafana OSS**
|
||||
|
||||
Continue to [Generate sample logs](#generate-sample-logs).
|
||||
To run a Grafana stack locally, ensure you have the following applications installed:
|
||||
|
||||
<!-- INTERACTIVE ignore END-->
|
||||
- [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
|
||||
- [Git](https://git-scm.com/)
|
||||
|
||||
### Grafana OSS users
|
||||
- **Interactive learning environment**
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
- Alternatively, you can [try out this example in our interactive learning environment](https://killercoda.com/grafana-labs/course/grafana/alerting-loki-logs). It's a fully configured environment with all the dependencies already installed.
|
||||
|
||||
In order to run a Grafana stack locally, ensure you have the following applications installed.
|
||||
|
||||
- [Docker Compose](https://docs.docker.com/get-docker/) (included in Docker for Desktop for macOS and Windows)
|
||||
- [Git](https://git-scm.com/)
|
||||
## Set up the Grafana stack
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
To demonstrate the observation of data using the Grafana stack, download the files to your local machine.
|
||||
To demonstrate the observation of data using the Grafana stack, download and run the following files.
|
||||
|
||||
1. Download and save a Docker compose file to run Grafana, Loki and Promtail.
|
||||
|
||||
@ -113,26 +109,14 @@ If you already have Grafana, Loki, or Prometheus running on your system, you mig
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
Alternatively, you can try out this example in our interactive learning environment: [Get started with Grafana Alerting](https://killercoda.com/grafana-labs/course/grafana/alerting-loki-logs).
|
||||
|
||||
It's a fully configured environment with all the dependencies already installed.
|
||||
|
||||
|
||||
|
||||
Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda).
|
||||
{{< /admonition >}}
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
<!-- INTERACTIVE page step1.md END -->
|
||||
|
||||
<!-- INTERACTIVE page step2.md START -->
|
||||
|
||||
## Generate sample logs
|
||||
|
||||
To demonstrate how to create alert rules based on logs, you’ll use a script that generates realistic log entries to simulate typical monitoring data in Grafana. Running this script outputs logs continuously, each containing a timestamp, HTTP method (either GET or POST), status code (200 for success or 500 for failures), and request duration in milliseconds.
|
||||
|
||||
1. Download and save a Python file that generates logs.
|
||||
|
||||
```bash
|
||||
@ -162,13 +146,13 @@ If you don't see the sample logs in Explore:
|
||||
|
||||
Besides being an open-source observability tool, Grafana has its own built-in alerting service. This means that you can receive notifications whenever there is an event of interest in your data, and even see these events graphed in your visualizations.
|
||||
|
||||
In this step, we'll set up a new [contact point](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). This contact point will use the _webhooks_ integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We will use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.
|
||||
In this step, we set up a new contact point. This contact point uses the [webhook integration](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier/). This contact point uses the _webhooks_ integration. In order to make this work, we also need an endpoint for our webhook integration to receive the alert. We can use [Webhook.site](https://webhook.site/) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere.
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
1. In your browser, **sign in** to your Grafana Cloud account.
|
||||
|
||||
OSS users: To log in, navigate to [http://localhost:3000](http://localhost:3000), where Grafana is running.
|
||||
OSS users: To log in, navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.
|
||||
|
||||
1. In another tab, go to [Webhook.site](https://webhook.site/).
|
||||
1. Copy Your unique URL.
|
||||
@ -176,7 +160,7 @@ In this step, we'll set up a new [contact point](https://grafana.com/docs/grafan
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
1. Navigate to [http://localhost:3000](http://localhost:3000), where Grafana is running.
|
||||
1. Navigate to [http://localhost:3000](http://localhost:3000), where Grafana should be running.
|
||||
1. In another tab, go to [Webhook.site](https://webhook.site/).
|
||||
1. Copy Your unique URL.
|
||||
{{< /docs/ignore >}}
|
||||
@ -186,12 +170,12 @@ Your webhook endpoint is now waiting for the first request.
|
||||
Next, let's configure a contact point in Grafana's Alerting UI to send notifications to our webhook endpoint.
|
||||
|
||||
1. Return to Grafana. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Click **+ Add contact point**.
|
||||
1. Click **+ Create contact point**.
|
||||
1. In **Name**, write **Webhook**.
|
||||
1. In **Integration**, choose **Webhook**.
|
||||
1. In **URL**, paste the endpoint to your webhook endpoint.
|
||||
1. Click **Test**, and then click **Send test notification** to send a test alert to your webhook endpoint.
|
||||
1. Navigate back to [Webhook.site](https://webhook.site/). On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.
|
||||
1. Navigate back to _Webhook.site_. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-webhook-detail.png" max-width="1200px" caption="A POST entry in Webhook.site" >}}
|
||||
|
||||
@ -209,52 +193,44 @@ Next, we'll establish an [alert rule](http://grafana.com/docs/grafana/next/alert
|
||||
|
||||
1. In Grafana, **navigate to Alerting** > **Alert rules**.
|
||||
1. Click on **New alert rule**.
|
||||
1. Enter alert rule name for your alert rule. Make it short and descriptive as this will appear in your alert notification. For instance, **web-requests-logs**
|
||||
1. Enter alert rule name for your alert rule. Make it short and descriptive as this appears in your alert notification. For instance, **web-requests-logs**
|
||||
|
||||
### Define query and alert condition
|
||||
|
||||
In this section, we define queries, expressions (used to manipulate the data), and the condition that must be met for the alert to be triggered.
|
||||
In this section, we use the default options for Grafana-managed alert rule creation. The default options let us define the query, a expression (used to manipulate the data -- the `WHEN` field in the UI), and the condition that must be met for the alert to be triggered (in default mode is the threshold).
|
||||
|
||||
1. Select the **Loki** datasource from the drop-down.
|
||||
2. In the Query editor, switch to Code mode by clicking the button on the right.
|
||||
3. Paste the query below.
|
||||
1. In the Query editor, switch to **Code** mode by clicking the button on the right.
|
||||
1. Paste the query below.
|
||||
|
||||
```
|
||||
sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != "status=200" | pattern "<_> <message> duration<_>" [10m]))
|
||||
```
|
||||
|
||||
This query will count the number of log lines with a status code that is not 200 (OK), then sum the result set by message type using an **instant query** and the time interval indicated in brackets. It uses the LogQL pattern parser to add a new label called `message` that contains the level, method, url, and status from the log line.
|
||||
This query counts the number of log lines with a status code that is not 200 (OK), then sum the result set by message type using an **instant query** and the time interval indicated in brackets. It uses the LogQL pattern parser to add a new label called `message` that contains the level, method, url, and status from the log line.
|
||||
|
||||
You can use the **explain query** toggle button for a full explanation of the query syntax. The optional log-generating script creates a sample log line similar to the one below:
|
||||
You can use the **explain query** toggle button for a full explanation of the query syntax. The optional log-generating script creates a sample log line similar to the one below:
|
||||
|
||||
```
|
||||
2023-04-22T02:49:32.562825+00:00 level=info method=GET url=test.com status=200 duration=171ms
|
||||
```
|
||||
```
|
||||
2023-04-22T02:49:32.562825+00:00 level=info method=GET url=test.com status=200 duration=171ms
|
||||
```
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
{{% admonition type="note" %}}
|
||||
If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
|
||||
{{% / admonition %}}
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
|
||||
{{< docs/ignore >}}
|
||||
If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
{{% / admonition %}}
|
||||
1. In the **Alert condition** section:
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
- Keep `Last` as the value for the reducer function (`WHEN`), and `0` as the threshold value. This is the value above which the alert rule should trigger.
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
If you're using your own logs, modify the LogQL query to match your own log message. Refer to the Loki docs to understand the [pattern parser](https://grafana.com/docs/loki/latest/logql/log_queries/#pattern).
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
4. Remove the ‘B’ **Reduce expression** (click the bin icon). The Reduce expression comes by default, and in this case, it is not needed since the queried data is already reduced. Note that the Threshold expression is now your **Alert condition**.
|
||||
|
||||
5. In the ‘C’ **Threshold expression**:
|
||||
|
||||
- Change the **Input** to **'A'** to select the data source.
|
||||
- Enter `0` as the threshold value. This is the value above which the alert rule should trigger.
|
||||
|
||||
6. Click **Preview** to run the queries.
|
||||
1. Click **Preview alert rule condition** to run the query.
|
||||
|
||||
It should return alert instances from log lines with a status code that is not 200 (OK), and that has met the alert condition. The condition for the alert rule to fire is any occurrence that goes over the threshold of `0`. Since the Loki query has returned more than zero alert instances, the alert rule is `Firing`.
|
||||
|
||||
@ -269,9 +245,9 @@ An [evaluation group](https://grafana.com/docs/grafana/latest/alerting/fundament
|
||||
|
||||
To set up the evaluation:
|
||||
|
||||
1. In **Folder**, click **+ New folder** and enter a name. For example: _web-server-alerts_. This folder will contain our alerts.
|
||||
1. In the **Evaluation group**, repeat the above step to create a new evaluation group. We will name it _1m-evaluation_.
|
||||
1. Choose an **Evaluation interval** (how often the alert will be evaluated).
|
||||
1. In **Folder**, click **+ New folder** and enter a name. For example: _web-server-alerts_. This folder contains our alerts.
|
||||
1. In the **Evaluation group**, repeat the above step to create a new evaluation group. Name it _1m-evaluation_.
|
||||
1. Choose an **Evaluation interval** (how often the alert are evaluated).
|
||||
For example, every `1m` (1 minute).
|
||||
1. Set the pending period to, `0s` (zero seconds), so the alert rule fires the moment the condition is met.
|
||||
|
||||
@ -288,7 +264,7 @@ Choose the contact point where you want to receive your alert notifications.
|
||||
|
||||
## Trigger the alert rule
|
||||
|
||||
Since the Python script will continue to generate log data that matches the alert rule condition, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.
|
||||
Since the Python script continues to generate log data that matches the alert rule condition, once the evaluation interval has concluded, you should receive an alert notification in the Webhook endpoint.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-webhook-firing-alert.png" max-width="1200px" caption="Firing alert notification details" >}}
|
||||
|
||||
@ -299,14 +275,14 @@ Since the Python script will continue to generate log data that matches the aler
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
Check out our [advanced alerting tutorial](https://grafana.com/tutorials/alerting-get-started-pt2/) to explore advanced topics such as alert instances and notification routing.
|
||||
In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
{{< /admonition >}}
|
||||
|
||||
<!-- INTERACTIVE ignore END -->
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
> Check out our [advanced alerting tutorial](https://grafana.com/tutorials/alerting-get-started-pt2/) to explore advanced topics such as alert instances and notification routing.
|
||||
> In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
|
||||
@ -153,19 +153,19 @@ Grafana is an open source platform for monitoring and observability that lets yo
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you will need to log in and provide credentials. In that case, the default username and password is `admin`.
|
||||
This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is `admin`.
|
||||
{{< /admonition >}}
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
> This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you will need to log in and provide credentials. In that case, the default username and password is `admin`.
|
||||
> This demo does not require a login page or credentials. However, if you choose to install Grafana locally, you need to log in and provide credentials. In that case, the default username and password is `admin`.
|
||||
> {{< /docs/ignore >}}
|
||||
|
||||
The first thing you see is the Home dashboard, which helps you get started.
|
||||
|
||||
In the top left corner, you can see the menu icon. Clicking it will open the _sidebar_, the main menu for navigating Grafana.
|
||||
In the top left corner, you can see the menu icon. Clicking it opens the _sidebar_, the main menu for navigating Grafana.
|
||||
|
||||
<!-- INTERACTIVE page step2.md END -->
|
||||
|
||||
@ -177,7 +177,7 @@ Grafana Explore is a workflow for troubleshooting and data exploration. In this
|
||||
|
||||
> Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query.
|
||||
|
||||
1. Click the menu icon and, in the sidebar, click **Explore**. A dropdown menu for the list of available data sources is on the upper-left side. The Prometheus data source will already be selected. If not, choose Prometheus.
|
||||
1. Click the menu icon and, in the sidebar, click **Explore**. A dropdown menu for the list of available data sources is on the upper-left side. The Prometheus data source should already be selected. If not, choose Prometheus.
|
||||
1. Confirm that you're in code mode by checking the **Builder/Code** toggle at the top right corner of the query panel.
|
||||
1. In the query editor, where it says _Enter a PromQL query…_, enter `tns_request_duration_seconds_count` and then press Shift + Enter.
|
||||
A graph appears.
|
||||
@ -259,7 +259,7 @@ Let's generate an error, and analyze it with Explore.
|
||||
|
||||
1. Click the log line that says `level=error msg="empty url"` to see more information about the error.
|
||||
|
||||
> **Note:** If you're in Live mode, clicking logs will not show more information about the error. Instead, stop and exit the live stream, then click the log line there.
|
||||
> **Note:** If you're in Live mode, clicking logs does not show more information about the error. Instead, stop and exit the live stream, then click the log line there.
|
||||
|
||||
Logs are helpful for understanding what went wrong. Later in this tutorial, you'll see how you can correlate logs with metrics from Prometheus to better understand the context of the error.
|
||||
|
||||
@ -302,7 +302,7 @@ Every panel consists of a _query_ and a _visualization_. The query defines _what
|
||||
|
||||
When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs.
|
||||
|
||||
In the next part of the tutorial, we will simulate some common use cases that someone would add annotations for.
|
||||
In the next part of the tutorial, we simulate some common use cases that someone would add annotations for.
|
||||
|
||||
1. To manually add an annotation, click anywhere in your graph, then click **Add annotation**.
|
||||
Note: you might need to save the dashboard first.
|
||||
@ -346,7 +346,7 @@ The log lines returned by your query are now displayed as annotations in the gra
|
||||
|
||||
Being able to combine data from multiple data sources in one graph allows you to correlate information from both Prometheus and Loki.
|
||||
|
||||
Annotations also work very well alongside alert rules. In the next and final section, we will set up an alert rules for our app `grafana.news` and then we will trigger it. This will provide a quick intro to our new Alerting platform.
|
||||
Annotations also work very well alongside alert rules. In the next and final section, we set up an alert rules for our app `grafana.news` and then we trigger it. This provides a quick intro to our new Alerting platform.
|
||||
|
||||
<!-- INTERACTIVE page step7.md END -->
|
||||
|
||||
@ -356,7 +356,7 @@ Annotations also work very well alongside alert rules. In the next and final sec
|
||||
|
||||
Alert rules allow you to identify problems in your system moments after they occur. By quickly identifying unintended changes in your system, you can minimize disruptions to your services.
|
||||
|
||||
Grafana's new alerting platform debuted with Grafana 8. A year later, with Grafana 9, it became the default alerting method. In this step we will create a Grafana-managed alert rule. Then we will trigger our new alert rule and send a test message to a dummy endpoint.
|
||||
Grafana's new alerting platform debuted with Grafana 8. A year later, with Grafana 9, it became the default alerting method. In this step we create a Grafana-managed alert rule. Then we trigger our new alert rule and send a test message to a dummy endpoint.
|
||||
|
||||
The most basic alert rule consists of two parts:
|
||||
|
||||
@ -392,7 +392,7 @@ To begin, let's set up a webhook contact point. Once we have a usable endpoint,
|
||||
|
||||
### Create a contact point for Grafana-managed alert rules
|
||||
|
||||
In this step, we set up a new contact point. This contact point will use the _webhooks_ channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert notification. We will use [requestbin.com](https://requestbin.com) to quickly set up that test endpoint. This way we can make sure that our alert manager is actually sending a notification somewhere.
|
||||
In this step, we set up a new contact point. This contact point uses the _webhooks_ channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert notification. We can use [requestbin.com](https://requestbin.com) to quickly set up that test endpoint. This way we can make sure that our alert manager is actually sending a notification somewhere.
|
||||
|
||||
1. Browse to [requestbin.com](https://requestbin.com).
|
||||
1. Under the **Create Request Bin** button, click the link to create a **public bin** instead.
|
||||
@ -437,9 +437,9 @@ Now that Grafana knows how to notify us, it's time to set up an alert rule:
|
||||
|
||||
1. Scroll down to bottom of section #2 and click the **Preview alert rule condition** button. You should see some data returned.
|
||||
1. Keep `Last` as the value for the reducer function (`WHEN`), and `0.2` as the threshold value. This is the value above which the alert rule should trigger. [You can read more about queries and conditions here](/docs/grafana/latest/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries).
|
||||
1. In **Section 3**, in Folder, create a new folder, by clicking `New folder` and typing a name for the folder. This folder will contain our alert rules. For example: `fundamentals`. Then, click `create`.
|
||||
1. In the Evaluation group, repeat the above step to create a new one. We will name it `fundamentals` too.
|
||||
1. Choose an Evaluation interval (how often the alert rule will be evaluated). For example, every `10s` (10 seconds).
|
||||
1. In **Section 3**, in Folder, create a new folder, by clicking `New folder` and typing a name for the folder. This folder contains our alert rules. For example: `fundamentals`. Then, click `create`.
|
||||
1. In the Evaluation group, repeat the above step to create a new one. Name it `fundamentals` too.
|
||||
1. Choose an Evaluation interval (how often the alert rule are evaluated). For example, every `10s` (10 seconds).
|
||||
1. Set the pending period. This is the time that a condition has to be met until the alert instance enters in Firing state and a notification is sent. Enter `0s`. For the purposes of this tutorial, the evaluation interval is intentionally short. This makes it easier to test. This setting makes Grafana wait until an alert instance has fired for a given time before Grafana sends the notification.
|
||||
1. In **Section 4**, choose **RequestBin** as the **Contact point**.
|
||||
1. Click **Save rule and exit** at the top of the page.
|
||||
@ -451,7 +451,7 @@ We have now configured an alert rule and a contact point. Now let's see if we ca
|
||||
1. Browse to [localhost:8081](http://localhost:8081).
|
||||
1. Add a new title and URL, repeatedly click the vote button, or refresh the page to generate a traffic spike.
|
||||
|
||||
Once the query `sum(rate(tns_request_duration_seconds_count[5m])) by(route)` returns a value greater than `0.2` Grafana will trigger our alert rule. Browse to the Request Bin we created earlier and find the sent Grafana alert notification with details and metadata.
|
||||
Once the query `sum(rate(tns_request_duration_seconds_count[5m])) by(route)` returns a value greater than `0.2` Grafana triggers our alert rule. Browse to the Request Bin we created earlier and find the sent Grafana alert notification with details and metadata.
|
||||
|
||||
<!-- INTERACTIVE ignore START -->
|
||||
|
||||
@ -494,7 +494,7 @@ Let's see how we can configure this.
|
||||
|
||||
{{< admonition type="tip" >}}
|
||||
|
||||
Check out our [advanced alerting tutorial](http://grafana.com/tutorials/alerting-get-started-pt2/) for more insights and tips.
|
||||
In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
@ -502,7 +502,7 @@ Check out our [advanced alerting tutorial](http://grafana.com/tutorials/alerting
|
||||
|
||||
{{< docs/ignore >}}
|
||||
|
||||
> Check out our [advanced alerting tutorial](http://grafana.com/tutorials/alerting-get-started-pt2/) for more insights and tips.
|
||||
> In [Get started with Grafana Alerting - Part 2](http://www.grafana.com/tutorials/alerting-get-started-pt2/) you can advance your skills by exploring alert instances and notification routing.
|
||||
|
||||
{{< /docs/ignore >}}
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user