diff --git a/docs/sources/tutorials/grafana-fundamentals/index.md b/docs/sources/tutorials/grafana-fundamentals/index.md index 1aa2197e4ee..dc7a25d6096 100644 --- a/docs/sources/tutorials/grafana-fundamentals/index.md +++ b/docs/sources/tutorials/grafana-fundamentals/index.md @@ -27,6 +27,13 @@ In this tutorial, you'll learn how to use Grafana to set up a monitoring solutio - Annotate dashboards - Set up alerts +Alternatively, you can also watch our Grafana for Beginners series where we discuss fundamental concepts to help you get started with Grafana. + +
+ +
+ {{% class "prerequisite-section" %}} ### Prerequisites @@ -128,7 +135,7 @@ To be able to visualize the metrics from Prometheus, you first need to add it as 1. In the URL box, enter **http\://prometheus:9090**. 1. Scroll to the bottom of the page and click **Save & test**. - Prometheus is now available as a data source in Grafana. +You should see the message "Successfully queried the Prometheus API." This means Prometheus is now available as a data source in Grafana. ## Explore your metrics @@ -136,7 +143,7 @@ Grafana Explore is a workflow for troubleshooting and data exploration. In this > Ad-hoc queries are queries that are made interactively, with the purpose of exploring data. An ad-hoc query is commonly followed by another, more specific query. -1. Click the menu icon and, in the sidebar, click **Explore**. The Prometheus data source that you added will already be selected. +1. Click the menu icon and, in the sidebar, click **Explore**. A dropdown menu for the list of available data sources is on the upper-left side. The Prometheus data source that you added will already be selected. If not, choose Prometheus. 1. Confirm that you're in code mode by checking the **Builder/Code** toggle at the top right corner of the query panel. 1. In the query editor, where it says _Enter a PromQL query…_, enter `tns_request_duration_seconds_count` and then press Shift + Enter. A graph appears. @@ -166,7 +173,7 @@ Grafana Explore is a workflow for troubleshooting and data exploration. In this 1. Back in Grafana, in the upper-right corner, click the _time picker_, and select **Last 5 minutes**. By zooming in on the last few minutes, it's easier to see when you receive new data. -Depending on your use case, you might want to group on other labels. Try grouping by other labels, such as `status_code`, by changing the `by(route)` part of the query. +Depending on your use case, you might want to group on other labels. Try grouping by other labels, such as `status_code`, by changing the `by(route)` part of the query to `by(status_code)`. ## Add a logging data source @@ -178,7 +185,7 @@ Grafana supports log data sources, like [Loki](/oss/loki/). Just like for metric 1. In the URL box, enter [http://loki:3100](http://loki:3100). 1. Scroll to the bottom of the page and click **Save & Test** to save your changes. -Loki is now available as a data source in Grafana. +You should see the message "Data source successfully connected." Loki is now available as a data source in Grafana. ## Explore your logs @@ -237,6 +244,10 @@ Every panel consists of a _query_ and a _visualization_. The query defines _what 1. Click the **Save dashboard** (disk) icon at the top of the dashboard to save your dashboard. 1. Enter a name in the **Dashboard name** field and then click **Save**. + You should now have a panel added to your dashboard. + + {{< figure src="/media/tutorials/grafana-fundamentals-dashboard.png" alt="A panel in a Grafana dashboard" caption="A panel in a Grafana dashboard" >}} + ## Annotate events When things go bad, it often helps if you understand the context in which the failure occurred. Time of last deploy, system changes, or database migration can offer insight into what might have caused an outage. Annotations allow you to represent such events directly on your graphs. @@ -254,11 +265,13 @@ Grafana also lets you annotate a time interval, with _region annotations_. Add a region annotation: -1. Press Ctrl (or Cmd on macOS), then click and drag across the graph to select an area. +1. Press Ctrl (or Cmd on macOS) and hold, then click and drag across the graph to select an area. 1. In **Description**, enter **Performed load tests**. 1. In **Tags**, enter **testing**. 1. Click **Save**. +### Using annotations to correlate logs with metrics + Manually annotating your dashboard is fine for those single events. For regularly occurring events, such as deploying a new release, Grafana supports querying annotations from one of your data sources. Let's create an annotation using the Loki data source we added earlier. 1. At the top of the dashboard, click the **Dashboard settings** (gear) icon. @@ -274,9 +287,13 @@ Manually annotating your dashboard is fine for those single events. For regularl 1. Click **Apply**. Grafana displays the Annotations list, with your new annotation. 1. Click on your dashboard name to return to your dashboard. 1. At the top of your dashboard, there is now a toggle to display the results of the newly created annotation query. Press it if it's not already enabled. +1. Click the **Save dashboard** icon to save the changes. +1. To test the changes, go back to the [sample application](http://localhost:8081), post a new link without a URL to generate an error in your browser that says `empty url`. The log lines returned by your query are now displayed as annotations in the graph. +{{< figure src="/media/tutorials/annotations-grafana-dashboard.png" alt="A panel in a Grafana dashboard with log queries from Loki displayed as annotations" caption="Displaying log queries from Loki as annotations" >}} + Being able to combine data from multiple data sources in one graph allows you to correlate information from both Prometheus and Loki. Annotations also work very well alongside alerts. In the next and final section, we will set up an alert for our app `grafana.news` and then we will trigger it. This will provide a quick intro to our new Alerting platform. @@ -289,7 +306,16 @@ Grafana's new alerting platform debuted with Grafana 8. A year later, with Grafa The most basic alert consists of two parts: -1. A _Contact point_ - A Contact point defines how Grafana delivers an alert. When the conditions of an _alert rule_ are met, Grafana notifies the contact points, or channels, configured for that alert. Some popular channels include email, webhooks, Slack notifications, and PagerDuty notifications. +1. A _Contact point_ - A Contact point defines how Grafana delivers an alert. When the conditions of an _alert rule_ are met, Grafana notifies the contact points, or channels, configured for that alert. + + Some popular channels include: + + - Email + - [Webhooks](#create-a-contact-point-for-grafana-managed-alerts) + - [Telegram](https://grafana.com/blog/2023/12/28/how-to-integrate-grafana-alerting-and-telegram/) + - Slack + - PagerDuty + 1. An _Alert rule_ - An Alert rule defines one or more _conditions_ that Grafana regularly evaluates. When these evaluations meet the rule's criteria, the alert is triggered. To begin, let's set up a webhook contact point. Once we have a usable endpoint, we'll write an alert rule and trigger a notification. @@ -299,12 +325,10 @@ To begin, let's set up a webhook contact point. Once we have a usable endpoint, In this step, we'll set up a new contact point. This contact point will use the _webhooks_ channel. In order to make this work, we also need an endpoint for our webhook channel to receive the alert. We will use [requestbin.com](https://requestbin.com) to quickly set up that test endpoint. This way we can make sure that our alert is actually sending a notification somewhere. 1. Browse to [requestbin.com](https://requestbin.com). -1. Under the **Create Request Bin** button -1. From RequestBin, Copy the endpoint URL. +1. Under the **Create Request Bin** button, click the link to create a **public bin** instead. +1. From Request Bin, copy the endpoint URL. -Your request bin is now waiting for the first request. - - +Your Request Bin is now waiting for the first request. Next, let's configure a Contact Point in Grafana's Alerting UI to send notifications to our Request Bin. @@ -314,8 +338,8 @@ Next, let's configure a Contact Point in Grafana's Alerting UI to send notificat 1. In **Integration**, choose **Webhook**. 1. In **URL**, paste the endpoint to your request bin. -1. Click **Test** to send a test alert to your request bin. -1. Navigate back to the request bin you created earlier. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent. +1. Click **Test**, and then click **Send test notification** to send a test alert to your request bin. +1. Navigate back to the Request Bin you created earlier. On the left side, there's now a `POST /` entry. Click it to see what information Grafana sent. 1. Return to Grafana and click **Save contact point**. We have now created a dummy webhook endpoint and created a new Alerting Contact Point in Grafana. Now we can create an alert rule and link it to this new channel. @@ -329,17 +353,24 @@ Now that Grafana knows how to notify us, it's time to set up an alert rule: 1. For **Section 1**, name the rule `fundamentals-test`. 1. For **Section 2**, Find the **query A** box. Choose your Prometheus datasource. Note that the rule type should automatically switch to Grafana-managed alert. 1. Switch to code mode by checking the Builder/Code toggle. -1. Enter the same query that we used in our earlier panel `sum(rate(tns_request_duration_seconds_count[5m])) by(route)` +1. Enter the same Prometheus query that we used in our earlier panel: + + ``` + sum(rate(tns_request_duration_seconds_count[5m])) by(route) + ``` + 1. Press **Preview**. You should see some data returned. -1. Keep expressions “B” and "C" as they are. These expressions (Reduce and Threshold, respectively) come by default when creating a new rule. Expression "B", selects the last value of our query “A”, while the Threshold expression "C" will check if the last value from expression "B" is above a specific value. In addition, the Threshold expression is the alert condition by default. Enter `0.2` as threshold value [You can read more about queries and conditions here](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries). -1. In **Section 3**, in Folder, create a new folder, by typing a name for the folder. This folder will contain our alerts. For example: `fundamentals`. Then, click + add new or hit enter twice. +1. Keep expressions “B” and "C" as they are. These expressions (Reduce and Threshold, respectively) come by default when creating a new rule. Expression "B", selects the last value of our query “A”, while the Threshold expression "C" will check if the last value from expression "B" is above a specific value. In addition, the Threshold expression is the alert condition by default. Enter `0.2` as threshold value. [You can read more about queries and conditions here](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/queries-conditions/#expression-queries). +1. In **Section 3**, in Folder, create a new folder, by clicking `New folder` and typing a name for the folder. This folder will contain our alerts. For example: `fundamentals`. Then, click `create`. 1. In the Evaluation group, repeat the above step to create a new one. We will name it `fundamentals` too. -1. Choose an Evaluation interval (how often the alert will be evaluated). For example, every `30s` (30 seconds). -1. Set the pending period . This is the time that a condition has to be met until the alert enters in Firing state and a notification is sent. Enter `0s`. For the purposes of this tutorial, the evaluation interval is intentionally short. This makes it easier to test. This setting makes Grafana wait until an alert has fired for a given time before Grafana sends the notification. +1. Choose an Evaluation interval (how often the alert will be evaluated). For example, every `10s` (10 seconds). +1. Set the pending period. This is the time that a condition has to be met until the alert enters in Firing state and a notification is sent. Enter `0s`. For the purposes of this tutorial, the evaluation interval is intentionally short. This makes it easier to test. This setting makes Grafana wait until an alert has fired for a given time before Grafana sends the notification. 1. In **Section 4**, you can optionally add some sample text to your summary message. [Read more about message templating here](/docs/grafana/latest/alerting/unified-alerting/message-templating/). 1. Click **Save rule and exit** at the top of the page. 1. In Grafana's sidebar, navigate to **Notification policies**. -1. Under **Default policy**, select **...** › **Edit** and change the **Default contact point** to **RequestBin**. +1. Under **Default policy**, select **...** › **Edit** and change the **Default contact point** from **grafana-default-email** to **RequestBin**. +1. Expand the **Timing options** dropdown and under **Group wait** and **Group interval** update the value to `30s` for testing purposes. Group wait is the time Grafana waits before sending the first notification for a new group of alerts. In contrast, group interval is the time Grafana waits before sending notifications about changes to the group. +1. Click **Update default policy**. As a system grows, admins can use the **Notification policies** setting to organize and match alert rules to specific contact points. @@ -349,10 +380,26 @@ Now that Grafana knows how to notify us, it's time to set up an alert rule: We have now configured an alert rule and a contact point. Now let's see if we can trigger a Grafana Managed Alert by generating some traffic on our sample application. 1. Browse to [localhost:8081](http://localhost:8081). -1. Repeatedly click the vote button or refresh the page to generate a traffic spike. +1. Add a new title and URL, repeatedly click the vote button, or refresh the page to generate a traffic spike. Once the query `sum(rate(tns_request_duration_seconds_count[5m])) by(route)` returns a value greater than `0.2` Grafana will trigger our alert. Browse to the Request Bin we created earlier and find the sent Grafana alert notification with details and metadata. +### Display Grafana Alerts to your dashboard + +In most cases, it's also valuable to display Grafana Alerts as annotations to your dashboard. Let's see how we can configure this. + +1. In Grafana's sidebar, hover over the **Alerting** (bell) icon and then click **Alert rules**. +1. Expand the `fundamentals > fundamentals` folder to view our created alert rule. +1. Click the **Edit** icon and scroll down to **Section 4**. +1. Click the **Link dashboard and panel** button and select the dashboard and panel to which you want the alert to be added as an annotation. +1. Click **Confirm** and **Save rule and exit** to save all the changes. +1. In Grafana's sidebar, navigate to the dashboard by clicking **Dashboards** and selecting the dashboard you created. +1. To test the changes, follow the steps listed to [trigger a Grafana Managed Alert](#trigger-a-grafana-managed-alert). + + You should now see a red, broken heart icon beside the panel name, signifying that the alert has been triggered. An annotation for the alert, represented as a vertical red line, is also displayed. + + {{< figure src="/media/tutorials/grafana-alert-on-dashboard.png" alt="A panel in a Grafana dashboard with alerting and annotations configured" caption="Displaying Grafana Alerts on a dashboard" >}} + ## Summary In this tutorial you learned about fundamental features of Grafana. To do so, we ran several Docker containers on your local machine. When you are ready to clean up this local tutorial environment, run the following command: