IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-25 18:55:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

Docs: Update Images in notifications docs to be more readable and instructive (#64227)

Browse Source
This commit is contained in:
George Robinson 2023-03-07 10:52:33 +00:00 committed by GitHub
parent 29982eb194
commit 9d9421154e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 38 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
docs/sources/alerting/manage-notifications/images-in-notifications.md
View File

@ -13,31 +13,26 @@ weight: 500
Images in notifications helps recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.
> **Note**: This feature is not supported for Mimir or Loki rules, or when Grafana sends alert notifications to an external Alertmanager.
> **Note**: This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager
When an alert is fired or resolved Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.
Because a number of contact points, such as email, do not support uploading screenshots at the time of sending a notification; Grafana can also upload the screenshot to a cloud storage service such as Amazon S3, Azure Blob Storage and Google Cloud Storage, where a link to the uploaded screenshot can be added to the notification. However, if using a cloud storage service is not an option then Grafana can be its own cloud storage service such that the screenshot is available under the same domain as Grafana.
Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert, instead you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.
Should either the cloud storage service, or Grafana if acting as its own cloud storage service, be protected by a firewall, gateway service or VPN, then screenshots might not be shown in notifications.
How to choose between uploading screenshots at the time of sending the notification, using a cloud storage service, or using Grafana as its own cloud storage service, depends on which contact points you plan to use and whether you use a firewall, gateway service or VPN.
For example, if a contact point supports uploading images at the time of notification is it not required to use cloud storage. Cloud storage is required when a contact point does not support uploading images at the time of sending a notification, such as email. We don't recommend using cloud storage if the cloud storage service is behind a firewall, gateway service, or VPN, as screenshots might not be shown in notifications.
Once a screenshot has been taken Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to it's internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.
Please refer to the table at the end of this page for a list of contact points and their support for images in notifications.
## Requirements
To use images in notifications, Grafana must be set up to use [image rendering](https://grafana.com/docs/grafana/next/setup-grafana/image-rendering/). You can either install the image rendering plugin or run it as a remote rendering service.
When a screenshot is taken it is saved to the [data]({{< relref "../../setup-grafana/configure-grafana/#paths" >}}) path. This is where screenshots are stored before being sent in a notification or uploaded to a cloud storage service. Grafana must have write-access to this path. If Grafana cannot write to this path then screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt.
If using a [cloud storage service](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#external_image_storage) such as Amazon S3, Azure Blob Storage or Google Cloud Storage, uploaded images need to be accessible outside of a firewall, gateway service or VPN for screenshots to be shown in notifications. Grafana will not delete screenshots from cloud storage. We recommend configuring a retention policy on the bucket to delete screenshots older than 1 month.
If using Grafana as its own cloud storage service then screenshots will be saved to `static_root_path/img/attachments`. `static_root_path` is a configuration option for Grafana and can be found in `defaults.ini`. However, like when using a cloud storage service, images need to be accessible outside of a firewall, gateway service or VPN for screenshots to be shown in notifications.
When using Grafana as its own cloud storage service screenshots are copied from [data]({{< relref "../../setup-grafana/configure-grafana/#paths" >}}) to `static_root_path/img/attachments`. Screenshots older than `temp_data_lifetime` are deleted from [data]({{< relref "../../setup-grafana/configure-grafana/#paths" >}}) but not from `static_root_path/images/attachments`. To delete screenshots from `static_root_path` after a certain amount of time we recommend setting up a CRON job.
1. To use images in notifications, Grafana must be set up to use [image rendering](https://grafana.com/docs/grafana/next/setup-grafana/image-rendering/). You can either install the image rendering plugin or run it as a remote rendering service.
2. When a screenshot is taken it is saved to the [data]({{< relref "../../setup-grafana/configure-grafana/#paths" >}}) folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder otherwise screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt.
3. You should configure Grafana to upload screenshots if sending alerts to integrations other than Discord, Email, Pushover, Slack or Telegram. If Grafana is behind a corporate network or VPN you should use a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage, otherwise you can configure Grafana to upload screenshots to its internal web server.
4. If uploading screenshots to a cloud storage service, uploaded images might need to be accessible outside of your VPN or corporate network as some instant messaging and communication platforms rewrite URLs to go via their CDN. If this is not an option for due to security concerns we recommend using integrations that [supports uploading images]({{<relref "#supported-contact-points">}}) or [disabling images in notifications]({{<relref "#configuration">}}) altogether.
5. When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess, but not impossible. If this is a security concern we recommend using integrations that [supports uploading images]({{<relref "#supported-contact-points">}}) or [disabling images in notifications]({{<relref "#configuration">}}) altogether.
6. Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.
7. If Grafana is configured to upload screenshots to its internal web server then this web server might need to be accessible via the Internet outside your VPN or corporate network as some instant messaging and communication platforms rewrite URLs to go via their CDN. You should understand there are risks involved with this, and at a minimum make sure Grafana is secured with https and that all Grafana users have strong passwords. If this is not an option for you due to security concerns we recommend using integrations that [supports uploading images]({{<relref "#supported-contact-points">}}) or [disabling images in notifications]({{<relref "#configuration">}}) altogether.
8. Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from `static_root_path/images/attachments` after a certain amount of time we recommend setting up a CRON job.
## Configuration
@ -73,36 +68,36 @@ We recommended that `max_concurrent_screenshots` is less than or equal to `concu
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5
## Support for images in contact points
## Supported contact points
Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading images at the time of sending the notification and images uploaded to cloud storage, including when Grafana is acting as its own cloud storage service.
Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.
| Name | Upload images from disk | Include images from URL |
| ----------------------- | --------------------------- | ------------------------- |
| DingDing | No | No |
| Discord | Yes | Yes |
| Email | Yes | Yes |
| Google Hangouts Chat | No | Yes |
| Kafka | No | No |
| Line | No | No |
| Microsoft Teams | No | Yes |
| Opsgenie | No | Yes |
| Pagerduty | No | Yes |
| Prometheus Alertmanager | No | No |
| Pushover | Yes | No |
| Sensu Go | No | No |
| Slack | Yes (when using Bot tokens) | Yes (when using webhooks) |
| Telegram | Yes | No |
| Threema | No | No |
| VictorOps | No | No |
| Webhook | No | Yes |
| Name | Upload from disk | Reference from cloud storage |
| ----------------------- | ---------------------------------------------------------- | -------------------------------------------------------- |
| DingDing | No | No |
| Discord | Yes (Maximum of 10 per notification) | Yes (Maximum of 10 per notification) |
| Email | Yes (Embedded in the email) | Yes |
| Google Hangouts Chat | No | Yes |
| Kafka | No | No |
| Line | No | No |
| Microsoft Teams | No | Yes |
| Opsgenie | No | Yes |
| Pagerduty | No | Yes |
| Prometheus Alertmanager | No | No |
| Pushover | Yes (Maximum of 1 per notification) | No |
| Sensu Go | No | No |
| Slack | Yes (when using Bot tokens, maximum of 5 per notification) | Yes (when using webhooks, maximum of 1 per notification) |
| Telegram | Yes | No |
| Threema | No | No |
| VictorOps | No | No |
| Webhook | No | Yes |
## Limitations
- This feature is not supported for Mimir or Loki rules, or when Grafana sends alert notifications to an external Alertmanager.
- When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown in random.
- Some contact points support at most one image per notification. In this case, the first image associated with an alert will be attached.
- We don't recommend using cloud storage if the cloud storage service is behind a firewall, gateway service, or VPN, as screenshots might not be shown in notifications.
- This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
- A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
- When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
- Screenshots might not be shown in notifications if your cloud storage is behind a firewall, gateway service, or VPN as some instant messaging and communication platforms rewrite URLs to go via their CDN.
## Troubleshooting
@ -113,9 +108,8 @@ If Grafana has been set up to send images in notifications, however notification
3. If the alert is not associated with a dashboard there will be logs for `Cannot take screenshot for alert rule as it is not associated with a dashboard`.
4. If the alert is associated with a dashboard, but no panel in the dashboard, there will be logs for `Cannot take screenshot for alert rule as it is not associated with a panel`.
5. If images cannot be taken because of mis-configuration or an issue with image rendering there will be logs for `Failed to take an image` including the Dashboard UID, Panel ID, and the error message.
6. Check that the contact point supports images in notifications, and the present configuration, as per the table.
7. If the image was uploaded to cloud storage make sure it is public.
8. If images are made available via Grafana's built in web server make sure it is accessible via the Internet.
6. Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.
7. If the image was uploaded to cloud storage make sure that it is public. Screenshots might not be shown in notifications if your cloud storage is behind a firewall, gateway service, or VPN as some instant messaging and communication platforms rewrite URLs to go via their CDN.
## Metrics