Docs: Move image renderer doc to its own section and add configuration doc (#38759)

* Move image renderer doc to its own section and add configuration doc

* revert unwanted changes

* remove unwanted formatting

* revert change

* Improve configuration section

* update default flags for 3.1.0 update

* Clean up and add some links

* fix broken links

* clean up notes & links and small updates

* fix after merge

* fix broken link

* fix typo

* Apply suggestions from code review

Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com>
Co-authored-by: Mitch Seaman <mjseaman@users.noreply.github.com>

* Apply PR feedback

* Update monitoring.md

* Apply suggestions from code review

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update troubleshooting.md

* add alias for old image rendering doc page

Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com>
Co-authored-by: Mitch Seaman <mjseaman@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
This commit is contained in:
Agnès Toulet 2021-09-16 15:53:43 +02:00 committed by GitHub
parent 10066cdb8d
commit b7eac0f655
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 671 additions and 130 deletions

View File

@ -1583,7 +1583,7 @@ ha_engine_address = 127.0.0.1:6379
## [plugin.grafana-image-renderer]
For more information, refer to [Image rendering]({{< relref "image_rendering.md" >}}).
For more information, refer to [Image rendering]({{< relref "../image-rendering/" >}}).
### rendering_timezone
@ -1619,7 +1619,7 @@ It can be useful to set this to `true` when troubleshooting.
### rendering_args
Additional arguments to pass to the headless browser instance. Default is --no-sandbox. The list of Chromium flags can be found at (https://peter.sh/experiments/chromium-command-line-switches/). Separate multiple arguments with commas.
Additional arguments to pass to the headless browser instance. Defaults are `--no-sandbox,--disable-gpu`. The list of Chromium flags can be found at (https://peter.sh/experiments/chromium-command-line-switches/). Separate multiple arguments with commas.
### rendering_chrome_bin

View File

@ -50,8 +50,8 @@ These examples show how often and when reminders are sent for a triggered alert.
## List of supported notifiers
| Name | Type | Supports images | Support alert rule tags |
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- | --- |
| Name | Type | Supports images | Supports alert rule tags |
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- |
| [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no |
| [Discord](#discord) | `discord` | yes | no |
| [Email](#email) | `email` | yes | no |
@ -70,7 +70,7 @@ These examples show how often and when reminders are sent for a triggered alert.
| Telegram | `telegram` | yes | no |
| Threema | `threema` | yes, external only | no |
| VictorOps | `victorops` | yes, external only | yes |
| [Webhook](#webhook) | `webhook` | yes, external only | yes | |
| [Webhook](#webhook) | `webhook` | yes, external only | yes |
### Email
@ -263,7 +263,7 @@ Grafana alert notifications can be sent to [Sensu](<(https://sensu.io)>) Go as e
## Enable images in notifications {#external-image-store}
Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure
[image rendering]({{< relref "../../administration/image_rendering/" >}}).
[image rendering]({{< relref "../../image-rendering/" >}}).
You must configure an [external image storage provider]({{< relref "../../administration/configuration/#external-image-storage" >}}) in order to receive images in alert notifications. If your notification channel requires that the image be publicly accessible (e.g. Slack, PagerDuty), configure a provider which uploads the image to a remote image store like Amazon S3, Webdav, Google Cloud Storage, or Azure Blob Storage. Otherwise, the local provider can be used to serve the image directly from Grafana.

View File

@ -20,7 +20,7 @@ Any changes you make to a dashboard used in a report are reflected the next time
## Requirements
- SMTP must be configured for reports to be sent. Refer to [SMTP]({{< relref "../administration/configuration.md#smtp" >}}) in [Configuration]({{< relref "../administration/configuration.md" >}}) for more information.
- The Image Renderer plugin must be installed or the remote rendering service must be set up. Refer to [Image rendering]({{< relref "../administration/image_rendering.md" >}}) for more information.
- The Image Renderer plugin must be installed or the remote rendering service must be set up. Refer to [Image rendering]({{< relref "../image-rendering/" >}}) for more information.
## Access control
@ -99,7 +99,7 @@ If the time zone is set differently between your Grafana server and its remote i
You can attach a CSV file to the report email for each table panel on the selected dashboard, along with the PDF report. By default, CSVs larger than 10Mb won't be sent to avoid email servers to reject the email. You can increase or decrease this limit in the [reporting configuration]({{< relref "#rendering-configuration" >}}).
This feature relies on the same plugin that supports the [image rendering]({{< relref "../administration/image_rendering.md" >}}) features.
This feature relies on the same plugin that supports the [image rendering]({{< relref "../image-rendering/" >}}) features.
When the CSV file is generated, it is temporarily written to the `csv` folder in the Grafana `data` folder.

View File

@ -0,0 +1,402 @@
+++
title = "Image rendering"
description = "Image rendering"
keywords = ["grafana", "image", "rendering", "plugin"]
aliases = ["/docs/grafana/latest/administration/image_rendering/"]
weight = 55
+++
# Image rendering
Grafana supports automatic rendering of panels as PNG images. This allows Grafana to automatically generate images of your panels to include in [alert notifications]({{< relref "../alerting/old-alerting/notifications.md" >}}), [PDF export]({{< relref "../enterprise/export-pdf.md" >}}), and [Reporting]({{< relref "../enterprise/reporting.md" >}}). PDF Export and Reporting are available only in [Grafana Enterprise]({{< relref "../enterprise/" >}}).
> **Note:** Image rendering of dashboards is not supported at this time.
While an image is being rendered, the PNG image is temporarily written to the file system. When the image is rendered, the PNG image is temporarily written to the `png` folder in the Grafana `data` folder.
A background job runs every 10 minutes and removes temporary images. You can configure how long an image should be stored before being removed by configuring the [temp_data_lifetime]({{< relref "../administration/configuration/#temp_data_lifetime" >}}) setting.
You can also render a PNG by clicking the dropdown arrow next to a panel title, then clicking **Share > Direct link rendered image**.
## Alerting and render limits
Alert notifications can include images, but rendering many images at the same time can overload the server where the renderer is running. For instructions of how to configure this, see [concurrent_render_limit]({{< relref "../administration/configuration/#concurrent_render_limit" >}}).
## Install Grafana Image Renderer plugin
> **Note:** Starting from Grafana v7.0.0, all PhantomJS support has been removed. Please use the Grafana Image Renderer plugin or remote rendering service.
To install the plugin, refer to the [Grafana Image Renderer Installation instructions](https://grafana.com/grafana/plugins/grafana-image-renderer#installation).
## Configuration
The Grafana Image Renderer plugin has a number of configuration options that are used in plugin or remote rendering modes.
In plugin mode, you can specify them directly in the [Grafana configuration file]({{< relref "../administration/configuration/#plugingrafana-image-renderer" >}}).
In remote rendering mode, you can specify them in a `.json` [configuration file](#configuration-file) or, for some of them, you can override the configuration defaults using environment variables.
### Configuration file
You can update your settings by using a configuration file, see [default.json](https://github.com/grafana/grafana-image-renderer/tree/master/default.json) for defaults. Note that any configured environment variable takes precedence over configuration file settings.
You can volume mount your custom configuration file when starting the docker container:
```bash
docker run -d --name=renderer --network=host -v /some/path/config.json:/usr/src/app/config.json grafana/grafana-image-renderer:latest
```
You can see a docker-compose example using a custom configuration file [here](https://github.com/grafana/grafana-image-renderer/tree/master/devenv/docker/custom-config).
### Rendering mode
You can instruct how headless browser instances are created by configuring a rendering mode. Default is `default`, other supported values are `clustered` and `reusable`.
#### Default
Default mode will create a new browser instance on each request. When handling multiple concurrent requests, this mode increases memory usage as it will launch multiple browsers at the same time. If you want to set a maximum number of browser to open, you'll need to use the [clustered mode](#clustered).
> **Note:** When using the `default` mode, it's recommended to not remove the default Chromium flag `--disable-gpu`. When receiving a lot of concurrent requests, not using this flag can cause Puppeteer `newPage` function to freeze, causing request timeouts and leaving browsers open.
```bash
RENDERING_MODE=default
```
```json
{
"rendering": {
"mode": "default"
}
}
```
#### Clustered
With the `clustered` mode, you can configure how many browser instances or incognito pages can execute concurrently. Default is `browser` and will ensure a maximum amount of browser instances can execute concurrently. Mode `context` will ensure a maximum amount of incognito pages can execute concurrently. You can also configure the maximum concurrency allowed, which per default is `5`.
Using a cluster of incognito pages is more performant and consumes less CPU and memory than a cluster of browsers. However, if one page crashes it can bring down the entire browser with it (making all the rendering requests happening at the same time fail). Also, each page isn't guaranteed to be totally clean (cookies and storage might bleed-through as seen [here](https://bugs.chromium.org/p/chromium/issues/detail?id=754576)).
```bash
RENDERING_MODE=clustered
RENDERING_CLUSTERING_MODE=browser
RENDERING_CLUSTERING_MAX_CONCURRENCY=5
```
```json
{
"rendering": {
"mode": "clustered",
"clustering": {
"mode": "browser",
"maxConcurrency": 5
}
}
}
```
#### Reusable (experimental)
When using the rendering mode `reusable`, one browser instance will be created and reused. A new incognito page will be opened for each request. This mode is experimental since, if the browser instance crashes, it will not automatically be restarted. You can achieve a similar behavior using `clustered` mode with a high `maxConcurrency` setting.
```bash
RENDERING_MODE=reusable
```
```json
{
"rendering": {
"mode": "reusable"
}
}
```
#### Optimize the performance, CPU and memory usage of the image renderer
The performance and resources consumption of the different modes depend a lot on the number of concurrent requests your service is handling. To understand how many concurrent requests your service is handling, [monitor your image renderer service]({{< relref "./monitoring/" >}}).
With no concurrent requests, the different modes show very similar performance and CPU / memory usage.
When handling concurrent requests, we see the following trends:
- To improve performance and reduce CPU and memory consumption, use [clustered](#clustered) mode with `RENDERING_CLUSTERING_MODE` set as `context`. This parallelizes incognito pages instead of browsers.
- If you use the [clustered](#clustered) mode with a `maxConcurrency` setting below your average number of concurrent requests, performance will drop as the rendering requests will need to wait for the other to finish before getting access to an incognito page / browser.
To achieve better performance, monitor the machine on which your service is running. If you don't have enough memory and / or CPU, every rendering step will be slower than usual, increasing the duration of every rendering request.
### Other available settings
> **Note:** Please note that not all settings are available using environment variables. If there is no example using environment variable below, it means that you need to update the configuration file.
#### HTTP host
Change the listening host of the HTTP server. Default is unset and will use the local host.
```bash
HTTP_HOST=localhost
```
```json
{
"service": {
"host": "localhost"
}
}
```
#### HTTP port
Change the listening port of the HTTP server. Default is `8081`. Setting `0` will automatically assign a port not in use.
```bash
HTTP_PORT=0
```
```json
{
"service": {
"port": 0
}
}
```
#### Enable Prometheus metrics
You can enable [Prometheus](https://prometheus.io/) metrics endpoint `/metrics` using the environment variable `ENABLE_METRICS`. Node.js and render request duration metrics are included, see [output example](./monitoring/#prometheus-metrics-endpoint-output-example) for details.
Default is `false`.
```bash
ENABLE_METRICS=true
```
```json
{
"service": {
"metrics": {
"enabled": true,
"collectDefaultMetrics": true,
"requestDurationBuckets": [1, 5, 7, 9, 11, 13, 15, 20, 30]
}
}
}
```
#### Log level
Change the log level. Default is `info` and will include log messages with level `error`, `warning` and `info`.
```bash
LOG_LEVEL=debug
```
```json
{
"service": {
"logging": {
"level": "debug",
"console": {
"json": false,
"colorize": true
}
}
}
}
```
#### Verbose logging
Instruct headless browser instance whether to capture and log verbose information when rendering an image. Default is `false` and will only capture and log error messages. When enabled (`true`) debug messages are captured and logged as well.
Note that you need to change log level to `debug`, see above, for the verbose information to be included in the logs.
```bash
RENDERING_VERBOSE_LOGGING=true
```
```json
{
"rendering": {
"verboseLogging": true
}
}
```
#### Capture browser output
Instruct headless browser instance whether to output its debug and error messages into running process of remote rendering service. Default is `false`.
This can be useful to enable (`true`) when troubleshooting.
```bash
RENDERING_DUMPIO=true
```
```json
{
"rendering": {
"dumpio": true
}
}
```
#### Custom Chrome/Chromium
If you already have [Chrome](https://www.google.com/chrome/) or [Chromium](https://www.chromium.org/)
installed on your system, then you can use this instead of the pre-packaged version of Chromium.
> **Note:** Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not compatible with the [Grafana Image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer).
You need to make sure that the Chrome/Chromium executable is available for the Grafana/image rendering service process.
```bash
CHROME_BIN="/usr/bin/chromium-browser"
```
```json
{
"rendering": {
"chromeBin": "/usr/bin/chromium-browser"
}
}
```
#### Start browser with additional arguments
Additional arguments to pass to the headless browser instance. Defaults are `--no-sandbox,--disable-gpu`. The list of Chromium flags can be found [here](https://peter.sh/experiments/chromium-command-line-switches/) and the list of flags used as defaults by Puppeteer can be found [there](https://github.com/puppeteer/puppeteer/blob/main/src/node/Launcher.ts#L172). Multiple arguments is separated with comma-character.
```bash
RENDERING_ARGS=--no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758
```
```json
{
"rendering": {
"args": [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
"--disable-accelerated-2d-canvas",
"--disable-gpu",
"--window-size=1280x758"
]
}
}
```
#### Ignore HTTPS errors
Instruct headless browser instance whether to ignore HTTPS errors during navigation. Per default HTTPS errors are not ignored.
Due to the security risk it's not recommended to ignore HTTPS errors.
```bash
IGNORE_HTTPS_ERRORS=true
```
```json
{
"rendering": {
"ignoresHttpsErrors": true
}
}
```
#### Default timezone
Instruct headless browser instance to use a default timezone when not provided by Grafana, .e.g. when rendering panel image of alert. See [ICUs metaZones.txt](https://cs.chromium.org/chromium/src/third_party/icu/source/data/misc/metaZones.txt?rcl=faee8bc70570192d82d2978a71e2a615788597d1) for a list of supported timezone IDs. Fallbacks to `TZ` environment variable if not set.
```bash
BROWSER_TZ=Europe/Stockholm
```
```json
{
"rendering": {
"timezone": "Europe/Stockholm"
}
}
```
#### Default language
Instruct headless browser instance to use a default language when not provided by Grafana, e.g. when rendering panel image of alert.
Refer to the HTTP header Accept-Language to understand how to format this value.
```json
{
"rendering": {
"acceptLanguage": "fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5"
}
}
```
#### Viewport width
Default viewport width when width is not specified in the rendering request. Default is `1000`.
```json
{
"rendering": {
"width": 1000
}
}
```
#### Viewport height
Default viewport height when height is not specified in the rendering request. Default is `500`.
```json
{
"rendering": {
"height": 500
}
}
```
#### Viewport maximum width
Limit the maximum viewport width that can be requested. Default is `3000`.
```json
{
"rendering": {
"maxWidth": 1000
}
}
```
#### Viewport maximum height
Limit the maximum viewport height that can be requested. Default is `3000`.
```json
{
"rendering": {
"maxHeight": 500
}
}
```
#### Device scale factor
Specify default device scale factor for rendering images. `2` is enough for monitor resolutions, `4` would be better for printed material. Setting a higher value affects performance and memory. Default is `1`.
This can be overridden in the rendering request.
```json
{
"rendering": {
"deviceScaleFactor": 2
}
}
```
#### Maximum device scale factor
Limit the maximum device scale factor that can be requested. Default is `4`.
```json
{
"rendering": {
"maxDeviceScaleFactor": 4
}
}
```

View File

@ -0,0 +1,210 @@
+++
title = "Monitoring the image renderer"
description = "Image rendering monitoring"
keywords = ["grafana", "image", "rendering", "plugin", "monitoring"]
weight = 100
+++
# Monitoring the image renderer
Rendering images requires a lot of memory, mainly because Grafana creates browser instances in the background for the actual rendering. Monitoring your service can help you allocate the right amount of resources to your rendering service and set the right [rendering mode]({{< relref "./#rendering-mode" >}}).
## Enable Prometheus metrics endpoint
Configure this service to expose a Prometheus metrics endpoint. For information on how to configure and monitor this service using Prometheus as a data source, refer to [Grafana Image Rendering Service dashboard](https://grafana.com/grafana/dashboards/12203).
**Metrics endpoint output example:**
```
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.536 1579444523566
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.064 1579444523566
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.6000000000000001 1579444523566
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1579444433
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 52686848 1579444523568
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2055344128 1579444523568
# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 1996390400 1579444523568
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 31 1579444523567
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1573877
# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0.000915922 1579444523567
# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="WriteStream"} 2 1579444523566
nodejs_active_handles{type="Server"} 1 1579444523566
nodejs_active_handles{type="Socket"} 9 1579444523566
nodejs_active_handles{type="ChildProcess"} 2 1579444523566
# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 14 1579444523567
# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge
nodejs_active_requests{type="FSReqCallback"} 2
# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 2 1579444523567
# HELP nodejs_heap_size_total_bytes Process heap size from node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 13725696 1579444523567
# HELP nodejs_heap_size_used_bytes Process heap size used from node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 12068008 1579444523567
# HELP nodejs_external_memory_bytes Nodejs external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 1728962 1579444523567
# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 262144 1579444523567
nodejs_heap_space_size_total_bytes{space="new"} 1048576 1579444523567
nodejs_heap_space_size_total_bytes{space="old"} 9809920 1579444523567
nodejs_heap_space_size_total_bytes{space="code"} 425984 1579444523567
nodejs_heap_space_size_total_bytes{space="map"} 1052672 1579444523567
nodejs_heap_space_size_total_bytes{space="large_object"} 1077248 1579444523567
nodejs_heap_space_size_total_bytes{space="code_large_object"} 49152 1579444523567
nodejs_heap_space_size_total_bytes{space="new_large_object"} 0 1579444523567
# HELP nodejs_heap_space_size_used_bytes Process heap space size used from node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 32296 1579444523567
nodejs_heap_space_size_used_bytes{space="new"} 601696 1579444523567
nodejs_heap_space_size_used_bytes{space="old"} 9376600 1579444523567
nodejs_heap_space_size_used_bytes{space="code"} 286688 1579444523567
nodejs_heap_space_size_used_bytes{space="map"} 704320 1579444523567
nodejs_heap_space_size_used_bytes{space="large_object"} 1064872 1579444523567
nodejs_heap_space_size_used_bytes{space="code_large_object"} 3552 1579444523567
nodejs_heap_space_size_used_bytes{space="new_large_object"} 0 1579444523567
# HELP nodejs_heap_space_size_available_bytes Process heap space size available from node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 229576 1579444523567
nodejs_heap_space_size_available_bytes{space="new"} 445792 1579444523567
nodejs_heap_space_size_available_bytes{space="old"} 417712 1579444523567
nodejs_heap_space_size_available_bytes{space="code"} 20576 1579444523567
nodejs_heap_space_size_available_bytes{space="map"} 343632 1579444523567
nodejs_heap_space_size_available_bytes{space="large_object"} 0 1579444523567
nodejs_heap_space_size_available_bytes{space="code_large_object"} 0 1579444523567
nodejs_heap_space_size_available_bytes{space="new_large_object"} 1047488 1579444523567
# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v14.16.1",major="14",minor="16",patch="1"} 1
# HELP grafana_image_renderer_service_http_request_duration_seconds duration histogram of http responses labeled with: status_code
# TYPE grafana_image_renderer_service_http_request_duration_seconds histogram
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="1",status_code="200"} 0
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="5",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="7",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="9",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="11",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="13",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="15",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="20",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="30",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_bucket{le="+Inf",status_code="200"} 4
grafana_image_renderer_service_http_request_duration_seconds_sum{status_code="200"} 10.492873834
grafana_image_renderer_service_http_request_duration_seconds_count{status_code="200"} 4
# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1
# HELP grafana_image_renderer_http_request_in_flight A gauge of requests currently being served by the image renderer.
# TYPE grafana_image_renderer_http_request_in_flight gauge
grafana_image_renderer_http_request_in_flight 1
# HELP grafana_image_renderer_step_duration_seconds duration histogram of browser steps for rendering an image labeled with: step
# TYPE grafana_image_renderer_step_duration_seconds histogram
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="launch"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="launch"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="launch"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="launch"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="launch"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="launch"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="launch"} 1
grafana_image_renderer_step_duration_seconds_sum{step="launch"} 0.7914972
grafana_image_renderer_step_duration_seconds_count{step="launch"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="newPage"} 1
grafana_image_renderer_step_duration_seconds_sum{step="newPage"} 0.2217868
grafana_image_renderer_step_duration_seconds_count{step="newPage"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="prepare"} 1
grafana_image_renderer_step_duration_seconds_sum{step="prepare"} 0.0819274
grafana_image_renderer_step_duration_seconds_count{step="prepare"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="navigate"} 0
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="navigate"} 1
grafana_image_renderer_step_duration_seconds_sum{step="navigate"} 15.3311258
grafana_image_renderer_step_duration_seconds_count{step="navigate"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_sum{step="panelsRendered"} 0.0205577
grafana_image_renderer_step_duration_seconds_count{step="panelsRendered"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="1",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="2",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="3",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="5",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="screenshot"} 1
grafana_image_renderer_step_duration_seconds_sum{step="screenshot"} 0.2866623
grafana_image_renderer_step_duration_seconds_count{step="screenshot"} 1
# HELP grafana_image_renderer_browser_info A metric with a constant '1 value labeled by version of the browser in use
# TYPE grafana_image_renderer_browser_info gauge
grafana_image_renderer_browser_info{version="HeadlessChrome/79.0.3945.0"} 1
```

View File

@ -1,124 +1,32 @@
+++
title = "Image rendering"
description = ""
keywords = ["grafana", "image", "rendering", "plugin"]
weight = 300
title = "Troubleshooting"
description = "Image rendering troubleshooting"
keywords = ["grafana", "image", "rendering", "plugin", "troubleshooting"]
weight = 115
+++
# Image rendering
# Troubleshoot image rendering
Grafana supports automatic rendering of panels as PNG images. This allows Grafana to automatically generate images of your panels to include in [alert notifications]({{< relref "../alerting/old-alerting/notifications.md" >}}).
In this section, you'll learn how to enable logging for the image renderer and you'll find the most common issues.
> **Note:** Image rendering of dashboards is not supported at this time.
## Enable debug logging
While an image is being rendered, the PNG image is temporarily written to the file system. When the image is rendered, the PNG image is temporarily written to the `png` folder in the Grafana `data` folder.
To troubleshoot the image renderer, different kind of logs are available.
A background job runs every 10 minutes and removes temporary images. You can configure how long an image should be stored before being removed by configuring the [temp-data-lifetime]({{< relref "../administration/configuration/#temp-data-lifetime" >}}) setting.
You can also render a PNG by clicking the dropdown arrow next to a panel title, then clicking **Share > Direct link rendered image**.
## Memory requirements
Minimum free memory recommendation is 16GB on the system doing the rendering.
Rendering images can require a lot of memory, mainly because Grafana creates browser instances in the background for the actual rendering. If multiple images are rendered in parallel, then the rendering has a bigger memory footprint. One advantage of using the remote rendering service is that the rendering will be done on the remote system, so your local system resources will not be affected by rendering.
## Alerting and render limits
Alert notifications can include images, but rendering many images at the same time can overload the server where the renderer is running. For instructions of how to configure this, see [concurrent_render_limit]({{< relref "../administration/configuration/#concurrent_render_limit" >}}).
## Install Grafana Image Renderer plugin
The [Grafana image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) is a plugin that runs on the backend and handles rendering panels and dashboards as PNG images using headless Chrome.
To install the plugin, refer to the [Grafana Image Renderer Installation instructions](https://grafana.com/grafana/plugins/grafana-image-renderer/?tab=installation).
## Run in custom Grafana Docker image
We recommend setting up another Docker container for rendering and using remote rendering. Refer to [Remote rendering service]({{< relref "#remote-rendering-service" >}}) for instructions.
If you still want to install the plugin in the Grafana Docker image, refer to [Build with Grafana Image Renderer plugin pre-installed]({{< relref "../installation/docker/#build-with-grafana-image-renderer-plugin-pre-installed" >}}).
## Remote rendering service
> Requires an internet connection.
The [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) can also be run as a remote HTTP rendering service. In this setup, Grafana renders an image by making a HTTP request to the remote rendering service, which in turn renders the image and returns it back in the HTTP response to Grafana.
You can run the remote HTTP rendering service using Docker or as a standalone Node.js application.
### Run in Docker
The following example shows how to run Grafana and the remote HTTP rendering service in two separate Docker containers using Docker Compose.
Create a `docker-compose.yml` with the following content:
```yaml
version: '2'
services:
grafana:
image: grafana/grafana:main
ports:
- '3000:3000'
environment:
GF_RENDERING_SERVER_URL: http://renderer:8081/render
GF_RENDERING_CALLBACK_URL: http://grafana:3000/
GF_LOG_FILTERS: rendering:debug
renderer:
image: grafana/grafana-image-renderer:latest
ports:
- 8081
```
And then run:
```bash
docker-compose up
```
## Run as standalone Node.js application
The following example describes how to build and run the remote HTTP rendering service as a standalone Node.js application and configure Grafana appropriately.
1. Clone the [Grafana image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) Git repository.
1. Install dependencies and build:
```bash
yarn install --pure-lockfile
yarn run build
```
1. Run the server:
```bash
node build/app.js server --port=8081
```
1. Update Grafana configuration:
```
[rendering]
server_url = http://localhost:8081/render
callback_url = http://localhost:3000/
```
1. Restart Grafana.
## PhantomJS
> Starting from Grafana v7.0.0, all PhantomJS support has been removed. Please use the Grafana Image Renderer plugin or remote rendering service.
## Troubleshoot image rendering
Enable debug log messages for rendering in the Grafana configuration file and inspect the Grafana server log.
You can enable debug log messages for rendering in the Grafana configuration file and inspect the Grafana server logs.
```bash
[log]
filters = rendering:debug
```
### Grafana image renderer plugin and remote rendering service
You can also enable more logs in image renderer service itself by:
- Increasing the [log level]({{< relref "./#log-level" >}}).
- Enabling [verbose logging]({{< relref "./#verbose-logging" >}}).
- [Capturing headless browser output]({{< relref "./#capture-browser-output" >}}).
## Missing libraries
The plugin and rendering service uses [Chromium browser](https://www.chromium.org/) which depends on certain libraries.
If you don't have all of those libraries installed in your system you may encounter errors when trying to render an image, e.g.
@ -175,7 +83,7 @@ On a minimal Centos installation, the following dependencies have been confirmed
libXcomposite libXdamage libXtst cups libXScrnSaver pango atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy liberation-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time trousers xdg-utils xkeyboard-config alsa-lib
```
### Certificate signed by internal certificate authorities
## Certificate signed by internal certificate authorities
In many cases, Grafana runs on internal servers and uses certificates that have not been signed by a CA ([Certificate Authority](https://en.wikipedia.org/wiki/Certificate_authority)) known to Chrome, and therefore cannot be validated. Chrome internally uses NSS ([Network Security Services](https://en.wikipedia.org/wiki/Network_Security_Services)) for cryptographic operations such as the validation of certificates.
@ -187,26 +95,47 @@ t=2019-12-04T12:39:22+0000 lvl=error msg="Rendering failed." logger=context user
t=2019-12-04T12:39:22+0000 lvl=error msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/render/d-solo/zxDJxNaZk/graphite-metrics status=500 remote_addr=192.168.106.101 time_ms=310 size=1722 referer="https://grafana.xxx-xxx/d/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m"
```
(The severity-level `error` in the above messages might be misspelled with a single `r`)
If this happens, then you have to add the certificate to the trust store. If you have the certificate file for the internal root CA in the file `internal-root-ca.crt.pem`, then use these commands to create a user specific NSS trust store for the Grafana user (`grafana` for the purpose of this example) and execute the following steps:
```[root@server ~]# [ -d /usr/share/grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb
**Linux:**
```
[root@server ~]# [ -d /usr/share/grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb
[root@merver ~]# certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n internal-root-ca -t C -i /etc/pki/tls/certs/internal-root-ca.crt.pem
[root@server ~]# chown -R grafana: /usr/share/grafana/.pki/nssdb
```
### Custom Chrome/Chromium
**Windows:**
```
certutil addstore "Root" <path>/internal-root-ca.crt.pem
```
## Custom Chrome/Chromium
As a last resort, if you already have [Chrome](https://www.google.com/chrome/) or [Chromium](https://www.chromium.org/)
installed on your system, then you can configure [Grafana Image renderer plugin](#grafana-image-renderer-plugin) to use this
installed on your system, then you can configure the [Grafana Image renderer plugin](../#custom-chromechromium) to use this
instead of the pre-packaged version of Chromium.
> Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not
> compatible with the [Grafana Image renderer plugin](#grafana-image-renderer-plugin).
> **Note:** Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not
> compatible with the [Grafana Image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer).
To override the path to the Chrome/Chromium executable, set an environment variable and make sure that it's available for the Grafana process. For example:
To override the path to the Chrome/Chromium executable in plugin mode, set an environment variable and make sure that it's available for the Grafana process. For example:
```bash
export GF_PLUGIN_RENDERING_CHROME_BIN="/usr/bin/chromium-browser"
```
In remote rendering mode, you need to set the environment variable or update the configuration file and make sure that it's available for the image rendering service process:
```bash
CHROME_BIN="/usr/bin/chromium-browser"
```
```json
{
"rendering": {
"chromeBin": "/usr/bin/chromium-browser"
}
}
```

View File

@ -141,7 +141,7 @@ Replace `Dockerfile` in above example with `ubuntu.Dockerfile` to build a custom
> Only available in Grafana v6.5 and later. This is experimental.
The [Grafana Image Renderer plugin]({{< relref "../administration/image_rendering/#grafana-image-renderer-plugin" >}}) does not currently work if it is installed in a Grafana Docker image. You can build a custom Docker image by using the `GF_INSTALL_IMAGE_RENDERER_PLUGIN` build argument. This installs additional dependencies needed for the Grafana Image Renderer plugin to run.
The [Grafana Image Renderer plugin]({{< relref "../image-rendering/#grafana-image-renderer-plugin" >}}) does not currently work if it is installed in a Grafana Docker image. You can build a custom Docker image by using the `GF_INSTALL_IMAGE_RENDERER_PLUGIN` build argument. This installs additional dependencies needed for the Grafana Image Renderer plugin to run.
Example of how to build and run:

View File

@ -33,7 +33,7 @@ Minimum recommended CPU: 1
Some features might require more memory or CPUs. Features require more resources include:
- [Server side rendering of images]({{< relref "../administration/image_rendering/#requirements" >}})
- [Server side rendering of images](https://grafana.com/grafana/plugins/grafana-image-renderer#requirements)
- [Alerting]({{< relref "../alerting" >}})
- [Data source proxy]({{< relref "../http_api/data_source" >}})

View File

@ -26,7 +26,7 @@ To share a direct link:
1. Send the copied URL to a Grafana user with authorization to view the link.
1. You also optionally click **Direct link rendered image** to share an image of the panel.
For more information, refer to the topic [Image rendering]({{< relref "../administration/image_rendering.md)" >}}).
For more information, refer to the topic [Image rendering]({{< relref "../image-rendering/" >}}).
Here is an example of a link to a server-side rendered PNG:

View File

@ -196,7 +196,7 @@ Read more about this new feature in [Generic OAuth Authentication]({{< relref ".
Since we announced the deprecation of PhantomJS and the new [Image Renderer Plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) in Grafana [6.4]({{< relref "whats-new-in-v6-4/#phantomjs-deprecation" >}}), weve received bug reports and valuable feedback.
In Grafana 6.5 weve updated documentation to make it easier to understand how to install and troubleshoot possible problems. Read more about [Image Rendering]({{< relref "../administration/image_rendering/" >}}).
In Grafana 6.5 weve updated documentation to make it easier to understand how to install and troubleshoot possible problems. Read more about [Image Rendering]({{< relref "../image-rendering/" >}}).
Please try the [Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) and let us know what you think.

View File

@ -169,7 +169,7 @@ Since then we have been working towards removing PhantomJS. In October 2019, whe
As a replacement for PhantomJS weve developed the [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) which is a plugin that runs on the backend and handles rendering panels and dashboards as PNG images using headless Chromium/Chrome. The [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) can either be installed as a Grafana plugin running in its own process side-by-side with Grafana. or runs as an external HTTP service, hosted using Docker or as a standalone application.
Read more about [Image Rendering]({{< relref "../administration/image_rendering/" >}}) in the documentation for further instructions.
Read more about [Image Rendering]({{< relref "../image-rendering/" >}}) in the documentation for further instructions.
## Query history in Explore out of beta