diff --git a/docs/sources/administration/configuration.md b/docs/sources/administration/configuration.md index bacf313b423..1b542fb42f2 100644 --- a/docs/sources/administration/configuration.md +++ b/docs/sources/administration/configuration.md @@ -1583,7 +1583,7 @@ ha_engine_address = 127.0.0.1:6379 ## [plugin.grafana-image-renderer] -For more information, refer to [Image rendering]({{< relref "image_rendering.md" >}}). +For more information, refer to [Image rendering]({{< relref "../image-rendering/" >}}). ### rendering_timezone @@ -1619,7 +1619,7 @@ It can be useful to set this to `true` when troubleshooting. ### rendering_args -Additional arguments to pass to the headless browser instance. Default is --no-sandbox. The list of Chromium flags can be found at (https://peter.sh/experiments/chromium-command-line-switches/). Separate multiple arguments with commas. +Additional arguments to pass to the headless browser instance. Defaults are `--no-sandbox,--disable-gpu`. The list of Chromium flags can be found at (https://peter.sh/experiments/chromium-command-line-switches/). Separate multiple arguments with commas. ### rendering_chrome_bin diff --git a/docs/sources/alerting/old-alerting/notifications.md b/docs/sources/alerting/old-alerting/notifications.md index ce953a19fd3..61de7d33a9c 100644 --- a/docs/sources/alerting/old-alerting/notifications.md +++ b/docs/sources/alerting/old-alerting/notifications.md @@ -50,8 +50,8 @@ These examples show how often and when reminders are sent for a triggered alert. ## List of supported notifiers -| Name | Type | Supports images | Support alert rule tags | -| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- | --- | +| Name | Type | Supports images | Supports alert rule tags | +| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- | | [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no | | [Discord](#discord) | `discord` | yes | no | | [Email](#email) | `email` | yes | no | @@ -70,7 +70,7 @@ These examples show how often and when reminders are sent for a triggered alert. | Telegram | `telegram` | yes | no | | Threema | `threema` | yes, external only | no | | VictorOps | `victorops` | yes, external only | yes | -| [Webhook](#webhook) | `webhook` | yes, external only | yes | | +| [Webhook](#webhook) | `webhook` | yes, external only | yes | ### Email @@ -263,7 +263,7 @@ Grafana alert notifications can be sent to [Sensu](<(https://sensu.io)>) Go as e ## Enable images in notifications {#external-image-store} Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure -[image rendering]({{< relref "../../administration/image_rendering/" >}}). +[image rendering]({{< relref "../../image-rendering/" >}}). You must configure an [external image storage provider]({{< relref "../../administration/configuration/#external-image-storage" >}}) in order to receive images in alert notifications. If your notification channel requires that the image be publicly accessible (e.g. Slack, PagerDuty), configure a provider which uploads the image to a remote image store like Amazon S3, Webdav, Google Cloud Storage, or Azure Blob Storage. Otherwise, the local provider can be used to serve the image directly from Grafana. diff --git a/docs/sources/enterprise/reporting.md b/docs/sources/enterprise/reporting.md index c64c5d691bc..4ba8e2134f2 100644 --- a/docs/sources/enterprise/reporting.md +++ b/docs/sources/enterprise/reporting.md @@ -20,7 +20,7 @@ Any changes you make to a dashboard used in a report are reflected the next time ## Requirements - SMTP must be configured for reports to be sent. Refer to [SMTP]({{< relref "../administration/configuration.md#smtp" >}}) in [Configuration]({{< relref "../administration/configuration.md" >}}) for more information. -- The Image Renderer plugin must be installed or the remote rendering service must be set up. Refer to [Image rendering]({{< relref "../administration/image_rendering.md" >}}) for more information. +- The Image Renderer plugin must be installed or the remote rendering service must be set up. Refer to [Image rendering]({{< relref "../image-rendering/" >}}) for more information. ## Access control @@ -99,7 +99,7 @@ If the time zone is set differently between your Grafana server and its remote i You can attach a CSV file to the report email for each table panel on the selected dashboard, along with the PDF report. By default, CSVs larger than 10Mb won't be sent to avoid email servers to reject the email. You can increase or decrease this limit in the [reporting configuration]({{< relref "#rendering-configuration" >}}). -This feature relies on the same plugin that supports the [image rendering]({{< relref "../administration/image_rendering.md" >}}) features. +This feature relies on the same plugin that supports the [image rendering]({{< relref "../image-rendering/" >}}) features. When the CSV file is generated, it is temporarily written to the `csv` folder in the Grafana `data` folder. diff --git a/docs/sources/image-rendering/_index.md b/docs/sources/image-rendering/_index.md new file mode 100644 index 00000000000..8bb9130431a --- /dev/null +++ b/docs/sources/image-rendering/_index.md @@ -0,0 +1,402 @@ ++++ +title = "Image rendering" +description = "Image rendering" +keywords = ["grafana", "image", "rendering", "plugin"] +aliases = ["/docs/grafana/latest/administration/image_rendering/"] +weight = 55 ++++ + +# Image rendering + +Grafana supports automatic rendering of panels as PNG images. This allows Grafana to automatically generate images of your panels to include in [alert notifications]({{< relref "../alerting/old-alerting/notifications.md" >}}), [PDF export]({{< relref "../enterprise/export-pdf.md" >}}), and [Reporting]({{< relref "../enterprise/reporting.md" >}}). PDF Export and Reporting are available only in [Grafana Enterprise]({{< relref "../enterprise/" >}}). + +> **Note:** Image rendering of dashboards is not supported at this time. + +While an image is being rendered, the PNG image is temporarily written to the file system. When the image is rendered, the PNG image is temporarily written to the `png` folder in the Grafana `data` folder. + +A background job runs every 10 minutes and removes temporary images. You can configure how long an image should be stored before being removed by configuring the [temp_data_lifetime]({{< relref "../administration/configuration/#temp_data_lifetime" >}}) setting. + +You can also render a PNG by clicking the dropdown arrow next to a panel title, then clicking **Share > Direct link rendered image**. + +## Alerting and render limits + +Alert notifications can include images, but rendering many images at the same time can overload the server where the renderer is running. For instructions of how to configure this, see [concurrent_render_limit]({{< relref "../administration/configuration/#concurrent_render_limit" >}}). + +## Install Grafana Image Renderer plugin + +> **Note:** Starting from Grafana v7.0.0, all PhantomJS support has been removed. Please use the Grafana Image Renderer plugin or remote rendering service. + +To install the plugin, refer to the [Grafana Image Renderer Installation instructions](https://grafana.com/grafana/plugins/grafana-image-renderer#installation). + +## Configuration + +The Grafana Image Renderer plugin has a number of configuration options that are used in plugin or remote rendering modes. + +In plugin mode, you can specify them directly in the [Grafana configuration file]({{< relref "../administration/configuration/#plugingrafana-image-renderer" >}}). + +In remote rendering mode, you can specify them in a `.json` [configuration file](#configuration-file) or, for some of them, you can override the configuration defaults using environment variables. + +### Configuration file + +You can update your settings by using a configuration file, see [default.json](https://github.com/grafana/grafana-image-renderer/tree/master/default.json) for defaults. Note that any configured environment variable takes precedence over configuration file settings. + +You can volume mount your custom configuration file when starting the docker container: + +```bash +docker run -d --name=renderer --network=host -v /some/path/config.json:/usr/src/app/config.json grafana/grafana-image-renderer:latest +``` + +You can see a docker-compose example using a custom configuration file [here](https://github.com/grafana/grafana-image-renderer/tree/master/devenv/docker/custom-config). + +### Rendering mode + +You can instruct how headless browser instances are created by configuring a rendering mode. Default is `default`, other supported values are `clustered` and `reusable`. + +#### Default + +Default mode will create a new browser instance on each request. When handling multiple concurrent requests, this mode increases memory usage as it will launch multiple browsers at the same time. If you want to set a maximum number of browser to open, you'll need to use the [clustered mode](#clustered). + +> **Note:** When using the `default` mode, it's recommended to not remove the default Chromium flag `--disable-gpu`. When receiving a lot of concurrent requests, not using this flag can cause Puppeteer `newPage` function to freeze, causing request timeouts and leaving browsers open. + +```bash +RENDERING_MODE=default +``` + +```json +{ + "rendering": { + "mode": "default" + } +} +``` + +#### Clustered + +With the `clustered` mode, you can configure how many browser instances or incognito pages can execute concurrently. Default is `browser` and will ensure a maximum amount of browser instances can execute concurrently. Mode `context` will ensure a maximum amount of incognito pages can execute concurrently. You can also configure the maximum concurrency allowed, which per default is `5`. + +Using a cluster of incognito pages is more performant and consumes less CPU and memory than a cluster of browsers. However, if one page crashes it can bring down the entire browser with it (making all the rendering requests happening at the same time fail). Also, each page isn't guaranteed to be totally clean (cookies and storage might bleed-through as seen [here](https://bugs.chromium.org/p/chromium/issues/detail?id=754576)). + +```bash +RENDERING_MODE=clustered +RENDERING_CLUSTERING_MODE=browser +RENDERING_CLUSTERING_MAX_CONCURRENCY=5 +``` + +```json +{ + "rendering": { + "mode": "clustered", + "clustering": { + "mode": "browser", + "maxConcurrency": 5 + } + } +} +``` + +#### Reusable (experimental) + +When using the rendering mode `reusable`, one browser instance will be created and reused. A new incognito page will be opened for each request. This mode is experimental since, if the browser instance crashes, it will not automatically be restarted. You can achieve a similar behavior using `clustered` mode with a high `maxConcurrency` setting. + +```bash +RENDERING_MODE=reusable +``` + +```json +{ + "rendering": { + "mode": "reusable" + } +} +``` + +#### Optimize the performance, CPU and memory usage of the image renderer + +The performance and resources consumption of the different modes depend a lot on the number of concurrent requests your service is handling. To understand how many concurrent requests your service is handling, [monitor your image renderer service]({{< relref "./monitoring/" >}}). + +With no concurrent requests, the different modes show very similar performance and CPU / memory usage. + +When handling concurrent requests, we see the following trends: + +- To improve performance and reduce CPU and memory consumption, use [clustered](#clustered) mode with `RENDERING_CLUSTERING_MODE` set as `context`. This parallelizes incognito pages instead of browsers. +- If you use the [clustered](#clustered) mode with a `maxConcurrency` setting below your average number of concurrent requests, performance will drop as the rendering requests will need to wait for the other to finish before getting access to an incognito page / browser. + +To achieve better performance, monitor the machine on which your service is running. If you don't have enough memory and / or CPU, every rendering step will be slower than usual, increasing the duration of every rendering request. + +### Other available settings + +> **Note:** Please note that not all settings are available using environment variables. If there is no example using environment variable below, it means that you need to update the configuration file. + +#### HTTP host + +Change the listening host of the HTTP server. Default is unset and will use the local host. + +```bash +HTTP_HOST=localhost +``` + +```json +{ + "service": { + "host": "localhost" + } +} +``` + +#### HTTP port + +Change the listening port of the HTTP server. Default is `8081`. Setting `0` will automatically assign a port not in use. + +```bash +HTTP_PORT=0 +``` + +```json +{ + "service": { + "port": 0 + } +} +``` + +#### Enable Prometheus metrics + +You can enable [Prometheus](https://prometheus.io/) metrics endpoint `/metrics` using the environment variable `ENABLE_METRICS`. Node.js and render request duration metrics are included, see [output example](./monitoring/#prometheus-metrics-endpoint-output-example) for details. + +Default is `false`. + +```bash +ENABLE_METRICS=true +``` + +```json +{ + "service": { + "metrics": { + "enabled": true, + "collectDefaultMetrics": true, + "requestDurationBuckets": [1, 5, 7, 9, 11, 13, 15, 20, 30] + } + } +} +``` + +#### Log level + +Change the log level. Default is `info` and will include log messages with level `error`, `warning` and `info`. + +```bash +LOG_LEVEL=debug +``` + +```json +{ + "service": { + "logging": { + "level": "debug", + "console": { + "json": false, + "colorize": true + } + } + } +} +``` + +#### Verbose logging + +Instruct headless browser instance whether to capture and log verbose information when rendering an image. Default is `false` and will only capture and log error messages. When enabled (`true`) debug messages are captured and logged as well. + +Note that you need to change log level to `debug`, see above, for the verbose information to be included in the logs. + +```bash +RENDERING_VERBOSE_LOGGING=true +``` + +```json +{ + "rendering": { + "verboseLogging": true + } +} +``` + +#### Capture browser output + +Instruct headless browser instance whether to output its debug and error messages into running process of remote rendering service. Default is `false`. +This can be useful to enable (`true`) when troubleshooting. + +```bash +RENDERING_DUMPIO=true +``` + +```json +{ + "rendering": { + "dumpio": true + } +} +``` + +#### Custom Chrome/Chromium + +If you already have [Chrome](https://www.google.com/chrome/) or [Chromium](https://www.chromium.org/) +installed on your system, then you can use this instead of the pre-packaged version of Chromium. + +> **Note:** Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not compatible with the [Grafana Image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer). + +You need to make sure that the Chrome/Chromium executable is available for the Grafana/image rendering service process. + +```bash +CHROME_BIN="/usr/bin/chromium-browser" +``` + +```json +{ + "rendering": { + "chromeBin": "/usr/bin/chromium-browser" + } +} +``` + +#### Start browser with additional arguments + +Additional arguments to pass to the headless browser instance. Defaults are `--no-sandbox,--disable-gpu`. The list of Chromium flags can be found [here](https://peter.sh/experiments/chromium-command-line-switches/) and the list of flags used as defaults by Puppeteer can be found [there](https://github.com/puppeteer/puppeteer/blob/main/src/node/Launcher.ts#L172). Multiple arguments is separated with comma-character. + +```bash +RENDERING_ARGS=--no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758 +``` + +```json +{ + "rendering": { + "args": [ + "--no-sandbox", + "--disable-setuid-sandbox", + "--disable-dev-shm-usage", + "--disable-accelerated-2d-canvas", + "--disable-gpu", + "--window-size=1280x758" + ] + } +} +``` + +#### Ignore HTTPS errors + +Instruct headless browser instance whether to ignore HTTPS errors during navigation. Per default HTTPS errors are not ignored. +Due to the security risk it's not recommended to ignore HTTPS errors. + +```bash +IGNORE_HTTPS_ERRORS=true +``` + +```json +{ + "rendering": { + "ignoresHttpsErrors": true + } +} +``` + +#### Default timezone + +Instruct headless browser instance to use a default timezone when not provided by Grafana, .e.g. when rendering panel image of alert. See [ICU’s metaZones.txt](https://cs.chromium.org/chromium/src/third_party/icu/source/data/misc/metaZones.txt?rcl=faee8bc70570192d82d2978a71e2a615788597d1) for a list of supported timezone IDs. Fallbacks to `TZ` environment variable if not set. + +```bash +BROWSER_TZ=Europe/Stockholm +``` + +```json +{ + "rendering": { + "timezone": "Europe/Stockholm" + } +} +``` + +#### Default language + +Instruct headless browser instance to use a default language when not provided by Grafana, e.g. when rendering panel image of alert. +Refer to the HTTP header Accept-Language to understand how to format this value. + +```json +{ + "rendering": { + "acceptLanguage": "fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5" + } +} +``` + +#### Viewport width + +Default viewport width when width is not specified in the rendering request. Default is `1000`. + +```json +{ + "rendering": { + "width": 1000 + } +} +``` + +#### Viewport height + +Default viewport height when height is not specified in the rendering request. Default is `500`. + +```json +{ + "rendering": { + "height": 500 + } +} +``` + +#### Viewport maximum width + +Limit the maximum viewport width that can be requested. Default is `3000`. + +```json +{ + "rendering": { + "maxWidth": 1000 + } +} +``` + +#### Viewport maximum height + +Limit the maximum viewport height that can be requested. Default is `3000`. + +```json +{ + "rendering": { + "maxHeight": 500 + } +} +``` + +#### Device scale factor + +Specify default device scale factor for rendering images. `2` is enough for monitor resolutions, `4` would be better for printed material. Setting a higher value affects performance and memory. Default is `1`. +This can be overridden in the rendering request. + +```json +{ + "rendering": { + "deviceScaleFactor": 2 + } +} +``` + +#### Maximum device scale factor + +Limit the maximum device scale factor that can be requested. Default is `4`. + +```json +{ + "rendering": { + "maxDeviceScaleFactor": 4 + } +} +``` diff --git a/docs/sources/image-rendering/monitoring.md b/docs/sources/image-rendering/monitoring.md new file mode 100644 index 00000000000..6c1f9ba71a7 --- /dev/null +++ b/docs/sources/image-rendering/monitoring.md @@ -0,0 +1,210 @@ ++++ +title = "Monitoring the image renderer" +description = "Image rendering monitoring" +keywords = ["grafana", "image", "rendering", "plugin", "monitoring"] +weight = 100 ++++ + +# Monitoring the image renderer + +Rendering images requires a lot of memory, mainly because Grafana creates browser instances in the background for the actual rendering. Monitoring your service can help you allocate the right amount of resources to your rendering service and set the right [rendering mode]({{< relref "./#rendering-mode" >}}). + +## Enable Prometheus metrics endpoint + +Configure this service to expose a Prometheus metrics endpoint. For information on how to configure and monitor this service using Prometheus as a data source, refer to [Grafana Image Rendering Service dashboard](https://grafana.com/grafana/dashboards/12203). + +**Metrics endpoint output example:** + +``` +# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds. +# TYPE process_cpu_user_seconds_total counter +process_cpu_user_seconds_total 0.536 1579444523566 + +# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds. +# TYPE process_cpu_system_seconds_total counter +process_cpu_system_seconds_total 0.064 1579444523566 + +# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds. +# TYPE process_cpu_seconds_total counter +process_cpu_seconds_total 0.6000000000000001 1579444523566 + +# HELP process_start_time_seconds Start time of the process since unix epoch in seconds. +# TYPE process_start_time_seconds gauge +process_start_time_seconds 1579444433 + +# HELP process_resident_memory_bytes Resident memory size in bytes. +# TYPE process_resident_memory_bytes gauge +process_resident_memory_bytes 52686848 1579444523568 + +# HELP process_virtual_memory_bytes Virtual memory size in bytes. +# TYPE process_virtual_memory_bytes gauge +process_virtual_memory_bytes 2055344128 1579444523568 + +# HELP process_heap_bytes Process heap size in bytes. +# TYPE process_heap_bytes gauge +process_heap_bytes 1996390400 1579444523568 + +# HELP process_open_fds Number of open file descriptors. +# TYPE process_open_fds gauge +process_open_fds 31 1579444523567 + +# HELP process_max_fds Maximum number of open file descriptors. +# TYPE process_max_fds gauge +process_max_fds 1573877 + +# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds. +# TYPE nodejs_eventloop_lag_seconds gauge +nodejs_eventloop_lag_seconds 0.000915922 1579444523567 + +# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name. +# TYPE nodejs_active_handles gauge +nodejs_active_handles{type="WriteStream"} 2 1579444523566 +nodejs_active_handles{type="Server"} 1 1579444523566 +nodejs_active_handles{type="Socket"} 9 1579444523566 +nodejs_active_handles{type="ChildProcess"} 2 1579444523566 + +# HELP nodejs_active_handles_total Total number of active handles. +# TYPE nodejs_active_handles_total gauge +nodejs_active_handles_total 14 1579444523567 + +# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name. +# TYPE nodejs_active_requests gauge +nodejs_active_requests{type="FSReqCallback"} 2 + +# HELP nodejs_active_requests_total Total number of active requests. +# TYPE nodejs_active_requests_total gauge +nodejs_active_requests_total 2 1579444523567 + +# HELP nodejs_heap_size_total_bytes Process heap size from node.js in bytes. +# TYPE nodejs_heap_size_total_bytes gauge +nodejs_heap_size_total_bytes 13725696 1579444523567 + +# HELP nodejs_heap_size_used_bytes Process heap size used from node.js in bytes. +# TYPE nodejs_heap_size_used_bytes gauge +nodejs_heap_size_used_bytes 12068008 1579444523567 + +# HELP nodejs_external_memory_bytes Nodejs external memory size in bytes. +# TYPE nodejs_external_memory_bytes gauge +nodejs_external_memory_bytes 1728962 1579444523567 + +# HELP nodejs_heap_space_size_total_bytes Process heap space size total from node.js in bytes. +# TYPE nodejs_heap_space_size_total_bytes gauge +nodejs_heap_space_size_total_bytes{space="read_only"} 262144 1579444523567 +nodejs_heap_space_size_total_bytes{space="new"} 1048576 1579444523567 +nodejs_heap_space_size_total_bytes{space="old"} 9809920 1579444523567 +nodejs_heap_space_size_total_bytes{space="code"} 425984 1579444523567 +nodejs_heap_space_size_total_bytes{space="map"} 1052672 1579444523567 +nodejs_heap_space_size_total_bytes{space="large_object"} 1077248 1579444523567 +nodejs_heap_space_size_total_bytes{space="code_large_object"} 49152 1579444523567 +nodejs_heap_space_size_total_bytes{space="new_large_object"} 0 1579444523567 + +# HELP nodejs_heap_space_size_used_bytes Process heap space size used from node.js in bytes. +# TYPE nodejs_heap_space_size_used_bytes gauge +nodejs_heap_space_size_used_bytes{space="read_only"} 32296 1579444523567 +nodejs_heap_space_size_used_bytes{space="new"} 601696 1579444523567 +nodejs_heap_space_size_used_bytes{space="old"} 9376600 1579444523567 +nodejs_heap_space_size_used_bytes{space="code"} 286688 1579444523567 +nodejs_heap_space_size_used_bytes{space="map"} 704320 1579444523567 +nodejs_heap_space_size_used_bytes{space="large_object"} 1064872 1579444523567 +nodejs_heap_space_size_used_bytes{space="code_large_object"} 3552 1579444523567 +nodejs_heap_space_size_used_bytes{space="new_large_object"} 0 1579444523567 + +# HELP nodejs_heap_space_size_available_bytes Process heap space size available from node.js in bytes. +# TYPE nodejs_heap_space_size_available_bytes gauge +nodejs_heap_space_size_available_bytes{space="read_only"} 229576 1579444523567 +nodejs_heap_space_size_available_bytes{space="new"} 445792 1579444523567 +nodejs_heap_space_size_available_bytes{space="old"} 417712 1579444523567 +nodejs_heap_space_size_available_bytes{space="code"} 20576 1579444523567 +nodejs_heap_space_size_available_bytes{space="map"} 343632 1579444523567 +nodejs_heap_space_size_available_bytes{space="large_object"} 0 1579444523567 +nodejs_heap_space_size_available_bytes{space="code_large_object"} 0 1579444523567 +nodejs_heap_space_size_available_bytes{space="new_large_object"} 1047488 1579444523567 + +# HELP nodejs_version_info Node.js version info. +# TYPE nodejs_version_info gauge +nodejs_version_info{version="v14.16.1",major="14",minor="16",patch="1"} 1 + +# HELP grafana_image_renderer_service_http_request_duration_seconds duration histogram of http responses labeled with: status_code +# TYPE grafana_image_renderer_service_http_request_duration_seconds histogram +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="1",status_code="200"} 0 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="5",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="7",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="9",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="11",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="13",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="15",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="20",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="30",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_bucket{le="+Inf",status_code="200"} 4 +grafana_image_renderer_service_http_request_duration_seconds_sum{status_code="200"} 10.492873834 +grafana_image_renderer_service_http_request_duration_seconds_count{status_code="200"} 4 + +# HELP up 1 = up, 0 = not up +# TYPE up gauge +up 1 + +# HELP grafana_image_renderer_http_request_in_flight A gauge of requests currently being served by the image renderer. +# TYPE grafana_image_renderer_http_request_in_flight gauge +grafana_image_renderer_http_request_in_flight 1 + +# HELP grafana_image_renderer_step_duration_seconds duration histogram of browser steps for rendering an image labeled with: step +# TYPE grafana_image_renderer_step_duration_seconds histogram +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="launch"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="launch"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="launch"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="launch"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="launch"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="launch"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="launch"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="launch"} 0.7914972 +grafana_image_renderer_step_duration_seconds_count{step="launch"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="newPage"} 0.2217868 +grafana_image_renderer_step_duration_seconds_count{step="newPage"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="prepare"} 0.0819274 +grafana_image_renderer_step_duration_seconds_count{step="prepare"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="navigate"} 0 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="navigate"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="navigate"} 15.3311258 +grafana_image_renderer_step_duration_seconds_count{step="navigate"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="panelsRendered"} 0.0205577 +grafana_image_renderer_step_duration_seconds_count{step="panelsRendered"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.3",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="0.5",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="1",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="2",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="3",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="5",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_bucket{le="+Inf",step="screenshot"} 1 +grafana_image_renderer_step_duration_seconds_sum{step="screenshot"} 0.2866623 +grafana_image_renderer_step_duration_seconds_count{step="screenshot"} 1 + +# HELP grafana_image_renderer_browser_info A metric with a constant '1 value labeled by version of the browser in use +# TYPE grafana_image_renderer_browser_info gauge +grafana_image_renderer_browser_info{version="HeadlessChrome/79.0.3945.0"} 1 +``` diff --git a/docs/sources/administration/image_rendering.md b/docs/sources/image-rendering/troubleshooting.md old mode 100755 new mode 100644 similarity index 50% rename from docs/sources/administration/image_rendering.md rename to docs/sources/image-rendering/troubleshooting.md index 2a1d9bca337..578e7f6e3ca --- a/docs/sources/administration/image_rendering.md +++ b/docs/sources/image-rendering/troubleshooting.md @@ -1,124 +1,32 @@ +++ -title = "Image rendering" -description = "" -keywords = ["grafana", "image", "rendering", "plugin"] -weight = 300 +title = "Troubleshooting" +description = "Image rendering troubleshooting" +keywords = ["grafana", "image", "rendering", "plugin", "troubleshooting"] +weight = 115 +++ -# Image rendering +# Troubleshoot image rendering -Grafana supports automatic rendering of panels as PNG images. This allows Grafana to automatically generate images of your panels to include in [alert notifications]({{< relref "../alerting/old-alerting/notifications.md" >}}). +In this section, you'll learn how to enable logging for the image renderer and you'll find the most common issues. -> **Note:** Image rendering of dashboards is not supported at this time. +## Enable debug logging -While an image is being rendered, the PNG image is temporarily written to the file system. When the image is rendered, the PNG image is temporarily written to the `png` folder in the Grafana `data` folder. +To troubleshoot the image renderer, different kind of logs are available. -A background job runs every 10 minutes and removes temporary images. You can configure how long an image should be stored before being removed by configuring the [temp-data-lifetime]({{< relref "../administration/configuration/#temp-data-lifetime" >}}) setting. - -You can also render a PNG by clicking the dropdown arrow next to a panel title, then clicking **Share > Direct link rendered image**. - -## Memory requirements - -Minimum free memory recommendation is 16GB on the system doing the rendering. - -Rendering images can require a lot of memory, mainly because Grafana creates browser instances in the background for the actual rendering. If multiple images are rendered in parallel, then the rendering has a bigger memory footprint. One advantage of using the remote rendering service is that the rendering will be done on the remote system, so your local system resources will not be affected by rendering. - -## Alerting and render limits - -Alert notifications can include images, but rendering many images at the same time can overload the server where the renderer is running. For instructions of how to configure this, see [concurrent_render_limit]({{< relref "../administration/configuration/#concurrent_render_limit" >}}). - -## Install Grafana Image Renderer plugin - -The [Grafana image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) is a plugin that runs on the backend and handles rendering panels and dashboards as PNG images using headless Chrome. - -To install the plugin, refer to the [Grafana Image Renderer Installation instructions](https://grafana.com/grafana/plugins/grafana-image-renderer/?tab=installation). - -## Run in custom Grafana Docker image - -We recommend setting up another Docker container for rendering and using remote rendering. Refer to [Remote rendering service]({{< relref "#remote-rendering-service" >}}) for instructions. - -If you still want to install the plugin in the Grafana Docker image, refer to [Build with Grafana Image Renderer plugin pre-installed]({{< relref "../installation/docker/#build-with-grafana-image-renderer-plugin-pre-installed" >}}). - -## Remote rendering service - -> Requires an internet connection. - -The [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) can also be run as a remote HTTP rendering service. In this setup, Grafana renders an image by making a HTTP request to the remote rendering service, which in turn renders the image and returns it back in the HTTP response to Grafana. - -You can run the remote HTTP rendering service using Docker or as a standalone Node.js application. - -### Run in Docker - -The following example shows how to run Grafana and the remote HTTP rendering service in two separate Docker containers using Docker Compose. - -Create a `docker-compose.yml` with the following content: - -```yaml -version: '2' - -services: - grafana: - image: grafana/grafana:main - ports: - - '3000:3000' - environment: - GF_RENDERING_SERVER_URL: http://renderer:8081/render - GF_RENDERING_CALLBACK_URL: http://grafana:3000/ - GF_LOG_FILTERS: rendering:debug - renderer: - image: grafana/grafana-image-renderer:latest - ports: - - 8081 -``` - -And then run: - -```bash -docker-compose up -``` - -## Run as standalone Node.js application - -The following example describes how to build and run the remote HTTP rendering service as a standalone Node.js application and configure Grafana appropriately. - -1. Clone the [Grafana image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) Git repository. -1. Install dependencies and build: - - ```bash - yarn install --pure-lockfile - yarn run build - ``` - -1. Run the server: - - ```bash - node build/app.js server --port=8081 - ``` - -1. Update Grafana configuration: - - ``` - [rendering] - server_url = http://localhost:8081/render - callback_url = http://localhost:3000/ - ``` - -1. Restart Grafana. - -## PhantomJS - -> Starting from Grafana v7.0.0, all PhantomJS support has been removed. Please use the Grafana Image Renderer plugin or remote rendering service. - -## Troubleshoot image rendering - -Enable debug log messages for rendering in the Grafana configuration file and inspect the Grafana server log. +You can enable debug log messages for rendering in the Grafana configuration file and inspect the Grafana server logs. ```bash [log] filters = rendering:debug ``` -### Grafana image renderer plugin and remote rendering service +You can also enable more logs in image renderer service itself by: + +- Increasing the [log level]({{< relref "./#log-level" >}}). +- Enabling [verbose logging]({{< relref "./#verbose-logging" >}}). +- [Capturing headless browser output]({{< relref "./#capture-browser-output" >}}). + +## Missing libraries The plugin and rendering service uses [Chromium browser](https://www.chromium.org/) which depends on certain libraries. If you don't have all of those libraries installed in your system you may encounter errors when trying to render an image, e.g. @@ -175,7 +83,7 @@ On a minimal Centos installation, the following dependencies have been confirmed libXcomposite libXdamage libXtst cups libXScrnSaver pango atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy liberation-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time trousers xdg-utils xkeyboard-config alsa-lib ``` -### Certificate signed by internal certificate authorities +## Certificate signed by internal certificate authorities In many cases, Grafana runs on internal servers and uses certificates that have not been signed by a CA ([Certificate Authority](https://en.wikipedia.org/wiki/Certificate_authority)) known to Chrome, and therefore cannot be validated. Chrome internally uses NSS ([Network Security Services](https://en.wikipedia.org/wiki/Network_Security_Services)) for cryptographic operations such as the validation of certificates. @@ -187,26 +95,47 @@ t=2019-12-04T12:39:22+0000 lvl=error msg="Rendering failed." logger=context user t=2019-12-04T12:39:22+0000 lvl=error msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/render/d-solo/zxDJxNaZk/graphite-metrics status=500 remote_addr=192.168.106.101 time_ms=310 size=1722 referer="https://grafana.xxx-xxx/d/zxDJxNaZk/graphite-metrics?orgId=1&refresh=1m" ``` -(The severity-level `error` in the above messages might be misspelled with a single `r`) - If this happens, then you have to add the certificate to the trust store. If you have the certificate file for the internal root CA in the file `internal-root-ca.crt.pem`, then use these commands to create a user specific NSS trust store for the Grafana user (`grafana` for the purpose of this example) and execute the following steps: -```[root@server ~]# [ -d /usr/share/grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb +**Linux:** + +``` +[root@server ~]# [ -d /usr/share/grafana/.pki/nssdb ] || mkdir -p /usr/share/grafana/.pki/nssdb [root@merver ~]# certutil -d sql:/usr/share/grafana/.pki/nssdb -A -n internal-root-ca -t C -i /etc/pki/tls/certs/internal-root-ca.crt.pem [root@server ~]# chown -R grafana: /usr/share/grafana/.pki/nssdb ``` -### Custom Chrome/Chromium +**Windows:** + +``` +certutil –addstore "Root" /internal-root-ca.crt.pem +``` + +## Custom Chrome/Chromium As a last resort, if you already have [Chrome](https://www.google.com/chrome/) or [Chromium](https://www.chromium.org/) -installed on your system, then you can configure [Grafana Image renderer plugin](#grafana-image-renderer-plugin) to use this +installed on your system, then you can configure the [Grafana Image renderer plugin](../#custom-chromechromium) to use this instead of the pre-packaged version of Chromium. -> Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not -> compatible with the [Grafana Image renderer plugin](#grafana-image-renderer-plugin). +> **Note:** Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not +> compatible with the [Grafana Image renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer). -To override the path to the Chrome/Chromium executable, set an environment variable and make sure that it's available for the Grafana process. For example: +To override the path to the Chrome/Chromium executable in plugin mode, set an environment variable and make sure that it's available for the Grafana process. For example: ```bash export GF_PLUGIN_RENDERING_CHROME_BIN="/usr/bin/chromium-browser" ``` + +In remote rendering mode, you need to set the environment variable or update the configuration file and make sure that it's available for the image rendering service process: + +```bash +CHROME_BIN="/usr/bin/chromium-browser" +``` + +```json +{ + "rendering": { + "chromeBin": "/usr/bin/chromium-browser" + } +} +``` diff --git a/docs/sources/installation/docker.md b/docs/sources/installation/docker.md index 89193f8986b..4032619d037 100755 --- a/docs/sources/installation/docker.md +++ b/docs/sources/installation/docker.md @@ -141,7 +141,7 @@ Replace `Dockerfile` in above example with `ubuntu.Dockerfile` to build a custom > Only available in Grafana v6.5 and later. This is experimental. -The [Grafana Image Renderer plugin]({{< relref "../administration/image_rendering/#grafana-image-renderer-plugin" >}}) does not currently work if it is installed in a Grafana Docker image. You can build a custom Docker image by using the `GF_INSTALL_IMAGE_RENDERER_PLUGIN` build argument. This installs additional dependencies needed for the Grafana Image Renderer plugin to run. +The [Grafana Image Renderer plugin]({{< relref "../image-rendering/#grafana-image-renderer-plugin" >}}) does not currently work if it is installed in a Grafana Docker image. You can build a custom Docker image by using the `GF_INSTALL_IMAGE_RENDERER_PLUGIN` build argument. This installs additional dependencies needed for the Grafana Image Renderer plugin to run. Example of how to build and run: diff --git a/docs/sources/installation/requirements.md b/docs/sources/installation/requirements.md index 4b76f034635..de889c6e54b 100644 --- a/docs/sources/installation/requirements.md +++ b/docs/sources/installation/requirements.md @@ -33,7 +33,7 @@ Minimum recommended CPU: 1 Some features might require more memory or CPUs. Features require more resources include: -- [Server side rendering of images]({{< relref "../administration/image_rendering/#requirements" >}}) +- [Server side rendering of images](https://grafana.com/grafana/plugins/grafana-image-renderer#requirements) - [Alerting]({{< relref "../alerting" >}}) - [Data source proxy]({{< relref "../http_api/data_source" >}}) diff --git a/docs/sources/sharing/share-panel.md b/docs/sources/sharing/share-panel.md index 00e224b0f5f..05b3cc10cde 100644 --- a/docs/sources/sharing/share-panel.md +++ b/docs/sources/sharing/share-panel.md @@ -26,7 +26,7 @@ To share a direct link: 1. Send the copied URL to a Grafana user with authorization to view the link. 1. You also optionally click **Direct link rendered image** to share an image of the panel. -For more information, refer to the topic [Image rendering]({{< relref "../administration/image_rendering.md)" >}}). +For more information, refer to the topic [Image rendering]({{< relref "../image-rendering/" >}}). Here is an example of a link to a server-side rendered PNG: diff --git a/docs/sources/whatsnew/whats-new-in-v6-5.md b/docs/sources/whatsnew/whats-new-in-v6-5.md index d49dfcd8ef1..eb3c8f0ed6c 100644 --- a/docs/sources/whatsnew/whats-new-in-v6-5.md +++ b/docs/sources/whatsnew/whats-new-in-v6-5.md @@ -196,7 +196,7 @@ Read more about this new feature in [Generic OAuth Authentication]({{< relref ". Since we announced the deprecation of PhantomJS and the new [Image Renderer Plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) in Grafana [6.4]({{< relref "whats-new-in-v6-4/#phantomjs-deprecation" >}}), we’ve received bug reports and valuable feedback. -In Grafana 6.5 we’ve updated documentation to make it easier to understand how to install and troubleshoot possible problems. Read more about [Image Rendering]({{< relref "../administration/image_rendering/" >}}). +In Grafana 6.5 we’ve updated documentation to make it easier to understand how to install and troubleshoot possible problems. Read more about [Image Rendering]({{< relref "../image-rendering/" >}}). Please try the [Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) and let us know what you think. diff --git a/docs/sources/whatsnew/whats-new-in-v7-0.md b/docs/sources/whatsnew/whats-new-in-v7-0.md index 25cf380d652..b7212bb359c 100644 --- a/docs/sources/whatsnew/whats-new-in-v7-0.md +++ b/docs/sources/whatsnew/whats-new-in-v7-0.md @@ -169,7 +169,7 @@ Since then we have been working towards removing PhantomJS. In October 2019, whe As a replacement for PhantomJS we’ve developed the [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) which is a plugin that runs on the backend and handles rendering panels and dashboards as PNG images using headless Chromium/Chrome. The [Grafana Image Renderer plugin](https://grafana.com/grafana/plugins/grafana-image-renderer) can either be installed as a Grafana plugin running in its own process side-by-side with Grafana. or runs as an external HTTP service, hosted using Docker or as a standalone application. -Read more about [Image Rendering]({{< relref "../administration/image_rendering/" >}}) in the documentation for further instructions. +Read more about [Image Rendering]({{< relref "../image-rendering/" >}}) in the documentation for further instructions. ## Query history in Explore out of beta