mirror of
https://github.com/grafana/grafana.git
synced 2024-11-26 02:40:26 -06:00
Docs: Add troubleshooting content (#26871)
* added folder and moved content * added content * updated content * added content * Update troubleshoot-dashboards.md * added content
This commit is contained in:
Diana Payton
GitHub
parent
02e4c2bb9e
commit
eb985ffbee
@ -1,100 +0,0 @@
|
||||
+++
|
||||
title = "Troubleshooting"
|
||||
description = "Guide to troubleshooting Grafana problems"
|
||||
keywords = ["grafana", "troubleshooting", "documentation", "guide"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
parent = "admin"
|
||||
weight = 8
|
||||
+++
|
||||
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
This page lists some useful tools to help troubleshoot common Grafana issues.
|
||||
|
||||
## Visualization and query issues
|
||||
|
||||
{{< imgbox max-width="40%" img="/img/docs/v45/query_inspector.png" caption="Query Inspector" >}}
|
||||
|
||||
The most common problems are related to the query and response from your data source. Even if it looks
|
||||
like a bug or visualization issue in Grafana, it is almost always a problem with the data source query or
|
||||
the data source response.
|
||||
|
||||
To check this you should use query inspector, which was added in Grafana 4.5. The query inspector shows query requests and responses. Refer to the data source page for more information.
|
||||
|
||||
For more on the query inspector read the Grafana Community article [Using Grafana’s Query Inspector to troubleshoot issues](https://community.grafana.com/t/using-grafanas-query-inspector-to-troubleshoot-issues/2630). For older versions of Grafana, refer to the [How troubleshoot metric query issue](https://community.grafana.com/t/how-to-troubleshoot-metric-query-issues/50/2) article.
|
||||
|
||||
## Logging
|
||||
|
||||
If you encounter an error or problem, then you can check the Grafana server log. Usually located at `/var/log/grafana/grafana.log` on Unix systems or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
|
||||
|
||||
You can enable more logging by changing log level in the Grafana configuration file.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
The `grafana-server` process can be instructed to enable certain diagnostics when it starts. This can be helpful
|
||||
when investigating certain performance problems. It's *not* recommended to have these enabled by default.
|
||||
|
||||
### Profiling
|
||||
|
||||
The `grafana-server` can be started with the arguments `-profile` to enable profiling and `-profile-port` to override
|
||||
the default HTTP port (`6060`) where the pprof debugging endpoints will be available, e.g.
|
||||
|
||||
```bash
|
||||
./grafana-server -profile -profile-port=8080
|
||||
```
|
||||
|
||||
Note that `pprof` debugging endpoints are served on a different port than the Grafana HTTP server.
|
||||
|
||||
You can configure or override profiling settings using environment variables:
|
||||
|
||||
```bash
|
||||
export GF_DIAGNOSTICS_PROFILING_ENABLED=true
|
||||
export GF_DIAGNOSTICS_PROFILING_PORT=8080
|
||||
```
|
||||
|
||||
Refer to [Go command pprof](https://golang.org/cmd/pprof/) for more information about how to collect and analyze profiling data.
|
||||
|
||||
### Server side image rendering (RPM-based Linux)
|
||||
|
||||
Server side image (png) rendering is a feature that is optional but very useful when sharing visualizations, for example in alert notifications.
|
||||
|
||||
If the image is missing text make sure you have font packages installed.
|
||||
|
||||
```bash
|
||||
sudo yum install fontconfig
|
||||
sudo yum install freetype*
|
||||
sudo yum install urw-fonts
|
||||
```
|
||||
|
||||
### Tracing
|
||||
|
||||
The `grafana-server` can be started with the arguments `-tracing` to enable tracing and `-tracing-file` to override the default trace file (`trace.out`) where trace result is written to. For example:
|
||||
|
||||
```bash
|
||||
./grafana-server -tracing -tracing-file=/tmp/trace.out
|
||||
```
|
||||
|
||||
You can configure or override profiling settings using environment variables:
|
||||
|
||||
```bash
|
||||
export GF_DIAGNOSTICS_TRACING_ENABLED=true
|
||||
export GF_DIAGNOSTICS_TRACING_FILE=/tmp/trace.out
|
||||
```
|
||||
|
||||
View the trace in a web browser (Go required to be installed):
|
||||
|
||||
```bash
|
||||
go tool trace <trace file>
|
||||
2019/11/24 22:20:42 Parsing trace...
|
||||
2019/11/24 22:20:42 Splitting trace...
|
||||
2019/11/24 22:20:42 Opening browser. Trace viewer is listening on http://127.0.0.1:39735
|
||||
```
|
||||
|
||||
See [Go command trace](https://golang.org/cmd/trace/) for more information about how to analyze trace files.
|
||||
|
||||
## FAQs
|
||||
|
||||
Check out the [FAQ section](https://community.grafana.com/c/howto/faq) on the Grafana Community page for answers to frequently
|
||||
asked questions.
|
||||
@ -95,8 +95,6 @@
|
||||
link: /tutorials/ha_setup/
|
||||
- name: Change home dashboard
|
||||
link: /administration/change-home-dashboard/
|
||||
- name: Troubleshooting
|
||||
link: /installation/troubleshooting/
|
||||
- name: Manage users
|
||||
link: /manage-users
|
||||
children:
|
||||
@ -435,6 +433,16 @@
|
||||
link: /http_api/team/
|
||||
- name: Users API
|
||||
link: /http_api/user/
|
||||
- name: Troubleshooting
|
||||
children:
|
||||
- name: Overview
|
||||
link: /troubleshooting/
|
||||
- name: Enable diagnostics
|
||||
link: /troubleshooting/diagnostics/
|
||||
- name: Troubleshoot dashboards
|
||||
link: /troubleshooting/troubleshoot-dashboards/
|
||||
- name: Troubleshoot queries
|
||||
link: /troubleshooting/troubleshoot-queries/
|
||||
- name: Developers
|
||||
children:
|
||||
- name: Plugins
|
||||
|
||||
@ -48,7 +48,7 @@ View raw query results in a table. This is the data returned by the query with t
|
||||
1. If your panel contains multiple queries or queries multiple nodes, then you have additional options.
|
||||
* **Select result -** Choose which result set data you want to view.
|
||||
* **Transform data**
|
||||
* **Join by time -** View raw data from all your queries at once, one result set per column. Click a column heading to reorder the data.
|
||||
- **Join by time -** View raw data from all your queries at once, one result set per column. Click a column heading to reorder the data.
|
||||
|
||||
View raw query results in a table with field options and options overrides applied:
|
||||
1. Open the **Data** tab in panel inspector.
|
||||
|
||||
43
docs/sources/troubleshooting/_index.md
Normal file
43
docs/sources/troubleshooting/_index.md
Normal file
@ -0,0 +1,43 @@
|
||||
+++
|
||||
title = "Troubleshooting"
|
||||
description = "Guide to troubleshooting Grafana problems"
|
||||
keywords = ["grafana", "troubleshooting", "documentation", "guide"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
weight = 100
|
||||
+++
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
This page lists some tools and advice to help troubleshoot common Grafana issues.
|
||||
|
||||
## Troubleshoot with logs
|
||||
|
||||
If you encounter an error or problem, then you can check the Grafana server log. Usually located at `/var/log/grafana/grafana.log` on Unix systems or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
|
||||
|
||||
You can enable more logging by changing log level in the Grafana configuration file.
|
||||
|
||||
For more information, refer to [Enable debug logging in Grafana CLI]({{< relref "../administration/cli.md#enable-debug-logging" >}}) and the [log section in Configuration]({{< relref "../administration/configuration.md#log" >}}).
|
||||
|
||||
## Troubleshoot transformations
|
||||
|
||||
Order of transformations matters. If the final data output from multiple transformations looks wrong, try changing the transformation order. Each transformation transforms data returned by the previous transformation, not the original raw data.
|
||||
|
||||
For more information, refer to [Debug transformations]({{< relref "../panels/transformations.md#debug-transformations" >}}).
|
||||
|
||||
## Text missing with server-side image rendering (RPM-based Linux)
|
||||
|
||||
Server-side image (png) rendering is a feature that is optional but very useful when sharing visualizations, for example in alert notifications.
|
||||
|
||||
If the image is missing text, then make sure you have font packages installed.
|
||||
|
||||
```bash
|
||||
sudo yum install fontconfig
|
||||
sudo yum install freetype*
|
||||
sudo yum install urw-fonts
|
||||
```
|
||||
|
||||
## FAQs
|
||||
|
||||
Check out the [FAQ section](https://community.grafana.com/c/howto/faq) on the Grafana Community page for answers to frequently
|
||||
asked questions.
|
||||
57
docs/sources/troubleshooting/diagnostics.md
Normal file
57
docs/sources/troubleshooting/diagnostics.md
Normal file
@ -0,0 +1,57 @@
|
||||
+++
|
||||
title = "Enable diagnostics"
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
weight = 200
|
||||
+++
|
||||
|
||||
# Enable diagnostics
|
||||
|
||||
You can set up the `grafana-server` process to enable certain diagnostics when it starts. This can be helpful
|
||||
when investigating certain performance problems. It's *not* recommended to have these enabled by default.
|
||||
|
||||
## Turn on profiling
|
||||
|
||||
The `grafana-server` can be started with the arguments `-profile` to enable profiling and `-profile-port` to override
|
||||
the default HTTP port (`6060`) where the `pprof` debugging endpoints are available, for example:
|
||||
|
||||
```bash
|
||||
./grafana-server -profile -profile-port=8080
|
||||
```
|
||||
|
||||
Note that `pprof` debugging endpoints are served on a different port than the Grafana HTTP server.
|
||||
|
||||
You can configure or override profiling settings using environment variables:
|
||||
|
||||
```bash
|
||||
export GF_DIAGNOSTICS_PROFILING_ENABLED=true
|
||||
export GF_DIAGNOSTICS_PROFILING_PORT=8080
|
||||
```
|
||||
|
||||
Refer to [Go command pprof](https://golang.org/cmd/pprof/) for more information about how to collect and analyze profiling data.
|
||||
|
||||
## Use tracing
|
||||
|
||||
The `grafana-server` can be started with the arguments `-tracing` to enable tracing and `-tracing-file` to override the default trace file (`trace.out`) where trace result is written to. For example:
|
||||
|
||||
```bash
|
||||
./grafana-server -tracing -tracing-file=/tmp/trace.out
|
||||
```
|
||||
|
||||
You can configure or override profiling settings using environment variables:
|
||||
|
||||
```bash
|
||||
export GF_DIAGNOSTICS_TRACING_ENABLED=true
|
||||
export GF_DIAGNOSTICS_TRACING_FILE=/tmp/trace.out
|
||||
```
|
||||
|
||||
View the trace in a web browser (Go required to be installed):
|
||||
|
||||
```bash
|
||||
go tool trace <trace file>
|
||||
2019/11/24 22:20:42 Parsing trace...
|
||||
2019/11/24 22:20:42 Splitting trace...
|
||||
2019/11/24 22:20:42 Opening browser. Trace viewer is listening on http://127.0.0.1:39735
|
||||
```
|
||||
|
||||
See [Go command trace](https://golang.org/cmd/trace/) for more information about how to analyze trace files.
|
||||
44
docs/sources/troubleshooting/troubleshoot-dashboards.md
Normal file
44
docs/sources/troubleshooting/troubleshoot-dashboards.md
Normal file
@ -0,0 +1,44 @@
|
||||
+++
|
||||
title = "Troubleshoot dashboards"
|
||||
description = "Guide to troubleshooting Grafana dashboards"
|
||||
keywords = ["grafana", "troubleshooting", "documentation", "dashboards"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
weight = 100
|
||||
+++
|
||||
|
||||
# Troubleshoot dashboards
|
||||
|
||||
This page provides information to solve common dashboard problems.
|
||||
|
||||
## Dashboard is slow
|
||||
|
||||
- Are you trying to render dozens (or hundreds or thousands) of time-series on a graph? This can cause the browser to lag and feel sluggish. Try using functions like `highestMax` (in Graphite) to reduce the returned series.
|
||||
- Sometimes the series names can be very large. This causes larger response sizes. Try using `alias` to reduce the size of the returned series names.
|
||||
- Are you querying many time-series or for a long range of time? Both of these can cause Grafana or your data source to pull in a lot of data, which may slow it down.
|
||||
- It could be high load on your network infrastructure. If the slowness isn't consistent, this may be the problem.
|
||||
|
||||
## Dashboard refresh rate issues
|
||||
|
||||
By default, Grafana queries your data source every 30 seconds. Setting a low refresh rate on your dashboards puts unnecessary stress on the backend. In many cases, querying this frequently makes no sense, because the data isn't being sent to the system such that changes would be seen.
|
||||
|
||||
We recommend the following:
|
||||
|
||||
- Do not enable auto-refreshing on dashboards, panels, or variables unless you need it. Users can refresh their browser manually, or you can set the refresh rate for a time period that makes sense (every ten minutes, every hour, and so on).
|
||||
- If it is required, then set the refresh rate to once a minute. Again, users can always refresh the dashboard manually.
|
||||
- If your dashboard has a longer time period (such as a week), then you really don't need automated refreshing.
|
||||
|
||||
### Handling or rendering null data is wrong/confusing/weird
|
||||
|
||||
Some applications publish data intermittently; for example, they only post a metric when an event occurs. By
|
||||
default, Grafana graphs connect lines between the data points. This can be very deceiving.
|
||||
|
||||
In the picture below we have enabled:
|
||||
- Points and 3-point radius to highlight where data points are actually present.
|
||||
- **Null value** is set to **connected**.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/troubleshooting/grafana_null_connected.png" max-width="1200px" >}}
|
||||
|
||||
In this graph, we set graph to show bars instead of lines and set the **Null value** to graph **null as zero**. There is a very big different in the visuals.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/troubleshooting/grafana_null_zero.png" max-width="1200px" >}}
|
||||
24
docs/sources/troubleshooting/troubleshoot-queries.md
Normal file
24
docs/sources/troubleshooting/troubleshoot-queries.md
Normal file
@ -0,0 +1,24 @@
|
||||
+++
|
||||
title = "Troubleshoot queries"
|
||||
description = "Guide to troubleshooting Grafana queries"
|
||||
keywords = ["grafana", "troubleshooting", "documentation", "guide", "queries"]
|
||||
type = "docs"
|
||||
[menu.docs]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Troubleshoot queries
|
||||
|
||||
This page provides information to solve common dashboard problems.
|
||||
|
||||
### I get different results when I rearrange my functions
|
||||
|
||||
Function order is very important. Just like in math, the order that you place your functions can affect the result.
|
||||
|
||||
## Inspect your query request and response
|
||||
|
||||
The most common problems are related to the query and response from your data source. Even if it looks
|
||||
like a bug or visualization issue in Grafana, it is almost always a problem with the data source query or
|
||||
the data source response. Start by inspecting your panel query and response.
|
||||
|
||||
For more information, refer to [Inspect a panel]({{< relref "../panels/inspect-panel.md" >}}).
|
||||
Loading…
Reference in New Issue
Block a user