mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Docs: Correlations (#66121)
* Add usage stats about correlations * Add stats.correlations.count to collected stats * Expose grafana_stat_totals_correlations metric * Organize imports * Correlations: Add documentation * Add info about the feature toggle and move it to beta * Remove incorrect bullet point * Remove a bullet point that is not a step * Update punctuation * Update punctuation * Fix typo * Fix typo * Grammar * Rephrase * Clarify * Update transformation.field documentation * Revert automatic merge formatting
This commit is contained in:
Piotr Jamróz
GitHub
parent
f9fdeafe35
commit
f48d31171e
35
docs/sources/administration/correlations/_index.md
Normal file
35
docs/sources/administration/correlations/_index.md
Normal file
@ -0,0 +1,35 @@
|
||||
---
|
||||
title: Correlations
|
||||
weight: 900
|
||||
keywords:
|
||||
- correlations
|
||||
- Explore
|
||||
---
|
||||
|
||||
# Correlations
|
||||
|
||||
You can create interactive links for Explore visualizations to run queries related to presented data by setting up Correlations.
|
||||
|
||||
A correlation defines how data in one [data source]({{< relref "/docs/grafana/latest/datasources/" >}}) is used to query data in another data source. Some examples:
|
||||
|
||||
- an application name returned in a logs data source can be used to query metrics related to that application in a metrics data source, or
|
||||
- a user name returned by an SQL data source can be used to query logs related to that particular user in a logs data source
|
||||
|
||||
[Explore]({{< relref "/docs/grafana/latest/explore/" >}}) takes user-defined correlations to display links inside the visualizations. You can click on a link to run the related query and see results in [Explore Split View]({{< relref "/docs/grafana/latest/explore/#split-and-compare" >}}).
|
||||
|
||||
Explore visualizations that currently support showing links based on correlations:
|
||||
|
||||
- [Logs Panel]({{< relref "./use-correlations-in-visualizations#correlations-in-logs-panel">}})
|
||||
- [Table]({{< relref "./use-correlations-in-visualizations#correlations-in-table">}})
|
||||
|
||||
You can configure correlations using [Administration > Correlation page]({{< relref "/docs/grafana/latest/administration/" >}}) or with [provisioning]({{< relref "/docs/grafana/latest/administration/provisioning" >}}).
|
||||
|
||||
> **Note:** Correlations are available in Grafana 10.0+ as an opt-in beta feature. Modify Grafana [configuration file]({{< relref "/docs/grafana/latest/setup-grafana/configure-grafana/#configuration-file-location" >}}) to enable the `correlations` [feature toggle]({{< relref "/docs/grafana/latest/setup-grafana/configure-grafana/#feature_toggles" >}}) to use it.
|
||||
|
||||
## Example of how links work in Explore once set up
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/correlations-in-explore-10-0.gif" caption="Correlations links in Explore" >}}
|
||||
|
||||
See also:
|
||||
|
||||
{{< section >}}
|
||||
@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Add permissions to create new correlations
|
||||
menuTitle: Add permissions
|
||||
weight: 30
|
||||
---
|
||||
|
||||
# Add permissions to create new correlations
|
||||
|
||||
## Before you begin
|
||||
|
||||
Adding access to create correlations for [Viewers and Editors]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/" >}}) is available with [Role-based access control]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/access-control/" >}}).
|
||||
|
||||
## Add permissions to create correlations
|
||||
|
||||
1. Go to the Administration section in Grafana.
|
||||
1. Open Users page.
|
||||
1. Select the user to be granted access to create correlations.
|
||||
1. Add _Data sources writer_ role.
|
||||
@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Correlation
|
||||
weight: 10
|
||||
---
|
||||
|
||||
# Correlation
|
||||
|
||||
Each correlation is configured with the following options:
|
||||
|
||||
**Label**
|
||||
: Link label, shown in the visualization
|
||||
|
||||
**Description**
|
||||
: Optional description
|
||||
|
||||
**Source data source**
|
||||
: The source of results that have links displayed
|
||||
|
||||
**Results field**
|
||||
: Defines where the link is shown in a visualization
|
||||
|
||||
**Target query**
|
||||
: The target query run when a link is clicked
|
||||
|
||||
**Transformations**
|
||||
: Optional manipulations to the source data included passed to the target query
|
||||
|
||||
Learn how to create correlations using the [Administration page]({{< relref "./create-a-new-correlation#create-a-correlation-in-administration-page" >}}) or with [provisioning]({{< relref "./create-a-new-correlation#create-a-correlation-with-provisioning" >}}).
|
||||
|
||||
## Source data source and result field
|
||||
|
||||
Links are shown in Explore visualizations for the results from the correlation’s source data source. A link is assigned to one of the fields from the result provided in the correlation configuration (the results field). Each visualization displays fields with links in a different way ([Correlations in Logs Panel]({{< relref "./use-correlations-in-visualizations#correlations-in-logs-panel">}}) and see [Correlations in Table]({{< relref "./use-correlations-in-visualizations#correlations-in-table">}}))
|
||||
|
||||
## Target query
|
||||
|
||||
The target query is run when a link is clicked in the visualization. You can use the query editor of the selected target data source to specify the target query. Source data results can be accessed inside the target query with variables.
|
||||
|
||||
### Correlation Variables
|
||||
|
||||
You can use variables inside the target query to access the source data related to the query. Correlations use [Grafana variable syntax]({{< relref "/docs/grafana/latest/dashboards/variables/variable-syntax" >}}). Variables are filled with values from the source results when the link is clicked. There are two types of variables you can use:
|
||||
|
||||
- [field variables]({{< relref "/docs/grafana/latest/panels-visualizations/configure-data-links#field-variables" >}}) (allows to access field values and labels)
|
||||
- correlation variables (allows to access field values and transformations)
|
||||
|
||||
Example: If source results contain a field called “employee”, the value of the field can be accessed with:
|
||||
|
||||
- A field variable ${\_\_data.fields.employee}
|
||||
- A correlation variable that maps the field value above to ${employee}
|
||||
|
||||
In addition to mapping field values to shorter variable names, more correlation variables can be created by applying transformations to existing fields
|
||||
|
||||
For more details, please see the example in [Use variables and transformations in a correlation]({{< relref "./use-variables-and-transformations" >}}).
|
||||
|
||||
### Correlation Transformations
|
||||
|
||||
Correlations provide a way to extract more variables out of field values. The output of transformations is a set of new variables that can be accessed as any other variable.
|
||||
|
||||
There are two types of transformations: logfmt and regular expression.
|
||||
|
||||
Each transformation uses a selected field value as the input. The output of a transformation is a set of new variables based on the type and options of the transformation.
|
||||
|
||||
For more details, please see the example in [Use variables and transformations in a correlation]({{< relref "./use-variables-and-transformations" >}}) for more details.
|
||||
|
||||
### Logfmt transformation
|
||||
|
||||
The logfmt transformation deconstructs a field value containing text formatted with [logfmt key/value pairs](https://brandur.org/logfmt). Each pair becomes a variable with the key being the name of the variable.
|
||||
|
||||
The logfmt transformation only requires specifying the input field name if you would like the transformation to apply to a different field than the results field.
|
||||
Example output variables for field = “host=srv001 endpoint=/test app=foo”:
|
||||
|
||||
| name | value |
|
||||
| :------- | :----- |
|
||||
| host | srv001 |
|
||||
| endpoint | /test |
|
||||
| app | foo |
|
||||
|
||||
### Regular expression transformation
|
||||
|
||||
The regular expression transformation deconstructs a field value based on the provided regular expression.
|
||||
|
||||
Regular expression transformation options:
|
||||
|
||||
**field**
|
||||
: Input field name
|
||||
|
||||
**expression**
|
||||
: Regular expression. Named capture groups are mapped to variables matching the group name. If non-named matching groups are used a variable is created out of the first match. The value overrides the variable matching the input field or a new variable is created if mapValue is provided (check the example below)
|
||||
|
||||
**mapValue**
|
||||
: Used with simple regex groups without named matching groups. By default, the first match overrides the variable with the name of the field that is used as the input. To change that default behavior you can specify the mapValue property. The provided name is used to create a new variable. This can be useful if your target query requires both the exact value and a part of the value extracted with the transformation.
|
||||
|
||||
Example: Assuming the selected field name is “employee” and the field value is “John Doe”.
|
||||
|
||||
Various output variables based on expression and mapValue options:
|
||||
|
||||
| expression | mapValue | output variables | comment |
|
||||
| :---------------- | :------- | :--------------------------- | :------------------------------------------------------------------------------------------------ |
|
||||
| /\\w+ (\\w+)/ | - | employee=Doe | No mapValue provided. The first matching is mapped to the existing field name variable (employee) |
|
||||
| /(\\w+) (\\w+)/ | name | name=John | The first matching is mapped to a new variable called “name” |
|
||||
| /(?\\w+) (?\\w+)/ | - | firstName=John, lastName=Doe | When named groups are used they are the names of the output variables and mapValue is ignored. |
|
||||
| /(?\\w+) (?\\w+)/ | name | firstName=John, lastName=Doe | Same as above |
|
||||
@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Permissions
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# Permissions
|
||||
|
||||
Users with [Viewer base role]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/" >}}) or with [datasources:query RBAC role]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/access-control/" >}}) can:
|
||||
|
||||
- Use correlations in Explore’s visualizations
|
||||
- List all available correlations in read-only mode
|
||||
|
||||
Users with [Admin base role]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/" >}}) or with [datasources:write RBAC role]({{< relref "/docs/grafana/latest/administration/roles-and-permissions/access-control/" >}}) can:
|
||||
|
||||
- Add, edit and delete correlations
|
||||
@ -0,0 +1,120 @@
|
||||
---
|
||||
title: Create a new correlation
|
||||
weight: 40
|
||||
---
|
||||
|
||||
# Create a new correlation
|
||||
|
||||
## Before you begin
|
||||
|
||||
Make sure you have permission to add new correlations. Only users with write permissions to data sources can define new correlations.
|
||||
|
||||
## Create a correlation in Administration page
|
||||
|
||||
1. Go to the Administration section in Grafana.
|
||||
1. Open Correlations page.
|
||||
1. Click the “Add” button in the top right corner.
|
||||
1. Provide a **label** for the correlation.
|
||||
1. Provide an optional **description**.
|
||||
1. Go to the next page.
|
||||
1. Provide **target data source**.
|
||||
1. Provide **target query** using variables.
|
||||
1. Go to the next page.
|
||||
1. Provide **source data source**.
|
||||
1. Provide **results field**.
|
||||
1. Add transformations if you need variables that are not fields in the source data source.
|
||||
1. Click “Add” to add a new transformation.
|
||||
1. Select the type of a transformation.
|
||||
1. Configure transformation depending on the selected type.
|
||||
1. Save correlation.
|
||||
|
||||
You can edit correlation in the same way, but you cannot change the selected data sources.
|
||||
|
||||
## Create a correlation with provisioning
|
||||
|
||||
Provision correlations by extending provisioned data sources. Correlations are defined as a subsection of the source data source configuration:
|
||||
|
||||
```yaml
|
||||
datasources:
|
||||
- name: Data source name # source data source
|
||||
...
|
||||
jsonData:
|
||||
...
|
||||
correlations:
|
||||
- targetUID: uid
|
||||
label: "test"
|
||||
description: "..."
|
||||
config:
|
||||
type: "query"
|
||||
target:
|
||||
expr: "..."
|
||||
field: "name"
|
||||
transformations:
|
||||
- type: regex
|
||||
field: "test"
|
||||
expression: /\w+/
|
||||
mapValue: "other"
|
||||
- type: logfmt
|
||||
field: "test"
|
||||
```
|
||||
|
||||
Description of provisioning properties:
|
||||
|
||||
**targetUID**
|
||||
: Target data source UID
|
||||
|
||||
**label**
|
||||
: Link label
|
||||
|
||||
**description**
|
||||
: Optional description
|
||||
|
||||
**config**
|
||||
: Config object
|
||||
|
||||
**config.type**
|
||||
: Correlation type. “query” is the only supported type at the moment
|
||||
|
||||
**config.target**
|
||||
: [Target query model]({{< relref "#determine-target-query-model-structure" >}})
|
||||
|
||||
**config.field**
|
||||
: Name of the field where link is shown
|
||||
|
||||
**config.transformations (list)**
|
||||
: List of transformation objects
|
||||
|
||||
**transformation.type**
|
||||
: regex, or logfmt
|
||||
|
||||
**transformation.field**
|
||||
: The field that will be transformed. If this is not defined, it will apply the transformation to the data from the correlation's config.field.
|
||||
|
||||
**transformation.expression**
|
||||
: Regex expression (regex transformation only)
|
||||
|
||||
**transformation.mapValue**
|
||||
: New name of the variable from the first regex match (regex transformation only)
|
||||
|
||||
### Determine target query model structure
|
||||
|
||||
When you set up a correlation with admin page you can use the target query editor. When you use provisioning you may need to know the structure of the target query which may not be well documented depending on the plugin. Here is a quick step-by-step guide on how to determine the target query model:
|
||||
|
||||
1. Open Explore.
|
||||
1. Select the data source you want to use as the target of the correlation.
|
||||
1. Open the inspector tab and select “Query”.
|
||||
1. Run a sample query.
|
||||
1. Inspect results.
|
||||
1. Look for the “queries” list object. Each object is created using the query model structure defined by the data source. You can use the same structure in your provisioning file.
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/determine-target-query-structure-inspector-10-0.png" max-width="600px" caption="Query inspector with target query structure" >}}
|
||||
|
||||
The query model in this example is represented by the first entry in the queries list. Properties “refId” and “datasource” are added to all queries in runtime and can be omitted:
|
||||
|
||||
```json
|
||||
{
|
||||
"scenario_id": "random_walk",
|
||||
"alias": "app",
|
||||
"seriesCount: 2
|
||||
}
|
||||
```
|
||||
@ -0,0 +1,29 @@
|
||||
---
|
||||
title: Use correlations in visualizations
|
||||
weight: 70
|
||||
---
|
||||
|
||||
# Use correlations in visualizations
|
||||
|
||||
## Correlations in Logs Panel
|
||||
|
||||
1. Setup a correlation.
|
||||
1. Open Explore.
|
||||
1. Select a data source that you chose as the source data source of the correlation.
|
||||
1. Run a query that results in data containing fields required to build variables in the target query.
|
||||
1. Expand log row details.
|
||||
1. If the selected row contains all the information required to build the target query a link appears in the “Links” section at the bottom.
|
||||
1. Additional information about used variables and their values is shown next to each link.
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/correlations-in-logs-panel-10-0.png" max-width="600px" caption="Correlation links in Logs panel" >}}
|
||||
|
||||
## Correlations in Table
|
||||
|
||||
1. Setup a correlation.
|
||||
1. Open Explore.
|
||||
1. Select a data source that you chose as the source data source of the correlation.
|
||||
1. Run a query that results in data containing fields required to build variables in the target query.
|
||||
1. Links are added to cell rows in the column representing the field with the assigned link ([the results field]({{< relref "/docs/grafana/latest/administration/correlations/correlation-configuration#source-data-source-and-result-field" >}}).
|
||||
1. Cells containing multiple links accessible with a context menu.
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/correlations-in-table-10-0.png" max-width="600px" caption="Correlations links in table" >}}
|
||||
@ -0,0 +1,135 @@
|
||||
---
|
||||
title: Use variables and transformations in a correlation
|
||||
weight: 60
|
||||
---
|
||||
|
||||
# Use variables and transformations in a correlation
|
||||
|
||||
## Before you begin
|
||||
|
||||
This example walks through creating a link in a test data source but the same principles apply to any data source.
|
||||
|
||||
The example emulates a scenario with two data sources:
|
||||
|
||||
- Logs containing lines in the format: “2020-01-01 10:00 level=error message=error service=app1.loginService” stored in a field named “msg”
|
||||
- Metrics for application included in the service name of a log line (e.g. app1)
|
||||
|
||||
Instructions below show how to set up a link that can run metrics query for the host included in each log line with provisioning and regex transformation. Additionally, a link with a query containing the full name of the service is set up to demonstrate the logfmt transformation.
|
||||
|
||||
## Use variables and transformations in provisioning
|
||||
|
||||
1. Add the following provisioning configuration to your Grafana:
|
||||
|
||||
```yaml
|
||||
datasources:
|
||||
- name: Target
|
||||
uid: test-target
|
||||
type: testdata
|
||||
|
||||
- name: Source
|
||||
uid: test-source
|
||||
type: testdata
|
||||
- name: Source
|
||||
uid: test-source
|
||||
type: testdata
|
||||
correlations:
|
||||
- targetUID: test-target
|
||||
label: App metrics
|
||||
description: Application HTTP request metrics
|
||||
config:
|
||||
type: query
|
||||
target:
|
||||
scenario_id: random_walk
|
||||
alias: $${application}
|
||||
field: msg
|
||||
transformations:
|
||||
- type: regex
|
||||
field: msg
|
||||
expression: service=(\w+)\.\w+
|
||||
mapValue: application
|
||||
- targetUID: test-target
|
||||
label: Service metrics
|
||||
description: Service metrics
|
||||
config:
|
||||
type: query
|
||||
target:
|
||||
scenario_id: random_walk
|
||||
alias: $${service}
|
||||
field: msg
|
||||
transformations:
|
||||
- type: logfmt
|
||||
field: msg
|
||||
```
|
||||
|
||||
Two data sources are created: Source (emulating logs data source) and Target (emulating metrics data source):
|
||||
|
||||
- A correlation called “App metrics” is created targeting the Target data source with its UID.
|
||||
- The label and description are provided as text
|
||||
- Each correlation contains the following configuration:
|
||||
- Required correlation type (query)
|
||||
- Target query matching test data source model
|
||||
- “App metrics” correlation contains the following configuration:
|
||||
- Alias is set to ${application} variable (note that in provisioning files $ is used to access environment variables so it has to be [escaped]({{< relref "/docs/grafana/latest/administration/provisioning#using-environment-variables" >}})).
|
||||
- Regular expression transformation is created to extract values from “msg” field
|
||||
- Regular expression transformation is used to capture the application name from the full name of the service stored in the log line.
|
||||
- The output of the transformation is mapped to a variable called “application”.
|
||||
- “Service metrics” correlation is created in a similar way but with logfmt transformation to break down log message and access full name of the service (e.g. “app1.loginService”).
|
||||
- For example, when a logline “2020-01-01 10:00 level=error message=error service=app1.loginService” is provided as the input, the transformation produces new variables: level, message, and service.
|
||||
- “service” variable is used as the alias in the target query.
|
||||
|
||||
1. Navigate to Explore and open “Source” data source.
|
||||
1. Select the “Raw Frames” scenario and provide the following data frames to emulate returning log lines:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"meta": {
|
||||
"preferredVisualisationType": "logs"
|
||||
},
|
||||
"fields": [
|
||||
{
|
||||
"name": "time",
|
||||
"values": [1, 2]
|
||||
},
|
||||
{
|
||||
"name": "msg",
|
||||
"values": [
|
||||
"level=error msg=error service=app1.loginService",
|
||||
"level=debug msg=info service=app2.userProfileService"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
1. Run the query and open log details by clicking on the log line.
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/correlations-in-logs-panel-10-0.png" max-width="600px" caption="Correlation links in Logs panel" >}}
|
||||
|
||||
A link “App metrics” and “Service metrics” show next to variables extracted out of the log line with transformations
|
||||
|
||||
1. Click on the “App metrics” link.
|
||||
1. A split view is opened and the target query is run.
|
||||
1. Notice how the application name from the log line is filled in as the alias property in the target query.
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/interpolated-target-query-10-0.png" max-width="600px" caption="Interpolated target query" >}}
|
||||
|
||||
This allows you to run a specific query based on the source results:
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/target-query-results-10-0.png" max-width="600px" caption="Interpolated target query results" >}}
|
||||
|
||||
1. Go back to the source query and change raw frames’ preferred visualization type to “table” to see how links are displayed in a Table visualization.
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"meta": {
|
||||
"preferredVisualisationType": "table"
|
||||
},
|
||||
"fields": [...]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
1. Run the query and notice how links are created in the Table cell:
|
||||
|
||||
{{< figure src="/static/img/docs/correlations/correlations-in-table-10-0.png" max-width="600px" caption="Correlations links in table" >}}
|
||||
@ -52,6 +52,7 @@ Some stable features are enabled by default. You can disable a stable feature by
|
||||
| `accessControlOnCall` | Access control primitives for OnCall |
|
||||
| `alertingNoNormalState` | Stop maintaining state of alerts that are not firing |
|
||||
| `renderAuthJWT` | Uses JWT-based auth for rendering instead of relying on remote cache |
|
||||
| `correlations` | Correlations page |
|
||||
|
||||
## Alpha feature toggles
|
||||
|
||||
@ -70,7 +71,6 @@ Alpha features might be changed or removed without prior notice.
|
||||
| `storage` | Configurable storage for dashboards, datasources, and resources |
|
||||
| `exploreMixedDatasource` | Enable mixed datasource in Explore |
|
||||
| `newTraceView` | Shows the new trace view design |
|
||||
| `correlations` | Correlations page |
|
||||
| `datasourceQueryMultiStatus` | Introduce HTTP 207 Multi Status for api/ds/query |
|
||||
| `traceToMetrics` | Enable trace to metrics links |
|
||||
| `prometheusWideSeries` | Enable wide series responses in the Prometheus datasource |
|
||||
|
||||
Loading…
Reference in New Issue
Block a user