IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-25 18:55:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

[DOC] Update Tempo and Pyroscope data source doc (#91704)

Browse Source 
Co-authored-by: Isabel Matwawana <76437239+imatwawana@users.noreply.github.com>
This commit is contained in:
Kim Nylander 2024-08-13 10:39:55 -04:00 committed by GitHub
parent d779dfb0a2
commit b03a709500
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 338 additions and 207 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

150
docs/sources/administration/provisioning/index.md
View File

@ -15,31 +15,34 @@ weight: 600
# Provision Grafana
In previous versions of Grafana, you could only use the API for provisioning data sources and dashboards. But that required the service to be running before you started creating dashboards and you also needed to set up credentials for the HTTP API. In v5.0 we decided to improve this experience by adding a new active provisioning system that uses config files. This will make GitOps more natural as data sources and dashboards can be defined via files that can be version controlled. We hope to extend this system to later add support for users and orgs as well.
Grafana has an active provisioning system that uses configuration files.
This makes GitOps more natural since data sources and dashboards can be defined using files that can be version controlled.
## Config File
## Configuration file
See [Configuration]({{< relref "../../setup-grafana/configure-grafana/" >}}) for more information on what you can configure in `grafana.ini`.
Refer to [Configuration]({{< relref "../../setup-grafana/configure-grafana/" >}}) for more information on what you can configure in `grafana.ini`.
### Config File Locations
### Configuration file locations
- Default configuration from `$WORKING_DIR/conf/defaults.ini`
- Custom configuration from `$WORKING_DIR/conf/custom.ini`
- The custom configuration file path can be overridden using the `--config` parameter
{{% admonition type="note" %}}
{{< admonition type="note" >}}
If you have installed Grafana using the `deb` or `rpm`
packages, then your configuration file is located at
`/etc/grafana/grafana.ini`. This path is specified in the Grafana
init.d script using `--config` file parameter.
{{% /admonition %}}
`init.d` script using the `--config` file parameter.
{{< /admonition >}}
### Using Environment Variables
### Environment variables
It is possible to use environment variable interpolation in all 3 provisioning configuration types. Allowed syntax
is either `$ENV_VAR_NAME` or `${ENV_VAR_NAME}` and can be used only for values not for keys or bigger parts
of the configurations. It is not available in the dashboard's definition files just the dashboard provisioning
You can use environment variable interpolation in all three provisioning configuration types.
The allowed syntax is either `$ENV_VAR_NAME` or `${ENV_VAR_NAME}`, and it can be used only for values, not for keys or larger parts
of the configurations.
It's not available in the dashboard's definition files, just the dashboard provisioning
configuration.
Example:
```yaml
@ -51,13 +54,13 @@ datasources:
password: $PASSWORD
```
If you have a literal `$` in your value and want to avoid interpolation, `$$` can be used.
You can use `$$` if you have a literal `$` in your value and want to avoid interpolation.
<hr />
## Configuration management tools
## Configuration Management Tools
Currently we do not provide any scripts/manifests for configuring Grafana. Rather than spending time learning and creating scripts/manifests for each tool, we think our time is better spent making Grafana easier to provision. Therefore, we heavily rely on the expertise of the community.
Currently, we don't provide any scripts or manifests for configuring Grafana.
Rather than spending time learning and creating scripts or manifests for each tool, we think our time is better spent making Grafana easier to provision.
Therefore, we heavily rely on the expertise of the community.
| Tool | Project |
| --------- | ------------------------------------------------------------------------------------------------------------------------------- |
@ -70,12 +73,8 @@ Currently we do not provide any scripts/manifests for configuring Grafana. Rathe
## Data sources
{{% admonition type="note" %}}
Available in Grafana v5.0 and higher.
{{% /admonition %}}
You can manage data sources in Grafana by adding YAML configuration files in the [`provisioning/datasources`]({{< relref "../../setup-grafana/configure-grafana#provisioning" >}}) directory.
Each config file can contain a list of `datasources` to add or update during startup.
You can manage data sources in Grafana by adding YAML configuration files in the [`provisioning/data sources`]({{< relref "../../setup-grafana/configure-grafana#provisioning" >}}) directory.
Each configuration file can contain a list of `datasources` to add or update during startup.
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.
The configuration file can also list data sources to automatically delete, called `deleteDatasources`.
@ -85,17 +84,17 @@ You can configure Grafana to automatically delete provisioned data sources when
To do so, add `prune: true` to the root of your provisioning file.
With this configuration, Grafana also removes the provisioned data sources if you remove the provisioning file entirely.
{{% admonition type="note" %}}
{{< admonition type="note" >}}
The `prune` parameter is available in Grafana v11.1 and higher.
{{% /admonition %}}
{{< /admonition >}}
### Running multiple Grafana instances
If you run multiple instances of Grafana, add a version number to each data source in the configuration and increase it when you update the configuration.
Grafana updates only data sources with the same or lower version number than specified in the config.
Grafana updates only data sources with the same or lower version number than specified in the configuration.
This prevents old configurations from overwriting newer ones if you have different versions of the `datasource.yaml` file that don't define version numbers, and then restart instances at the same time.
### Example data source config file
### Example data source configuration file
This example provisions a [Graphite data source]({{< relref "../../datasources/graphite" >}}):
@ -179,16 +178,16 @@ datasources:
For provisioning examples of specific data sources, refer to that [data source's documentation]({{< relref "../../datasources" >}}).
#### JSON Data
#### JSON data
Since not all data sources have the same configuration settings, we include only the most common ones as fields.
Not all data sources have the same configuration settings. Only the most common fields are included in examples.
To provision the rest of a data source's settings, include them as a JSON blob in the `jsonData` field.
Common settings in the [built-in core data sources]({{< relref "../../datasources#built-in-core-data-sources" >}}) include:
{{% admonition type="note" %}}
{{< admonition type="note" >}}
Data sources tagged with _HTTP\*_ communicate using the HTTP protocol, which includes all core data source plugins except MySQL, PostgreSQL, and MSSQL.
{{% /admonition %}}
{{< /admonition >}}
| Name | Type | Data source | Description |
| ----------------------------- | ------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@ -249,11 +248,14 @@ For examples of specific data sources' JSON data, refer to that [data source's d
#### Secure JSON Data
Secure JSON data is a map of settings that will be encrypted with [secret key]({{< relref "../../setup-grafana/configure-grafana#secret_key" >}}) from the Grafana config. The purpose of this is only to hide content from the users of the application. This should be used for storing TLS Cert and password that Grafana will append to the request on the server side. All of these settings are optional.
Secure JSON data is a map of settings that are encrypted with a [secret key]({{< relref "../../setup-grafana/configure-grafana#secret_key" >}}) from the Grafana configuration.
The encryption hides content from the users of the application.
This should be used for storing the TLS Cert and password that Grafana appends to the request on the server side.
All of these settings are optional.
{{% admonition type="note" %}}
The _HTTP\*_ tag denotes data sources that communicate using the HTTP protocol, including all core data source plugins except MySQL, PostgreSQL, and MSSQL.
{{% /admonition %}}
{{< admonition type="note" >}}
The _HTTP\*_ tag denotes data sources that communicate using the HTTP protocol, including all core data source plugins except MySQL, PostgreSQL, and MS SQL.
{{< /admonition >}}
| Name | Type | Data source | Description |
| ----------------- | ------ | ---------------------------------- | -------------------------------------------------------- |
@ -270,7 +272,7 @@ The _HTTP\*_ tag denotes data sources that communicate using the HTTP protocol,
#### Custom HTTP headers for data sources
Data sources managed with provisioning can be configured to add HTTP headers to all requests.
The header name is configured in the `jsonData` field and the header value is configured in `secureJsonData`.
Configure the header name in the `jsonData` field and the header value in `secureJsonData`.
```yaml
apiVersion: 1
@ -287,16 +289,14 @@ datasources:
## Plugins
{{% admonition type="note" %}}
Available in Grafana v7.1 and higher.
{{% /admonition %}}
You can manage plugin applications in Grafana by adding one or more YAML configuration files in the [`provisioning/plugins`]({{< relref "../../setup-grafana/configure-grafana#provisioning" >}}) directory.
Each configuration file can contain a list of `apps` that update during start up.
Grafana updates each app to match the configuration file.
You can manage plugin applications in Grafana by adding one or more YAML config files in the [`provisioning/plugins`]({{< relref "../../setup-grafana/configure-grafana#provisioning" >}}) directory. Each config file can contain a list of `apps` that will be updated during start up. Grafana updates each app to match the configuration file.
{{% admonition type="note" %}}
{{< admonition type="note" >}}
This feature enables you to provision plugin configurations, not the plugins themselves.
The plugins must already be installed on the Grafana instance.
{{% /admonition %}}
{{< /admonition >}}
### Example plugin configuration file
@ -324,9 +324,10 @@ apps:
## Dashboards
You can manage dashboards in Grafana by adding one or more YAML config files in the [`provisioning/dashboards`]({{< relref "../../setup-grafana/configure-grafana#dashboards" >}}) directory. Each config file can contain a list of `dashboards providers` that load dashboards into Grafana from the local filesystem.
You can manage dashboards in Grafana by adding one or more YAML configuration files in the [`provisioning/dashboards`]({{< relref "../../setup-grafana/configure-grafana#dashboards" >}}) directory.
Each configuration file can contain a list of `dashboards providers` that load dashboards into Grafana from the local filesystem.
The dashboard provider config file looks somewhat like this:
The dashboard provider configuration file looks somewhat like this:
```yaml
apiVersion: 1
@ -355,40 +356,47 @@ providers:
foldersFromFilesStructure: true
```
When Grafana starts, it will update/insert all dashboards available in the configured path. Then later on poll that path every **updateIntervalSeconds** and look for updated json files and update/insert those into the database.
When Grafana starts, it updates and inserts all dashboards available in the configured path.
Then later on, Grafana polls that path every **updateIntervalSeconds**, looks for updated JSON files, and updates and inserts those into the database.
> **Note:** Dashboards are provisioned to the root level if the `folder` option is missing or empty.
#### Making changes to a provisioned dashboard
It's possible to make changes to a provisioned dashboard in the Grafana UI. However, it is not possible to automatically save the changes back to the provisioning source.
If `allowUiUpdates` is set to `true` and you make changes to a provisioned dashboard, you can `Save` the dashboard then changes will be persisted to the Grafana database.
While you can change a provisioned dashboard in the Grafana UI, those changes can't be saved back to the provisioning source.
If `allowUiUpdates` is set to `true` and you make changes to a provisioned dashboard, you can `Save` the dashboard, then changes persist to the Grafana database.
> **Note:**
> If a provisioned dashboard is saved from the UI and then later updated from the source, the dashboard stored in the database will always be overwritten. The `version` property in the JSON file will not affect this, even if it is lower than the existing dashboard.
>
> If a provisioned dashboard is saved from the UI and the source is removed, the dashboard stored in the database will be deleted unless the configuration option `disableDeletion` is set to true.
{{< admonition type="note" >}}
If a provisioned dashboard is saved from the UI and then later updated from the source, the dashboard stored in the database will always be overwritten. The `version` property in the JSON file won't affect this, even if it's lower than the version of the existing dashboard.
If a provisioned dashboard is saved from the UI and the source is removed, the dashboard stored in the database is deleted unless the configuration option `disableDeletion` is set to `true`.
{{< /admonition >}}
If `allowUiUpdates` is configured to `false`, you are not able to make changes to a provisioned dashboard. When you click `Save`, Grafana brings up a _Cannot save provisioned dashboard_ dialog. The screenshot below illustrates this behavior.
Grafana offers options to export the JSON definition of a dashboard. Either `Copy JSON to Clipboard` or `Save JSON to file` can help you synchronize your dashboard changes back to the provisioning source.
Note: The JSON definition in the input field when using `Copy JSON to Clipboard` or `Save JSON to file` will have the `id` field automatically removed to aid the provisioning workflow.
{{< admonition type="note" >}}
The JSON definition in the input field when using `Copy JSON to Clipboard` or `Save JSON to file` has the `id` field automatically removed to aid the provisioning workflow.
{{< /admonition >}}
{{< figure src="/static/img/docs/v51/provisioning_cannot_save_dashboard.png" max-width="500px" class="docs-image--no-shadow" >}}
### Reusable Dashboard URLs
### Reusable dashboard URLs
If the dashboard in the JSON file contains an [UID]({{< relref "../../dashboards/build-dashboards/view-dashboard-json-model" >}}), Grafana forces insert/update on that UID. This allows you to migrate dashboards between Grafana instances and provisioning Grafana from configuration without breaking the URLs given because the new dashboard URL uses the UID as identifier.
When Grafana starts, it updates/inserts all dashboards available in the configured folders. If you modify the file, then the dashboard is also updated.
By default, Grafana deletes dashboards in the database if the file is removed. You can disable this behavior using the `disableDeletion` setting.
If the dashboard in the JSON file contains an [UID]({{< relref "../../dashboards/build-dashboards/view-dashboard-json-model" >}}), Grafana forces insert/update on that UID.
This allows you to migrate dashboards between Grafana instances and provisioning Grafana from configuration without breaking the URLs given because the new dashboard URL uses the UID as identifier.
When Grafana starts, it updates and inserts all dashboards available in the configured folders.
If you modify the file, then the dashboard is also updated.
By default, Grafana deletes dashboards in the database if the file is removed.
You can disable this behavior using the `disableDeletion` setting.
{{% admonition type="note" %}}
{{< admonition type="note" >}}
Provisioning allows you to overwrite existing dashboards
which leads to problems if you reuse settings that are supposed to be unique.
Be careful not to reuse the same `title` multiple times within a folder
or `uid` within the same installation as this will cause weird behaviors.
{{% /admonition %}}
or `uid` within the same installation as this causes weird behaviors.
{{< /admonition >}}
### Provision folders structure from filesystem to Grafana
@ -406,7 +414,7 @@ For example, to replicate these dashboards structure from the filesystem to Graf
└── /resources_dashboard.json
```
you need to specify just this short provision configuration file.
You need to specify just this short provision configuration file.
```yaml
apiVersion: 1
@ -420,32 +428,24 @@ providers:
foldersFromFilesStructure: true
```
`server` and `application` will become new folders in Grafana menu.
{{% admonition type="note" %}}
`folder` and `folderUid` options should be empty or missing to make `foldersFromFilesStructure` work.
{{% /admonition %}}
{{% admonition type="note" %}}
To provision dashboards to the root level, store them in the root of your `path`.
{{% /admonition %}}
In this example, `server` and `application` become new folders in the Grafana menu.
{{< admonition type="note" >}}
This feature doesn't currently allow you to create nested folder structures, that is, where you have folders within folders.
The `folder` and `folderUid` options should be empty or missing to make `foldersFromFilesStructure` work.
To provision dashboards to the root level, store them in the root of your `path`.
You can't create nested folders structures, where you have folders within folders.
{{< /admonition >}}
## Alerting
For information on provisioning Grafana Alerting, refer to [Provision Grafana Alerting resources]({{< relref "../../alerting/set-up/provision-alerting-resources/" >}}).
### Supported Settings
### Supported settings
The following sections detail the supported settings and secure settings for each alert notification type. Secure settings are stored encrypted in the database and you add them to `secure_settings` in the YAML file instead of `settings`.
{{% admonition type="note" %}}
Secure settings is supported since Grafana v7.2.
{{% /admonition %}}
#### Alert notification `pushover`
| Name | Secure setting |

4
docs/sources/datasources/pyroscope/_index.md
View File

@ -16,8 +16,8 @@ labels:
- cloud
- enterprise
- oss
title: Grafana Pyroscope
weight: 1150
title: Pyroscope
weight: 1350
refs:
flame-graph:
- pattern: /docs/grafana/

111
docs/sources/datasources/pyroscope/configure-pyroscope-data-source.md
View File

@ -33,98 +33,93 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/
- pattern: /docs/grafana-cloud/
destination: docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/configure-tempo-data-source/
provisioning-data-sources:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
- pattern: /docs/grafana-cloud/provision
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
---
# Configure the Grafana Pyroscope data source
To configure basic settings for the data source, complete the following steps:
The Pyroscope data source sets how Grafana connects to your Pyroscope database.
1. Click **Connections** in the left-side menu.
1. Under Your connections, click **Data sources**.
1. Enter `Grafana Pyroscope` in the search bar.
1. Select **Add new data source**.
1. Click **Grafana Pyroscope** to display the **Settings** tab of the data source.
1. Set the data source's basic configuration options.
1. Select **Save & test**.
## Configuration options
You can configure several options for the Pyroscope data source, including the name, HTTP, authentication, querying, and private data source connect.
You can configure the data source using either the data source interface in Grafana or using a configuration file.
This page explains how to set up and enable the data source capabilities using Grafana.
If you make any changes, select **Save & test** to preserve those changes.
![Configuration options for the Pyroscope data source](/media/docs/grafana/data-sources/screenshot-pyroscope-data-source-config.png)
If you're using your own installation of Grafana, you can provision the Pyroscope data source using a YAML configuration file.
For more information about provisioning and available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources).
### Name and default
## Before you begin
**Name**
: Enter a name to specify the data source in panels, queries, and Explore.
To configure a Pyroscope data source, you need administrator rights to your Grafana instance and a Pyroscope instance configured to send data to Grafana.
**Default**
: The default data source is pre-selected for new panels.
If you're provisioning a Pyroscope data source, then you also need administrative rights on the server hosting your Grafana instance.
### HTTP
## Add or modify a data source
The HTTP section is shown in number 1 in the screenshot.
You can use these procedures to configure a new Pyroscope data source or to edit an existing one.
**URL**
: The URL of the Grafana Pyroscope instance, for example, `https://localhost:4100`.
### Create a new data source
**Allowed cookies**
: The Grafana Proxy deletes forwarded cookies. Use this field to specify cookies by name that should be forwarded to the data source.
To configure basic settings for the data source, complete the following steps:
**Timeout**
: HTTP request timeout in seconds.
1. Select **Connections** in the main menu.
1. Enter `Grafana Pyroscope` in the search bar.
1. Select **Grafana Pyroscope**.
1. Select **Add new data source** in the top-right corner of the page.
1. On the **Settings** tab, complete the **Name**, **Connection**, and **Authentication** sections.
### Auth
- Use the **Name** field to specify the name used for the data source in panels, queries, and Explore. Toggle the **Default** switch for the data source to be pre-selected for new panels.
- Under **Connection**, enter the **URL** of the Pyroscope instance. For example, `https://example.com:4100`.
- Complete the [**Authentication** section](#authentication).
The Auth section is shown in number 2 in the screenshot.
1. Optional: Use **Additional settings** to configure other options.
1. Select **Save & test**.
**Basic auth**
: Enable basic authentication to the data source. When activated, it provides **User** and **Password** fields.
### Update an existing data source
**With Credentials**
: Whether credentials, such as cookies or auth headers, should be sent with cross-site requests.
To modify an existing Pyroscope data source:
**TLS Client Auth**
: Toggle on to use client authentication. When enabled, it adds the **Server name**, **Client cert**, and **Client key** fields. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server. These details are encrypted and stored in the Grafana database.
1. Select **Connections** in the main menu.
1. Select **Data sources** to view a list of configured data sources.
1. Select the Pyroscope data source you wish to modify.
1. Optional: Use **Additional settings** to configure or modify other options.
1. After completing your updates, select **Save & test**.
**With CA Cert**
: Activate this option to verify self-signed TLS certificates.
## Authentication
**Skip TLS Verify**
: When activated, it bypasses TLS certificate verification.
Use this section to select an authentication method to access the data source.
**Forward OAuth Identity**
: When activated, the user’s upstream OAuth 2.0 identity is forwarded to the data source along with their access token.
{{< admonition type="note" >}}
Use Transport Layer Security (TLS) for an additional layer of security when working with Pyroscope.
For additional information on setting up TLS encryption with Pyroscope, refer to [Pyroscope configuration](https://grafana.com/docs/pyroscope/<PYROSCOPE_VERSION>/configure-server/reference-configuration-parameters/).
{{< /admonition >}}
**Custom HTTP Headers**
: Select Add header to add Header and Value fields.
[//]: # 'Shared content for authentication section procedure in data sources'
**Header**
: Add a custom header. This allows custom headers to be passed based on the needs of your Pyroscope instance.
{{< docs/shared source="grafana" lookup="datasources/datasouce-authentication.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}
**Value**
: The value of the header.
## Additional settings
Use the down arrow to expand the **Additional settings** section to view these options.
### Advanced HTTP settings
The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies that should be forwarded to the data source by name.
The **Timeout** field sets the HTTP request timeout in seconds.
### Querying
The **Querying** section is shown in number 3 in the screenshot.
**Minimum step** is used for queries returning time-series data. The default value is 15 seconds.
Adjusting this option can help prevent gaps when you zoom in to profiling data.
### Private data source connect
The **Private data source connect** section is shown in number 4 in the screenshot.
[//]: # 'Shared content for authentication section procedure in data sources'
This feature is only available in Grafana Cloud.
This option lets you query data that lives within a secured network without opening the network to inbound traffic from Grafana Cloud.
Use the drop-down box to select a configured private data sources.
Select **Manage private data source connect** to configure and manage any private data sources you have configured.
For more information, refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/).
{{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}

150
docs/sources/datasources/tempo/configure-tempo-data-source.md
View File

@ -47,7 +47,7 @@ refs:
provisioning-data-sources:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
- pattern: /docs/grafana-cloud/
- pattern: /docs/grafana-cloud/provision
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
explore:
- pattern: /docs/grafana/
@ -56,66 +56,108 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
---
# Configure the Tempo data source
# Configure a Tempo data source
The Tempo data source sets how Grafana connects to your Tempo database and lets you configure features and integrations with other telemetry signals.
To configure basic settings for the Tempo data source, complete the following steps:
You can configure the data source using either the data source interface in Grafana or using a configuration file.
This page explains how to set up and enable the data source capabilities using Grafana.
1. Click **Connections** in the left-side menu.
1. Under Your connections, click **Data sources**.
1. Enter `Tempo` in the search bar.
1. Select **Tempo**.
If you're using your own installation of Grafana, you can provision the Tempo data source using a YAML configuration file.
1. On the **Settings** tab, set the data source's basic configuration options:
Depending upon your tracing environment, you may have more than one Tempo instance.
Grafana supports multiple Tempo data sources.
| Name | Description |
| -------------- | ------------------------------------------------------------------------ |
| **Name** | Sets the name you use to refer to the data source in panels and queries. |
| **Default** | Sets the data source that's pre-selected for new panels. |
| **URL** | Sets the URL of the Tempo instance, such as `http://tempo`. |
| **Basic Auth** | Enables authentication to the Tempo data source. |
| **User** | Sets the user name for basic authentication. |
| **Password** | Sets the password for basic authentication. |
## Before you begin
You can also configure settings specific to the Tempo data source.
To configure a Tempo data source, you need administrator rights to your Grafana instance and a Tempo instance configured to send tracing data to Grafana.
This video explains how to add data sources, including Loki, Tempo, and Mimir, to Grafana and Grafana Cloud. Tempo data source set up starts at 4:58 in the video.
If you're provisioning a Tempo data source, then you also need administrative rights on the server hosting your Grafana instance.
Refer to [Provision the data source](#provision-the-data-source) for next steps.
{{< youtube id="cqHO0oYW6Ic" start="298" >}}
![Provisioned data source warning](/media/docs/grafana/data-sources/tempo/tempo-data-source-provisioned-error.png)
## Add or modify a data source
You can use these procedures to configure a new Tempo data source or to edit an existing one.
### Add a new data source
Follow these steps to set up a new Tempo data source:
1. Select **Connections** in the main menu.
1. Enter `Tempo` in the search bar.
1. Select **Tempo**.
1. Select **Add new data source** in the top-right corner of the page.
1. On the **Settings** tab, complete the **Name**, **Connection**, and **Authentication** sections.
- Use the **Name** field to specify the name used for the data source in panels, queries, and Explore. Toggle the **Default** switch for the data source to be pre-selected for new panels.
- Under **Connection**, enter the **URL** of the Tempo instance, for example, `https://example.com:4100`.
- Complete the [**Authentication** section](#authentication).
1. Optional: Configure other sections to add capabilities to your tracing data. Refer to the additional procedures for instructions.
1. Select **Save & test**.
### Update an existing data source
To modify an existing Tempo data source:
1. Select **Connections** in the main menu.
1. Select **Data sources** to view a list of configured data sources.
1. Select the Tempo data source you wish to modify.
1. Configure or update additional sections to add capabilities to your tracing data. Refer to the additional procedures for instructions.
1. After completing your updates, select **Save & test**.
## Authentication
Use this section to select an authentication method to access the data source.
{{< admonition type="note" >}}
Use Transport Layer Security (TLS) for an additional layer of security when working with Tempo.
For additional information on setting up TLS encryption with Tempo, refer to [Configure TLS communication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/network/tls/) and [Tempo configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/).
{{< /admonition >}}
[//]: # 'Shared content for authentication section procedure in data sources'
{{< docs/shared source="grafana" lookup="datasources/datasouce-authentication.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}
## Streaming
<!-- The traceQLStreaming toggle will be deprecated in Grafana 11.2 and removed in 11.3. -->
Streaming enables TraceQL query results to be displayed as they become available. Without streaming, no results are displayed until all results have returned.
Streaming enables TraceQL query results to be displayed as they become available.
Without streaming, no results are displayed until all results have returned.
{{< docs/public-preview product="TraceQL streaming results" >}}
### Requirements
To use streaming, you need to:
- Be running Tempo version 2.2 or newer, or Grafana Enterprise Traces (GET) version 2.2 or newer, or be using Grafana Cloud Traces.
- Run Tempo version 2.2 or newer, or Grafana Enterprise Traces (GET) version 2.2 or newer, or use Grafana Cloud Traces.
- For self-managed Tempo or GET instances: If your Tempo or GET instance is behind a load balancer or proxy that doesn't supporting gRPC or HTTP2, streaming may not work and should be disabled.
### Activate streaming
For streaming to work for a particular Tempo data source, set your Grafana's `traceQLStreaming` [feature toggle](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/) to true and set **Streaming** to enabled in your Tempo data source configuration.
You can activate streaming by either setting the `traceQLStreaming` [feature toggle](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/) to true or by activating the **Streaming** toggle in the Tempo data source.
![Streaming section in Tempo data source](/media/docs/grafana/data-sources/tempo-data-source-streaming-v11.2.png)
If you are using Grafana Cloud, the `traceQLStreaming` feature toggle is already set to `true` by default.
If the Tempo data source is set to allow streaming but the `traceQLStreaming` feature toggle is set to `false` in Grafana, no streaming will occur.
If the Tempo data source is set to allow streaming but the `traceQLStreaming` feature toggle is set to `false` in Grafana, streaming occurs.
If the data source has streaming disabled and `traceQLStreaming` is set to `true`, no streaming will happen for that data source.
If the data source has streaming disabled and `traceQLStreaming` is set to `true`, streaming happens for that data source.
When streaming is active, it's shows as **Enabled** in **Explore**.
To check the status, select Explore in the menu, select your Tempo data source, and expand the **Options** section.
![The Explore screen shows the Tempo data source with streaming active](/media/docs/grafana/data-sources/tempo/tempo-query-stream-active.png)
## Trace to logs
The **Trace to logs** setting configures [trace to logs](ref:explore-trace-integration) that's available when you integrate Grafana with Tempo.
Trace to logs can also be used with other tracing data sources, such as Jaeger and Zipkin.
![Trace to logs settings](/media/docs/tempo/tempo-trace-to-logs-9-4.png)
![Trace to logs settings](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-logs.png)
There are two ways to configure the trace to logs feature:
@ -171,7 +213,9 @@ There are two ways to configure the trace to metrics feature:
- Use a basic configuration with a default query, or
- Configure one or more custom queries where you can use a [template language](ref:variable-syntax) to interpolate variables from the trace or span.
Refer to the Trace to metrics configuration options section to learn about the available options.
Refer to the [Trace to metrics configuration options](#trace-tometrics-configuration-options) section to learn about the available options.
![Trace to metrics settings in the Tempo data source](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-metrics.png)
### Set up a simple configuration
@ -233,7 +277,7 @@ To use custom queries with the configuration, follow these steps:
[//]: # 'Shared content for Trace to profiles in the Tempo data source'
{{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" leveloffset="+1" version="<GRAFANA_VERSION>" >}}
{{< docs/shared source="grafana" lookup="datasources/tempo-traces-to-profiles.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}
## Custom query variables
@ -252,27 +296,37 @@ For example, `${__span.name}`.
| **\_\_trace.duration** | The duration of the trace. |
| **\_\_trace.name** | The name of the trace. |
## Service Graph
## Additional settings
The **Service Graph** setting configures the [Service Graph](/docs/tempo/latest/metrics-generator/service_graphs/enable-service-graphs/) feature.
Use the down arrow to expand the **Additional settings** section to view these options.
### Advanced HTTP settings
The Grafana Proxy deletes forwarded cookies. Use the **Allowed cookies** field to specify cookies by name that should be forwarded to the data source.
The **Timeout** field sets the HTTP request timeout in seconds.
### Service graph
The **Service graph** setting configures the [Service Graph](/docs/tempo/latest/metrics-generator/service_graphs/enable-service-graphs/) data.
Configure the **Data source** setting to define in which Prometheus instance the Service Graph data is stored.
To use the Service Graph, refer to the [Service Graph documentation](#use-the-service-graph).
## Node Graph
### Node graph
The **Node Graph** setting enables the [node graph visualization](ref:node-graph), which is disabled by default.
The **Node graph** setting enables the [node graph visualization](ref:node-graph), which isn't activated by default.
Once enabled, Grafana displays the node graph above the trace view.
Once activated, Grafana displays the node graph above the trace view.
## Tempo search
### Tempo search
The **Search** setting configures [Tempo search](/docs/tempo/latest/configuration/#search).
You can configure the **Hide search** setting to hide the search query option in **Explore** if search is not configured in the Tempo instance.
## TraceID query
### TraceID query
The **TraceID query** setting modifies how TraceID queries are run. The time range can be used when there are performance issues or timeouts since it will narrow down the search to the defined range. This setting is disabled by default.
@ -284,7 +338,7 @@ You can configure this setting as follows:
| **Time shift start** | Time shift for start of search. Default: `30m`. |
| **Time shift end** | Time shift for end of search. Default: `30m`. |
## Span bar
### Span bar
The **Span bar** setting helps you display additional information in the span bar row.
@ -296,12 +350,30 @@ You can choose one of three options:
| **Duration** | _(Default)_ Displays the span duration on the span bar row. |
| **Tag** | Displays the span tag on the span bar row. You must also specify which tag key to use to get the tag value, such as `component`. |
### Private data source connect
[//]: # 'Shared content for authentication section procedure in data sources'
{{< docs/shared source="grafana" lookup="datasources/datasouce-private-ds-connect.md" leveloffset="+2" version="<GRAFANA_VERSION>" >}}
## Provision the data source
You can define and configure the Tempo data source in YAML files as part of Grafana's provisioning system.
You can define and configure the Tempo data source in YAML files as part of the Grafana provisioning system.
Provisioning is primarily used Grafana instances that don't use Grafana Cloud.
You can use version control, like git, to track and manage file changes.
Changes can be updated or rolled back as needed.
For more information about provisioning and available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources).
Example provision YAML file:
{{< admonition type="note" >}}
You can't modify a provisioned data source using the Tempo data source settings in Grafana.
Grafana displays a message for provisioned data sources.
{{< /admonition >}}
### Example file
This example provision YAML file sets up the equivalents of the options available in the Tempo data source user interface.
```yaml
apiVersion: 1

41
docs/sources/shared/datasources/datasouce-authentication.md Normal file
View File

@ -0,0 +1,41 @@
---
headless: true
labels:
products:
- enterprise
- oss
---
[//]: # 'This file documents the Authentication section for data sources.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/pyroscope/configure-pyroscope-data-source.md'
[//]: # '/grafana/docs/sources/datasources/tempo/configure-tempo-data-source.md'
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'
<!-- Authentication procedure from shared file -->
To set up authentication:
1. Select an authentication method from the drop-down list:
- **Basic authentication**: Authenticates your data source using a username and password
- **Forward OAuth identity**: Forwards the OAuth access token and the OIDC ID token, if available, of the user querying to the data source
- **No authentication**: No authentication is required to access the data source
1. For **Basic authentication** only: Enter the **User** and **Password**.
1. Optional: Complete the **TLS settings** for additional security methods.
**TLS Client Authentication**
: Toggle on to use client authentication. When enabled, it adds the **Server name**, **Client cert**, and **Client key** fields. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server. These details are encrypted and stored in the Grafana database.
**Add self-signed certificate**
: Activate this option to use a self-signed TLS certificate. You can add your own Certificate Authority (CA) certificate on top of one generated by the certificate authorities for additional security measure.
**Skip TLS certification validation**
: When activated, it bypasses TLS certificate verification. Not recommended, unless absolutely necessary for testing.
![Authentication section showing the TLS client certificate options](/media/docs/grafana/data-sources/tempo/tempo-data-source-authentication.png)
1. Optional: Add **HTTP Headers**. You can pass along additional context and metadata data about the request and response. Select **Add header** to add **Header** and **Value** fields.
1. Select **Save & test** to preserve your changes.

28
docs/sources/shared/datasources/datasouce-private-ds-connect.md Normal file
View File

@ -0,0 +1,28 @@
---
headless: true
labels:
products:
- enterprise
- oss
---
[//]: # 'This file documents the Private data source section for data sources.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/pyroscope/configure-pyroscope-data-source.md'
[//]: # '/grafana/docs/sources/datasources/tempo/configure-tempo-data-source.md'
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.'
<!-- Procedure for using private data source connect section in the data sources -->
{{< admonition type="note" >}}
This feature is only available in Grafana Cloud.
{{< /admonition >}}
Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud.
Refer to [Private data source connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) for more information on how PDC works and [Configure Grafana private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc) for steps on setting up a PDC connection.
Use the drop-down list to select a configured private data source. If you make changes, select **Test & save** to preserve your changes.
Use **Manage private data source connect** to configure and manage any private data sources you have configured.

61
docs/sources/shared/datasources/tempo-traces-to-profiles.md
View File

@ -19,7 +19,13 @@ labels:
Using Trace to profiles, you can use Grafana’s ability to correlate different signals by adding the functionality to link between traces and profiles.
**Trace to profiles** lets you link your Grafana Pyroscope data source to tracing data.
When configured, this connection lets you run queries from a trace span into the profile data.
When configured, this connection lets you run queries from a trace span into the profile data using **Explore**.
Each span links to your queries. Clicking a link runs the query in a split panel.
If tags are configured, Grafana dynamically inserts the span attribute values into the query.
The query runs over the time range of the (span start time - 60) to (span end time + 60 seconds).
Embedded flame graphs are also inserted into each span details section that has a linked profile.
This lets you see resource consumption in a flame graph visualization for each span without having to navigate away from the current view.
{{< youtube id="AG8VzfFMLxo" >}}
@ -28,37 +34,13 @@ There are two ways to configure the trace to profiles feature:
- Use a basic configuration with default query, or
- Configure a custom query where you can use a template language to interpolate variables from the trace or span.
{{< admonition type="note">}}
Traces to profile requires a Tempo data source with Traces to profiles configured and a Pyroscope data source.
![Traces to profiles section in the Tempo data source](/media/docs/grafana/data-sources/tempo/tempo-data-source-trace-to-profiles.png)
## Before you begin
Traces to profile requires a Tempo data source with Traces to profiles configured and a [Grafana Pyroscope data source](/docs/grafana/<GRAFANA_VERSION>/datasources/grafana-pyroscope/).
As with traces, your application needs to be instrumented to emit profiling data. For more information, refer to [Linking tracing and profiling with span profiles](/docs/pyroscope/<PYROSCOPE_VERSION>/configure-client/trace-span-profiles/).
{{< /admonition >}}
To use trace to profiles, navigate to **Explore** and query a trace.
Each span links to your queries. Clicking a link runs the query in a split panel.
If tags are configured, Grafana dynamically inserts the span attribute values into the query.
The query runs over the time range of the (span start time - 60) to (span end time + 60 seconds).
![Selecting a link in the span queries the profile data source](/media/docs/tempo/profiles/tempo-trace-to-profile.png)
To use trace to profiles, you must have a configured Grafana Pyroscope data source.
For more information, refer to the [Grafana Pyroscope data source](/docs/grafana/<GRAFANA_VERSION>/datasources/grafana-pyroscope/) documentation.
**Embedded flame graphs** are also inserted into each span details section that has a linked profile.
This lets you see resource consumption in a flame graph visualization for each span without having to navigate away from the current view.
Hover over a particular block in the flame graph to see more details about the consumed resources.
## Configuration options
The following table describes options for configuring your Trace to profiles settings:
| Setting name | Description |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data source** | Defines the target data source. You can select a Pyroscope \[profiling\] data source. |
| **Tags** | Defines the tags to use in the profile query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name for example to remove dots from the name if they're not allowed in the target data source. For example, map `http.status` to `http_status`. |
| **Profile type** | Defines the profile type that used in the query. |
| **Use custom query** | Toggles use of custom query with interpolation. |
| **Query** | Input to write custom query. Use variable interpolation to customize it with variables from span. |
## Use a basic configuration
@ -77,7 +59,6 @@ To use a basic configuration, follow these steps:
1. Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.
The profile type or app must be selected for the query to be valid. Grafana doesn't show any data if the profile type or app isn’t selected when a query runs.
![Traces to profile configuration options in the Tempo data source](/media/docs/tempo/profiles/Tempo-data-source-profiles-Settings.png)
1. Select **Save and Test**.
@ -94,9 +75,23 @@ To use a custom query with the configuration, follow these steps:
1. Select a Pyroscope data source in the **Data source** drop-down.
1. Optional: Choose any tags to use in the query. If left blank, the default values of `service.name` and `service.namespace` are used.
These tags can be used in the custom query with `${__tags}` variable. This variable interpolates the mapped tags as list in an appropriate syntax for the data source. Only the tags that were present in the span are included; tags that aren't present are omitted. You can also configure a new name for the tag. This is useful in cases where the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. If you don’t map any tags here, you can still use any tag in the query, for example: `method="${__span.tags.method}"`. You can learn more about custom query variables [here](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#custom-query-variables).
These tags can be used in the custom query with `${__tags}` variable. This variable interpolates the mapped tags as list in an appropriate syntax for the data source. Only tags present in the span are included. Tags that aren't present are omitted.
You can also configure a name for the tag. Tag names are useful where the tag has dots in the name and the target data source doesn't allow using dots in labels. For example, you can remap `service.name` to `service_name`. If you don’t map any tags here, you can still use any tag in the query, for example: `method="${__span.tags.method}"`. Learn more about [custom query variables](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#custom-query-variables).
1. Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.
1. Switch on **Use custom query** to enter a custom query.
1. Specify a custom query to be used to query profile data. You can use various variables to make that query relevant for current span. The link is shown only if all the variables are interpolated with non-empty values to prevent creating an invalid query. You can interpolate the configured tags using the `$__tags` keyword.
1. Specify a custom query to be used to query profile data. You can use various variables to make that query relevant for current span. The link shows only if all the variables are interpolated with non-empty values to prevent creating an invalid query. You can interpolate the configured tags using the `$__tags` keyword.
1. Select **Save and Test**.
## Configuration options
The following table describes options for configuring your **Trace to profiles** settings:
| Setting name | Description |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Data source | Defines the target data source. You can select a Pyroscope \[profiling\] data source. |
| Tags | Defines the tags to use in the profile query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name for example to remove dots from the name if they're not allowed in the target data source. For example, map `http.status` to `http_status`. |
| Profile type | Defines the profile type that used in the query. |
| Use custom query | Toggles use of custom query with interpolation. |
| Query | Input to write custom query. Use variable interpolation to customize it with variables from span. |