mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
[DOC] Update TraceQL query editor doc (#91910)
Co-authored-by: Isabel Matwawana <76437239+imatwawana@users.noreply.github.com>
This commit is contained in:
Kim Nylander
GitHub
parent
24afc7a5b3
commit
f063088188
@ -26,86 +26,194 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
service-graph:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/service-graph/
|
||||
recorded-queries:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
query-history-management:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/query-management/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/query-management/
|
||||
query-inspector:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/explore-inspector/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/explore-inspector/
|
||||
---
|
||||
|
||||
# Query tracing data
|
||||
|
||||
The Tempo data source's query editor helps you query and display traces from Tempo in [Explore](ref:explore).
|
||||
The queries use [TraceQL](/docs/tempo/latest/traceql), the query language designed specifically for tracing.
|
||||
|
||||
This topic explains configuration and queries specific to the Tempo data source.
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data](ref:query-transform-data).
|
||||
For general documentation on querying data sources in Grafana, refer to [Query and transform data](ref:query-transform-data).
|
||||
|
||||
## Before you begin
|
||||
|
||||
You can compose TraceQL queries in Grafana and Grafana Cloud using **Explore** and a Tempo data source.
|
||||
|
||||
### TraceQL knowledge helpful, but not required
|
||||
|
||||
You don't have to know TraceQL to create a query.
|
||||
You can use the **Search** query builder's user interface to select options to search your data.
|
||||
These selections generate a TraceQL query.
|
||||
Any query generated using **Search** query builder can be transferred to the **TraceQL** query editor, where you can edit the query directly.
|
||||
|
||||
To learn more about how to query by TraceQL, refer to the [TraceQL documentation](/docs/tempo/latest/traceql).
|
||||
|
||||
## Choose a query editing mode
|
||||
|
||||
The query editor has three **Query types** that you can use to explore your tracing data.
|
||||
You can use these modes by themselves or in combination to create building blocks to generate custom queries.
|
||||
Adding another query adds a new query block.
|
||||
Refer to [Use query types together](#use-query-types-together) for more information.
|
||||
|
||||
|
||||
|
||||
The three query types are:
|
||||
|
||||
- **Search** query builder - Provides a user interface for building a TraceQL query.
|
||||
- **TraceQL** query editor - Lets you write your own TraceQL query with assistance from autocomplete.
|
||||
- **Service Graph** view - Displays a visual relationship between services. Refer to the [Service graph](ref:service-graph) documentation for more information.
|
||||
|
||||
### Search query builder
|
||||
|
||||
The **Search** query builder provides drop-down lists and text fields to help you write a query.
|
||||
The query builder is ideal for people who aren't familiar with or want to learn TraceQL.
|
||||
|
||||
Refer to the [Search using the TraceQL query builder documentation]({{< relref "./traceql-search" >}}) to learn more about creating queries using convenient drop-down menus.
|
||||
|
||||
|
||||
|
||||
### TraceQL query editor
|
||||
|
||||
The **TraceQL** query editor lets you search by trace ID and write TraceQL queries using autocomplete.
|
||||
|
||||
Refer to the [TraceQL query editor documentation]({{< relref "./traceql-editor" >}}) to learn more about constructing queries using a code-editor-like experience.
|
||||
|
||||
|
||||
|
||||
You can also search for a trace ID by entering it into the query field.
|
||||
|
||||
### Service graph view
|
||||
|
||||
Grafana’s **Service Graph** view uses metrics to display span request rates, error rates, and durations, as well as service graphs.
|
||||
Once the requirements are set up, this preconfigured view is immediately available.
|
||||
|
||||
Using the service graph view, you can:
|
||||
|
||||
- Discover spans which are consistently erroring and the rates at which they occur.
|
||||
- Get an overview of the overall rate of span calls throughout your services.
|
||||
- Determine how long the slowest queries in your service take to complete.
|
||||
- Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals).
|
||||
|
||||
For more information about the service graph, refer to [Service graph](../service-graph/).
|
||||
|
||||
|
||||
|
||||
## Use TraceQL panels in dashboards
|
||||
|
||||
To add TraceQL panels to your dashboard, refer to the [Traces panel documentation](/docs/grafana/latest/panels-visualizations/visualizations/traces/).
|
||||
|
||||
To learn more about Grafana dashboards, refer to the [Use dashboards documentation](/docs/grafana/latest/dashboards/use-dashboards/).
|
||||
|
||||
## Write TraceQL queries in Grafana
|
||||
## Set options for query builder and editor
|
||||
|
||||
You can compose TraceQL queries in Grafana and Grafana Cloud using **Explore** and a Tempo data source. You can use either the **Query type** > **Search** (the TraceQL query builder) or the **TraceQL** tab (the TraceQL query editor).
|
||||
Both of these methods let you build queries and drill-down into result sets.
|
||||
The following options are available for the **Search** and **TraceQL** query types.
|
||||
You can modify these settings in the **Options** section.
|
||||
|
||||
To learn more about how to query by TraceQL, refer to the [TraceQL documentation](/docs/tempo/latest/traceql).
|
||||
|
||||
|
||||
### TraceQL query builder
|
||||
After changing any option, re-run the query to apply the updates.
|
||||
|
||||
The TraceQL query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-downs and text fields to help you write a query.
|
||||
Limit
|
||||
: Determines the maximum number of traces to return. Default value is `20`.
|
||||
|
||||
Refer to the [Search using the TraceQL query builder documentation]({{< relref "./traceql-search" >}}) to learn more about creating queries using convenient drop-down menus.
|
||||
Span Limit
|
||||
: Sets the maximum number of spans to return for each spanset. Default value is `3`.
|
||||
|
||||
|
||||
Table Format
|
||||
: Determines whether the query results table is displayed focused on **Traces** or **Spans**. **Traces** is the default selection. When **Traces** is selected, the results table starts with the trace ID. When **Spans** is selected, the table starts with the trace service.
|
||||
|
||||
### TraceQL query editor
|
||||
Step
|
||||
: Defines the step for metrics queries. Use duration notation, for example, `30ms` or `1m`.
|
||||
|
||||
The TraceQL query editor, located on the **Explore** > **TraceQL** tab in Grafana, lets you search by trace ID and write TraceQL queries using autocomplete.
|
||||
Streaming
|
||||
: Indicates if streaming is active. Streaming lets you view partial query results before the entire query completes. Activating streaming adds the **Table - Streaming Progress** section to the query results.
|
||||
|
||||
Refer to the [TraceQL query editor documentation]({{< relref "./traceql-editor" >}}) to learn more about constructing queries using a code-editor-like experience.
|
||||
## Use query types together
|
||||
|
||||
|
||||
You can use **+ Add query** to create customized queries that use one or more of the query types together.
|
||||
Each time you add a new query, it adds a new section, or query block, that contains **Search**, **TraceQL**, or **Service Graph** user interface.
|
||||
|
||||
## Query by search (deprecated)
|
||||
The added query and results table appear in the navigation under **Queries** and **Tables** respectively.
|
||||
You can use the navigation to view query, results table, and service graph blocks.
|
||||
|
||||
{{% admonition type="caution" %}}
|
||||
Starting with Grafana v10.2, this query type has been deprecated. It will be removed in Grafana v10.3.
|
||||
{{% /admonition %}}
|
||||
{{< video-embed src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-editor.mp4" max-width="800px" class="my-cool-video" caption="Navigating through the query blocks" align="center" >}}
|
||||
|
||||
Use this to search for traces by service name, span name, duration range, or process-level attributes that are included in your application's instrumentation, such as HTTP status code and customer ID.
|
||||
To add a query block:
|
||||
|
||||
To configure Tempo and the Tempo data source for search, refer to [Configure the data source]({{< relref "../#configure-the-data-source" >}}).
|
||||
1. Select **+ Add query**.
|
||||
1. Choose a query type: **Search**, **TraceQL**, or **Service Graph**.
|
||||
|
||||
To search for traces:
|
||||
To remove a query block, select the **Remove query** trash can icon.
|
||||
|
||||
1. Select **Search** from the **Query** type selector.
|
||||
1. Fill out the search form:
|
||||
To rename a block, select the **Rename** edit icon next to the query block name.
|
||||
The name changes in the queries and table list.
|
||||
|
||||
| Name | Description |
|
||||
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Service Name** | Returns a list of services. |
|
||||
| **Span Name** | Returns a list of span names. |
|
||||
| **Tags** | Sets tags with values in the [logfmt](https://brandur.org/logfmt) format, such as `error=true db.statement="select * from User"`. |
|
||||
| **Min Duration** | Filters all traces with a duration higher than the set value. Possible values are `1.2s`, `100ms`, `500us`. |
|
||||
| **Max Duration** | Filters all traces with a duration lower than the set value. Possible values are `1.2s`, `100ms`, `500us`. |
|
||||
| **Limit** | Limits the number of traces returned. |
|
||||
### Additional query block options
|
||||
|
||||
{{< figure src="/static/img/docs/explore/tempo-search.png" class="docs-image--no-shadow" max-width="750px" caption="Screenshot of the Tempo search feature with a trace rendered in the right panel" >}}
|
||||
Each query block has a set of icons in the right top corner.
|
||||
|
||||
### Search recent traces
|
||||
|
||||
|
||||
You can search recent traces held in Tempo's ingesters.
|
||||
By default, ingesters store the last 15 minutes of tracing data.
|
||||
These icons include:
|
||||
|
||||
To configure your Tempo data source to use this feature, refer to the [Tempo documentation](/docs/tempo/latest/getting-started/tempo-in-grafana/#search-of-recent-traces).
|
||||
Show data source help
|
||||
: Displays the **Tempo Cheat Sheet** with links to documentation.
|
||||
|
||||
### Search the backend datastore
|
||||
Create recorded query
|
||||
: Lets you save the current query block as a recorded query. This option is available in Grafana Cloud and Grafana Enterprise. For more information, refer to [Recorded queries](ref:recorded-queries).
|
||||
|
||||
Tempo includes the ability to search the entire backend datastore.
|
||||
Duplicate query
|
||||
: Copies the current block and adds a new identical block.
|
||||
|
||||
To configure your Tempo data source to use this feature, refer to the [Tempo documentation](/docs/tempo/latest/getting-started/tempo-in-grafana/#search-of-the-backend-datastore).
|
||||
Remove query
|
||||
: Deletes the query block.
|
||||
|
||||
## Query by TraceID
|
||||
### Use query history and query inspector
|
||||
|
||||
To query a particular trace:
|
||||
**Explore** provides a history of all queries you've used within a data source and an inspector that lets you view stats, inspect queries, view JSON, and general information for your data source queries.
|
||||
|
||||
1. Select the **TraceQL** query type.
|
||||
1. Enter the trace's ID into the query field.
|
||||
For more information, refer to the [Query inspector in Explore](ref:query-inspector) and [Query management in Explore](ref:query-history-management) documentation.
|
||||
|
||||
{{< figure src="/static/img/docs/tempo/query-editor-traceid.png" class="docs-image--no-shadow" max-width="750px" caption="Screenshot of the Tempo TraceID query type" >}}
|
||||
## Cross-tenant TraceQL queries
|
||||
|
||||
If you've configured a multi-stack Tempo data source, you can perform TraceQL queries across those stacks and tenants.
|
||||
|
||||
Queries performed using the cross-tenant configured data source, in either **Explore** or inside of dashboards,
|
||||
are performed across all the tenants that you specified in the **X-Scope-OrgID** header.
|
||||
|
||||
<!-- vale Grafana.Spelling = NO -->
|
||||
|
||||
TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,
|
||||
|
||||
<!-- vale Grafana.Quotes = YES -->
|
||||
|
||||
```
|
||||
{ span.attr1 = "bar" } && { span.attr2 = "foo" }
|
||||
```
|
||||
|
||||
TraceQL evaluates a contiguously stored trace.
|
||||
If these two conditions are satisfied in separate tenants, then Tempo doesn't return the trace.
|
||||
|
||||
Refer to [Set up a multi-stack Tempo data source in Grafana](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/multi-stack-data-sources/#set-up-a-multi-stack-tempo-data-source-in-grafana) for information about configuring the Tempo data source.
|
||||
|
||||
For information about Tempo configuration requirements, refer to the [Cross-tenant query](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/cross_tenant_query/) and [Enable multitenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/multitenancy/) documentation.
|
||||
|
||||
@ -13,6 +13,27 @@ labels:
|
||||
menuTitle: Write TraceQL queries
|
||||
title: Write TraceQL queries with the editor
|
||||
weight: 300
|
||||
refs:
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
service-graph:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/service-graph/
|
||||
recorded-queries:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
tempo-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/query-editor/
|
||||
---
|
||||
|
||||
# Write TraceQL queries with the editor
|
||||
|
||||
@ -11,11 +11,32 @@ labels:
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Search traces
|
||||
title: Search traces using TraceQL query builder
|
||||
title: Investigate traces using Search query builder
|
||||
weight: 300
|
||||
refs:
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
service-graph:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/service-graph/
|
||||
recorded-queries:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/recorded-queries/
|
||||
tempo-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/query-editor/
|
||||
---
|
||||
|
||||
# Search traces using TraceQL query builder
|
||||
# Investigate traces using Search query builder
|
||||
|
||||
Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces.
|
||||
TraceQL provides a method for formulating precise queries so you can zoom in to the data you need.
|
||||
@ -23,9 +44,9 @@ Query results are returned faster because the queries limit what is searched.
|
||||
|
||||
To learn more about how to query by TraceQL, refer to the [TraceQL documentation](/docs/tempo/latest/traceql).
|
||||
|
||||
The TraceQL query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-downs and text fields to help you write a query.
|
||||
The **Search** query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-down lists and text fields to help you write a query.
|
||||
|
||||
|
||||
|
||||
|
||||
## Enable Search with the query builder
|
||||
|
||||
|
||||
@ -68,7 +68,7 @@ Each node on the graph represents a service such as an API or database.
|
||||
|
||||
You use the Service Graph to detect performance issues; track increases in error, fault, or throttle rates in services; and investigate root causes by viewing corresponding traces.
|
||||
|
||||
{{< figure src="/static/img/docs/node-graph/node-graph-8-0.png" class="docs-image--no-shadow" max-width="500px" caption="Screenshot of a Node Graph" >}}
|
||||
{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-node-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Screenshot of a Node Graph" >}}
|
||||
|
||||
## Display the Service Graph
|
||||
|
||||
@ -100,9 +100,9 @@ Each circle's color represents the percentage of requests in each state:
|
||||
|
||||
Service graph view displays a table of request rate, error rate, and duration metrics (RED) calculated from your incoming spans. It also includes a node graph view built from your spans.
|
||||
|
||||
{{< figure src="/static/img/docs/tempo/apm-table.png" class="docs-image--no-shadow" max-width="500px" caption="Screenshot of the Service Graph view" >}}
|
||||
{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png" class="docs-image--no-shadow" max-width="500px" alt="Screenshot of the Service Graph view" >}}
|
||||
|
||||
For details, refer to the [Service Graph view documentation](/docs/tempo/latest/metrics-generator/service-graph-view/).
|
||||
For details, refer to the [Service Graph view documentation](/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/).
|
||||
|
||||
To open the Service Graph view:
|
||||
|
||||
@ -120,4 +120,6 @@ These metrics must exist in your Prometheus data source.
|
||||
|
||||
To open a query in Prometheus with the span name of that row automatically set in the query, click a row in the **rate**, **error rate**, or **duration** columns.
|
||||
|
||||
|
||||
|
||||
To open a query in Tempo with the span name of that row automatically set in the query, click a row in the **links** column.
|
||||
|
||||
@ -23,7 +23,7 @@ weight: 100
|
||||
|
||||
Node graphs can visualize directed graphs or networks. They use a directed force layout to effectively position the nodes, so they can display complex infrastructure maps, hierarchies, or execution diagrams.
|
||||
|
||||
|
||||
|
||||
|
||||
## Configure a node graph visualization
|
||||
|
||||
@ -75,9 +75,11 @@ Node graphs can show only 1,500 nodes. If this limit is crossed a warning will b
|
||||
|
||||
Usually, nodes show two statistical values inside the node and two identifiers just below the node, usually name and type. Nodes can also show another set of values as a color circle around the node, with sections of different color represents different values that should add up to 1.
|
||||
|
||||
For example, you can have the percentage of errors represented by a red portion of the circle. Additional details can be displayed in a context menu which is displayed when you click on the node. There also can be additional links in the context menu that can target either other parts of Grafana or any external link.
|
||||
For example, you can have the percentage of errors represented by a red portion of the circle.
|
||||
Additional details can be displayed in a context menu which is displayed when you click on the node.
|
||||
There also can be additional links in the context menu that can target either other parts of Grafana or any external link.
|
||||
|
||||
|
||||
|
||||
|
||||
### Edges
|
||||
|
||||
@ -107,7 +109,9 @@ The number of nodes shown at a given time is limited to maintain a reasonable vi
|
||||
|
||||
You can switch to the grid view to have a better overview of the most interesting nodes in the graph. Grid view shows nodes in a grid without edges and can be sorted by stats shown inside the node or by stats represented by the a colored border of the nodes.
|
||||
|
||||
|
||||
<!-- Screenshot from v11.2 -->
|
||||
|
||||
|
||||
|
||||
To sort the nodes, click on the stats inside the legend. The marker next to the stat name shows which stat is currently used for sorting and sorting direction.
|
||||
|
||||
|
||||
@ -23,19 +23,24 @@ Query results are returned faster because the queries limit what is searched.
|
||||
|
||||
To learn more about how to query by TraceQL, refer to the [TraceQL documentation](https://grafana.com/docs/tempo/latest/traceql/).
|
||||
|
||||
The TraceQL query editor, located on the **Explore** > **TraceQL** tab in Grafana, lets you search by trace ID and write TraceQL queries using autocomplete.
|
||||
The TraceQL query editor in Grafana **Explore** lets you search by trace ID and write TraceQL queries using autocomplete.
|
||||
|
||||
|
||||
|
||||
|
||||
## Enable the query editor
|
||||
## Before you begin
|
||||
|
||||
This feature is automatically available in Grafana 10 (and newer) and Grafana Cloud.
|
||||
|
||||
To use the TraceQL query editor in self-hosted Grafana 9.3.2 and older, you need to [enable the `traceqlEditor` feature toggle](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/).
|
||||
|
||||
If trying to query a self-managed Grafana Tempo or Grafana Enterprise Traces database with a gateway (e.g., nginx) in front of it from your hosted Grafana, that gateway (e.g., nginx) must allow gRPC connections. If it does not, streaming will not work and queries will fail to return results.
|
||||
### Streaming and gRPC
|
||||
|
||||
If you cannot configure your gateway to allow gRPC, open a support escalation to request streaming query results be disabled in your hosted Grafana.
|
||||
If you're trying to query a self-managed Grafana Tempo or Grafana Enterprise Traces database with a gateway, such as nginx, in front of it from your hosted Grafana, that gateway (for example, nginx) must allow gRPC connections.
|
||||
If it doesn't, streaming won't work and queries will fail to return results.
|
||||
|
||||
If you can't configure your gateway to allow gRPC, deactivate streaming in your hosted Grafana.
|
||||
In Grafana 11.2 and newer, you can deactivate the **Streaming** option in your Tempo data source settings from **Connections** > **Data sources** in the Grafana main menu.
|
||||
You can also open a support escalation to request streaming query results be disabled in your hosted Grafana.
|
||||
|
||||
## Write TraceQL queries using the query editor
|
||||
|
||||
@ -44,13 +49,19 @@ The Tempo data source’s TraceQL query editor helps you query and display trace
|
||||
To access the query editor, follow these steps:
|
||||
|
||||
1. Sign into Grafana or Grafana Cloud.
|
||||
1. Select your Tempo data source.
|
||||
1. From the menu, choose **Explore** and select the **TraceQL** tab.
|
||||
1. Select **Explore** from the main menu.
|
||||
1. Select a Tempo data source.
|
||||
1. Select the **TraceQL** tab.
|
||||
1. Start your query on the text line by entering `{`. For help with TraceQL syntax, refer to the [Construct a TraceQL query documentation](https://grafana.com/docs/tempo/latest/traceql/#construct-a-traceql-query).
|
||||
1. Optional: Use the Time picker drop-down to change the time and range for the query (refer to the [documentation for instructions](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range)).
|
||||
1. Once you have finished your query, select **Run query**.
|
||||
|
||||
This video provides and example of creating a TraceQL query using the custom tag grouping.
|
||||
Optional: Select **Copy query from Search** to transfer a builder query to the editor.
|
||||
|
||||
1. Optional: Use the **Time picker** drop-down list to change the time and range for the query (refer to the [documentation for instructions](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range)).
|
||||
1. Once you've finished your query, select **Run query**.
|
||||
|
||||
|
||||
|
||||
This video provides an example of creating a TraceQL query using the custom tag grouping.
|
||||
|
||||
{{< youtube id="fraepWra00Y" >}}
|
||||
|
||||
@ -62,15 +73,15 @@ To query a particular trace by its trace ID:
|
||||
1. Enter the trace ID into the query field. For example: `41928b92edf1cdbe0ba6594baee5ae9`
|
||||
1. Click **Run query** or use the keyboard shortcut Shift + Enter.
|
||||
|
||||
|
||||
|
||||
|
||||
## Use autocomplete to write queries
|
||||
|
||||
You can use the query editor’s autocomplete suggestions to write queries.
|
||||
The editor detects span sets to provide relevant autocomplete options.
|
||||
It uses regular expressions (regex) to detect where it's inside a spanset and provide attribute names, scopes, intrinsic names, logic operators, or attribute values from Tempo's API, depending on what's expected for the current situation.
|
||||
The editor detects spansets to provide relevant autocomplete options.
|
||||
It uses regular expressions (regex) to detect where it is inside a spanset and provide attribute names, scopes, intrinsic names, logic operators, or attribute values from the Tempo API, depending on what's expected for the current situation.
|
||||
|
||||
|
||||
|
||||
|
||||
To create a query using autocomplete, follow these steps:
|
||||
|
||||
@ -84,17 +95,37 @@ To create a query using autocomplete, follow these steps:
|
||||
|
||||
## View query results
|
||||
|
||||
Query results for both the editor and the builder are returned in a table. Selecting the Trace ID or Span ID provides more detailed information.
|
||||
Query results appear in a table, such as **Table - Traces**, under the query editor.
|
||||
Each span (and the trace it belongs to) matching the query conditions is returned by the query.
|
||||
If there are no filter conditions, all spans are matching and thus returned with their associated traces.
|
||||
|
||||
Selecting the trace ID from the returned results opens a trace diagram. Selecting a span from the returned results opens a trace diagram and reveals the relevant span in the trace diagram (the highlighted blue line).
|
||||
A query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time).
|
||||
The query response is also limited by the number of traces (**Limit**) and spans per spanset (**Span Limit**).
|
||||
|
||||
In the trace diagram, the bold text on the left side of each span indicates the service name, for example `mythical-requester: requester`, and it is hidden when subsequent spans have the same service name (nested spans).
|
||||
Each service has a color assigned to it, which is visible to the left of the name and timeline in the graph.
|
||||
Spans with the same color belong to the same service. The grey text to the right of the service name indicates the span name.
|
||||
|
||||
|
||||
|
||||
1. TraceQL query editor
|
||||
1. Query options: **Limit**, **Span Limit** and **Table Format** (Traces or Spans).
|
||||
1. Trace (by Trace ID). The **Name** and **Service** columns are displaying the trace root span name and associated service.
|
||||
1. Spans associated with the Trace.
|
||||
|
||||
### Streaming results
|
||||
Selecting the trace ID from the returned results opens a trace diagram.
|
||||
Selecting a span from the returned results opens a trace diagram and reveals the relevant span in the trace diagram.
|
||||
|
||||
For more information on span details, refer to [Traces in Explore](https://grafana.com/docs/grafana/latest/explore/trace-integration/#span-details).
|
||||
|
||||
|
||||
|
||||
### Focus on traces or spans
|
||||
|
||||
Under **Options**, you can choose to display the table as **Traces** or **Spans** focused.
|
||||
|
||||
When the **Table Type** option is set to **Spans**, the traces and spansets are flattened into a list of spans.
|
||||
The trace service and trace name are added to the row of each span to add context.
|
||||
|
||||
Using the **Spans** option makes it easier access the spans to apply transformations and plot them in dashboards.
|
||||
|
||||
### Stream results
|
||||
|
||||
The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.
|
||||
|
||||
|
||||
@ -17,45 +17,44 @@ labels:
|
||||
|
||||
# Write TraceQL queries using Search
|
||||
|
||||
The TraceQL query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-downs and text fields to help you write a query.
|
||||
The **Search** query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-down lists and text fields to help you write a query.
|
||||
The selections you make automatically generate a [TraceQL query](/docs/tempo/latest/traceql).
|
||||
|
||||
|
||||
|
||||
|
||||
The builder lets you run the most common queries in as few clicks as possible. You don't need to know the underlying query language or database architecture to use it.
|
||||
|
||||
The builder supports a subset of TraceQL capabilities. For example, if you wish to use structural operators (`>>`, `>`, `~`), you need to use the query editor on the **TraceQL** tab.
|
||||
|
||||
You can use the query builder’s drop-downs to compose TraceQL queries. The selections you make automatically generate a [TraceQL query](/docs/tempo/latest/traceql).
|
||||
The builder supports a subset of TraceQL capabilities, including some structural operators (`>>`, `>`, `=~`, `!=`).
|
||||
|
||||
To access **Search**, select your Tempo data source, and then choose **Explore** and select **Query type** > **Search**.
|
||||
You can use the query builder to search trace data by resource service name, span name, duration, and one or more tags. The examples on this page use the default filters.
|
||||
You can use the query builder to search trace data by resource service name, span name, duration, one or more tags. The examples on this page use the default filters.
|
||||
|
||||
In addition, you can add query builder blocks, view the query history, and use the **Inspector** to see details.
|
||||
|
||||
{{< figure src="/media/docs/tempo/traceql/screenshot-tempods-query-search.png" class="docs-image--no-shadow" max-width="750px" caption="Screenshot of the Tempo Search query type" >}}
|
||||
{{< figure src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-builder-v11.png" class="docs-image--no-shadow" max-width="750px" caption="Screenshot of the Tempo Search query type" >}}
|
||||
|
||||
## Perform a search
|
||||
|
||||
To perform a search, you need to select filters and/or tags and then run the query. The results appear underneath the query builder.
|
||||
To perform a search, you need to select filters and then run the query. The results appear underneath the query builder.
|
||||
The screenshot identifies the areas used to perform a search.
|
||||
|
||||
{{< figure src="/media/docs/tempo/traceql/screenshot-tempods-query-search-parts.png" class="docs-image--no-shadow" max-width="750px" caption="Parts of Tempo Search query type" >}}
|
||||
|
||||
|
||||
| Number | Name | Action | Comment |
|
||||
| :----- | :----------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | Data source | Use the data source drop-down to select a Tempo data source. | Each data source has its own version of search. This Search is specific to the Tempo data source. |
|
||||
| 2 | Query type tab | Select Search. | |
|
||||
| 3 | Choose filter | Choose whether to filter using **Resource Service Name**, **Span Name**, and/or **Duration**. | Optional. You can execute an empty query in the Search tab. In TraceQL, `{}` is a valid query and is the default query until you choose options. |
|
||||
| 4 | Filters conditions | Select options for one or more filters. For example, you can define a filter where **Resource Service Name** (`resource.service.name`) equals (`=`) `cloud-logs-archiver`. | Optional. At least one tag or filter must be defined. |
|
||||
| 5 | Tags | Add tags for span, resource, or unscoped and define their conditions. | Optional. |
|
||||
| 6 | TraceQL query | Displays the TraceQL query constructed by your selections. | This TraceQL query is executed when you select **Run query**. |
|
||||
| Number | Name | Action | Comment |
|
||||
| :----- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | Data source | Use the data source drop-down list to select a Tempo data source. | Each data source has its own version of search. This **Search** is specific to the Tempo data source. |
|
||||
| 2 | Query type | Select **Search**. | |
|
||||
| 3 | Choose filter | Choose one or more of the filters. | Optional. You can execute an empty query in the Search tab. In TraceQL, `{}` is a valid query and is the default query to provide a list of all traces or spans. |
|
||||
| 4 | Filters conditions | Select options for one or more filters. For example, you can define a filter where **Service Name** (`resource.service.name`) equals (`=`) `user`. | Optional. At least one tag or filter must be defined. |
|
||||
| 5 | Tags and Aggregate by | Add tags for span, resource, or unscoped and define their conditions. Use **Aggregate by** to group results. | Optional. |
|
||||
| 6 | TraceQL query | Displays the TraceQL query constructed by your selections. | This TraceQL query is executed when you select **Run query**. |
|
||||
|
||||
Every query searches the data for the selected time frame.
|
||||
By default, queries run against data from the last hour.
|
||||
Select Time range to the left of Run query to choose the time range for the data your query runs against.
|
||||
Select **Time range** to the left of **Run query** to choose the time range for the data your query runs against.
|
||||
Read the [dashboard time range](/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range) documentation to learn more.
|
||||
|
||||
To access Search, use the following steps:
|
||||
To access the **Search** query builder, use the following steps:
|
||||
|
||||
1. Sign into Grafana.
|
||||
1. Select your Tempo data source.
|
||||
@ -63,16 +62,20 @@ To access Search, use the following steps:
|
||||
|
||||
## Define filters
|
||||
|
||||
Using filters, you refine the data returned from the query by selecting **Resource Service Name**, **Span Name**, or **Duration**. The **Duration** represents span time, calculated by subtracting the end time from the start time of the span.
|
||||
Using filters, you refine the data returned from the query by selecting **Service Name**, **Span Name**, **Status**, or **Duration**.
|
||||
**Duration** represents span time, calculated by subtracting the end time from the start time of the span.
|
||||
|
||||
Grafana administrators can change the default filters using the Tempo data source configuration.
|
||||
Filters can be limited by the operators. The available operators are determined by the field type.
|
||||
For example, **Span Name** and **Resource Service Name** are string fields so the comparison operators are equals (`=`), not equal (`!=`), or regular expressions (`=~`).
|
||||
Filters can be limited by the operators.
|
||||
The field type determines the available operators.
|
||||
For example, **Span Name** and **Service Name** are string fields so the comparison operators are equals (`=`), not equal (`!=`), matches regular expressions (`=~`), or doesn't match regular expression (`!~`).
|
||||
**Duration** is a duration field type and uses range selections (`>`, `>=`, `<`, `<=`).
|
||||
|
||||
When you select multiple values for the same filter, Grafana automatically changes the operator to the regex operator `=~` and concatenates the values with a `|`. This capability only applies to fields with drop-down value selection.
|
||||
When you select multiple values for the same filter, Grafana automatically changes the operator to the regular expression (regex) operator `=~` and concatenates the values with a `|`.
|
||||
This capability only applies to fields with drop-down value selection.
|
||||
|
||||
For example, if you choose **Span Name** `= get` and then **Span Name** `= log_results_cache,` operator drop-down changes from `=` to `=~` and both `get` and `log_results_cache` are listed in the **Span Name** field. The resulting query is updated with this:
|
||||
For example, if you choose **Span Name** `= get` and then **Span Name** `= log_results_cache,` operator drop-down list changes from `=` to `=~` and both `get` and `log_results_cache` are listed in the **Span Name** field.
|
||||
The resulting query is updated with this:
|
||||
|
||||
`{duration>5ms && duration<10ms && name=~"get|log_results_cache"}`
|
||||
|
||||
@ -80,7 +83,7 @@ To define filters, follow these steps:
|
||||
|
||||
1. Choose one of the filters.
|
||||
1. Select a comparison operator from the drop-down.
|
||||
1. **Resource Service Name** and **Span Name** only: Select one or more values from the drop-down.
|
||||
1. **Service Name**, **Span Name**, and **Status** only: Select one or more values from the drop-down.
|
||||
1. **Duration** only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (`ns`), milliseconds (`ms`), seconds (`s`), minutes (`m`), and hours (`h`).
|
||||
|
||||
You can either select **Run query** to execute the query or define tags and then run the query.
|
||||
@ -97,9 +100,9 @@ To add a tag, follow these steps:
|
||||
1. Select a tag from the **Select tag** drop-down.
|
||||
1. Select a comparison operator.
|
||||
1. Select a value from the **Select value** drop-down. This field is populated based upon the tag.
|
||||
1. Optional: Select **+** to add an additional tag.
|
||||
1. Optional: Select **+** to add another tag.
|
||||
|
||||
## Optional: Use Aggregate by
|
||||
### Optional: Use Aggregate by
|
||||
|
||||
{{% admonition type="warning" %}}
|
||||
**Aggregate by** is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided.
|
||||
@ -127,7 +130,7 @@ The results are shown in the same order used in **Aggregate by**.
|
||||
For example, **Aggregate by** lists `intrinsic.name` followed by `span.http.user_agent`.
|
||||
The first column in the results Table shows **name** and then **span.http.user_agent**.
|
||||
|
||||
|
||||
|
||||
|
||||
To use this capability:
|
||||
|
||||
@ -141,22 +144,26 @@ To use this capability:
|
||||
|
||||
{{< youtube id="g97CjKOZqT4" >}}
|
||||
|
||||
### Optional: Add queries
|
||||
### Optional: Add query and service graph blocks
|
||||
|
||||
Using **Add query**, you can have successive queries that run in sequential order.
|
||||
Using **Add query**, you can have successive query or service node blocks that run in sequential order.
|
||||
For example, query A runs and then query B.
|
||||
You can reorder the queries by dragging and dropping them above or below other queries.
|
||||
Select **+ Add query** to add another query block.
|
||||
|
||||
### Run queries and view results
|
||||
For more information, refer to [Use query types together](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/).
|
||||
|
||||
## Run queries and view results
|
||||
|
||||
Select **Run query** to run the TraceQL query (1 in the screenshot).
|
||||
|
||||
Queries can take a little while to return results. The results appear in a table underneath the query builder. Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).
|
||||
Queries can take a little while to return results. The results appear in a table underneath the query builder.
|
||||
Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).
|
||||
**Span Filters** (4 in the screenshot) provide an additional to refine the query results.
|
||||
|
||||
{{< figure src="/media/docs/tempo/traceql/screenshot-tempods-query-results.png" class="docs-image--no-shadow" max-width="750px" caption="Tempo Search query type results" >}}
|
||||
|
||||
|
||||
#### Streaming results
|
||||
### Stream results
|
||||
|
||||
The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.
|
||||
|
||||
@ -171,3 +178,16 @@ Streaming is available for both the **Search** and **TraceQL** query types.
|
||||
You'll get immediate visibility of incoming traces on the results table.
|
||||
|
||||
{{< video-embed src="/media/docs/grafana/data-sources/tempo-streaming-v2.mp4" >}}
|
||||
|
||||
### Use filters and tags on spans
|
||||
|
||||
Using **Span Filters**, you can use filters to refine results when viewing span details.
|
||||
These filters are available when viewing details for a trace.
|
||||
|
||||
You can continue to apply filters until you have narrowed down your resulting spans to the select few you are most interested in.
|
||||
|
||||
**Service Name**, **Span Name**, **Duration**, and **Tags** have the same function and operation as the filters of the same name in the **Search** query builder.
|
||||
|
||||
In addition, you can search for a keyword, opt to **Show matches only**, opt to **Show critical path only**, and browse matches using **Prev** and **Next**.
|
||||
|
||||
Use **Clear** to reset the filters.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user