mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Docs: Updates to Traces in Explore doc (#90774)
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> Co-authored-by: Haris Rozajac <58232930+harisrozajac@users.noreply.github.com>
This commit is contained in:
Larissa Wandzura
GitHub
parent
3952f627eb
commit
6efd52eac8
@ -1,5 +1,4 @@
|
||||
---
|
||||
description: Tracing in Explore
|
||||
keywords:
|
||||
- explore
|
||||
- trace
|
||||
@ -8,167 +7,161 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Tracing in Explore
|
||||
title: Traces in Explore
|
||||
weight: 20
|
||||
---
|
||||
|
||||
# Tracing in Explore
|
||||
# Traces in Explore
|
||||
|
||||
You can use Explore to query and visualize traces from tracing data sources.
|
||||
You can use Explore to query and visualize traces from tracing data sources. Supported data sources include:
|
||||
|
||||
Supported data sources are:
|
||||
|
||||
- [Tempo]({{< relref "../datasources/tempo/" >}}) (supported ingestion formats: OpenTelemetry, Jaeger, and Zipkin)
|
||||
- [Jaeger]({{< relref "../datasources/jaeger/" >}})
|
||||
- [Zipkin]({{< relref "../datasources/zipkin/" >}})
|
||||
- [Tempo](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/)
|
||||
- [Jaeger](/docs/grafana/<GRAFANA_VERSION>/datasources/jaeger/)
|
||||
- [Zipkin](/docs/grafana/<GRAFANA_VERSION>/datasources/zipkin/)
|
||||
- [X-Ray](https://grafana.com/grafana/plugins/grafana-x-ray-datasource)
|
||||
- [Azure Monitor Application Insights]({{< relref "../datasources/azure-monitor/" >}})
|
||||
- [Azure Monitor](/docs/grafana/latest/datasources/azure-monitor/)
|
||||
- [ClickHouse](https://github.com/grafana/clickhouse-datasource)
|
||||
- [New Relic](https://grafana.com/grafana/plugins/grafana-newrelic-datasource)
|
||||
- [Infinity](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource)
|
||||
- [New Relic](/docs/plugins/grafana-newrelic-datasource/latest/)
|
||||
- [Infinity](/docs/plugins/yesoreyeram-infinity-datasource/latest/)
|
||||
|
||||
For information on how to configure queries for the data sources listed above, refer to the documentation for specific data source.
|
||||
Here are some references to learn more about traces and how you can use them:
|
||||
|
||||
## Query editor
|
||||
- [Introduction to tracing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/)
|
||||
- [Trace structure](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/trace-structure/#trace-structure)
|
||||
- [Traces and telemetry](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/telemetry/)
|
||||
- [Using traces to find solutions to problems](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/solutions-with-traces/)
|
||||
- [Best practices for tracing](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/tracing-best-practices/)
|
||||
|
||||
You can query and search tracing data using a data source's query editor.
|
||||
## Query editors
|
||||
|
||||
Each data source can have it's own query editor. The query editor for the Tempo data source is slightly different than the query editor for the Jaeger data source.
|
||||
You can query and search tracing data using a data source's query editor. Note that data sources in Grafana have unique query editors.
|
||||
|
||||
For information on querying each data source, refer to their documentation:
|
||||
|
||||
- [Tempo query editor]({{< relref "../datasources/tempo/query-editor" >}})
|
||||
- [Jaeger query editor]({{< relref "../datasources/jaeger/#query-the-data-source" >}})
|
||||
- [Zipkin query editor]({{< relref "../datasources/zipkin/#query-the-data-source" >}})
|
||||
- [Azure Monitor Application Insights query editor]({{< relref "../datasources/azure-monitor/query-editor/#query-application-insights-traces" >}})
|
||||
- [ClickHouse query editor](https://clickhouse.com/docs/en/integrations/grafana/query-builder#traces)
|
||||
For information on how to use the query editor to create queries for tracing data sources, refer to the documentation for each individual data source.
|
||||
|
||||
## Trace view
|
||||
|
||||
This section explains the elements of the Trace View.
|
||||
Grafana's trace view provides an overview of a request as it travels through your system. The following sections provide detail on various elements of the trace view.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the trace view" >}}
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view" >}}
|
||||
|
||||
### Header
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-header.png" class="docs-image--no-shadow" max-width= "750px" caption="Screenshot of the trace view header" >}}
|
||||
The trace view header includes the following:
|
||||
|
||||
- Header title: Shows the name of the root span and trace ID.
|
||||
- Search: Highlights spans containing the searched text.
|
||||
- Metadata: Various metadata about the trace.
|
||||
- **Header title** - Shows the name of the root span and trace ID.
|
||||
- **Search** - Highlights spans containing the searched text.
|
||||
- **Metadata** - Various metadata about the trace.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-header.png" class="docs-image--no-shadow" max-width= "750px" caption="Trace view header" >}}
|
||||
|
||||
### Minimap
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-minimap.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the trace view minimap" >}}
|
||||
**Minimap** displays a condensed view of the trace timeline. Drag your mouse over the minimap to zoom into a smaller time range. This also updates the main timeline, making it easier to view shorter spans
|
||||
When zoomed in, hovering over the minimap displays **Reset selection**, which resets the zoom.
|
||||
|
||||
Shows condensed view or the trace timeline. Drag your mouse over the minimap to zoom into smaller time range. Zooming will also update the main timeline, so it is easy to see shorter spans. Hovering over the minimap, when zoomed, will show Reset Selection button which resets the zoom.
|
||||
|
||||
### Span filters
|
||||
|
||||
|
||||
|
||||
Using span filters, you can filter your spans in the trace timeline viewer. The more filters you add, the more specific are the filtered spans.
|
||||
|
||||
You can add one or more of the following filters:
|
||||
|
||||
- Resource service name
|
||||
- Span name
|
||||
- Duration
|
||||
- Tags (which include tags, process tags, and log fields)
|
||||
|
||||
To only show the spans you have matched, you can press the `Show matches only` toggle.
|
||||
|
||||
{{< youtube id="VP2XV3IIc80" >}}
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-minimap.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view minimap example" >}}
|
||||
|
||||
### Timeline
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-timeline.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the trace view timeline" >}}
|
||||
Timeline shows list of spans within the trace. Each span row consists of the following components:
|
||||
|
||||
Shows list of spans within the trace. Each span row consists of these components:
|
||||
- **Expand children** - Expands or collapses all the children spans of the selected span.
|
||||
- **Service name** - Name of the service logged the span.
|
||||
- **Operation name** - Name of the operation that this span represents.
|
||||
- **Span duration bar** - Visual representation of the operation duration within the trace.
|
||||
|
||||
- Expand children button: Expands or collapses all the children spans of selected span.
|
||||
- Service name: Name of the service logged the span.
|
||||
- Operation name: Name of the operation that this span represents.
|
||||
- Span duration bar: Visual representation of the operation duration within the trace.
|
||||
Click anywhere on the span row to reveal span details.
|
||||
|
||||
Clicking anywhere on the span row shows span details.
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-timeline.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view timeline" >}}
|
||||
|
||||
### Span details
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-span-details.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the trace view span details" >}}
|
||||
Traces are composed of one or more spans.
|
||||
A span is a unit of work within a trace that has a start time relative to the beginning of the trace, a duration and an operation name for the unit of work.
|
||||
It usually has a reference to a parent span, unless it’s the first span, the root span, in a trace.
|
||||
It frequently includes key/value attributes that are relevant to the span itself, for example the HTTP method used in the request, as well as other metadata such as the service name, sub-span events, or links to other spans.
|
||||
|
||||
- Operation name.
|
||||
- Span metadata.
|
||||
- Tags: Any tags associated with this span.
|
||||
- Process metadata: Metadata about the process that logged this span.
|
||||
- Logs: List of logs logged by this span and associated key values. In case of Zipkin logs section shows Zipkin annotations.
|
||||
You can expand any span in a trace and view the details, including the span and resource attributes.
|
||||
|
||||
For more information about spans and traces, refer to [Introduction to tracing](https://grafana.com/docs/tempo/latest/introduction/) in the Tempo documentation.
|
||||
|
||||
Span details include:
|
||||
|
||||
- **Span attributes** - Key/value pairs that provides context for spans. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key `http.url`) and the HTTP status code returned (as the span attribute `http.status_code`).
|
||||
|
||||
- **Resource attributes** - Key/value pairs that describe the context of how the span was collected.
|
||||
|
||||
Refer to [Span and resource attributes](/docs/tempo/<TEMPO_VERSION>/operations/best-practices/#span-and-resource-attributes) for more detail.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-span-details.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace view span details" >}}
|
||||
|
||||
### Span filters
|
||||
|
||||
Span filters allow you to refine the spans displayed in the trace timeline viewer.
|
||||
The more filters you add, the more specific the filtered spans become.
|
||||
Click on a trace to access Span filters.
|
||||
|
||||
|
||||
|
||||
You can add one or more of the following filters:
|
||||
|
||||
- **Service name** - Filter by selecting a service name from the dropdown.
|
||||
- **Span name** - Filter by selecting a span name from the dropdown.
|
||||
- **Duration** - Filter by duration. Accepted units include ns, us, ms, s, m, h.
|
||||
- **Tags** - Filter by tags, process tags, or log fields in your span.
|
||||
|
||||
To only show the spans you have matched, toggle **Show matches only**.
|
||||
|
||||
Refer to [Span filters](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/span-filters/) for more in depth information.
|
||||
|
||||
Watch the following video to learn more about filtering trace spans in Grafana:
|
||||
{{< youtube id="VP2XV3IIc80" >}}
|
||||
|
||||
### Trace to logs
|
||||
|
||||
You can navigate from a span in a trace view directly to logs relevant for that span. This feature is available for Tempo, Jaeger, and Zipkin data sources. Refer to their [relevant documentation](/docs/grafana/latest/datasources/tempo/#trace-to-logs) for configuration instructions.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-trace-to-logs.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the trace view in Explore with icon next to the spans" >}}
|
||||
You can navigate from a span in a trace view directly to logs relevant for that span.
|
||||
This feature is available for the Tempo, Jaeger, and Zipkin data sources.
|
||||
Refer to each individual data source's documentation for configuration instructions.
|
||||
|
||||
Click the document icon to open a split view in Explore with the configured data source and query relevant logs for the span.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-trace-view-trace-to-logs.png" class="docs-image--no-shadow" max-width= "900px" caption="Trace to logs" >}}
|
||||
|
||||
### Trace to metrics
|
||||
|
||||
{{% admonition type="note" %}}
|
||||
This feature is currently in beta and behind the `traceToMetrics` feature toggle.
|
||||
{{% /admonition %}}
|
||||
You can navigate from a span in a trace view directly to metrics relevant for that span.
|
||||
This feature is available for the Tempo, Jaeger, and Zipkin data sources.
|
||||
|
||||
You can navigate from a span in a trace view directly to metrics relevant for that span. This feature is available for Tempo, Jaeger, and Zipkin data sources. Refer to their [relevant documentation](/docs/grafana/latest/datasources/tempo/configure-tempo-data-source/#trace-to-metrics) for configuration instructions.
|
||||
Refer to each individual data source's documentation for configuration instructions.
|
||||
For Tempo, refer to [Trace to metrics configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/#trace-to-metrics).
|
||||
|
||||
### Trace to profiles
|
||||
|
||||
Using Trace to profiles, you can use Grafana’s ability to correlate different signals by adding the functionality to link between traces and profiles.
|
||||
Refer to the [relevant documentation](/docs/grafana/latest/datasources/tempo/configure-tempo-data-source#trace-to-profiles) for configuration instructions.
|
||||
|
||||
For Tempo refer to [Trace to profiles](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source#trace-to-profiles) for configuration instructions.
|
||||
|
||||
{{< figure src="/static/img/docs/tempo/profiles/tempo-trace-to-profile.png" max-width="900px" class="docs-image--no-shadow" alt="Selecting a link in the span queries the profile data source" >}}
|
||||
|
||||
## Node graph
|
||||
|
||||
You can optionally expand the node graph for the displayed trace. Depending on the data source, this can show spans of the trace as nodes in the graph, or as some additional context like service graph based on the current trace.
|
||||
You can also expand the node graph for a displayed trace. If the data source supports it, this displays spans of the trace as nodes in the graph, or provides additional context, such as a service graph based on the current trace.
|
||||
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-node-graph.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the node graph" >}}
|
||||
Refer to [Node graph](/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/) for additional information.
|
||||
|
||||
## Service Graph
|
||||
{{< admonition type="note" >}}
|
||||
The node graph requires data to be returned from the data source in a specific format to display correctly. Refer to [Data API](/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/node-graph/#data-api), [Nodes data frame structure](/docs/grafana/latest/panels-visualizations/visualizations/node-graph/#nodes-data-frame-structure) and [Node graph data requirements](/docs/grafana/latest/panels-visualizations/visualizations/node-graph/#data-requirements) for additional information and configuration instructions.
|
||||
{{< /admonition >}}
|
||||
|
||||
The Service Graph visualizes the span metrics (traces data for rates, error rates, and durations (RED)) and service graphs.
|
||||
Once the requirements are set up, this pre-configured view is immediately available.
|
||||
{{< figure src="/media/docs/tempo/screenshot-grafana-node-graph.png" class="docs-image--no-shadow" max-width= "900px" caption="Node graph" >}}
|
||||
|
||||
For more information, refer to the [Service Graph view section]({{< relref "../datasources/tempo/#open-the-service-graph-view" >}}) of the Tempo data source page and the [service graph view page](/docs/tempo/latest/metrics-generator/service-graph-view/) in the Tempo documentation.
|
||||
## Service graph
|
||||
|
||||
{{< figure src="/static/img/docs/grafana-cloud/apm-overview.png" class="docs-image--no-shadow" max-width= "900px" caption="Screenshot of the Service Graph view" >}}
|
||||
A service graph visualizes span metrics, including rates, error rates, and durations (RED), along with service relationships. Once the requirements are configured, this pre-configured view is immediately available.
|
||||
|
||||
## Data API
|
||||
For additional information refer to the following documentation:
|
||||
|
||||
This visualization needs a specific shape of the data to be returned from the data source in order to correctly display it.
|
||||
- [Service Graph and Service Graph view](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/service-graph/)
|
||||
- [Service graph view](/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/) in Tempo documentation
|
||||
|
||||
Data source needs to return data frame and set `frame.meta.preferredVisualisationType = 'trace'`.
|
||||
|
||||
### Data frame structure
|
||||
|
||||
Required fields:
|
||||
|
||||
| Field name | Type | Description |
|
||||
| ------------ | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| traceID | string | Identifier for the entire trace. There should be only one trace in the data frame. |
|
||||
| spanID | string | Identifier for the current span. SpanIDs should be unique per trace. |
|
||||
| parentSpanID | string | SpanID of the parent span to create child parent relationship in the trace view. Can be `undefined` for root span without a parent. |
|
||||
| serviceName | string | Name of the service this span is part of. |
|
||||
| serviceTags | TraceKeyValuePair[] | List of tags relevant for the service. |
|
||||
| startTime | number | Start time of the span in millisecond epoch time. |
|
||||
| duration | number | Duration of the span in milliseconds. |
|
||||
|
||||
Optional fields:
|
||||
|
||||
| Field name | Type | Description |
|
||||
| -------------- | ------------------- | ------------------------------------------------------------------ |
|
||||
| logs | TraceLog[] | List of logs associated with the current span. |
|
||||
| tags | TraceKeyValuePair[] | List of tags associated with the current span. |
|
||||
| warnings | string[] | List of warnings associated with the current span. |
|
||||
| stackTraces | string[] | List of stack traces associated with the current span. |
|
||||
| errorIconColor | string | Color of the error icon in case span is tagged with `error: true`. |
|
||||
|
||||
For details about the types see [TraceSpanRow](https://github.com/grafana/grafana/blob/main/packages/grafana-data/src/types/trace.ts#L28), [TraceKeyValuePair](https://github.com/grafana/grafana/blob/main/packages/grafana-data/src/types/trace.ts#L4) and [TraceLog](https://github.com/grafana/grafana/blob/main/packages/grafana-data/src/types/trace.ts#L12).
|
||||
{{< figure src="/static/img/docs/grafana-cloud/apm-overview.png" class="docs-image--no-shadow" max-width= "900px" caption="Service graph view" >}}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user