mirror of
https://github.com/grafana/grafana.git
synced 2025-02-25 18:55:37 -06:00
Docs: Add distributed tracing plugin developer guide (#65898)
* Docs: Add plugin developer guide on distributed tracing * Docs: Fix typo * Docs: Plugins: Tracing: Address TODOs and other changes * Docs: Plugins: Tracing: Fix typo * Docs: Plugins: Tracing: Add note about min SDK version and update command * Docs: Plugins: Tracing: Fix some typos, rephrase some sentences * Docs: Plugins: Traacing: Removed manual instance management instructions * Update docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md Co-authored-by: Joseph Perez <45749060+josmperez@users.noreply.github.com> * Docs: Plugins: Tracing: Use markdown instead of HTML --------- Co-authored-by: Joseph Perez <45749060+josmperez@users.noreply.github.com>
This commit is contained in:
parent
1855bca2fa
commit
d1f1beaef9
@ -52,6 +52,7 @@ Improve an existing plugin with one of our guides:
|
||||
- [Add support for Explore queries]({{< relref "add-support-for-explore-queries.md" >}})
|
||||
- [Add support for variables]({{< relref "add-support-for-variables.md" >}})
|
||||
- [Add a query editor help component]({{< relref "add-query-editor-help.md" >}})
|
||||
- [Add distributed tracing for backend plugins]({{< relref "add-distributed-tracing-for-backend-plugins.md" >}})
|
||||
- [Build a logs data source plugin]({{< relref "build-a-logs-data-source-plugin.md" >}})
|
||||
- [Build a streaming data source plugin]({{< relref "build-a-streaming-data-source-plugin.md" >}}/)
|
||||
- [Error handling]({{< relref "error-handling.md" >}})
|
||||
|
@ -0,0 +1,123 @@
|
||||
---
|
||||
title: Add distributed tracing for backend plugins
|
||||
---
|
||||
|
||||
# Add distributed tracing for backend plugins
|
||||
|
||||
> **Note:** This feature requires at least Grafana 9.5.0, and your plugin needs to be built at least with
|
||||
> grafana-plugins-sdk-go v0.157.0. If you run a plugin with tracing features on an older version of Grafana,
|
||||
> tracing will be disabled.
|
||||
|
||||
## Introduction
|
||||
|
||||
Distributed tracing allows backend plugin developers to create custom spans in their plugins, and send them to the same endpoint
|
||||
and with the same propagation format as the main Grafana instance. The tracing context is also propagated from the Grafana instance
|
||||
to the plugin, so the plugin's spans will be correlated to the correct trace.
|
||||
|
||||
## Configuration
|
||||
|
||||
> **Note:** Only OpenTelemetry is supported. If Grafana is configured to use a deprecated tracing system (Jaeger or OpenTracing),
|
||||
> tracing will be disabled in the plugin. Please note that OpenTelemetry + Jaeger propagator is supported.
|
||||
|
||||
OpenTelemetry must be enabled and configured for the Grafana instance. Please refer to [this section](
|
||||
{{< relref "../../setup-grafana/configure-grafana/#tracingopentelemetry" >}}) for more information.
|
||||
|
||||
As of Grafana 9.5.0, plugins tracing must be enabled manually on a per-plugin basis, by specifying `tracing = true` in the plugin's config section:
|
||||
|
||||
```ini
|
||||
[plugin.myorg-myplugin-datasource]
|
||||
tracing = true
|
||||
```
|
||||
|
||||
## Implementing tracing in your plugin
|
||||
|
||||
> **Note:** Make sure you are using at least grafana-plugin-sdk-go v0.157.0. You can update with `go get -u github.com/grafana/grafana-plugin-sdk-go`.
|
||||
|
||||
When OpenTelemetry tracing is enabled on the main Grafana instance and tracing is enabeld for a plugin,
|
||||
the OpenTelemetry endpoint address and propagation format will be passed to the plugin during startup,
|
||||
which will be used to configure a global tracer.
|
||||
|
||||
1. The global tracer is configured automatically if you use <code>datasource.Manage</code> or <code>app.Manage</code> to run your plugin.
|
||||
|
||||
This also allows you to specify custom attributes for the default tracer:
|
||||
|
||||
```go
|
||||
func main() {
|
||||
if err := datasource.Manage("MY_PLUGIN_ID", plugin.NewDatasource, datasource.ManageOpts{
|
||||
TracingOpts: tracing.Opts{
|
||||
// Optional custom attributes attached to the tracer's resource.
|
||||
// The tracer will already have some SDK and runtime ones pre-populated.
|
||||
CustomAttributes: []attribute.KeyValue{
|
||||
attribute.String("my_plugin.my_attribute", "custom value"),
|
||||
},
|
||||
},
|
||||
}); err != nil {
|
||||
log.DefaultLogger.Error(err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
1. Once tracing is configured, you can access the global tracer with:
|
||||
|
||||
```go
|
||||
tracing.DefaultTracer()
|
||||
```
|
||||
|
||||
this returns an [OpenTelemetry trace.Tracer](https://pkg.go.dev/go.opentelemetry.io/otel/trace#Tracer), and can be used to create spans.
|
||||
|
||||
For example:
|
||||
|
||||
```go
|
||||
func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) {
|
||||
ctx, span := tracing.DefaultTracer().Start(
|
||||
ctx,
|
||||
"query processing",
|
||||
trace.WithAttributes(
|
||||
attribute.String("query.ref_id", query.RefID),
|
||||
attribute.String("query.type", query.QueryType),
|
||||
attribute.Int64("query.max_data_points", query.MaxDataPoints),
|
||||
attribute.Int64("query.interval_ms", query.Interval.Milliseconds()),
|
||||
attribute.Int64("query.time_range.from", query.TimeRange.From.Unix()),
|
||||
attribute.Int64("query.time_range.to", query.TimeRange.To.Unix()),
|
||||
),
|
||||
)
|
||||
defer span.End()
|
||||
log.DefaultLogger.Debug("query", "traceID", trace.SpanContextFromContext(ctx).TraceID())
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Refer to the [OpenTelemetry Go SDK](https://pkg.go.dev/go.opentelemetry.io/otel) for in-depth documentation about all the features provided by OpenTelemetry.
|
||||
|
||||
If tracing is disabled in Grafana, `backend.DefaultTracer()` returns a no-op tracer.
|
||||
|
||||
### Tracing GRPC calls
|
||||
|
||||
A new span is created automatically for each GRPC call (`QueryData`, `CheckHealth`, etc), both on Grafana's side and
|
||||
on the plugin's side.
|
||||
|
||||
This also injects the trace context into the `context.Context` passed to those methods.
|
||||
|
||||
This allows you to retrieve the [trace.SpanContext](https://pkg.go.dev/go.opentelemetry.io/otel/trace#SpanContext) by using `tracing.SpanContextFromContext` by passing the original `context.Context` to it:
|
||||
|
||||
```go
|
||||
func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) {
|
||||
spanCtx := trace.SpanContextFromContext(ctx)
|
||||
traceID := spanCtx.TraceID()
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Tracing HTTP requests
|
||||
|
||||
When tracing is enabled, a `TracingMiddleware` is also added to the default middleware stack to all HTTP clients created
|
||||
using the `httpclient.New` or `httpclient.NewProvider`, unless custom middlewares are specified.
|
||||
|
||||
This middleware creates spans for each outgoing HTTP request and provides some useful attributes and events related to the request's lifecycle.
|
||||
|
||||
## Complete plugin example
|
||||
|
||||
You can refer to the [datasource-http-backend plugin example](https://github.com/grafana/grafana-plugin-examples/tree/main/examples/datasource-http-backend) for a complete example of a plugin that has full tracing support.
|
Loading…
Reference in New Issue
Block a user