opentofu/docs/plugin-protocol

OpenTF Plugin Protocol

This directory contains documentation about the physical wire protocol that OpenTF Core uses to communicate with provider plugins.

Most providers are not written directly against this protocol. Instead, prefer to use an SDK that implements this protocol and write the provider against the SDK's API.

---

If you want to write a plugin for OpenTF, please refer to Extending OpenTF instead.

This documentation is for those who are developing OpenTF SDKs, rather than those implementing plugins.

---

From OpenTF v0.12.0 onwards, OpenTF's plugin protocol is built on gRPC. This directory contains .proto definitions of different versions of OpenTF's protocol.

Only .proto files published as part of OpenTF release tags are actually official protocol versions. If you are reading this directory on the main branch or any other development branch then it may contain protocol definitions that are not yet finalized and that may change before final release.

RPC Plugin Model

OpenTF plugins are normal executable programs that, when launched, expose gRPC services on a server accessed via the loopback interface. OpenTF Core discovers and launches plugins, waits for a handshake to be printed on the plugin's stdout, and then connects to the indicated port number as a gRPC client.

For this reason, we commonly refer to OpenTF Core itself as the plugin "client" and the plugin program itself as the plugin "server". Both of these processes run locally, with the server process appearing as a child process of the client. OpenTF Core controls the lifecycle of these server processes and will terminate them when they are no longer required.

The startup and handshake protocol is not currently documented. We hope to document it here or to link to external documentation on it in future.

Versioning Strategy

The Plugin Protocol uses a versioning strategy that aims to allow gradual enhancements to the protocol while retaining compatibility, but also to allow more significant breaking changes from time to time while allowing old and new plugins to be used together for some period.

The versioning strategy described below was introduced with protocol version 5.0 in OpenTF v0.12. Prior versions of OpenTF and prior protocol versions do not follow this strategy.

The authoritative definition for each protocol version is in this directory as a Protocol Buffers (protobuf) service definition. The files follow the naming pattern tfpluginX.Y.proto, where X is the major version and Y is the minor version.

Major and minor versioning

The minor version increases for each change introducing optional new functionality that can be ignored by implementations of prior versions. For example, if a new field were added to an response message, it could be a minor release as long as OpenTF Core can provide some default behavior when that field is not populated.

The major version increases for any significant change to the protocol where compatibility is broken. However, OpenTF Core and an SDK may both choose to support multiple major versions at once: the plugin handshake includes a negotiation step where client and server can work together to select a mutually-supported major version.

The major version number is encoded into the protobuf package name: major version 5 uses the package name tfplugin5, and one day major version 6 will switch to tfplugin6. This change of name allows a plugin server to implement multiple major versions at once, by exporting multiple gRPC services. Minor version differences rely instead on feature-detection mechanisms, so they are not represented directly on the wire and exist primarily as a human communication tool to help us easily talk about which software supports which features.

Version compatibility for Core, SDK, and Providers

A particular version of OpenTF Core has both a minimum minor version it requires and a maximum major version that it supports. A particular version of OpenTF Core may also be able to optionally use a newer minor version when available, but fall back on older behavior when that functionality is not available.

Likewise, each provider plugin release is compatible with a set of versions. The compatible versions for a provider are a list of major and minor version pairs, such as "4.0", "5.2", which indicates that the provider supports the baseline features of major version 4 and supports major version 5 including the enhancements from both minor versions 1 and 2. This provider would therefore be compatible with a OpenTF Core release that supports only protocol version 5.0, since major version 5 is supported and the optional 5.1 and 5.2 enhancements will be ignored.

If OpenTF Core and the plugin do not have at least one mutually-supported major version, OpenTF Core will return an error from opentf init during plugin installation:

Provider "aws" v1.0.0 is not compatible with OpenTF v0.12.0.

Provider version v2.0.0 is the earliest compatible version.
Select it with the following version constraint:

    version = "~> 2.0.0"

Provider "aws" v3.0.0 is not compatible with OpenTF v0.12.0.
Provider version v2.34.0 is the latest compatible version. Select 
it with the following constraint:

    version = "~> 2.34.0"

Alternatively, upgrade to the latest version of OpenTF for compatibility with newer provider releases.

The above messages are for plugins installed via opentf init from a OpenTF registry, where the registry API allows OpenTF Core to recognize the protocol compatibility for each provider release. For plugins that are installed manually to a local plugin directory, OpenTF Core has no way to suggest specific versions to upgrade or downgrade to, and so the error message is more generic:

The installed version of provider "example" is not compatible with OpenTF v0.12.0.

This provider was loaded from:
     /usr/local/bin/terraform-provider-example_v0.1.0

Adding/removing major version support in SDK and Providers

The set of supported major versions is decided by the SDK used by the plugin. Over time, SDKs will add support for new major versions and phase out support for older major versions.

In doing so, the SDK developer passes those capabilities and constraints on to any provider using their SDK, and that will in turn affect the compatibility of the plugin in ways that affect its semver-based version numbering:

• If an SDK upgrade adds support for a new provider protocol, that will usually be considered a new feature and thus warrant a new minor version.
• If an SDK upgrade removes support for an old provider protocol, that is always a breaking change and thus requires a major release of the provider.

For this reason, SDK developers must be clear in their release notes about the addition and removal of support for major versions.

OpenTF Core also makes an assumption about major version support when it produces actionable error messages for users about incompatibilities: a particular protocol major version is supported for a single consecutive range of provider releases, with no "gaps".

Using the protobuf specifications in an SDK

If you wish to build an SDK for OpenTF plugins, an early step will be to copy one or more .proto files from this directory into your own repository (depending on which protocol versions you intend to support) and use the protoc protocol buffers compiler (with gRPC extensions) to generate suitable RPC stubs and types for your target language.

For example, if you happen to be targeting Python, you might generate the stubs using a command like this:

protoc --python_out=. --grpc_python_out=. tfplugin5.1.proto

You can find out more about the tool usage for each target language in the gRPC Quick Start guides.

The protobuf specification for a version is immutable after it has been included in at least one OpenTF release. Any changes will be documented in a new .proto file establishing a new protocol version.

The protocol buffer compiler will produce some sort of library object appropriate for the target language, which depending on the language might be called a module, or a package, or something else. We recommend to include the protocol major version in your module or package name so that you can potentially support multiple versions concurrently in future. For example, if you are targeting major version 5 you might call your package or module tfplugin5.

To upgrade to a newer minor protocol version, copy the new .proto file from this directory into the same location as your previous version, delete the previous version, and then run the protocol buffers compiler again against the new .proto file. Because minor releases are backward-compatible, you can simply update your previous stubs in-place rather than creating a new set alongside.

To support a new major protocol version, create a new package or module and copy the relevant .proto file into it, creating a separate set of stubs that can in principle allow your SDK to support both major versions at the same time. We recommend supporting both the previous and current major versions together for a while across a major version upgrade so that users can avoid having to upgrade both OpenTF Core and all of their providers at the same time, but you can delete the previous major version stubs once you remove support for that version.

Note: Some of the .proto files contain statements about being updated in-place for minor versions. This reflects an earlier version management strategy which is no longer followed. The current process is to create a new file in this directory for each new minor version and consider all previously-tagged definitions as immutable. The outdated comments in those files are retained in order to keep the promise of immutability, even though it is now incorrect.