IntenseWebs/opentofu
3
0
Fork 0
mirror of https://github.com/opentofu/opentofu.git synced 2025-02-25 18:45:20 -06:00
Code Issues Packages Projects Releases Wiki Activity

State Encryption Documentation and Partial Implementation (#1227)

Browse Source 
Signed-off-by: StephanHCB <sbs_github_u43a@packetloss.de>
Signed-off-by: Christian Mesh <christianmesh1@gmail.com>
Signed-off-by: Janos <86970079+janosdebugs@users.noreply.github.com>
Signed-off-by: James Humphries <james@james-humphries.co.uk>
Co-authored-by: StephanHCB <sbs_github_u43a@packetloss.de>
Co-authored-by: Janos <86970079+janosdebugs@users.noreply.github.com>
Co-authored-by: Christian Mesh <christianmesh1@gmail.com>
This commit is contained in:
James Humphries 2024-02-16 14:59:19 +00:00 committed by GitHub
parent 7054dda96e
commit cbab4bee83
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
36 changed files with 3014 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

351
docs/state_encryption.md Normal file
View File

@ -0,0 +1,351 @@
# State encryption
This document details our intended implementation of the state and plan encryption feature.
## Notes
This document mentions `HCL` as a short-form for OpenTofu code. Unless otherwise specified, everything written also applies to the [JSON-equivalent of the HCL code](https://opentofu.org/docs/language/syntax/json/).
## Goals
The goal of this feature is to allow OpenTofu users to fully encrypt state files when they are stored on the local disk or transferred to a remote backend. The feature should also allow reading from an encrypted remote backend using the `terraform_remote_state` data source. The encrypted version should still be a valid JSON file, but not necessarily a valid state file.
Furthermore, this feature should allow users to encrypt plan files when they are stored. However, plan files are not JSON, they are undocumented binary files and should be treated as such.
For the encryption key, users should be able to specify a key directly, use a remote key provider (such as AWS KMS, etc.), or create derivative keys from another key source. The primary encryption method should be AES-GCM, but the implementation should be open to different encryption methods. The user should also have the ability to decrypt a state or plan file with one (older) key and then re-encrypt data with a newer key. Multiple fallbacks should be avoided in the implementation.
To enable use cases where multiple teams need to collaborate, the user should be able to specify separate encryption methods and keys for individual uses, especially for the `terraform_remote_state` data source. However, to simplify configuration, the user should be able to specify a default configuration for all remote state data sources.
It is the goal of this feature to let users specify their encryption configuration both in (HCL) code and in environment variables. The latter is necessary to allow users the reuse of code for both encrypted and unencrypted state storage.
Finally, it is the goal of the encryption feature to make available a library that third party tooling can use to encrypt and decrypt state. This may be implemented as a package within the OpenTofu repository, or as a standalone repository.
## Possible future goals
This section describes possible future goals. However, these goals are merely aspirations, and we may or may not implement them, or implement them differently based on community feedback. We describe these aspirations here to make clear which features we intentionally left out of scope for the current implementation.
Users use CI/CD systems or security scanners that need to read the state or plan files, but may not fully trust these systems. In the future, the user should be able to specify partial encryption. This encryption type would only encrypt sensitive values instead of the whole state file.
At this time, due to the limitations on passing providers through to modules, encryption configuration is global. However, in the future, the user should be able to create a module that carries along their own encryption method and how it relates to the `terraform_remote_state` data sources. This is important so individual teams can ship ready-to-use modules to other teams that access their state. However, due to the constraints on passing resources to modules this is currently out of scope for this proposal.
Finally, it is a future goal to enable providers to provide their own key providers and encryption methods. Users may also want to create additional, encryption-related, such as merely signing plan files, which this functionality would enable.
## Non-goals
In this section we describe the features that are out of scope for state and plan encryption. We do not aspire to solve these problems with the same implementation, and they must be addressed separately if the community chooses to support these endeavours.
The primary goal of this feature is to protect state and plan files **at rest**. It is not the goal of this feature to protect other channels secrets may be accessed through, such as the JSON output. As such, it is not a goal of this feature to encrypt any output on the standard output, or file output that is not a state or plan file.
It is also not a goal of this feature to protect the state file against the operator of the device running `tofu`. The operator already has access to the encryption key and can decrypt the data without the `tofu` binary being present if they so chose.
## User-facing effects
Unless the user explicitly specifies encryption options, no encryption will take place and OpenTofu will continue to function as before. No forced encryption will take place. Furthermore, regardless of the encryption status, other functionality, such as state management CLI functions, JSON output, etc. remain unaffected and will be readable in plain text if they were readable as plain text before. Only state and plan files will be affected by the encryption.
Users will be able to specify their encryption configuration both in code and via environment variables. Both configurations are equivalent and will be merged at execution time. For more details, see the [environment configuration](#environment-configuration) section below.
When a user wants to enable encryption, they must specify the following block:
```hcl2
terraform {
encryption {
// Encryption options
}
}
```
The mere presence of the `encryption` block alone should not enable encryption because the user should explicitly specify what key and method to use. The implementation should error and alert the user if the encryption block is present but has no configuration.
The encryption relies on an encryption key, or a composite encryption key, which the user can provide directly or via a key management system. The user must provide at least one `key_provider` block with the settings described below. These key providers serve the purpose of creating or providing the encryption key. For example, the user could hard-code a static key named foo:
```hcl2
terraform {
encryption {
key_provider "static" "foo" {
key = "6f6f706830656f67686f6834616872756f3751756165686565796f6f72653169"
}
}
}
```
> [!NOTE]
> The user is responsible for keeping this key safe and follow disaster recovery best practices. OpenTofu is a read-only consumer for the key the user provided and will not perform tasks on the key management system like key rotation.
The user also has to specify at least one encryption method referencing the key provider. This encryption method determines how the encryption takes place. It is the user's responsibility to make sure that they provided a key that is suitable for the encryption method.
```hcl2
terraform {
encryption {
//...
method "aes_gcm" "bar" {
key_provider = key_provider.static.foo
}
}
}
```
Finally, the user must reference the method for use in their state file, plan file, etc. This enables creating a different configuration for different purposes.
```hcl2
terraform {
encryption {
//...
statefile {
method = method.aes_gcm.abc
}
planfile {
method = method.aes_gcm.cde
}
backend {
method = method.some_derivative_key_provider.efg
}
remote_data_sources {
default {
method = method.aes_gcm.ghi
}
remote_data_source "some_module.remote_data_source.foo" {
method = method.aes_gcm.ijk
}
}
}
}
```
To facilitate key and method rollover, the user can specify a fallback configuration for state and plan decryption. When the user specifies a `fallback` block, `tofu` will first attempt to decrypt any state or plan it reads with the primary method and then fall back to the `fallback` method. When `tofu` writes a state or plan, it will not use the `fallback` method and always writes with the primary method.
```hcl2
terraform {
encryption {
//...
statefile {
method = method.aes_gcm.bar
fallback {
method = method.aes_gcm.baz
}
}
}
}
```
> [!NOTE]
> The `fallback` is a block because the future goals may require adding additional options.
> [!NOTE]
> Multiple `fallback` blocks should not be supported or should be discouraged because they would be detrimental to performance and encourage keeping old encryption keys in the configuration.
In the situation where `enforce` is not set to true, and no `method` or `fallback` is specified, no encryption will take place. If the user does not specify a method, but specifies a fallback, the next apply command will disable the encryption given that `enforce` is not set to true. This is to allow the user to disable encryption without removing the encryption configuration from the code.
### Composite keys
When a user desires to create a composite key, such as for creating a passphrase-based derivative key or a shared custody key, they may avail themselves of a key provider that supports multiple inputs. The user can chain key providers together:
```hcl2
terraform {
encryption {
key_provider "static" "my_passphrase" {
key = "this is my encryption key"
}
key_provider "some_derivative_key_provider" "my_derivative_key" {
key_providers = [key_provider.static.my_passphrase]
}
method "aes_gcm" "foo" {
key_provider = key_provider.some_derivative_key_provider.my_derivative_key
}
//...
}
}
```
> [!NOTE]
> The specific implementation of the derivative key must pay attention to combine the keys securely if it supports multiple key providers as inputs, such as using HMAC to combine keys.
### Environment configuration
As mentioned above, users can configure encryption in environment variables, either as HCL or JSON. To do this, the user has to specify the encryption configuration fragment in either of the two formats. The following two examples are equivalent:
```hcl2
key_provider "static" "my_key" {
key = "this is my encryption key"
}
method "aes_gcm" "foo" {
key_provider = key_provider.static.my_key
}
statefile {
method = method.aes_gcm.foo
}
```
```json
{
"key_provider" : {
"static": {
"my_key": {
"key": "this is my encryption key"
}
}
},
"method": {
"aes_gcm": {
"foo": {
"key_provider": "${key_provider.static.my_key}"
}
}
},
"statefile": {
"method": "${method.aes_gcm.foo}"
}
}
```
The user can set either of these structures in the `TF_ENCRYPTION` environment variable:
```bash
export TF_ENCRYPTION='{"key_provider":{...},"method":{...},"statefile":{...}}'
```
When the user specifies both an environment and a code configuration, `tofu` merges the two configurations. If two values conflict, the environment configuration takes precedence.
To ensure that the encryption cannot be accidentally forgotten or disabled and the data stored unencrypted, the user can specify the `enforced` option in the HCL configuration:
```hcl2
terraform {
encryption {
//...
statefile {
enforced = true
}
planfile {
enforced = true
}
backend {
enforced = true
}
}
}
```
> [!NOTE]
> The `enforced` option is also available in the environment configuration and works as intended, but doesn't make much sense because its primary purpose is to guard against environment variable omission.
## Encrypted state format
When `tofu` encrypts a state file, the encrypted state is still a JSON file. Any implementations can distinguish encrypted files by the `encryption` key being present in the JSON structure. However, there may be other keys in the state file depending on the encryption method and type.
For example (not final):
```json
{
"encryption": {
"method": "aes_gcm",
"key_provider": "static.my_key"
},
// ... Additional keys here
}
```
> [!WARNING]
> Tools working with state files should not make assumptions about the type or structure of the `encryption` field as it may vary from implementation to implementation.
## Encrypted plan format
An unencrypted plan file in OpenTofu is an opaque binary. This specification makes no rules for how the encrypted format should look like and all non-encryption routines should treat the value as opaque.
## Implementation
When implementing the encryption tooling, the implementation should be split in two parts: the library and the OpenTofu implementation. The library should rely on the cty types and hcl as a means to specify schema, but should not be otherwise tied to the OpenTofu codebase. This is necessary to enable encryption capabilities for third party tooling that may need to work with state and plan files.
### Library implementation
The encryption library should create an interface that other projects and OpenTofu itself can use to encrypt and decrypt state. The library should express its schema needs (e.g. for key provider config) using [cty](https://github.com/zclconf/go-cty) and [hcl](https://github.com/hashicorp/hcl), but be otherwise independent of the OpenTofu codebase. Ideally, the OpenTofu project should provide this library as a standalone dependency that does not pull in the entire OpenTofu dependency tree.
#### Encryption interface
The main component of the library should be the `Encryption` interface. This interface should provide methods to request an encryption tool for each individual purpose, such as:
```go
type Encryption interface {
StateFile() StateEncryption
PlanFile() PlanEncryption
Backend() StateEncryption
RemoteState(string) ReadOnlyStateEncryption
}
```
Each of the returned encryption tools should provide methods to encrypt the data of the specified purpose, such as:
```go
type ReadOnlyStateEncryption interface {
DecryptState([]byte) ([]byte, error)
}
type StateEncryption interface {
ReadOnlyStateEncryption
EncryptState([]byte) ([]byte, error)
}
```
The encryption routines should assume that they get passed a valid state or plan file and encrypt it as described in this document. Conversely, the decryption routines should assume that their input will be an encrypted state or plan file and should attempt to decrypt. The state decryption function should follow the fallback process described in this document.
#### Key providers
The main responsibility of a key provider is providing a key in a `[]byte`. It may consume structured configuration, which may also include references to other key providers. However, an implementation of a key provider should never have to deal with resolving these dependencies. Instead, the library should correctly resolve the key provider order and look up the keys in the right order and pass the already-resolved data in as [configuration](#configuration).
In addition to the encryption key, key providers may also emit additional metadata. The library must store this metadata alongside the encrypted data and pass it to the key provider when initializing the key provider for decryption in a subsequent run. The key provider is responsible for ensuring that no sensitive data is stored in the metadata.
> [!NOTE]
> Since a user can chain key providers, the library must make sure to store metadata from all key providers in the encrypted form. However, when the user renames the key provider the library may fail to decrypt the state or plan files if the user fails to provide an adequate fallback with the correct naming. The documentation for this feature should encourage users to create new key providers if they change the parameters in a backwards-incompatible manner, and they want to decrypt older state or plan files.
#### Methods
The responsibility of a method is to encrypt and decrypt an opaque block of data. The method is not responsible for understanding the structure of the data. Instead, the library core should take care of traversing the state or plan files and deciding specifically what to encrypt. A method must implement the encrypted format in such a way that it can determine if a subsequent decryption failed or not. Methods that cannot decide on decryption success without validating the underlying data, such as rot13, are not supported.
Similar to [key providers](#key-providers), the method may need configuration but should not have to deal with lookup up key providers itself.
#### Registering key providers and methods
The library should be modular. Anyone using the library, including OpenTofu, should be able to add new key providers and methods and read the configuration for these without modifying the library code. To that end, the library should provide a registry for key providers and methods.
The library should also not force any included key providers or methods onto its user, so the registry should not be global. Instead, every library user should configure their own registry. However, the library should provide a way to obtain a preconfigured registry with built-in key providers and methods.
#### Configuration
In order to ensure consistency between OpenTofu and other library users, the library should provide a method to parse an HCL or JSON block and turn it into configuration structures. In parallel, the library should also make it as simple as possible for implementers to safely provide new key providers and methods, which is why the library should also use struct tags in Go to convert the incoming configuration.
```go
type Config struct {
Key string `hcl:"key"`
}
```
### OpenTofu integration
Currently, the OpenTofu code is in large parts procedural and has globally scoped state. Launching a second instance of OpenTofu is impossible. As it is a much-requested feature to embed OpenTofu as a library, this will need to change in the future. Therefore, the integration of the library should do its best to not introduce more global variables if possible.
The implementation may avail itself of either a singleton, or choose to pass the Encryption interface along several function calls. Both have benefits and drawbacks. However, singletons make parallelized testing very difficult and should therefore be avoided in tests as much as possible.
#### Singleton
A singleton takes an otherwise instance-based struct and stores it in a global variable. As such, it can carry information across otherwise procedural code.
**Pros:**
* Simpler access to the singleton from calling code
* Less refactoring as the singleton is either available or not.
**Cons:**
* Harder to trace when/where the configuration is initialized.
* Easy to introduce new code paths that access the singleton before it is available.
#### Passed Instance
This approach requires changing large parts of the OpenTofu code in order to pass along the `Encryption` object. This can cause additional bugs and seduce the inexperienced coder into introducing a super-object (also often referred to as context) to hold everything. Instead, a more granular approach would be desirable, but may not be possible given the state of the code.
**Pros:**
* Easy to trace where a given encryption instance comes from
* Hard to introduce new code paths without passing the correct encryption interface.
Cons:
* More in-depth refactoring is required / more of the codebase edited in this work

2
internal/command/cliconfig/provider_installation.go
View File

@ -260,7 +260,7 @@ func decodeProviderInstallationFromConfig(hclFile *hclast.File) ([]*ProviderInst
devOverrides[addr] = getproviders.PackageLocalDir(dirPath)
}
continue // We won't add anything to pi.Methods for this one
continue // We won't add anything to pi.MethodConfigs for this one
default:
diags = diags.Append(tfdiags.Sourceless(

196
internal/encryption/base.go Normal file
View File

@ -0,0 +1,196 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"encoding/json"
"fmt"
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
"github.com/hashicorp/hcl/v2"
)
const (
encryptionVersion = "v0"
)
type baseEncryption struct {
enc *encryption
target *config.TargetConfig
enforced bool
name string
}
func newBaseEncryption(enc *encryption, target *config.TargetConfig, enforced bool, name string) *baseEncryption {
return &baseEncryption{
enc: enc,
target: target,
enforced: enforced,
name: name,
}
}
type basedata struct {
Meta map[keyprovider.Addr][]byte `json:"meta"`
Data []byte `json:"encrypted_data"`
Version string `json:"encryption_version"` // This is both a sigil for a valid encrypted payload and a future compatability field
}
func (s *baseEncryption) encrypt(data []byte) ([]byte, hcl.Diagnostics) {
if s.target == nil {
return data, nil
}
es := basedata{
Meta: make(map[keyprovider.Addr][]byte),
Version: encryptionVersion,
}
// Mutates es.Meta
methods, diags := s.buildTargetMethods(es.Meta)
if diags.HasErrors() {
return nil, diags
}
var encryptor method.Method = nil
if len(methods) != 0 {
encryptor = methods[0]
}
if encryptor == nil {
// ensure that the method is defined when Enforced is true
if s.enforced {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Encryption method required",
Detail: fmt.Sprintf("%q is enforced, and therefore requires a method to be provided", s.name),
})
return nil, diags
}
return data, nil
}
encd, err := encryptor.Encrypt(data)
if err != nil {
return nil, append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Encryption failed for " + s.name,
Detail: err.Error(),
})
}
es.Data = encd
jsond, err := json.Marshal(es)
if err != nil {
return nil, append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unable to encode encrypted data as json",
Detail: err.Error(),
})
}
return jsond, diags
}
// TODO Find a way to make these errors actionable / clear
func (s *baseEncryption) decrypt(data []byte, validator func([]byte) error) ([]byte, hcl.Diagnostics) {
if s.target == nil {
return data, nil
}
es := basedata{}
err := json.Unmarshal(data, &es)
if err != nil {
return nil, hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid data format for decryption",
Detail: err.Error(),
}}
}
if len(es.Version) == 0 {
// Not a valid payload, might be already decrypted
err = validator(data)
if err == nil {
// Yep, it's already decrypted
return data, nil
} else {
// Nope, just bad input
return nil, hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unable to determine data structure during decryption",
}}
}
}
if es.Version != encryptionVersion {
return nil, hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid encrypted payload version",
Detail: fmt.Sprintf("%s != %s", es.Version, encryptionVersion),
}}
}
methods, diags := s.buildTargetMethods(es.Meta)
if diags.HasErrors() {
return nil, diags
}
if len(methods) == 0 {
err = validator(data)
if err == nil {
// No methods/fallbacks specified and data is valid payload
return data, diags
} else {
// TODO improve this error message
return nil, append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: err.Error(),
})
}
}
var methodDiags hcl.Diagnostics
for _, method := range methods {
if method == nil {
// No method specified for this target
err = validator(data)
if err == nil {
return data, diags
}
// toDO improve this error message
methodDiags = append(methodDiags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Attempted decryption failed for " + s.name,
Detail: err.Error(),
})
continue
}
uncd, err := method.Decrypt(es.Data)
if err == nil {
// Success
return uncd, diags
}
// Record the failure
methodDiags = append(methodDiags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Attempted decryption failed for " + s.name,
Detail: err.Error(),
})
}
// Record the overall failure
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Decryption failed",
Detail: "All methods of decryption provided failed for " + s.name,
})
return nil, append(diags, methodDiags...)
}

103
internal/encryption/config/config.go Normal file
View File

@ -0,0 +1,103 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package config
import (
"github.com/hashicorp/hcl/v2"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
)
// Config describes the terraform.encryption HCL block you can use to configure the state and plan encryption.
// The individual fields of this struct match the HCL structure directly.
type Config struct {
KeyProviderConfigs []KeyProviderConfig `hcl:"key_provider,block"`
MethodConfigs []MethodConfig `hcl:"method,block"`
Backend *EnforcableTargetConfig `hcl:"backend,block"`
StateFile *EnforcableTargetConfig `hcl:"statefile,block"`
PlanFile *EnforcableTargetConfig `hcl:"planfile,block"`
Remote *RemoteConfig `hcl:"remote_data_source,block"`
}
// Merge returns a merged configuration with the current config and the specified override combined, the override
// taking precedence.
func (c *Config) Merge(override *Config) *Config {
return MergeConfigs(c, override)
}
// KeyProviderConfig describes the terraform.encryption.key_provider.* block you can use to declare a key provider for
// encryption. The Body field will contain the remaining undeclared fields the key provider can consume.
type KeyProviderConfig struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
Body hcl.Body `hcl:",remain"`
}
// Addr returns a keyprovider.Addr from the current configuration.
func (k KeyProviderConfig) Addr() (keyprovider.Addr, hcl.Diagnostics) {
return keyprovider.NewAddr(k.Type, k.Name)
}
// MethodConfig describes the terraform.encryption.method.* block you can use to declare the encryption method. The Body
// field will contain the remaining undeclared fields the method can consume.
type MethodConfig struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
Body hcl.Body `hcl:",remain"`
}
func (m MethodConfig) Addr() (method.Addr, hcl.Diagnostics) {
return method.NewAddr(m.Type, m.Name)
}
// RemoteConfig describes the terraform.encryption.remote block you can use to declare encryption for remote state data
// sources.
type RemoteConfig struct {
Default *TargetConfig `hcl:"default,block"`
Targets []NamedTargetConfig `hcl:"remote_data_source,block"`
}
// TargetConfig describes the target.encryption.state, target.encryption.plan, etc blocks.
type TargetConfig struct {
Method hcl.Expression `hcl:"method,optional"`
Fallback *TargetConfig `hcl:"fallback,block"`
}
// EnforcableTargetConfig is an extension of the TargetConfig that supports the enforced form.
//
// Note: This struct is copied because gohcl does not support embedding.
type EnforcableTargetConfig struct {
Enforced bool `hcl:"enforced,optional"`
Method hcl.Expression `hcl:"method,optional"`
Fallback *TargetConfig `hcl:"fallback,block"`
}
// AsTargetConfig converts the struct into its parent TargetConfig.
func (e EnforcableTargetConfig) AsTargetConfig() *TargetConfig {
return &TargetConfig{
Method: e.Method,
Fallback: e.Fallback,
}
}
// NamedTargetConfig is an extension of the TargetConfig that describes a
// terraform.encryption.remote.remote_state_data.* block.
//
// Note: This struct is copied because gohcl does not support embedding.
type NamedTargetConfig struct {
Name string `hcl:"name,label"`
Method hcl.Expression `hcl:"method,optional"`
Fallback *TargetConfig `hcl:"fallback,block"`
}
// AsTargetConfig converts the struct into its parent TargetConfig.
func (n NamedTargetConfig) AsTargetConfig() *TargetConfig {
return &TargetConfig{
Method: n.Method,
Fallback: n.Fallback,
}
}

35
internal/encryption/config/config_load.go Normal file
View File

@ -0,0 +1,35 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package config
import (
"strings"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/hclsyntax"
"github.com/hashicorp/hcl/v2/json"
)
// LoadConfigFromString loads a configuration from a string. The sourceName is used to identify the source of the
// configuration in error messages.
// This method serves as an example for how someone using this library might want to load a configuration.
// if they were not using gohcl directly.
// However! Right now, this method should only be used in tests, as OpenTofu should be using gohcl to parse the configuration.
func LoadConfigFromString(sourceName string, rawInput string) (*Config, hcl.Diagnostics) {
var diags hcl.Diagnostics
var file *hcl.File
if strings.TrimSpace(rawInput)[0] == '{' {
file, diags = json.Parse([]byte(rawInput), sourceName)
} else {
file, diags = hclsyntax.ParseConfig([]byte(rawInput), sourceName, hcl.Pos{Byte: 0, Line: 1, Column: 1})
}
cfg, cfgDiags := DecodeConfig(file.Body, hcl.Range{Filename: sourceName})
diags = append(diags, cfgDiags...)
return cfg, diags
}

166
internal/encryption/config/config_merge.go Normal file
View File

@ -0,0 +1,166 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package config
import (
"github.com/hashicorp/hcl/v2"
"github.com/opentofu/opentofu/internal/configs"
)
// MergeConfigs merges two Configs together, with the override taking precedence.
func MergeConfigs(cfg *Config, override *Config) *Config {
return &Config{
KeyProviderConfigs: mergeKeyProviderConfigs(cfg.KeyProviderConfigs, override.KeyProviderConfigs),
MethodConfigs: mergeMethodConfigs(cfg.MethodConfigs, override.MethodConfigs),
StateFile: mergeEnforcableTargetConfigs(cfg.StateFile, override.StateFile),
PlanFile: mergeEnforcableTargetConfigs(cfg.PlanFile, override.PlanFile),
Backend: mergeEnforcableTargetConfigs(cfg.Backend, override.Backend),
Remote: mergeRemoteConfigs(cfg.Remote, override.Remote),
}
}
func mergeMethodConfigs(configs []MethodConfig, overrides []MethodConfig) []MethodConfig {
// Initialize a copy of configs to preserve the original entries.
merged := make([]MethodConfig, len(configs))
copy(merged, configs)
for _, override := range overrides {
wasOverridden := false
// Attempt to find a match based on type/name
for i, method := range merged {
if method.Type == override.Type && method.Name == override.Name {
// Override the existing method.
merged[i].Body = mergeBody(method.Body, override.Body)
wasOverridden = true
break
}
}
// If no existing method was overridden, append the new override.
if !wasOverridden {
merged = append(merged, override)
}
}
return merged
}
func mergeKeyProviderConfigs(configs []KeyProviderConfig, overrides []KeyProviderConfig) []KeyProviderConfig {
// Initialize a copy of configs to preserve the original entries.
merged := make([]KeyProviderConfig, len(configs))
copy(merged, configs)
for _, override := range overrides {
wasOverridden := false
// Attempt to find a match based on type/name
for i, keyProvider := range merged {
if keyProvider.Type == override.Type && keyProvider.Name == override.Name {
// Override the existing key provider.
merged[i].Body = mergeBody(keyProvider.Body, override.Body)
wasOverridden = true
break
}
}
// If no existing key provider was overridden, append the new override.
if !wasOverridden {
merged = append(merged, override)
}
}
return merged
}
func mergeTargetConfigs(cfg *TargetConfig, override *TargetConfig) *TargetConfig {
if cfg == nil {
return override
}
if override == nil {
return cfg
}
merged := &TargetConfig{}
if override.Method != nil {
merged.Method = override.Method
} else {
merged.Method = cfg.Method
}
if override.Fallback != nil {
merged.Fallback = override.Fallback
} else {
merged.Fallback = cfg.Fallback
}
return merged
}
func mergeEnforcableTargetConfigs(cfg *EnforcableTargetConfig, override *EnforcableTargetConfig) *EnforcableTargetConfig {
if cfg == nil {
return override
}
if override == nil {
return cfg
}
mergeTarget := mergeTargetConfigs(cfg.AsTargetConfig(), override.AsTargetConfig())
return &EnforcableTargetConfig{
Enforced: cfg.Enforced || override.Enforced,
Method: mergeTarget.Method,
Fallback: mergeTarget.Fallback,
}
}
func mergeRemoteConfigs(cfg *RemoteConfig, override *RemoteConfig) *RemoteConfig {
if cfg == nil {
return override
}
if override == nil {
return cfg
}
merged := &RemoteConfig{
Default: mergeTargetConfigs(cfg.Default, override.Default),
Targets: make([]NamedTargetConfig, len(cfg.Targets)),
}
copy(merged.Targets, cfg.Targets)
for _, overrideTarget := range override.Targets {
found := false
for i, t := range merged.Targets {
found = t.Name == overrideTarget.Name
if found {
// gohcl does not support struct embedding
mergeTarget := mergeTargetConfigs(t.AsTargetConfig(), overrideTarget.AsTargetConfig())
merged.Targets[i] = NamedTargetConfig{
Name: t.Name,
Method: mergeTarget.Method,
Fallback: mergeTarget.Fallback,
}
break
}
}
if !found {
merged.Targets = append(merged.Targets, overrideTarget)
}
}
return merged
}
func mergeBody(base hcl.Body, override hcl.Body) hcl.Body {
if base == nil {
return override
}
if override == nil {
return base
}
return configs.MergeBodies(base, override)
}

335
internal/encryption/config/config_merge_test.go Normal file
View File

@ -0,0 +1,335 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package config
import (
"reflect"
"testing"
"github.com/davecgh/go-spew/spew"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/hcltest"
"github.com/opentofu/opentofu/internal/configs"
"github.com/zclconf/go-cty/cty"
)
func TestMergeMethodConfigs(t *testing.T) {
makeMethodConfig := func(typeName, name, key, value string) MethodConfig {
return MethodConfig{
Type: typeName,
Name: name,
Body: configs.SynthBody("method", map[string]cty.Value{
key: cty.StringVal(value),
}),
}
}
schema := &hcl.BodySchema{Attributes: []hcl.AttributeSchema{{Name: "key"}}}
tests := []struct {
name string
configSchema *hcl.BodySchema
input []MethodConfig
override []MethodConfig
expected []MethodConfig
}{
{
name: "empty",
configSchema: nil,
input: []MethodConfig{},
override: []MethodConfig{},
expected: []MethodConfig{},
},
{
name: "override one method config body",
configSchema: schema,
input: []MethodConfig{
makeMethodConfig("type", "name", "key", "value"),
},
override: []MethodConfig{
makeMethodConfig("type", "name", "key", "override"),
},
expected: []MethodConfig{
makeMethodConfig("type", "name", "key", "override"),
},
},
{
name: "initial config is empty",
configSchema: schema,
input: []MethodConfig{},
override: []MethodConfig{
makeMethodConfig("type", "name", "key", "override"),
},
expected: []MethodConfig{
makeMethodConfig("type", "name", "key", "override"),
},
},
{
name: "override multiple method configs",
configSchema: schema,
input: []MethodConfig{
makeMethodConfig("type", "name", "key", "value"),
makeMethodConfig("type", "name2", "key", "value"),
makeMethodConfig("type", "name3", "key", "value"),
},
override: []MethodConfig{
makeMethodConfig("type", "name", "key", "override1"),
makeMethodConfig("type", "name2", "key", "override2"),
},
expected: []MethodConfig{
makeMethodConfig("type", "name", "key", "override1"),
makeMethodConfig("type", "name2", "key", "override2"),
makeMethodConfig("type", "name3", "key", "value"),
},
},
{
name: "override config is empty",
configSchema: schema,
input: []MethodConfig{
makeMethodConfig("type", "name", "key", "value"),
},
override: []MethodConfig{},
expected: []MethodConfig{
makeMethodConfig("type", "name", "key", "value"),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
output := mergeMethodConfigs(test.input, test.override)
// for each of the expected methods, check if it exists in the output
for _, expectedMethod := range test.expected {
found := false
for _, method := range output {
if method.Type == expectedMethod.Type && method.Name == expectedMethod.Name {
found = true
expectedContent, _ := expectedMethod.Body.Content(test.configSchema)
actualContent, diags := method.Body.Content(test.configSchema)
if diags.HasErrors() {
t.Fatalf("unexpected diagnostics: %v", diags)
}
// Only compare the attributes here, so that we don't look at things like the MissingItemRange on the hcl.Body
if !reflect.DeepEqual(expectedContent.Attributes, actualContent.Attributes) {
t.Errorf("expected %v, got %v", spew.Sdump(expectedContent.Attributes), spew.Sdump(actualContent.Attributes))
}
}
}
if !found {
t.Errorf("expected method %v not found in output", spew.Sdump(expectedMethod))
}
}
})
}
}
func TestMergeKeyProviderConfigs(t *testing.T) {
makeKeyProviderConfig := func(typeName, name, key, value string) KeyProviderConfig {
return KeyProviderConfig{
Type: typeName,
Name: name,
Body: configs.SynthBody("key_provider", map[string]cty.Value{
key: cty.StringVal(value),
}),
}
}
schema := &hcl.BodySchema{Attributes: []hcl.AttributeSchema{{Name: "key"}}}
tests := []struct {
name string
configSchema *hcl.BodySchema
input []KeyProviderConfig
override []KeyProviderConfig
expected []KeyProviderConfig
}{
{
name: "empty",
configSchema: nil,
input: []KeyProviderConfig{},
override: []KeyProviderConfig{},
expected: []KeyProviderConfig{},
},
{
name: "override one key provider config body",
configSchema: schema,
input: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "value"),
},
override: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override"),
},
expected: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override"),
},
},
{
name: "initial config is empty",
configSchema: schema,
input: []KeyProviderConfig{},
override: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override"),
},
expected: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override"),
},
},
{
name: "override multiple key provider configs",
configSchema: schema,
input: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "value"),
makeKeyProviderConfig("type", "name2", "key", "value"),
},
override: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override1"),
makeKeyProviderConfig("type", "name2", "key", "override2"),
},
expected: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "override1"),
makeKeyProviderConfig("type", "name2", "key", "override2"),
},
},
{
name: "override config is empty",
configSchema: schema,
input: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "value"),
},
override: []KeyProviderConfig{},
expected: []KeyProviderConfig{
makeKeyProviderConfig("type", "name", "key", "value"),
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
output := mergeKeyProviderConfigs(test.input, test.override)
// for each of the expected key providers, check if it exists in the output
for _, expectedKeyProvider := range test.expected {
found := false
for _, keyProvider := range output {
if keyProvider.Type == expectedKeyProvider.Type && keyProvider.Name == expectedKeyProvider.Name {
found = true
expectedContent, _ := expectedKeyProvider.Body.Content(test.configSchema)
actualContent, diags := keyProvider.Body.Content(test.configSchema)
if diags.HasErrors() {
t.Fatalf("unexpected diagnostics: %v", diags)
}
// Only compare the attributes here, so that we don't look at things like the MissingItemRange on the hcl.Body
if !reflect.DeepEqual(expectedContent.Attributes, actualContent.Attributes) {
t.Errorf("expected %v, got %v", spew.Sdump(expectedContent.Attributes), spew.Sdump(actualContent.Attributes))
}
}
}
if !found {
t.Errorf("expected key provider %v not found in output", spew.Sdump(expectedKeyProvider))
}
}
})
}
}
func TestMergeTargetConfigs(t *testing.T) {
makeTargetConfig := func(enforced bool, method hcl.Expression, fallback *TargetConfig) *TargetConfig {
return &TargetConfig{
Method: method,
Fallback: fallback,
}
}
makeEnforcableTargetConfig := func(enforced bool, method hcl.Expression, fallback *TargetConfig) *EnforcableTargetConfig {
return &EnforcableTargetConfig{
Enforced: enforced,
Method: method,
Fallback: fallback,
}
}
expressionOne := hcltest.MockExprLiteral(cty.UnknownVal(cty.Set(cty.String)))
expressionTwo := hcltest.MockExprLiteral(cty.UnknownVal(cty.Set(cty.Bool)))
tests := []struct {
name string
input *EnforcableTargetConfig
override *EnforcableTargetConfig
expected *EnforcableTargetConfig
}{
{
name: "both nil",
input: nil,
override: nil,
expected: nil,
},
{
name: "input is nil",
input: nil,
override: makeEnforcableTargetConfig(true, expressionOne, nil),
expected: makeEnforcableTargetConfig(true, expressionOne, nil),
},
{
name: "override is nil",
input: makeEnforcableTargetConfig(true, expressionOne, nil),
override: nil,
expected: makeEnforcableTargetConfig(true, expressionOne, nil),
},
{
name: "override target config method",
input: makeEnforcableTargetConfig(true, expressionOne, nil),
override: makeEnforcableTargetConfig(true, expressionTwo, nil),
expected: makeEnforcableTargetConfig(true, expressionTwo, nil),
},
{
name: "override target config fallback",
input: makeEnforcableTargetConfig(true, expressionOne, makeTargetConfig(true, expressionOne, nil)),
override: makeEnforcableTargetConfig(true, expressionOne, makeTargetConfig(true, expressionTwo, nil)),
expected: makeEnforcableTargetConfig(true, expressionOne, makeTargetConfig(true, expressionTwo, nil)),
},
{
name: "override target config fallback",
input: makeEnforcableTargetConfig(true, expressionOne, nil),
override: makeEnforcableTargetConfig(true, expressionOne, makeTargetConfig(true, expressionTwo, nil)),
expected: makeEnforcableTargetConfig(true, expressionOne, makeTargetConfig(true, expressionTwo, nil)),
},
{
name: "override target config enforced - should be true if any are true",
input: makeEnforcableTargetConfig(true, expressionOne, nil),
override: makeEnforcableTargetConfig(false, expressionOne, nil),
expected: makeEnforcableTargetConfig(true, expressionOne, nil),
},
{
name: "override target config enforced - should be true if any are true",
input: makeEnforcableTargetConfig(false, expressionOne, nil),
override: makeEnforcableTargetConfig(true, expressionOne, nil),
expected: makeEnforcableTargetConfig(true, expressionOne, nil),
},
{
name: "override target config enforced - should be false if both are false",
input: makeEnforcableTargetConfig(false, expressionOne, nil),
override: makeEnforcableTargetConfig(false, expressionOne, nil),
expected: makeEnforcableTargetConfig(false, expressionOne, nil),
},
{
name: "override enforced, method and fallback",
input: makeEnforcableTargetConfig(false, expressionOne, makeTargetConfig(true, expressionOne, nil)),
override: makeEnforcableTargetConfig(true, expressionTwo, makeTargetConfig(true, expressionTwo, nil)),
expected: makeEnforcableTargetConfig(true, expressionTwo, makeTargetConfig(true, expressionTwo, nil)),
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
output := mergeEnforcableTargetConfigs(test.input, test.override)
if !reflect.DeepEqual(output, test.expected) {
t.Errorf("expected %v, got %v", spew.Sdump(test.expected), spew.Sdump(output))
}
})
}
}

76
internal/encryption/config/config_parse.go Normal file
View File

@ -0,0 +1,76 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package config
import (
"fmt"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
)
// DecodeConfig takes a hcl.Body and decodes it into a Config struct.
// This method is here as an example for how someone using this library might want to decode a configuration.
// if they were not using gohcl directly.
// Right now for real world use this is only intended to be used in tests, until we publish this publicly.
func DecodeConfig(body hcl.Body, rng hcl.Range) (*Config, hcl.Diagnostics) {
cfg := &Config{}
diags := gohcl.DecodeBody(body, nil, cfg)
if diags.HasErrors() {
return nil, diags
}
for i, kp := range cfg.KeyProviderConfigs {
for j, okp := range cfg.KeyProviderConfigs {
if i != j && kp.Type == okp.Type && kp.Name == okp.Name {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Duplicate key_provider",
Detail: fmt.Sprintf("Found multiple instances of key_provider.%s.%s", kp.Type, kp.Name),
Subject: rng.Ptr(),
})
break
}
}
}
for i, m := range cfg.MethodConfigs {
for j, om := range cfg.MethodConfigs {
if i != j && m.Type == om.Type && m.Name == om.Name {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Duplicate method",
Detail: fmt.Sprintf("Found multiple instances of method.%s.%s", m.Type, m.Name),
Subject: rng.Ptr(),
})
break
}
}
}
if cfg.Remote != nil {
for i, t := range cfg.Remote.Targets {
for j, ot := range cfg.Remote.Targets {
if i != j && t.Name == ot.Name {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Duplicate remote_data_source",
Detail: fmt.Sprintf("Found multiple instances of remote_data_source.%s", t.Name),
Subject: rng.Ptr(),
})
break
}
}
}
}
if diags.HasErrors() {
return nil, diags
}
return cfg, diags
}

74
internal/encryption/encryption.go Normal file
View File

@ -0,0 +1,74 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/registry"
)
// Encryption contains the methods for obtaining a StateEncryption or PlanEncryption correctly configured for a specific
// purpose. If no encryption configuration is present, it should return a pass through method that doesn't do anything.
type Encryption interface {
// StateFile produces a StateEncryption overlay for encrypting and decrypting state files for local storage.
StateFile() StateEncryption
// PlanFile produces a PlanEncryption overlay for encrypting and decrypting plan files.
PlanFile() PlanEncryption
// Backend produces a StateEncryption overlay for storing state files on remote backends, such as an S3 bucket.
Backend() StateEncryption
// RemoteState produces a ReadOnlyStateEncryption for reading remote states using the terraform_remote_state data
// source.
RemoteState(string) ReadOnlyStateEncryption
}
type encryption struct {
// Inputs
cfg *config.Config
reg registry.Registry
}
// New creates a new Encryption provider from the given configuration and registry.
func New(reg registry.Registry, cfg *config.Config) Encryption {
return &encryption{
cfg: cfg,
reg: reg,
}
}
func (e *encryption) StateFile() StateEncryption {
return &stateEncryption{
base: newBaseEncryption(e, e.cfg.StateFile.AsTargetConfig(), e.cfg.StateFile.Enforced, "statefile"),
}
}
func (e *encryption) PlanFile() PlanEncryption {
return &planEncryption{
base: newBaseEncryption(e, e.cfg.PlanFile.AsTargetConfig(), e.cfg.PlanFile.Enforced, "planfile"),
}
}
func (e *encryption) Backend() StateEncryption {
return &stateEncryption{
base: newBaseEncryption(e, e.cfg.StateFile.AsTargetConfig(), e.cfg.StateFile.Enforced, "backend"),
}
}
func (e *encryption) RemoteState(name string) ReadOnlyStateEncryption {
for _, remoteTarget := range e.cfg.Remote.Targets {
if remoteTarget.Name == name {
return &stateEncryption{
// TODO the addr here should be generated in one place.
base: newBaseEncryption(e, remoteTarget.AsTargetConfig(), false, "remote.remote_state_datasource."+remoteTarget.Name),
}
}
}
return &stateEncryption{
base: newBaseEncryption(e, e.cfg.Remote.Default, false, "remote.default"),
}
}

96
internal/encryption/example_test.go Normal file
View File

@ -0,0 +1,96 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption_test
import (
"fmt"
"github.com/hashicorp/hcl/v2"
"github.com/opentofu/opentofu/internal/encryption"
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/keyprovider/static"
"github.com/opentofu/opentofu/internal/encryption/method/aesgcm"
"github.com/opentofu/opentofu/internal/encryption/registry/lockingencryptionregistry"
)
var (
ConfigA = `
backend {
enforced = true
}
`
ConfigB = `
key_provider "static" "basic" {
key = "6f6f706830656f67686f6834616872756f3751756165686565796f6f72653169"
}
method "aes_gcm" "example" {
cipher = key_provider.static.basic
}
statefile {
method = method.aes_gcm.example
}
backend {
method = method.aes_gcm.example
}
`
)
// This example demonstrates how to use the encryption package to encrypt and decrypt data.
func Example() {
// Construct a new registry
// the registry is where we store the key providers and methods
reg := lockingencryptionregistry.New()
if err := reg.RegisterKeyProvider(static.New()); err != nil {
panic(err)
}
if err := reg.RegisterMethod(aesgcm.New()); err != nil {
panic(err)
}
// Load the 2 different configurations
cfgA, diags := config.LoadConfigFromString("Test Source A", ConfigA)
handleDiags(diags)
cfgB, diags := config.LoadConfigFromString("Test Source B", ConfigB)
handleDiags(diags)
// Merge the configurations
cfg := config.MergeConfigs(cfgA, cfgB)
// Construct the encryption object
enc := encryption.New(reg, cfg)
// Encrypt the data, for this example we will be using the string "test",
// but in a real world scenario this would be the plan file.
sourceData := []byte("test")
encrypted, diags := enc.StateFile().EncryptState(sourceData)
handleDiags(diags)
if string(encrypted) == "test" {
panic("The data has not been encrypted!")
}
println(string(encrypted))
// Decrypt
decryptedState, err := enc.StateFile().DecryptState(encrypted)
if err != nil {
panic(err)
}
fmt.Printf("%s\n", decryptedState)
// Output: test
}
func handleDiags(diags hcl.Diagnostics) {
for _, d := range diags {
println(d.Error())
}
if diags.HasErrors() {
panic(diags.Error())
}
}

197
internal/encryption/keyprovider.go Normal file
View File

@ -0,0 +1,197 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"errors"
"fmt"
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/registry"
"github.com/opentofu/opentofu/internal/varhcl"
"github.com/zclconf/go-cty/cty"
)
// setupKeyProviders sets up the key providers for encryption. It returns a list of diagnostics if any of the key providers
// are invalid.
func (e *targetBuilder) setupKeyProviders() hcl.Diagnostics {
var diags hcl.Diagnostics
e.keyValues = make(map[string]map[string]cty.Value)
for _, keyProviderConfig := range e.cfg.KeyProviderConfigs {
diags = append(diags, e.setupKeyProvider(keyProviderConfig, nil)...)
}
// Regenerate the context now that the key provider is loaded
kpMap := make(map[string]cty.Value)
for name, kps := range e.keyValues {
kpMap[name] = cty.ObjectVal(kps)
}
e.ctx.Variables["key_provider"] = cty.ObjectVal(kpMap)
return diags
}
func (e *targetBuilder) setupKeyProvider(cfg config.KeyProviderConfig, stack []config.KeyProviderConfig) hcl.Diagnostics {
// Ensure cfg.Type is in keyValues, if it isn't then add it in preparation for the next step
if _, ok := e.keyValues[cfg.Type]; !ok {
e.keyValues[cfg.Type] = make(map[string]cty.Value)
}
// Check if we have already setup this Descriptor (due to dependency loading)
// if we've already setup this key provider, then we don't need to do it again
// and we can return early
if _, ok := e.keyValues[cfg.Type][cfg.Name]; ok {
return nil
}
// Mark this key provider as partially handled. This value will be replaced below once it is actually known.
// The goal is to allow an early return via the above if statement to prevent duplicate errors if errors are encoutered in the key loading stack.
e.keyValues[cfg.Type][cfg.Name] = cty.UnknownVal(cty.DynamicPseudoType)
// Check for circular references, this is done by inspecting the stack of key providers
// that are currently being setup. If we find a key provider in the stack that matches
// the current key provider, then we have a circular reference and we should return an error
// to the user.
for _, s := range stack {
if s == cfg {
addr, diags := keyprovider.NewAddr(cfg.Type, cfg.Name)
diags = diags.Append(
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Circular reference detected",
// TODO add the stack trace to the detail message
Detail: fmt.Sprintf("Can not load %q due to circular reference", addr),
},
)
return diags
}
}
stack = append(stack, cfg)
// Pull the meta key out for error messages and meta storage
metakey, diags := cfg.Addr()
if diags.HasErrors() {
return diags
}
// Lookup the KeyProviderDescriptor from the registry
id := keyprovider.ID(cfg.Type)
keyProviderDescriptor, err := e.reg.GetKeyProviderDescriptor(id)
if err != nil {
if errors.Is(err, &registry.KeyProviderNotFoundError{}) {
return hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unknown key_provider type",
Detail: fmt.Sprintf("Can not find %q", cfg.Type),
}}
}
return hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: fmt.Sprintf("Error fetching key_provider %q", cfg.Type),
Detail: err.Error(),
}}
}
// Now that we know we have the correct Descriptor, we can decode the configuration
// and build the KeyProvider
keyProviderConfig := keyProviderDescriptor.ConfigStruct()
// Locate all the dependencies
deps, diags := varhcl.VariablesInBody(cfg.Body, keyProviderConfig)
if diags.HasErrors() {
return diags
}
// Required Dependencies
for _, dep := range deps {
// Key Provider references should be in the form key_provider.type.name
if len(dep) != 3 {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid key_provider reference",
Detail: "Expected reference in form key_provider.type.name",
Subject: dep.SourceRange().Ptr(),
})
continue
}
// TODO this should be more defensive
depRoot := (dep[0].(hcl.TraverseRoot)).Name
depType := (dep[1].(hcl.TraverseAttr)).Name
depName := (dep[2].(hcl.TraverseAttr)).Name
if depRoot != "key_provider" {
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid key_provider reference",
Detail: "Expected reference in form key_provider.type.name",
Subject: dep.SourceRange().Ptr(),
})
continue
}
for _, kpc := range e.cfg.KeyProviderConfigs {
// Find the key provider in the config
if kpc.Type == depType && kpc.Name == depName {
depDiags := e.setupKeyProvider(kpc, stack)
diags = append(diags, depDiags...)
break
}
}
}
if diags.HasErrors() {
// We should not continue now if we have any diagnostics that are errors
// as we may end up in an inconsistent state.
// The reason we collate the diags here and then show them instead of showing them as they arise
// is to ensure that the end user does not have to play whack-a-mole with the errors one at a time.
return diags
}
// Initialize the Key Provider
decodeDiags := gohcl.DecodeBody(cfg.Body, e.ctx, keyProviderConfig)
diags = append(diags, decodeDiags...)
if diags.HasErrors() {
return diags
}
// Build the Key Provider from the configuration
keyProvider, err := keyProviderConfig.Build()
if err != nil {
return append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unable to build encryption key data",
Detail: fmt.Sprintf("%s failed with error: %s", metakey, err.Error()),
})
}
meta := e.keyProviderMetadata[metakey]
data, newmeta, err := keyProvider.Provide(meta)
if err != nil {
return append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unable to fetch encryption key data",
Detail: fmt.Sprintf("%s failed with error: %s", metakey, err.Error()),
})
}
e.keyProviderMetadata[metakey] = newmeta
// Convert the data into it's cty equivalent
ctyData := make([]cty.Value, len(data))
for i, d := range data {
ctyData[i] = cty.NumberIntVal(int64(d))
}
e.keyValues[cfg.Type][cfg.Name] = cty.ListVal(ctyData)
return nil
}

73
internal/encryption/keyprovider/addr.go Normal file
View File

@ -0,0 +1,73 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package keyprovider
import (
"fmt"
"regexp"
"github.com/hashicorp/hcl/v2"
)
// TODO is there a generalized way to regexp-check names?
var addrRe = regexp.MustCompile(`^key_provider\.([a-zA-Z_0-9-]+)\.([a-zA-Z_0-9-]+)$`)
var nameRe = regexp.MustCompile("^([a-zA-Z_0-9-]+)$")
// Addr is a type-alias for key provider address strings that identify a specific key provider configuration.
// The Addr is an opaque value. Do not perform string manipulation on it outside the functions supplied by the
// keyprovider package.
type Addr string
// Validate validates the Addr for formal naming conformance, but does not check if the referenced key provider actually
// exists in the configuration.
func (a Addr) Validate() hcl.Diagnostics {
if !addrRe.MatchString(string(a)) {
return hcl.Diagnostics{
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid key provider address",
Detail: fmt.Sprintf(
"The supplied key provider address does not match the required form of %s",
addrRe.String(),
),
},
}
}
return nil
}
// NewAddr creates a new Addr type from the provider and name supplied. The Addr is a type-alias for key provider
// address strings that identify a specific key provider configuration. You should treat the value as opaque and not
// perform string manipulation on it outside the functions supplied by the keyprovider package.
func NewAddr(provider string, name string) (addr Addr, err hcl.Diagnostics) {
if !nameRe.MatchString(provider) {
err = err.Append(
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "The provided key provider type is invalid",
Detail: fmt.Sprintf(
"The supplied key provider type (%s) does not match the required form of %s.",
provider,
nameRe.String(),
),
},
)
}
if !nameRe.MatchString(name) {
err = err.Append(
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "The provided key provider name is invalid",
Detail: fmt.Sprintf(
"The supplied key provider name (%s) does not match the required form of %s.",
name,
nameRe.String(),
),
},
)
}
return Addr(fmt.Sprintf("key_provider.%s.%s", provider, name)), err
}

15
internal/encryption/keyprovider/id.go Normal file
View File

@ -0,0 +1,15 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package keyprovider
// ID is a type alias to make passing the wrong ID into a key provider harder.
type ID string
// Validate validates the key provider ID for correctness.
func (i ID) Validate() error {
// TODO implement format checking
return nil
}

28
internal/encryption/keyprovider/keyprovider.go Normal file
View File

@ -0,0 +1,28 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package keyprovider
type Config interface {
Build() (KeyProvider, error)
}
type Descriptor interface {
// ID returns the unique identifier used when parsing HCL or JSON configs.
ID() ID
// ConfigStruct creates a new configuration struct pointer annotated with hcl tags. The Build() receiver on
// this struct must be able to build a KeyProvider from the configuration:
//
// Common errors:
// - Returning a struct without a pointer
// - Returning a non-struct
ConfigStruct() Config
}
type KeyProvider interface {
// Provide provides an encryption key. If the process fails, it returns an error.
Provide(metadata []byte) ([]byte, []byte, error)
}

25
internal/encryption/keyprovider/static/config.go Normal file
View File

@ -0,0 +1,25 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package static
import (
"encoding/hex"
"fmt"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
)
type Config struct {
Key string `hcl:"key"`
}
func (c Config) Build() (keyprovider.KeyProvider, error) {
decodedData, err := hex.DecodeString(c.Key)
if err != nil {
return nil, fmt.Errorf("failed to hex-decode the provided key (%w)", err)
}
return &staticKeyProvider{decodedData}, nil
}

25
internal/encryption/keyprovider/static/descriptor.go Normal file
View File

@ -0,0 +1,25 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package static
import (
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
)
func New() keyprovider.Descriptor {
return &descriptor{}
}
type descriptor struct {
}
func (f descriptor) ID() keyprovider.ID {
return "static"
}
func (f descriptor) ConfigStruct() keyprovider.Config {
return &Config{}
}

73
internal/encryption/keyprovider/static/example_test.go Normal file
View File

@ -0,0 +1,73 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package static_test
import (
"fmt"
"github.com/hashicorp/hcl/v2/gohcl"
config2 "github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/keyprovider/static"
)
var config = `key_provider "static" "foo" {
key = "6f6f706830656f67686f6834616872756f3751756165686565796f6f72653169"
}
method "aes_gcm" "foo" {
cipher = key_provider.static.foo
}
planfile {
method = method.aes_gcm.foo
}
`
// This example is a bare-bones configuration for a static key provider.
// It is mainly intended to demonstrate how you can use parse configuration
// and construct a static key provider from in.
// And is not intended to be used as a real-world example.
func Example() {
// TODO: Rename from ConfigStruct
staticConfig := static.New().ConfigStruct()
// Parse the config:
parsedConfig, diags := config2.LoadConfigFromString("config.hcl", config)
if diags.HasErrors() {
panic(diags)
}
// TODO: Rename KeyProviderConfigs to KeyProviderConfigs
if len(parsedConfig.KeyProviderConfigs) != 1 {
panic("Expected 1 key provider")
}
// Grab the KeyProvider from the parsed config:
keyProvider := parsedConfig.KeyProviderConfigs[0]
// assert the Type is "static" and the Name is "foo"
if keyProvider.Type != "static" {
panic("Expected key provider type to be 'static'")
}
if keyProvider.Name != "foo" {
panic("Expected key provider name to be 'foo'")
}
// Use gohcl to parse the hcl block from parsedConfig into the static configuration struct
// This is not the intended path and it should be handled by the implementation of the Encryption
// interface
// This is just an example of how to use the static configuration struct, and this is how testing
// may be carried out.
if err := gohcl.DecodeBody(parsedConfig.KeyProviderConfigs[0].Body, nil, staticConfig); err != nil {
panic(err)
}
// Cast the static configuration struct to a static.Config so that we can assert against the key
// value
s := staticConfig.(*static.Config)
fmt.Printf("%s\n", s.Key)
// Output: 6f6f706830656f67686f6834616872756f3751756165686565796f6f72653169
}

19
internal/encryption/keyprovider/static/provider.go Normal file
View File

@ -0,0 +1,19 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
// Package static contains a key provider that emits a static key.
package static
type staticKeyProvider struct {
key []byte
}
func (p staticKeyProvider) Provide(metadata []byte) ([]byte, []byte, error) {
if metadata == nil {
metadata = []byte("magic")
}
return p.key, metadata, nil
}

78
internal/encryption/keyprovider/static/provider_test.go Normal file
View File

@ -0,0 +1,78 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package static_test
import (
"testing"
"github.com/opentofu/opentofu/internal/encryption/keyprovider/static"
)
func TestKeyProvider(t *testing.T) {
// TODO: Rework to check the expected errors and not just expectSuccess
type testCase struct {
name string
key string
expectSuccess bool
expectedData string // The key as a string taken from the hex value of the key
expectedMeta string
}
testCases := []testCase{
{
name: "Empty",
expectSuccess: true,
expectedData: "",
expectedMeta: "magic", // We currently always output the metadata "magic"
},
{
name: "InvalidInput",
key: "G",
expectSuccess: false,
},
{
name: "Success",
key: "48656c6c6f20776f726c6421",
expectSuccess: true,
expectedData: "Hello world!", // "48656c6c6f20776f726c6421" in hex is "Hello world!"
expectedMeta: "magic", // We currently always output the metadata "magic"
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
descriptor := static.New()
c := descriptor.ConfigStruct().(*static.Config)
// Set key if provided
if tc.key != "" {
c.Key = tc.key
}
keyProvider, buildErr := c.Build()
if tc.expectSuccess {
if buildErr != nil {
t.Fatalf("unexpected error: %v", buildErr)
}
data, newMetadata, err := keyProvider.Provide(nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if string(data) != tc.expectedData {
t.Fatalf("unexpected key output: got %v, want %v", data, tc.expectedData)
}
if string(newMetadata) != tc.expectedMeta {
t.Fatalf("unexpected metadata: got %v, want %v", newMetadata, tc.expectedMeta)
}
} else {
if buildErr == nil {
t.Fatalf("expected an error but got none")
}
}
})
}
}

73
internal/encryption/method/addr.go Normal file
View File

@ -0,0 +1,73 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package method
import (
"fmt"
"regexp"
"github.com/hashicorp/hcl/v2"
)
// TODO is there a generalized way to regexp-check names?
var addrRe = regexp.MustCompile(`^method\.([a-zA-Z_0-9-]+)\.([a-zA-Z_0-9-]+)$`)
var nameRe = regexp.MustCompile("^([a-zA-Z_0-9-]+)$")
// Addr is a type-alias for method address strings that identify a specific encryption method configuration.
// The Addr is an opaque value. Do not perform string manipulation on it outside the functions supplied by the
// method package.
type Addr string
// Validate validates the Addr for formal naming conformance, but does not check if the referenced method actually
// exists in the configuration.
func (a Addr) Validate() hcl.Diagnostics {
if !addrRe.MatchString(string(a)) {
return hcl.Diagnostics{
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Invalid encryption method address",
Detail: fmt.Sprintf(
"The supplied encryption method address does not match the required form of %s",
addrRe.String(),
),
},
}
}
return nil
}
// NewAddr creates a new Addr type from the provider and name supplied. The Addr is a type-alias for encryption method
// address strings that identify a specific encryption method configuration. You should treat the value as opaque and
// not perform string manipulation on it outside the functions supplied by the method package.
func NewAddr(method string, name string) (addr Addr, err hcl.Diagnostics) {
if !nameRe.MatchString(method) {
err = err.Append(
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "The provided encryption method type is invalid",
Detail: fmt.Sprintf(
"The supplied encryption method type (%s) does not match the required form of %s.",
method,
nameRe.String(),
),
},
)
}
if !nameRe.MatchString(name) {
err = err.Append(
&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "The provided encryption method name is invalid",
Detail: fmt.Sprintf(
"The supplied encryption method name (%s) does not match the required form of %s.",
name,
nameRe.String(),
),
},
)
}
return Addr(fmt.Sprintf("method.%s.%s", method, name)), err
}

61
internal/encryption/method/aesgcm/aesgcm.go Normal file
View File

@ -0,0 +1,61 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package aesgcm
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
)
type aesgcm struct {
key []byte
}
// Inspired by https://bruinsslot.jp/post/golang-crypto/
func (a aesgcm) Encrypt(data []byte) ([]byte, error) {
// TODO change this to the implementation in Stephan's PR.
blockCipher, err := aes.NewCipher(a.key)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(blockCipher)
if err != nil {
return nil, err
}
nonce := make([]byte, gcm.NonceSize())
if _, err = rand.Read(nonce); err != nil {
return nil, err
}
ciphertext := gcm.Seal(nonce, nonce, data, nil)
return ciphertext, nil
}
func (a aesgcm) Decrypt(data []byte) ([]byte, error) {
blockCipher, err := aes.NewCipher(a.key)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(blockCipher)
if err != nil {
return nil, err
}
nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]
plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
if err != nil {
return nil, err
}
return plaintext, nil
}

25
internal/encryption/method/aesgcm/config.go Normal file
View File

@ -0,0 +1,25 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package aesgcm
import (
"fmt"
"github.com/opentofu/opentofu/internal/encryption/method"
)
type Config struct {
Key []byte `hcl:"cipher"`
}
func (c Config) Build() (method.Method, error) {
if len(c.Key) != 32 {
return nil, fmt.Errorf("AES-GCM requires a 32-byte key")
}
return &aesgcm{
c.Key,
}, nil
}

24
internal/encryption/method/aesgcm/descriptor.go Normal file
View File

@ -0,0 +1,24 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package aesgcm
import "github.com/opentofu/opentofu/internal/encryption/method"
// New creates a new descriptor for the AES-GCM encryption method, which requires a 32-byte key.
func New() method.Descriptor {
return &descriptor{}
}
type descriptor struct {
}
func (f *descriptor) ID() method.ID {
return "aes_gcm"
}
func (f *descriptor) ConfigStruct() method.Config {
return &Config{}
}

15
internal/encryption/method/id.go Normal file
View File

@ -0,0 +1,15 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package method
// ID is a type alias to make passing the wrong ID into a method ID harder.
type ID string
// Validate validates the key provider ID for correctness.
func (i ID) Validate() error {
// TODO implement format checking
return nil
}

37
internal/encryption/method/method.go Normal file
View File

@ -0,0 +1,37 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package method
type Config interface {
// Build takes the configuration and builds an encryption method.
Build() (Method, error)
}
type Descriptor interface {
// ID returns the unique identifier used when parsing HCL or JSON configs.
ID() ID
// ConfigStruct creates a new configuration struct annotated with hcl tags. The Build() receiver on
// this struct must be able to build a Method from the configuration.
//
// Common errors:
// - Returning a struct without a pointer
// - Returning a non-struct
ConfigStruct() Config
}
// Method is a low-level encryption method interface that is responsible for encrypting a binary blob of data. It should
// not try to interpret what kind of data it is encrypting.
type Method interface {
// Encrypt encrypts the specified data with the set configuration. This method should treat any data passed as
// opaque and should not try to interpret its contents. The interpretation is the job of the encryption.Encryption
// interface.
Encrypt(data []byte) ([]byte, error)
// Decrypt decrypts the specified data with the set configuration. This method should treat any data passed as
// opaque and should not try to interpret its contents. The interpretation is the job of the encryption.Encryption
// interface.
Decrypt(data []byte) ([]byte, error)
}

93
internal/encryption/methods.go Normal file
View File

@ -0,0 +1,93 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"errors"
"fmt"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/method"
"github.com/opentofu/opentofu/internal/encryption/registry"
"github.com/zclconf/go-cty/cty"
)
func (e *targetBuilder) setupMethods() hcl.Diagnostics {
var diags hcl.Diagnostics
e.methodValues = make(map[string]map[string]cty.Value)
e.methods = make(map[method.Addr]method.Method)
for _, m := range e.cfg.MethodConfigs {
diags = append(diags, e.setupMethod(m)...)
}
// Regenerate the context now that the method is loaded
mMap := make(map[string]cty.Value)
for name, ms := range e.methodValues {
mMap[name] = cty.ObjectVal(ms)
}
e.ctx.Variables["method"] = cty.ObjectVal(mMap)
return diags
}
// setupMethod sets up a single method for encryption. It returns a list of diagnostics if the method is invalid.
func (e *targetBuilder) setupMethod(cfg config.MethodConfig) hcl.Diagnostics {
addr, diags := cfg.Addr()
if diags.HasErrors() {
return diags
}
// Ensure cfg.Type is in methodValues
if _, ok := e.methodValues[cfg.Type]; !ok {
e.methodValues[cfg.Type] = make(map[string]cty.Value)
}
// Lookup the definition of the encryption method from the registry
encryptionMethod, err := e.reg.GetMethod(method.ID(cfg.Type))
if err != nil {
// Handle if the method was not found
var notFoundError *registry.MethodNotFoundError
if errors.Is(err, notFoundError) {
return hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Unknown encryption method type",
Detail: fmt.Sprintf("Can not find %q", cfg.Type),
}}
}
// Or, we don't know the error type, so we'll just return it as a generic error
return hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: fmt.Sprintf("Error fetching encryption method %q", cfg.Type),
Detail: err.Error(),
}}
}
// TODO: we could use varhcl here to provider better error messages
methodConfig := encryptionMethod.ConfigStruct()
diags = gohcl.DecodeBody(cfg.Body, e.ctx, methodConfig)
if diags.HasErrors() {
return diags
}
e.methodValues[cfg.Type][cfg.Name] = cty.StringVal(string(addr))
m, err := methodConfig.Build()
if err != nil {
// TODO this error handling could use some work
return hcl.Diagnostics{&hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Encryption method configuration failed",
Detail: err.Error(),
}}
}
e.methods[addr] = m
return nil
}

63
internal/encryption/plan.go Normal file
View File

@ -0,0 +1,63 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"fmt"
"github.com/hashicorp/hcl/v2"
)
// PlanEncryption describes the methods that you can use for encrypting a plan file. Plan files are opaque values with
// no standardized format, so the encrypted form should be treated equally an opaque value.
type PlanEncryption interface {
// EncryptPlan encrypts a plan file and returns the encrypted form.
//
// When implementing this function:
//
// Plan files are opaque values. You may expect a valid plan file on the input, but you can produce binary data
// that is not necessarily a valid plan file. If no encryption is configured, this function should pass through
// any data it receives without modification, even if the plan file is invalid.
//
// When using this function:
//
// Make sure that you pass a valid plan file as an input. Failing to provide a valid plan file may result in an
// error. However, output values may not be valid plan files and you should not pass the encrypted plan file to any
// additional functions that normally work with plan files.
EncryptPlan([]byte) ([]byte, hcl.Diagnostics)
// DecryptPlan decrypts an encrypted plan file.
//
// When implementing this function:
//
// If the user has configured no encryption, pass through any input unmodified regardless if the input is a valid
// plan file. If the user configured encryption, decrypt the plan file and return the decrypted plan file as a
// binary without further evaluating its validity.
//
// When using this function:
//
// Pass a potentially encrypted plan file as an input, and you will receive the decrypted plan file or an error as
// a result.
DecryptPlan([]byte) ([]byte, hcl.Diagnostics)
}
type planEncryption struct {
base *baseEncryption
}
func (p planEncryption) EncryptPlan(data []byte) ([]byte, hcl.Diagnostics) {
return p.base.encrypt(data)
}
func (p planEncryption) DecryptPlan(data []byte) ([]byte, hcl.Diagnostics) {
return p.base.decrypt(data, func(data []byte) error {
// Check magic bytes
if len(data) < 4 || string(data[:4]) != "PK" {
return fmt.Errorf("Invalid plan file")
}
return nil
})
}

89
internal/encryption/registry/errors.go Normal file
View File

@ -0,0 +1,89 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package registry
import (
"fmt"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
)
// InvalidKeyProviderError indicates that the supplied keyprovider.Descriptor is invalid/misbehaving. Check the error
// message for details.
type InvalidKeyProviderError struct {
KeyProvider keyprovider.Descriptor
Cause error
}
func (k InvalidKeyProviderError) Error() string {
return fmt.Sprintf("the supplied key provider %T is invalid (%v)", k.KeyProvider, k.Cause)
}
func (k InvalidKeyProviderError) Unwrap() error {
return k.Cause
}
// KeyProviderNotFoundError indicates that the requested key provider was not found in the registry.
type KeyProviderNotFoundError struct {
ID keyprovider.ID
}
func (k KeyProviderNotFoundError) Error() string {
return fmt.Sprintf("key provider with ID %s not found", k.ID)
}
// KeyProviderAlreadyRegisteredError indicates that the requested key provider was already registered in the registry.
type KeyProviderAlreadyRegisteredError struct {
ID keyprovider.ID
CurrentProvider keyprovider.Descriptor
PreviousProvider keyprovider.Descriptor
}
func (k KeyProviderAlreadyRegisteredError) Error() string {
return fmt.Sprintf(
"error while registering key provider ID %s to %T, this ID is already registered by %T",
k.ID, k.CurrentProvider, k.PreviousProvider,
)
}
// InvalidMethodError indicates that the supplied method.Descriptor is invalid/misbehaving. Check the error message for
// details.
type InvalidMethodError struct {
Method method.Descriptor
Cause error
}
func (k InvalidMethodError) Error() string {
return fmt.Sprintf("the supplied encryption method %T is invalid (%v)", k.Method, k.Cause)
}
func (k InvalidMethodError) Unwrap() error {
return k.Cause
}
// MethodNotFoundError indicates that the requested encryption method was not found in the registry.
type MethodNotFoundError struct {
ID method.ID
}
func (m MethodNotFoundError) Error() string {
return fmt.Sprintf("encryption method with ID %s not found", m.ID)
}
// MethodAlreadyRegisteredError indicates that the requested encryption method was already registered in the registry.
type MethodAlreadyRegisteredError struct {
ID method.ID
CurrentMethod method.Descriptor
PreviousMethod method.Descriptor
}
func (m MethodAlreadyRegisteredError) Error() string {
return fmt.Sprintf(
"error while registering encryption method ID %s to %T, this ID is already registered by %T",
m.ID, m.CurrentMethod, m.PreviousMethod,
)
}

79
internal/encryption/registry/lockingencryptionregistry/new.go Normal file
View File

@ -0,0 +1,79 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package lockingencryptionregistry
import (
"sync"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
"github.com/opentofu/opentofu/internal/encryption/registry"
)
// New returns a new encryption registry that locks for parallel access.
func New() registry.Registry {
return &lockingRegistry{
lock: &sync.RWMutex{},
providers: map[keyprovider.ID]keyprovider.Descriptor{},
methods: map[method.ID]method.Descriptor{},
}
}
type lockingRegistry struct {
lock *sync.RWMutex
providers map[keyprovider.ID]keyprovider.Descriptor
methods map[method.ID]method.Descriptor
}
func (l *lockingRegistry) RegisterKeyProvider(keyProvider keyprovider.Descriptor) error {
l.lock.Lock()
defer l.lock.Unlock()
id := keyProvider.ID()
if err := id.Validate(); err != nil {
return &registry.InvalidKeyProviderError{KeyProvider: keyProvider, Cause: err}
}
if _, ok := l.providers[id]; ok {
return &registry.KeyProviderAlreadyRegisteredError{ID: id}
}
l.providers[id] = keyProvider
return nil
}
func (l *lockingRegistry) RegisterMethod(method method.Descriptor) error {
l.lock.Lock()
defer l.lock.Unlock()
id := method.ID()
if err := id.Validate(); err != nil {
return &registry.InvalidMethodError{Method: method, Cause: err}
}
if previousMethod, ok := l.methods[id]; ok {
return &registry.MethodAlreadyRegisteredError{ID: id, CurrentMethod: method, PreviousMethod: previousMethod}
}
l.methods[id] = method
return nil
}
func (l *lockingRegistry) GetKeyProviderDescriptor(id keyprovider.ID) (keyprovider.Descriptor, error) {
l.lock.RLock()
defer l.lock.RUnlock()
provider, ok := l.providers[id]
if !ok {
return nil, &registry.KeyProviderNotFoundError{ID: id}
}
return provider, nil
}
func (l *lockingRegistry) GetMethod(id method.ID) (method.Descriptor, error) {
l.lock.RLock()
defer l.lock.RUnlock()
foundMethod, ok := l.methods[id]
if !ok {
return nil, &registry.MethodNotFoundError{ID: id}
}
return foundMethod, nil
}

31
internal/encryption/registry/registry.go Normal file
View File

@ -0,0 +1,31 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package registry
import (
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
)
// Registry collects all encryption methods and key providers
type Registry interface {
// RegisterKeyProvider registers a key provider. Use the keyprovider.Any().
// This function returns a *KeyProviderAlreadyRegisteredError error if a key provider with the
// same ID is already registered.
RegisterKeyProvider(keyProvider keyprovider.Descriptor) error
// RegisterMethod registers an encryption method. Use the method.Any() function to convert your method into a
// suitable format. This function returns a *MethodAlreadyRegisteredError error if a key provider with the same ID is
// already registered.
RegisterMethod(method method.Descriptor) error
// GetKeyProvider returns the key provider with the specified ID. If the key provider is not registered,
// it will return a *KeyProviderNotFoundError error.
GetKeyProviderDescriptor(id keyprovider.ID) (keyprovider.Descriptor, error)
// GetMethod returns the method with the specified ID.
// If the method is not registered, it will return a *MethodNotFoundError.
GetMethod(id method.ID) (method.Descriptor, error)
}

86
internal/encryption/state.go Normal file
View File

@ -0,0 +1,86 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"encoding/json"
"fmt"
"github.com/hashicorp/hcl/v2"
)
const StateEncryptionMarkerField = "encryption"
// ReadOnlyStateEncryption is an encryption layer for reading encrypted state files.
type ReadOnlyStateEncryption interface {
// DecryptState decrypts a potentially encrypted state file and returns a valid JSON-serialized state file.
//
// When implementing this function:
//
// If the user configured no encryption, also return the input as-is regardless if the state file is valid. If the
// user configured encryption unserialize the input as JSON and check for the presence of the field specified in the
// StateEncryptionMarkerField. If the field is not present, return the input as-is and return a warning that an
// unexpected unencrypted state file was encountered. Otherwise, decrypt the state file and return the decrypted
// state file as serialized JSON. If the state file cannot be decrypted, return an error in the diagnostics.
//
// When using this function:
//
// After reading the state file from its source (local file, remote backend, etc.), pass in the state file to this
// function. Do not attempt to determine if the state file is encrypted as this function will take care of any
// and all encryption-related matters. After the function returns, use the returned byte array as a normal state
// file.
DecryptState([]byte) ([]byte, hcl.Diagnostics)
}
// StateEncryption describes the interface for encrypting state files.
type StateEncryption interface {
ReadOnlyStateEncryption
// EncryptState encrypts a state file and returns the encrypted form.
//
// When implementing this function:
//
// The file should take a JSON-serialized state file as an input and encrypt it according to the configuration.
// The encrypted form should also return a JSON which contains, at least, the key specified in
// StateEncryptionMarkerField to identify the state file as encrypted. This is necessary because some backends
// expect a state file to always be a JSON file.
//
// If the user configured no encryption, this function should be a no-op and return the input unchanged. If the
// input is not a valid state file, this function should return an error in the diagnostics return.
//
// When using this function:
//
// Pass in a valid JSON-serialized state file as an input and store the output. Note that you should not pass the
// output to any additional functions that require a valid state file as it may not contain the fields typically
// present in a state file.
EncryptState([]byte) ([]byte, hcl.Diagnostics)
}
type stateEncryption struct {
base *baseEncryption
}
func (s *stateEncryption) EncryptState(plainState []byte) ([]byte, hcl.Diagnostics) {
return s.base.encrypt(plainState)
}
func (s *stateEncryption) DecryptState(encryptedState []byte) ([]byte, hcl.Diagnostics) {
return s.base.decrypt(encryptedState, func(data []byte) error {
tmp := struct {
FormatVersion string `json:"format_version"`
}{}
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
if len(tmp.FormatVersion) == 0 {
// Not a state file
return fmt.Errorf("Given payload is not a state file")
}
// Probably a state file
return nil
})
}

104
internal/encryption/targets.go Normal file
View File

@ -0,0 +1,104 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package encryption
import (
"fmt"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/opentofu/opentofu/internal/encryption/config"
"github.com/opentofu/opentofu/internal/encryption/keyprovider"
"github.com/opentofu/opentofu/internal/encryption/method"
"github.com/opentofu/opentofu/internal/encryption/registry"
"github.com/zclconf/go-cty/cty"
)
type targetBuilder struct {
cfg *config.Config
reg registry.Registry
// Used to evaluate hcl expressions
ctx *hcl.EvalContext
keyProviderMetadata map[keyprovider.Addr][]byte
// Used to build EvalContext (and related mappings)
keyValues map[string]map[string]cty.Value
methodValues map[string]map[string]cty.Value
methods map[method.Addr]method.Method
}
func (base *baseEncryption) buildTargetMethods(meta map[keyprovider.Addr][]byte) ([]method.Method, hcl.Diagnostics) {
var diags hcl.Diagnostics
builder := &targetBuilder{
cfg: base.enc.cfg,
reg: base.enc.reg,
ctx: &hcl.EvalContext{
Variables: map[string]cty.Value{},
},
keyProviderMetadata: meta,
}
diags = append(diags, builder.setupKeyProviders()...)
if diags.HasErrors() {
return nil, diags
}
diags = append(diags, builder.setupMethods()...)
if diags.HasErrors() {
return nil, diags
}
return builder.build(base.target, base.name)
}
// build sets up a single target for encryption. It returns the primary and fallback methods for the target, as well
// as a list of diagnostics if the target is invalid.
// The targetName parameter is used for error messages only.
func (e *targetBuilder) build(target *config.TargetConfig, targetName string) (methods []method.Method, diags hcl.Diagnostics) {
// gohcl has some weirdness around attributes that are not provided, but are hcl.Expressions
// They will set the attribute field to a static null expression
// https://github.com/hashicorp/hcl/blob/main/gohcl/decode.go#L112-L118
// Descriptor referenced by this target
var methodIdent *string
decodeDiags := gohcl.DecodeExpression(target.Method, e.ctx, &methodIdent)
diags = append(diags, decodeDiags...)
// Only attempt to fetch the method if the decoding was successful
if !decodeDiags.HasErrors() {
if methodIdent != nil {
if method, ok := e.methods[method.Addr(*methodIdent)]; ok {
methods = append(methods, method)
} else {
// We can't continue if the method is not found
diags = append(diags, &hcl.Diagnostic{
Severity: hcl.DiagError,
Summary: "Undefined encryption method",
Detail: fmt.Sprintf("Can not find %q for %q", *methodIdent, targetName),
Subject: target.Method.Range().Ptr(),
})
}
} else {
// nil is a nop method
methods = append(methods, nil)
}
}
// Attempt to fetch the fallback method if it's been configured
if target.Fallback != nil {
fallback, fallbackDiags := e.build(target.Fallback, targetName+".fallback")
diags = append(diags, fallbackDiags...)
methods = append(methods, fallback...)
}
return methods, diags
}

2
internal/plugin/discovery/meta_set.go
View File

@ -7,7 +7,7 @@ package discovery
// A PluginMetaSet is a set of PluginMeta objects meeting a certain criteria.
//
// Methods on this type allow filtering of the set to produce subsets that
// MethodConfigs on this type allow filtering of the set to produce subsets that
// meet more restrictive criteria.
type PluginMetaSet map[PluginMeta]struct{}

84
internal/varhcl/example_test.go Normal file
View File

@ -0,0 +1,84 @@
// Copyright (c) The OpenTofu Authors
// SPDX-License-Identifier: MPL-2.0
// Copyright (c) 2023 HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package varhcl
import (
"fmt"
"testing"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
"github.com/zclconf/go-cty/cty"
)
var data = `
inner "foo" "bar" {
val = magic.foo.bar
data = {
"z" = nested.value
}
}
`
type InnerBlock struct {
Type string `hcl:"type,label"`
Name string `hcl:"name,label"`
Value string `hcl:"val"`
Data map[string]string `hcl:"data"`
}
type OuterBlock struct {
Contents InnerBlock `hcl:"inner,block"`
}
func Test(t *testing.T) {
println("> Parse HCL")
file, diags := hclsyntax.ParseConfig([]byte(data), "INLINE", hcl.Pos{Byte: 0, Line: 1, Column: 1})
println(diags.Error())
ob := &OuterBlock{}
println()
println("> Detect Variables")
vars, diags := VariablesInBody(file.Body, ob)
println(diags.Error())
for _, v := range vars {
ident := ""
for _, p := range v {
if root, ok := p.(hcl.TraverseRoot); ok {
ident += root.Name
}
if attr, ok := p.(hcl.TraverseAttr); ok {
ident += "." + attr.Name
}
}
println("Required: " + ident)
}
println()
println("> Decode Body")
ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"magic": cty.ObjectVal(map[string]cty.Value{
"foo": cty.ObjectVal(map[string]cty.Value{
"bar": cty.StringVal("BAR IS BEST BAR"),
}),
}),
"nested": cty.ObjectVal(map[string]cty.Value{
"value": cty.StringVal("ZISHERE"),
}),
},
}
diags = gohcl.DecodeBody(file.Body, ctx, ob)
println(diags.Error())
fmt.Printf("%#v\n", ob)
}

85
internal/varhcl/schema.go Normal file
View File

@ -0,0 +1,85 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
// This is a partial copy of hashicorp/hcl/gohcl/schema.go to allow access to internal variables.
// vardecode.go should be upstreamed instead.
package varhcl
import (
"fmt"
"reflect"
"strings"
)
type fieldTags struct {
Attributes map[string]int
Blocks map[string]int
Labels []labelField
Remain *int
Body *int
Optional map[string]bool
}
type labelField struct {
FieldIndex int
Name string
}
func getFieldTags(ty reflect.Type) *fieldTags {
ret := &fieldTags{
Attributes: map[string]int{},
Blocks: map[string]int{},
Optional: map[string]bool{},
}
ct := ty.NumField()
for i := 0; i < ct; i++ {
field := ty.Field(i)
tag := field.Tag.Get("hcl")
if tag == "" {
continue
}
comma := strings.Index(tag, ",")
var name, kind string
if comma != -1 {
name = tag[:comma]
kind = tag[comma+1:]
} else {
name = tag
kind = "attr"
}
switch kind {
case "attr":
ret.Attributes[name] = i
case "block":
ret.Blocks[name] = i
case "label":
ret.Labels = append(ret.Labels, labelField{
FieldIndex: i,
Name: name,
})
case "remain":
if ret.Remain != nil {
panic("only one 'remain' tag is permitted")
}
idx := i // copy, because this loop will continue assigning to i
ret.Remain = &idx
case "body":
if ret.Body != nil {
panic("only one 'body' tag is permitted")
}
idx := i // copy, because this loop will continue assigning to i
ret.Body = &idx
case "optional":
ret.Attributes[name] = i
ret.Optional[name] = true
default:
panic(fmt.Sprintf("invalid hcl field tag kind %q on %s %q", kind, field.Type.String(), field.Name))
}
}
return ret
}

98
internal/varhcl/vardecode.go Normal file
View File

@ -0,0 +1,98 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
// This is a fork of hashicorp/hcl/gohcl/decode.go that pulls out variable dependencies in attributes
package varhcl
import (
"fmt"
"reflect"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
)
func VariablesInBody(body hcl.Body, val interface{}) ([]hcl.Traversal, hcl.Diagnostics) {
rv := reflect.ValueOf(val)
if rv.Kind() != reflect.Ptr {
panic(fmt.Sprintf("target value must be a pointer, not %s", rv.Type().String()))
}
return findVariablesInBody(body, rv.Elem())
}
func findVariablesInBody(body hcl.Body, val reflect.Value) ([]hcl.Traversal, hcl.Diagnostics) {
et := val.Type()
switch et.Kind() {
case reflect.Struct:
return findVariablesInBodyStruct(body, val)
case reflect.Map:
return findVariablesInBodyMap(body, val)
default:
panic(fmt.Sprintf("target value must be pointer to struct or map, not %s", et.String()))
}
}
func findVariablesInBodyStruct(body hcl.Body, val reflect.Value) ([]hcl.Traversal, hcl.Diagnostics) {
var variables []hcl.Traversal
schema, partial := gohcl.ImpliedBodySchema(val.Interface())
var content *hcl.BodyContent
var diags hcl.Diagnostics
if partial {
content, _, diags = body.PartialContent(schema)
} else {
content, diags = body.Content(schema)
}
if content == nil {
return variables, diags
}
tags := getFieldTags(val.Type())
for name := range tags.Attributes {
attr := content.Attributes[name]
variables = append(variables, attr.Expr.Variables()...)
}
blocksByType := content.Blocks.ByType()
for typeName, fieldIdx := range tags.Blocks {
blocks := blocksByType[typeName]
field := val.Type().Field(fieldIdx)
ty := field.Type
if ty.Kind() == reflect.Slice {
ty = ty.Elem()
}
if ty.Kind() == reflect.Ptr {
ty = ty.Elem()
}
for _, block := range blocks {
blockVars, blockDiags := findVariablesInBody(block.Body, reflect.New(ty).Elem())
variables = append(variables, blockVars...)
diags = append(diags, blockDiags...)
}
}
return variables, diags
}
func findVariablesInBodyMap(body hcl.Body, v reflect.Value) ([]hcl.Traversal, hcl.Diagnostics) {
var variables []hcl.Traversal
attrs, diags := body.JustAttributes()
if attrs == nil {
return variables, diags
}
for _, attr := range attrs {
variables = append(variables, attr.Expr.Variables()...)
}
return variables, diags
}