opentofu/helper/schema/resource.go
Martin Atkins bd1a215580 helper/resource: Ignore Removed attributes for ImportStateVerify
Due to the lossiness of our legacy models for diff and state, shimming a
diff and then creating a state from it produces a different result than
shimming a state directly. That means that ImportStateVerify no longer
works as expected if there are any Computed attributes in the schema where
d.Set isn't called during Read.

Fixing that for every case would require some risky changes to the shim
behavior, so we're instead going to ask provider developers to address it
by adding `d.Set` calls where needed, since that is the contract for
"Computed" anyway -- a default value should be produced during Create, and
thus by extension during Import.

However, since a common situation where this occurs is attributes marked
as "Removed", since all of the code that deals with them has generally
been deleted, we'll avoid problems in that case here by treating Removed
attributes as ignored for the purposes of ImportStateVerify.

This required exporting some functionality that was formerly unexported
in helper/schema, but it's a relatively harmless schema introspection
function so shouldn't be a big deal to export it.
2019-04-16 11:14:49 -07:00

824 lines
25 KiB
Go

package schema
import (
"errors"
"fmt"
"log"
"strconv"
"github.com/hashicorp/terraform/config"
"github.com/hashicorp/terraform/terraform"
"github.com/zclconf/go-cty/cty"
)
// Resource represents a thing in Terraform that has a set of configurable
// attributes and a lifecycle (create, read, update, delete).
//
// The Resource schema is an abstraction that allows provider writers to
// worry only about CRUD operations while off-loading validation, diff
// generation, etc. to this higher level library.
//
// In spite of the name, this struct is not used only for terraform resources,
// but also for data sources. In the case of data sources, the Create,
// Update and Delete functions must not be provided.
type Resource struct {
// Schema is the schema for the configuration of this resource.
//
// The keys of this map are the configuration keys, and the values
// describe the schema of the configuration value.
//
// The schema is used to represent both configurable data as well
// as data that might be computed in the process of creating this
// resource.
Schema map[string]*Schema
// SchemaVersion is the version number for this resource's Schema
// definition. The current SchemaVersion stored in the state for each
// resource. Provider authors can increment this version number
// when Schema semantics change. If the State's SchemaVersion is less than
// the current SchemaVersion, the InstanceState is yielded to the
// MigrateState callback, where the provider can make whatever changes it
// needs to update the state to be compatible to the latest version of the
// Schema.
//
// When unset, SchemaVersion defaults to 0, so provider authors can start
// their Versioning at any integer >= 1
SchemaVersion int
// MigrateState is deprecated and any new changes to a resource's schema
// should be handled by StateUpgraders. Existing MigrateState implementations
// should remain for compatibility with existing state. MigrateState will
// still be called if the stored SchemaVersion is less than the
// first version of the StateUpgraders.
//
// MigrateState is responsible for updating an InstanceState with an old
// version to the format expected by the current version of the Schema.
//
// It is called during Refresh if the State's stored SchemaVersion is less
// than the current SchemaVersion of the Resource.
//
// The function is yielded the state's stored SchemaVersion and a pointer to
// the InstanceState that needs updating, as well as the configured
// provider's configured meta interface{}, in case the migration process
// needs to make any remote API calls.
MigrateState StateMigrateFunc
// StateUpgraders contains the functions responsible for upgrading an
// existing state with an old schema version to a newer schema. It is
// called specifically by Terraform when the stored schema version is less
// than the current SchemaVersion of the Resource.
//
// StateUpgraders map specific schema versions to a StateUpgrader
// function. The registered versions are expected to be ordered,
// consecutive values. The initial value may be greater than 0 to account
// for legacy schemas that weren't recorded and can be handled by
// MigrateState.
StateUpgraders []StateUpgrader
// The functions below are the CRUD operations for this resource.
//
// The only optional operation is Update. If Update is not implemented,
// then updates will not be supported for this resource.
//
// The ResourceData parameter in the functions below are used to
// query configuration and changes for the resource as well as to set
// the ID, computed data, etc.
//
// The interface{} parameter is the result of the ConfigureFunc in
// the provider for this resource. If the provider does not define
// a ConfigureFunc, this will be nil. This parameter should be used
// to store API clients, configuration structures, etc.
//
// If any errors occur during each of the operation, an error should be
// returned. If a resource was partially updated, be careful to enable
// partial state mode for ResourceData and use it accordingly.
//
// Exists is a function that is called to check if a resource still
// exists. If this returns false, then this will affect the diff
// accordingly. If this function isn't set, it will not be called. It
// is highly recommended to set it. The *ResourceData passed to Exists
// should _not_ be modified.
Create CreateFunc
Read ReadFunc
Update UpdateFunc
Delete DeleteFunc
Exists ExistsFunc
// CustomizeDiff is a custom function for working with the diff that
// Terraform has created for this resource - it can be used to customize the
// diff that has been created, diff values not controlled by configuration,
// or even veto the diff altogether and abort the plan. It is passed a
// *ResourceDiff, a structure similar to ResourceData but lacking most write
// functions like Set, while introducing new functions that work with the
// diff such as SetNew, SetNewComputed, and ForceNew.
//
// The phases Terraform runs this in, and the state available via functions
// like Get and GetChange, are as follows:
//
// * New resource: One run with no state
// * Existing resource: One run with state
// * Existing resource, forced new: One run with state (before ForceNew),
// then one run without state (as if new resource)
// * Tainted resource: No runs (custom diff logic is skipped)
// * Destroy: No runs (standard diff logic is skipped on destroy diffs)
//
// This function needs to be resilient to support all scenarios.
//
// If this function needs to access external API resources, remember to flag
// the RequiresRefresh attribute mentioned below to ensure that
// -refresh=false is blocked when running plan or apply, as this means that
// this resource requires refresh-like behaviour to work effectively.
//
// For the most part, only computed fields can be customized by this
// function.
//
// This function is only allowed on regular resources (not data sources).
CustomizeDiff CustomizeDiffFunc
// Importer is the ResourceImporter implementation for this resource.
// If this is nil, then this resource does not support importing. If
// this is non-nil, then it supports importing and ResourceImporter
// must be validated. The validity of ResourceImporter is verified
// by InternalValidate on Resource.
Importer *ResourceImporter
// If non-empty, this string is emitted as a warning during Validate.
DeprecationMessage string
// Timeouts allow users to specify specific time durations in which an
// operation should time out, to allow them to extend an action to suit their
// usage. For example, a user may specify a large Creation timeout for their
// AWS RDS Instance due to it's size, or restoring from a snapshot.
// Resource implementors must enable Timeout support by adding the allowed
// actions (Create, Read, Update, Delete, Default) to the Resource struct, and
// accessing them in the matching methods.
Timeouts *ResourceTimeout
}
// ShimInstanceStateFromValue converts a cty.Value to a
// terraform.InstanceState.
func (r *Resource) ShimInstanceStateFromValue(state cty.Value) (*terraform.InstanceState, error) {
// Get the raw shimmed value. While this is correct, the set hashes don't
// match those from the Schema.
s := terraform.NewInstanceStateShimmedFromValue(state, r.SchemaVersion)
// We now rebuild the state through the ResourceData, so that the set indexes
// match what helper/schema expects.
data, err := schemaMap(r.Schema).Data(s, nil)
if err != nil {
return nil, err
}
s = data.State()
if s == nil {
s = &terraform.InstanceState{}
}
return s, nil
}
// See Resource documentation.
type CreateFunc func(*ResourceData, interface{}) error
// See Resource documentation.
type ReadFunc func(*ResourceData, interface{}) error
// See Resource documentation.
type UpdateFunc func(*ResourceData, interface{}) error
// See Resource documentation.
type DeleteFunc func(*ResourceData, interface{}) error
// See Resource documentation.
type ExistsFunc func(*ResourceData, interface{}) (bool, error)
// See Resource documentation.
type StateMigrateFunc func(
int, *terraform.InstanceState, interface{}) (*terraform.InstanceState, error)
type StateUpgrader struct {
// Version is the version schema that this Upgrader will handle, converting
// it to Version+1.
Version int
// Type describes the schema that this function can upgrade. Type is
// required to decode the schema if the state was stored in a legacy
// flatmap format.
Type cty.Type
// Upgrade takes the JSON encoded state and the provider meta value, and
// upgrades the state one single schema version. The provided state is
// deocded into the default json types using a map[string]interface{}. It
// is up to the StateUpgradeFunc to ensure that the returned value can be
// encoded using the new schema.
Upgrade StateUpgradeFunc
}
// See StateUpgrader
type StateUpgradeFunc func(rawState map[string]interface{}, meta interface{}) (map[string]interface{}, error)
// See Resource documentation.
type CustomizeDiffFunc func(*ResourceDiff, interface{}) error
// Apply creates, updates, and/or deletes a resource.
func (r *Resource) Apply(
s *terraform.InstanceState,
d *terraform.InstanceDiff,
meta interface{}) (*terraform.InstanceState, error) {
data, err := schemaMap(r.Schema).Data(s, d)
if err != nil {
return s, err
}
// Instance Diff shoould have the timeout info, need to copy it over to the
// ResourceData meta
rt := ResourceTimeout{}
if _, ok := d.Meta[TimeoutKey]; ok {
if err := rt.DiffDecode(d); err != nil {
log.Printf("[ERR] Error decoding ResourceTimeout: %s", err)
}
} else if s != nil {
if _, ok := s.Meta[TimeoutKey]; ok {
if err := rt.StateDecode(s); err != nil {
log.Printf("[ERR] Error decoding ResourceTimeout: %s", err)
}
}
} else {
log.Printf("[DEBUG] No meta timeoutkey found in Apply()")
}
data.timeouts = &rt
if s == nil {
// The Terraform API dictates that this should never happen, but
// it doesn't hurt to be safe in this case.
s = new(terraform.InstanceState)
}
if d.Destroy || d.RequiresNew() {
if s.ID != "" {
// Destroy the resource since it is created
if err := r.Delete(data, meta); err != nil {
return r.recordCurrentSchemaVersion(data.State()), err
}
// Make sure the ID is gone.
data.SetId("")
}
// If we're only destroying, and not creating, then return
// now since we're done!
if !d.RequiresNew() {
return nil, nil
}
// Reset the data to be stateless since we just destroyed
data, err = schemaMap(r.Schema).Data(nil, d)
// data was reset, need to re-apply the parsed timeouts
data.timeouts = &rt
if err != nil {
return nil, err
}
}
err = nil
if data.Id() == "" {
// We're creating, it is a new resource.
data.MarkNewResource()
err = r.Create(data, meta)
} else {
if r.Update == nil {
return s, fmt.Errorf("doesn't support update")
}
err = r.Update(data, meta)
}
return r.recordCurrentSchemaVersion(data.State()), err
}
// Diff returns a diff of this resource.
func (r *Resource) Diff(
s *terraform.InstanceState,
c *terraform.ResourceConfig,
meta interface{}) (*terraform.InstanceDiff, error) {
t := &ResourceTimeout{}
err := t.ConfigDecode(r, c)
if err != nil {
return nil, fmt.Errorf("[ERR] Error decoding timeout: %s", err)
}
instanceDiff, err := schemaMap(r.Schema).Diff(s, c, r.CustomizeDiff, meta, true)
if err != nil {
return instanceDiff, err
}
if instanceDiff != nil {
if err := t.DiffEncode(instanceDiff); err != nil {
log.Printf("[ERR] Error encoding timeout to instance diff: %s", err)
}
} else {
log.Printf("[DEBUG] Instance Diff is nil in Diff()")
}
return instanceDiff, err
}
func (r *Resource) simpleDiff(
s *terraform.InstanceState,
c *terraform.ResourceConfig,
meta interface{}) (*terraform.InstanceDiff, error) {
t := &ResourceTimeout{}
err := t.ConfigDecode(r, c)
if err != nil {
return nil, fmt.Errorf("[ERR] Error decoding timeout: %s", err)
}
instanceDiff, err := schemaMap(r.Schema).Diff(s, c, r.CustomizeDiff, meta, false)
if err != nil {
return instanceDiff, err
}
if instanceDiff == nil {
log.Printf("[DEBUG] Instance Diff is nil in SimpleDiff()")
return nil, err
}
// Make sure the old value is set in each of the instance diffs.
// This was done by the RequiresNew logic in the full legacy Diff.
for k, attr := range instanceDiff.Attributes {
if attr == nil {
continue
}
if s != nil {
attr.Old = s.Attributes[k]
}
}
if err := t.DiffEncode(instanceDiff); err != nil {
log.Printf("[ERR] Error encoding timeout to instance diff: %s", err)
}
return instanceDiff, err
}
// Validate validates the resource configuration against the schema.
func (r *Resource) Validate(c *terraform.ResourceConfig) ([]string, []error) {
warns, errs := schemaMap(r.Schema).Validate(c)
if r.DeprecationMessage != "" {
warns = append(warns, r.DeprecationMessage)
}
return warns, errs
}
// ReadDataApply loads the data for a data source, given a diff that
// describes the configuration arguments and desired computed attributes.
func (r *Resource) ReadDataApply(
d *terraform.InstanceDiff,
meta interface{},
) (*terraform.InstanceState, error) {
// Data sources are always built completely from scratch
// on each read, so the source state is always nil.
data, err := schemaMap(r.Schema).Data(nil, d)
if err != nil {
return nil, err
}
err = r.Read(data, meta)
state := data.State()
if state != nil && state.ID == "" {
// Data sources can set an ID if they want, but they aren't
// required to; we'll provide a placeholder if they don't,
// to preserve the invariant that all resources have non-empty
// ids.
state.ID = "-"
}
return r.recordCurrentSchemaVersion(state), err
}
// RefreshWithoutUpgrade reads the instance state, but does not call
// MigrateState or the StateUpgraders, since those are now invoked in a
// separate API call.
// RefreshWithoutUpgrade is part of the new plugin shims.
func (r *Resource) RefreshWithoutUpgrade(
s *terraform.InstanceState,
meta interface{}) (*terraform.InstanceState, error) {
// If the ID is already somehow blank, it doesn't exist
if s.ID == "" {
return nil, nil
}
rt := ResourceTimeout{}
if _, ok := s.Meta[TimeoutKey]; ok {
if err := rt.StateDecode(s); err != nil {
log.Printf("[ERR] Error decoding ResourceTimeout: %s", err)
}
}
if r.Exists != nil {
// Make a copy of data so that if it is modified it doesn't
// affect our Read later.
data, err := schemaMap(r.Schema).Data(s, nil)
data.timeouts = &rt
if err != nil {
return s, err
}
exists, err := r.Exists(data, meta)
if err != nil {
return s, err
}
if !exists {
return nil, nil
}
}
data, err := schemaMap(r.Schema).Data(s, nil)
data.timeouts = &rt
if err != nil {
return s, err
}
err = r.Read(data, meta)
state := data.State()
if state != nil && state.ID == "" {
state = nil
}
return r.recordCurrentSchemaVersion(state), err
}
// Refresh refreshes the state of the resource.
func (r *Resource) Refresh(
s *terraform.InstanceState,
meta interface{}) (*terraform.InstanceState, error) {
// If the ID is already somehow blank, it doesn't exist
if s.ID == "" {
return nil, nil
}
rt := ResourceTimeout{}
if _, ok := s.Meta[TimeoutKey]; ok {
if err := rt.StateDecode(s); err != nil {
log.Printf("[ERR] Error decoding ResourceTimeout: %s", err)
}
}
if r.Exists != nil {
// Make a copy of data so that if it is modified it doesn't
// affect our Read later.
data, err := schemaMap(r.Schema).Data(s, nil)
data.timeouts = &rt
if err != nil {
return s, err
}
exists, err := r.Exists(data, meta)
if err != nil {
return s, err
}
if !exists {
return nil, nil
}
}
// there may be new StateUpgraders that need to be run
s, err := r.upgradeState(s, meta)
if err != nil {
return s, err
}
data, err := schemaMap(r.Schema).Data(s, nil)
data.timeouts = &rt
if err != nil {
return s, err
}
err = r.Read(data, meta)
state := data.State()
if state != nil && state.ID == "" {
state = nil
}
return r.recordCurrentSchemaVersion(state), err
}
func (r *Resource) upgradeState(s *terraform.InstanceState, meta interface{}) (*terraform.InstanceState, error) {
var err error
needsMigration, stateSchemaVersion := r.checkSchemaVersion(s)
migrate := needsMigration && r.MigrateState != nil
if migrate {
s, err = r.MigrateState(stateSchemaVersion, s, meta)
if err != nil {
return s, err
}
}
if len(r.StateUpgraders) == 0 {
return s, nil
}
// If we ran MigrateState, then the stateSchemaVersion value is no longer
// correct. We can expect the first upgrade function to be the correct
// schema type version.
if migrate {
stateSchemaVersion = r.StateUpgraders[0].Version
}
schemaType := r.CoreConfigSchema().ImpliedType()
// find the expected type to convert the state
for _, upgrader := range r.StateUpgraders {
if stateSchemaVersion == upgrader.Version {
schemaType = upgrader.Type
}
}
// StateUpgraders only operate on the new JSON format state, so the state
// need to be converted.
stateVal, err := StateValueFromInstanceState(s, schemaType)
if err != nil {
return nil, err
}
jsonState, err := StateValueToJSONMap(stateVal, schemaType)
if err != nil {
return nil, err
}
for _, upgrader := range r.StateUpgraders {
if stateSchemaVersion != upgrader.Version {
continue
}
jsonState, err = upgrader.Upgrade(jsonState, meta)
if err != nil {
return nil, err
}
stateSchemaVersion++
}
// now we need to re-flatmap the new state
stateVal, err = JSONMapToStateValue(jsonState, r.CoreConfigSchema())
if err != nil {
return nil, err
}
return r.ShimInstanceStateFromValue(stateVal)
}
// InternalValidate should be called to validate the structure
// of the resource.
//
// This should be called in a unit test for any resource to verify
// before release that a resource is properly configured for use with
// this library.
//
// Provider.InternalValidate() will automatically call this for all of
// the resources it manages, so you don't need to call this manually if it
// is part of a Provider.
func (r *Resource) InternalValidate(topSchemaMap schemaMap, writable bool) error {
if r == nil {
return errors.New("resource is nil")
}
if !writable {
if r.Create != nil || r.Update != nil || r.Delete != nil {
return fmt.Errorf("must not implement Create, Update or Delete")
}
// CustomizeDiff cannot be defined for read-only resources
if r.CustomizeDiff != nil {
return fmt.Errorf("cannot implement CustomizeDiff")
}
}
tsm := topSchemaMap
if r.isTopLevel() && writable {
// All non-Computed attributes must be ForceNew if Update is not defined
if r.Update == nil {
nonForceNewAttrs := make([]string, 0)
for k, v := range r.Schema {
if !v.ForceNew && !v.Computed {
nonForceNewAttrs = append(nonForceNewAttrs, k)
}
}
if len(nonForceNewAttrs) > 0 {
return fmt.Errorf(
"No Update defined, must set ForceNew on: %#v", nonForceNewAttrs)
}
} else {
nonUpdateableAttrs := make([]string, 0)
for k, v := range r.Schema {
if v.ForceNew || v.Computed && !v.Optional {
nonUpdateableAttrs = append(nonUpdateableAttrs, k)
}
}
updateableAttrs := len(r.Schema) - len(nonUpdateableAttrs)
if updateableAttrs == 0 {
return fmt.Errorf(
"All fields are ForceNew or Computed w/out Optional, Update is superfluous")
}
}
tsm = schemaMap(r.Schema)
// Destroy, and Read are required
if r.Read == nil {
return fmt.Errorf("Read must be implemented")
}
if r.Delete == nil {
return fmt.Errorf("Delete must be implemented")
}
// If we have an importer, we need to verify the importer.
if r.Importer != nil {
if err := r.Importer.InternalValidate(); err != nil {
return err
}
}
for k, f := range tsm {
if isReservedResourceFieldName(k, f) {
return fmt.Errorf("%s is a reserved field name", k)
}
}
}
lastVersion := -1
for _, u := range r.StateUpgraders {
if lastVersion >= 0 && u.Version-lastVersion > 1 {
return fmt.Errorf("missing schema version between %d and %d", lastVersion, u.Version)
}
if u.Version >= r.SchemaVersion {
return fmt.Errorf("StateUpgrader version %d is >= current version %d", u.Version, r.SchemaVersion)
}
if !u.Type.IsObjectType() {
return fmt.Errorf("StateUpgrader %d type is not cty.Object", u.Version)
}
if u.Upgrade == nil {
return fmt.Errorf("StateUpgrader %d missing StateUpgradeFunc", u.Version)
}
lastVersion = u.Version
}
if lastVersion >= 0 && lastVersion != r.SchemaVersion-1 {
return fmt.Errorf("missing StateUpgrader between %d and %d", lastVersion, r.SchemaVersion)
}
// Data source
if r.isTopLevel() && !writable {
tsm = schemaMap(r.Schema)
for k, _ := range tsm {
if isReservedDataSourceFieldName(k) {
return fmt.Errorf("%s is a reserved field name", k)
}
}
}
return schemaMap(r.Schema).InternalValidate(tsm)
}
func isReservedDataSourceFieldName(name string) bool {
for _, reservedName := range config.ReservedDataSourceFields {
if name == reservedName {
return true
}
}
return false
}
func isReservedResourceFieldName(name string, s *Schema) bool {
// Allow phasing out "id"
// See https://github.com/terraform-providers/terraform-provider-aws/pull/1626#issuecomment-328881415
if name == "id" && (s.Deprecated != "" || s.Removed != "") {
return false
}
for _, reservedName := range config.ReservedResourceFields {
if name == reservedName {
return true
}
}
return false
}
// Data returns a ResourceData struct for this Resource. Each return value
// is a separate copy and can be safely modified differently.
//
// The data returned from this function has no actual affect on the Resource
// itself (including the state given to this function).
//
// This function is useful for unit tests and ResourceImporter functions.
func (r *Resource) Data(s *terraform.InstanceState) *ResourceData {
result, err := schemaMap(r.Schema).Data(s, nil)
if err != nil {
// At the time of writing, this isn't possible (Data never returns
// non-nil errors). We panic to find this in the future if we have to.
// I don't see a reason for Data to ever return an error.
panic(err)
}
// load the Resource timeouts
result.timeouts = r.Timeouts
if result.timeouts == nil {
result.timeouts = &ResourceTimeout{}
}
// Set the schema version to latest by default
result.meta = map[string]interface{}{
"schema_version": strconv.Itoa(r.SchemaVersion),
}
return result
}
// TestResourceData Yields a ResourceData filled with this resource's schema for use in unit testing
//
// TODO: May be able to be removed with the above ResourceData function.
func (r *Resource) TestResourceData() *ResourceData {
return &ResourceData{
schema: r.Schema,
}
}
// SchemasForFlatmapPath tries its best to find a sequence of schemas that
// the given dot-delimited attribute path traverses through in the schema
// of the receiving Resource.
func (r *Resource) SchemasForFlatmapPath(path string) []*Schema {
return SchemasForFlatmapPath(path, r.Schema)
}
// Returns true if the resource is "top level" i.e. not a sub-resource.
func (r *Resource) isTopLevel() bool {
// TODO: This is a heuristic; replace with a definitive attribute?
return (r.Create != nil || r.Read != nil)
}
// Determines if a given InstanceState needs to be migrated by checking the
// stored version number with the current SchemaVersion
func (r *Resource) checkSchemaVersion(is *terraform.InstanceState) (bool, int) {
// Get the raw interface{} value for the schema version. If it doesn't
// exist or is nil then set it to zero.
raw := is.Meta["schema_version"]
if raw == nil {
raw = "0"
}
// Try to convert it to a string. If it isn't a string then we pretend
// that it isn't set at all. It should never not be a string unless it
// was manually tampered with.
rawString, ok := raw.(string)
if !ok {
rawString = "0"
}
stateSchemaVersion, _ := strconv.Atoi(rawString)
// Don't run MigrateState if the version is handled by a StateUpgrader,
// since StateMigrateFuncs are not required to handle unknown versions
maxVersion := r.SchemaVersion
if len(r.StateUpgraders) > 0 {
maxVersion = r.StateUpgraders[0].Version
}
return stateSchemaVersion < maxVersion, stateSchemaVersion
}
func (r *Resource) recordCurrentSchemaVersion(
state *terraform.InstanceState) *terraform.InstanceState {
if state != nil && r.SchemaVersion > 0 {
if state.Meta == nil {
state.Meta = make(map[string]interface{})
}
state.Meta["schema_version"] = strconv.Itoa(r.SchemaVersion)
}
return state
}
// Noop is a convenience implementation of resource function which takes
// no action and returns no error.
func Noop(*ResourceData, interface{}) error {
return nil
}
// RemoveFromState is a convenience implementation of a resource function
// which sets the resource ID to empty string (to remove it from state)
// and returns no error.
func RemoveFromState(d *ResourceData, _ interface{}) error {
d.SetId("")
return nil
}