core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.
This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.
Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.
However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.
My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
- The "terraform graph" command is no longer hooked directly into the
core graph builders, because that's no longer part of the public API.
However, I did include a couple new Context functions whose contract
is to produce a UI-oriented graph, and _for now_ those continue to
return the physical graph we use for those operations. There's no
exported API for generating the "validate" and "eval" graphs, because
neither is particularly interesting in its own right, and so
"terraform graph" no longer supports those graph types.
- terraform.NewContext no longer has the responsibility for collecting
all of the provider schemas up front. Instead, we wait until we need
them. However, that means that some of our error messages now have a
slightly different shape due to unwinding through a differently-shaped
call stack. As of this commit we also end up reloading the schemas
multiple times in some cases, which is functionally acceptable but
likely represents a performance regression. I intend to rework this to
use caching, but I'm saving that for a later commit because this one is
big enough already.
The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.
However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-24 14:06:38 -05:00
|
|
|
package terraform
|
|
|
|
|
|
|
|
import (
|
|
|
|
"log"
|
|
|
|
|
|
|
|
"github.com/hashicorp/terraform/internal/configs"
|
|
|
|
"github.com/hashicorp/terraform/internal/instances"
|
|
|
|
"github.com/hashicorp/terraform/internal/plans"
|
|
|
|
"github.com/hashicorp/terraform/internal/refactoring"
|
|
|
|
"github.com/hashicorp/terraform/internal/states"
|
|
|
|
"github.com/hashicorp/terraform/internal/tfdiags"
|
|
|
|
)
|
|
|
|
|
|
|
|
// graphWalkOpts captures some transient values we use (and possibly mutate)
|
|
|
|
// during a graph walk.
|
|
|
|
//
|
|
|
|
// The way these options get used unfortunately varies between the different
|
|
|
|
// walkOperation types. This is a historical design wart that dates back to
|
|
|
|
// us using the same graph structure for all operations; hopefully we'll
|
|
|
|
// make the necessary differences between the walk types more explicit someday.
|
|
|
|
type graphWalkOpts struct {
|
|
|
|
InputState *states.State
|
|
|
|
Changes *plans.Changes
|
|
|
|
Config *configs.Config
|
|
|
|
|
core: Handle root and child module input variables consistently
Previously we had a significant discrepancy between these two situations:
we wrote the raw root module variables directly into the EvalContext and
then applied type conversions only at expression evaluation time, while
for child modules we converted and validated the values while visiting
the variable graph node and wrote only the _final_ value into the
EvalContext.
This confusion seems to have been the root cause for #29899, where
validation rules for root module variables were being applied at the wrong
point in the process, prior to type conversion.
To fix that bug and also make similar mistakes less likely in the future,
I've made the root module variable handling more like the child module
variable handling in the following ways:
- The "raw value" (exactly as given by the user) lives only in the graph
node representing the variable, which mirrors how the _expression_
for a child module variable lives in its graph node. This means that
the flow for the two is the same except that there's no expression
evaluation step for root module variables, because they arrive as
constant values from the caller.
- The set of variable values in the EvalContext is always only "final"
values, after type conversion is complete. That in turn means we no
longer need to do "just in time" conversion in
evaluationStateData.GetInputVariable, and can just return the value
exactly as stored, which is consistent with how we handle all other
references between objects.
This diff is noisier than I'd like because of how much it takes to wire
a new argument (the raw variable values) through to the plan graph builder,
but those changes are pretty mechanical and the interesting logic lives
inside the plan graph builder itself, in NodeRootVariable, and
the shared helper functions in eval_variable.go.
While here I also took the opportunity to fix a historical API wart in
EvalContext, where SetModuleCallArguments was built to take a set of
variable values all at once but our current caller always calls with only
one at a time. That is now just SetModuleCallArgument singular, to match
with the new SetRootModuleArgument to deal with root module variables.
2021-11-10 19:29:45 -06:00
|
|
|
MoveResults refactoring.MoveResults
|
core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.
This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.
Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.
However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.
My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
- The "terraform graph" command is no longer hooked directly into the
core graph builders, because that's no longer part of the public API.
However, I did include a couple new Context functions whose contract
is to produce a UI-oriented graph, and _for now_ those continue to
return the physical graph we use for those operations. There's no
exported API for generating the "validate" and "eval" graphs, because
neither is particularly interesting in its own right, and so
"terraform graph" no longer supports those graph types.
- terraform.NewContext no longer has the responsibility for collecting
all of the provider schemas up front. Instead, we wait until we need
them. However, that means that some of our error messages now have a
slightly different shape due to unwinding through a differently-shaped
call stack. As of this commit we also end up reloading the schemas
multiple times in some cases, which is functionally acceptable but
likely represents a performance regression. I intend to rework this to
use caching, but I'm saving that for a later commit because this one is
big enough already.
The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.
However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-24 14:06:38 -05:00
|
|
|
}
|
|
|
|
|
|
|
|
func (c *Context) walk(graph *Graph, operation walkOperation, opts *graphWalkOpts) (*ContextGraphWalker, tfdiags.Diagnostics) {
|
|
|
|
log.Printf("[DEBUG] Starting graph walk: %s", operation.String())
|
|
|
|
|
|
|
|
walker := c.graphWalker(operation, opts)
|
|
|
|
|
|
|
|
// Watch for a stop so we can call the provider Stop() API.
|
|
|
|
watchStop, watchWait := c.watchStop(walker)
|
|
|
|
|
|
|
|
// Walk the real graph, this will block until it completes
|
|
|
|
diags := graph.Walk(walker)
|
|
|
|
|
|
|
|
// Close the channel so the watcher stops, and wait for it to return.
|
|
|
|
close(watchStop)
|
|
|
|
<-watchWait
|
|
|
|
|
|
|
|
return walker, diags
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *Context) graphWalker(operation walkOperation, opts *graphWalkOpts) *ContextGraphWalker {
|
|
|
|
var state *states.SyncState
|
|
|
|
var refreshState *states.SyncState
|
|
|
|
var prevRunState *states.SyncState
|
|
|
|
|
|
|
|
// NOTE: None of the SyncState objects must directly wrap opts.InputState,
|
|
|
|
// because we use those to mutate the state object and opts.InputState
|
|
|
|
// belongs to our caller and thus we must treat it as immutable.
|
|
|
|
//
|
|
|
|
// To account for that, most of our SyncState values created below end up
|
|
|
|
// wrapping a _deep copy_ of opts.InputState instead.
|
|
|
|
inputState := opts.InputState
|
|
|
|
if inputState == nil {
|
|
|
|
// Lots of callers use nil to represent the "empty" case where we've
|
|
|
|
// not run Apply yet, so we tolerate that.
|
|
|
|
inputState = states.NewState()
|
|
|
|
}
|
|
|
|
|
|
|
|
switch operation {
|
|
|
|
case walkValidate:
|
|
|
|
// validate should not use any state
|
|
|
|
state = states.NewState().SyncWrapper()
|
|
|
|
|
|
|
|
// validate currently uses the plan graph, so we have to populate the
|
|
|
|
// refreshState and the prevRunState.
|
|
|
|
refreshState = states.NewState().SyncWrapper()
|
|
|
|
prevRunState = states.NewState().SyncWrapper()
|
|
|
|
|
|
|
|
case walkPlan, walkPlanDestroy:
|
|
|
|
state = inputState.DeepCopy().SyncWrapper()
|
|
|
|
refreshState = inputState.DeepCopy().SyncWrapper()
|
|
|
|
prevRunState = inputState.DeepCopy().SyncWrapper()
|
|
|
|
|
|
|
|
default:
|
|
|
|
state = inputState.DeepCopy().SyncWrapper()
|
|
|
|
// Only plan-like walks use refreshState and prevRunState
|
|
|
|
}
|
|
|
|
|
|
|
|
changes := opts.Changes
|
|
|
|
if changes == nil {
|
|
|
|
// Several of our non-plan walks end up sharing codepaths with the
|
|
|
|
// plan walk and thus expect to generate planned changes even though
|
|
|
|
// we don't care about them. To avoid those crashing, we'll just
|
|
|
|
// insert a placeholder changes object which'll get discarded
|
|
|
|
// afterwards.
|
|
|
|
changes = plans.NewChanges()
|
|
|
|
}
|
|
|
|
|
|
|
|
if opts.Config == nil {
|
|
|
|
panic("Context.graphWalker call without Config")
|
|
|
|
}
|
|
|
|
|
|
|
|
return &ContextGraphWalker{
|
core: Handle root and child module input variables consistently
Previously we had a significant discrepancy between these two situations:
we wrote the raw root module variables directly into the EvalContext and
then applied type conversions only at expression evaluation time, while
for child modules we converted and validated the values while visiting
the variable graph node and wrote only the _final_ value into the
EvalContext.
This confusion seems to have been the root cause for #29899, where
validation rules for root module variables were being applied at the wrong
point in the process, prior to type conversion.
To fix that bug and also make similar mistakes less likely in the future,
I've made the root module variable handling more like the child module
variable handling in the following ways:
- The "raw value" (exactly as given by the user) lives only in the graph
node representing the variable, which mirrors how the _expression_
for a child module variable lives in its graph node. This means that
the flow for the two is the same except that there's no expression
evaluation step for root module variables, because they arrive as
constant values from the caller.
- The set of variable values in the EvalContext is always only "final"
values, after type conversion is complete. That in turn means we no
longer need to do "just in time" conversion in
evaluationStateData.GetInputVariable, and can just return the value
exactly as stored, which is consistent with how we handle all other
references between objects.
This diff is noisier than I'd like because of how much it takes to wire
a new argument (the raw variable values) through to the plan graph builder,
but those changes are pretty mechanical and the interesting logic lives
inside the plan graph builder itself, in NodeRootVariable, and
the shared helper functions in eval_variable.go.
While here I also took the opportunity to fix a historical API wart in
EvalContext, where SetModuleCallArguments was built to take a set of
variable values all at once but our current caller always calls with only
one at a time. That is now just SetModuleCallArgument singular, to match
with the new SetRootModuleArgument to deal with root module variables.
2021-11-10 19:29:45 -06:00
|
|
|
Context: c,
|
|
|
|
State: state,
|
|
|
|
Config: opts.Config,
|
|
|
|
RefreshState: refreshState,
|
|
|
|
PrevRunState: prevRunState,
|
|
|
|
Changes: changes.SyncWrapper(),
|
|
|
|
InstanceExpander: instances.NewExpander(),
|
|
|
|
MoveResults: opts.MoveResults,
|
|
|
|
Operation: operation,
|
|
|
|
StopContext: c.runContext,
|
core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.
This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.
Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.
However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.
My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
- The "terraform graph" command is no longer hooked directly into the
core graph builders, because that's no longer part of the public API.
However, I did include a couple new Context functions whose contract
is to produce a UI-oriented graph, and _for now_ those continue to
return the physical graph we use for those operations. There's no
exported API for generating the "validate" and "eval" graphs, because
neither is particularly interesting in its own right, and so
"terraform graph" no longer supports those graph types.
- terraform.NewContext no longer has the responsibility for collecting
all of the provider schemas up front. Instead, we wait until we need
them. However, that means that some of our error messages now have a
slightly different shape due to unwinding through a differently-shaped
call stack. As of this commit we also end up reloading the schemas
multiple times in some cases, which is functionally acceptable but
likely represents a performance regression. I intend to rework this to
use caching, but I'm saving that for a later commit because this one is
big enough already.
The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.
However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-24 14:06:38 -05:00
|
|
|
}
|
|
|
|
}
|