Start fixing plan tests that don't expect data sources to be in the
plan. A few were just checking that Read was never called, and some
expected the data source to be nil.
In order to udpate data sources correctly when their configuration
changes, they need to be evaluated during plan. Since the plan working
state isn't saved, store any data source reads as plan changes to be
applied later. This is currently abusing the Update plan action to
indicate that the data source was read and needs to be applied to state.
We can possibly add a Store action for data sources if this approach
works out. The Read action still indicates that the data source was
deferred to the Apply phase.
We also fully handle any data source depends_on changes. Now that all
the transitive resource dependencies are known at the time of
evaluation, we can check the plan to determine if there are any changes
in the dependencies and selectively defer reading the data source.
We need to load the state during refresh, so that even if the data
source can't be read due to `depends_on`, the state can be saved back
again to prevent it from being lost altogether.
This is a step towards having data sources refresh like resources, which
will be from their saved state value.
This transformer is what will provider the data sources with the
transitive dependencies needed to determine if they can read during plan
or must be deferred.
That name tag was left in only to reduce the diff when during
implementation. Fix the naming now for these nodes so it is correct, and
prevent any possible name collision between types.
Adding a transformer to attach any transitive DependsOn references to
data sources during plan. Refactored the ReferenceMap from the
ReferenceTransformer so it can be reused for both.
GraphNodeAttachDependsOn give us a method for adding all transitive resource
dependencies found through depends_on references, so that data source
can determine if they can be read during plan. This will be done by
inspecting the changes of all dependency resources, and delaying read
until apply if any changes are planned.
* Include eval in output walk
This allows outputs to be evaluated in the evalwalk,
impacting terraform console. Outputs are still not evaluated
for terraform console in the root module, so this has
no impact on writing to state (as child module outputs are not
written to state). Also adds test coverage to the console command,
including for evaluating locals (another use of the evalwalk)
Add the expansion transformer to the eval graph,
which is used in rare scenarios which includes running
terraform console. Prevents panic when running terraform
console in contexts with module expansion
Since objects and tuples have fixed numbers of elements, we can't return
an unknown version of those during validation. While we could return a
DyanmicVal (which was used previously), that prevents the validation of
outputs and attributes in config references.
Instead, we can return a synthetic type made from a List or Map based
on the configuration, which will allow us to more precisely validate
indexes, attributes, and outputs.
Because tuple types have a fixed number of elements, and we may not know
the number of expanded instances, we can't use an unknown tuple type to
validate index expressions.
Since evaluation is driven only by the configuration (i.e. you can't
interpolate what's not in the config), the resource evaluation should
also follow configuration rather than state. Determining the each mode
solely from the config, and applying that to the state and changes
removes the need for EachMode in the resource state. This bypasses the
awkward dance around getting the correct EachMode set in and retrieved
from state during plan when it changes in the config.
A side effect of the various changes to the provider installer included losing the initialization required error message which would occur if a user removed or modified the .terraform directory.
Previously, plugin factories were created after the configuration was loaded, in terraform.NewContext. Terraform would compare the required providers (from config and state) to the available providers and return the aforementioned error if a provider was missing.
Provider factories are now loaded at the beginning of any terraform command, before terraform even loads the configuration, and therefore before terraform has a list of required providers.
This commit replaces the current error when a providers' schema cannot be found in the provider factories with the init error, and adds a command test (to plan tests, for no real reason other than that's what I thought of first).
There is no codepath that can use this any longer, since we need to
evaluate the modules as whole objects.
This means we're going to have to live for now with invalid module
output references returning "object" errors rather that "module".
The evaluationStateData needs the change to the GetModule method to work
with the new evaluator. This is using a deep copy of module instances,
which we will clean up after some changes to the states package.
Stop evaluating count and for each if they aren't set in the config.
Remove "Resource" from the function names, as they are also now used
with modules.
While we don't have any expansion info during validation, we can try to
evaluate variable expressions to catch some basic errors. Do this by
creating module instance RepetitionData with unknown values. This
unfortunately will still miss the incorrect usage of count/each values,
but that would require the module call's each mode, which is not
available at this time.
The variable nodes are not only used during plan and apply, so remove
those from there names. The "plan" node is now
`nodeExpandModuleVariable` and the "apply" node is now just
`nodeModuleVariable`.
Remove unnecessary methods, as the nodeModuleVariable is no longer used
in the full graph transformations.
NodeModuleRemoved is redundant now with the concept of
nodeCloseModule, so we can replace it within the graph. This does mean
that nodeCloseModule needs to know if it's evaluating an orphaned module
that can't be expanded, but the overhead to checking this isn't too
bad.
Now that nodeModuleClose is referenceable, and we can ensure it's always
in the graph at the correct time, we can eliminate the need to connect
each resource to every single node within a module it references, and
instead connect only to the nodeModuleClose, which acts as the module
root. Since module expansion can cause exponential growth in the number
of edges in graphs, this will help with performance problems when
transforming and reducing these graphs by eliminating the bulk of
redundant edges. This will also help with general debugging, making the
graphs easier to read.
This is all that is required to make module reference ordering work
during apply, by adding and edge to the nodeCloseModule node, which will
be the last node evaluated in the module.
These will now use "default" provider addresses, rather than "legacy"
ones, so that they can cooperate with the rest of Terraform that has been
updated to no longer use legacy provider addresses.
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".
This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
* terraform: add helper functions for creating test state
testSetResourceInstanceCurrent and testSetResourceInstanceTainted are
wrapper functions around states.Module.SetResourceInstanceCurrent()
used to set a resource in state. They work with current, non-deposed
resources with no dependencies.
testSetResourceInstanceDeposed can be used to set a desosed resource in state.
* terraform: update all tests to use modern providers and state
These are cases where we were using the legacy string only to produce a
message to the user or to write to the log. It's enough to make some
basic Terraform commands like "terraform validate" not panic and get far
enough along to see that provider startup is working.
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.
However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.
The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.
This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
We cannot evaluate expansion during validation, since the values may not
be known at that time.
Inject a nodeValidateModule, using the "Concrete" pattern used for other
node types during graph building. This node will always evaluate to a
single module instance, so that we have a valid context within which to
evaluate all sub resources.
Make the expansion logic easier to follow, keeping the evaluation and
registration local to switch cases. We don't validate anything between
count or for_each (config loading should handle that), and we don't need
to keep relying on the count == -1 sentinel value.
Replace the graphNodeRoot for the main graph with a nodeCloseModule for
the root module. USe a new transformer as well, so as to not change any
behavior of DynamicExpand graphs.
Closing out the root module like we do with sub modules means we no
longer need the OrphanResourceTransformer, or the NodeDestroyResource.
The old resource destroy logic has mostly moved into the instance nodes,
and the remaining resource node was just for cleanup, which need to be
done again by the module since there isn't always a NodeDestroyResource
to be evaluated.
The more-correct state caused a few tests to fail, which need to be
cleaned up to match the state without empty resource husks.
There is not one more non-dependent type to look for when pruning unused
values. This fixes the oversight, but still leaves the ugly concrete
type checking which we need to remove.
During plan, anything dependent on a module can connect to the module
expansion node, because all instance nodes are created during
DynamicExpand. During apply the instance nodes are created from the
diff, so we need a root module to terminate the logical module subgraph.
Besides providing an anchor for the completion of a module, the
nodeCloseModule can also be used to cleanup the orphan resource and
module placeholders in the state.
NodeDestroyResource does not require a provider, and to avoid this a
temporary GraphNodeNoProvider was used to differentiate it from other
resource nodes. We can now de-couple the destroy node from the abstract
resource which was adding the ProvidedBy method, and remove the
NoProvider method.
Remove the shims where they aren't necessary from the Init and Close
provider nodes. This also removed some provider path checks from the
builtin eval context, which cannot be resolved since the context may not
be created with a ModuleInstance path.
Use the new addrs type here.
Also remove the uniqueMap from the config transformer. We enforce
uniqueness during config loading, and this is more likely to have false
positives due to stringification than anything.
While the Expander itself now handles the recursive expansion of
modules, Resources themselves still need to be expanded twice, because
the evaluation of the Resource, which entails evaluating the for_each or
count expressions, is separate from the ResourceInstance expansion.
Add a nodeExpandPlannableResource to do handle this expansion to allow
all NodePlannableResources to call EvalWriteResourceState with an
absolute address.
As the Graph is walked, the current way to set the context path was to
have the walker return a context from EnterPath. This required that
every node know it's absolute path, which can no longer be the case
during plan when modules have not been expanded.
This introduces a new method called WithPath, which returns a copy of
the context with the internal path updated to reflect the method
argument. Any use of the EvalContext that requires knowing the path will
now panic if it wasn't explicitly set to ensure that evaluations always
occur in the correct path.
Add EvalContext to the GraphWalker interface.
EvalContext returns an EvalContext that has not yet set a path. This
will allow us to enforce that all context operations requiring a module
instance path will require that a path be explicitly set rather than
evaluating within the wrong path.
This also calls ExpandModuleResource in one location, because the logic
is not yet updated to handle actual module expansion, but that will be
fixed in a forthcoming PR.
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.
Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
* import: remove Config from ImportOpts
`Config` in ImportOpts was any provider configuration provided by the
user on the command line. This option has already been removed in favor
of only taking the provider from the configuration loaded in the current
context.
* terrafrom: add Config to ImportStateTransformer and refactor Transform
to get the resource provider FQN from the Config
* terraform: large refactor to use Provider from configs.Resource
configs.Resource.ImpliedProvider() now returns a string; it is the
callers' responsibility to turn that into an addrs.Provider if needed.
GraphNodeProviderConsumer ProvidedBy() no longer returns nil (reverting
to earlier, pre-provider-fqn behavior): it will return either the
provider set in config, provider set in state, or the default provider.
Make the interface name reflect the new return type of the method.
Remove the confusingly named and unused ResourceAddress method from the
resource nodes as well.
Remove unused variables, sync.Once, and init in BuiltinEvalContext.
Replace some shim calls.
GraphNodeAttachProvider doesn't need to be a NodeModuleInstance.
* configs: parse provider source string during module merge
This was the smallest unit of work needed to start writing provider
source tests!
* Update configs/parser_test.go
Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Remove the requirement for most *Resource types to be a
GraphNodeModuleInstance, ensuring that they never call ctx.Path while
being evaluated. This gets rid of most of the direct need for the Path
method currently implemented by NodeResourceAbstract, leaving the
provider and schema related calls for a subsequent PR.
This adds more shimming into that node itself, but allows us to pull it
out of the config transformer, and ensure we can create the resources
correctly from the config. The shimmed address usage can then be raised
out of the abstract resource, into the expanded node types.
Using this in the same manner as NodePlannableOutput, which expands the
local values within modules. All thee output and local types are used in
both plan and apply, we may rename these to better reflect their usage
in expanding. That wait until we are certain that apply won't need any
extra machinery for handling values that aren't stored in the plan.
