Just like in the destroy apply, we can skip the inter-provider cycle
check when creating the destroy plan, which can be expensive when there
are a lot of resource instances with dependencies from another provider.
* checks: filter out check diagnostics during certain plans
* wrap diagnostics produced by check blocks in a dedicated check block diagnostic
* address comments
When we plan to destroy an instance, the change recorded should use the
correct type for the resource rather than `DynamicPseudoType`. Most of
the time this is hidden when the change is encoded in the plan, because
any `null` is always encoded to the same value, and when decoded it will
be converted to the schema type. However when apply requires creating a
second plan for an instance's replacement that value is not going to be
encoded, and remains a dynamic value which is sent to the provider.
Most providers won't see that either, as the grpc request also encodes
and decodes the value to conform with the correct schema. The builtin
terraform provider does get the raw cty value though, and when that
dynamic value is returned validation fails when the type does not match.
If a resource has a change in marks from the prior state, we need to
notify the user that an update is going to be necessary to at least
store that new value in the state. If the provider however returns the
prior state value in lieu of a new config value, we need to be sure to
filter any new marks for comparison as well. The comparison of the prior
marks and new marks must take into account whether those new marks could
even be applied, because if the value is unchanged the new marks may be
completely irrelevant.
* Add support for scoped resources
* refactor existing checks addrs and add check block addr
* Add configuration for check blocks
* introduce check blocks into the terraform node and transform graph
* address comments
* address comments
* don't execute checks during destroy operations
* don't even include check nodes for destroy operations
In the case where a provider has been upgraded, and there are external
changes to resources outside of terraform, and -target is being used,
and resources which are not targeted require a schema migration; the
untargeted resources will not have been migrated and cannot be decoded for the
external changes report.
Since there is no way to decode the resources which have been excluded
via -target, we can only skip over them when inspecting
driftedResources. Return warnings for now to indicate that these
resources could not be decoded to help indicate that users will need to
eventually apply these changes.
Module outputs are evaluated from state, so in order to have detailed
information about sensitivity from non-root module outputs, we need to
store the value along with all sensitive marks. This aligns with the
usage of state being the in-memory store for other temporary values like
locals and variables.
When planning encounters an error we were returning early without
cleaning out any planed data sources which cannot be serialized. Move
the cleanup to the common walkPlan method where the PriorState is
assigned so that it cannot be missed.
This test was previously not taking into account the fact that the
"Stopping" hook gets sent in the goroutine that calls ctx.Stop, whereas
all of the others get called from inside ctx.Apply, and so there are no
ordering guarantees for that event in relation to the others.
We now handle the stopping event as a special case that is allowed to
appear anywhere in the sequence as long as it appears. The other events
are still strongly ordered because their ordering is important for
correctness of Terraform Core's own behavior.
As some extra insurance we also now check whether the provider's
ApplyResourceChange and Stop functions both ran and reached a suitable
point of execution related to the stop request, which help to ensure not
only that something called Stop but that Terraform Core correctly
interacted with the provider to handle the stop.
This is a mostly mechanical refactor with a handful of changes which
are necessary due to the semantic difference between earlyconfig and
configs.
When parsing root and descendant modules in the module installer, we now
check the core version requirements inline. If the Terraform version is
incompatible, we drop any other module loader diagnostics. This ensures
that future language additions don't clutter the output and confuse the
user.
We also add two new checks during the module load process:
* Don't try to load a module with a `nil` source address. This is a
necessary change due to the move away from earlyconfig.
* Don't try to load a module with a blank name (i.e. `module ""`).
Because our module loading manifest uses the stringified module path
as its map key, this causes a collision with the root module, and a
later panic. This is the bug which triggered this refactor in the
first place.
Terraform Core emits a hook event every time it writes a change into the
in-memory state. Previously the local backend would just copy that into
the transient storage of the state manager, but for most state storage
implementations that doesn't really do anything useful because it just
makes another copy of the state in memory.
We originally added this hook mechanism with the intent of making
Terraform _persist_ the state each time, but we backed that out after
finding that it was a bit too aggressive and was making the state snapshot
history much harder to use in storage systems that can preserve historical
snapshots.
However, sometimes Terraform gets killed mid-apply for whatever reason and
in our previous implementation that meant always losing that transient
state, forcing the user to edit the state manually (or use "import") to
recover a useful state.
In an attempt at finding a sweet spot between these extremes, here we
change the rule so that if an apply runs for longer than 20 seconds then
we'll try to persist the state to the backend in an update that arrives
at least 20 seconds after the first update, and then again for each
additional 20 second period as long as Terraform keeps announcing new
state snapshots.
This also introduces a special interruption mode where if the apply phase
gets interrupted by SIGINT (or equivalent) then the local backend will
try to persist the state immediately in anticipation of a
possibly-imminent SIGKILL, and will then immediately persist any
subsequent state update that arrives until the apply phase is complete.
After interruption Terraform will not start any new operations and will
instead just let any already-running operations run to completion, and so
this will persist the state once per resource instance that is able to
complete before being killed.
This does mean that now long-running applies will generate intermediate
state snapshots where they wouldn't before, but there should still be
considerably fewer snapshots than were created when we were persisting
for each individual state change. We can adjust the 20 second interval
in future commits if we find that this spot isn't as sweet as first
assumed.
Resource instances with no current object in state should not have
orphan nodes added to the graph, as deposed objects are handled
separately. This was previously handled correctly for the non-expanded
case, but expanded resources were missing the appropriate check for a
current object.
Also update the comment in the non-expanded case to hopefully clarify
that we're checking for the presence of a current object, not the
absence of any deposed objects. An instance may have both a current
object and zero or more deposed objects in some circumstances, and if
so, we still want an orphan node to be added if the instance is not in
configuration.
* Add failing test case for the given issue
* pause
* don't use local when sending PR for review
* go get github.com/hashicorp/hcl/v2@v2.16.0
* Update go.mod
---------
Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Although they are not serialized to the final stored state, all module
outputs must be saved in the state for evaluation. There is no defined
schema which is used to identify the overall type of module outputs, so
all outputs must exist in the state to build the correct type for proper
evaluation.
Output references must also include the error_message expression.
Fix the early return in referencesForOutput, which could skip
preconditions. The small slice allocation optimization is not really
needed here, since this is not a hot path at all.
Legacy providers may return null values for nested blocks during
refresh. Because the ReadResource call needs to accept any value to
allow the provider to report external changes, we allowed all changes to
the value as long as the underlying cty.Type was correct, allowing
null block values to be inserted into the state.
While technically invalid, we needed to accept these null values for
compatibility, and they were mostly seen as a nuisance, causing noise in
external changes and plan output. These null block values however can be
inserted into the effective configuration with the use of
`ignore_changes`, which can cause problems where the configuration is
assumed to be completely valid.
Rather than accept the null values, we can insert empty container values
for these blocks when refreshing the instance, which will prevent any
invalid values from entering state at all. Because these must still be
accepted for compatibility, we can only log the difference as a warning.
Currently the NormalizeObjectFromLegacySDK does not report which
specific blocks it fixed, so we just log a generic message.
In any situation where we return a plan object along with some errors
we'll also explicitly annotate the plan object as being errored so that
we can catch if someone accidentally tries to apply that incomplete plan.
At the moment this situation is impossible to reach but in a later commit
we'll make it possible to save errored plans to disk for further
inspection, at which point it'll become important to not allow applying
them.
For some kinds of plan failure we will already have successfully completed
planning for at least one upstream object before encountering a downstream
error.
Since a downstream failure can be caused by an already-recorded action
from upstream, it might be helpful to inspect the actions planned so far
in order to understand better why the error occurred.
This doesn't yet make this result visible anywhere, and is backward
compatible with existing callers because they currently entirely ignore
the returned plan pointer if the diagnostics contains at least one error.
Outputs were being evaluated from changes, even during apply. Make sure
we update the state correctly, and remove the existing change. This
requires adding more Planning fields to the output nodes to
differentiate whether the output is being planned or applied because the
same type handles both cases. We can evaluate separately whether new
types should be introduced to deal with both cases.
The module node cleanup was also prematurely removing module outputs
from the state before evaluation. This was not noticed before because
the evaluation was always falling back to changes. Have the root module
node do the final cleanup for all its children.
It turns out sensitive was also being handled incorrectly, and only
sensitive from configuration was being considered. Make sure to mark the
output as sensitive when storing sensitive values into state, and OR
sensitive marks with the state config when evaluating the output values.
When output values are updated in the refreshed state, we don't need to
re-set the changes which were already set in conjunction with the
current state.
Once the ProviderTransformer has resolved and set the exact provider,
the ProvidedBy method should return that exact provider again.
We can hoist the stored provider addr into the AbstractInstance and
avoid the method duplication and slight differences between the
implementations.
If when refreshing an orphaned instance the provider indicates it has
already been deleted, there is no reason to create a change for that
instance. A NoOp change should only represent an object that exists and
is not changing.
This was likely left in before in order to try and provide a record of
the change for external consumers of the plan, but newer plans also
contain all changes made outside of Terraform which better accounts for
the difference. The NoOp change now can cause problems, because it may
represent an instance with conditions to check even though that instance
does not exist.
In a heavily-connected graph with lots of inter-dependent providers, the
cycle checks for destroy edges across providers can seriously impact
performance. Since the specific cases we need to avoid will involve
create/update nodes, skip the extra checks during a full destroy
operation. Once we find a way to better track these dependencies, the
transformer will not need to do the cycle checks in the first place.
Data resource dependencies are not stored in the state, so we need to
take the latest dependency set to use for any direct connections to
destroy nodes.
Some prior refactors left the detroyPlan method a bit confusing, and ran
into a case where the previous run state could be returned as nil.
Get rid of the no longer used pendingPlan value, and track the prior and
prev states directly, making sure we always have a value for both.
In order to complete the terraform destroy command, a refresh must first
be done to update state and remove any instances which have already been
deleted externally. This was being done with a refresh plan, which will
avoid any condition evaluations and avoid planning new instances. That
however can fail due to invalid references from resources that are
already missing from the state.
A new plan type to handle the concept of the pre-destroy-refresh is
needed here, which should probably be incorporated directly into the
destroy plan, just like the original refresh walk was incorporated into
the normal planning process. That however is major refactoring that is
not appropriate for a patch release.
Instead we make two discrete changes here to prevent blocking a destroy
plan. The first is to use a normal plan to refresh, which will enable
evaluation because missing and inconsistent instances will be planned
for creation and updates, allowing them to be evaluated. That is not
optimal of course, but does revert to the method used by previous
Terraform releases until a better method can be implemented.
The second change is adding a preDestroyRefresh flag to the planning
process. This is checked in any location which evalCheckRules is called,
and lets us change the diagnosticSeverity of the output to only be
warnings, matching the behavior of a normal refresh plan.
