Commit Graph

1595 Commits

Author SHA1 Message Date
Martin Atkins
efdc6e52bc cloud: Skip intermediate state snapshots in Terraform Cloud/Enterprise
We've seen some concern about the additional storage usage implied by
creating intermediate state snapshots for particularly long apply phases
that can arise when managing a large number of resource instances together
in a single workspace.

This is an initial coarse approach to solving that concern, just restoring
the original behavior when running inside Terraform Cloud or Enterprise
for now and not creating snapshots at all.

This is here as a solution of last resort in case we cannot find a better
compromise before the v1.5.0 final release. Hopefully a future commit
will implement a more subtle take on this which still gets some of the
benefits when running in a Terraform Enterprise environment but in a way
that will hopefully be less concerning for Terraform Enterprise
administrators.

This does not affect any other state storage implementation except the
Terraform Cloud integration and the "remote" backend's state storage when
running inside a TFC/TFE-driven remote execution environment.
2023-05-23 15:25:48 -07:00
Martin Atkins
8884bef59d backend/local: Allow storage impls to customize intermediate persistence
Previously we just always used the same intermediate state persistence
behavior for all state storages. However, some storages might have access
to additional information that allows them to tailor when they persist,
such as reacting to API rate limit status headers in responses, or just
knowing that a particular storage isn't suited to intermediate snapshots
at all for some reason.

This commit doesn't actually change any observable behavior yet, but it
introduces an optional means for a state storage to customize the behavior
which we may make use of in certain storage implementations in future
commits.
2023-05-23 15:25:48 -07:00
CJ Horton
258bdbe89f
Merge pull request #33238 from hashicorp/radditude/import-plan-plumbing
plannable import: correct plumbing when using a plan output file
2023-05-23 10:36:19 -07:00
James Bardin
2f308cf948
Merge pull request #32962 from hashicorp/jbardin/validate-unknown-coll-attrs
validate unknown nested attribute collections
2023-05-23 11:38:13 -04:00
CJ Horton
40ff381887 plumb import changes to and from binary plan 2023-05-22 22:19:42 -07:00
James Bardin
0a921976cd destroy locals referenced by root outputs
When planning a destroy operations, locals only referenced by root
outputs do not need to be kept in the graph, because the root output
does not get evaluated. Rather than try and prune the local based on
this condition, we can prevent the connection from being created by
ensuring that a root output destroy node has no references.

The separate plan+apply destroy fields used for outputs can be
simplified by combining, since they are only ever referenced together.
2023-05-22 13:03:49 -04:00
Masayuki Morita
53755180fd
Fix an error message for import block with moved block (#33221)
Fixes #33220
2023-05-19 13:47:46 -07:00
kmoe
4015f1aa30
genconfig: do not generate null NestingSingle blocks (#33213)
* genconfig: fix nil nested block panic

* always InternalValidate test schemas

* genconfig: null NestingSingle blocks should be absent

A NestingSingle block that is null in state should be completely absent from config.
2023-05-19 11:32:28 -07:00
kmoe
b4d1146f58
plannable import: improve gen config human plan output (#33194)
* renderer: remove hard-coded config gen path

* mention config gen file in plan next steps
2023-05-15 15:21:41 +01:00
kmoe
789e30dfc5
error if importing to invalid keyed address (#33191)
Import addresses targeting expanded resource instances must target instances that already exist in configuration.
2023-05-13 00:57:51 +01:00
CJ Horton
adcecddb4f
Merge pull request #33193 from hashicorp/radditude/no-import-blocks-in-child-modules
plannable import: disallow import blocks in child modules
2023-05-12 16:44:12 -07:00
kmoe
1172d40d7b
error if import target is move source (#33192)
It is invalid for any import block to have a "to" argument matching any moved block's "from" argument.
2023-05-13 00:30:15 +01:00
CJ Horton
2dd89d9776 import blocks are only allowed in the root module 2023-05-12 16:04:47 -07:00
CJ Horton
bd6ba6cf99
check for duplicate import blocks (#33190)
Importing to the same target address twice or importing the same ID
to multiple different resources of the same type is not allowed.
2023-05-12 23:14:44 +01:00
Liam Cervante
d5fed58fc5
plannable import: write generated config to out flag (#33186)
* plannable import: write generated config to out flag

* Add example command to diagnostic
2023-05-12 23:05:00 +01:00
kmoe
2b71e9edf3
terraform: config-driven import is idempotent (#33188)
If a resource is already in state, do not attempt to import it again. Resources already in state are filtered out of the plan's import targets.

A change is only considered "importing" if it is adding a new resource instance to the state.
2023-05-12 21:31:29 +01:00
CJ Horton
5d7864316e
Merge pull request #33160 from hashicorp/radditude/apply-counts
Populate import counts during applies and clean up output
2023-05-12 09:33:33 -07:00
CJ Horton
e5a6806206 clarify apply hook usage 2023-05-11 19:02:59 -07:00
Liam Cervante
cd06543b39
plannable import: fix config generation printing empty collections instead of null values (#33183) 2023-05-11 20:18:25 +02:00
Liam Cervante
192cb255a6
checks: no longer experimental (#33184) 2023-05-11 20:17:49 +02:00
Liam Cervante
5d6c5a9a33
plannable import: add a provider argument to the import block (#33175)
* command: keep our promises

* remove some nil config checks

Remove some of the safety checks that ensure plan nodes have config attached at the appropriate time.

* add GeneratedConfig to plan changes objects

Add a new GeneratedConfig field alongside Importing in plan changes.

* add config generation package

The genconfig package implements HCL config generation from provider state values.

Thanks to @mildwonkey whose implementation of terraform add is the basis for this package.

* generate config during plan

If a resource is being imported and does not already have config, attempt to generate that config during planning. The config is generated from the state as an HCL string, and then parsed back into an hcl.Body to attach to the plan graph node.

The generated config string is attached to the change emitted by the plan.

* complete config generation prototype, and add tests

* plannable import: add a provider argument to the import block

* Update internal/configs/config.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/configs/config.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/configs/config.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* fix formatting and tests

---------

Co-authored-by: Katy Moe <katy@katy.moe>
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
2023-05-11 09:04:39 +02:00
Liam Cervante
4d837df546
Plannable import: Add generated config to JSON and human-readable plan output (#33154)
* command: keep our promises

* remove some nil config checks

Remove some of the safety checks that ensure plan nodes have config attached at the appropriate time.

* add GeneratedConfig to plan changes objects

Add a new GeneratedConfig field alongside Importing in plan changes.

* add config generation package

The genconfig package implements HCL config generation from provider state values.

Thanks to @mildwonkey whose implementation of terraform add is the basis for this package.

* generate config during plan

If a resource is being imported and does not already have config, attempt to generate that config during planning. The config is generated from the state as an HCL string, and then parsed back into an hcl.Body to attach to the plan graph node.

The generated config string is attached to the change emitted by the plan.

* complete config generation prototype, and add tests

* Plannable import: Add generated config to json and human-readable plan output

---------

Co-authored-by: Katy Moe <katy@katy.moe>
2023-05-11 08:50:03 +02:00
Liam Cervante
79f7f59155
Plannable import: Generate config for imported resources during the plan. (#33153)
* command: keep our promises

* remove some nil config checks

Remove some of the safety checks that ensure plan nodes have config attached at the appropriate time.

* add GeneratedConfig to plan changes objects

Add a new GeneratedConfig field alongside Importing in plan changes.

* add config generation package

The genconfig package implements HCL config generation from provider state values.

Thanks to @mildwonkey whose implementation of terraform add is the basis for this package.

* generate config during plan

If a resource is being imported and does not already have config, attempt to generate that config during planning. The config is generated from the state as an HCL string, and then parsed back into an hcl.Body to attach to the plan graph node.

The generated config string is attached to the change emitted by the plan.

* complete config generation prototype, and add tests

---------

Co-authored-by: Katy Moe <katy@katy.moe>
2023-05-11 08:38:37 +02:00
CJ Horton
bc084858b1 add import hooks for plan and apply
Separate hooks used for the legacy import command for those used by
the new import mechanism; also add apply output for imports.
2023-05-10 20:53:44 -07:00
CJ Horton
9904f62bfd
Merge pull request #33171 from hashicorp/revert-33155-liamcervante/plannable-import/streamed-logs
Revert "Plannable import: Make the streamed logs more consistent during planning"
2023-05-10 20:53:14 -07:00
James Bardin
2e0efe7321
Merge pull request #33047 from hashicorp/jbardin/destroy-provider-pruning
prune unused providers within modules
2023-05-10 11:54:10 -04:00
Liam Cervante
2793af042c Revert "Plannable import: Make the streamed logs more consistent during a plan operation (#33155)"
This reverts commit 3c20f7b340.
2023-05-10 11:00:45 +02:00
Liam Cervante
3c20f7b340
Plannable import: Make the streamed logs more consistent during a plan operation (#33155) 2023-05-10 08:27:15 +02:00
hashicorp-copywrite[bot]
bb36298b21 [COMPLIANCE] Add Copyright and License Headers 2023-05-08 15:47:42 -07:00
Rees Pozzi
fed6538dec
core: Correct typo in "Moved resource instances excluded by targeting" error message 2023-05-05 13:36:05 -07:00
James Bardin
76737a8966
Merge pull request #33151 from hashicorp/jbardin/import-refresh
Import: only refresh an imported state once
2023-05-05 08:09:33 -04:00
James Bardin
be682f1d29 only refresh an import state once
The imported resource was being stored in the wrong state, and only
ended up in the refresh state because ReadResource was being called a
second time in the normal refresh path.

Make sure to only refresh the imported resource once. This is still done
separately within importState so that we can handle the error slightly
differently to let the user know if an imported instance does not exist.
2023-05-04 14:20:45 -04:00
Liam Cervante
77f10c4f68
Imports should come before Adds in change summaries (#33147) 2023-05-04 15:06:48 +02:00
Liam Cervante
81eb73731d
[Plannable Import] Implement streamed logs for plan (#33106)
* [plannable import] embed the resource id within the changes

* [Plannable Import] Implement streamed logs for -json plan

* use latest structs

* remove implementation plans from TODO
2023-05-04 10:02:06 +02:00
Liam Cervante
54c1c1162f
[Plannable Import] Implement human-readable plan rendering (#33113)
* [plannable import] embed the resource id within the changes

* add the plannable imports to the json and human plans

* latest importing struct
2023-05-03 18:50:04 +02:00
Alisdair McDiarmid
b5658a46a2 Rebuild protobuf 2023-05-02 11:44:23 -04:00
hashicorp-copywrite[bot]
325d18262e [COMPLIANCE] Add Copyright and License Headers 2023-05-02 15:33:06 +00:00
Liam Cervante
4210d905c0
[plannable import] embed the resource id within the changes (#33134)
* [plannable import] embed the resource id within the changes

* make pointers and update docs
2023-05-02 16:04:51 +02:00
Liam Cervante
b5576159da
Migrate 'state show' command to new renderer (#33116)
* Migrate 'state show' command to new renderer

* handle error
2023-05-02 15:27:59 +02:00
James Bardin
2c09ae4f3d prune unused providers within modules
The logic used to prune unused providers was only taking into account
the common case of providers in the root module. The quick check of
looking for up edges doesn't work within a module, because the module
structures will create non-resource nodes connected to the providers.
Use a deeper check of looking for any dependent resources which may
require that provider to be configured.
2023-05-01 10:38:13 -04:00
kmoe
28643516b2
Plannable import 3: Make import plannable (#33085)
During a plan, Terraform now checks for the presence of import blocks.

For each resource in config, if an import block is present with a matching address, planning that node will now trigger an ImportResourceState and ReadResource. The resulting state is treated as the node's "refresh state", and planning proceeds as normal from there.

The walkImport operation is now only used for the legacy "terraform import" CLI command. This is the only case under which the plan should produce graphNodeImportStates.
2023-04-28 23:45:43 +01:00
Roberto Hidalgo
1b7f772f8b
fix breaking code
this is what i get for submitting PRs late at night
2023-04-27 10:13:38 -06:00
Roberto Hidalgo
0dda1d5c61
Return debuggeble CAS errors on consul state put
ran into this error while running terraform on a container and saving state to Consul. I suspect my policy needs tweaking but it's impossible to tell with an error like this:

```
╷
│ Error: Failed to save state
│ 
│ Error saving state: consul CAS failed with transaction errors:
│ [0xc0006e93c8]
╵
```

This PR makes the will include the error messaage in the details so I can continue debugging
2023-04-26 23:44:39 -06:00
Zac Clifton
3bb6a5896b
Feat: Add strcontains function and documentation (#33069)
* add strcontains function and documentation
2023-04-26 20:41:32 +01:00
Alisdair McDiarmid
0ffdd07980 Fix goimports discrepancies 2023-04-26 14:33:13 -04:00
kmoe
c6400fabb1
configs: add import block (#33081) 2023-04-26 16:28:11 +01:00
kmoe
531efd303b
add types for plannable import (#33080) 2023-04-25 15:19:48 +01:00
Lauren
50c3f53595 add resource_drift logtype 2023-04-24 15:08:33 -04:00
James Bardin
d47d475a03
Merge pull request #33079 from hashicorp/jbardin/destroy-plan-faster-providers
skip inter-provider cycle check in destroy plan
2023-04-24 12:29:05 -04:00
James Bardin
583350a5c4 skip inter-provider cycle check in destroy plan
Just like in the destroy apply, we can skip the inter-provider cycle
check when creating the destroy plan, which can be expensive when there
are a lot of resource instances with dependencies from another provider.
2023-04-24 12:12:40 -04:00
Liam Cervante
b54668f44c
make the unknown attribute renderer consider the force replacement metadata (#33065) 2023-04-24 11:02:32 +02:00
Liam Cervante
14123e277c
Include sensitive metadata from the schema when building the json state output (#33059)
* include sensitive metadata from the schema when building the json state output

* found another test case
2023-04-24 10:52:44 +02:00
Liam Cervante
519a18aedf
Propagate unknown and sensitive metadata to dynamic attributes (#33057)
* propagate unknown and sensitive metadata to dynamic attributes

* update goimports and add some comments
2023-04-24 10:28:21 +02:00
Liam Cervante
357012a2f3
Refactor of differ to make code reuse easier (#33054)
* refactor of differ to make code reuse easier

* fix imports
2023-04-21 09:51:55 +02:00
Liam Cervante
aae6990d38
Add native plantimestamp function to Terraform (#32980)
* add plantimestamp function

* documentation

* add missing links

* fix typo
2023-04-21 09:47:28 +02:00
Rémi Lapeyre
af571b2642
Improve environment variable support for the pg backend (#33045)
* Improve environment variable support for the pg backend

This patch does two things:
  - it adds environment variable support to the parameters that did
    not have it (and uses `PG_CONN_STR` instead of `PGDATABASE` which is
    actually more appropriate to match the behavior of other PostgreSQL
    utilities)
  - better documents how to give the connection parameters as environment
    variables for the ones that were already supported based on the
	recommendation of @bsouth00

I will prepare a backport of the documentation part of this once it is
merged.

Closes https://github.com/hashicorp/terraform/issues/33024

* Remove global variable in test of the PG backend
2023-04-21 08:39:19 +02:00
Glenn Sarti
7e2e834aff
Emit warnings for certain run events in cloud backend (#33020)
The cloud backend, which communicates with TFC like APIs, can create
runs which may have one more configuration parameters altered. These
alterations are emitted as run-events on the run so that API clients
can consume and display them to users. This commit adds a step in
plan operation to query the run-events once a run is created and then
emit specific run-event descriptions to the console as warnings for
the user.
2023-04-17 08:53:47 -07:00
Sebastian Rivera
e6c3aab6c5
Merge pull request #33018 from hashicorp/tf-5529-sro-tfe-version-check 2023-04-14 11:55:12 -04:00
Liam Cervante
2c624acea1
Fix rendering unknown values in map and null string primitives (#33029)
* fix rendering unknown values in map and null string primitives

* Update map.go

* fix code consistency checks
2023-04-14 09:56:32 +02:00
Sebastian Rivera
300a60f393 Fix typo in format version check 2023-04-13 18:30:56 -04:00
Sebastian Rivera
5634ae3e18 Unit tests to ensure renderer is appropriately called 2023-04-13 18:30:56 -04:00
Sebastian Rivera
b23cfaefe8 Refactor SRO check to prevent duplicate plan output 2023-04-13 18:30:56 -04:00
Liam Cervante
9c87006c34
checks: hide check diagnostics during plans that will not wait for approval (#32938)
* checks: filter out check diagnostics during certain plans

* wrap diagnostics produced by check blocks in a dedicated check block diagnostic

* address comments
2023-04-11 10:54:30 +02:00
James Bardin
acbcbcb311
Merge pull request #32988 from hashicorp/jbardin/destroy-plan-null-type
the destroy plan should use correct type
2023-04-06 09:53:55 -04:00
janaurka
1f603b1a7f
backend/pg: Accept connection string in PGDATABASE environment variable 2023-04-05 15:42:44 -07:00
James Bardin
c872cd6d96 the destroy plan should use correct type
When we plan to destroy an instance, the change recorded should use the
correct type for the resource rather than `DynamicPseudoType`. Most of
the time this is hidden when the change is encoded in the plan, because
any `null` is always encoded to the same value, and when decoded it will
be converted to the schema type. However when apply requires creating a
second plan for an instance's replacement that value is not going to be
encoded, and remains a dynamic value which is sent to the provider.

Most providers won't see that either, as the grpc request also encodes
and decodes the value to conform with the correct schema. The builtin
terraform provider does get the raw cty value though, and when that
dynamic value is returned validation fails when the type does not match.
2023-04-05 10:19:26 -04:00
Liam Cervante
84dc498b90
checks: always reference the nested data source from the check graph node (#32946)
* checks: always reference the nested data source from the check block graph node

* goimports
2023-04-05 08:48:42 +02:00
James Bardin
8ab7af8c5f validate unknown nested attribute collections
It is not valid for a provider to return an unknown value for a
configured nested collection, but we need to check for unknowns before
comparing the number of values in the collection.
2023-04-03 13:31:36 -04:00
James Bardin
81b74cdb22 don't compare plan marks for missing values
If a resource has a change in marks from the prior state, we need to
notify the user that an update is going to be necessary to at least
store that new value in the state. If the provider however returns the
prior state value in lieu of a new config value, we need to be sure to
filter any new marks for comparison as well. The comparison of the prior
marks and new marks must take into account whether those new marks could
even be applied, because if the value is unchanged the new marks may be
completely irrelevant.
2023-03-28 15:55:15 -04:00
James Bardin
a4e92f3fca
Merge pull request #32876 from hashicorp/jbardin/state-serialize-plan-error
Remove planned data source objects from state on error
2023-03-28 15:50:59 -04:00
James Bardin
fdb00b9a46
Merge pull request #32900 from hashicorp/jbardin/target-drift-upgrade
External changes report can fail with schema migrations and `-target`
2023-03-28 15:50:38 -04:00
Liam Cervante
c06db2aadd
checks: don't iterate through all the nodes when there is nothing to search for (#32927) 2023-03-28 18:14:27 +02:00
Liam Cervante
5f97f88025
ensure checks are reported before executed (#32925) 2023-03-28 15:20:54 +02:00
Liam Cervante
978263efe9
Checks: Introduce check blocks into the terraform node and transform graph (#32735)
* Add support for scoped resources

* refactor existing checks addrs and add check block addr

* Add configuration for check blocks

* introduce check blocks into the terraform node and transform  graph

* address comments

* address comments

* don't execute checks during destroy operations

* don't even include check nodes for destroy operations
2023-03-23 16:07:31 +01:00
Liam Cervante
3827120c25
Checks: Add configuration for check blocks (#32734)
* Add support for scoped resources

* refactor existing checks addrs and add check block addr

* Add configuration for check blocks

* address comments
2023-03-23 09:12:53 +01:00
Liam Cervante
87c457781d
Checks: Refactor existing check addrs and add new check block addr (#32733)
* Add support for scoped resources

* refactor existing checks addrs and add check block addr

* address comments
2023-03-23 09:04:21 +01:00
James Bardin
240e345b45 test for untargeted schema mismatch 2023-03-21 12:55:26 -04:00
James Bardin
b10c4c54d9 prevent errors with -target and provider upgrades
In the case where a provider has been upgraded, and there are external
changes to resources outside of terraform, and -target is being used,
and resources which are not targeted require a schema migration; the
untargeted resources will not have been migrated and cannot be decoded for the
external changes report.

Since there is no way to decode the resources which have been excluded
via -target, we can only skip over them when inspecting
driftedResources. Return warnings for now to indicate that these
resources could not be decoded to help indicate that users will need to
eventually apply these changes.
2023-03-21 12:49:59 -04:00
James Bardin
defd7f0cde test that module outputs maintain sensitive marks 2023-03-20 14:22:08 -04:00
James Bardin
d33e627514 remove old comments 2023-03-20 14:22:08 -04:00
James Bardin
425c6bead2 store non-root sensitive outputs in state
Module outputs are evaluated from state, so in order to have detailed
information about sensitivity from non-root module outputs, we need to
store the value along with all sensitive marks. This aligns with the
usage of state being the in-memory store for other temporary values like
locals and variables.
2023-03-20 13:27:53 -04:00
James Bardin
1ca631bda0 remove planned objects from state on error
When planning encounters an error we were returning early without
cleaning out any planed data sources which cannot be serialized. Move
the cleanup to the common walkPlan method where the PriorState is
assigned so that it cannot be missed.
2023-03-17 08:45:42 -04:00
Martin Atkins
9f827f57ae go.mod: Revert cty minor version v1.13 upgrade
We inadvertently incorporated the new minor release of cty into the 1.4
branch, and that's introduced some more refined handling of unknown values
that is too much of a change to introduce in a patch release.

Therefore this reverts back to the previous minor release for the v1.4
series, and then we'll separately get the main branch ready to work
correctly with the new cty before Terraform v1.5.

This reverts just the upgrade and the corresponding test changes from
#32775, while retaining the HCL upgrade and the new test case it
introduced for that bug it was trying to fix. That new test is still
passing so it seems that the cty upgrade is not crucial to that fix.
2023-03-16 11:48:41 -04:00
Martin Atkins
4e6e0bc47b core: Fix test flake in TestContext2Apply_stop
This test was previously not taking into account the fact that the
"Stopping" hook gets sent in the goroutine that calls ctx.Stop, whereas
all of the others get called from inside ctx.Apply, and so there are no
ordering guarantees for that event in relation to the others.

We now handle the stopping event as a special case that is allowed to
appear anywhere in the sequence as long as it appears. The other events
are still strongly ordered because their ordering is important for
correctness of Terraform Core's own behavior.

As some extra insurance we also now check whether the provider's
ApplyResourceChange and Stop functions both ran and reached a suitable
point of execution related to the stop request, which help to ensure not
only that something called Stop but that Terraform Core correctly
interacted with the provider to handle the stop.
2023-03-15 13:44:33 -07:00
James Bardin
86c0c3bf80
Merge pull request #32846 from hashicorp/fix-32793
emit a warning msg if invalid CLI configuration file location
2023-03-14 14:58:47 -04:00
Reda Khaled
bf723bd65a fix(init_test.go): in TestInitProvidersLocalOnly tst, set TF_CLI_CONFIG_FILE env var to an empty val 2023-03-14 14:47:43 -04:00
Reda Khaled
fa9d044c58 bugfix(cliconfig.go): emit a warning msg if bad CLI config file path 2023-03-14 14:47:43 -04:00
James Bardin
06f1ceb026 add some output values to the show state test 2023-03-14 10:04:37 -04:00
James Bardin
5ed7f2a798 always use the RenderHuman constructor 2023-03-14 10:04:28 -04:00
Liam Cervante
15ecdb66c8
Fix no-op outputs causing the plan renderer to skip the 'no changes' message (#32820)
* Fix no-op outputs causing the plan renderer to skip the 'no changes' message

* fix imports
2023-03-10 16:19:33 +01:00
James Bardin
843befff29
Merge pull request #32818 from hashicorp/jbardin/nil-plan
return early from opPlan when the plan is nil
2023-03-10 09:46:55 -05:00
James Bardin
c02e7e8754 return early from opPlan when the plan is nil
While the returned plan is checked for nil in most cases, there was
a single point where the plan was dereferenced which could panic. Rather
than always guarding the dereferences, return early when the plan is
nil.
2023-03-10 09:34:47 -05:00
Liam Cervante
af05cbb645
Add support for scoped resources (#32732) 2023-03-10 11:11:10 +01:00
Liam Cervante
32f151f20b
Update HCL to handle type mismatches when applying defaults (#32775)
* add test reproducing terraform crash

* pull latest hcl

* add missing go.sum file

* also fix tests broken by go-cty update
2023-03-09 14:29:58 +01:00
Eugene Dementyev
36aac6b498
Fixes depends_on when overriding modules (#32796)
Fixes #32795
2023-03-09 11:53:13 +01:00
Alisdair McDiarmid
b088c67c60
Merge pull request #32781 from hashicorp/alisdair/rip-earlyconfig
initwd: Port from `earlyconfig` to `configs`
2023-03-06 15:02:06 -05:00
Alisdair McDiarmid
584811b829 initwd: Add fake range for fake root module
This isn't currently used anywhere downstream, but it easily could be in
the future, so populating the range with some zero-ish data is more
robust.
2023-03-06 14:35:06 -05:00
Alisdair McDiarmid
ca53ca00f4 Remove failing DNS query in cloud tests
This test case was making a real DNS call in a non-acceptance test, and
since it was intended to fail it would introduce a several second delay.
This commit replaces the test with a similar one which uses the mocked
disco services for a non-TFE host.

Also restructure the test to use t.Run for clarity.
2023-03-06 09:32:03 -05:00
Alisdair McDiarmid
19e635bfc8 Remove polling delays in run tasks test
These delays are not relevant to the test and result in increased test
execution time.
2023-03-06 09:14:59 -05:00
Alisdair McDiarmid
d9eae48a75 Remove redundant CheckCoreVersionRequirements 2023-03-06 09:14:28 -05:00
Alisdair McDiarmid
60ea68edc7 Remove earlyconfig 2023-03-06 09:14:28 -05:00
Alisdair McDiarmid
8df065a2fe initwd: Switch from earlyconfig to configs
This is a mostly mechanical refactor with a handful of changes which
are necessary due to the semantic difference between earlyconfig and
configs.

When parsing root and descendant modules in the module installer, we now
check the core version requirements inline. If the Terraform version is
incompatible, we drop any other module loader diagnostics. This ensures
that future language additions don't clutter the output and confuse the
user.

We also add two new checks during the module load process:

* Don't try to load a module with a `nil` source address. This is a
  necessary change due to the move away from earlyconfig.

* Don't try to load a module with a blank name (i.e. `module ""`).
  Because our module loading manifest uses the stringified module path
  as its map key, this causes a collision with the root module, and a
  later panic. This is the bug which triggered this refactor in the
  first place.
2023-03-06 09:14:28 -05:00
kmoe
76b34e891c
remove unused actionforchange (#32759) 2023-03-01 14:15:26 +00:00
Reda Khaled
100c44b6c3
bugfix: issue-28274, terraform providers mirror command should honor terraform lock file 2023-02-25 00:16:46 +01:00
Martin Atkins
a86cef4d50 cliconfig: Allow breaking the dependency lock file using the environment
Since it's already possible to activate the dependency lock file using an
environment variable, we should allow opting in to it having broken
behavior using the environment too.

It's kinda odd in retrospect that TF_PLUGIN_CACHE_DIR is the only setting
we allow to be configured both in the environment and the CLI
configuration. That means that the infrastructure for dealing with that
situation was relatively immature here and so I did some light refactoring
to make it unit-testable without actually modifying the test program's
environment.
2023-02-22 12:23:56 -08:00
Brandon Croft
3d1a58d5b5
Merge pull request #32708 from mrinalirao/mr/taskStage-race
Bug Fix where CLI exits if run is not confirmable.
2023-02-21 18:17:51 -07:00
CJ Horton
3c54e42080
Merge pull request #32695 from hashicorp/radditude/init-config-warning
keep errors friendly when init encounters syntax problems
2023-02-21 16:09:05 -08:00
Brandon Croft
ec4f62078a
Merge pull request #32666 from hashicorp/sebasslash/fix-credentials-sourcing-backend
Fix SRO authorization bug when token is fetched from cloud config
2023-02-21 16:40:53 -07:00
kmoe
b435b4ccde
cliconfig: more provider_installation err detail (#32722) 2023-02-21 18:18:57 +00:00
Sebastian Rivera
3f23a9e70a
Merge pull request #32696 from hashicorp/sebasslash/sro-provisioner-logs
Handle provisioner log types when rendering structured logs
2023-02-21 11:44:16 -05:00
Daniel Banck
f29156cdef
Replace HTML entities in function descriptions (#32710) 2023-02-20 14:11:06 +00:00
Mrinali Rao
afc9235f17
Update internal/cloud/backend_taskStages.go
Co-authored-by: Glenn Sarti <glennsarti@users.noreply.github.com>
2023-02-20 18:35:58 +11:00
mrinalirao
3104811c7d add test to see if override continues to poll on success 2023-02-20 17:50:25 +11:00
mrinalirao
190858b9e4 Bug Fix where CLI exits if run is not confirmable. 2023-02-20 17:18:13 +11:00
James Bardin
014a425d0e
Merge pull request #32683 from hashicorp/jbardin/import-terraform-data
allow terraform_data to import
2023-02-17 09:51:02 -05:00
Sebastian Rivera
6d0a191ec4 Handle provisioner log types in renderer 2023-02-16 10:54:58 -05:00
CJ Horton
30f8b014f8 keep the friendly error message whenever possible 2023-02-15 21:40:46 -08:00
CJ Horton
727e22e762 add tests for init syntax error handling
With the demise of the early config loader, we want to show core
version errors first, followed by backend errors, and only then
show other errors with the configuration.
2023-02-15 21:01:27 -08:00
Martin Atkins
f0de9b60c1 backend/local: Periodically persist intermediate state snapshots
Terraform Core emits a hook event every time it writes a change into the
in-memory state. Previously the local backend would just copy that into
the transient storage of the state manager, but for most state storage
implementations that doesn't really do anything useful because it just
makes another copy of the state in memory.

We originally added this hook mechanism with the intent of making
Terraform _persist_ the state each time, but we backed that out after
finding that it was a bit too aggressive and was making the state snapshot
history much harder to use in storage systems that can preserve historical
snapshots.

However, sometimes Terraform gets killed mid-apply for whatever reason and
in our previous implementation that meant always losing that transient
state, forcing the user to edit the state manually (or use "import") to
recover a useful state.

In an attempt at finding a sweet spot between these extremes, here we
change the rule so that if an apply runs for longer than 20 seconds then
we'll try to persist the state to the backend in an update that arrives
at least 20 seconds after the first update, and then again for each
additional 20 second period as long as Terraform keeps announcing new
state snapshots.

This also introduces a special interruption mode where if the apply phase
gets interrupted by SIGINT (or equivalent) then the local backend will
try to persist the state immediately in anticipation of a
possibly-imminent SIGKILL, and will then immediately persist any
subsequent state update that arrives until the apply phase is complete.
After interruption Terraform will not start any new operations and will
instead just let any already-running operations run to completion, and so
this will persist the state once per resource instance that is able to
complete before being killed.

This does mean that now long-running applies will generate intermediate
state snapshots where they wouldn't before, but there should still be
considerably fewer snapshots than were created when we were persisting
for each individual state change. We can adjust the 20 second interval
in future commits if we find that this spot isn't as sweet as first
assumed.
2023-02-14 15:17:56 -08:00
James Bardin
3b953d3bd8 allow terraform_data to import
The terraform provider was panicking on import, because it didn't
previously have a resource type which could be imported at all. Add a
stub import function for terraform_data as a placeholder to allow the
call to complete successfully. While there's no need to actually import
a terraform_data resource, users will inevitably use this to construct
examples of import actions for learning purposes or bug reports.

This still isn't very useful even for examples however, because the
state-only nature of the terraform_data resource type means that we
can't fill in the state from only the import ID. This means that any
value in `trigger_replace` or `input` will cause a change in the next
plan. Once configuration data is available during import we can extend
this to create a logical final state based on config.
2023-02-14 09:37:21 -05:00
Daniel Banck
4fa77727b5
Introduce metadata functions command (#32487)
* Add metadata functions command skeleton

* Export functions as JSON via cli command

* Add metadata command

* Add tests to jsonfunction package

* WIP: Add metadata functions test

* Change return_type & type in JSON to json.RawMessage

This enables easier deserialisation of types when parsing the JSON.

* Skip is_nullable when false

* Update cli docs with metadata command

* Use tfdiags to report function marshal errors

* Ignore map, list and type functions

* Test Marshal function with diags

* Test metadata functions command output

* Simplify type marshaling by using cty.Type

* Add static function signatures for can and try

* Update internal/command/jsonfunction/function_test.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

---------

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
2023-02-14 14:08:47 +00:00
yokomotod
a62f4f0763
sort modules.json for stable order (#32618) 2023-02-13 18:26:16 +01:00
Alisdair McDiarmid
b7042e818f
Merge pull request #32664 from hashicorp/alisdair/do-not-add-orphan-nodes-for-deposed-instances
Do not add orphan nodes for deposed instances
2023-02-13 09:50:48 -05:00
Alisdair McDiarmid
12d00e6571
Merge pull request #32663 from hashicorp/alisdair/read-resource-instance-state-nil-checks
Fix panic when planning orphaned deposed instances
2023-02-13 09:50:28 -05:00
Liam Cervante
d212a72d1d
structured run output: impose canonical ordering on jsonstate and jsonplan packages (#32649) 2023-02-13 15:40:17 +01:00
Yin Luo
e9b066f514
Support assume role for cos backend (#32631)
* go get github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sts/v20180813@v1.0.588

* feat:support assume_role for COS backend

* update go.mod and go.sum

* change secret_id and secret_key from required to optional

* update cos doc

* update logic by comments

* rm sensitive info in log
2023-02-13 10:25:58 +01:00
Brandon Croft
26e85685d1
Merge pull request #32647 from hashicorp/TF-4390-backport-initial-remote-state-serial-fix
Begin cloud remote state with serial > 0
2023-02-10 15:35:37 -07:00
Sebastian Rivera
71dac9bf5e Add token field to cloud backend 2023-02-10 16:58:14 -05:00
Alisdair McDiarmid
7ecb0b8ffb Do not add orphan nodes for deposed instances
Resource instances with no current object in state should not have
orphan nodes added to the graph, as deposed objects are handled
separately. This was previously handled correctly for the non-expanded
case, but expanded resources were missing the appropriate check for a
current object.

Also update the comment in the non-expanded case to hopefully clarify
that we're checking for the presence of a current object, not the
absence of any deposed objects. An instance may have both a current
object and zero or more deposed objects in some circumstances, and if
so, we still want an orphan node to be added if the instance is not in
configuration.
2023-02-10 16:25:11 -05:00
Alisdair McDiarmid
8997e5b8c6 Fix panic when planning orphaned deposed instances 2023-02-10 15:31:21 -05:00
megan07
d7d8a2262c
Merge pull request #32557 from bharathkkb/gcs-backend-impersonate-envvar
Add a GCS backend specific env var for impersonation
2023-02-09 09:28:26 -06:00
Sebastian Rivera
de574ae6d4
Initial implementation of structured logging in cloud backend (#32504)
* Implementation of structured logging.

These are the changes that enable the cloud backend to consume
structured logs and make use of the new plan renderer. This will enable
CLI-driven runs to view the structured output in the Terraform Cloud UI.

* Cloud structured logging unit tests

* Remove deferred logs logic, fix minor issues

Color formatting fixes, log type stop lists, default behavior for logs
that are unknown

* Use service disco path in redacted plan url
2023-02-09 13:35:48 +01:00
Brandon Croft
de7304cacb
Begin cloud remote state with serial > 0 2023-02-08 14:43:57 -07:00
Liam Cervante
4fa7cd0a68
structured renderer: fix closing json tags when rendering with no symbols (#32642) 2023-02-08 16:47:12 +01:00
James Bardin
1307317457
Merge pull request #32614 from yardbirdsax/fix/30670
make remote state initial behavior the same as local state
2023-02-07 17:43:28 -05:00
James Bardin
2d9e3da983
Update internal/states/remote/state.go
Fix from review

Co-authored-by: Nathan Mische <nmische@gmail.com>
2023-02-07 17:35:56 -05:00
Liam Cervante
d818d7850d
Structured Renderer: use the new renderer when rendering the state in addition to the plan (#32629)
* Use the new renderer when rendering the state

* remove confusing and unneeded comment
2023-02-07 09:14:14 +01:00
zetHannes
c70244426a
Fix for no json output of state locking actions for --json flag (#32451)
* Add viewType to Meta object and use it at the call sites

* Assign viewType passed from flags to state-locking cli commands

* Remove temp files

* Set correct mode for statelocker depending on json flag passed to commands

* Add StateLocker interface conformation check for StateLockerJSON

* Remove empty line at end of comment

* Pass correct ViewType to StateLocker from Backend call chain

* Pass viewType to backend migration and initialization functions

* Remove json processing info in process comment

* Restore documentation style of backendMigrateOpts
2023-02-07 09:06:12 +01:00
Joshua Feierman
d45ebfbdef chore: clean-up tests & logging 2023-02-01 13:54:59 -05:00
Joshua Feierman
2576544db8 fix: remote state behavior
This makes the behavior of remote state consistent with local state in regards to the initial serial number of the generated / pushed state. Previously remote state's initial push would have a serial number of 0, whereas local state had a serial of > 0. This causes issues with the logic around, for example, ensuring that a plan file cannot be applied if state is stale (see https://github.com/hashicorp/terraform/issues/30670 for example).
2023-02-01 13:54:40 -05:00
Joshua Feierman
68e227d93d wip: refactored tests 2023-02-01 13:34:42 -05:00
Joshua Feierman
09e0dffe6c wip: failing unit test for condition 2023-02-01 13:34:42 -05:00
Brandon Croft
81a4e5ae44
Backport support for generic hostname to backend/remote 2023-01-31 15:10:53 -07:00
Brandon Croft
0cafe0dfdb
Merge pull request #32571 from hashicorp/TF-3527-detect-alias-localterraform-com-during-terraform-init
Add support for Terraform Enterprise generic hostname localterraform.com
2023-01-31 10:59:12 -07:00
Liam Cervante
24b88b7a72
Reverse the order of conversion/defaults, and update HCL with more flexible defaults package (#32454)
* Add failing test case for the given issue

* pause

* don't use local when sending PR for review

* go get github.com/hashicorp/hcl/v2@v2.16.0

* Update go.mod

---------

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2023-01-31 06:37:24 -05:00
Brandon Croft
2fe3a23094
Add generic hostname (localterraform.com) support to cloud backend
Aliases the backend hostname config as "localterraform.com" and duplicates any existing auth credentials during cloud backend configuration.
2023-01-30 17:21:24 -07:00
Brandon Croft
9fd76e56cd
Factor out terraform-config-inspect/tfconfig during init
As explained by the deleted comments, this package was used to identify situations where the `terraform 0.12upgrade` command can help migrate 0.11 syntax. Current versions of terraform don't include this command, and it's not likely that users are attempting upgrades from 0.11 to 1.4+

The replacement init swaps the order of the module and backend initialization in order to prepare for the next commit.

Config initialization now takes the following approach:
1. Load the root module, but withhold diagnostic errors until after version check
2. Initialize the backend, but withhold diagnostic errors until after version check
3. Get modules
4. Load all config (root and modules)
5. Check terraform version requirements (this can be defined by nested modules) and display any errors. It's important to show these first because prior errors could be the result of a newer terraform version syntax
6. Finally, show any errors related to backed init or config loading
2023-01-30 17:21:23 -07:00
James Bardin
fc8fed0047
Merge pull request #32583 from hashicorp/jbardin/store-null-module-outputs
save null module outputs in state
2023-01-26 15:35:48 -05:00
James Bardin
47fed6d31e save null module outputs in state
Although they are not serialized to the final stored state, all module
outputs must be saved in the state for evaluation. There is no defined
schema which is used to identify the overall type of module outputs, so
all outputs must exist in the state to build the correct type for proper
evaluation.
2023-01-26 11:03:23 -05:00
Sheridan C Rawlins
75e5ae27a2
[fixes 31700] Add mTLS support for http backend by way of client cert & key, as well as enterprise cacert. (#31699)
* Add mTLS support for http backend by way of client cert & key, as well as enterprise cacert.

* Fix style.

* Skip cert validation to be sure error is related to missing client cert; not untrusted server cert.

* Remove misplaced err check.

* Fix the size of test using http backend.

* Just for correctness, include all certs in the pem encoded cert - sometimes certs come with a chain of their signers.

* Adjusted names as recommended in PR comments.

* Adjusted names to be full-length and more descriptive.

* Added full-fledged testing with mTLS http server

* Fix goimports.

* Fix the names of the backend config.

* Exclusive lock for write and delete.

* Revert "Fix goimports."

This reverts commit 7d40f6099fbbb675fb2e25e35ee40aeafe3d0a22.

* goimports just for server test.

* Added the go:generation for the mock.

* Move the TLS configuration out to make it more readable - don't replace the HTTPClient as the retryablehttp already creates one - just configure its TLS.

* Just switch the client/data params - felt more natural this way.

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/testdata/gencerts.sh

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* the location of the file name is not sensitive.

* Added error if only one of client_certificate_pem and client_private_key_pem are set.

* Remove testify from test cases; use t.Error* for assert and t.Fatal* for require.

* Fixed import consistency

* Just use default openssl.

* Since file(...) is so trivial to use, changed the client cert, key, and ca cert to be the data.

See also https://github.com/hashicorp/terraform-provider-http/pull/211

Co-authored-by: Sheridan C Rawlins <scr@ouryahoo.com>
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
2023-01-26 14:08:07 +00:00