Commit Graph

1379 Commits

Author SHA1 Message Date
mrinalirao
3104811c7d add test to see if override continues to poll on success 2023-02-20 17:50:25 +11:00
mrinalirao
190858b9e4 Bug Fix where CLI exits if run is not confirmable. 2023-02-20 17:18:13 +11:00
James Bardin
014a425d0e
Merge pull request #32683 from hashicorp/jbardin/import-terraform-data
allow terraform_data to import
2023-02-17 09:51:02 -05:00
Martin Atkins
f0de9b60c1 backend/local: Periodically persist intermediate state snapshots
Terraform Core emits a hook event every time it writes a change into the
in-memory state. Previously the local backend would just copy that into
the transient storage of the state manager, but for most state storage
implementations that doesn't really do anything useful because it just
makes another copy of the state in memory.

We originally added this hook mechanism with the intent of making
Terraform _persist_ the state each time, but we backed that out after
finding that it was a bit too aggressive and was making the state snapshot
history much harder to use in storage systems that can preserve historical
snapshots.

However, sometimes Terraform gets killed mid-apply for whatever reason and
in our previous implementation that meant always losing that transient
state, forcing the user to edit the state manually (or use "import") to
recover a useful state.

In an attempt at finding a sweet spot between these extremes, here we
change the rule so that if an apply runs for longer than 20 seconds then
we'll try to persist the state to the backend in an update that arrives
at least 20 seconds after the first update, and then again for each
additional 20 second period as long as Terraform keeps announcing new
state snapshots.

This also introduces a special interruption mode where if the apply phase
gets interrupted by SIGINT (or equivalent) then the local backend will
try to persist the state immediately in anticipation of a
possibly-imminent SIGKILL, and will then immediately persist any
subsequent state update that arrives until the apply phase is complete.
After interruption Terraform will not start any new operations and will
instead just let any already-running operations run to completion, and so
this will persist the state once per resource instance that is able to
complete before being killed.

This does mean that now long-running applies will generate intermediate
state snapshots where they wouldn't before, but there should still be
considerably fewer snapshots than were created when we were persisting
for each individual state change. We can adjust the 20 second interval
in future commits if we find that this spot isn't as sweet as first
assumed.
2023-02-14 15:17:56 -08:00
James Bardin
3b953d3bd8 allow terraform_data to import
The terraform provider was panicking on import, because it didn't
previously have a resource type which could be imported at all. Add a
stub import function for terraform_data as a placeholder to allow the
call to complete successfully. While there's no need to actually import
a terraform_data resource, users will inevitably use this to construct
examples of import actions for learning purposes or bug reports.

This still isn't very useful even for examples however, because the
state-only nature of the terraform_data resource type means that we
can't fill in the state from only the import ID. This means that any
value in `trigger_replace` or `input` will cause a change in the next
plan. Once configuration data is available during import we can extend
this to create a logical final state based on config.
2023-02-14 09:37:21 -05:00
Daniel Banck
4fa77727b5
Introduce metadata functions command (#32487)
* Add metadata functions command skeleton

* Export functions as JSON via cli command

* Add metadata command

* Add tests to jsonfunction package

* WIP: Add metadata functions test

* Change return_type & type in JSON to json.RawMessage

This enables easier deserialisation of types when parsing the JSON.

* Skip is_nullable when false

* Update cli docs with metadata command

* Use tfdiags to report function marshal errors

* Ignore map, list and type functions

* Test Marshal function with diags

* Test metadata functions command output

* Simplify type marshaling by using cty.Type

* Add static function signatures for can and try

* Update internal/command/jsonfunction/function_test.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

---------

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
2023-02-14 14:08:47 +00:00
yokomotod
a62f4f0763
sort modules.json for stable order (#32618) 2023-02-13 18:26:16 +01:00
Alisdair McDiarmid
b7042e818f
Merge pull request #32664 from hashicorp/alisdair/do-not-add-orphan-nodes-for-deposed-instances
Do not add orphan nodes for deposed instances
2023-02-13 09:50:48 -05:00
Alisdair McDiarmid
12d00e6571
Merge pull request #32663 from hashicorp/alisdair/read-resource-instance-state-nil-checks
Fix panic when planning orphaned deposed instances
2023-02-13 09:50:28 -05:00
Liam Cervante
d212a72d1d
structured run output: impose canonical ordering on jsonstate and jsonplan packages (#32649) 2023-02-13 15:40:17 +01:00
Yin Luo
e9b066f514
Support assume role for cos backend (#32631)
* go get github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sts/v20180813@v1.0.588

* feat:support assume_role for COS backend

* update go.mod and go.sum

* change secret_id and secret_key from required to optional

* update cos doc

* update logic by comments

* rm sensitive info in log
2023-02-13 10:25:58 +01:00
Brandon Croft
26e85685d1
Merge pull request #32647 from hashicorp/TF-4390-backport-initial-remote-state-serial-fix
Begin cloud remote state with serial > 0
2023-02-10 15:35:37 -07:00
Alisdair McDiarmid
7ecb0b8ffb Do not add orphan nodes for deposed instances
Resource instances with no current object in state should not have
orphan nodes added to the graph, as deposed objects are handled
separately. This was previously handled correctly for the non-expanded
case, but expanded resources were missing the appropriate check for a
current object.

Also update the comment in the non-expanded case to hopefully clarify
that we're checking for the presence of a current object, not the
absence of any deposed objects. An instance may have both a current
object and zero or more deposed objects in some circumstances, and if
so, we still want an orphan node to be added if the instance is not in
configuration.
2023-02-10 16:25:11 -05:00
Alisdair McDiarmid
8997e5b8c6 Fix panic when planning orphaned deposed instances 2023-02-10 15:31:21 -05:00
megan07
d7d8a2262c
Merge pull request #32557 from bharathkkb/gcs-backend-impersonate-envvar
Add a GCS backend specific env var for impersonation
2023-02-09 09:28:26 -06:00
Sebastian Rivera
de574ae6d4
Initial implementation of structured logging in cloud backend (#32504)
* Implementation of structured logging.

These are the changes that enable the cloud backend to consume
structured logs and make use of the new plan renderer. This will enable
CLI-driven runs to view the structured output in the Terraform Cloud UI.

* Cloud structured logging unit tests

* Remove deferred logs logic, fix minor issues

Color formatting fixes, log type stop lists, default behavior for logs
that are unknown

* Use service disco path in redacted plan url
2023-02-09 13:35:48 +01:00
Brandon Croft
de7304cacb
Begin cloud remote state with serial > 0 2023-02-08 14:43:57 -07:00
Liam Cervante
4fa7cd0a68
structured renderer: fix closing json tags when rendering with no symbols (#32642) 2023-02-08 16:47:12 +01:00
James Bardin
1307317457
Merge pull request #32614 from yardbirdsax/fix/30670
make remote state initial behavior the same as local state
2023-02-07 17:43:28 -05:00
James Bardin
2d9e3da983
Update internal/states/remote/state.go
Fix from review

Co-authored-by: Nathan Mische <nmische@gmail.com>
2023-02-07 17:35:56 -05:00
Liam Cervante
d818d7850d
Structured Renderer: use the new renderer when rendering the state in addition to the plan (#32629)
* Use the new renderer when rendering the state

* remove confusing and unneeded comment
2023-02-07 09:14:14 +01:00
zetHannes
c70244426a
Fix for no json output of state locking actions for --json flag (#32451)
* Add viewType to Meta object and use it at the call sites

* Assign viewType passed from flags to state-locking cli commands

* Remove temp files

* Set correct mode for statelocker depending on json flag passed to commands

* Add StateLocker interface conformation check for StateLockerJSON

* Remove empty line at end of comment

* Pass correct ViewType to StateLocker from Backend call chain

* Pass viewType to backend migration and initialization functions

* Remove json processing info in process comment

* Restore documentation style of backendMigrateOpts
2023-02-07 09:06:12 +01:00
Joshua Feierman
d45ebfbdef chore: clean-up tests & logging 2023-02-01 13:54:59 -05:00
Joshua Feierman
2576544db8 fix: remote state behavior
This makes the behavior of remote state consistent with local state in regards to the initial serial number of the generated / pushed state. Previously remote state's initial push would have a serial number of 0, whereas local state had a serial of > 0. This causes issues with the logic around, for example, ensuring that a plan file cannot be applied if state is stale (see https://github.com/hashicorp/terraform/issues/30670 for example).
2023-02-01 13:54:40 -05:00
Joshua Feierman
68e227d93d wip: refactored tests 2023-02-01 13:34:42 -05:00
Joshua Feierman
09e0dffe6c wip: failing unit test for condition 2023-02-01 13:34:42 -05:00
Brandon Croft
81a4e5ae44
Backport support for generic hostname to backend/remote 2023-01-31 15:10:53 -07:00
Brandon Croft
0cafe0dfdb
Merge pull request #32571 from hashicorp/TF-3527-detect-alias-localterraform-com-during-terraform-init
Add support for Terraform Enterprise generic hostname localterraform.com
2023-01-31 10:59:12 -07:00
Liam Cervante
24b88b7a72
Reverse the order of conversion/defaults, and update HCL with more flexible defaults package (#32454)
* Add failing test case for the given issue

* pause

* don't use local when sending PR for review

* go get github.com/hashicorp/hcl/v2@v2.16.0

* Update go.mod

---------

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2023-01-31 06:37:24 -05:00
Brandon Croft
2fe3a23094
Add generic hostname (localterraform.com) support to cloud backend
Aliases the backend hostname config as "localterraform.com" and duplicates any existing auth credentials during cloud backend configuration.
2023-01-30 17:21:24 -07:00
Brandon Croft
9fd76e56cd
Factor out terraform-config-inspect/tfconfig during init
As explained by the deleted comments, this package was used to identify situations where the `terraform 0.12upgrade` command can help migrate 0.11 syntax. Current versions of terraform don't include this command, and it's not likely that users are attempting upgrades from 0.11 to 1.4+

The replacement init swaps the order of the module and backend initialization in order to prepare for the next commit.

Config initialization now takes the following approach:
1. Load the root module, but withhold diagnostic errors until after version check
2. Initialize the backend, but withhold diagnostic errors until after version check
3. Get modules
4. Load all config (root and modules)
5. Check terraform version requirements (this can be defined by nested modules) and display any errors. It's important to show these first because prior errors could be the result of a newer terraform version syntax
6. Finally, show any errors related to backed init or config loading
2023-01-30 17:21:23 -07:00
James Bardin
fc8fed0047
Merge pull request #32583 from hashicorp/jbardin/store-null-module-outputs
save null module outputs in state
2023-01-26 15:35:48 -05:00
James Bardin
47fed6d31e save null module outputs in state
Although they are not serialized to the final stored state, all module
outputs must be saved in the state for evaluation. There is no defined
schema which is used to identify the overall type of module outputs, so
all outputs must exist in the state to build the correct type for proper
evaluation.
2023-01-26 11:03:23 -05:00
Sheridan C Rawlins
75e5ae27a2
[fixes 31700] Add mTLS support for http backend by way of client cert & key, as well as enterprise cacert. (#31699)
* Add mTLS support for http backend by way of client cert & key, as well as enterprise cacert.

* Fix style.

* Skip cert validation to be sure error is related to missing client cert; not untrusted server cert.

* Remove misplaced err check.

* Fix the size of test using http backend.

* Just for correctness, include all certs in the pem encoded cert - sometimes certs come with a chain of their signers.

* Adjusted names as recommended in PR comments.

* Adjusted names to be full-length and more descriptive.

* Added full-fledged testing with mTLS http server

* Fix goimports.

* Fix the names of the backend config.

* Exclusive lock for write and delete.

* Revert "Fix goimports."

This reverts commit 7d40f6099fbbb675fb2e25e35ee40aeafe3d0a22.

* goimports just for server test.

* Added the go:generation for the mock.

* Move the TLS configuration out to make it more readable - don't replace the HTTPClient as the retryablehttp already creates one - just configure its TLS.

* Just switch the client/data params - felt more natural this way.

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/testdata/gencerts.sh

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* Update internal/backend/remote-state/http/backend.go

Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>

* the location of the file name is not sensitive.

* Added error if only one of client_certificate_pem and client_private_key_pem are set.

* Remove testify from test cases; use t.Error* for assert and t.Fatal* for require.

* Fixed import consistency

* Just use default openssl.

* Since file(...) is so trivial to use, changed the client cert, key, and ca cert to be the data.

See also https://github.com/hashicorp/terraform-provider-http/pull/211

Co-authored-by: Sheridan C Rawlins <scr@ouryahoo.com>
Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>
2023-01-26 14:08:07 +00:00
James Bardin
b6906f3572
Merge pull request #32563 from hashicorp/jbardin/optional-computed-comparison-next
a new method of `ProposedNew` set comparison
2023-01-25 15:05:50 -05:00
James Bardin
799f4a76ab
Merge pull request #32554 from hashicorp/jbardin/optional-computed-comparison
refactor ProposedNew
2023-01-25 15:05:39 -05:00
James Bardin
33c1ffa7c2
Merge pull request #32553 from hashicorp/jbardin/nesting-map-objchange
Fix crashes with NestingMap values
2023-01-25 15:05:27 -05:00
James Bardin
f6af5c1ef7
Merge pull request #32551 from hashicorp/jbardin/optional-computed-null
better determine when to plan optional+computed
2023-01-25 15:05:06 -05:00
James Bardin
aacbc167eb
Merge pull request #32536 from hashicorp/jbardin/optional-computed-nested-objects
Plan correct optional and computed attributes in nested objects and sets
2023-01-25 15:04:37 -05:00
James Bardin
d1d55653e4 update comment for fixed test 2023-01-25 12:11:31 -05:00
Pooya
6bbb76eca3
Added Quiet value (#32116)
* Added Quiet value

* Added Quiet value

* Added Quiet value

* Removed comments on 170 171 173

Co-authored-by: itspooya <fallenangel201190@gmail.com>
2023-01-25 16:51:14 +00:00
Martin Atkins
e2380b1038 cliconfig: Allow forcing use of the plugin cache despite the lock file
Currently Terraform will use an entry from the global plugin cache only if
it matches a checksum already recorded in the dependency lock file. This
allows Terraform to produce a complete lock file entry on the first
encounter with a new provider, whereas using the cache in that case would
cause the lock file to only cover the single package in the cache and
thereefore be unusable on any other operating system or CPU architecture.

This temporary CLI config option is a pragmatic exception to support those
who cannot currently correctly use the dependency lock file but who still
want to benefit from the plugin cache. With this setting enabled,
Terraform has permission to produce a dependency lock file that is only
suitable for the current system if that would allow use of an existing
entry in the plugin cache.

We are introducing this option to resolve a conflict between the needs of
folks who are using the dependency lock file as expected and the needs of
folks who cannot use the dependency lock file for some reason. The hope
then is to give respite to those who need this exception in the meantime
while we understand better why they cannot use the dependency lock file
and improve its design so that everyone will be able to use it
successfully in a future version of Terraform. This option will become a
silent no-op in a future version of Terraform, once the dependency lock
file behavior is sufficient for all supported Terraform development
workflows.
2023-01-25 08:23:01 -08:00
Anna Winkler
3cc7e55465
Expand error when discovery document request fails (#32394)
* Expand error when discovery document request fails.
* Use error type from svchost library
* Use svchost v0.0.1
2023-01-24 15:19:12 -07:00
James Bardin
60d6e52021 a new method of ProposedNew set comparison
The existing set comparison method uses the prior elements with the computed
portions nulled out to find candidates to match the configuration. This
has the shortcoming of always removing optional+computed attributes,
because we have not yet found the configuration to know if attribute was
set or not.

Rather than having to take the most pessimistic value before comparison
to precompute the nulled values, we can compare each candidate directly,
walking the values in tandem. Each prior value is compared against the
config and checked to see if it could have been derived from that
configuration value, which allows us to treat optional+computed as
optional if there is config and computed if there is not.

This removes the ambiguity from having optional+computed attributes
within sets, giving us consistent plans when all values are known.
Unknown values of course are still undecidable, as are edge cases were
providers refresh with altered values or retained changed prior values
plan that were deemed not functionally significant.
2023-01-24 14:05:57 -05:00
bharathkkb
05ff3c3278 Add a GCS backend specific env var for impersonation 2023-01-21 05:20:31 +00:00
James Bardin
9459f0b214 refactor ProposedNew
Unify the ProposedNew paths for Blocks and Objects. Break out the
individual case blocks into functions, then use a common interface to
dispatch the object creation to the correct function based on schema
type. This cuts the code in half, and prevents the block and object
behavior from diverging.
2023-01-20 15:59:32 -05:00
Nick Fagerlund
f26786d6d9
Merge pull request #32545 from hashicorp/nf/jan23-cloud-backend-test-typo
Fix typo in cloud backend's `TestCloud_setConfigurationFields`
2023-01-20 11:40:52 -08:00
James Bardin
375c2da3e3 update NestingMap logic
Simplify the logic in the NestingMap cases. Prevent uninitialized
cty.NilVal from appearing in block case.
2023-01-20 13:51:10 -05:00
James Bardin
fcbfc365e6 fix panics when handling null values in maps
NestingMap structures are not well tested, and we panic in many
situations when null crops up. Fix the first test cases and start
refactoring best we can. This probably won't go so far as making all the
objchange functions generic over Block and Object, but we can simplify a
lot and verify parity in implementations for now.
2023-01-20 13:38:55 -05:00
Nick Fagerlund
3b26f68737
Merge pull request #32543 from hashicorp/nf/jan23-attr-path-value-marks-corruption
Fix accidental mutation of shared `cty.Path`s in ValueMarks funcs
2023-01-20 10:17:58 -08:00