Commit Graph

308 Commits

Author SHA1 Message Date
hellertang
0dbf0711a0
cos backend support accelerate (#31425) 2022-07-15 17:07:41 +01:00
Tom Bamford
dc1f5bccf8
Merge pull request #31070 from hashicorp/f/removing-adal-auth
backend/azurerm: exclusively using Microsoft Graph/MSAL and removing Azure Active Directory Graph/ADAL
2022-07-12 12:18:29 +01:00
James Bardin
96c720517f fixup broken test fixtures
some of the minimal test provider implementations didn't check for null
values.
2022-07-06 13:47:35 -04:00
James Bardin
d61d3e9fd8 remove deprecated etcdv3 backend 2022-06-30 15:32:43 -04:00
James Bardin
953c448f9a add simple error indicating backend removal
There are no good options for inserting diagnostics into the backend
lookup, or creating a backend which reports it's removal because none of
the init or GetSchema functions return any errors.

Keep a registry of the removed backend so that we can at least notify
users that a backend was removed vs an invalid name.
2022-06-28 13:58:22 -04:00
James Bardin
799ab6c951 remove deprecated etcd backend
This allows us to remove the manual replace directives
github.com/dgrijalva/jwt-go and google.golang.org/grpc, so that we can
remove the CVE warnings and update the grpc packages.

While the etcdv3 backend is also marked as deprecated, the changes here
are done in a manner to keep that backend working for the time being.
2022-06-27 15:01:21 -04:00
Nicolas Vanheuverzwijn
bb1c134b94 http-backend: http backend lock error return LockError instead of generic error 2022-06-16 11:01:01 -04:00
Kagashino
cab6cee385 fix: backend/cos - remove cos object write lock either when unlock state file 2022-06-10 20:14:58 +08:00
James Bardin
0c7d39aca5 deprecate unmaintained backends 2022-06-08 18:09:25 -04:00
tombuildsstuff
05528e8c8a backend/azurerm: removing ADAL support 2022-05-18 16:56:10 +02:00
Anna Winkler
22c121df86
Bump compatibility version to 1.3.0 for terraform core release (#30988)
* Bump compatibility version to 1.3.0 for terraform core release

Co-authored-by: Brandon Croft <brandon.croft@gmail.com>
2022-05-03 12:28:41 -06:00
James Bardin
df0a70bfb6 check for cancellation before apply confirmation
When executing an apply with no plan, it's possible for a cancellation
to arrive during the final batch of provider operations, resulting in no
errors in the plan. The run context was next checked during the
confirmation for apply, but in the case of -auto-approve that
confirmation is skipped, resulting in the canceled plan being applied.

Make sure we directly check for cancellation before confirming the plan.
2022-05-02 14:09:47 -04:00
Tom Bamford
d08bc4463d
Azure Backend: support OIDC authentication 2022-04-27 19:27:30 +01:00
Keith Clawson
23dffee568 Always use token in backend config when provided 2022-04-22 14:01:00 -04:00
Tom Harvey
2eb9118cd1
backend/remote-state/azure: defaulting the Azure Backend to use MSAL (#30891)
* backend/remote-state/azure: defaulting the Azure Backend to use MSAL

Fixes #30881

* backend/remote-state/azurerm: deprecating `use_microsoft_graph`
2022-04-20 17:31:44 +02:00
Eng Zer Jun
fedd315275
test: use T.TempDir to create temporary test directory (#30803)
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-08 17:34:16 +01:00
Lars Lehtonen
88c9b90c93
internal/backend/remote-state/oss: fix dropped error (#30352) 2022-04-01 08:07:31 +01:00
Luces Huayhuaca
bdc7d8c0a1
Upgrade go-tfe dependency to use 1.0 version. It contains breaking changes, so we are updating method signatures, method names and the type of optional parameters, as needed. (#30626) 2022-03-23 13:58:47 -07:00
Theo Chupp
d15a2bc024
fix: local variables should not be overridden by remote variables during terraform import (#29972)
* fix: local variables should not be overridden by remote variables during `terraform import`

* chore: applied the same fix in the 'internal/cloud' package

* backport changes from cloud package to remote package

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Co-authored-by: uturunku1 <luces.huayhuaca@gmail.com>
2022-03-15 14:42:11 -07:00
Krista LaFentres (she/her)
6dcf00aefc
Merge pull request #30344 from hashicorp/lafentres/refactor-show-command
cli: Refactor show command & migrate to command arguments and views
2022-01-13 13:58:53 -06:00
Krista LaFentres
64e1241ae3 backend/local: Remove unused DisablePlanFileStateLineageChecks flag
Now that show command has been refactored to remove its dependence
on a local backend and local run, this flag is no longer needed to
fix #30195.
2022-01-13 11:00:10 -06:00
Martin Atkins
36c4d4c241 core and backend: remove redundant handling of default variable values
Previously we had three different layers all thinking they were
responsible for substituting a default value for an unset root module
variable:
 - the local backend, via logic in backend.ParseVariableValues
 - the context.Plan function (and other similar functions) trying to
   preprocess the input variables using
   terraform.mergeDefaultInputVariableValues .
 - the newer prepareFinalInputVariableValue, which aims to centralize all
   of the variable preparation logic so it can be common to both root and
   child module variables.

The second of these was also trying to handle type constraint checking,
which is also the responsibility of the central function and not something
we need to handle so early.

Only the last of these consistently handles both root and child module
variables, and so is the one we ought to keep. The others are now
redundant and are causing prepareFinalInputVariableValue to get a slightly
corrupted view of the caller's chosen variable values.

To rectify that, here we remove the two redundant layers altogether and
have unset root variables pass through as cty.NilVal all the way to the
central prepareFinalInputVariableValue function, which will then handle
them in a suitable way which properly respects the "nullable" setting.

This commit includes some test changes in the terraform package to make
those tests no longer rely on the mergeDefaultInputVariableValues logic
we've removed, and to instead explicitly set cty.NilVal for all unset
variables to comply with our intended contract for PlanOpts.SetVariables,
and similar. (This is so that we can more easily catch bugs in callers
where they _don't_ correctly handle input variables; it allows us to
distinguish between the caller explicitly marking a variable as unset vs.
not describing it at all, where the latter is a bug in the caller.)
2022-01-10 12:26:54 -08:00
Alisdair McDiarmid
768741c0f7 command/show: Disable plan state lineage checks
When showing a saved plan, we do not need to check the state lineage
against current state, because the plan cannot be applied. This is
relevant when plan and apply specify a `-state` argument to choose a
non-default state file. In this case, the stored prior state in the plan
will not match the default state file, so a lineage check will always
error.
2021-12-17 17:46:42 -05:00
James Bardin
8c4031ef15 don't persist a nil state from Apply
Apply should not return a nil state to be persisted.
2021-12-17 14:00:59 -05:00
Chris Arcand
f521ba6cd7 Remove 'enhanced' backend type distinction
As explained in the changes: The 'enhanced' backend terminology, which
only truly pertains to the 'remote' backend with a single API (Terraform
Cloud/Enterprise's), has been found to be a confusing vestige which need
only be explained in the context of the 'remote' backend.

These changes reorient the explanation(s) of backends to pertain more
directly to their primary purpose, which is storage of state snapshots
(and not implementing operations).

That Terraform operations are still _implemented_ by the literal
`Backend` and `Enhanced` interfaces is inconsequential a user of
Terraform, an internal detail.
2021-12-07 16:29:51 -06:00
Katy Moe
b1c071bb57
pass hclog logger to retryable client 2021-11-30 00:09:01 +00:00
Tom Harvey
a1c7c0dfa4
Update internal/backend/remote-state/azure/backend.go
Co-authored-by: Tom Bamford <tbamford@hashicorp.com>
2021-11-17 20:24:22 +01:00
tombuildsstuff
ffe9a41468 internal/backend/remote-state/azure: adding tests covering MSAL 2021-11-17 18:45:44 +01:00
tombuildsstuff
a2cad7d579 internal/backend/remote-state/azure: prefixing the existing tests with ADAL 2021-11-17 18:43:39 +01:00
tombuildsstuff
9f710558ce internal/backend/remote-state/azure: adding a feature-toggle to use MSAL 2021-11-17 18:42:01 +01:00
tombuildsstuff
27a34d6c2c internal/backend/remote-state/azure: updating the link to the client secret
This redirects to the registry so we should point there directly
2021-11-17 18:28:47 +01:00
tombuildsstuff
5133d30b71 internal/backend/remote-state/azure: code changes for go-azure-helpers v0.18.0 2021-11-17 18:26:23 +01:00
Jared Ledvina
61a5808680 k8s-backend - Initial context support for library update
Signed-off-by: Jared Ledvina <jared.ledvina@datadoghq.com>
2021-11-16 14:43:50 -05:00
Chris Arcand
779c958fbf cloud: Add streamlined 'remote' backend state migration path
For Terraform Cloud users using the 'remote' backend, the existing
'pattern' prompt should work just fine - but because their workspaces
are already present in TFC, the 'migration' here is really just
realigning their local workspaces with Terraform Cloud. Instead of
forcing users to do the mental gymnastics of what it means to migrate
from 'prefix' - and because their remote workspaces probably already exist and
already conform to Terraform Cloud's naming concerns - streamline the
process for them and calculate the necessary pattern to migrate as-is,
without any user intervention necessary.
2021-11-03 15:07:33 -05:00
xiaozhu36
e0a2716601 backend/oss: Removes the ConflictWith tag which on the attributes assume_role* to fix the incompatible error 2021-11-03 11:17:35 +08:00
xiaozhu36
5d19a34f88 backend/oss: Supports the new attribute sts_endpoint 2021-11-01 22:52:45 +08:00
James Bardin
2ebdc099ac
Merge pull request #29307 from hayorov/main
Flatten `assume_role` block for OSS backend
2021-11-01 09:18:25 -04:00
James Bardin
81e709d185
Merge pull request #29825 from hashicorp/jbardin/no-panicwrap
Remove the use of panicwrap in the Terraform CLI
2021-10-29 14:41:05 -04:00
Brandon Croft
dbbfae5a1c refactor ParseVariableValues into separate operations
1. ParseDeclaredValues: parses unparsed variables into terraform.InputValues
2. ProbeUndeclaredVariableValues: compares variable declarations with unparsed values to warn/error about undeclared variables
2021-10-28 19:29:12 -05:00
Chris Arcand
1791b71196 cloud: TestCloud_backendWithTags
Implementing this test was quite a rabbithole, as in order to satisfy
backendTestBackendStates() the workspaces returned from
backend.Workspaces() must match exactly, and the shortcut taken to test
pagination in 3cc58813f0 created an
impossible circumstance that got plastered over with the fact that
prefix filtering is done clientside, not by the API as it should be.

Tagging does not rely on clientside filtering, and expects that the
request made to the TFC API returns exactly those workspaces with the
given tags.

These changes include a better way to test pagination, wherein we
actually create over a page worth of valid workspaces in the mock client
and implement a simplified pagination behavior to match how the TFC API
actually works.
2021-10-28 19:29:10 -05:00
Chris Arcand
471dd479e8 Update go-tfe to 26689e
These changes include additions to fulfill the interface for the client
mock, plus moving all that logic (which needn't be duplicated across
both the remote and cloud packages) over to the cloud package under a
dedicated mock client file.
2021-10-28 19:29:10 -05:00
Chris Arcand
ea8ad0b15a Initial commit of 'cloud' package
The cloud package intends to implement a new integration for
Terraform Cloud/Enterprise. The purpose of this integration is to better
support TFC users; it will shed some overly generic UX and architecture,
behavior changes that are otherwise backwards incompatible in the remote
backend, and technical debt - all of which are vestiges from before
Terraform Cloud existed.

This initial commit is largely a porting of the existing 'remote'
backend, which will serve as an underlying implementation detail and not
be a typical user-level backend. This is because to re-implement the
literal backend interface is orthogonal to the purpose of this
integration, and can always be migrated away from later.

As this backend is considered an implementation detail, it will not be
registered as a declarable backend. Within these changes it is, for easy
of initial development and a clean diff.
2021-10-28 19:29:09 -05:00
James Bardin
d03a037567 insert panic handlers 2021-10-28 11:51:39 -04:00
Alex Khaerov
6f1eceb0e2
Revert autoupdate 2021-10-21 21:42:13 +08:00
xiaozhu36
7afaea4cf2 backend/oss: Fixes the nil pointer panic error when missing access key or secret key 2021-10-20 16:05:00 +08:00
Alex Khaerov
14f366dbf4
Update documentation 2021-10-18 12:54:40 +08:00
Alex Khaerov
f32702c5c2
Support deprecated assume_role block 2021-10-18 12:32:57 +08:00
Alex Khaerov
de8810cdd9
Merge branch 'hashicorp:main' into main 2021-10-18 11:42:37 +08:00
Martin Atkins
39779e7021 backend/remote-state/cos: Don't use github.com/likexian/gokit
We don't use this library anywhere else in Terraform, and this backend was
using it only for trivial helpers that are easy to express inline anyway.
The new direct code is also type-checkable, whereas these helper functions
seem to be written using reflection.

This gives us one fewer dependency to worry about and makes the test code
for this backend follow a similar assertions style as the rest of this
codebase.
2021-10-14 10:50:05 -07:00
James Bardin
ef3c98466d
Merge pull request #29755 from hashicorp/jbardin/first-plan-lineage
Check for stale plan with no state metadata
2021-10-14 13:31:30 -04:00
James Bardin
5ffa0839f9 only check serial when applying the first plan
Ensure that we still check for a stale plan even when it was created
with no previous state.

Create separate errors for incorrect lineage vs incorrect serial.

To prevent confusion when applying a first plan multiple times, only
report it as a stale plan rather than different lineage.
2021-10-13 17:17:51 -04:00
Martin Atkins
bee7403f3e command/workspace_delete: Allow deleting a workspace with empty husks
Previously we would reject attempts to delete a workspace if its state
contained any resources at all, even if none of the resources had any
resource instance objects associated with it.

Nowadays there isn't any situation where the normal Terraform workflow
will leave behind resource husks, and so this isn't as problematic as it
might've been in the v0.12 era, but nonetheless what we actually care
about for this check is whether there might be any remote objects that
this state is tracking, and for that it's more precise to look for
non-nil resource instance objects, rather than whole resources.

This also includes some adjustments to our error messaging to give more
information about the problem and to use terminology more consistent with
how we currently talk about this situation in our documentation and
elsewhere in the UI.

We were also using the old State.HasResources method as part of some of
our tests. I considered preserving it to avoid changing the behavior of
those tests, but the new check seemed close enough to the intent of those
tests that it wasn't worth maintaining this method that wouldn't be used
in any main code anymore. I've therefore updated those tests to use
the new HasResourceInstanceObjects method instead.
2021-10-13 13:54:11 -07:00
Megan Bang
9b9b26a3cd update error message for invalid json 2021-10-13 13:51:07 -05:00
megan07
7dda3366a6
Update internal/backend/remote-state/gcs/backend.go
Co-authored-by: appilon <apilon@hashicorp.com>
2021-10-08 10:02:05 -05:00
Megan Bang
81201d69a3 check valid json 2021-10-07 16:33:21 -05:00
Megan Bang
348c761bea add better error if credentials are invalid 2021-10-07 16:28:47 -05:00
Martin Atkins
df578afd7e backend/local: Check dependency lock consistency before any operations
In historical versions of Terraform the responsibility to check this was
inside the terraform.NewContext function, along with various other
assorted concerns that made that function particularly complicated.

More recently, we reduced the responsibility of the "terraform" package
only to instantiating particular named plugins, assuming that its caller
is responsible for selecting appropriate versions of any providers that
_are_ external. However, until this commit we were just assuming that
"terraform init" had correctly selected appropriate plugins and recorded
them in the lock file, and so nothing was dealing with the problem of
ensuring that there haven't been any changes to the lock file or config
since the most recent "terraform init" which would cause us to need to
re-evaluate those decisions.

Part of the game here is to slightly extend the role of the dependency
locks object to also carry information about a subset of provider
addresses whose lock entries we're intentionally disregarding as part of
the various little edge-case features we have for overridding providers:
dev_overrides, "unmanaged providers", and the testing overrides in our
own unit tests. This is an in-memory-only annotation, never included in
the serialized plan files on disk.

I had originally intended to create a new package to encapsulate all of
this plugin-selection logic, including both the version constraint
checking here and also the handling of the provider factory functions, but
as an interim step I've just made version constraint consistency checks
the responsibility of the backend/local package, which means that we'll
always catch problems as part of preparing for local operations, while
not imposing these additional checks on commands that _don't_ run local
operations, such as "terraform apply" when in remote operations mode.
2021-10-01 14:43:58 -07:00
Martin Atkins
6a98e4720c plans/planfile: Create takes most arguments via a struct type
Previously the planfile.Create function had accumulated probably already
too many positional arguments, and I'm intending to add another one in
a subsequent commit and so this is preparation to make the callsites more
readable (subjectively) and make it clearer how we can extend this
function's arguments to include further components in a plan file.

There's no difference in observable functionality here. This is just
passing the same set of arguments in a slightly different way.
2021-10-01 14:43:58 -07:00
Martin Atkins
8d193ad268 core: Simplify and centralize plugin availability checks
Historically the responsibility for making sure that all of the available
providers are of suitable versions and match the appropriate checksums has
been split rather inexplicably over multiple different layers, with some
of the checks happening as late as creating a terraform.Context.

We're gradually iterating towards making that all be handled in one place,
but in this step we're just cleaning up some old remnants from the
main "terraform" package, which is now no longer responsible for any
version or checksum verification and instead just assumes it's been
provided with suitable factory functions by its caller.

We do still have a pre-check here to make sure that we at least have a
factory function for each plugin the configuration seems to depend on,
because if we don't do that up front then it ends up getting caught
instead deep inside the Terraform runtime, often inside a concurrent
graph walk and thus it's not deterministic which codepath will happen to
catch it on a particular run.

As of this commit, this actually does leave some holes in our checks: the
command package is using the dependency lock file to make sure we have
exactly the provider packages we expect (exact versions and checksums),
which is the most crucial part, but we don't yet have any spot where
we make sure that the lock file is consistent with the current
configuration, and we are no longer preserving the provider checksums as
part of a saved plan.

Both of those will come in subsequent commits. While it's unusual to have
a series of commits that briefly subtracts functionality and then adds
back in equivalent functionality later, the lock file checking is the only
part that's crucial for security reasons, with everything else mainly just
being to give better feedback when folks seem to be using Terraform
incorrectly. The other bits are therefore mostly cosmetic and okay to be
absent briefly as we work towards a better design that is clearer about
where that responsibility belongs.
2021-10-01 14:43:58 -07:00
James Bardin
8ce2254ad6
Merge pull request #29647 from hashicorp/jbardin/test-temp-cleanup
temp path clean for some backend tests
2021-09-24 12:27:53 -04:00
Alisdair McDiarmid
57318ef561 backend/remote: Support interop from 0.14 to 1.1
The previous conservative guarantee that we would not make backwards
incompatible changes to the state file format until at least Terraform
1.1 can now be extended. Terraform 0.14 through 1.1 will be able to
interoperably use state files, so we can update the remote backend
version compatibility check accordingly.
2021-09-24 09:26:09 -04:00
James Bardin
9c078c27cf temp path clean for some backend tests 2021-09-23 17:16:33 -04:00
Martin Atkins
f0034beb33 core: refactoring.ImpliedMoveStatements replaces NodeCountBoundary
Going back a long time we've had a special magic behavior which tries to
recognize a situation where a module author either added or removed the
"count" argument from a resource that already has instances, and to
silently rename the zeroth or no-key instance so that we don't plan to
destroy and recreate the associated object.

Now we have a more general idea of "move statements", and specifically
the idea of "implied" move statements which replicates the same heuristic
we used to use for this behavior, we can treat this magic renaming rule as
just another "move statement", special only in that Terraform generates it
automatically rather than it being written out explicitly in the
configuration.

In return for wiring that in, we can now remove altogether the
NodeCountBoundary graph node type and its associated graph transformer,
CountBoundaryTransformer. We handle moves as a preprocessing step before
building the plan graph, so we no longer need to include any special nodes
in the graph to deal with that situation.

The test updates here are mainly for the graph builders themselves, to
acknowledge that indeed we're no longer inserting the NodeCountBoundary
vertices. The vertices that NodeCountBoundary previously depended on now
become dependencies of the special "root" vertex, although in many cases
here we don't see that explicitly because of the transitive reduction
algorithm, which notices when there's already an equivalent indirect
dependency chain and removes the redundant edge.

We already have plenty of test coverage for these "count boundary" cases
in the context tests whose names start with TestContext2Plan_count and
TestContext2Apply_resourceCount, all of which continued to pass here
without any modification and so are not visible in the diff. The test
functions particularly relevant to this situation are:
 - TestContext2Plan_countIncreaseFromNotSet
 - TestContext2Plan_countDecreaseToOne
 - TestContext2Plan_countOneIndex
 - TestContext2Apply_countDecreaseToOneCorrupted

The last of those in particular deals with the situation where we have
both a no-key instance _and_ a zero-key instance in the prior state, which
is interesting here because to exercises an intentional interaction
between refactoring.ImpliedMoveStatements and refactoring.ApplyMoves,
where we intentionally generate an implied move statement that produces
a collision and then expect ApplyMoves to deal with it in the same way as
it would deal with all other collisions, and thus ensure we handle both
the explicit and implied collisions in the same way.

This does affect some UI-level tests, because a nice side-effect of this
new treatment of this old feature is that we can now report explicitly
in the UI that we're assigning new addresses to these objects, whereas
before we just said nothing and hoped the user would just guess what had
happened and why they therefore weren't seeing a diff.

The backend/local plan tests actually had a pre-existing bug where they
were using a state with a different instance key than the config called
for but getting away with it because we'd previously silently fix it up.
That's still fixed up, but now done with an explicit mention in the UI
and so I made the state consistent with the configuration here so that the
tests would be able to recognize _real_ differences where present, as
opposed to the errant difference caused by that inconsistency.
2021-09-20 09:06:22 -07:00
Martin Atkins
332ea1f233 backend/local: TestLocal_plan_context_error to fail terraform.NewContext
The original intent of this test was to verify that we properly release
the state lock if terraform.NewContext fails. This was in response to a
bug in an earlier version of Terraform where that wasn't true.

In the recent refactoring that made terraform.NewContext no longer
responsible for provider constraint/checksum verification, this test began
testing a failed plan operation instead, which left the error return path
from terraform.NewContext untested.

An invalid parallelism value is the one remaining case where
terraform.NewContext can return an error, so as a localized fix for this
test I've switched it to just intentionally set an invalid parallelism
value. This is still not ideal because it's still testing an
implementation detail, but I've at least left a comment inline to try to
be clearer about what the goal is here so that we can respond in a more
appropriate way if future changes cause this test to fail again.

In the long run I'd like to move this last remaining check out to be the
responsibility of the CLI layer, with terraform.NewContext either just
assuming the value correct or panicking when it isn't, but the handling
of this CLI option is currently rather awkwardly spread across the
command and backend packages so we'll save that refactoring for a later
date.
2021-09-14 10:35:08 -07:00
Martin Atkins
718fa3895f backend: Remove Operation.Parallelism field
The presence of this field was confusing because in practice the local
backend doesn't use it for anything and the remote backend was using it
only to return an error if it's set to anything other than the default,
under the assumption that it would always match ContextOpts.Parallelism.

The "command" package is the one actually responsible for handling this
option, and it does so by placing it into the partial ContextOpts which it
passes into the backend when preparing for a local operation. To make that
clearer, here we remove Operation.Parallelism and change the few uses of
it to refer to ContextOpts.Parallelism instead, so that everyone is
reading and writing this value from the same place.
2021-09-14 10:35:08 -07:00
James Bardin
d0993b0e80 fix temp directory handling in some tests
Cleanup some more test fixtures to use t.TempDir

Use EvalSymlinks with temp dir paths to help with MacOS errors from
various terraform components.
2021-09-13 13:45:04 -04:00
James Bardin
8ed9a270e5
Merge pull request #29559 from hashicorp/jbardin/optional-attrs
Prevent `ObjectWithOptionalAttrs` from escaping
2021-09-13 08:58:11 -04:00
James Bardin
53a73a8ab6 configs: add ConstraintType to config.Variable
In order to handle optional attributes, the Variable type needs to keep
track of the type constraint for decoding and conversion, as well as the
concrete type for creating values and type comparison.

Since the Type field is referenced throughout the codebase, and for
future refactoring if the handling of optional attributes changes
significantly, the constraint is now loaded into an entirely new field
called ConstraintType. This prevents types containing
ObjectWithOptionalAttrs from escaping the decode/conversion codepaths
into the rest of the codebase.
2021-09-13 08:51:32 -04:00
Martin Atkins
343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
Valeriy Pastushenko
7c7e5d8f0a Don't show data while input if sensitive 2021-09-03 18:33:10 +03:00
James Bardin
863963e7a6 de-linting 2021-09-01 11:36:21 -04:00
Martin Atkins
89b05050ec core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.

This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.

Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.

However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.

My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
 - The "terraform graph" command is no longer hooked directly into the
   core graph builders, because that's no longer part of the public API.
   However, I did include a couple new Context functions whose contract
   is to produce a UI-oriented graph, and _for now_ those continue to
   return the physical graph we use for those operations. There's no
   exported API for generating the "validate" and "eval" graphs, because
   neither is particularly interesting in its own right, and so
   "terraform graph" no longer supports those graph types.
 - terraform.NewContext no longer has the responsibility for collecting
   all of the provider schemas up front. Instead, we wait until we need
   them. However, that means that some of our error messages now have a
   slightly different shape due to unwinding through a differently-shaped
   call stack. As of this commit we also end up reloading the schemas
   multiple times in some cases, which is functionally acceptable but
   likely represents a performance regression. I intend to rework this to
   use caching, but I'm saving that for a later commit because this one is
   big enough already.

The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.

However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-30 13:59:14 -07:00
Alex Khaerov
9d5f1752c8
oss backend: flattern assume_role block 2021-08-03 14:26:15 +08:00
James Bardin
97a2694528
Merge pull request #28838 from remilapeyre/consul-size-limit
Fix handling large states in the Consul backend
2021-07-30 14:18:34 -04:00
Rémi Lapeyre
da6717761e Merge remote-tracking branch 'origin/main' into update-consul 2021-07-28 12:18:01 +02:00
James Bardin
570b70b02f
Merge pull request #28078 from jasons42/configure-etcdv3-client-max-request-size
Expose etcd client MaxCallSendMsgSize config
2021-07-20 15:49:14 -04:00
James Bardin
8dd722ece0
Merge pull request #29167 from xiaozhu36/xiaozhu
backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug
2021-07-20 15:12:31 -04:00
James Bardin
431aa0280e
Merge pull request #29157 from remilapeyre/unique-constraint
Add uniqueness constraint on workspaces name for the pg backend
2021-07-20 15:11:35 -04:00
Jason Smith
d1608d7a7f Expose etcd client MaxCallSendMsgSize config
The etcdv3 client has a default request send limit of 2.0 MiB. This change
exposes the configuration option to increase that limit enabling larger
state using the etcdv3 backend.

This also requires that the corresponding --max-request-bytes flag be
increased on the server side. The default there is 1.5 MiB.

Fixes https://github.com/hashicorp/terraform/issues/25745
2021-07-20 14:04:45 -05:00
James Bardin
dd330e5194
Merge pull request #29200 from hashicorp/jbardin/rebase-25554
manual rebase of #25554
2021-07-20 14:29:56 -04:00
Kevin Burke
c047958b57 go.mod,backend: update coreos/etcd dependency to release-3.4 branch
etcd rewrote its import path from coreos/etcd to go.etcd.io/etcd.
Changed the imports path in this commit, which also updates the code
version.

This lets us remove the github.com/ugorji/go/codec dependency, which
was pinned to a fairly old version. The net change is a loss of 30,000
lines of code in the vendor directory. (I first noticed this problem
because the outdated go/codec dependency was causing a dependency
failure when I tried to put Terraform and another project in the same
vendor directory.)

Note the version shows up funkily in go.mod, but I verified
visually it's the same commit as the "release-3.4" tag in
github.com/coreos/etcd. The etcd team plans to fix the release version
tagging in v3.5, which should be released soon.
2021-07-20 12:27:22 -04:00
James Bardin
9c20ed6185 StateMgr must be able to return with locked state
The current usage of internal remote state backends requires that
`StateMgr` be able to return an instance of `statemgr.Full` even if the
state is currently locked.
2021-07-20 12:10:34 -04:00
xiaozhu36
c495caafeb backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug 2021-07-15 16:07:18 +08:00
Rémi Lapeyre
3c34e15d40 Update the Consul API client
Some users would want to use Consul namespaces when using the Consul
backend but the version of the Consul API client we use is too old and
don't support them. In preparation for this change this patch just update
it the client and replace testutil.NewTestServerConfig() by
testutil.NewTestServerConfigT() in the tests.
2021-07-14 14:49:13 +02:00
Rémi Lapeyre
b8e0d6f418 Add UNIQUE constraint in the states table for the pg backend 2021-07-14 11:59:14 +02:00
Rémi Lapeyre
3bf053a5fc Add a test to ensure that workspaces must have a unique name in pg backend 2021-07-14 11:57:28 +02:00
Nick Fagerlund
ab70879b9b Remote backend: Stop setting message when creating runs
Historically, we've used TFC's default run messages as a sort of dumping
ground for metadata about the run. We've recently decided to mostly stop
doing that, in favor of:

- Only specifying the run's source in the default message.
- Letting TFC itself handle the default messages.

Today, the remote backend explicitly sets a run message, overriding
any default that TFC might set. This commit removes that explicit message
so we can allow TFC to sort it out.

This shouldn't have any bad effect on TFE out in the wild, because it's
known how to set a default message for remote backend runs since late 2018.
2021-06-09 11:07:39 -07:00
Rémi Lapeyre
5f444a6862 Fix handling large states in the Consul backend 2021-05-28 21:31:07 +02:00
Rémi Lapeyre
e5fb84a01a Add test for storing state just under the size limit of the Consul backend
Storing a state just a bit smaller than the limit of the KV API fails
because we are using the Transaction API which has an overhead.
2021-05-28 21:30:56 +02:00
CJ Horton
2f980f1dfe
Merge pull request #28820 from hashicorp/radditude/refresh-error-message
backend/remote: point refresh users towards -refresh-only
2021-05-27 07:21:37 -07:00
CJ Horton
314d322b59 backend/remote: encourage use of -refresh-only 2021-05-26 23:08:39 -07:00
Alisdair McDiarmid
a3b1764fd7 backend/local: Fix lock when applying stale plan
When returning from the context method, a deferred function call checked
for error diagnostics in the `diags` variable, and unlocked the state if
any exist. This means that we need to be extra careful to mutate that
variable when returning errors, rather than returning a different set of
diags in the same position.

Previously this would result in an invalid plan file causing a lock to
become stuck.
2021-05-26 16:05:58 -04:00
CJ Horton
ab5fec9aec backend/remote: test new options and modes
Tests for -refresh=false, -refresh-only, and -replace,
as well as tests to ensure we error appropriately on
mismatched API versions.
2021-05-18 15:35:37 -07:00
CJ Horton
eb1f113693 backend/remote: validate API version for refresh and replace 2021-05-18 15:35:37 -07:00
Chris Arcand
2171f8a1a6 backend/remote: Support replacements in plans 2021-05-17 16:31:34 -07:00
Chris Arcand
a436c7fa2f backend/remote: Support refresh-only plans 2021-05-17 16:31:33 -07:00
Chris Arcand
cb49a4c8d6 backend/remote: Remove the -refresh=false restriction 2021-05-17 16:31:32 -07:00
James Bardin
4d55e2604f save apply diagnostics on backend failure
Make sure to collect apply diagnostics before persisting the backend
state so runtime errors are not lost.
2021-05-17 17:22:04 -04:00
Martin Atkins
36d0a50427 Move terraform/ to internal/terraform/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
f40800b3a4 Move states/ to internal/states/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
034e944070 Move plans/ to internal/plans/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
31349a9c3a Move configs/ to internal/configs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
ffe056bacb Move command/ to internal/command/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
b9a93a0fe7 Move addrs/ to internal/addrs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
1409f30f9c Move providers/ to internal/providers/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
05caff2ca3 Move tfdiags/ to internal/tfdiags/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
4c254cc2be Move httpclient/ to internal/httpclient/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins
73dda868cc Move backend/ to internal/backend/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00