As part of ongoing research into Terraform testing we'd like to use an
experimental feature to validate our current understanding that expressing
tests as part of the Terraform language, as opposed to in some other
language run alongside, is a good and viable way to write practical
module integration tests.
This initial experimental incarnation of that idea is implemented as a
provider, just because that's an easier extension point for research
purposes than a first-class language feature would be. Whether this would
ultimately emerge as a provider similar to this or as custom language
constructs will be a matter for future research, if this first
experiment confirms that tests written in the Terraform language are the
best direction to take.
The previous incarnation of this experiment was an externally-developed
provider apparentlymart/testing, listed on the Terraform Registry. That
helped with showing that there are some useful tests that we can write
in the Terraform language, but integrating such a provider into Terraform
will allow us to make use of it in the also-experimental "terraform test"
command, which will follow in subsequent commits, to see how this might
fit into a development workflow.
Changes:
```
* backend/s3: Support for AWS Single-Sign On (SSO) cached credentials
```
Updated via:
```
go get github.com/aws/aws-sdk-go@v1.37.0
go mod tidy
```
Please note that Terraform CLI will not initiate or perform the AWS SSO login flow. It is expected that you have already performed the SSO login flow using AWS CLI using the `aws sso login` command, or by some other mechanism before executing Terraform. More precisely, this credential handling must find a valid non-expired access token for the AWS SSO user portal URL in `~/.aws/sso/cache`. If a cached token is not found, is expired, or the file is malformed an error will be returned.
You can use configure AWS SSO credentials from the AWS shared configuration file by specifying the required keys in the profile:
```
sso_account_id
sso_region
sso_role_name
sso_start_url
```
For example, the following defines a profile "devsso" and specifies the AWS SSO parameters that defines the target account, role, sign-on portal, and the region where the user portal is located. Note: all SSO arguments must be provided, or an error will be returned.
```
[profile devsso]
sso_start_url = https:my-sso-portal.awsapps.com/start
sso_role_name = SSOReadOnlyRole
sso_region = us-east-1
sso_account_id = 123456789012
```
Additional Resources
* [Configuring the AWS CLI to use AWS Single Sign-On](https:docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html)
* [AWS Single Sign-On User Guide](https:docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html)
Here we propagate in the initialized terminal.Streams from package main,
and then onwards to backends running in CLI mode.
This also replaces our use of helper/wrappedstreams to determine whether
stdin is a terminal or a pipe. helper/wrappedstreams returns incorrect
file descriptors on Windows, causing StdinPiped to always return false on
that platform and thus causing one of the odd behaviors discussed in
Finally, this includes some wrappers around the ability to look up the
number of columns in the terminal in preparation for use elsewhere. These
wrappers deal with the fact that our unit tests typically won't populate
meta.Streams.
This is a helper package that creates a very thin abstraction over
terminal setup, with the main goal being to deal with all of the extra
setup we need to do in order to get a UTF-8-supporting virtual terminal
on a Windows system.
The upstream bug with opening a browser on Windows Subsystem for Linux
has been fixed, so this reverts our local patch for this. The approach
upstream adds fallback support for x-www-browser and www-browser if
xdg-open fails, and this fixes the problem on WSL.
This reverts commit 12e090ce48.
This is needed to make it possible to use the scram-sha-256
authentication method for the pg backend. It's not easy to write
unit-tests for this since it requires a specific configuration of the
PostgreSQL server, I did test it manually thought and everything seems
to work like it should.
Closes https://github.com/hashicorp/terraform/issues/24016
* github.com/Azure/azure-sdk-for-go to v47.1.0
* github.com/Azure/go-autorest to v0.11.10
* github.com/hashicorp/go-azure-helpers to v0.13.0
* github.com/tombuildsstuff/giovanni to v0.14.0
Added an Off level to hclog, so we can individually disable logging at
various levels.
Added IndependentLevels so that sublogger levels are not linked to their
parents.
The main process is now handling what output to print, so it doesn't do
any good to try and run it through prefixedio, which is only adding
extra coordination to echo the same data.
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).
I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.
helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).
There's also a `go mod tidy` included.
Update tests to match the fix in mitchellh/cli#71, which aligns MockUi
with BasicUi and allows newlines in user input.
We are not using the new ErrorWriter, added in mitchellh/cli#81, as it
does not appear to interact correctly with panicwrap. All error output
from CLI parsing will continue to appear on stdout, not stderr.
We've not been using HIL in the main codepaths since Terraform 0.12, but
some references to it (and some supporting functionality in Terraform)
stuck around due to interactions with types we'd kept around to support
legacy shims.
However, removing the configs.RawConfig field from
terraform.ResourceConfig disconnects that subtree of dependencies from
everything else, allowing us to remove it. This is safe because the only
remaining uses of terraform.ResourceConfig are shims from values that
were already evaluated using the HCL 2 API, and thus they never need
the "just in time" HIL evaluation that ResourceConfig.interpolateForce
used to do.
We also had some HIL references in configs/hcl2shim that were previously
in support of the "terraform 0.12upgrade" command, but the implementation
of that command is now removed.
There was one remaining reference to HIL in a now-unused function in the
helper/schema package, which I removed entirely here.
This then allows us to remove the HIL dependency entirely, and also to
clean up some remaining old remants of the legacy "config" package that
we'd recently moved into the "configs" package pending further pruning.
* updating `github.com/Azure/azure-sdk-for-go` to `v45.0.0`
* updating `github.com/Azure/go-autorest` to `v0.11.3`
* updating `github.com/hashicorp/go-azure-helpers` to `v0.12.0`
* updating `github.com/tombuildsstuff/giovanni` to `v0.12.0`
This is for consistency with other commands which use prompts, all of
which require "yes" rather than "y" to confirm.
We also migrate the login command to use UIInput, which now supports
securely asking for passwords or secrets via the speakeasy library.
This new version permits omitting the space between the operator and the
boundary in a ruby-style version constraint, like ">1.0.0" instead of
"> 1.0.0".
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.
To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.
As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.
This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.
This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.
Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
* update vendored azure sdk
* vendor giovanni storage sdk
* Add giovanni clients
* go mod vendor
* Swap to new storage sdk
* workable tests
* update .go-version to 1.14.2
* Tests working minus SAS
* Add SAS Token support
* Update vendor
* Passing tests
* Add date randomizer
* Captalize RG
* Remove random bits
* Update client var name
Co-authored-by: kt <kt@katbyte.me>
This includes a new TargetAddrs field on both Run and RunCreateOptions
which we'll use to send resource addresses that were specified using
-target on the CLI command line when using the remote backend.
There were some unrelated upstream breaking changes compared to the last
version we had vendored, so this commit also includes some changes to the
backend/remote package to work with this new API, which now requires the
remote backend to be aware of the remote system's opaque workspace id.
* update github.com/aws/aws-sdk-go to v1.30.9
* deps: github.com/aws/aws-sdk-go@v1.30.12
Reference: https://github.com/hashicorp/terraform/issues/24710
Reference: https://github.com/hashicorp/terraform/issues/24741
Changes:
```
NOTES:
* backend/s3: Region validation now automatically supports the new `af-south-1` (Africa (Cape Town)) region. For AWS operations to work in the new region, the region must be explicitly enabled as outlined in the [AWS Documentation](https://docs.aws.amazon.com/general/latest/gr/rande-manage.html#rande-manage-enable). When the region is not enabled, the Terraform S3 Backend will return errors during credential validation (e.g. `error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid`).
ENHANCEMENTS:
* backend/s3: Support automatic region validation for `af-south-1`
```
Updated via:
```console
$ go get github.com/aws/aws-sdk-go@v1.30.12
$ go mod tidy
$ go mod vendor
```
Output from acceptance testing:
```console
$ TF_ACC=1 go test -v ./backend/remote-state/s3 | grep '^--- '
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (1.68s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackend (15.07s)
--- PASS: TestBackendLocked (26.40s)
--- PASS: TestBackendSSECustomerKey (16.99s)
--- PASS: TestBackendExtraPaths (12.05s)
--- PASS: TestBackendPrefixInWorkspace (5.55s)
--- PASS: TestKeyEnv (45.07s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (5.39s)
--- PASS: TestRemoteClientLocks (14.30s)
--- PASS: TestForceUnlock (20.08s)
--- PASS: TestRemoteClient_clientMD5 (16.43s)
--- PASS: TestRemoteClient_stateChecksum (24.58s)
```
Co-authored-by: Nicola Senno <nicola.senno@workday.com>
This fixes several bugs:
- `substr("abc", 0, 0)` would previously return `"abc"`, despite the
length argument being `0`. This has been changed to return an empty
string when length is zero.
- `ceil(1/0)` and `floor(1/0)` would previously return a large integer
value, rather than infinity. This has been fixed.
This library implements the user-specific directory layout specifications
for various platforms (XDG on Unix, "Known Folders" on Windows, etc).
We'll use this in a subsequent commit to add additional system-specific
search directories for provider plugins, and perhaps later on also
CLI configuration directories.
It seems that the checksum for v3.3.10+incompatible has changed at some
point, causing "go mod vendor" to fail now.
We can see by the fact that no files within "vendor" have changed that the
change in checksum is not the result of any material change in the module
code, and therefore presumably resulted from some change in metadata or
a change in the Go module hashing algorithm since Go 1.12.
Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries.
Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump.
Co-authored-by: Paul Tyng <paul@paultyng.net>
* add TencentCloud COS backend for remote state
* add vendor of dependence
* fixed error not handle and remove default value for prefix argument
* get appid from TF_COS_APPID environment variables
This brings in the new HCL extension functions "try", "can", and
"convert", along with the underlying HCL and cty infrastructure that allow
them to work.
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block
With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
This also includes an upgrade to cty v1.1.1 because HCL calls for it.
The changes in these two libraries are mainly to codepaths that don't
directly affect Terraform, but including this upgrade will cause some
small improvements to Terraform's error messages for type conversion
problems.
In order to make this work reasonably we can't avoid using some funny
heuristics, which are somewhat reasonable to apply within the context of
Terraform itself but would not be good to add to the general "logutils".
Specifically, this is adding the additional heuristic that lines starting
with spaces are continuation lines and so should inherit the log level
of the most recent non-continuation line.
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.
This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.
For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
This new version includes Solaris support, the lack of which previously
caused us to disable readline-using features ("terraform console") on
Solaris builds.
We'll use this to call the Windows ReplaceFile API for safe file
replacement when updating credentials.tf.json in "terraform login" and
"terraform logout".
The dependencies here are dated and are causing conflicts with the
ACME provider, namely the version of the top-level autorest package.
This explicitly updates the Azure SDK and autorest packages, with the
separately versioned sub-packages being added automatically.
Notable changes (from Terraform AWS Provider CHANGELOG):
```
NOTES:
* backend/s3: Region validation now automatically supports the new `me-south-1` Middle East (Bahrain) region. For AWS operations to work in the new region, the region must be explicitly enabled as outlined in the [previous new region announcement blog post](https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-hong-kong-region/). When the region is not enabled, the Terraform S3 Backend will return errors during credential validation (e.g. `error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid`).
* backend/s3: After this update, the AWS Go SDK will prefer credentials found via the `AWS_PROFILE` environment variable when both the `AWS_PROFILE` environment variable and the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables are statically defined. Previously the SDK would ignore the `AWS_PROFILE` environment variable, if static environment credentials were also specified. This is listed as a bug fix in the AWS Go SDK release notes.
ENHANCEMENTS:
* backend/s3: Add support for assuming role via web identity token via the `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` environment variables
* backend/s3: Support automatic region validation for `me-south-1`
BUG FIXES:
* backend/s3: Load credentials via the `AWS_PROFILE` environment variable (if available) when `AWS_PROFILE` is defined along with `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
```
Updated via:
```
go get github.com/aws/aws-sdk-go@v1.21.7
go mod tidy
go mod vendor
```
Verification with this update:
```hcl
terraform {
backend "s3" {
bucket = "me-south-1-testing"
key = "test"
region = "me-south-1"
}
}
output "test" {
value = timestamp()
}
```
Outputs:
```
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
test = 2019-07-30T12:49:19Z
```
If the new region is not properly enabled for the account, errors like the below will be received:
```
$ terraform init
Initializing the backend...
Error: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
```
To use this region before this update:
```hcl
terraform {
# ... potentially other configuration ...
backend "s3" {
# ... other configuration ...
region = "me-south-1"
skip_region_validation = true
}
}
```
This includes a fix to make sure that an expression with a static string
index, like foo["bar"], will be parsed as a traversal rather than as a
dynamic index expression.
This module contains a YAML parser and encoder tailored to cty, though we
are mostly interested in it for its YAMLEncode and YAMLDecode cty
functions, which we can make available in Terraform.
This includes a small fix to ensure the parser doesn't produce an invalid
body for block parsing syntax errors, and instead produces an incomplete
result that calling applications like Terraform can still analyze.
The problem here was affecting our version-constraint-sniffing code, which
intentionally tried to find a core version constraint even if there's a
syntax error so that it can report that a new version of Terraform is a
likely cause of the syntax error. It was working in most cases, unless
it was the "terraform" block itself that contained the error, because then
we'd try to analyze a broken hcl.Block with a nil body.
This includes a new test for "terraform init" that exercises this
recovery codepath.
This corrects a bug in the HCL 2 scanner where a $ or % symbol would cause
incorrect tokenization if appearing immediately before a " .
This also includes some updates to Go extension libraries that the HCL
update brings in. Some of these changes update to support Unicode 11, but
only when compiling with Go 1.13, so we won't see the effect of these
changes until we start building Terraform with Go 1.13.
This contains an adjustment to how the dynamic blocks extension expands
a dynamic block whose for_each expression is unknown: it now produces an
block whose leaf attributes are all unknown, which is what Terraform had
previously been expecting but it wasn't actually true in practice.
cty now guarantees that sets of primitive values will iterate in a
reasonable order. Previously it was the caller's responsibility to deal
with that, but we invariably neglected to do so, causing inconsistent
ordering. Since cty prioritizes consistent behavior over performance, it
now imposes its own sort on set elements as part of iterating over them so
that calling applications don't have to worry so much about it.
This change also causes cty to consistently push unknown and null values
in sets to the end of iteration, where before that was undefined. This
means that our diff output will now consistently list additions before
removals when showing sets, rather than the ordering being undefined as
before.
The ordering of known, non-null, non-primitive values is still not
contractually fixed but remains consistent for a particular version of
cty.
By upgrading this to a version that supports Go Modules, we can avoid
the redundant dependencies labix.org/v2/mgo and launchpad.net/gocheck.
These dependencies come via hashicorp/consul, which we have vendored for
the Consul backend.
There are no changes to the go-msgpack code here; we're just adopting a
newer version that has a go.mod present so that its dependencies can be
determined more accurately.
This gives us an extra hook in the dynblock variables analysis that should
allow us to also make it subject also to the lang/blocktoattr fixup, to
ensure we'll find all the references in spite of these various
pre-processing wrappers.
This includes a fix to prevent unintentional infinite recursion when
trying to unify multiple object types to a single type for conversion to
list(any).
Sadly I wasn't able to reproduce the problem as reported (in #20728), so
therefore I wasn't able to write a Terraform test for it, but I have
confirmed that the cty behavior here was incorrect anyway (recursively
calling the same function we're already in with the same arguments is
clearly not productive) and so this change will allow whatever situation
that was to terminate with a type conversion error, rather than causing a
stack overflow.
It's likely that there is another bug lurking under this, since the
problematic code here was supposed to be unreachable, but avoiding the
crash is the priority for now. If the problem re-surfaces then it should
at least be an error message with some additional context about what the
goal of the caller was.
This also includes an unrelated fix for the gocty package, which doesn't
affect Terraform because it makes very little use of that package.
This includes improved functionality for HCL's "dynamic block extension",
which will allow us (in a subsequent commit) to properly detect
dependencies inside nested "dynamic" blocks, where currently they get
missed.
For this commit though, we just upgrade HCL to a version that includes it
and make a small change to our "lang" package to align with an upstream
renaming.
This includes two upstream fixes:
- Handle explicit JSON "null" consistently during decode of JSON syntax.
- Properly detect the end of a "heredoc" when formatting to avoid messing
up indentation of other lines following the heredoc.
This includes a fix for the parsing of object for expressions in newline-
sensitive contexts like block bodies.
It also includes a change to the JSON syntax decoder that cause it to
consider an explicit null to be equivalent to a property not being set at
all when interpreting a property value as a nested block. (It was
previously doing tha only when interpreting the property value as an
attribute value.)
Previously, configupgrade would panic if it encountered a HEREDOC. For
the time being, we will simply print out the HEREDOC as-is.
Unfortunately, we discovered that terraform 0.11's version of HCL
allowed for HEREDOCs with the termination delimiter inline (instead of
on a newline, which is technically correct). Since 0.12configupgrade
needs to be bug-compatible with terraform 0.11, we must roll back to the
same version of HCL used in terraform 0.11.
This fixes a bug in the TestConformance function that was generating false
positives when given two object types with the same number of attributes
but not identical attribute names.
The cty change here fixes a panic situation when cty.Path.Apply is given
a null value, making it now correctly return an error.
However, the HCL2 change includes an alternative to cty.Path.Apply that
uses HCL-level rules rather than cty-level rules, so the result behaves
like an HCL expression would. Most uses of cty.Path.Apply ought to use
hcl.ApplyPath instead, to ensure that the behavior is consistent with what
users expect in the main language.
An earlier commit incorrectly updated some versions in go.mod without also
updating the vendor tree, so this also rolls those back to where they used
to be so that we can roll them forward carefully and make sure the tests
actually pass. (If we just accept these new versions as specified the
tests do not pass, so some work is required to fix those regressions.)
This fixes some consistency problems with how number strings were parsed
in the msgpack decoder vs other situations.
This commit also includes an upgrade of HCL2 to use this new cty function,
though there's no change in behavior here since the new function is
functionally equivalent to what it replaced.
There are no code changes in this update. This just catches up with the
latest master commit in the upstream repo, which has only seen
documentation updates since the prior commit.
This includes a missing feature for the splat syntax (null.* returns an
empty tuple) and also fixes a bug in the source code formatter where it
was inserting spaces between two consecutive interpolation sequences, like
"${foo}${bar}".
This includes a change to accept and ignore a UTF-8 BOM at the start of
any given native syntax configuration.
Although a BOM is redundant in UTF-8, we learned in #18618 that several
software products on Windows will produce a BOM whenever they save as
UTF-8, so accepting it avoids friction when using those tools to author
or generate Terraform configuration files.
This fixes#18618.
This includes:
- An additional check in the format stdlib function to fail if there are
too many arguments given, rather than silently ignoring.
- Refinements for the type unification behavior to allow unification of
object/tuple types into weaker map/list types when no other unification
is possible.
- Improvements to the error messages for failed type conversions on
collection and structural types to talk about mismatching element types
where possible, rather than the outer value.
This includes a fix to hcl.RelTraversalForExpr where it would
inadvertantly modify the internals of a traversal AST node as part of
relativizing the traversal in order to return it.
This includes a number of upstream bug fixes, which in turn fix a number
of issues here in Terraform:
- New-style "full splat" operator now working correctly (#19181)
- The weird HCL1-ish single-line block syntax is now supported (#19153)
- Formatting of single-line blocks adds spaces around the braces (#19154)
This also includes a number of other upstream fixes that were not tracked
as issues in the Terraform repository. The highlights of those are:
- A for expression with the "for" keyword wrapped onto a newline after its
opening bracket now parses correctly.
- In JSON syntax, interpolation sequences in properties of objects that
are representing expressions now have their variables properly detected.
- The "flush" heredoc variant is now functional again after being broken
in some (much-)earlier rework of the template parser.
Notable changes:
* backend/s3: Automatic validation of `eu-north-1` region
* backend/s3: Support for `credential_process` handling in AWS configuration file
Updated via:
```
go get github.com/aws/aws-sdk-go@v1.16.4
go get github.com/terraform-providers/terraform-provider-aws@v1.52.0
go mod tidy
go mod vendor
```
* Upgrading to 2.0.0 of github.com/hashicorp/go-azure-helpers
* Support for authenticating using Azure CLI
* backend/azurerm: support for authenticating using the Azure CLI
