mirror of
https://github.com/opentofu/opentofu.git
synced 2025-01-12 00:52:35 -06:00
cliconfig: Allow forcing use of the plugin cache despite the lock file
Currently Terraform will use an entry from the global plugin cache only if it matches a checksum already recorded in the dependency lock file. This allows Terraform to produce a complete lock file entry on the first encounter with a new provider, whereas using the cache in that case would cause the lock file to only cover the single package in the cache and thereefore be unusable on any other operating system or CPU architecture. This temporary CLI config option is a pragmatic exception to support those who cannot currently correctly use the dependency lock file but who still want to benefit from the plugin cache. With this setting enabled, Terraform has permission to produce a dependency lock file that is only suitable for the current system if that would allow use of an existing entry in the plugin cache. We are introducing this option to resolve a conflict between the needs of folks who are using the dependency lock file as expected and the needs of folks who cannot use the dependency lock file for some reason. The hope then is to give respite to those who need this exception in the meantime while we understand better why they cannot use the dependency lock file and improve its design so that everyone will be able to use it successfully in a future version of Terraform. This option will become a silent no-op in a future version of Terraform, once the dependency lock file behavior is sufficient for all supported Terraform development workflows.
This commit is contained in:
Martin Atkins
parent
3cc7e55465
commit
e2380b1038
@ -95,6 +95,8 @@ func initCommands(
|
||||
CLIConfigDir: configDir,
|
||||
PluginCacheDir: config.PluginCacheDir,
|
||||
|
||||
PluginCacheMayBreakDependencyLockFile: config.PluginCacheMayBreakDependencyLockFile,
|
||||
|
||||
ShutdownCh: makeShutdownCh(),
|
||||
|
||||
ProviderSource: providerSrc,
|
||||
|
||||
@ -38,6 +38,17 @@ type Config struct {
|
||||
// avoid repeatedly re-downloading over the Internet.
|
||||
PluginCacheDir string `hcl:"plugin_cache_dir"`
|
||||
|
||||
// PluginCacheMayBreakDependencyLockFile is an interim accommodation for
|
||||
// those who wish to use the Plugin Cache Dir even in cases where doing so
|
||||
// will cause the dependency lock file to be incomplete.
|
||||
//
|
||||
// This is likely to become a silent no-op in future Terraform versions but
|
||||
// is here in recognition of the fact that the dependency lock file is not
|
||||
// yet a good fit for all Terraform workflows and folks in that category
|
||||
// would prefer to have the plugin cache dir's behavior to take priority
|
||||
// over the requirements of the dependency lock file.
|
||||
PluginCacheMayBreakDependencyLockFile bool `hcl:"plugin_cache_may_break_dependency_lock_file"`
|
||||
|
||||
Hosts map[string]*ConfigHost `hcl:"host"`
|
||||
|
||||
Credentials map[string]map[string]interface{} `hcl:"credentials"`
|
||||
|
||||
@ -103,6 +103,22 @@ type Meta struct {
|
||||
// into the given directory.
|
||||
PluginCacheDir string
|
||||
|
||||
// PluginCacheMayBreakDependencyLockFile is a temporary CLI configuration-based
|
||||
// opt out for the behavior of only using the plugin cache dir if its
|
||||
// contents match checksums recorded in the dependency lock file.
|
||||
//
|
||||
// This is an accommodation for those who currently essentially ignore the
|
||||
// dependency lock file -- treating it only as transient working directory
|
||||
// state -- and therefore don't care if the plugin cache dir causes the
|
||||
// checksums inside to only be sufficient for the computer where Terraform
|
||||
// is currently running.
|
||||
//
|
||||
// We intend to remove this exception again (making the CLI configuration
|
||||
// setting a silent no-op) in future once we've improved the dependency
|
||||
// lock file mechanism so that it's usable for everyone and there are no
|
||||
// longer any compelling reasons for folks to not lock their dependencies.
|
||||
PluginCacheMayBreakDependencyLockFile bool
|
||||
|
||||
// ProviderSource allows determining the available versions of a provider
|
||||
// and determines where a distribution package for a particular
|
||||
// provider version can be obtained.
|
||||
|
||||
@ -63,6 +63,7 @@ func (m *Meta) providerInstallerCustomSource(source getproviders.Source) *provid
|
||||
inst := providercache.NewInstaller(targetDir, source)
|
||||
if globalCacheDir != nil {
|
||||
inst.SetGlobalCacheDir(globalCacheDir)
|
||||
inst.SetGlobalCacheDirMayBreakDependencyLockFile(m.PluginCacheMayBreakDependencyLockFile)
|
||||
}
|
||||
var builtinProviderTypes []string
|
||||
for ty := range m.internalProviders() {
|
||||
|
||||
@ -3,6 +3,7 @@ package providercache
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
@ -36,6 +37,12 @@ type Installer struct {
|
||||
// version between different configurations on the same system.
|
||||
globalCacheDir *Dir
|
||||
|
||||
// globalCacheDirMayBreakDependencyLockFile allows a temporary exception to
|
||||
// the rule that an entry in globalCacheDir can normally only be used if
|
||||
// its validity is already confirmed by an entry in the dependency lock
|
||||
// file.
|
||||
globalCacheDirMayBreakDependencyLockFile bool
|
||||
|
||||
// builtInProviderTypes is an optional set of types that should be
|
||||
// considered valid to appear in the special terraform.io/builtin/...
|
||||
// namespace, which we use for providers that are built in to Terraform
|
||||
@ -103,6 +110,19 @@ func (i *Installer) SetGlobalCacheDir(cacheDir *Dir) {
|
||||
i.globalCacheDir = cacheDir
|
||||
}
|
||||
|
||||
// SetGlobalCacheDirMayBreakDependencyLockFile activates or deactivates our
|
||||
// temporary exception to the rule that the global cache directory can be used
|
||||
// only when entries are confirmed by existing entries in the dependency lock
|
||||
// file.
|
||||
//
|
||||
// If this is set then if we install a provider for the first time from the
|
||||
// cache then the dependency lock file will include only the checksum from
|
||||
// the package in the global cache, which means the lock file won't be portable
|
||||
// to Terraform running on another operating system or CPU architecture.
|
||||
func (i *Installer) SetGlobalCacheDirMayBreakDependencyLockFile(mayBreak bool) {
|
||||
i.globalCacheDirMayBreakDependencyLockFile = mayBreak
|
||||
}
|
||||
|
||||
// HasGlobalCacheDir returns true if someone has previously called
|
||||
// SetGlobalCacheDir to configure a global cache directory for this installer.
|
||||
func (i *Installer) HasGlobalCacheDir() bool {
|
||||
@ -375,6 +395,41 @@ NeedProvider:
|
||||
}
|
||||
}
|
||||
|
||||
if !acceptablePackage && i.globalCacheDirMayBreakDependencyLockFile {
|
||||
// The "may break dependency lock file" setting effectively
|
||||
// means that we'll accept any matching package that's
|
||||
// already in the cache, regardless of whether it matches
|
||||
// what's in the dependency lock file.
|
||||
//
|
||||
// That means two less-ideal situations might occur:
|
||||
// - If this provider is not currently tracked in the lock
|
||||
// file at all then after installation the lock file will
|
||||
// only accept the package that was already present in
|
||||
// the cache as a valid checksum. That means the generated
|
||||
// lock file won't be portable to other operating systems
|
||||
// or CPU architectures.
|
||||
// - If the provider _is_ currently tracked in the lock file
|
||||
// but the checksums there don't match what was in the
|
||||
// cache then the LinkFromOtherCache call below will
|
||||
// fail with a checksum error, and the user will need to
|
||||
// either manually remove the entry from the lock file
|
||||
// or remove the mismatching item from the cache,
|
||||
// depending on which of these they prefer to use as the
|
||||
// source of truth for the expected contents of the
|
||||
// package.
|
||||
//
|
||||
// If the lock file already includes this provider and the
|
||||
// cache entry matches one of the locked checksums then
|
||||
// there's no problem, but in that case we wouldn't enter
|
||||
// this branch because acceptablePackage would already be
|
||||
// true from the check above.
|
||||
log.Printf(
|
||||
"[WARN] plugin_cache_may_break_dependency_lock_file: Using global cache dir package for %s v%s even though it doesn't match this configuration's dependency lock file",
|
||||
provider.String(), version.String(),
|
||||
)
|
||||
acceptablePackage = true
|
||||
}
|
||||
|
||||
// TODO: Should we emit an event through the events object
|
||||
// for "there was an entry in the cache but we ignored it
|
||||
// because the checksum didn't match"? We can't use
|
||||
|
||||
@ -3,6 +3,7 @@ package providercache
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@ -809,6 +810,281 @@ func TestEnsureProviderVersions(t *testing.T) {
|
||||
}
|
||||
},
|
||||
},
|
||||
"successful initial install of one provider through a warm global cache without a lock file entry but allowing the cache to break the lock file": {
|
||||
Source: getproviders.NewMockSource(
|
||||
[]getproviders.PackageMeta{
|
||||
{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.0.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.1.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
},
|
||||
nil,
|
||||
),
|
||||
LockFile: `
|
||||
# (intentionally empty)
|
||||
`,
|
||||
Prepare: func(t *testing.T, inst *Installer, dir *Dir) {
|
||||
globalCacheDirPath := tmpDir(t)
|
||||
globalCacheDir := NewDirWithPlatform(globalCacheDirPath, fakePlatform)
|
||||
_, err := globalCacheDir.InstallPackage(
|
||||
context.Background(),
|
||||
getproviders.PackageMeta{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.1.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to populate global cache: %s", err)
|
||||
}
|
||||
inst.SetGlobalCacheDir(globalCacheDir)
|
||||
inst.SetGlobalCacheDirMayBreakDependencyLockFile(true)
|
||||
},
|
||||
Mode: InstallNewProvidersOnly,
|
||||
Reqs: getproviders.Requirements{
|
||||
beepProvider: getproviders.MustParseVersionConstraints(">= 2.0.0"),
|
||||
},
|
||||
Check: func(t *testing.T, dir *Dir, locks *depsfile.Locks) {
|
||||
if allCached := dir.AllAvailablePackages(); len(allCached) != 1 {
|
||||
t.Errorf("wrong number of cache directory entries; want only one\n%s", spew.Sdump(allCached))
|
||||
}
|
||||
if allLocked := locks.AllProviders(); len(allLocked) != 1 {
|
||||
t.Errorf("wrong number of provider lock entries; want only one\n%s", spew.Sdump(allLocked))
|
||||
}
|
||||
|
||||
gotLock := locks.Provider(beepProvider)
|
||||
wantLock := depsfile.NewProviderLock(
|
||||
beepProvider,
|
||||
getproviders.MustParseVersion("2.1.0"),
|
||||
getproviders.MustParseVersionConstraints(">= 2.0.0"),
|
||||
[]getproviders.Hash{beepProviderHash},
|
||||
)
|
||||
if diff := cmp.Diff(wantLock, gotLock, depsfile.ProviderLockComparer); diff != "" {
|
||||
t.Errorf("wrong lock entry\n%s", diff)
|
||||
}
|
||||
|
||||
gotEntry := dir.ProviderLatestVersion(beepProvider)
|
||||
wantEntry := &CachedProvider{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.1.0"),
|
||||
PackageDir: filepath.Join(dir.BasePath(), "example.com/foo/beep/2.1.0/bleep_bloop"),
|
||||
}
|
||||
if diff := cmp.Diff(wantEntry, gotEntry); diff != "" {
|
||||
t.Errorf("wrong cache entry\n%s", diff)
|
||||
}
|
||||
},
|
||||
WantEvents: func(inst *Installer, dir *Dir) map[addrs.Provider][]*testInstallerEventLogItem {
|
||||
return map[addrs.Provider][]*testInstallerEventLogItem{
|
||||
noProvider: {
|
||||
{
|
||||
Event: "PendingProviders",
|
||||
Args: map[addrs.Provider]getproviders.VersionConstraints{
|
||||
beepProvider: getproviders.MustParseVersionConstraints(">= 2.0.0"),
|
||||
},
|
||||
},
|
||||
},
|
||||
beepProvider: {
|
||||
{
|
||||
Event: "QueryPackagesBegin",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Constraints string
|
||||
Locked bool
|
||||
}{">= 2.0.0", false},
|
||||
},
|
||||
{
|
||||
Event: "QueryPackagesSuccess",
|
||||
Provider: beepProvider,
|
||||
Args: "2.1.0",
|
||||
},
|
||||
{
|
||||
Event: "LinkFromCacheBegin",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Version string
|
||||
CacheRoot string
|
||||
}{
|
||||
"2.1.0",
|
||||
inst.globalCacheDir.BasePath(),
|
||||
},
|
||||
},
|
||||
{
|
||||
Event: "ProvidersLockUpdated",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Version string
|
||||
Local []getproviders.Hash
|
||||
Signed []getproviders.Hash
|
||||
Prior []getproviders.Hash
|
||||
}{
|
||||
"2.1.0",
|
||||
[]getproviders.Hash{"h1:2y06Ykj0FRneZfGCTxI9wRTori8iB7ZL5kQ6YyEnh84="},
|
||||
nil,
|
||||
nil,
|
||||
},
|
||||
},
|
||||
{
|
||||
Event: "LinkFromCacheSuccess",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Version string
|
||||
LocalDir string
|
||||
}{
|
||||
"2.1.0",
|
||||
filepath.Join(dir.BasePath(), "/example.com/foo/beep/2.1.0/bleep_bloop"),
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
},
|
||||
},
|
||||
"failing install of one provider through a warm global cache with an incorrect locked checksum while allowing the cache to break the lock file": {
|
||||
Source: getproviders.NewMockSource(
|
||||
[]getproviders.PackageMeta{
|
||||
{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.0.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.1.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
},
|
||||
nil,
|
||||
),
|
||||
LockFile: `
|
||||
# The existing cache entry is valid only if it matches a
|
||||
# checksum already recorded in the lock file, but this
|
||||
# test is overriding that rule using a special setting.
|
||||
provider "example.com/foo/beep" {
|
||||
version = "2.1.0"
|
||||
constraints = ">= 1.0.0"
|
||||
hashes = [
|
||||
"h1:wrong-not-matchy",
|
||||
]
|
||||
}
|
||||
`,
|
||||
Prepare: func(t *testing.T, inst *Installer, dir *Dir) {
|
||||
globalCacheDirPath := tmpDir(t)
|
||||
globalCacheDir := NewDirWithPlatform(globalCacheDirPath, fakePlatform)
|
||||
_, err := globalCacheDir.InstallPackage(
|
||||
context.Background(),
|
||||
getproviders.PackageMeta{
|
||||
Provider: beepProvider,
|
||||
Version: getproviders.MustParseVersion("2.1.0"),
|
||||
TargetPlatform: fakePlatform,
|
||||
Location: beepProviderDir,
|
||||
},
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to populate global cache: %s", err)
|
||||
}
|
||||
inst.SetGlobalCacheDir(globalCacheDir)
|
||||
inst.SetGlobalCacheDirMayBreakDependencyLockFile(true)
|
||||
},
|
||||
Mode: InstallNewProvidersOnly,
|
||||
Reqs: getproviders.Requirements{
|
||||
beepProvider: getproviders.MustParseVersionConstraints(">= 2.0.0"),
|
||||
},
|
||||
Check: func(t *testing.T, dir *Dir, locks *depsfile.Locks) {
|
||||
if allCached := dir.AllAvailablePackages(); len(allCached) != 0 {
|
||||
t.Errorf("wrong number of cache directory entries; want none\n%s", spew.Sdump(allCached))
|
||||
}
|
||||
if allLocked := locks.AllProviders(); len(allLocked) != 1 {
|
||||
t.Errorf("wrong number of provider lock entries; want only one\n%s", spew.Sdump(allLocked))
|
||||
}
|
||||
|
||||
gotLock := locks.Provider(beepProvider)
|
||||
wantLock := depsfile.NewProviderLock(
|
||||
// The lock file entry hasn't changed because the cache
|
||||
// entry didn't match the existing lock file entry.
|
||||
beepProvider,
|
||||
getproviders.MustParseVersion("2.1.0"),
|
||||
getproviders.MustParseVersionConstraints(">= 1.0.0"),
|
||||
[]getproviders.Hash{"h1:wrong-not-matchy"},
|
||||
)
|
||||
if diff := cmp.Diff(wantLock, gotLock, depsfile.ProviderLockComparer); diff != "" {
|
||||
t.Errorf("wrong lock entry\n%s", diff)
|
||||
}
|
||||
|
||||
// The provider wasn't installed into the local cache directory
|
||||
// because that would make the local cache mismatch the
|
||||
// lock file.
|
||||
gotEntry := dir.ProviderLatestVersion(beepProvider)
|
||||
wantEntry := (*CachedProvider)(nil)
|
||||
if diff := cmp.Diff(wantEntry, gotEntry); diff != "" {
|
||||
t.Errorf("wrong cache entry\n%s", diff)
|
||||
}
|
||||
},
|
||||
WantErr: `doesn't match any of the checksums`,
|
||||
WantEvents: func(inst *Installer, dir *Dir) map[addrs.Provider][]*testInstallerEventLogItem {
|
||||
return map[addrs.Provider][]*testInstallerEventLogItem{
|
||||
noProvider: {
|
||||
{
|
||||
Event: "PendingProviders",
|
||||
Args: map[addrs.Provider]getproviders.VersionConstraints{
|
||||
beepProvider: getproviders.MustParseVersionConstraints(">= 2.0.0"),
|
||||
},
|
||||
},
|
||||
},
|
||||
beepProvider: {
|
||||
{
|
||||
Event: "QueryPackagesBegin",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Constraints string
|
||||
Locked bool
|
||||
}{">= 2.0.0", true},
|
||||
},
|
||||
{
|
||||
Event: "QueryPackagesSuccess",
|
||||
Provider: beepProvider,
|
||||
Args: "2.1.0",
|
||||
},
|
||||
{
|
||||
Event: "LinkFromCacheBegin",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Version string
|
||||
CacheRoot string
|
||||
}{
|
||||
"2.1.0",
|
||||
inst.globalCacheDir.BasePath(),
|
||||
},
|
||||
},
|
||||
{
|
||||
Event: "LinkFromCacheFailure",
|
||||
Provider: beepProvider,
|
||||
Args: struct {
|
||||
Version string
|
||||
Error string
|
||||
}{
|
||||
"2.1.0",
|
||||
fmt.Sprintf(
|
||||
"the provider cache at %s has a copy of example.com/foo/beep 2.1.0 that doesn't match any of the checksums recorded in the dependency lock file",
|
||||
dir.BasePath(),
|
||||
),
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
},
|
||||
},
|
||||
"successful reinstall of one previously-locked provider": {
|
||||
Source: getproviders.NewMockSource(
|
||||
[]getproviders.PackageMeta{
|
||||
@ -1972,8 +2248,8 @@ func TestEnsureProviderVersions(t *testing.T) {
|
||||
if test.WantErr != "" {
|
||||
if instErr == nil {
|
||||
t.Errorf("succeeded; want error\nwant: %s", test.WantErr)
|
||||
} else if got, want := instErr.Error(), test.WantErr; got != want {
|
||||
t.Errorf("wrong error\ngot: %s\nwant: %s", got, want)
|
||||
} else if got, want := instErr.Error(), test.WantErr; !strings.Contains(got, want) {
|
||||
t.Errorf("wrong error\ngot: %s\nwant substring: %s", got, want)
|
||||
}
|
||||
} else if instErr != nil {
|
||||
t.Errorf("unexpected error\ngot: %s", instErr.Error())
|
||||
|
||||
@ -387,6 +387,53 @@ grow to contain several unused versions which you must delete manually.
|
||||
safe. The provider installer's behavior in environments with multiple `terraform
|
||||
init` calls is undefined.
|
||||
|
||||
## Allowing the Provider Plugin Cache to break the dependency lock file
|
||||
|
||||
~> **Note:** The option described in is for unusual and exceptional situations
|
||||
only. Do not set this option unless you are sure you need it and you fully
|
||||
understand the consequences of enabling it.
|
||||
|
||||
By default Terraform will use packages from the global cache directory only
|
||||
if they match at least one of the checksums recorded in the
|
||||
[dependency lock file](https://developer.hashicorp.com/terraform/language/files/dependency-lock)
|
||||
for that provider. This ensures that Terraform can always
|
||||
generate a complete and correct dependency lock file entry the first time you
|
||||
use a new provider in a particular configuration.
|
||||
|
||||
However, we know that in some special situations teams have been unable to use
|
||||
the dependency lock file as intended, and so they don't include it in their
|
||||
version control as recommended and instead let Terraform re-generate it each
|
||||
time it installs providers.
|
||||
|
||||
For those teams that don't preserve the dependency lock file in their version
|
||||
control systems between runs, Terraform allows an additional CLI Configuration
|
||||
setting which tells Terraform to always treat a package in the cache directory
|
||||
as valid even if there isn't already an entry in the dependency lock file
|
||||
to confirm it:
|
||||
|
||||
```hcl
|
||||
plugin_cache_may_break_dependency_lock_file = true
|
||||
```
|
||||
|
||||
Setting this option gives Terraform CLI permission to create an incomplete
|
||||
dependency lock file entry for a provider if that would allow Terraform to
|
||||
use the cache to install that provider. In that situation the dependency lock
|
||||
file will be valid for use on the current system but may not be valid for use on
|
||||
another computer with a different operating system or CPU architecture, because
|
||||
it will include only a checksum of the package in the global cache.
|
||||
|
||||
We recommend that most users leave this option unset, in which case Terraform
|
||||
will always install a provider from upstream the first time you use it with
|
||||
a particular configuration, but can then re-use the cache entry on later runs
|
||||
once the dependency lock file records valid checksums for the provider package.
|
||||
|
||||
~> **Note:** The Terraform team intends to improve the dependency lock file
|
||||
mechanism in future versions so that it will be usable in more situations. At
|
||||
that time this option will become silently ignored. If your workflow relies on
|
||||
the use of this option, please open a GitHub issue to share details about your
|
||||
situation so that we can consider how to support it without breaking the
|
||||
dependency lock file.
|
||||
|
||||
### Development Overrides for Provider Developers
|
||||
|
||||
-> **Note:** Development overrides work only in Terraform v0.14 and later.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user