rfc/oci-registries: Implementation-related content from earlier drafts

Signed-off-by: Martin Atkins <mart@degeneration.co.uk>

rfc/20241206-oci-registries.md

@@ -32,4 +32,12 @@ In order to make this RFC easier to read, we have split it into several parts:
 5. [Providers in OCI](20241206-oci-registries/5-providers.md)
 6. [Modules in OCI](20241206-oci-registries/6-modules.md)
 7. [Authentication](20241206-oci-registries/7-authentication.md)
-8. [Open questions](20241206-oci-registries/8-open-questions.md)
+8. [Open questions](20241206-oci-registries/8-open-questions.md)
+
+### Appendices
+
+The following additional sections discuss potential implementation details of the features described in the earlier chapters. These are primarily for core team reference purposes, but potentially interesting to others too.
+
+9. [Authentication-related implementation details](20241206-oci-registries/9-auth-implementation-details.md)
+10. [Provider installation implementation details](20241206-oci-registries/10-provider-implementation-details.md)
+11. [Module installation implementation details](20241206-oci-registries/11-module-implementation-details.md)

rfc/20241206-oci-registries/10-provider-implementation-details.md

@@ -0,0 +1,34 @@
+# Provider installation implementation details
+
+---
+
+This document is part of the [OCI registries RFC](../20241206-oci-registries.md).
+
+| [« Previous](9-auth-implementation-details.md) | [Up](../20241206-oci-registries.md) | [Next »](10-provider-implementation-details.md) |
+
+---
+
+This appendix discusses implementation details related to [installing provider packages from OCI registries](5-providers.md).
+
+> [!WARNING]
+> This appendix is still under construction, subject to change based on feedback on the earlier chapters, and may not yet be up-to-date with the latest changes in the earlier chapters.
+
+## OCI Mirror Provider Installation Source
+
+> **TODO:** Describe something like what we did for OCI Mirror sources in [the earlier prototype](https://github.com/opentofu/opentofu/pull/2170), with adjustments for the latest modifications to the proposed artifact layout.
+>
+> For now we can ignore the parts of that prototype related to treating an OCI registry as if it is an OpenTofu registry directly, since we're not planning to implement that in the first round.
+
+## Package checksums and signing
+
+OpenTofu tracks in its [dependency lock file](https://opentofu.org/docs/language/files/dependency-lock/) the single selected version of each previously-installed provider, and a set of checksums that are considered acceptable for that version of that provider.
+
+Additionally, whenever possible OpenTofu retrieves a GPG signature and associated public key for a provider (with both decided by its origin registry) and uses that signature to verify that the fetched provider package matches one of the checksums that was covered by the GPG signature. If that verification succeeds, OpenTofu assumes that all other checksums covered by the same signature are also trusted, and so records _all_ of them in the dependency lock file for future use. This mechanism is important to ensure that the dependency lock file ends up recording checksums for _all_ platform-specific packages offered for that provider version, even though OpenTofu only downloads and verifies the package for the current platform.
+
+OpenTofu's existing provider registry protocol always uses `.zip` archives as provider packages and requires the provider developer's signature to cover a document containing SHA256 checksums of all of the `.zip` archives for a particular version. These translate into `zh:`-schemed checksums in the dependency lock file, but since those checksums can only be verified against a not-yet-unpacked `.zip` archive OpenTofu also generates a more general `h1:` checksum based on the _contents_ of the package it downloaded, which it can then use to verify already-unpacked provider packages in the local filesystem.
+
+[Our OCI artifact layout for provider packages](5-providers.md#storage-in-oci) intentionally uses `archive/zip` layers so that we can use byte-for-byte identical copies of a provider developer's signed `.zip` packages as the layer blobs, and therefore we can directly translate the `sha256:` digest of each layer into a `zh:`-style checksum for dependency locking purposes, without having to download the package and recalculate the checksum locally.
+
+We are not intending to support OCI artifact signing in our first implementation, since we are focusing initially only on the "OCI mirror" use-case. Without any signatures, we'll capture in the dependency lock file only the checksum of the specific artifact we downloaded, for consistency with the guarantees we make from installing from unsigned sources in today's OpenTofu. When we later add support for optionally signing the index manifest, we can begin using those signatures to justify including the `zh:` checksums from _all_ of the per-platform artifacts, announcing the ID of the signing key as part of the `tofu init` output just as we do for registry-distributed signatures today. It remains the user's responsibility to verify that the key ID is one they expected before committing the new checksums in the dependency lock file.
+
+As with OpenTofu's current provider registry protocol, an OCI provider artifact cannot provide us any trustworthy representation of the `h1:` checksum of a provider package, and so OpenTofu will calculate that locally based on the already-downloaded package(s), assuming that they also match one of the previously-discovered `zh:` checksums, as usual.

rfc/20241206-oci-registries/11-module-implementation-details.md

@@ -0,0 +1,16 @@
+# Module installation implementation details
+
+---
+
+This document is part of the [OCI registries RFC](../20241206-oci-registries.md).
+
+| [« Previous](10-provider-implementation-details.md) | [Up](../20241206-oci-registries.md) |
+
+---
+
+This appendix discusses implementation details related to [installing module packages from OCI registries](6-modules.md).
+
+> [!WARNING]
+> This section is still under construction, subject to change based on feedback on the earlier chapters, and may not yet be up-to-date with the latest changes in the earlier chapters.
+
+> **TODO:** Write this!

rfc/20241206-oci-registries/8-open-questions.md

@@ -26,6 +26,29 @@ Users working on multiple projects may want to use different credentials for the
 
 Currently, this RFC [proscribes registry semantics](6-modules.md) for module versions. Do we want to use this or switch to Git-like semantics?
 
+## Provider Source Addresses with Unsupported Unicode Characters
+
+OpenTofu's provider source address syntax allows a wide variety of Unicode characters in all three components, following the [RFC 3491 "Nameprep"](https://datatracker.ietf.org/doc/rfc3491/) rules.
+
+However, the OCI Distribution specification has a considerably more restrictive allowed character set for repository names: it supports only ASCII letters and digits along with a small set of punctuation.
+
+Because of this, there some valid OpenTofu provider source addresses that cannot be translated mechanically to valid OCI Distribution repository addresses via template substitution alone. A provider source address that, for example, has a Japanese alphabet character in its "type" portion would be projected into a syntactically-invalid OCI repository address.
+
+Our initial prototype assumed that in practice non-ASCII characters in these addresses are very rare, and so just returns an error message whenever this situation arises:
+
+```
+requested provider address example.com/foo/ほげ contains characters that
+are not valid in an OCI distribution repository name, so this provider
+cannot be installed from an OCI repository as
+ghcr.io/examplecom-otf-providers/foo-ほげ
+```
+
+Of course, we cannot see into every organization to know whether they have in-house providers that are named with non-ASCII characters, and the fact that the OpenTofu project works primarily in English means that we are less likely to hear from those whose typical working language is not English.
+
+If we learn in future that supporting non-ASCII characters in provider source addresses installed from OCI registries is important, we could potentially force a specific scheme for automatically transforming those names into ones that are compatible with the OCI repository name requirements, such as applying a "[Punycode](https://en.wikipedia.org/wiki/Punycode)-like" encoding to them before rendering them into the template.
+
+However, Punycode in particular is not generally human-readable and so translation strategies like this often require some UI support to automatically transcode the data back into human-readable form for display. Any OpenTofu-specific mapping strategy we might invent is unlikely to be handled automatically by the UI associated with any general-purpose OCI registry implementation.
+
 ---
 
 | [« Previous](7-authentication.md) | [Up](../20241206-oci-registries.md) |

rfc/20241206-oci-registries/9-auth-implementation-details.md

@@ -0,0 +1,58 @@
+# Authentication-related Implementation Details
+
+---
+
+This document is part of the [OCI registries RFC](../20241206-oci-registries.md).
+
+| [Up](../20241206-oci-registries.md) | [Next »](10-provider-implementation-details.md) |
+
+---
+
+This appendix discusses implementation details related to [OCI Registry Authentication](7-authentication.md).
+
+> [!WARNING]
+> This appendix is still under construction, subject to change based on feedback on the earlier chapters, and may not yet be up-to-date with the latest changes in the earlier chapters.
+
+## Registry authentication is a cross-cutting concern
+
+While most of what we've discussed in this RFC is defined separately for provider and module package installation, the _authentication_ implementation should be shared between both and designed so that it could potentially be used for any other OCI registry interactions we might implement in future, such as hypothetical support for storing OpenTofu state snapshots as OCI artifacts.
+
+Therefore our primary concern for implementation is in centrally-managing the credentials settings and then making them available to the relevant components of both the provider and module package installers.
+
+## OpenTofu CLI Configuration and `package main`
+
+Although there is considerable existing legacy code not following this pattern, the current intended design for OpenTofu is to follow the [dependency inversion principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle) with `package main` acting as the ultimate arbiter of how different subsystems are configured to work together. The main package in turn uses `package cliconfig` (`internal/command/cliconfig`) to decide most of the locally-user-configurable settings that can affect those dependency resolution decisions.
+
+We will continue that design approach by teaching `package cliconfig` to decode and validate the `oci` block in the CLI configuration, returning the discovered information as part of the overall `cliconfig.Config` object generated by that package.
+
+The implicit configuration mode acts as an alternative way to populate the same settings from Docker CLI or other container system configuration files, and so would also be implemented in `package cliconfig` by mapping the concepts from the Docker CLI configuration language to the same internal data types that we would decode our explicit configuration into, so that the rest of the system does not need to be concerned about how that information was discovered.
+
+`package main` is responsible for using the information returned from the CLI configuration to configure and instantiate the [`ociclient.OCIClient`](https://pkg.go.dev/github.com/opentofu/libregistry@v0.0.0-20241121135917-6f06a9a60bb5/registryprotocols/ociclient#OCIClient) that will then be passed as a dependency into both the provider installer and the module installer, which will then encapsulate all of the OCI registry interactions including the collection and inclusion of credentials when making requests.
+
+The fine details of how we will find and decode Docker CLI-style and Podman CLI-style configuration files are essentially to follow the rules implemented in those codebases as closely as possible, and those are defined as code rather than as specification so are hard to capture as prose here. We have [an experimental initial implementation](https://github.com/opentofu/opentofu/blob/f4c82859864d2ee3397e2f26875cfa73c796c28b/internal/command/cliconfig/oci_registry.go#L137) that illustrates the overall shape of the problem.
+
+> **TODO:** Translate that prototype code into a specification for the subset of the format we intend to support, and the discovery rules we intend to follow to find files in that format.
+
+## OCI Client for the Provider Installer
+
+The provider installation components already follow the dependency inversion principle, with `package main` constructing various implementations of [`getproviders.Source`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/getproviders#Source) based on the `provider_installation` block in the CLI configuration, or the implied fallbacks thereof.
+
+The full details of how OCI registries will be realized as new provider source types is discussed in [Provider implementation details](10-provider-implementation-details.md). For authentication, any new implementatinos of `getproviders.Source` that interact with OCI registries must take a preconfigured `ociclient.OCIClient` as a dependency during instantiation.
+
+The existing logic for instantiating the provider installation methods in `package main` can then be extended to pass the shared OCI registry client to those sources when instantiating them.
+
+## OCI Client for the Module Package Installer
+
+The module installation mechanisms in OpenTofu are considerably older and have not yet been adapted to follow the dependency inversion principle. Therefore some refactoring of that subsystem will be required to implement this proposal. We will take inspiration from the design of the provider installation process to improve the consistency between these two subsystems.
+
+Currently `package getmodules` (`internal/getmodules`) contains some statically-initialized data structures that act as configuration for the third-party library [`go-getter`](https://pkg.go.dev/github.com/hashicorp/go-getter), which OpenTofu relies on for all module package retrieval. Those static data structures are exposed to external callers only indirectly through [`getmodules.PackageFetcher`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/getmodules#PackageFetcher), whose instantiation function currently takes no arguments because all of its dependencies are statically configured inside the package.
+
+`getmodules.PackageFetcher` instances are currently instantiated inline within some of the functions of [`package initwd`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/initwd), as an implementation detail. `package initwd` has seen _some_ efforts to adopt a dependency-inversion-style approach, with `initwd.ModuleInstaller` taking the modules directory, config loader, and module registry protocol client as arguments rather than instantiating them directly itself.
+
+To continue that evolution, we will extend `initwd.NewModuleInstaller` to also take a `getmodules.PackageFetcher` as an argument rather than instantiating it directly inline. We will then extend `package command`'s `Meta` type to include a field for a provided `getmodules.PackageFetcher`, alongside [the existing field for a `getproviders.Source`](https://github.com/opentofu/opentofu/blob/ffa43acfcdc4431f139967198faa2dd20a2752ea/internal/command/meta.go#L127-L130).
+
+[`command.Meta.installModules` currently calls `initwd.NewModuleInstaller` directly](https://github.com/opentofu/opentofu/blob/ffa43acfcdc4431f139967198faa2dd20a2752ea/internal/command/meta_config.go#L294), and so we will extend that call to pass in the provided `getmodules.PackageFetcher` alongside the module registry client and other dependency objects.
+
+[`package main` directly instantiates `command.Meta`](https://github.com/opentofu/opentofu/blob/ffa43acfcdc4431f139967198faa2dd20a2752ea/cmd/tofu/commands.go#L89-L115) as its primary way of injecting dependencies into the CLI command layer, including the population of the `ProviderSource` field described above. We will therefore also pass the centrally-instantiated `getmodules.PackageFetcher` in the same way, completing the chain of dependency passing all the way from `package main` to the module installer.
+
+The support for OCI registries as a module installation source will involve the addition of a new implementation of `go-getter`'s `Getter` interface, which will include the preconfigured `ociclient.OCIClient` as one of its fields. For more information, refer to [Module implementation details](11-module-implementation-details.md).