opentofu/WEEKLY_UPDATES.md

Weekly Updates

2023-11-02

Hey!

This week was still mostly spent on discussing and implementing proof of concepts for some stable registry designs, as well as work on the S3 backend.

• Current Status
  • OpenTofu Day will be happening as a KubeCon + CloudNativeCon Europe CNCF-hosted Co-located Event, learn more here!
  • The technical steering committee is meeting today (2023-11-02) to pick a stable registry design.
    • There are three registry RFCs right now
  • Extensive work is happening around updating the S3 state backend configuration to be in line with the AWS provider configuration. Only two pull requests left here and we'll be done.
  • A fix for the interesting bug report mentioned last week in the update has been finished and merged.
• Up next
  • The technical steering committee will pick a stable registry design later today, and the core engineering team will focus most of its time on that in the upcoming weeks (2023-11-06 and onwards).
• How can I help?
  • Since the alpha release is out, right now the best way to help is to take the alpha for a test drive and see if there are any bugs / issues.
  • Other than that, the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally - this very document was a result of such feedback! We're available on Slack, via GitHub issues, or even in the pull request updating this file.

2023-10-24

Hey!

This week was again spent on discussing and PoC'in stable registry designs, as well as work on the S3 backend.

• Current Status
  • There are two registry RFCs right now, and both will most likely have a PoC by the end of the week.
  • Extensive work is happening around updating the S3 state backend configuration to be in line with the AWS provider configuration.
  • OpenTofu is available on Homebrew Core, so you can just brew install opentofu to use it!
  • A bug report has surfaced an interesting issue when migrating from Terraform to OpenTofu with an existing state file. An initial fix has already been implemented and will be further polished and released soon.
• Up next
  • The steering committee will pick a registry RFC in the week of 30th Oct - 3rd Nov.
    • Intense implementation work will start on the 6th of November.
    • This should let us have a stable registry in place around mid-November - mid-December.
• How can I help?
  • Since the alpha release is out, right now the best way to help is to take the alpha for a test drive and see if there are any bugs / issues.
  • You can also submit and discuss RFCs related to the stable registry.
  • Other than that, the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally - this very document was a result of such feedback! We're available on Slack, via GitHub issues, or even in the pull request updating this file.

2023-10-17

Hello there!

This past week was mostly spent on creating stable registry designs, which is a prerequisite to building a stable provider/module registry and releasing a stable version of OpenTofu.

• Current Status
  • The registry replacement issue has now been updated to include a list of stable registry requirements.
    • There is now time until the 27th October to submit and discuss RFCs.
    • You can find relevant RFCs by using GitHub Issue search.
    • After the 27th October the Technical Steering Committee will pick a single RFC which will then be implemented.
  • A setup-opentofu GitHub Action is now available, it's a great way to take OpenTofu for a spin on GitHub Actions. Thanks a lot to @kislerdm for contributing this!
  • Work on TF 1.6 compatibility has been done.
  • We've closed a lot of issues that were stable release blockers, you can take a look at the current stable milestone status here.
• Up next
  • The main TF 1.6 compatibility issue left, on which work has already been started and will continue for a while is the new S3 state backend configuration schema. You can track this issue here.
  • Until the 27th October deadline, work and discussion on RFCs will happen, as well as the implementation of PoCs for some of these RFCs.
• How can I help?
  • Since the alpha release is out, right now the best way to help is to take the alpha for a test drive and see if there are any bugs / issues.
  • You can also submit and discuss RFCs related to the stable registry.
  • Other than that, the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally - this very document was a result of such feedback! We're available on Slack, via GitHub issues, or even in the pull request updating this file.

2023-10-11

Hey!

Last week we've released the first alpha release (and another one the next day). This has been a huge milestone for the OpenTofu community, and means that you can actually download OpenTofu and play with it. It also marks the beginning of the stable registry design discussions.

Just to give you some metrics about the release:

• The alpha releases of OpenTofu have over 2k downloads.
• Some registry stats
  • Uptime: 100%
  • Max RPS: 105
  • Total Requests: 78869
  • Average latency: 19.15ms
  • Total unique providers used: 194

We'll also have our first office hours today! Make sure you don't miss that.

• Current Status
  • First Alpha Release
    • We've managed to get the alpha release out, including a working registry.
  • Second Alpha Release
    • A bug was reported to OpenTofu and later also to Terraform. We've deemed it high priority and possibly impacting the ability of the community to test OpenTofu with existing codebases, so we've worked on fixing it asap. The root cause was in the hcl library. We've made a fix in OpenTofu, released a new alpha, and afterward the fix got accepted to the hcl library.
• Up next
  • Stable Registry Design (see the tracking issue)
    • Even though there's a registry included with the alpha release, it's not stable. It was mostly a "get it working" design, and we will now be working on a stable design.
    • Right now we're working on a list of requirements, which will most likely be shared tomorrow in the tracking issue above.
    • Then, there will be an approximate 2 weeks to submit and discuss RFCs, after which the steering committee will choose the final design, which we'll begin implementing right away.
    • Please discuss in the issue and incoming RFCs, we want to hear all your voices! Please also feel free to submit your own RFCs.
• How can I help?
  • Since the alpha release is out, right now the best way to help is to take the alpha for a test drive and see if there are any bugs / issues.
  • Other than that, the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally - this very document was a result of such feedback! We're available on Slack, via GitHub issues, or even in the pull request updating this file.

2023-10-02

Hey!

Another week, another update! This last week has been all about the alpha release, which is planned to be released this week, ideally tomorrow (Tuesday).

• Current Status
  • Alpha release is blocked only by the initial registry work
    • There's a PR open for using the OpenTofu registry as the default in OpenTofu.
    • There's been extensive work on caching, discussed in a GitHub issue and a Slack channel.
      • One PR introduced a better version listing cache in DynamoDB, to reuse the cache across provider listing and provider downloading.
      • Another PR introduced http-level caching for all requests to GitHub, which generally avoids making the same request twice in quick succession.
    • There's a bug in the provider mirroring scripts which makes them not properly trigger GitHub Actions. We're working on a fix.
    • There's a bug with module downloading, for which both tags in the form of x.y.z and vx.y.z are supported, while we only support one of them.
• Up next
  • The alpha release is planned for this week, using our alpha registry.
  • As described last week, once the alpha release is out, we'll be kicking off discussions about the stable registry design.
    • One such issue has already been opened and explores the problem space of handling signatures and public keys.
• How can I help?
  • Right now the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu. Additionally, the alpha release is coming out this week. We'd appreciate you taking it for a test drive and letting us know about any issues you encounter.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally! We're available on Slack, via GitHub issues, or even in the pull request creating this very file.

2023-09-26

Hello!

We’ve received some feedback that it’s currently hard to track the status of OpenTofu and what is currently being worked on. We hear you, and we’ll do better. We’re slowly moving discussions from our internal slack to public GitHub issues, to make them more inclusive. In addition, we’ll now be publishing an update like this once a week to give you a breakdown of where we’re at, what we’re currently working on, how you can best help, and what’s up next.

• Current Status
  • Right now we’re primarily working towards the first alpha release. There are two remaining blockers here:
    • Finishing the renaming to OpenTofu. There's only a single PR left here.
    • Preparing the alpha registry replacement to be ready for use. The main current issue here is #97, described in a bit more detail below.
  • The current registry design is a glorified GitHub redirector.
    • This is by no means the final design! It’s meant to be a “get it working” design for the alpha release. Soon (~once the alpha release is out) we’ll start public discussions regarding the “stable” design for the registry, and we’re not ruling out any options here yet.
    • Right now the main issue is caching. Since the registry is a GitHub redirector, it is affected by GitHub rate limits. The rate limit is 5k requests per hour. We already do extensive caching here, but we can still do better and want to do better prior to the alpha release. This issue contains more details about the work being done here.
    • Also, even though OpenTofu alpha will skip signature validation when the key is not available, we’re collecting provider signing keys for the registry to serve. So, if you’re a provider author, please make sure to create a Pull Request to the registry repo and add your public key. Here's the latest example of such a PR.
    • We’ve also made a mirror of all official HashiCorp providers so that they’re hosted on the OpenTofu GitHub, like any other 3rd-party provider. This included getting all historic versions to build (which was quite a challenge!) but we’re done with it and the end result is that out of 2260 provider versions we only have 16 failures, which are generally old and broken versions, so you should be able to easily work around this.
      • The reason for this is that the registry being a GitHub redirector needs all artifacts on GitHub, while HashiCorp hosts these providers directly on their registry and doesn't include them in GitHub releases.
      • Long-term we’re planning to host these most-used providers on Fastly.
• Up next
  • Early next week we’re planning to make available an alpha release of OpenTofu, with a fully drop-in working registry for providers and modules.
  • With the above, we’ll also kick off the discussion around the stable registry.
    • We’ll publish an initial list of requirements for the registry in the original registry issue. The list itself will be open to changes based on further discussion.
    • Based on those requirements, we will be creating RFC’s for possible solutions. If you’d like to propose a solution, feel free to post an RFC too!
    • After having discussions there and possibly doing some PoC’s, the technical steering committee will make the final call which approach to pursue.
• How can I help?
  • Right now the best way to help is to create issues, discuss on issues, and spread the word about OpenTofu.
  • There are some occasional minor issues which are accepted and open to external contribution, esp. ones outside the release-blocking path. We’re also happy to accept any minor refactors or linter fixes. Please see the contributing guide for more details.
  • We have multiple engineers available full-time in the core team, so we’re generally trying to own any issues that are release blockers - this way we can make sure we get to the release as soon as possible.
  • The amount of pending-decision-labeled issues on the repository might be a bit off-putting. The reason for that is that right now we’re prioritizing the alpha and stable release. Only after we have a stable release in place do we aim to start actually accepting enhancement proposals and getting them implemented/merged. Still, we encourage you to open those issues and discuss them!
    • Issues and Pull Requests with enhancements or major changes will generally be frozen until we have the first stable release out. We will introduce a milestone to mark them as such more clearly.
  • Once we make the alpha release available next week, the best way to help will be by test-driving that release and creating GitHub issues for any problems you find.

Please let us know if you have any feedback on what we could improve, either with these updates or more generally! We're available on Slack, via GitHub issues, or even in the pull request creating this very file.