opentofu/internal/command/jsonformat/README.md

jsonformat

This package contains functionality around formatting and displaying the JSON structured output produced by adding the -json flag to various Terraform commands.

Terraform Structured Plan Renderer

As of January 2023, this package contains only a single structure: the Renderer.

The renderer accepts the JSON structured output produced by the terraform show <plan-file> -json command and writes it in a human-readable format.

Implementation details and decisions for the Renderer are discussed in the following sections.

Implementation

There are two subpackages within the jsonformat renderer package. The differ package compares the before and after values of the given plan and produces Diff objects from the computed package.

This approach is aimed at ensuring the process by which the plan difference is calculated is separated from the rendering itself. In this way it should be possible to modify the rendering or add new renderer formats without being concerned with the complex diff calculations.

The differ package

The differ package operates on Change objects. These are produced from jsonplan.Change objects (which are produced by the terraform show command). Each jsonplan.Change object represents a single resource within the overall Terraform configuration.

The differ package will iterate through the Change objects and produce a single Diff that represents a processed summary of the changes described by the Change. You will see that the produced changes are nested so a change to a list attribute will contain a slice of changes, this is discussed in the "The computed package" section.

The Change object

The Change objects contain raw Golang representations of JSON objects (generic interface{} fields). These are produced by parsing the json.RawMessage objects within the provided changes.

The fields the differ cares about from the provided changes are:

• Before: The value before the proposed change.
• After: The value after the proposed change.
• Unknown: If the value is being computed during the change.
• BeforeSensitive: If the value was sensitive before the change.
• AfterSensitive: If the value is sensitive after the change.
• ReplacePaths: If the change is causing the overall resource to be replaced.

In addition, the changes define two additional meta fields that they set and manipulate internally:

• BeforeExplicit: If the value in Before is explicit or an implied result due to a change elsewhere.
• AfterExplicit: If the value in After is explicit or an implied result due to a change elsewhere.

The actual concrete type of each of the generic fields is determined by the overall schema. The changes are also recursive, this means as we iterate through the Change we create relevant child values based on the schema for the given resource.

For example, the initial change is always a block type which means the Before and After values will actually be map[string]interface{} types mapping each attribute and block to their relevant values. The Unknown, BeforeSensitive, AfterSensitive values will all be either a map[string]interface{} which maps each attribute or nested block to their unknown and sensitive status, or it could simply be a boolean which generally means the entire block and all children are sensitive or computed.

In total, a Change can represent the following types:

• Attribute
  • map: Values will typically be map[string]interface{}.
  • list: Values will typically be []interface{}.
  • set: Values will typically be []interface{}.
  • object: Values will typically be map[string]interface{}.
  • tuple: Values will typically be []interface{}.
  • bool: Values will typically be a bool.
  • number: Values will typically be a float64.
  • string: Values will typically be a string.
• Block: Values will typically be map[string]interface{}, but they can be split between nested blocks and attributes.
• Output
  • Outputs are interesting as we don't have a schema for them, as such they can be any JSON type.
  • We also use the Output type to represent dynamic attributes, since in both cases we work out the type based on the JSON representation instead of the schema.

The ReplacePaths field is unique in that it's value doesn't actually change based on the schema - it's always a slice of index slices. An index in this context will either be an integer pointing to a child of a set or a list or a string pointing to the child of a map, object or block. As we iterate through the value we manipulate the outer slice to remove child slices where the index doesn't match and propagate paths that do match onto the children.

Quick note on explicit vs implicit: In practice, it is only possible to get implicit changes when you manipulate a collection. That is to say child values of a modified collection will insert nil entries into the relevant before or after fields of their child changes to represent their values being deleted or created. It is also possible for users to explicitly put null values into their collections, and this behaviour is different to deleting an item in the collection. With the BeforeExplicit and AfterExplicit values we can tell the difference between whether this value was removed from a collection or this value was set to null in a collection.

Quick note on the go-cty Value and Type objects: The Before and After fields are actually go-cty values, but we cannot convert them directly because of the Terraform Cloud redacted endpoint. The redacted endpoint turns sensitive values into strings regardless of their types. Because of this, we cannot just do a direct conversion using the ctyjson package. We would have to iterate through the schema first, find the sensitive values and their mapped types, update the types inside the schema to strings, and then go back and do the overall conversion. This isn't including any of the more complicated parts around what happens if something was sensitive before and isn't sensitive after or vice versa. This would mean the type would need to change between the before and after value. It is in fact just easier to iterate through the values as generic JSON interfaces, and obfuscate the sensitive values as we never need to print them anyway.

Iterating through changes

The differ package will recursively create child Change objects for the complex objects.

There are two key subtypes of a Change: SliceChange and MapChange. SliceChange values are used by list, set, and tuple attributes. MapChange values are used by map and object attributes, and blocks. For what it is worth outputs and dynamic types can end up using both, but they're kind of special as the processing for dynamic types works out the type from the JSON struct and then just passes it into the relevant real types for actual processing.

The two subtypes implement GetChild functions that retrieve a child change for a relevant index (int for slice, string for map). These functions build an entirely populated Change object, and the package will then recursively compute the change for the child (and all other children). When a complex change has all the children changes, it then passes that into the relevant complex diff type.

The computed package

A computed Diff should contain all the relevant information it needs to render itself.

The Diff itself contains the action (eg. Create, Delete, Update), and whether this change is causing the overall resource to be replaced (read from the ReplacePaths field discussed in the previous section). The actual content of the diffs is passed directly into the internal renderer field. The internal renderer is then an implementation that knows the actual content of the changes and what they represent.

For example to instantiate a diff resulting from updating a list of primitives:

    listDiff := computed.NewDiff(renderers.List([]computed.Diff{
        computed.NewDiff(renderers.Primitive(0.0, 0.0, cty.Number), plans.NoOp, false),
        computed.NewDiff(renderers.Primitive(1.0, nil, cty.Number), plans.Delete, false),
        computed.NewDiff(renderers.Primitive(nil, 4.0, cty.Number), plans.Create, false),
        computed.NewDiff(renderers.Primitive(2.0, 2.0, cty.Number), plans.NoOp, false)
    }, plans.Update, false))

The RenderHuman function

Currently, there is only one way to render a change, and it is implemented via the RenderHuman function. In the future, there may be additional rendering capabilities, but for now the RenderHuman function just passes the call directly onto the internal renderer.

Rendering the above diff with: listDiff.RenderHuman(0, RenderOpts{}) would produce:

[
    0,
  - 1 -> null,
  + 4, 
    2,
]    

Note, the render function itself doesn't print out metadata about its own change (eg. there's no ~ symbol in front of the opening bracket). The expectation is that parent changes control how child changes are rendered, so are responsible for deciding on their opening indentation, whether they have a key (as in maps, objects, and blocks), or how the action symbol is displayed.

In the above example, the primitive renderer would print out only 1 -> null while the surrounding list renderer is providing the indentation, the symbol and the line ending commas.

Implementing new diff types

To implement a new diff type, you must implement the internal Renderer functionality. To do this you create a new implementation of the computed.DiffRenderer, make sure it accepts all the data you need, and implement the RenderHuman function (and any other additional render functions that may exist).

Some changes publish warnings that should be displayed alongside them. If your new change has no warnings you can use the NoWarningsRenderer to avoid implementing the additional Warnings function.

If/when new Renderer types are implemented, additional Render like functions will be added. You should implement all of these with your new change type.

Implementing new renderer types for changes

As of January 2023, there is only a single type of renderer (the human-readable) renderer. As such, the Diff structure provides a single RenderHuman function.

To implement a new renderer:

1. Add a new render function onto the internal DiffRenderer interface.
2. Add a new render function onto the Diff struct that passes the call onto the internal renderer.
3. Implement the new function on all the existing internal interfaces.

Since each internal renderer contains all the information it needs to provide change information about itself, your new Render function should pass in anything it needs.

New types of Renderer

In the future, we may wish to add in different kinds of renderer, such as a compact renderer, or an interactive renderer. To do this, you'll need to modify the Renderer struct or create a new type of Renderer.

The logic around creating the Diff structures will be shared (ie. calling into the differ package should be consistent across renderers). But when it comes to rendering the changes, I'd expect the Diff structures to implement additional functions that allow them to internally organise the data as required and return a relevant object. For the existing human-readable renderer that is simply a string, but for a future interactive renderer it might be a model from an MVC pattern.