opentofu/backend
Rémi Lapeyre e680211bc0 Split the state in chunks when they outgrow the limit of Consul KV store
The Consul KV store limits the size of the values in the KV store to 524288
bytes. Once the state reaches this limit Consul will refuse to save it. It is
currently possible to try to bypass this limitation by enable Gzip but the issue
will manifest itself later. This is particularly inconvenient as it is possible
for the state to reach this limit without changing the Terraform configuration
as datasources or computed attributes can suddenly return more data than they
used to. Several users already had issues with this.

To fix the problem once and for all we now split the payload in chunks of 524288
bytes when they are to large and store them separatly in the KV store. A small
JSON payload that references all the chunks so we can retrieve them later and
concatenate them to reconstruct the payload.

While this has the caveat of requiring multiple calls to Consul that cannot be
done as a single transaction as those have the same size limit, we use unique
paths for the chunks and CAS when setting the last payload so possible issues
during calls to Put() should not result in unreadable states.

Closes https://github.com/hashicorp/terraform/issues/19182
2020-08-14 17:38:18 +02:00
..
atlas state: remove deprecated state package (#25490) 2020-08-11 11:43:01 -04:00
init Add kubernetes backend 2020-06-04 02:29:57 -04:00
local Merge pull request #25779 from hashicorp/jbardin/remove-state-attrs 2020-08-12 10:49:44 -04:00
remote state: remove deprecated state package (#25490) 2020-08-11 11:43:01 -04:00
remote-state Split the state in chunks when they outgrow the limit of Consul KV store 2020-08-14 17:38:18 +02:00
backend.go state: remove deprecated state package (#25490) 2020-08-11 11:43:01 -04:00
cli.go backend: Update interface and implementations for new config loader 2018-10-16 18:39:12 -07:00
nil_test.go backend: introduce the backend set of interfaces 2017-01-26 14:33:49 -08:00
nil.go Change backend.ValidateConfig to PrepareConfig 2019-02-25 18:37:20 -05:00
operation_type.go Version tools per Go convention under tools.go 2019-10-17 22:23:39 +02:00
operationtype_string.go stringer: Regenerate files with latest version 2019-05-13 15:34:27 +01:00
testing.go state: remove deprecated state package (#25490) 2020-08-11 11:43:01 -04:00
unparsed_value_test.go backend/local: Handle interactive prompts for variables in UI layer 2019-10-10 10:07:01 -07:00
unparsed_value.go Avoid leaking sensitive values in undeclared warnings 2019-11-05 17:32:57 -08:00