Encryption: Enable envelope encryption by default (#49301)

* Encryption: Enable envelope encryption by default

* Stop relying on feature toggles from settings (deprecated)

* Database encryption docs (envelope encryption)

* Remove deprecated (and no longer used) FT

* Apply suggestions from code review

Co-authored-by: Tania <yalyna.ts@gmail.com>

docs/sources/administration/_index.md

@@ -13,4 +13,5 @@ This section includes information for Grafana administrators, team administrator
 - [Configuration]({{< relref "configuration" >}})
 - [Configure Docker image]({{< relref "configure-docker" >}})
 - [Security]({{< relref "security" >}})
+- [Database encryption]({{< relref "database-encryption" >}})
 - [Service accounts]({{< relref "service-accounts" >}})

docs/sources/administration/database-encryption.md

@@ -12,21 +12,91 @@ Grafana’s database contains secrets, which are used to query data sources, sen
 
 Grafana encrypts these secrets before they are written to the database, by using a symmetric-key encryption algorithm called Advanced Encryption Standard (AES), and using a [secret key]({{< relref "../administration/configuration/#secret_key" >}}) that you can change when you configure a new Grafana instance.
 
-You can choose to use [envelope encryption](#envelope-encryption), which adds a layer of indirection to the encryption process.
+Since Grafana v9.0, it uses [envelope encryption](#envelope-encryption) by default, which adds a layer of indirection to the
+encryption process that represents an [**implicit breaking change**](#implicit-breaking-change) for older versions of Grafana.
+
+For further details about how to operate a Grafana instance with envelope encryption, see the [Operational work](#operational work) section below.
 
 > **Note:** In Grafana Enterprise, you can also choose to [encrypt secrets in AES-GCM mode]({{< relref "../enterprise/enterprise-encryption/#changing-your-encryption-mode-to-aes-gcm" >}}) instead of AES-CFB.
 
 # Envelope encryption
 
-In Grafana, you can choose to use envelope encryption. Instead of
-encrypting all secrets with a single key, Grafana uses a set of keys
-called data encryption keys (DEKs) to encrypt them. These data
-encryption keys are themselves encrypted with a single key encryption
-key (KEK).
+> **Note:** Since Grafana v9.0, you can turn it off by adding the term `disableEnvelopeEncryption` to the list of
+> feature toggles in your [Grafana configuration]({{< relref "../administration/configuration/#feature_toggles" >}}).
 
-To turn on envelope encryption, add the term `envelopeEncryption` to the list of feature toggles in your [Grafana configuration]({{< relref "../administration/configuration/#feature_toggles" >}}).
+Instead of encrypting all secrets with a single key, Grafana uses a set of keys called data encryption keys (DEKs) to
+encrypt them. These data encryption keys are themselves encrypted with a single key encryption key (KEK), configured
+through the `secret_key` attribute in your
+[Grafana configuration]({{< relref "../administration/configuration/#secret_key" >}}) or with a
+[KMS integration](#kms-integration).
 
-> **Note:** Avoid turning off envelope encryption once you have turned it on, and back up your database before turning it on for the first time. If you turn envelope encryption on, create new secrets or update your existing secrets (for example, by creating a new data source or alert notification channel), and then turn envelope encryption off, then those data sources, alert notification channels, and other resources using envelope encryption will stop working and you will experience errors. This is because the secrets encrypted with envelope encryption cannot be decrypted or used by Grafana when envelope encryption is turned off.
+## Implicit breaking change
+
+As stated above, envelope encryption represents an implicit breaking change because it changes the way secrets stored
+in the Grafana database are encrypted. That means Grafana administrators will be able to transition to Grafana v9.0
+with no action required from the database encryption perspective, but will need to be extremely careful if they need
+to roll back to a previous version (e.g. Grafana v8.5) after being updated, because secrets created or modified after upgrade
+the update to Grafana v9.0 won't be decryptable in previous versions.
+
+Fortunately though, envelope encryption was added in Grafana v8.3 behind a feature toggle. So, in case of emergency,
+Grafana administrators will be able to downgrade up to Grafana v8.3 and enable envelope encryption as a workaround.
+
+> **Note:** In Grafana releases between v8.3 and v8.5, you can turn envelope encryption on by adding the term
+> `envelopeEncryption` to the list of feature toggles in your
+> [Grafana configuration]({{< relref "../administration/configuration/#feature_toggles" >}}).
+
+# Operational work
+
+From the database encryption perspective, there are several operations that Grafana administrator may want to perform:
+
+- [**Re-encrypt secrets**](#re-encrypt-secrets): re-encrypt secrets with envelope encryption and a fresh data key.
+- [**Roll back secrets**](#roll-back-secrets): decrypt secrets encrypted with envelope encryption and re-encrypt them with legacy encryption.
+- [**Re-encrypt data keys**](#re-encrypt-data-keys): re-encrypt data keys with a fresh key encryption key and a [KMS integration](#kms-integration).
+- [**Rotate data keys**](#rotate-data-keys): disable active data keys and stop using them for encryption in favor of a fresh one.
+
+Find more details about each of those below.
+
+## Re-encrypt secrets
+
+Secrets re-encryption can be performed when a Grafana administrator wants to either:
+
+- Move forward already existing secrets' encryption from legacy to envelope encryption.
+- Re-encrypt secrets after a [data keys rotation](#rotate-data-keys).
+
+> **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration re-encrypt`
+> command. It's safe to run more than once. Recommended to run under maintenance mode.
+
+## Roll back secrets
+
+Used to roll back secrets encrypted with envelope encryption to legacy encryption. It can be used to downgrade to
+a Grafana version earlier than Grafana v9.0 after an unsuccessful upgrade.
+
+> **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration rollback`
+> command. It's safe to run more than once. Recommended to run under maintenance mode.
+
+## Re-encrypt data keys
+
+Used to re-encrypt data keys encrypted with a specific key encryption key (KEK). It can be used to either re-encrypt
+existing data keys with a new key encryption key version (see [KMS integration](#kms-integration) rotation) or to
+re-encrypt them with a completely different key encryption key.
+
+> **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration re-encrypt-data-keys`
+> command. It's safe to run more than once. Recommended to run under maintenance mode.
+
+## Rotate data keys
+
+Data keys rotation can be performed to disable the active data key and therefore stop using them for encryption operations.
+For high-availability setups, you may need to wait until the data keys cache's TTL (time-to-live) expires to ensure that all
+rotated data keys are no longer being used for encryption operations.
+
+New data keys for encryption operations are generated on-demand.
+
+> **Note:** It does not imply secrets re-encryption. Therefore, rotated data keys will continue being used to decrypt
+> those secrets still encrypted with it. Look at [secrets re-encryption](#re-encrypt-secrets) to completely stop using
+> rotated data keys for both encryption and decryption.
+
+> **Note:** This operation is available through Grafana [Admin API]({{< relref "../developers/configuration/admin/#rotate-data-encryption-keys" >}}).
+> It's safe to run more than once.
 
 # KMS integration
 

packages/grafana-data/src/types/featureToggles.gen.ts

@@ -17,7 +17,7 @@ export interface FeatureToggles {
   [name: string]: boolean | undefined; // support any string value
 
   trimDefaults?: boolean;
-  envelopeEncryption?: boolean;
+  disableEnvelopeEncryption?: boolean;
   httpclientprovider_azure_auth?: boolean;
   serviceAccounts?: boolean;
   database_metrics?: boolean;

pkg/cmd/grafana-cli/commands/secretsmigrations/reencrypt_deks.go

@@ -9,7 +9,7 @@ import (
 )
 
 func ReEncryptDEKS(_ utils.CommandLine, runner runner.Runner) error {
-	if !runner.Features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	if runner.Features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		logger.Warn("Envelope encryption is not enabled, quitting...")
 		return nil
 	}

pkg/cmd/grafana-cli/commands/secretsmigrations/reencrypt_secrets.go

@@ -267,7 +267,7 @@ func (s alertingSecret) reencrypt(ctx context.Context, secretsSrv *manager.Secre
 }
 
 func ReEncryptSecrets(_ utils.CommandLine, runner runner.Runner) error {
-	if !runner.Features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	if runner.Features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		logger.Warn("Envelope encryption is not enabled, quitting...")
 		return nil
 	}

pkg/cmd/grafana-cli/commands/secretsmigrations/rollback_secrets.go

@@ -291,7 +291,7 @@ func (s alertingSecret) rollback(
 }
 
 func RollBackSecrets(_ utils.CommandLine, runner runner.Runner) error {
-	if !runner.Features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	if runner.Features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		logger.Warn("Envelope encryption is not enabled, quitting...")
 		return nil
 	}

pkg/services/featuremgmt/registry.go

@@ -14,9 +14,9 @@ var (
 			State:       FeatureStateBeta,
 		},
 		{
-			Name:        "envelopeEncryption",
-			Description: "encrypt secrets",
-			State:       FeatureStateBeta,
+			Name:        "disableEnvelopeEncryption",
+			Description: "Disable envelope encryption (emergency only)",
+			State:       FeatureStateStable,
 		},
 		{
 			Name:        "httpclientprovider_azure_auth",

pkg/services/featuremgmt/toggles_gen.go

@@ -11,9 +11,9 @@ const (
 	// Use cue schema to remove values that will be applied automatically
 	FlagTrimDefaults = "trimDefaults"
 
-	// FlagEnvelopeEncryption
-	// encrypt secrets
-	FlagEnvelopeEncryption = "envelopeEncryption"
+	// FlagDisableEnvelopeEncryption
+	// Disable envelope encryption (emergency only)
+	FlagDisableEnvelopeEncryption = "disableEnvelopeEncryption"
 
 	// FlagHttpclientproviderAzureAuth
 	// Experimental. Allow datasources to configure Azure authentication directly via JsonData

pkg/services/kmsproviders/osskmsproviders/osskmsproviders.go

@@ -24,7 +24,7 @@ func ProvideService(enc encryption.Internal, settings setting.Provider, features
 }
 
 func (s Service) Provide() (map[secrets.ProviderID]secrets.Provider, error) {
-	if !s.features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	if s.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		return nil, nil
 	}
 

pkg/services/secrets/manager/helpers.go

@@ -9,7 +9,6 @@ import (
 	"github.com/grafana/grafana/pkg/services/kmsproviders/osskmsproviders"
 	"github.com/grafana/grafana/pkg/services/secrets"
 	"github.com/grafana/grafana/pkg/setting"
-	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"gopkg.in/ini.v1"
 )
@@ -29,14 +28,11 @@ func SetupTestService(tb testing.TB, store secrets.Store) *SecretsService {
 		data_keys_cache_cleanup_interval = 1ns`))
 	require.NoError(tb, err)
 
-	features := featuremgmt.WithFeatures(featuremgmt.FlagEnvelopeEncryption)
+	features := featuremgmt.WithFeatures()
 
 	cfg := &setting.Cfg{Raw: raw}
-	cfg.IsFeatureToggleEnabled = features.IsEnabled
 
 	settings := &setting.OSSImpl{Cfg: cfg}
-	assert.True(tb, settings.IsFeatureToggleEnabled(featuremgmt.FlagEnvelopeEncryption))
-	assert.True(tb, features.IsEnabled(featuremgmt.FlagEnvelopeEncryption))
 
 	encryption := ossencryption.ProvideService()
 	secretsService, err := ProvideSecretsService(

pkg/services/secrets/manager/manager.go

@@ -54,7 +54,7 @@ func ProvideSecretsService(
 	}
 
 	logger := log.New("secrets")
-	enabled := features.IsEnabled(featuremgmt.FlagEnvelopeEncryption)
+	enabled := !features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption)
 	currentProviderID := kmsproviders.NormalizeProviderID(secrets.ProviderID(
 		settings.KeyValue("security", "encryption_provider").MustString(kmsproviders.Default),
 	))
@@ -94,7 +94,7 @@ func (s *SecretsService) registerUsageMetrics() {
 
 		// Enabled / disabled
 		usageMetrics["stats.encryption.envelope_encryption_enabled.count"] = 0
-		if s.features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+		if !s.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 			usageMetrics["stats.encryption.envelope_encryption_enabled.count"] = 1
 		}
 
@@ -131,8 +131,8 @@ func (s *SecretsService) Encrypt(ctx context.Context, payload []byte, opt secret
 }
 
 func (s *SecretsService) EncryptWithDBSession(ctx context.Context, payload []byte, opt secrets.EncryptionOptions, sess *xorm.Session) ([]byte, error) {
-	// Use legacy encryption service if envelopeEncryptionFeatureToggle toggle is off
-	if !s.features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	// Use legacy encryption service if featuremgmt.FlagDisableEnvelopeEncryption toggle is on
+	if s.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		return s.enc.Encrypt(ctx, payload, setting.SecretKey)
 	}
 
@@ -291,8 +291,8 @@ func newRandomDataKey() ([]byte, error) {
 }
 
 func (s *SecretsService) Decrypt(ctx context.Context, payload []byte) ([]byte, error) {
-	// Use legacy encryption service if featuremgmt.FlagEnvelopeEncryption toggle is off
-	if !s.features.IsEnabled(featuremgmt.FlagEnvelopeEncryption) {
+	// Use legacy encryption service if featuremgmt.FlagDisableEnvelopeEncryption toggle is on
+	if s.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
 		return s.enc.Decrypt(ctx, payload, setting.SecretKey)
 	}
 

pkg/services/secrets/manager/manager_test.go

@@ -190,32 +190,26 @@ func TestSecretsService_UseCurrentProvider(t *testing.T) {
 		raw, err := ini.Load([]byte(rawCfg))
 		require.NoError(t, err)
 
-		features := featuremgmt.WithFeatures(featuremgmt.FlagEnvelopeEncryption)
-		providerID := secrets.ProviderID("fakeProvider.v1")
-		settings := &setting.OSSImpl{
-			Cfg: &setting.Cfg{
-				Raw:                    raw,
-				IsFeatureToggleEnabled: features.IsEnabled,
-			},
-		}
-		encr := ossencryption.ProvideService()
-		kms := newFakeKMS(osskmsproviders.ProvideService(encr, settings, features))
+		encryptionService := ossencryption.ProvideService()
+		settings := &setting.OSSImpl{Cfg: &setting.Cfg{Raw: raw}}
+		features := featuremgmt.WithFeatures()
+		kms := newFakeKMS(osskmsproviders.ProvideService(encryptionService, settings, features))
 		secretStore := database.ProvideSecretsStore(sqlstore.InitTestDB(t))
 
-		svcEncrypt, err := ProvideSecretsService(
+		secretsService, err := ProvideSecretsService(
 			secretStore,
 			&kms,
-			encr,
+			encryptionService,
 			settings,
 			features,
 			&usagestats.UsageStatsMock{T: t},
 		)
 		require.NoError(t, err)
 
-		assert.Equal(t, providerID, svcEncrypt.currentProviderID)
-		assert.Equal(t, 2, len(svcEncrypt.GetProviders()))
+		assert.Equal(t, secrets.ProviderID("fakeProvider.v1"), secretsService.currentProviderID)
+		assert.Equal(t, 2, len(secretsService.GetProviders()))
 
-		encrypted, _ := svcEncrypt.Encrypt(context.Background(), []byte{}, secrets.WithoutScope())
+		encrypted, _ := secretsService.Encrypt(context.Background(), []byte{}, secrets.WithoutScope())
 		assert.True(t, kms.fake.encryptCalled)
 
 		// secret service tries to find a DEK in a cache first before calling provider's decrypt
@@ -223,7 +217,7 @@ func TestSecretsService_UseCurrentProvider(t *testing.T) {
 		svcDecrypt, err := ProvideSecretsService(
 			secretStore,
 			&kms,
-			encr,
+			encryptionService,
 			settings,
 			features,
 			&usagestats.UsageStatsMock{T: t},