Files
grafana/pkg/services/secrets/kvstore/migrations/migrator.go

106 lines
3.7 KiB
Go
Raw Normal View History

Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
package migrations
import (
"context"
"reflect"
"time"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/infra/serverlock"
"github.com/grafana/grafana/pkg/registry"
"github.com/grafana/grafana/pkg/setting"
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
)
var logger = log.New("secret.migration")
const actionName = "secret migration task "
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
// SecretMigrationService is used to migrate legacy secrets to new unified secrets.
type SecretMigrationService interface {
Migrate(ctx context.Context) error
}
type SecretMigrationProvider interface {
registry.BackgroundService
TriggerPluginMigration(ctx context.Context, toPlugin bool) error
}
type SecretMigrationProviderImpl struct {
services []SecretMigrationService
ServerLockService *serverlock.ServerLockService
migrateToPluginService *MigrateToPluginService
migrateFromPluginService *MigrateFromPluginService
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
}
func ProvideSecretMigrationProvider(
cfg *setting.Cfg,
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
serverLockService *serverlock.ServerLockService,
dataSourceSecretMigrationService *DataSourceSecretMigrationService,
migrateToPluginService *MigrateToPluginService,
migrateFromPluginService *MigrateFromPluginService,
) *SecretMigrationProviderImpl {
services := make([]SecretMigrationService, 0)
services = append(services, dataSourceSecretMigrationService)
// Plugin migration should always be last; should either migrate to or from, not both
// This is because the migrateTo checks for use_plugin = true, in which case we should always
// migrate by default to ensure users don't lose access to secrets. If migration has
// already occurred, the migrateTo function will be called but it won't do anything
if cfg.SectionWithEnvOverrides("secrets").Key("migrate_from_plugin").MustBool(false) {
services = append(services, migrateFromPluginService)
} else {
services = append(services, migrateToPluginService)
}
return &SecretMigrationProviderImpl{
ServerLockService: serverLockService,
services: services,
migrateToPluginService: migrateToPluginService,
migrateFromPluginService: migrateFromPluginService,
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
}
}
func (s *SecretMigrationProviderImpl) Run(ctx context.Context) error {
return s.Migrate(ctx)
}
// Migrate Run migration services. This will block until all services have exited.
// This should only be called once at startup
func (s *SecretMigrationProviderImpl) Migrate(ctx context.Context) error {
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
// Start migration services.
err := s.ServerLockService.LockExecuteAndRelease(ctx, actionName, time.Minute*10, func(context.Context) {
for _, service := range s.services {
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
serviceName := reflect.TypeOf(service).String()
logger.Debug("Starting secret migration service", "service", serviceName)
err := service.Migrate(ctx)
if err != nil {
logger.Error("Stopped secret migration service", "service", serviceName, "reason", err)
}
logger.Debug("Finished secret migration service", "service", serviceName)
}
})
if err != nil {
logger.Error("Server lock for secret migration already exists")
}
return nil
Secrets: Improve unified secrets migration and implement compatibility flag (#50463) * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2022-07-12 17:27:37 -03:00
}
// TriggerPluginMigration Kick off a migration to or from the plugin. This will block until all services have exited.
func (s *SecretMigrationProviderImpl) TriggerPluginMigration(ctx context.Context, toPlugin bool) error {
// Don't migrate if there is already one happening
return s.ServerLockService.LockExecuteAndRelease(ctx, actionName, time.Minute*10, func(context.Context) {
var err error
if toPlugin {
err = s.migrateToPluginService.Migrate(ctx)
} else {
err = s.migrateFromPluginService.Migrate(ctx)
}
if err != nil {
direction := "from_plugin"
if toPlugin {
direction = "to_plugin"
}
logger.Error("Failed to migrate plugin secrets", "direction", direction, "error", err.Error())
}
})
}