Encryption: Expose secrets migrations through HTTP API (#51707)

* Encryption: Move secrets migrations into secrets.Migrator

* Encryption: Refactor secrets.Service initialization

* Encryption: Add support to run secrets migrations even when EE is disabled

* Encryption: Expose secrets migrations through HTTP API

* Update docs

* Fix docs links

* Some adjustments to makes errors explicit through HTTP response

docs/sources/developers/http_api/admin.md

@@ -718,11 +718,7 @@ Content-Type: application/json
 
 `POST /api/admin/encryption/rotate-data-keys`
 
-Rotates data encryption keys, so all the active keys are disabled
+[Rotates]({{< relref "../../setup-grafana/configure-security/configure-database-encryption/#rotate-data-keys" >}}) data encryption keys.
-and no longer used for encryption but kept for decryption operations.
-
-Secrets encrypted with one of the deactivated keys need to be re-encrypted
-to actually stop using those keys for both encryption and decryption.
 
 **Example Request**:
 
@@ -738,3 +734,66 @@ Content-Type: application/json
 HTTP/1.1 204
 Content-Type: application/json
 ```
+
+## Re-encrypt data encryption keys
+
+`POST /api/admin/encryption/reencrypt-data-keys`
+
+[Re-encrypts]({{< relref "../../setup-grafana/configure-security/configure-database-encryption/#re-encrypt-data-keys" >}}) data encryption keys.
+
+**Example Request**:
+
+```http
+POST /api/admin/encryption/reencrypt-data-keys HTTP/1.1
+Accept: application/json
+Content-Type: application/json
+```
+
+**Example Response**:
+
+```http
+HTTP/1.1 204
+Content-Type: application/json
+```
+
+## Re-encrypt secrets
+
+`POST /api/admin/encryption/reencrypt-secrets`
+
+[Re-encrypts]({{< relref "../../setup-grafana/configure-security/configure-database-encryption/#re-encrypt-secrets" >}}) secrets.
+
+**Example Request**:
+
+```http
+POST /api/admin/encryption/reencrypt-secrets HTTP/1.1
+Accept: application/json
+Content-Type: application/json
+```
+
+**Example Response**:
+
+```http
+HTTP/1.1 204
+Content-Type: application/json
+```
+
+## Roll back secrets
+
+`POST /api/admin/encryption/rollback-secrets`
+
+[Rolls back]({{< relref "../../setup-grafana/configure-security/configure-database-encryption/#roll-back-secrets" >}}) secrets.
+
+**Example Request**:
+
+```http
+POST /api/admin/encryption/rollback-secrets HTTP/1.1
+Accept: application/json
+Content-Type: application/json
+```
+
+**Example Response**:
+
+```http
+HTTP/1.1 204
+Content-Type: application/json
+```

docs/sources/setup-grafana/configure-security/configure-database-encryption/_index.md

@@ -18,7 +18,7 @@ Grafana encrypts these secrets before they are written to the database, by using
 Since Grafana v9.0, it uses [envelope encryption](#envelope-encryption) by default, which adds a layer of indirection to the
 encryption process that represents an [**implicit breaking change**](#implicit-breaking-change) for older versions of Grafana.
 
-For further details about how to operate a Grafana instance with envelope encryption, see the [Operational work]({{< relref "/#operational-work" >}}) section below.
+For further details about how to operate a Grafana instance with envelope encryption, see the [Operational work](#operational-work) section below.
 
 > **Note:** In Grafana Enterprise, you can also choose to [encrypt secrets in AES-GCM mode]({{< relref "#changing-your-encryption-mode-to-aes-gcm" >}}) instead of AES-CFB.
 
@@ -31,7 +31,7 @@ Instead of encrypting all secrets with a single key, Grafana uses a set of keys
 encrypt them. These data encryption keys are themselves encrypted with a single key encryption key (KEK), configured
 through the `secret_key` attribute in your
 [Grafana configuration]({{< relref "../../configure-grafana/#secret_key" >}}) or with a
-[KMS integration](#kms-integration).
+[KMS integration](#encrypting-your-database-with-a-key-from-a-key-management-system-kms).
 
 ## Implicit breaking change
 
@@ -67,7 +67,8 @@ Secrets re-encryption can be performed when a Grafana administrator wants to eit
 - Re-encrypt secrets after a [data keys rotation](#rotate-data-keys).
 
 > **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration re-encrypt`
-> command. It's safe to run more than once. Recommended to run under maintenance mode.
+> command and through Grafana [Admin API]({{< relref "../../../developers/http_api/admin/#re-encrypt-secrets" >}}).
+> It's safe to run more than once. Recommended to run under maintenance mode.
 
 ## Roll back secrets
 
@@ -75,16 +76,18 @@ Used to roll back secrets encrypted with envelope encryption to legacy encryptio
 a Grafana version earlier than Grafana v9.0 after an unsuccessful upgrade.
 
 > **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration rollback`
-> command. It's safe to run more than once. Recommended to run under maintenance mode.
+> command and through Grafana [Admin API]({{< relref "../../../developers/http_api/admin/#roll-back-secrets" >}}).
+> It's safe to run more than once. Recommended to run under maintenance mode.
 
 ## Re-encrypt data keys
 
 Used to re-encrypt data keys encrypted with a specific key encryption key (KEK). It can be used to either re-encrypt
-existing data keys with a new key encryption key version (see [KMS integration](#kms-integration) rotation) or to
+existing data keys with a new key encryption key version (see [KMS integration](#encrypting-your-database-with-a-key-from-a-key-management-system-kms) rotation) or to
 re-encrypt them with a completely different key encryption key.
 
 > **Note:** This operation is available through Grafana CLI by running `grafana-cli admin secrets-migration re-encrypt-data-keys`
-> command. It's safe to run more than once. Recommended to run under maintenance mode.
+> command and through Grafana [Admin API]({{< relref "../../../developers/http_api/admin/#re-encrypt-data-encryption-keys" >}}).
+> It's safe to run more than once. Recommended to run under maintenance mode.
 
 ## Rotate data keys
 

pkg/api/admin_encryption.go

@@ -9,8 +9,42 @@ import (
 
 func (hs *HTTPServer) AdminRotateDataEncryptionKeys(c *models.ReqContext) response.Response {
 	if err := hs.SecretsService.RotateDataKeys(c.Req.Context()); err != nil {
-		return response.Error(http.StatusInternalServerError, "Failed to rotate data key", err)
+		return response.Error(http.StatusInternalServerError, "Failed to rotate data keys", err)
 	}
 
 	return response.Respond(http.StatusNoContent, "")
 }
+
+func (hs *HTTPServer) AdminReEncryptEncryptionKeys(c *models.ReqContext) response.Response {
+	if err := hs.SecretsService.ReEncryptDataKeys(c.Req.Context()); err != nil {
+		return response.Error(http.StatusInternalServerError, "Failed to re-encrypt data keys", err)
+	}
+
+	return response.Respond(http.StatusOK, "Data encryption keys re-encrypted successfully")
+}
+
+func (hs *HTTPServer) AdminReEncryptSecrets(c *models.ReqContext) response.Response {
+	success, err := hs.secretsMigrator.ReEncryptSecrets(c.Req.Context())
+	if err != nil {
+		return response.Error(http.StatusInternalServerError, "Failed to re-encrypt secrets", err)
+	}
+
+	if !success {
+		return response.Error(http.StatusPartialContent, "Something unexpected happened, refer to the server logs for more details", err)
+	}
+
+	return response.Respond(http.StatusOK, "Secrets re-encrypted successfully")
+}
+
+func (hs *HTTPServer) AdminRollbackSecrets(c *models.ReqContext) response.Response {
+	success, err := hs.secretsMigrator.RollBackSecrets(c.Req.Context())
+	if err != nil {
+		return response.Error(http.StatusInternalServerError, "Failed to rollback secrets", err)
+	}
+
+	if !success {
+		return response.Error(http.StatusPartialContent, "Something unexpected happened, refer to the server logs for more details", err)
+	}
+
+	return response.Respond(http.StatusOK, "Secrets rolled back successfully")
+}

pkg/api/api.go

@@ -575,6 +575,9 @@ func (hs *HTTPServer) registerRoutes() {
 		}
 
 		adminRoute.Post("/encryption/rotate-data-keys", reqGrafanaAdmin, routing.Wrap(hs.AdminRotateDataEncryptionKeys))
+		adminRoute.Post("/encryption/reencrypt-data-keys", reqGrafanaAdmin, routing.Wrap(hs.AdminReEncryptEncryptionKeys))
+		adminRoute.Post("/encryption/reencrypt-secrets", reqGrafanaAdmin, routing.Wrap(hs.AdminReEncryptSecrets))
+		adminRoute.Post("/encryption/rollback-secrets", reqGrafanaAdmin, routing.Wrap(hs.AdminRollbackSecrets))
 
 		adminRoute.Post("/provisioning/dashboards/reload", authorize(reqGrafanaAdmin, ac.EvalPermission(ActionProvisioningReload, ScopeProvisionersDashboards)), routing.Wrap(hs.AdminProvisioningReloadDashboards))
 		adminRoute.Post("/provisioning/plugins/reload", authorize(reqGrafanaAdmin, ac.EvalPermission(ActionProvisioningReload, ScopeProvisionersPlugins)), routing.Wrap(hs.AdminProvisioningReloadPlugins))

pkg/cmd/grafana-cli/commands/secretsmigrations/secretsmigrations.go

@@ -12,9 +12,11 @@ func ReEncryptDEKS(_ utils.CommandLine, runner runner.Runner) error {
 }
 
 func ReEncryptSecrets(_ utils.CommandLine, runner runner.Runner) error {
-	return runner.SecretsMigrator.ReEncryptSecrets(context.Background())
+	_, err := runner.SecretsMigrator.ReEncryptSecrets(context.Background())
+	return err
 }
 
 func RollBackSecrets(_ utils.CommandLine, runner runner.Runner) error {
-	return runner.SecretsMigrator.RollBackSecrets(context.Background())
+	_, err := runner.SecretsMigrator.RollBackSecrets(context.Background())
+	return err
 }

pkg/services/secrets/migrator/migrator.go

@@ -37,14 +37,14 @@ func ProvideSecretsMigrator(
 	}
 }
 
-func (m *SecretsMigrator) ReEncryptSecrets(ctx context.Context) error {
+func (m *SecretsMigrator) ReEncryptSecrets(ctx context.Context) (bool, error) {
 	err := m.initProvidersIfNeeded()
 	if err != nil {
-		return err
+		return false, err
 	}
 
 	toReencrypt := []interface {
-		reencrypt(context.Context, *manager.SecretsService, *sqlstore.SQLStore)
+		reencrypt(context.Context, *manager.SecretsService, *sqlstore.SQLStore) bool
 	}{
 		simpleSecret{tableName: "dashboard_snapshot", columnName: "dashboard_encrypted"},
 		b64Secret{simpleSecret: simpleSecret{tableName: "user_auth", columnName: "o_auth_access_token"}, encoding: base64.StdEncoding},
@@ -56,30 +56,21 @@ func (m *SecretsMigrator) ReEncryptSecrets(ctx context.Context) error {
 		alertingSecret{},
 	}
 
-	for _, r := range toReencrypt {
+	var anyFailure bool
-		r.reencrypt(ctx, m.secretsSrv, m.sqlStore)
-	}
-
-	return nil
-}
-
-func (m *SecretsMigrator) initProvidersIfNeeded() error {
-	if m.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
-		logger.Info("Envelope encryption is not enabled but trying to init providers anyway...")
 
-		if err := m.secretsSrv.InitProviders(); err != nil {
+	for _, r := range toReencrypt {
-			logger.Error("Envelope encryption providers initialization failed", "error", err)
+		if success := r.reencrypt(ctx, m.secretsSrv, m.sqlStore); !success {
-			return err
+			anyFailure = true
 		}
 	}
 
-	return nil
+	return !anyFailure, nil
 }
 
-func (m *SecretsMigrator) RollBackSecrets(ctx context.Context) error {
+func (m *SecretsMigrator) RollBackSecrets(ctx context.Context) (bool, error) {
 	err := m.initProvidersIfNeeded()
 	if err != nil {
-		return err
+		return false, err
 	}
 
 	toRollback := []interface {
@@ -110,11 +101,26 @@ func (m *SecretsMigrator) RollBackSecrets(ctx context.Context) error {
 
 	if anyFailure {
 		logger.Warn("Some errors happened, not cleaning up data keys table...")
-		return nil
+		return false, nil
 	}
 
-	if _, sqlErr := m.sqlStore.NewSession(ctx).Exec("DELETE FROM data_keys"); sqlErr != nil {
+	_, sqlErr := m.sqlStore.NewSession(ctx).Exec("DELETE FROM data_keys")
+	if sqlErr != nil {
 		logger.Warn("Error while cleaning up data keys table...", "error", sqlErr)
+		return false, nil
+	}
+
+	return true, nil
+}
+
+func (m *SecretsMigrator) initProvidersIfNeeded() error {
+	if m.features.IsEnabled(featuremgmt.FlagDisableEnvelopeEncryption) {
+		logger.Info("Envelope encryption is not enabled but trying to init providers anyway...")
+
+		if err := m.secretsSrv.InitProviders(); err != nil {
+			logger.Error("Envelope encryption providers initialization failed", "error", err)
+			return err
+		}
 	}
 
 	return nil

pkg/services/secrets/migrator/reencrypt.go

@@ -12,7 +12,7 @@ import (
 	"github.com/grafana/grafana/pkg/services/sqlstore"
 )
 
-func (s simpleSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) {
+func (s simpleSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) bool {
 	var rows []struct {
 		Id     int
 		Secret []byte
@@ -20,7 +20,7 @@ func (s simpleSecret) reencrypt(ctx context.Context, secretsSrv *manager.Secrets
 
 	if err := sqlStore.NewSession(ctx).Table(s.tableName).Select(fmt.Sprintf("id, %s as secret", s.columnName)).Find(&rows); err != nil {
 		logger.Warn("Could not find any secret to re-encrypt", "table", s.tableName)
-		return
+		return false
 	}
 
 	var anyFailure bool
@@ -62,9 +62,11 @@ func (s simpleSecret) reencrypt(ctx context.Context, secretsSrv *manager.Secrets
 	} else {
 		logger.Info(fmt.Sprintf("Column %s from %s has been re-encrypted successfully", s.columnName, s.tableName))
 	}
+
+	return !anyFailure
 }
 
-func (s b64Secret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) {
+func (s b64Secret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) bool {
 	var rows []struct {
 		Id     int
 		Secret string
@@ -72,7 +74,7 @@ func (s b64Secret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsSer
 
 	if err := sqlStore.NewSession(ctx).Table(s.tableName).Select(fmt.Sprintf("id, %s as secret", s.columnName)).Find(&rows); err != nil {
 		logger.Warn("Could not find any secret to re-encrypt", "table", s.tableName)
-		return
+		return false
 	}
 
 	var anyFailure bool
@@ -128,9 +130,11 @@ func (s b64Secret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsSer
 	} else {
 		logger.Info(fmt.Sprintf("Column %s from %s has been re-encrypted successfully", s.columnName, s.tableName))
 	}
+
+	return !anyFailure
 }
 
-func (s jsonSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) {
+func (s jsonSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) bool {
 	var rows []struct {
 		Id             int
 		SecureJsonData map[string][]byte
@@ -138,7 +142,7 @@ func (s jsonSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsSe
 
 	if err := sqlStore.NewSession(ctx).Table(s.tableName).Cols("id", "secure_json_data").Find(&rows); err != nil {
 		logger.Warn("Could not find any secret to re-encrypt", "table", s.tableName)
-		return
+		return false
 	}
 
 	var anyFailure bool
@@ -184,9 +188,11 @@ func (s jsonSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsSe
 	} else {
 		logger.Info(fmt.Sprintf("Secure json data secrets from %s have been re-encrypted successfully", s.tableName))
 	}
+
+	return !anyFailure
 }
 
-func (s alertingSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) {
+func (s alertingSecret) reencrypt(ctx context.Context, secretsSrv *manager.SecretsService, sqlStore *sqlstore.SQLStore) bool {
 	var results []struct {
 		Id                        int
 		AlertmanagerConfiguration string
@@ -195,7 +201,7 @@ func (s alertingSecret) reencrypt(ctx context.Context, secretsSrv *manager.Secre
 	selectSQL := "SELECT id, alertmanager_configuration FROM alert_configuration"
 	if err := sqlStore.NewSession(ctx).SQL(selectSQL).Find(&results); err != nil {
 		logger.Warn("Could not find any alert_configuration secret to re-encrypt")
-		return
+		return false
 	}
 
 	var anyFailure bool
@@ -261,4 +267,6 @@ func (s alertingSecret) reencrypt(ctx context.Context, secretsSrv *manager.Secre
 	} else {
 		logger.Info("Alerting configuration secrets have been re-encrypted successfully")
 	}
+
+	return !anyFailure
 }

pkg/services/secrets/secrets.go

@@ -72,6 +72,14 @@ type BackgroundProvider interface {
 
 // Migrator is responsible for secrets migrations like re-encrypting or rolling back secrets.
 type Migrator interface {
-	ReEncryptSecrets(ctx context.Context) error
+	// ReEncryptSecrets decrypts and re-encrypts the secrets with most recent
-	RollBackSecrets(ctx context.Context) error
+	// available data key. If a secret-specific decryption / re-encryption fails,
+	// it does not stop, but returns false as the first return (success or not)
+	// at the end of the process.
+	ReEncryptSecrets(ctx context.Context) (bool, error)
+	// RollBackSecrets decrypts and re-encrypts the secrets using the legacy
+	// encryption. If a secret-specific decryption / re-encryption fails, it
+	// does not stop, but returns false as the first return (success or not)
+	// at the end of the process.
+	RollBackSecrets(ctx context.Context) (bool, error)
 }