94 Commits (a901c512a0dd5773303745453e6f8acd77cf4c07)

Author	SHA1	Message	Date
lean.dev	e1785f4eb4	Secret migration from Sql KV Store to Secret Plugin (#52191) ... * Created PluginSecretMigrationService to be able to migrate from the secrets table from the database to the secret plugin. Added migration which takes all the secrets at the sql store and stores it in the plugin. Then deletes all the secrets from the sql * Added secretsKVStoreSQL.GetAll() method to return all the secrets at the sql table * Renaming kvstore_test.go as sql_test.go, adding GetAll test case. Fixing decryption of keys	3 years ago
Joan López de la Franca Beltran	9abe9fa702	Encryption: Expose secrets migrations through HTTP API (#51707) ... * Encryption: Move secrets migrations into secrets.Migrator * Encryption: Refactor secrets.Service initialization * Encryption: Add support to run secrets migrations even when EE is disabled * Encryption: Expose secrets migrations through HTTP API * Update docs * Fix docs links * Some adjustments to makes errors explicit through HTTP response	3 years ago
Joan López de la Franca Beltran	7b40322bbe	Encryption: Add support to run secrets migrations even when EE is disabled (#51705) ... * Encryption: Move secrets migrations into secrets.Migrator * Encryption: Refactor secrets.Service initialization * Encryption: Add support to run secrets migrations even when EE is disabled * Init EE providers on-demand (only when needed) * Add multiple tests + some adjustments * Apply feedback	3 years ago
Joan López de la Franca Beltran	973b3112d7	Encryption: Re-order error check (#52251)	3 years ago
Guilherme Caulada	2d8a91a846	Secrets: Improve unified secrets migration and implement compatibility flag (#50463) ... * Implement disableSecretsCompatibility flag * Allow secret deletion right after migration * Use dialect.Quote for secure_json_data on secret deletion Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Set secure_json_data to NULL instead of empty json * Run toggles_gen_test and use generated flag variable * Add ID to delete data source secrets command on function call Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Remove extra query to get datasource on secret deletion * Fix linting issues with CHANGELOG.md * Use empty json string when deleting secure json data * Implement secret migration as a background process * Refactor secret migration as a background service * Refactor migration to be inside secret store * Re-add secret deletion function removed on merge * Try using transaction to fix db lock during tests * Disable migration for pipeline debugging * Try adding sleep to fix database lock * Remove unecessary time sleep from migration * Fix merge issue, replace models with datasources * Try event listener approach * Fix merge issue, replace models with datasources * Fix linting issues with unchecked error * Remove unecessary trainling new line * Increase wait interval on background secret migration * Rename secret store migration folder for consistency * Convert background migration to blocking * Fix number of arguments on server tests * Check error value of secret migration provider * Fix linting issue with method varaible * Revert unintended change on background services * Move secret migration service provider to wire.go * Remove unecessary else from datasource service * Move transaction inside loop on secret migration * Remove unecessary GetServices function * Remove unecessary interface after method removal * Rename Run to Migrate on secret migration interface * Rename secret migrations service variable on server * Use MustBool on datasource secret migration * Revert changes to GetDataSources * Implement GetAllDataSources function * Remove DeleteDataSourceSecrets function * Move datasource secret migration to datasource service * Remove unecessary properties from datasource secret migration * Make DecryptLegacySecrets a private method * Remove context canceled check on secret migrator * Log error when fail to unmarshal datasource secret * Add necessary fields to update command on migration * Handle high availability on secret migration * Use kvstore for datasource secret migration status * Add error check for migration status set on kvstore * Remove NewSecretMigrationService from server tests * Use const for strings on datasource secrets migration * Test all cases for datasources secret migrations Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>	3 years ago
Joan López de la Franca Beltran	81753526bd	Encryption: Refactor secrets.Service initialization (#51091) ... * Encryption: Refactor secrets.Service initialization	3 years ago
Joan López de la Franca Beltran	38bcd37fba	Encryption: Move secrets migrations into secrets.Migrator (#51014)	3 years ago
Guilherme Caulada	d5185f8ab9	Secrets: Implement unified secrets short lived cache (#51275) ... * Implement unified secrets short lived cache * Improve debug logging for unified secrets cache * Re-add decryption cache to sql secret kvstore * Remove cache from remote secret store plugin * Revert secret store helpers implementation * Remove cache from secret store plugin struct * Update secret store cache to implement interface * Set secret store cache value on get * Fix issues with sql secret store decryption cache * Increase clean up interval on cached secret store	3 years ago
Kristin Laemmert	945f015770	backend/datasources: move datasources models into the datasources service package (#51267) ... * backend/datasources: move datasources models into the datasources service pkg	3 years ago
Michael Mandrus	c043a8818a	Secrets: add better error handling for secret plugin failures when updating datasources (#50542) ... * Add protobuf config and generated code, and client wrapper * wire up loading of secretsmanager plugin, using renderer plugin as a model * update kvstore provider to check if we should use the grpc plugin. return false always in OSS * add OSS remote plugin check * refactor wire gen file * log which secrets manager is being used * Fix argument types for remote checker * Turns out if err != nil, then the result is always nil. Return empty values if there is an error. * remove duplicate import * ensure atomicity by adding secret management as a step to sql operations and rolling back if necessary * Update pkg/services/secrets/kvstore/kvstore.go Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> * Update pkg/services/secrets/kvstore/kvstore.go Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> * refactor RemotePluginCheck interface to just return the Plugin client directly * rename struct to something less silly * add special error handling for remote secrets management * switch to errors.as instead of type inference * remove unnecessary rollback call * just declare error once * refactor .proto file according to prior PR suggestions * re-generate protobuf files and fix compilation errors * only wrap (ergo display in the front end) errors that are user friendly from the plugin * rename error type to suggest user friendly only * rename plugin functions to be more descriptive * change delete message name * Revert "change delete message name" This reverts commit 8ca978301e. * Revert "rename plugin functions to be more descriptive" This reverts commit 4355c9b9ff. * fix pointer to pointer problem * change plugin user error to just hold a string * fix sequencing problem with datasource updates * clean up some return statements * need to wrap multiple transactions with the InTransaction() func in order to keep the lock * make linter happy * revert input var name Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>	3 years ago
Michael Mandrus	d886141d69	Plugins: Refactor secrets plugin .proto file (#50722) ... * refactor .proto file according to prior PR suggestions * re-generate protobuf files and fix compilation errors * rename proto functions to be more descriptive	3 years ago
Joan López de la Franca Beltran	97baa6911d	Metrics: Expose functions to initialize counters at zero (#50122)	3 years ago
Michael Mandrus	f376c33903	WIP: Add private Secret Manager Plugins support to plugin platform (#49544) ... * Add protobuf config and generated code, and client wrapper * wire up loading of secretsmanager plugin, using renderer plugin as a model * update kvstore provider to check if we should use the grpc plugin. return false always in OSS * add OSS remote plugin check * refactor wire gen file * log which secrets manager is being used * Fix argument types for remote checker * Turns out if err != nil, then the result is always nil. Return empty values if there is an error. * remove duplicate import * Update pkg/services/secrets/kvstore/kvstore.go Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> * Update pkg/services/secrets/kvstore/kvstore.go Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> * refactor RemotePluginCheck interface to just return the Plugin client directly * rename struct to something less silly * Update pkg/plugins/backendplugin/secretsmanagerplugin/secretsmanager.proto Co-authored-by: Will Browne <wbrowne@users.noreply.github.com> Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>	3 years ago
Tania	4f8111e24e	Encryption: Fix multiple data keys migration (#49848) ... * Add migration * Migrator: Extend support to rename columns * Fix getting current key * Fix column name in migration * Fix deks reencryption * Fix caching * Add back separate caches for byName and byPrefix * Do not concatenate prefix with uid * Rename DataKey struc fields * SQLStore: Add deprecation comments for breaking migrations * Add comment * Minor corrections Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>	3 years ago
Tania	7a614fd8a1	Encryption: Stop decrypting EE encrypted secrets with legacy encryption (#50090)	3 years ago
Joan López de la Franca Beltran	3e4b4dba46	Encryption: Enable envelope encryption by default (#49301) ... * Encryption: Enable envelope encryption by default * Stop relying on feature toggles from settings (deprecated) * Database encryption docs (envelope encryption) * Remove deprecated (and no longer used) FT * Apply suggestions from code review Co-authored-by: Tania <yalyna.ts@gmail.com>	3 years ago
Joan López de la Franca Beltran	e43879e55d	Encryption: Add support for multiple data keys per day (#47765) ... * Add database migrations * Use short uids as data key ids * Add support for manual data key rotation * Fix duplicated mutex unlocks * Fix migration * Manage current data keys per name * Adjust key re-encryption and test * Modify rename column migration for MySQL compatibility * Refactor secrets manager and data keys cache * Multiple o11y adjustments * Fix stats query * Apply suggestions from code review Co-authored-by: Tania <yalyna.ts@gmail.com> * Fix linter * Docs: Rotate data encryption keys API endpoint Co-authored-by: Tania <yalyna.ts@gmail.com>	3 years ago
Joan López de la Franca Beltran	9826a694a8	Encryption: Add Prometheus metrics (#48603)	3 years ago
Guilherme Caulada	a367ad730c	Secrets: Implement basic unified secret store service (#45804) ... * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test	3 years ago
Joan López de la Franca Beltran	2081f37e95	Encryption: Make DEKs cache TTL & cleanup interval configurable (#46042) ... * Make DEKs cache TTL & cleanup interval configurable * Improve 'data_keys_cache_ttl' setting description * Fix test	3 years ago
Joan López de la Franca Beltran	e6a85826e9	Encryption: Refine secrets-related commands (#45201) ... * CLI: Adjust 're-encrypt-data-keys' command * CLI: Adjust 're-encrypt' command * Multiple improvements on re-encrypt secrets migration * Another bunch of code improvements * Lint fixes	3 years ago
Joan López de la Franca Beltran	b2655750e8	Encryption: Add support for data keys re-encryption (#43548) ... * Encryption: Add support for data keys re-encryption * Add tests for data keys re-encryption * Update code after refactorings Co-authored-by: Leonard Gram <leo@xlson.com>	3 years ago
Joan López de la Franca Beltran	3b4e812449	Encryption: Keep legacy default provider id on providers map (#44721) ... * Encryption: Keep legacy default provider id on providers map * Minor fixes * Refactor	3 years ago
Ryan McKinley	7ee38af95a	FeatureToggls: remove IsFeatureToggleEnabled from SettingsProvider (#44574)	3 years ago
Tania B	ca24b95b49	Encryption: Handle encryption key provider being a background service (#44007) ... * Encryption: Handle encryption key provider being a background service * Sort imports * Cleanup accidental changes * Add proper error handling * Apply review feedback	3 years ago
Ryan McKinley	5d66194ec5	FeatureFlags: define features outside settings.Cfg (take 3) (#44443)	3 years ago
Agnès Toulet	65bdb3a899	FeatureFlags: Revert managing feature flags outside of settings.Cfg (#44382) ... * Revert "FeatureToggles: register all enterprise feature toggles (#44336)" This reverts commit f53b3fb007. * Revert "FeatureFlags: manage feature flags outside of settings.Cfg (#43692)" This reverts commit f94c0decbd.	3 years ago
Ryan McKinley	f94c0decbd	FeatureFlags: manage feature flags outside of settings.Cfg (#43692)	3 years ago
Joan López de la Franca Beltran	532e71554f	Usage Stats: Add metrics to count enabled kms providers per kind (#43640) ... * Usage Insights: Add metrics to count enabled kms providers per kind * Add backwards compatibility	3 years ago
Joan López de la Franca Beltran	80e0dd74d2	Encryption: Increase context timeout on flaky test (#43553)	3 years ago
Joan López de la Franca Beltran	83bc445d3e	Encryption: Fix DEKs cache (#43129) ... * Encryption: Fix DEKs cache * Clarify tests	3 years ago
Tania B	0dc86bf7f4	Chore: Move default encryption provider to kmsproviders package (#43331)	3 years ago
Tania B	58978dcf96	Encryption: Add usage stats to secrets service (#42437) ... * Encryption: Add usage stats to secrets service * Sort imports	4 years ago
Tania B	4014891971	Encryption: Cleanup and add logging (#42084) ... * Encryption: Add more logs * Add logging and checks * Removed unused methods from secrets service * Refactor and update tests * Address review feedback	4 years ago
Joan López de la Franca Beltran	d49230d291	Grafana CLI Wire Runner (#41012) ... * Set up Wire build graph * Remove enterprise Wire set * Move runner package outside commands * Update Makefile (gen-go path) * Minor prettier fix * Include new Wire enterprise file into .gitignore * Update Wire deps * Update the grabpl version Co-authored-by: Dan Cech <dcech@grafana.com>	4 years ago
Tania B	bc60ae3c66	Encryption: Refactor secrets service (#41771) ... * Refactor kmsproviders pkg * Update tests * Fix linting Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>	4 years ago
Joan López de la Franca Beltran	d3e19b1b3b	Encryption: Improve the DX of encryption operations within database transactions (#41654) ... * Move user oauth info encryption away from db transaction * Add encryption methods with support for db session reusability	4 years ago
Joan López de la Franca Beltran	44837fc592	Replace encryption.Service usages by secrets.Service (#41625) ... * Replace encryption.Service by secrets.Service on expr.Service * Replace encryption.Service by secrets.Service on live pkg * Rename encryption.Service to encryption.Internal to clarify it must be not used	4 years ago
Tania B	f6545ab8f4	Chore: Add current provider test for secrets service (#41387) ... * Chore: Add current provider test for secrets service * Refactor the test * Fix linting issue	4 years ago
Tania B	e81d434edf	Encryption: Extend secrets service to support registering key providers (#40626) ... * Draft adding kms providers * Rename defaultProvider to currentProvider * Add getting current provider from config * Remove comments * Make current provider service struct field * Add methods to secrets service * Test getting current provider * Implements missing methods for fake secrets service * Remove accidental changes * Fix linter issue * Update configuration examples * Rename CurrentProvider method * Split service interface * Update wire Co-authored-by: spinillos <selenepinillos@gmail.com>	4 years ago
Tania B	5652bde447	Encryption: Use secrets service (#40251) ... * Use secrets service in pluginproxy * Use secrets service in pluginxontext * Use secrets service in pluginsettings * Use secrets service in provisioning * Use secrets service in authinfoservice * Use secrets service in api * Use secrets service in sqlstore * Use secrets service in dashboardshapshots * Use secrets service in tsdb * Use secrets service in datasources * Use secrets service in alerting * Use secrets service in ngalert * Break cyclic dependancy * Refactor service * Break cyclic dependancy * Add FakeSecretsStore * Setup Secrets Service in sqlstore * Fix * Continue secrets service refactoring * Fix cyclic dependancy in sqlstore tests * Fix secrets service references * Fix linter errors * Add fake secrets service for tests * Refactor SetupTestSecretsService * Update setting up secret service in tests * Fix missing secrets service in multiorg_alertmanager_test * Use fake db in tests and sort imports * Use fake db in datasources tests * Fix more tests * Fix linter issues * Attempt to fix plugin proxy tests * Pass secrets service to getPluginProxiedRequest in pluginproxy tests * Fix pluginproxy tests * Revert using secrets service in alerting and provisioning * Update decryptFn in alerting migration * Rename defaultProvider to currentProvider * Use fake secrets service in alert channels tests * Refactor secrets service test helper * Update setting up secrets service in tests * Revert alerting changes in api * Add comments * Remove secrets service from background services * Convert global encryption functions into vars * Revert "Convert global encryption functions into vars" This reverts commit 498eb19859. * Add feature toggle for envelope encryption * Rename toggle Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com> Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>	4 years ago
Tania B	f59aabbd3b	Chore: Refactor secrets service (#40331)	4 years ago
Joan López de la Franca Beltran	722c414fef	Encryption: Refactor securejsondata.SecureJsonData to stop relying on global functions (#38865) ... * Encryption: Add support to encrypt/decrypt sjd * Add datasources.Service as a proxy to datasources db operations * Encrypt ds.SecureJsonData before calling SQLStore * Move ds cache code into ds service * Fix tlsmanager tests * Fix pluginproxy tests * Remove some securejsondata.GetEncryptedJsonData usages * Add pluginsettings.Service as a proxy for plugin settings db operations * Add AlertNotificationService as a proxy for alert notification db operations * Remove some securejsondata.GetEncryptedJsonData usages * Remove more securejsondata.GetEncryptedJsonData usages * Fix lint errors * Minor fixes * Remove encryption global functions usages from ngalert * Fix lint errors * Minor fixes * Minor fixes * Remove securejsondata.DecryptedValue usage * Refactor the refactor * Remove securejsondata.DecryptedValue usage * Move securejsondata to migrations package * Move securejsondata to migrations package * Minor fix * Fix integration test * Fix integration tests * Undo undesired changes * Fix tests * Add context.Context into encryption methods * Fix tests * Fix tests * Fix tests * Trigger CI * Fix test * Add names to params of encryption service interface * Remove bus from CacheServiceImpl * Add logging * Add keys to logger Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com> * Add missing key to logger Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com> * Undo changes in markdown files * Fix formatting * Add context to secrets service * Rename decryptSecureJsonData to decryptSecureJsonDataFn * Name args in GetDecryptedValueFn * Add template back to NewAlertmanagerNotifier * Copy GetDecryptedValueFn to ngalert * Add logging to pluginsettings * Fix pluginsettings test Co-authored-by: Tania B <yalyna.ts@gmail.com> Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>	4 years ago
Tania B	62689ec804	Security: Add secrets service (#39418) ... * Add secrets service * Revert accidental changes in util encryption * Make minor changes Move functional options to models Revert renaming types to models * Add context * Minor change in GetDataKey * Use CreateDataKeyWithDBSession in CreateDataKey * Handle empty DEK name in DeleteDataKey * Rename defaultProvider * Remove secrets store service	4 years ago