The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
grafana/pkg/services/secrets/kvstore/sql_test.go

305 lines
6.7 KiB

Secrets: Implement basic unified secret store service (#45804) * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072e9aff601fb9d639127d468c03f97ef. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test
3 years ago
package kvstore
import (
"context"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana/pkg/infra/db"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/services/secrets/fakes"
"github.com/grafana/grafana/pkg/services/secrets/manager"
Secrets: Implement basic unified secret store service (#45804) * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072e9aff601fb9d639127d468c03f97ef. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test
3 years ago
)
type TestCase struct {
OrgId int64
Namespace string
Type string
Revision int64
}
func (t *TestCase) Value() string {
return fmt.Sprintf("%d:%s:%s:%d", t.OrgId, t.Namespace, t.Type, t.Revision)
}
func TestSecretsKVStoreSQL(t *testing.T) {
sqlStore := db.InitTestDB(t)
secretsService := manager.SetupTestService(t, fakes.NewFakeSecretsStore())
kv := NewSQLSecretsKVStore(sqlStore, secretsService, log.New("test.logger"))
Secrets: Implement basic unified secret store service (#45804) * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072e9aff601fb9d639127d468c03f97ef. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test
3 years ago
ctx := context.Background()
testCases := []*TestCase{
{
OrgId: 0,
Namespace: "namespace1",
Type: "testing1",
},
{
OrgId: 0,
Namespace: "namespace2",
Type: "testing2",
},
{
OrgId: 1,
Namespace: "namespace1",
Type: "testing1",
},
{
OrgId: 1,
Namespace: "namespace3",
Type: "testing3",
},
}
for _, tc := range testCases {
err := kv.Set(ctx, tc.OrgId, tc.Namespace, tc.Type, tc.Value())
require.NoError(t, err)
}
t.Run("get existing keys", func(t *testing.T) {
for _, tc := range testCases {
value, ok, err := kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.NoError(t, err)
require.True(t, ok)
require.Equal(t, tc.Value(), value)
}
})
t.Run("get nonexistent keys", func(t *testing.T) {
tcs := []*TestCase{
{
OrgId: 0,
Namespace: "namespace3",
Type: "testing3",
},
{
OrgId: 1,
Namespace: "namespace2",
Type: "testing2",
},
{
OrgId: 2,
Namespace: "namespace1",
Type: "testing1",
},
}
for _, tc := range tcs {
value, ok, err := kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.Nil(t, err)
require.False(t, ok)
require.Equal(t, "", value)
}
})
t.Run("modify existing key", func(t *testing.T) {
tc := testCases[0]
value, ok, err := kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.NoError(t, err)
require.True(t, ok)
assert.Equal(t, tc.Value(), value)
tc.Revision += 1
err = kv.Set(ctx, tc.OrgId, tc.Namespace, tc.Type, tc.Value())
require.NoError(t, err)
value, ok, err = kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.NoError(t, err)
require.True(t, ok)
assert.Equal(t, tc.Value(), value)
})
t.Run("use fixed client", func(t *testing.T) {
tc := testCases[0]
client := With(kv, tc.OrgId, tc.Namespace, tc.Type)
fmt.Println(client.Namespace, client.OrgId, client.Type)
value, ok, err := client.Get(ctx)
require.NoError(t, err)
require.True(t, ok)
require.Equal(t, tc.Value(), value)
tc.Revision += 1
err = client.Set(ctx, tc.Value())
require.NoError(t, err)
value, ok, err = client.Get(ctx)
require.NoError(t, err)
require.True(t, ok)
assert.Equal(t, tc.Value(), value)
})
t.Run("deleting keys", func(t *testing.T) {
var stillHasKeys bool
for _, tc := range testCases {
if _, ok, err := kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type); err == nil && ok {
stillHasKeys = true
break
}
}
require.True(t, stillHasKeys,
"we are going to test key deletion, but there are no keys to delete in the database")
for _, tc := range testCases {
err := kv.Del(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.NoError(t, err)
}
for _, tc := range testCases {
_, ok, err := kv.Get(ctx, tc.OrgId, tc.Namespace, tc.Type)
require.NoError(t, err)
require.False(t, ok, "all keys should be deleted at this point")
}
})
t.Run("listing existing keys", func(t *testing.T) {
sqlStore := db.InitTestDB(t)
secretsService := manager.SetupTestService(t, fakes.NewFakeSecretsStore())
kv := NewSQLSecretsKVStore(sqlStore, secretsService, log.New("test.logger"))
Secrets: Implement basic unified secret store service (#45804) * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072e9aff601fb9d639127d468c03f97ef. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test
3 years ago
ctx := context.Background()
namespace, typ := "listtest", "listtest"
testCases := []*TestCase{
{
OrgId: 1,
Type: typ,
Namespace: namespace,
},
{
OrgId: 2,
Type: typ,
Namespace: namespace,
},
{
OrgId: 3,
Type: typ,
Namespace: namespace,
},
{
OrgId: 4,
Type: typ,
Namespace: namespace,
},
{
OrgId: 1,
Type: typ,
Namespace: "other_key",
},
{
OrgId: 4,
Type: typ,
Namespace: "another_one",
},
}
for _, tc := range testCases {
err := kv.Set(ctx, tc.OrgId, tc.Namespace, tc.Type, tc.Value())
require.NoError(t, err)
}
keys, err := kv.Keys(ctx, AllOrganizations, namespace, typ)
require.NoError(t, err)
require.Len(t, keys, 4)
found := 0
for _, key := range keys {
for _, tc := range testCases {
if key.OrgId == tc.OrgId && key.Namespace == tc.Namespace && key.Type == tc.Type {
found++
break
}
}
}
require.Equal(t, 4, found, "querying for all orgs should return 4 records")
keys, err = kv.Keys(ctx, 1, namespace, typ)
require.NoError(t, err)
require.Len(t, keys, 1, "querying for a specific org should return 1 record")
keys, err = kv.Keys(ctx, AllOrganizations, "not_existing_namespace", "not_existing_type")
require.NoError(t, err, "querying a not existing namespace should not throw an error")
require.Len(t, keys, 0, "querying a not existing namespace should return an empty slice")
})
t.Run("getting all secrets", func(t *testing.T) {
sqlStore := db.InitTestDB(t)
secretsService := manager.SetupTestService(t, fakes.NewFakeSecretsStore())
kv := NewSQLSecretsKVStore(sqlStore, secretsService, log.New("test.logger"))
ctx := context.Background()
namespace, typ := "listtest", "listtest"
testCases := []*TestCase{
{
OrgId: 1,
Type: typ,
Namespace: namespace,
},
{
OrgId: 2,
Type: typ,
Namespace: namespace,
},
{
OrgId: 3,
Type: typ,
Namespace: namespace,
},
{
OrgId: 4,
Type: typ,
Namespace: namespace,
},
{
OrgId: 1,
Type: typ,
Namespace: "other_key",
},
{
OrgId: 4,
Type: typ,
Namespace: "another_one",
},
}
for _, tc := range testCases {
err := kv.Set(ctx, tc.OrgId, tc.Namespace, tc.Type, tc.Value())
require.NoError(t, err)
}
secrets, err := kv.GetAll(ctx)
require.NoError(t, err)
require.Len(t, secrets, 6)
found := 0
for _, s := range secrets {
for _, tc := range testCases {
if *s.OrgId == tc.OrgId &&
*s.Namespace == tc.Namespace &&
*s.Type == tc.Type {
require.Equal(t, tc.Value(), s.Value, "secret found but value is not equals")
found++
break
}
}
}
require.Equal(t, 6, found, "querying for all secrets should return 6 records")
})
Secrets: Implement basic unified secret store service (#45804) * wip: Implement kvstore for secrets * wip: Refactor kvstore for secrets * wip: Add format key function to secrets kvstore sql * wip: Add migration for secrets kvstore * Remove unused Key field from secrets kvstore * Remove secret values from debug logs * Integrate unified secrets with datasources * Fix minor issues and tests for kvstore * Create test service helper for secret store * Remove encryption tests from datasources * Move secret operations after datasources * Fix datasource proxy tests * Fix legacy data tests * Add Name to all delete data source commands * Implement decryption cache on sql secret store * Fix minor issue with cache and tests * Use secret type on secret store datasource operations * Add comments to make create and update clear * Rename itemFound variable to isFound * Improve secret deletion and cache management * Add base64 encoding to sql secret store * Move secret retrieval to decrypted values function * Refactor decrypt secure json data functions * Fix expr tests * Fix datasource tests * Fix plugin proxy tests * Fix query tests * Fix metrics api tests * Remove unused fake secrets service from query tests * Add rename function to secret store * Add check for error renaming secret * Remove bus from tests to fix merge conflicts * Add background secrets migration to datasources * Get datasource secure json fields from secrets * Move migration to secret store * Revert "Move migration to secret store" This reverts commit 7c3f872072e9aff601fb9d639127d468c03f97ef. * Add secret service to datasource service on tests * Fix datasource tests * Remove merge conflict on wire * Add ctx to data source http transport on prometheus stats collector * Add ctx to data source http transport on stats collector test
3 years ago
}