107 Commits (JayEkin-patch-6)

Author	SHA1	Message	Date
owensmallwood	5e3e7cca0e	Search PoC: Adds traces to indexer (#95372) ... * adds traces to indexer * fix linting errors	9 months ago
owensmallwood	ecebaf1bdf	Unified Storage: Add traces to poller (#95369) ... add traces to poller	9 months ago
Diego Augusto Molina	257aca5340	Unistore Chore: Fix span name (#94965) ... fix span name	9 months ago
Claudiu Dragalina-Paraipan	830600dab0	AuthN: Optionally use tokens for unified storage client authentication (#91665) ... * extracted in-proc mode to #93124 * allow insecure conns in dev mode + refactoring * removed ModeCloud, relying on ModeGrpc and stackID instead to discover if we're running in Cloud * remove the NamespaceAuthorizer would fail in legacy mode. It will be added back in the future. * use FlagAppPlatformGrpcClientAuth to enable new behavior, instead of legacy * extracted authz package changes in #95120 * extracted server side changes in #95086 --------- Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com> Co-authored-by: gamab <gabriel.mabille@grafana.com> Co-authored-by: Dan Cech <dcech@grafana.com>	9 months ago
owensmallwood	2ffb88b0ee	Unified Storage: Add index for poller query (#95293) ... add index for poller query	9 months ago
Gabriel MABILLE	b68b69c2b4	AuthN: Use tokens for unified storage server authentication (#95086) ... * Extract server code --------- Co-authored-by: Claudiu Dragalina-Paraipan <drclau@users.noreply.github.com>	9 months ago
Leonor Oliveira	a03652494c	Dual Writer simplification (#93852) ... * All objects should have an UID * Now with a different error message * Simplify create on DW 2: use the same object to write to both storages * Run only one test * Add check for status code * Add name if it's not present in mode2 * Populate UID in legacy * Remove logs and commented code * Change dualwriter1 * Remove commented code * Fix list test * remove get on update from dualwriter 2 * Get object before updating. Better var renaming * Finish rebasing * Comment test * Uncomment tests * Update legacy first. Add preconditions * Remove preconditions * Fix update test * copy RV from unified to legacy objects * revert changes to playlist xorm store * Improve logging. Add go routines for mode3 * Add tests for async funcs in mode3 * Lint * Lint * Lint. Start to fix tests * Fix watcher tests * Fix store tests * Fiinish fixing watcher tests * Fix server tests * add name check * Update pkg/apiserver/rest/dualwriter_mode1.go Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com> * All objects should have an UID * Now with a different error message * Simplify create on DW 2: use the same object to write to both storages * Run only one test * Add check for status code * Add name if it's not present in mode2 * Populate UID in legacy * Remove logs and commented code * Change dualwriter1 * Remove commented code * Fix list test * remove get on update from dualwriter 2 * Get object before updating. Better var renaming * Finish rebasing * Comment test * Uncomment tests * Fix update test * revert changes to playlist xorm store * Improve logging. Add go routines for mode3 * Lint * Fix watcher tests * Fiinish fixing watcher tests * Add mode 5 with etcd test case. Add early check to fail on populated RV in payload * we can't set RV to the found object when updating * Lint * Don't fail on update playlists * Name should not be different when updating and it should be not empty on creating * Fix tests * Update pkg/apiserver/rest/dualwriter_mode2.go Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com> * Lint * Fix mode 5 tests * Lint * Add generateName condition on every mode. Fix tests * Lint * Add condition on where name or generate name have to be set * Fix test * Lint * Fix folders test * We dont need to send name for mode1 * Fail if UID is not present * Remove change from not running test * Remove unused line * Lint * Update pkg/storage/unified/apistore/store.go Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com> * Improve error message * Fix broken watcher test * Fail on name mismatch on update * Remove log * Make sure UIDs match on create in both stores * Lint * Write first to unified storage * Remove uid setting * Remove RV only in mode2 * Fix test. Remove log line * test * No need to asser on RV in mode3 * Remove RV check due to race condition * Update dualwriter.go Co-authored-by: Georges Chaudy <chaudyg@gmail.com> * Update pkg/storage/unified/client.go * remove unused parameter * log an error for object is missing UID instead of returning an error * remove obj.SetResourceVersion("") * log an error for object is missing UID instead of returning an error * FInalise merge * Move RV check to where it was * Remove name check * Remove server check for backwards compatibility * Remove unused fn * Move test checks for another PR * Dont commit go work sum changes * Only log error if RV is present for now. --------- Co-authored-by: Todd Treece <todd.treece@grafana.com> Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com> Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com> Co-authored-by: Georges Chaudy <chaudyg@gmail.com>	9 months ago
owensmallwood	d9bc4f7395	Search poc: Add debug log for metric and update metric buckets (#95217) ... * adds debug log for metric * updates buckets	9 months ago
owensmallwood	9763199b53	Search Poc: Add metrics (#95111)	9 months ago
Andrej Ocenas	5b1d99b8c4	Sidecar: Remove extension helpers and use the sidecar service directly (#94979)	9 months ago
owensmallwood	bda27ec8c6	Unified Storage: Add watch latency metric (#93509) ... * adds metric for watch latency * registers storage metrics when creating a new ResourceServer * defines the latency (in milliseconds) as the diff between now and the RV. Still need to wait until PR for switching RV to millisecond timestamp is rolled out. * should be micro seconds not milli * for watch latency, use diff between now and resource version and convert to seconds * fix typo	9 months ago
owensmallwood	fb9cfd0d1b	Search PoC: Indexmapping per resource and transform to intermediate IndexedResource (#94906) ... * Transforms raw US resource into an intermediate IndexableResource and indexes that. Pulls index mapping code out into different file. For now, we will hardcode which spec fields are indexed, per resource. * Fixes a few bugs with field casing and timestamps not being formatted right (or not existing). * adds readme section for using search with US * extracts to function to transform from search hit to IndexedResource * get folders when building index	9 months ago
Scott Lepper	5700b4b13a	[search] - restart watch (#94981) ... undo go sum change	9 months ago
Diego Augusto Molina	3e1f5559a6	Unistore Chore: Add database-level observability (#92266) ... * add testing harness * fix mockery and linters * WIP * wip * fix transactions * fix transaction tracing; add tracing by default * rename package * move WithTx to simplify logic of DB implementations * fix potential issue with context deadline * add db instrumentation to dbutil * add otel tests * improve naming * minor fix in semantics and add comprehensive OTel testing * fix naming * instrument resourceVersionAtomicInc * provide a default testing tracer * fix docs * fix typo in docs * add semconv for k8s	9 months ago
Diego Augusto Molina	cf08f6762d	Unistore Chore: Add OTEL testing harness (#94835) ... * add testing harness * fix mockery and linters * fix data race in tests * fix data race in tests * reduce cardinality of data	9 months ago
Ryan McKinley	3457f219be	Storage: Add blob storage interfaces (#90932) ... Co-authored-by: Jean-Philippe Quémémer <jeanphilippe.quemener@grafana.com>	9 months ago
Diego Augusto Molina	44282134da	Unistore Chore: Make it easier to implement DB interface (#94680) ... make it easier to implement DB interface	9 months ago
Prem Saraswat	e2672021bc	[unified-storage/apistore] Fix GuranteedUpdate skipping updates when `tryUpdate` is passed (#94557) ... `GuranteedUpdate` method of `apistore.Storage` had a bug, where it would errorneously conclude that the object is unchanged, in case a `tryUpdate` function is passed that modifies the existing object itself (as it is the case in many core types in K8s upstream). The modified `existingObj` was compared with `updatedObj`, which would essentially be same and this lead to the update being skipped. This patch fixes this by always passing a copy of the `existingObj`. Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>	9 months ago
Georges Chaudy	d999b415df	UniStore: Use epoch with microsecond resolution as RV (#92638) ... * Use epoch with microsecond resolution as RV * fix backend tests * Add solution for when the clock goes back * Add solution for when the clock goes back * generate mocks * go lint * remove comment * Use Greatest instead of max in msyql and postgres * update tests * Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com> * cast to bigint * add additional round trip * increment the RV using 2 sql round trips instead of 3 * cleanup comments * cast unix timestamp to integer * fix postgres query * remove old increment test data * remove greatest * cast unix_timestamp to signed * Use statement_timestamp instead of clock_timestamp --------- Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>	9 months ago
owensmallwood	3d59d3b40f	Search poc: Initial index paginates list response (#94582)	9 months ago
owensmallwood	d96baaa878	Search PoC: Add logging (#94567)	9 months ago
Prem Saraswat	5c03c14b25	resource-api: Loosen name validation to match K8s requirements (#93404) ... * resource-api: Loosen name validation to match K8s requirements This patch modifies some of the requirements for name validation of objects in Resource API to match Kubernetes. The limit we have on characters in name is 64, but some resources allow upto 253 characters. Similarly we also include `:` in the regex, as many objects in default K8s setup use it in the name (the group `system:masters` for example) Signed-off-by: Prem Kumar <prem.saraswat@grafana.com> * Update the name column length in migrator and update e2e test to verify --------- Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>	9 months ago
Diego Augusto Molina	4a800eda9f	Unistore Chore: add inocuous preparative changes for otel-based db observability (#94473) ... add inocuous preparative changes for otel tracing	9 months ago
owensmallwood	612b864772	Indexing PoC: Add search/browse (#94126) ... * adds Filter gRPC and make protobuf * adds route for querying the filter gRPC * wires up Filter gRPC call * [WIP] index from start * renames gRPC endpoint to "Search" * adds /apis/search route into k8s routes. Hacky for now. * updates readme - wrong casing * adds feature toggle for unified storage search * hides US search behind feature flag. Clean up print statements. * removes indexer - will be added in another PR * Search: Add API Builder * adds required method * implementing UpdateAPIGroupInfo (WIP) * adds groupversion * commenting out for now * remove unneeded code from experimenting and update register.go to match interface required * list resources and load into index * pass context * namespaces search route * lint * watch * add todo * add todo * merge * cleanup * add todo * gen protobuf * lint; fix migration issue * Updates index mapping function to map unified storage object Value * Changes Index() to pointer receiver - fixes panic * add delete * cleanup * gets search/browse functioning. Results show up as base64 encoded. Still a WIP. * Doesnt json re-encode gRPC response in search handler * add kind to SearchRequest proto * Updates query interface to be more generic. Make proto. Parses query params in api server. * make protobuf * removes unused method and imports * Returns all indexed fields in search results. Adds pagination support (limit + offset). * remove comment * remove unused struct * gets tenant in search k8s api handler * adds hardcoded spec field mappings - starting with playlists * adds all spec fields to search results * moved helper function for field mappings into index * only includes allowed spec fields in search results * cleans up error handling * removes debug log --------- Co-authored-by: leonorfmartins <leonorfmartins@gmail.com> Co-authored-by: Todd Treece <todd.treece@grafana.com> Co-authored-by: Scott Lepper <scott.lepper@gmail.com>	9 months ago
Scott Lepper	a3764ebeba	[Search] fix: add and delete (#94438) ... [search] fix: add and delete from index	9 months ago
Scott Lepper	c2fb2dcfbe	wire up unified search from the ui; add basic search support (#94358) ... * wire up search from the ui; add basic search support	9 months ago
Scott Lepper	52b70ca976	Grafana indexing poc - load and watch index (#93843) ... * list resources and load into index * watch for changes and update index	10 months ago
Arati R.	ea5cf7c51f	Unified Storage /Folders: Allow Unified Storage subfolders creation (#94327) ... * Add parents field to folder DTO * Allow subfolder creation when folder flag is enabled * Update UnstructuredToLegacyFolder * Include parents field when creating folder	10 months ago
Matheus Macabu	4bc7a35f56	UnifiedStorage: fix watcher test (#94301)	10 months ago
Georges Chaudy	03f55e5288	Unistore : Ensure Watch works in HA mode (#94120) ... * Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)" This reverts commit 7c3fc2f261. * make previous_resource_version nullable * handle nil case	10 months ago
Diego Augusto Molina	7d32d5eff4	Unistore: Reuse MySQL and Postgres Grafana core config instead of the object (#94223) ... * Reuse MySQL and Postgres Grafana config instead of the object - Only reuse the Grafana DB object for SQLite. Support for SQLite will be added in a different PR - Fail when reusing the Grafana DB object if it is using DB instrumentation - In the case that we have to reuse a Grafana DB with its instrumentation, fail with an error that describes a workaround - Add regression tests to reproduce incident 2144 * remove temp file * fix linter * fix linter x2 * fix linter x3	10 months ago
Todd Treece	a4d919c157	Chore: Update k8s.io dependencies to v0.31.1 (#93696)	10 months ago
Georges Chaudy	7c3fc2f261	Revert "Unistore : Ensure Watch works in HA mode." (#94097) ... Revert "Unistore : Ensure Watch works in HA mode. (#93428)" This reverts commit 0a26c9e9ae.	10 months ago
owensmallwood	6a3eb276ef	Grafana Indexing PoC: Adds feature flag and gRPC endpoint (#93356) ... * adds Filter gRPC and make protobuf * adds route for querying the filter gRPC * wires up Filter gRPC call * [WIP] index from start * renames gRPC endpoint to "Search" * adds /apis/search route into k8s routes. Hacky for now. * updates readme - wrong casing * adds feature toggle for unified storage search * hides US search behind feature flag. Clean up print statements. * removes indexer - will be added in another PR * Search: Add API Builder * adds required method * implementing UpdateAPIGroupInfo (WIP) * adds groupversion * commenting out for now * remove unneeded code from experimenting and update register.go to match interface required * namespaces search route --------- Co-authored-by: leonorfmartins <leonorfmartins@gmail.com> Co-authored-by: Todd Treece <todd.treece@grafana.com>	10 months ago
Georges Chaudy	0a26c9e9ae	Unistore : Ensure Watch works in HA mode. (#93428) ... * Replace Watch with WatchNext * remove watchset * fix previous page and closing the channel * Remove the broadcaster cache to prevent dupplicated events * add watch bookmark * add watch bookmark * cleanup comments * disable the tests for bookmarks for now * Ensure we send previosu events * lint * re-introduce the cache * load from cache * disabling legacy test * disabling legacy test * Update pkg/storage/unified/resource/server.go Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com> * Could not read previous events * add proper migration * Add previous_resource_version to both history and resource * First event should have an RV of 2 and not 1 * Test both storage backends * fix the inital RV for the sql backend * ensure graceful stop of the stream decoder * gocyclo --------- Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>	10 months ago
Ryan McKinley	cdbc04ab2b	GRPC/Authn: Remove org name header (#93780)	10 months ago
Claudiu Dragalina-Paraipan	a8b07b0c81	[authn] use authlib client+interceptors for in-proc mode (#93124) ... * Add authlib gRPC authenticators for in-proc mode * implement `StaticRequester` signing in the unified resource client - [x] when the `claims.AuthInfo` value type is `identity.StaticRequester`, and there's no ID token set, create an internal token and sign it with symmetrical key. This is a workaround for `go-jose` not offering the possibility to create an unsigned token. - [x] update `IDClaimsWrapper` to support the scenario above - [x] Switch to using `claims.From()` in `dashboardSqlAccess.SaveDashboard()` --------- Co-authored-by: gamab <gabriel.mabille@grafana.com>	10 months ago
owensmallwood	8bbfbc61be	Unified Storage: Adds a few readme updates from issues I ran into with local dev (#93553) ... adds a few readme updates from issues I ran into with local dev	10 months ago
Georges Chaudy	216b63549d	unistore: fix watch events forever looping (#93517)	10 months ago
Andres Martinez Gotor	74b2d38e37	Chore: Update grafana-plugin-sdk (#93426)	10 months ago
Ryan McKinley	542105b680	ResourceClient: Exercise resource client in k8s apis tests (#93473)	10 months ago
Todd Treece	d1ffcc22d9	Playlists: Migrate to App SDK codegen (#93246)	10 months ago
Ryan McKinley	45eb72e95a	K8s: Use wire to initalize the resource client (#93221)	10 months ago
Ryan McKinley	7efadb0a00	Testing: Integration tests for unified-grpc start a local gRPC server (#93201)	10 months ago
Arati R.	b12a29a1da	K8s: Implement partial folders api with k8s client (#93089) ... * Add kubernetes folder feature toggle * Add kubernetes routes for getting and creating a folder * Add documentation for interacting with k8s folders	10 months ago
Claudiu Dragalina-Paraipan	3aeb8d390e	[authn] update `authlib` and `claims` versions (#93098) ... * update authlib version to latest * make update-workspace -- 2nd run * manual cleanup of old version	10 months ago
ismail simsek	6548ea377d	Chore: Bump grafana-plugin-sdk-go version to v0.247.0 (#93095) ... * Bump grafana-plugin-sdk-go version to v0.247.0 * make update-workspace * make update-workspace	10 months ago
Dave Henderson	e1090db5d9	Chore: Bump Go to 1.23.1 (#93007) ... Signed-off-by: Dave Henderson <dave.henderson@grafana.com>	11 months ago
Ryan McKinley	3cf84d1921	Chore: update centrifuge library (#93020)	11 months ago
Claudiu Dragalina-Paraipan	1ce43b4c42	[authn] update `authlib` and `claims` versions (#92987) ... * update authlib and claims versions * 2nd make update-workspace * manual fix	11 months ago