31 Commits (75b0788377d95e2079bce5dfd7598d4dba00c98a)

Author	SHA1	Message	Date
Will Browne	75b0788377	Plugins: Remove registry dependency from process manager (#73241) ... simplify	2 years ago
Will Browne	72da44db0e	Plugins: Add validation stage to plugin loader pipeline (#73053) ... * first pass * change validation signature * err tracking * fix * undo golden * 1 more * fix * adjust doc * add test helper * fix linter	2 years ago
Will Browne	60b4a0b2a4	Plugins: Add termination stage to plugin loader pipeline (#72822) ... * add termination stage * uid -> pluginID (for now) * also fix fakes * add simple test * Fix logger name Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com> * inline stop func call Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com> --------- Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>	2 years ago
Will Browne	ad2705fa0b	Plugins: Add inititialization stage to plugin loader pipeline (#72667) ... * first pass * migrate tests * simplify * fix comments * fix linter * nil checks * remove comment	2 years ago
Will Browne	758d9884bc	Plugins: Plugins loader pipeline (#71438) ... * discovery * flesh out * add docs * remove unused func * bootstrap stage * fix docs * update docs * undo unnecessary changes * add end tag * update doc * fix linter * fix * tidy * update docs * add class to filter func * apply PR feedback * fix test	2 years ago
Isabella Siu	56913fbd95	Redshift: Support caching async aws queries (#71682) ... Co-authored-by: Sarah Zinger <sarah.zinger@grafana.com>	2 years ago
Giuseppe Guerra	a8d2a9ae2b	Plugins: Angular detector: Add database cache store for remote patterns (#70693) ... * Plugins: Angular detector: Remote patterns fetching * Renamed PatternType to GCOMPatternType * Renamed files * Renamed more files * Moved files again * Add type checks, unexport GCOM structs * Cache failures, update log messages, fix GCOM URL * Fail silently for unknown pattern types, update docstrings * Fix tests * Rename gcomPattern.Value to gcomPattern.Pattern * Refactoring * Add FlagPluginsRemoteAngularDetectionPatterns feature flag * Fix tests * Re-generate feature flags * Add TestProvideInspector, renamed TestDefaultStaticDetectorsInspector * Add TestProvideInspector * Add TestContainsBytesDetector and TestRegexDetector * Renamed getter to provider * More tests * TestStaticDetectorsProvider, TestSequenceDetectorsProvider * GCOM tests * Lint * Made detector.detect unexported, updated docstrings * Allow changing grafana.com URL * Fix API path, add more logs * Update tryUpdateRemoteDetectors docstring * Use angulardetector http client * Return false, nil if module.js does not exist * Chore: Split angualrdetector into angularinspector and angulardetector packages Moved files around, changed references and fixed tests: - Split the old angulardetector package into angular/angulardetector and angular/angularinspector - angulardetector provides the detection structs/interfaces (Detector, DetectorsProvider...) - angularinspector provides the actual angular detection service used directly in pluginsintegration - Exported most of the stuff that was private and now put into angulardetector, as it is not required by angularinspector * Renamed detector.go -> angulardetector.go and inspector.go -> angularinspector.go Forgot to rename those two files to match the package's names * Renamed angularinspector.ProvideInspector to angularinspector.ProvideService * Renamed "harcoded" to "static" and "remote" to "dynamic" from PR review, matches the same naming schema used for signing keys fetching * WIP: Angular: cache patterns in db, moved gcom into pluginsintegration More similar to signing keys fetching * Rename package, refactoring * try to solve circular import * Fix merge conflict on updated angular patterns * Fix circular imports * Fix wire gen * Add docstrings, refactoring * Removed angualrdetectorsprovider dependency into angularpatternsstore * Moved GCOM test files * Removed GCOM cache * Renamed Detect to DetectAngular and Detector to AngularDetector * Fix call to NewGCOMDetectorsProvider in newDynamicInspector * Removed unused test function newError500GCOMScenario * Added angularinspector service definition in pluginsintegration * refactoring * lint * Fix angularinspector TestProvideService * cleanup * Await initial restore * Register dynamicAngularDetector background service * Removed static detectors provider from pluginsintegration * Add tests for kvstore * Add more tests * order imports in dynamic_test.go * Fix potential panic in dynamic_test * Add "runs the job periodically" test * lint * add timeout to test * refactoring * Removed context.Context from DetectorsProvider * Refactoring, ensure angular dynamic background service is not started if feature flag is off * Fix deadlock on startup * Fix angulardetectorsprovider tests * Revert "Removed context.Context from DetectorsProvider" This reverts commit 4e8c6dded7. * Fix wrong argument number in dynamic_teset * Standardize gcom http client * Reduce context timeout for angular inspector in plugins loader * Simplify initial restore logic * Fix dynamic detectors provider tests * Chore: removed angulardetector/provider.go * Add more tests * Removed backgroundJob interface, PR review feedback * Update tests * PR review feedback: remove ErrNoCachedValue from kv store Get * Update tests * PR review feedback: add IsDisabled and remove nop background srevice * Update tests * Remove initialRestore channel, use mux instead * Removed backgroundJobInterval, use package-level variable instead * Add TestDynamicAngularDetectorsProviderBackgroundService * Removed timeouts * pr review feedback: restore from store before returning the service * Update tests * Log duration on startup restore and cron run * Switch cron job start log to debug level * Do not attempt to restore if disabled	2 years ago
Andres Martinez Gotor	4ff0abd0d1	SaveExternalService (OAuth) on plugin load (#69764)	2 years ago
Giuseppe Guerra	cca9d89733	Plugins: Angular detector: Remote patterns fetching (#69843) ... * Plugins: Angular detector: Remote patterns fetching * Renamed PatternType to GCOMPatternType * Renamed files * Renamed more files * Moved files again * Add type checks, unexport GCOM structs * Cache failures, update log messages, fix GCOM URL * Fail silently for unknown pattern types, update docstrings * Fix tests * Rename gcomPattern.Value to gcomPattern.Pattern * Refactoring * Add FlagPluginsRemoteAngularDetectionPatterns feature flag * Fix tests * Re-generate feature flags * Add TestProvideInspector, renamed TestDefaultStaticDetectorsInspector * Add TestProvideInspector * Add TestContainsBytesDetector and TestRegexDetector * Renamed getter to provider * More tests * TestStaticDetectorsProvider, TestSequenceDetectorsProvider * GCOM tests * Lint * Made detector.detect unexported, updated docstrings * Allow changing grafana.com URL * Fix API path, add more logs * Update tryUpdateRemoteDetectors docstring * Use angulardetector http client * Return false, nil if module.js does not exist * Chore: Split angualrdetector into angularinspector and angulardetector packages Moved files around, changed references and fixed tests: - Split the old angulardetector package into angular/angulardetector and angular/angularinspector - angulardetector provides the detection structs/interfaces (Detector, DetectorsProvider...) - angularinspector provides the actual angular detection service used directly in pluginsintegration - Exported most of the stuff that was private and now put into angulardetector, as it is not required by angularinspector * Renamed detector.go -> angulardetector.go and inspector.go -> angularinspector.go Forgot to rename those two files to match the package's names * Renamed angularinspector.ProvideInspector to angularinspector.ProvideService * Renamed "harcoded" to "static" and "remote" to "dynamic" from PR review, matches the same naming schema used for signing keys fetching * Fix merge conflict on updated angular patterns * Removed GCOM cache * Renamed Detect to DetectAngular and Detector to AngularDetector * Fix call to NewGCOMDetectorsProvider in newDynamicInspector * Removed unused test function newError500GCOMScenario * Added angularinspector service definition in pluginsintegration * Moved dynamic inspector into pluginsintegration * Move gcom angulardetectorsprovider into pluginsintegration * Log errUnknownPatternType at debug level * re-generate feature flags * fix error log	2 years ago
Giuseppe Guerra	f148b5fb28	Plugins: Forbid loading Angular plugins when Angular is disabled (#69679) ... * Plugins: Forbid loading Angular plugins when Angular is disabled * Plugins: Made angulardetector a service, add tests for angular loader cases * Fix missing import * Add nolint:gocyclo to loadPlugins	2 years ago
Giuseppe Guerra	7a132680ef	Revert "Plugins: Forbid loading Angular plugins when Angular is disabled (#69225)" (#69657) ... This reverts commit ff34279ff4.	2 years ago
Giuseppe Guerra	ff34279ff4	Plugins: Forbid loading Angular plugins when Angular is disabled (#69225) ... * Plugins: Forbid loading Angular plugins when Angular is disabled * Plugins: Made angulardetector a service, add tests for angular loader cases * Fix missing import	2 years ago
Andres Martinez Gotor	aa9838bd25	Chore: Refactor manifest verifier (#67218)	2 years ago
Giuseppe Guerra	bfe1b68800	Plugins: Fs: Add option to access unallowed files in dev mode (#66492) ... * Plugins: Fs: Add option to access unallowed files in dev mode * Plugins: Fs: allow accessing unallowed files only when in dev mode * Plugins: Fs: Add ProvideLocalFinder * Plugins: FS: Pass whole config in NewLocalFinder() * Plugins: FS: Add AllowListLocalFS * Plugins: FS: Fix some tests * Plugins: FS: Update tests * Plugins: FS: Removed dead code * Plugins: FS: Add tests for AllowListFS * Plugins: FS: Update comments * Plugins: FS: Use variadic arguments for allow list rather than map * Plugins: FS: Remove unnecessary log * Plugins: FS: Do not escape plugin root dir * Fix merge conflict * Plugins: FS: Update comments * Plugins: FS: PR review changes * Fix merge conflict * Fix tests * Cleanup * Fix flaky test * Changes from PR review * Lint * Add comment to LocalFS.Remove * Fix Windows * Renamed devMode to production	2 years ago
Marcus Efraimsson	73920b1e34	Plugins: Refactor cleaning of call resource response headers (#67145) ... First part of #66889 moving cleaning of call resource response headers within plugin management client.	2 years ago
Andres Martinez Gotor	9d7c3a101d	Chore: Use KVStore for the manifest public key (#66839)	2 years ago
Andres Martinez Gotor	98c695c68f	Chore: Use Grafana API to retrieve the public key to validate plugins (#66439)	2 years ago
Michael Mandrus	5626461b3c	Caching: Refactor enterprise query caching middleware to a wire service (#65616) ... * define initial service and add to wire * update caching service interface * add skipQueryCache header handler and update metrics query function to use it * add caching service as a dependency to query service * working caching impl * propagate cache status to frontend in response * beginning of improvements suggested by Lean - separate caching logic from query logic. * more changes to simplify query function * Decided to revert renaming of function * Remove error status from cache request * add extra documentation * Move query caching duration metric to query package * add a little bit of documentation * wip: convert resource caching * Change return type of query service QueryData to a QueryDataResponse with Headers * update codeowners * change X-Cache value to const * use resource caching in endpoint handlers * write resource headers to response even if it's not a cache hit * fix panic caused by lack of nil check * update unit test * remove NONE header - shouldn't show up in OSS * Convert everything to use the plugin middleware * revert a few more things * clean up unused vars * start reverting resource caching, start to implement in plugin middleware * revert more, fix typo * Update caching interfaces - resource caching now has a separate cache method * continue wiring up new resource caching conventions - still in progress * add more safety to implementation * remove some unused objects * remove some code that I left in by accident * add some comments, fix codeowners, fix duplicate registration * fix source of panic in resource middleware * Update client decorator test to provide an empty response object * create tests for caching middleware * fix unit test * Update pkg/services/caching/service.go Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com> * improve error message in error log * quick docs update * Remove use of mockery. Update return signature to return an explicit hit/miss bool * create unit test for empty request context * rename caching metrics to make it clear they pertain to caching * Update pkg/services/pluginsintegration/clientmiddleware/caching_middleware.go Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> * Add clarifying comments to cache skip middleware func * Add comment pointing to the resource cache update call * fix unit tests (missing dependency) * try to fix mystery syntax error * fix a panic * Caching: Introduce feature toggle to caching service refactor (#66323) * introduce new feature toggle * hide calls to new service behind a feature flag * remove licensing flag from toggle (misunderstood what it was for) * fix unit tests * rerun toggle gen --------- Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com> Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>	2 years ago
Will Browne	bff9f4c890	Plugins: Move config factory to pluginsintegration package (#65716) ... * move config to pluginsintegration package * change to all plugin toggle * fixes * fixes * fix lerna * fix test	2 years ago
Will Browne	7bbe255150	Plugins: Add file store abstraction for handling plugin files (#65432) ... * add file store * fix markdown fetch bug * add markdown tests * fix var name	2 years ago
Giuseppe Guerra	a89202eab2	Plugins: Improve instrumentation by adding metrics and tracing (#61035) ... * WIP: Plugins tracing * Trace ID middleware * Add prometheus metrics and tracing to plugins updater * Add TODOs * Add instrumented http client * Add tracing to grafana update checker * Goimports * Moved plugins tracing to middleware * goimports, fix tests * Removed X-Trace-Id header * Fix comment in NewTracingHeaderMiddleware * Add metrics to instrumented http client * Add instrumented http client options * Removed unused function * Switch to contextual logger * Refactoring, fix tests * Moved InstrumentedHTTPClient and PrometheusMetrics to their own package * Tracing middleware: handle errors * Report span status codes when recording errors * Add tests for tracing middleware * Moved fakeSpan and fakeTracer to pkg/infra/tracing * Add TestHTTPClientTracing * Lint * Changes after PR review * Tests: Made "ended" in FakeSpan private, allow calling End only once * Testing: panic in FakeSpan if span already ended * Refactoring: Simplify Grafana updater checks * Refactoring: Simplify plugins updater error checks and logs * Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates * Tests: Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates * Log update checks duration, use Info log level for check succeeded logs * Add plugin context span attributes in tracing_middleware * Refactor prometheus metrics as httpclient middleware * Fix call to ProvidePluginsService in plugins_test.go * Propagate context to update checker outgoing http requests * Plugin client tracing middleware: Removed operation name in status * Fix tests * Goimports tracing_middleware.go * Goimports * Fix imports * Changed span name to plugins client middleware * Add span name assertion in TestTracingMiddleware * Removed Prometheus metrics middleware from grafana and plugins updatechecker * Add span attributes for ds name, type, uid, panel and dashboard ids * Fix http header reading in tracing middlewares * Use contexthandler.FromContext, add X-Query-Group-Id * Add test for RunStream * Fix imports * Changes from PR review * TestTracingMiddleware: Changed assert to require for didPanic assertion * Lint * Fix imports	2 years ago
Will Browne	31d6416157	Plugins: Migrate licensing and access control to pkg/services/pluginsintegration package (#65258) ... * migrate licensing + access control * update package name	2 years ago
Will Browne	ee2dd62a1f	Plugins: Refactor loader + finder to support multiple sourcing methods (#64735) ... * it's cdn time * tidy body closing * auto signed * fix close * update log name * remove comments	2 years ago
Will Browne	94f39e69a3	Plugins: Migrate `plugincontext`, `adapters` and `pluginsettings` to pkg/services/pluginsintegration package (#64154) ... * migrate plugincontext, adapter and pluginsettings * add to CODEOWNERS * fix imports * fix CODEOWNERS * take pluginsettings * migrate wire stuff	2 years ago
Will Browne	ab8de1a0e3	Plugins: Add simple plugin sources service (#63814) ... add simple plugin sources svc	2 years ago
Giuseppe Guerra	af1e2d68da	Plugins: Allow loading panel plugins from a CDN (#59096) ... * POC: Plugins CDN reverse proxy * CDN proxy POC: changed env var names * Add authorization: false for /public path in frontend plugin loader * Moved CDN settings to Cfg, add some comments * Fix error 500 in asset fetch if plugin is not using CDN * Fix EnterpriseLicensePath declared twice * Fix linter complaining about whitespaces * Plugins CDN: Skip signature verification for CDN plugins * Plugins CDN: Skip manifest and signature check for cdn plugins * Plugins: use IsValid() and IsInternal() rather than equality checks * Plugins CDN: remove comment * Plugins CDN: Fix seeker can't seek when serving plugins from local fs * Plugins CDN: add back error codes in getLocalPluginAssets * Plugins CDN: call asset.Close() rather than asset.readSeekCloser.Close() * Plugins CDN: Fix panic in JsonApiErr when errorMessageCoder wraps a nil error * Plugins CDN: Add error handling to proxyCDNPluginAsset * Plugins CDN: replace errorMessageCoder with errutil * Plugins CDN POC: expose cdn plugin paths to frontend for system.js * Plugins CDN: Fix cdn plugins showing as unsigned in frontend * WIP: Add support for formatted URL * Fix missing cdnPluginsBaseURLs in GrafanaConfig * Plugins CDN: Remove reverse proxy mode and reverse proxy references * Plugins CDN: Simplify asset serving logic * Plugins CDN: sanitize redirect path * Plugins CDN: Removed unused pluginAsset type * Plugins CDN: Removed system.js changes * Plugins CDN: Return different system.js baseURL and module for cdn plugins * Plugins CDN: Ensure CDN is disabled for non-external plugins * lint * Plugins CDN: serve images and screenshots from CDN, refactoring * Lint * Plugins CDN: Fix URLs for system.js (baseUrl and module) * Plugins CDN: Add more tests for RelativeURLForSystemJS * Plugins CDN: Iterate only on apps when preloading * Plugins CDN: Refactoring * Plugins CDN: Add comments to url_constructor.go * Plugins CDN: Update defaultHGPluginsCDNBaseURL * Plugins CDN: undo extract meta from system js config * refactor(plugins): migrate systemjs css plugin to typescript * feat(plugins): introduce systemjs cdn loader plugin * feat(plugins): add systemjs load type * Plugins CDN: Removed RelativeURLForSystemJS * Plugins CDN: Log backend redirect hits along with plugin info * Plugins CDN: Add pluginsCDNBasePath to getFrontendSettingsMap * feat(plugins): introduce cdn loading for angular plugins * refactor(plugins): move systemjs cache buster into systemjsplugins directory * Plugins CDN: Rename pluginsCDNBasePath to pluginsCDNBaseURL * refactor(plugins): introduce pluginsCDNBaseURL to the frontend * Plugins CDN: Renamed "cdn base path" to "cdn url template" in backend * Plugins CDN: lint * merge with main * Instrumentation: Add prometheus counter for backend hits, log from Info to Warn * Config: Changed key from plugins_cdn.url to plugins.plugins_cdn_base_url * CDN: Add backend tests * Lint: goimports * Default CDN URL to empty string, * Do not use CDN in setImages and module if the url template is empty * CDN: Backend: Add test for frontend settings * CDN: Do not log missing module.js warn if plugin is being loaded from CDN * CDN: Add backend test for CDN plugin loader * Removed 'cdn' signature level, switch to 'valid' * Fix pfs.TestParseTreeTestdata for cdn plugin testdata dir * Fix TestLoader_Load * Fix gocyclo complexity of loadPlugins * Plugins CDN: Moved prometheus metric to api package, removed asset_path label * Fix missing in config * Changes after review * Add pluginscdn.Service * Fix tests * Refactoring * Moved all remaining CDN checks inside pluginscdn.Service * CDN url constructor: Renamed stringURLFor to stringPath * CDN: Moved asset URL functionality to assetpath service * CDN: Renamed HasCDN() to IsEnabled() * CDN: Replace assert with require * CDN: Changes after review * Assetpath: Handle url.Parse error * Fix plugin_resource_test * CDN: Change fallback redirect from 302 to 307 * goimports * Fix tests * Switch to contextmodel.ReqContext in plugins.go Co-authored-by: Will Browne <will.browne@grafana.com> Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>	2 years ago
Will Browne	c54aa18cd8	Plugins: Add tailored interface for plugins licensing needs (#61045) ... * tailored licensing service * appease linter * fix * retrigger	2 years ago
Giedrius Statkevičius	0485cf34cd	Plugins: Pass through dashboard/contextual HTTP headers to plugins/datasources (#60301) ... `X-Dashboard-Uid`, `X-Datasource-Uid`, `X-Grafana-Org-Id`, `X-Panel-Id` are very useful headers set by Grafana front-end that we would like to see on the data source as well. This is so that it would be possible to pinpoint from where slow queries are coming in Mimir/Thanos/Cortex/etc., for example. Relevant Mimir code lines: 0a94f26203/pkg/frontend/transport/handler.go (L182-L184) Tested manually that with these changes the headers are visible.	3 years ago
Marcus Efraimsson	c35c689a96	Plugins: Automatically forward plugin request HTTP headers in outgoing HTTP requests (#60417) ... Automatically forward core plugin request HTTP headers in outgoing HTTP requests. Core datasource plugin authors don't have to specifically handle forwarding of HTTP headers, e.g. do not have to "hardcode" the header-names in the datasource plugin, if not having custom needs. Fixes #57065	3 years ago
Marcus Efraimsson	6478d0a5ef	Plugins: Forward user header (X-Grafana-User) in backend plugin requests (#58646) ... Grafana would forward the X-Grafana-User header to backend plugin request when dataproxy.send_user_header is enabled. In addition, X-Grafana-User will be automatically forwarded in outgoing HTTP requests for core/builtin HTTP datasources. Use grafana-plugin-sdk-go v0.147.0. Fixes #47734 Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>	3 years ago
Marcus Efraimsson	6dbe3b555f	Plugins: Refactor forward of cookies, OAuth token and header modifications by introducing client middlewares (#58132) ... Adding support for backend plugin client middlewares. This allows headers in outgoing backend plugin and HTTP requests to be modified using client middlewares. The following client middlewares added: Forward cookies: Will forward incoming HTTP request Cookies to outgoing plugins.Client and HTTP requests if the datasource has enabled forwarding of cookies (keepCookies). Forward OAuth token: Will set OAuth token headers on outgoing plugins.Client and HTTP requests if the datasource has enabled Forward OAuth Identity (oauthPassThru). Clear auth headers: Will clear any outgoing HTTP headers that was part of the incoming HTTP request and used when authenticating to Grafana. The current suggested way to register client middlewares is to have a separate package, pluginsintegration, responsible for bootstrap/instantiate the backend plugin client with middlewares and/or longer term bootstrap/instantiate plugin management. Fixes #54135 Related to #47734 Related to #57870 Related to #41623 Related to #57065	3 years ago