Commit Graph

58 Commits

Author SHA1 Message Date
linoman
1b8e9b51b2 Replace signed in user for identity.requester (#74048)
* Make identity.Requester available at Context

* Clean pkg/services/guardian/guardian.go

* Clean guardian provider and guardian AC

* Clean pkg/api/team.go

* Clean ctxhandler, datasources, plugin and live

* Clean dashboards and guardian

* Implement NewUserDisplayDTOFromRequester

* Change status code numbers for http constants

* Upgrade signature of ngalert services

* log parsing errors instead of throwing error
2023-08-30 16:51:18 +02:00
Will Browne
557b1654f9 Plugins: Ensure service registration occurs in right order (#74001)
* make sure service registration occurs in right order

* fix test
2023-08-29 14:55:08 +02:00
Jo
a307582212 Revert "Replace signed in user for identity.requester (#73750)" (#73962)
This reverts commit 9b9c9e83dc.
2023-08-28 21:05:59 +02:00
linoman
9b9c9e83dc Replace signed in user for identity.requester (#73750)
* Make identity.Requester available at Context

* Clean pkg/services/guardian/guardian.go

* Clean guardian provider and guardian AC

* Clean pkg/api/team.go

* Clean ctxhandler, datasources, plugin and live

* Question: what to do with the UserDisplayDTO?

* Clean dashboards and guardian

* Remove identity.Requester from ReqContext

* Implement NewUserDisplayDTOFromRequester

* Fix tests

* Change status code numbers for http constants

* Upgrade signature of ngalert services

* log parsing errors instead of throwing error

* Fix tests and add logs

* linting
2023-08-28 12:04:36 -05:00
Giuseppe Guerra
4ef98449ff Plugins: Dynamic angular patterns: Add random offset to GCOM API calls, handle HTTP errors (#73494)
* Plugins: Dynamic angular patterns: Return error for != 2xx status code

* Add test for status code check

* Plugins: Dynamic angular patterns: Add random skew to periocic GCOM api calls

* Add test for random skew

* Changed randomSkew signature, ensure it is always positive
2023-08-28 11:34:05 +02:00
Torkel Ödegaard
3ee26df41e PublicDashboards: Variables refactor (#73476)
Co-authored-by: Juan Cabanas <juan.cabanas@grafana.com>
Co-authored-by: Ezequiel Victorero <ezequiel.victorero@grafana.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-08-25 13:56:02 -05:00
Hugo Kiyodi Oshiro
c69dd41bf9 Plugins: Forward feature toggles to plugins (#72995) 2023-08-25 15:01:54 +02:00
Will Browne
3c50db328d Plugins: Make Installer responsible for removing plugins from file system (#73323)
* installer is responsible for removing from file system

* take plugin as arg

* remove resolve step

* return plugin in test
2023-08-16 15:44:20 +02:00
Will Browne
75b0788377 Plugins: Remove registry dependency from process manager (#73241)
simplify
2023-08-16 10:46:00 +02:00
Will Browne
c5e9a82ccb Plugins: Do not fail bootstrap stage if single decorate step fails (#73147)
* don't fail all if decorate step fails

* fix casing

* include err too

* cover pluginsintegration too
2023-08-10 14:46:38 +02:00
Will Browne
d29f4a8f76 Plugins: Add context to StaticRouteResolver and ErrorResolver interfaces (#73121)
* add ctx

* fix tests
2023-08-10 10:32:12 +02:00
Will Browne
72da44db0e Plugins: Add validation stage to plugin loader pipeline (#73053)
* first pass

* change validation signature

* err tracking

* fix

* undo golden

* 1 more

* fix

* adjust doc

* add test helper

* fix linter
2023-08-09 18:25:28 +02:00
Sarah Zinger
deeb1d85f8 Config: making [aws][external_id] part of env (#72062)
* Config: making [aws][external_id] part of env

* Fix go.sum

* Add a test
2023-08-04 16:06:01 -04:00
Will Browne
98f827537b Plugins: Add remaining steps to inititialization stage (#72894)
add remainder of init steps
2023-08-04 12:35:57 +02:00
Will Browne
60b4a0b2a4 Plugins: Add termination stage to plugin loader pipeline (#72822)
* add termination stage

* uid -> pluginID (for now)

* also fix fakes

* add simple test

* Fix logger name

Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>

* inline stop func call

Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>

---------

Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
2023-08-04 11:57:49 +02:00
Will Browne
ad2705fa0b Plugins: Add inititialization stage to plugin loader pipeline (#72667)
* first pass

* migrate tests

* simplify

* fix comments

* fix linter

* nil checks

* remove comment
2023-08-02 16:29:12 +00:00
Andres Martinez Gotor
a331c892a0 Chore: Refactor error when retrieving the plugin manifest key (#72492) 2023-07-28 10:22:21 +02:00
Will Browne
758d9884bc Plugins: Plugins loader pipeline (#71438)
* discovery

* flesh out

* add docs

* remove unused func

* bootstrap stage

* fix docs

* update docs

* undo unnecessary changes

* add end tag

* update doc

* fix linter

* fix

* tidy

* update docs

* add class to filter func

* apply PR feedback

* fix test
2023-07-27 15:29:13 +02:00
Isabella Siu
56913fbd95 Redshift: Support caching async aws queries (#71682)
Co-authored-by: Sarah Zinger <sarah.zinger@grafana.com>
2023-07-21 11:34:07 -04:00
Giuseppe Guerra
a8d2a9ae2b Plugins: Angular detector: Add database cache store for remote patterns (#70693)
* Plugins: Angular detector: Remote patterns fetching

* Renamed PatternType to GCOMPatternType

* Renamed files

* Renamed more files

* Moved files again

* Add type checks, unexport GCOM structs

* Cache failures, update log messages, fix GCOM URL

* Fail silently for unknown pattern types, update docstrings

* Fix tests

* Rename gcomPattern.Value to gcomPattern.Pattern

* Refactoring

* Add FlagPluginsRemoteAngularDetectionPatterns feature flag

* Fix tests

* Re-generate feature flags

* Add TestProvideInspector, renamed TestDefaultStaticDetectorsInspector

* Add TestProvideInspector

* Add TestContainsBytesDetector and TestRegexDetector

* Renamed getter to provider

* More tests

* TestStaticDetectorsProvider, TestSequenceDetectorsProvider

* GCOM tests

* Lint

* Made detector.detect unexported, updated docstrings

* Allow changing grafana.com URL

* Fix API path, add more logs

* Update tryUpdateRemoteDetectors docstring

* Use angulardetector http client

* Return false, nil if module.js does not exist

* Chore: Split angualrdetector into angularinspector and angulardetector packages

Moved files around, changed references and fixed tests:
- Split the old angulardetector package into angular/angulardetector and angular/angularinspector
- angulardetector provides the detection structs/interfaces (Detector, DetectorsProvider...)
- angularinspector provides the actual angular detection service used directly in pluginsintegration
- Exported most of the stuff that was private and now put into angulardetector, as it is not required by angularinspector

* Renamed detector.go -> angulardetector.go and inspector.go -> angularinspector.go

Forgot to rename those two files to match the package's names

* Renamed angularinspector.ProvideInspector to angularinspector.ProvideService

* Renamed "harcoded" to "static" and "remote" to "dynamic"

from PR review, matches the same naming schema used for signing keys fetching

* WIP: Angular: cache patterns in db, moved gcom into pluginsintegration

More similar to signing keys fetching

* Rename package, refactoring

* try to solve circular import

* Fix merge conflict on updated angular patterns

* Fix circular imports

* Fix wire gen

* Add docstrings, refactoring

* Removed angualrdetectorsprovider dependency into angularpatternsstore

* Moved GCOM test files

* Removed GCOM cache

* Renamed Detect to DetectAngular and Detector to AngularDetector

* Fix call to NewGCOMDetectorsProvider in newDynamicInspector

* Removed unused test function newError500GCOMScenario

* Added angularinspector service definition in pluginsintegration

* refactoring

* lint

* Fix angularinspector TestProvideService

* cleanup

* Await initial restore

* Register dynamicAngularDetector background service

* Removed static detectors provider from pluginsintegration

* Add tests for kvstore

* Add more tests

* order imports in dynamic_test.go

* Fix potential panic in dynamic_test

* Add "runs the job periodically" test

* lint

* add timeout to test

* refactoring

* Removed context.Context from DetectorsProvider

* Refactoring, ensure angular dynamic background service is not started if feature flag is off

* Fix deadlock on startup

* Fix angulardetectorsprovider tests

* Revert "Removed context.Context from DetectorsProvider"

This reverts commit 4e8c6dded7.

* Fix wrong argument number in dynamic_teset

* Standardize gcom http client

* Reduce context timeout for angular inspector in plugins loader

* Simplify initial restore logic

* Fix dynamic detectors provider tests

* Chore: removed angulardetector/provider.go

* Add more tests

* Removed backgroundJob interface, PR review feedback

* Update tests

* PR review feedback: remove ErrNoCachedValue from kv store Get

* Update tests

* PR review feedback: add IsDisabled and remove nop background srevice

* Update tests

* Remove initialRestore channel, use mux instead

* Removed backgroundJobInterval, use package-level variable instead

* Add TestDynamicAngularDetectorsProviderBackgroundService

* Removed timeouts

* pr review feedback: restore from store before returning the service

* Update tests

* Log duration on startup restore and cron run

* Switch cron job start log to debug level

* Do not attempt to restore if disabled
2023-07-06 17:34:27 +03:00
Andres Martinez Gotor
025465e611 Chore: Update plugin schema with service registration info (#70692) 2023-06-27 08:47:25 +02:00
Andres Martinez Gotor
4ff0abd0d1 SaveExternalService (OAuth) on plugin load (#69764) 2023-06-26 16:38:43 +02:00
Giuseppe Guerra
cca9d89733 Plugins: Angular detector: Remote patterns fetching (#69843)
* Plugins: Angular detector: Remote patterns fetching

* Renamed PatternType to GCOMPatternType

* Renamed files

* Renamed more files

* Moved files again

* Add type checks, unexport GCOM structs

* Cache failures, update log messages, fix GCOM URL

* Fail silently for unknown pattern types, update docstrings

* Fix tests

* Rename gcomPattern.Value to gcomPattern.Pattern

* Refactoring

* Add FlagPluginsRemoteAngularDetectionPatterns feature flag

* Fix tests

* Re-generate feature flags

* Add TestProvideInspector, renamed TestDefaultStaticDetectorsInspector

* Add TestProvideInspector

* Add TestContainsBytesDetector and TestRegexDetector

* Renamed getter to provider

* More tests

* TestStaticDetectorsProvider, TestSequenceDetectorsProvider

* GCOM tests

* Lint

* Made detector.detect unexported, updated docstrings

* Allow changing grafana.com URL

* Fix API path, add more logs

* Update tryUpdateRemoteDetectors docstring

* Use angulardetector http client

* Return false, nil if module.js does not exist

* Chore: Split angualrdetector into angularinspector and angulardetector packages

Moved files around, changed references and fixed tests:
- Split the old angulardetector package into angular/angulardetector and angular/angularinspector
- angulardetector provides the detection structs/interfaces (Detector, DetectorsProvider...)
- angularinspector provides the actual angular detection service used directly in pluginsintegration
- Exported most of the stuff that was private and now put into angulardetector, as it is not required by angularinspector

* Renamed detector.go -> angulardetector.go and inspector.go -> angularinspector.go

Forgot to rename those two files to match the package's names

* Renamed angularinspector.ProvideInspector to angularinspector.ProvideService

* Renamed "harcoded" to "static" and "remote" to "dynamic"

from PR review, matches the same naming schema used for signing keys fetching

* Fix merge conflict on updated angular patterns

* Removed GCOM cache

* Renamed Detect to DetectAngular and Detector to AngularDetector

* Fix call to NewGCOMDetectorsProvider in newDynamicInspector

* Removed unused test function newError500GCOMScenario

* Added angularinspector service definition in pluginsintegration

* Moved dynamic inspector into pluginsintegration

* Move gcom angulardetectorsprovider into pluginsintegration

* Log errUnknownPatternType at debug level

* re-generate feature flags

* fix error log
2023-06-26 15:33:21 +02:00
Andres Martinez Gotor
7c1c196031 Plugins: Periodically update public signing key (#70080) 2023-06-16 11:20:30 +02:00
Will Browne
91b0cdc871 Plugins: Account for nil user when constructing plugin context (#69811)
cater for nil user
2023-06-08 19:36:41 +03:00
Will Browne
624777258b Plugins: Refactor creation of plugin context to dedicated service (#66451)
* first pass

* fix tests

* return errs

* change signature

* tidy

* delete unnecessary fields from test

* tidy

* fix tests

* simplify

* separate error check in API

* apply nits
2023-06-08 13:59:51 +02:00
Giuseppe Guerra
f148b5fb28 Plugins: Forbid loading Angular plugins when Angular is disabled (#69679)
* Plugins: Forbid loading Angular plugins when Angular is disabled

* Plugins: Made angulardetector a service, add tests for angular loader cases

* Fix missing import

* Add nolint:gocyclo to loadPlugins
2023-06-07 11:08:01 +02:00
Giuseppe Guerra
7a132680ef Revert "Plugins: Forbid loading Angular plugins when Angular is disabled (#69225)" (#69657)
This reverts commit ff34279ff4.
2023-06-06 18:09:41 +02:00
Giuseppe Guerra
ff34279ff4 Plugins: Forbid loading Angular plugins when Angular is disabled (#69225)
* Plugins: Forbid loading Angular plugins when Angular is disabled

* Plugins: Made angulardetector a service, add tests for angular loader cases

* Fix missing import
2023-06-06 17:14:31 +02:00
Andres Martinez Gotor
932136807b Chore: Allow to force the download of the public key (#67486) 2023-04-28 16:00:48 +02:00
Marcus Efraimsson
4cbda914bd Plugins: Refactor call resource API handling (#67234)
Moving call resource API stream handling within plugin management as a utility/wrapper. 

Closes #66889

Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
2023-04-28 14:02:27 +02:00
Andres Martinez Gotor
aa9838bd25 Chore: Refactor manifest verifier (#67218) 2023-04-27 17:54:28 +02:00
Serge Zaitsev
6d8f9c5bf4 Chore: Remove opentracing and use opentelemetry instead (#67200)
* remove opentracing and use otel instead

* add various samplers for jaeger

* remove useless test that is covered in otel now

* we do not need a struct there

* remove old tests

* restore tests that parse various configurations

* check errors in tests

* Update pkg/infra/tracing/tracing_test.go

fix typo

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* add test for both legacy and new config formats

* use named constants

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2023-04-27 15:04:43 +02:00
Giuseppe Guerra
bfe1b68800 Plugins: Fs: Add option to access unallowed files in dev mode (#66492)
* Plugins: Fs: Add option to access unallowed files in dev mode

* Plugins: Fs: allow accessing unallowed files only when in dev mode

* Plugins: Fs: Add ProvideLocalFinder

* Plugins: FS: Pass whole config in NewLocalFinder()

* Plugins: FS: Add AllowListLocalFS

* Plugins: FS: Fix some tests

* Plugins: FS: Update tests

* Plugins: FS: Removed dead code

* Plugins: FS: Add tests for AllowListFS

* Plugins: FS: Update comments

* Plugins: FS: Use variadic arguments for allow list rather than map

* Plugins: FS: Remove unnecessary log

* Plugins: FS: Do not escape plugin root dir

* Fix merge conflict

* Plugins: FS: Update comments

* Plugins: FS: PR review changes

* Fix merge conflict

* Fix tests

* Cleanup

* Fix flaky test

* Changes from PR review

* Lint

* Add comment to LocalFS.Remove

* Fix Windows

* Renamed devMode to production
2023-04-27 10:26:15 +02:00
Marcus Efraimsson
73920b1e34 Plugins: Refactor cleaning of call resource response headers (#67145)
First part of #66889 moving cleaning of call resource response headers 
within plugin management client.
2023-04-25 19:44:32 +02:00
Michael Mandrus
1421f388ae Caching: Fix concurrent HTTP Header read/write in caching middleware (#67231)
read the response header synchronously, defer the metric only
2023-04-25 13:05:37 -04:00
Andres Martinez Gotor
9d7c3a101d Chore: Use KVStore for the manifest public key (#66839) 2023-04-25 13:01:49 +02:00
Michael Mandrus
a29cfe5d46 Caching: Consolidate resource cache checking and updating in plugin middleware (#67002)
* Update the HandleResourceRequest function to mimic the HandleQueryRequest function

* Remove CacheResourceResponse function from interface

* revert additional thing I missed
2023-04-21 13:03:49 -04:00
Andres Martinez Gotor
98c695c68f Chore: Use Grafana API to retrieve the public key to validate plugins (#66439) 2023-04-18 16:12:05 +02:00
Michael Mandrus
5626461b3c Caching: Refactor enterprise query caching middleware to a wire service (#65616)
* define initial service and add to wire

* update caching service interface

* add skipQueryCache header handler and update metrics query function to use it

* add caching service as a dependency to query service

* working caching impl

* propagate cache status to frontend in response

* beginning of improvements suggested by Lean - separate caching logic from query logic.

* more changes to simplify query function

* Decided to revert renaming of function

* Remove error status from cache request

* add extra documentation

* Move query caching duration metric to query package

* add a little bit of documentation

* wip: convert resource caching

* Change return type of query service QueryData to a QueryDataResponse with Headers

* update codeowners

* change X-Cache value to const

* use resource caching in endpoint handlers

* write resource headers to response even if it's not a cache hit

* fix panic caused by lack of nil check

* update unit test

* remove NONE header - shouldn't show up in OSS

* Convert everything to use the plugin middleware

* revert a few more things

* clean up unused vars

* start reverting resource caching, start to implement in plugin middleware

* revert more, fix typo

* Update caching interfaces - resource caching now has a separate cache method

* continue wiring up new resource caching conventions - still in progress

* add more safety to implementation

* remove some unused objects

* remove some code that I left in by accident

* add some comments, fix codeowners, fix duplicate registration

* fix source of panic in resource middleware

* Update client decorator test to provide an empty response object

* create tests for caching middleware

* fix unit test

* Update pkg/services/caching/service.go

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>

* improve error message in error log

* quick docs update

* Remove use of mockery. Update return signature to return an explicit hit/miss bool

* create unit test for empty request context

* rename caching metrics to make it clear they pertain to caching

* Update pkg/services/pluginsintegration/clientmiddleware/caching_middleware.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Add clarifying comments to cache skip middleware func

* Add comment pointing to the resource cache update call

* fix unit tests (missing dependency)

* try to fix mystery syntax error

* fix a panic

* Caching: Introduce feature toggle to caching service refactor (#66323)

* introduce new feature toggle

* hide calls to new service behind a feature flag

* remove licensing flag from toggle (misunderstood what it was for)

* fix unit tests

* rerun toggle gen

---------

Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2023-04-12 12:30:33 -04:00
Stephanie Hingtgen
536fad5b61 Plugins: add env variables needed for the secure socks proxy in the plugins sdk (#65878) 2023-04-07 08:20:25 -06:00
Will Browne
bff9f4c890 Plugins: Move config factory to pluginsintegration package (#65716)
* move config to pluginsintegration package

* change to all plugin toggle

* fixes

* fixes

* fix lerna

* fix test
2023-04-05 14:40:08 +02:00
Isabella Siu
92965578b3 CloudWatch Logs: Fix running logs queries with expressions (#65306) 2023-03-30 17:40:01 -04:00
Will Browne
7bbe255150 Plugins: Add file store abstraction for handling plugin files (#65432)
* add file store

* fix markdown fetch bug

* add markdown tests

* fix var name
2023-03-29 12:55:55 +02:00
Giuseppe Guerra
a89202eab2 Plugins: Improve instrumentation by adding metrics and tracing (#61035)
* WIP: Plugins tracing

* Trace ID middleware

* Add prometheus metrics and tracing to plugins updater

* Add TODOs

* Add instrumented http client

* Add tracing to grafana update checker

* Goimports

* Moved plugins tracing to middleware

* goimports, fix tests

* Removed X-Trace-Id header

* Fix comment in NewTracingHeaderMiddleware

* Add metrics to instrumented http client

* Add instrumented http client options

* Removed unused function

* Switch to contextual logger

* Refactoring, fix tests

* Moved InstrumentedHTTPClient and PrometheusMetrics to their own package

* Tracing middleware: handle errors

* Report span status codes when recording errors

* Add tests for tracing middleware

* Moved fakeSpan and fakeTracer to pkg/infra/tracing

* Add TestHTTPClientTracing

* Lint

* Changes after PR review

* Tests: Made "ended" in FakeSpan private, allow calling End only once

* Testing: panic in FakeSpan if span already ended

* Refactoring: Simplify Grafana updater checks

* Refactoring: Simplify plugins updater error checks and logs

* Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates

* Tests: Fix wrong call to checkForUpdates -> instrumentedCheckForUpdates

* Log update checks duration, use Info log level for check succeeded logs

* Add plugin context span attributes in tracing_middleware

* Refactor prometheus metrics as httpclient middleware

* Fix call to ProvidePluginsService in plugins_test.go

* Propagate context to update checker outgoing http requests

* Plugin client tracing middleware: Removed operation name in status

* Fix tests

* Goimports tracing_middleware.go

* Goimports

* Fix imports

* Changed span name to plugins client middleware

* Add span name assertion in TestTracingMiddleware

* Removed Prometheus metrics middleware from grafana and plugins updatechecker

* Add span attributes for ds name, type, uid, panel and dashboard ids

* Fix http header reading in tracing middlewares

* Use contexthandler.FromContext, add X-Query-Group-Id

* Add test for RunStream

* Fix imports

* Changes from PR review

* TestTracingMiddleware: Changed assert to require for didPanic assertion

* Lint

* Fix imports
2023-03-28 11:01:06 +02:00
Will Browne
31d6416157 Plugins: Migrate licensing and access control to pkg/services/pluginsintegration package (#65258)
* migrate licensing + access control

* update package name
2023-03-27 11:15:37 +02:00
Will Browne
ee2dd62a1f Plugins: Refactor loader + finder to support multiple sourcing methods (#64735)
* it's cdn time

* tidy body closing

* auto signed

* fix close

* update log name

* remove comments
2023-03-20 14:35:49 +01:00
Sven Grossmann
7261c6f7cd DataSourceWithBackend: Add queryGroupId to find correlated/related queries (#64587)
* add `correlationId` to queries

* trace correlation id in backend

* add correlation id to loki's span

* add correlation id to query chunks

* fix test

* fix DataSourceWithBackend test

* change to `queryGroupId`

* remove empty line

* fix test in `DataSourceWithBackend`
2023-03-17 15:48:13 +01:00
Will Browne
94f39e69a3 Plugins: Migrate plugincontext, adapters and pluginsettings to pkg/services/pluginsintegration package (#64154)
* migrate plugincontext, adapter and pluginsettings

* add to CODEOWNERS

* fix imports

* fix CODEOWNERS

* take pluginsettings

* migrate wire stuff
2023-03-07 11:22:30 -05:00
Will Browne
ab8de1a0e3 Plugins: Add simple plugin sources service (#63814)
add simple plugin sources svc
2023-02-28 15:27:11 +01:00