* move original stats service into a separate package
* add stats service to wire
* move GetAdminStats
* switch to using stats.Service
* add missing package
* fix api tests
* Plugins: Remove support for V1 manifests
* Plugins: Make proxy endpoints not leak sensitive HTTP headers
* Security: Fix do not forward login cookie in outgoing requests
(cherry picked from commit 4539c33fce)
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* chore: add alias for InitTestDB and Session
Adds an alias for the sqlstore InitTestDB and Session, and updates tests using these to reduce dependencies on the sqlstore.Store.
* next pass of removing sqlstore imports
* last little bit
* remove mockstore where possible
* SQLStore: Ensure that sessions are always closed
Delete `NewSession()` in favour of `WithDbSession()`
* Add WithDbSessionForceNewSession to the interface
* Apply suggestions from code review
* NewIA: Plugin nav config
* progress
* Progress
* Things are working
* Add monitoring node
* Add alerts and incidents
* added experiment with standalone page
* Refactoring by adding a type for navtree root
* First test working
* More tests
* more tests
* Progress on richer config and sorting
* Sort weight working
* Path config
* Improving logic for not including admin or cfg nodes, making it the last step so that enterprise can add admin nodes without having to worry about the section not existing
* fixed index routes
* removed file
* Fixes
* Fixing tests
* Fixing more tests and adding support for weight config
* Updates
* Remove unused fake
* More fixes
* Minor tweak
* Minor fix
* Can now control position using sortweight even when existing items have no sortweight
* Added tests for frontend standalone page logic
* more tests
* Remove unused fake and fixed lint issue
* Moving reading settings to navtree impl package
* remove nav_id setting prefix
* Remove old test file
* Fix trailing newline
* Fixed bug with adding nil node
* fixing lint issue
* remove some code we have to rethink
* move read settings to PrivideService and switch to util.SplitString
Introduces a FromContext method on the log.Logger interface that
allows contextual key/value pairs to be attached, e.g. per request,
so that any logger using this API will automatically get the per request
context attached. The proposal makes the traceID available for
contextual logger , if available, and would allow logs originating from
a certain HTTP request to be correlated with traceID.
In addition, when tracing not enabled, skip adding
traceID=00000000000000000000000000000000
to logs.
* WIP
* Set public_suffix to a pre Ruby 2.6 version
* we don't need to install python
* Stretch->Buster
* Bump versions in lib.star
* Manually update linter
Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.
Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.
* undo reformatting
* Various lint improvements
* More from the linter
* goimports -w ./pkg/
* Disable gocritic
* Add/modify linter exceptions
* lint + flatten nested list
Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
* initial cut at migration from plugin
* create new migration from plugin
* only migrate to or from, not both
* remove cfg check from plugin migration itself
* update comments, clean up secret after migration
* add better error handling
* hook up REST API with migrations
* Minor fixes
* fix wire injection issue
* modify migrator to access plugin calls directly. create unit tests
* change pre-migration checks in admin api
* stop plugin after migrating from it
* fix compile issues after merge
* add comment about migration
* fix linting issue
* bleh, fix unit test
* fix another unit test
* update plugin error fatal flag after a migration from the plugin
* add extra logging to migration
* make linter happy
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
* Chore: Add user service method SetUsingOrg
* Chore: Add user service method GetSignedInUserWithCacheCtx
* Use method GetSignedInUserWithCacheCtx from user service
* Fix lint after rebase
* Fix lint
* Fix lint error
* roll back some changes
* Roll back changes in api and middleware
* Add xorm tags to SignedInUser ID fields
* add depguard rule for ioutil
* replace ioutil.ReadDir with os.ReadDir
* use legacy option in depguard supported in golangci-lint v1.40
* replace ioutil.ReadDir with os.ReadDir
* return error for file info
* Move SignedInUser to user service and RoleType and Roles to org
* Use go naming convention for roles
* Fix some imports and leftovers
* Fix ldap debug test
* Fix lint
* Fix lint 2
* Fix lint 3
* Fix type and not needed conversion
* Clean up messages in api tests
* Clean up api tests 2
* Created PluginSecretMigrationService to be able to migrate from the secrets table from the database to the secret plugin. Added migration which takes all the secrets at the sql store and stores it in the plugin. Then deletes all the secrets from the sql
* Added secretsKVStoreSQL.GetAll() method to return all the secrets at the sql table
* Renaming kvstore_test.go as sql_test.go, adding GetAll test case. Fixing decryption of keys
* UsageStats: fixed elasticsearch version number
- The version numbering was changed from plain numbers to a semver-ish approach
* added missing version assertion
* adapted tests
* Extend kvstore to retrieve all items
* Fix comment
* Fix tests
* Change test order
* Move test outside to avoid order conditions
* Update Items to GetAll function and return a map
* Add explanation of map result
* Add description comment
Co-authored-by: Tania B <yalyna.ts@gmail.com>
In #50420 a few summary metrics were converted to histograms,
but the Help text in a couple of them still referred to summaries.
This fixes that help text.
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Because Summary metrics can not be aggreated, convert them to histograms
so that users with HA deployments can use these metrics.
* Convert metrics registration to promauto.
* Improve help text style.
Signed-off-by: SuperQ <superq@gmail.com>
* Chore: Exclude integration tests from running on test-backend step
* Remove -v from go test command
* Add check to skip integration tests before each integration test
* Try to restart pipeline
* Retrying to make pipeline run
* pkg/web: store http.Handler internally
* pkg/web: remove injection
Removes any injection code from pkg/web.
It already was no longer functional, as we already only injected into
`http.Handler`, meaning we only inject ctx.Req and ctx.Resp.
Any other types (*Context, *ReqContext) were already accessed using the
http.Request.Context.Value() method.
* *: remove type mappings
Removes any call to the previously removed TypeMapper, as those were
non-functional already.
* pkg/web: remove Context.Invoke
was no longer used outside of pkg/web and also no longer functional
* Add database migrations
* Use short uids as data key ids
* Add support for manual data key rotation
* Fix duplicated mutex unlocks
* Fix migration
* Manage current data keys per name
* Adjust key re-encryption and test
* Modify rename column migration for MySQL compatibility
* Refactor secrets manager and data keys cache
* Multiple o11y adjustments
* Fix stats query
* Apply suggestions from code review
Co-authored-by: Tania <yalyna.ts@gmail.com>
* Fix linter
* Docs: Rotate data encryption keys API endpoint
Co-authored-by: Tania <yalyna.ts@gmail.com>
Makes `pkg/web` only accept handles from the following set:
```go
handlerStd = func(http.ResponseWriter, *http.Request)
handlerStdCtx = func(http.ResponseWriter, *http.Request, *web.Context)
handlerStdReqCtx = func(http.ResponseWriter, *http.Request, *models.ReqContext)
handlerReqCtx = func(*models.ReqContext)
handlerReqCtxRes = func(*models.ReqContext) Response
handlerCtx = func(*web.Context)
```
This is a first step to reducing above set to only `http.Handler`.
---
Due to a cyclic import situation between `pkg/models` and `pkg/web`, parts of this PR were put into `pkg/api/response`, even though they definitely do not belong there. This however is _temporary_ until we untangle `models.ReqContext`.
* #45498: add String util to ListResponse for better UX
* #45498: refactor db_filestorage FS API backend - use path_hash in DB schema
* #45498: enable DB backend fs api tests
* #45498: add comment
* #45498: enable Storage feature flag during integration tests
* remove fmt.println
* #45498: reduce sizes of hash columns
* separate conditions
* #45498: make it easy to ignore backends when running fs api integration tests
* #45498: quote `key` column name
* #45498: reduce path_hash size
* #45498: verify `/{orgId}/{storageName}/` prefix convention in integration tests
* #45498: add etag to the sql table
* #45498: add etag to the sql table
* remove feature flag check (storage isn't dev-mode only)
* add cacheControl and content disposition
* add comments
* add path_hash comment
* explicitly set `path` column collation in `file` table for postgres
* wip: Implement kvstore for secrets
* wip: Refactor kvstore for secrets
* wip: Add format key function to secrets kvstore sql
* wip: Add migration for secrets kvstore
* Remove unused Key field from secrets kvstore
* Remove secret values from debug logs
* Integrate unified secrets with datasources
* Fix minor issues and tests for kvstore
* Create test service helper for secret store
* Remove encryption tests from datasources
* Move secret operations after datasources
* Fix datasource proxy tests
* Fix legacy data tests
* Add Name to all delete data source commands
* Implement decryption cache on sql secret store
* Fix minor issue with cache and tests
* Use secret type on secret store datasource operations
* Add comments to make create and update clear
* Rename itemFound variable to isFound
* Improve secret deletion and cache management
* Add base64 encoding to sql secret store
* Move secret retrieval to decrypted values function
* Refactor decrypt secure json data functions
* Fix expr tests
* Fix datasource tests
* Fix plugin proxy tests
* Fix query tests
* Fix metrics api tests
* Remove unused fake secrets service from query tests
* Add rename function to secret store
* Add check for error renaming secret
* Remove bus from tests to fix merge conflicts
* Add background secrets migration to datasources
* Get datasource secure json fields from secrets
* Move migration to secret store
* Revert "Move migration to secret store"
This reverts commit 7c3f872072.
* Add secret service to datasource service on tests
* Fix datasource tests
* Remove merge conflict on wire
* Add ctx to data source http transport on prometheus stats collector
* Add ctx to data source http transport on stats collector test
* use common traceID context value for opentracing and opentelemetry
* support sampled trace IDs as well
* inject traceID into NormalResponse on errors
* Finally the test passed
* fix the test
* fix linter
* change the function parameter
Co-authored-by: Ying WANG <ying.wang@grafana.com>
* add feature toggle with new format
* fix some comments ❤️
* Update pkg/infra/log/term/terminal_logger.go
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Remove specific stats from usage stats service
* Create statscollector service
* refactor
* Update and move tests
Mostly equivalent tests to before, but they've been divided over the two
services and removed the behavior driven legacy from GoConvey to
reduce the complexity of the tests.
* Collect featuremgmr metrics (copied over from #47407)
I removed the metrics registration from the feature manager in the merge
and re-add them in this commit. Separated to make things easier to
review.
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Stats: do not count SAs as users
* Stats: implement basic service account metrics
* Stats: do not count service account tokens as api keys
* Stats: fix metric names
* Stats: add SA stats test
* rename user to sa
* silence errors
* s3 fix - don't retrieve files with path equal to the root
* Storage: unify list queries
* Storage: add `IsFolder` method to file obj
* Storage: API consistency - always refer `File` as a pointer rather than a value
Fixing a couple bugs, adds some tests and hopefully decrease
lock contention when logging.
Switching from using sync.RWMutex to go-kit SwapLogger.
Fixes bug when creating a new logger from an existing one that
screwed up the keyvals and/or lost the logger name.
Ref #44681
* Separate Tracer interface to TracerService and Tracer
* Fix lint
* Fix:Make it possible to start spans for both opentracing and opentelemetry in ds proxy
* Add span methods, use span interface for rest of tracing
* Fix logs in tracing
* Fix tests that are related to tracing
* Fix resourcepermissions test
* Fix some tests
* Fix more tests
* Add TracingService to wire cli runner
* Remove GlobalTracer from bus
* Renaming test function
* Remove GlobalTracer from TSDB
* Replace GlobalTracer in api
* Adjust tests to the InitializeForTests func
* Remove GlobalTracer from services
* Remove GlobalTracer
* Remove bus.NewTest
* Remove Tracer interface
* Add InitializeForBus
* Simplify tests
* Clean up tests
* Rename TracerService to Tracer
* Update pkg/middleware/request_tracing.go
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
* Initialize tracer before passing it to SQLStore initialization in commands
* Remove tests for opentracing
* Set span attributes correctly, remove unnecessary trace initiliazation form test
* Add tracer instance to newSQLStore
* Fix changes due to rebase
* Add modified tracing middleware test
* Fix opentracing implementation tags
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
* add caller and stack Valuer functions
* Add WithPrefix and WithSuffix similar to what go-kit offers
* replace New with just `with`. Remove filter wrapper because the first argument of the context argument is not logger but additional context.
* update Xorm logger to use custom depth to display the datastore code instead of xorm
* Add interface Tracer, add Opentelemetry
* Fix lint
* Fix failing tests and return error if config not parsed fo opentelemetry
* Update defaults.ini
Add comment with jaeger url
* go mod tidy
* Remove comments that are not needed
* Move OpentracingSpan to tracing.go
* Add opentelemetry to sample.ini
When a request going through Grafana data source proxy responds with a websocket
upgrade response we cannot override the response body since it produces an error.
This problem seems to have been introduced in Grafana v8.0 by #38962.
In addition #40303 added same problem.
Fixes#41292
* context all the things
* apply feedback
* rollback some alerting changes
* rollback some alerting changes #2
* more rollbacks
* more rollbacks #2
* more rollbacks #3
* more rollbacks #4
* fix integration test
* add missing context
* add missing and remove incorrect dispatch
* Add context to star and stats
* Use WithTransactionalDbSession
* Add additional ctx
* Remove convey
* Fix star handler name
* Use WithDbSession, use DispatchCtx
* Remove xorm from star
* added errors in plugin list.
* added error to details page.
* adding badge on details page.
* added some more tests.
* Renamed to disabled and will handle the scenario in the plugin catalog.
* Update public/app/features/plugins/admin/components/PluginDetailsDisabledError.tsx
Co-authored-by: Levente Balogh <balogh.levente.hu@gmail.com>
* fixing some nits
* added missing isDisabeld to the mock.
* adding tests to verify scenarios when plugin is disabled.
* fixed issue with formatting after file changed on GH.
Co-authored-by: Levente Balogh <balogh.levente.hu@gmail.com>
* warn on linux
* add warning for grpc plugins
* add windows support
* update go.mod
* reorganize imports
* update naming
* remove Windows logic
* simplify and add check for when UID and EUID don't match
* fix build
* tidy go.mod
* feedback
* cleanup + migrate
* Introduce response_limit for datasource responses
* Fix lint
* Fix tests
* Add case where limit <= 0 - added parametrized tests
* Add max_bytes_reader.go
* Use new httpclient.MaxBytesReader instead of net/http one
* Fixes according to reviewer's comments
* Add tests for max_bytes_reader
* Add small piece in configuration.md
* Further fixes according to reviewer's comments
* Fix linting - fix test
* add key/value store service
* don't export kvStoreSQL, consumers should interact with KVStore & NamespacedKVStore
* add del method, avoid ErrNotFound (#38627)
* switch value column to medium text
Co-authored-by: Alexander Emelin <frvzmb@gmail.com>
Fixes#30144
Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
* Azure middleware in HttpClientProxy
* Azure authentication under feature flag
* Minor fixes
* Add prefixes to not clash with JsonData
* Return error if JsonData cannot be parsed
* Return original string if URL invalid
* Tests for datasource_cache
Make sure that default timeout settings are based on configuration
parameters. This now applies for core data sources using old TSDB
contracts and new SDK contracts. Before it was only applied for old TSDB
contracts.
Also moves global setting variables to non-global (setting.Cfg).
* Usage Stats: Rename service to use a more idiomatic name
* Usage Stats: Update MetricsFunc definition and implementations
* Revert "Usage Stats: Rename service to use a more idiomatic name"
This reverts commit 910ecce3e8.
* Usage Stats: Update MetricsFunc definition and implementations
Uses new httpclient package from grafana-plugin-sdk-go introduced
via grafana/grafana-plugin-sdk-go#328.
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the
SDK contracts and using httpclient.Provider for creating HTTP clients and such.
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
This patch adds metrics to support instrumenting the accesscontrols package.
It also instruments the accesscontrol evaluator and the permissions function.
Co-authored-by: Vardan Torosyan <vardants@gmail.com>
* add uninstall flow
* add install flow
* small cleanup
* smaller-footprint solution
* cleanup + make bp start auto
* fix interface contract
* improve naming
* accept version arg
* ensure use of shared logger
* make installer a field
* add plugin decommissioning
* add basic error checking
* fix api docs
* making initialization idempotent
* add mutex
* fix comment
* fix test
* add test for decommission
* improve existing test
* add more test coverage
* more tests
* change test func to use read lock
* refactoring + adding test asserts
* improve purging old install flow
* improve dupe checking
* change log name
* skip over dupe scanned
* make test assertion more flexible
* remove trailing line
* fix pointer receiver name
* update comment
* add context to API
* add config flag
* add base http api test + fix update functionality
* simplify existing check
* clean up test
* refactor tests based on feedback
* add single quotes to errs
* use gcmp in tests + fix logo issue
* make plugin list testing more flexible
* address feedback
* fix API test
* fix linter
* undo preallocate
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* fix linting issue in test
* add docs placeholder
* update install notes
* Update docs/sources/plugins/marketplace.md
Co-authored-by: Marcus Olsson <marcus.olsson@hey.com>
* update access wording
* add more placeholder docs
* add link to more info
* PR feedback - improved errors, refactor, lock fix
* improve err details
* propagate plugin version errors
* don't autostart renderer
* add H1
* fix imports
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
Co-authored-by: Marcus Olsson <marcus.olsson@hey.com>
* Rendering: add CSV rendering support
* Rendering: save csv files into a separate folder
* add missing field
* Renderer: get filename from renderer plugin
* apply PR suggestions
* Rendering: remove old PhantomJS error
* Rendering: separate RenderCSV and Render functions
* fix alerting test
* Rendering: fix handling error in HTTP mode
* apply PR feedback
* Update pkg/services/rendering/http_mode.go
Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>
* apply PR feedback
* Update rendering metrics with type label
* Rendering: return error if not able to parse header
* Rendering: update grpc generated file
* Rendering: use context.WithTimeout to render CSV too
Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>
* use a common way to redact sensitive values before logging them
* fix panic on missing testCase.err, simplify require checks
* fix a silly typo
* combine readConfig and buildConnectionString methods, as they are closely related
* check for jaeger env variables if tracing address configuration is empty
* add tests to ensure jaeger host/port variables override the settings
* allow default host and custom port, too
* disallow missing tracing.jaeger section in the config
* check for all errors in tests
* make parseSettings() return an error
* FrontendMetrics: Adds new backend api that frontend can use to push frontend measurements and counters to prometheus
* FrontendMetrics: Adds new backend api that frontend can use to push frontend measurements and counters to prometheus
* Fix naming
* change to histogram
* Fixed go lint
* Chore: Collect elasticsearch version usage stats
* Fix lint error
* use GetDataSources from sqlstore
* Apply review suggestions
* Return error if datasource type is not specified
* Update pkg/services/sqlstore/datasource.go
* fix undefined var
Usage Stats: Calculate concurrent users as a histogram
Metric will help to understand if users are sharing their credentials or using one account for concurrent sessions. This will help to make more informed decisions when enforcing any license limitations on credentials usage.
