* Encryption: Add support for data keys re-encryption
* Add tests for data keys re-encryption
* Update code after refactorings
Co-authored-by: Leonard Gram <leo@xlson.com>
* Separate Tracer interface to TracerService and Tracer
* Fix lint
* Fix:Make it possible to start spans for both opentracing and opentelemetry in ds proxy
* Add span methods, use span interface for rest of tracing
* Fix logs in tracing
* Fix tests that are related to tracing
* Fix resourcepermissions test
* Fix some tests
* Fix more tests
* Add TracingService to wire cli runner
* Remove GlobalTracer from bus
* Renaming test function
* Remove GlobalTracer from TSDB
* Replace GlobalTracer in api
* Adjust tests to the InitializeForTests func
* Remove GlobalTracer from services
* Remove GlobalTracer
* Remove bus.NewTest
* Remove Tracer interface
* Add InitializeForBus
* Simplify tests
* Clean up tests
* Rename TracerService to Tracer
* Update pkg/middleware/request_tracing.go
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
* Initialize tracer before passing it to SQLStore initialization in commands
* Remove tests for opentracing
* Set span attributes correctly, remove unnecessary trace initiliazation form test
* Add tracer instance to newSQLStore
* Fix changes due to rebase
* Add modified tracing middleware test
* Fix opentracing implementation tags
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
* Working version of migrate secrets CLI command
* Move user oauth info encryption away from db transaction
* Enable a mechanism for re-use db session on encryption
* De-duplicate shared code between db and runner commands
* Set up Wire build graph
* Remove enterprise Wire set
* Fix cylic dependency: sqlstore.DBSession <-> xorm.Session
* Minor fix (add missing base64 encoding)
* Extract CLI 'secrets-migration' commands from 'data-migration' ones
* Move runner package outside commands
* Update Makefile (gen-go path)
* Minor prettier fix
* Some minor XORM related refactors
* Include new Wire enterprise file into .gitignore
* Update Wire deps
Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>
Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>
* Set up Wire build graph
* Remove enterprise Wire set
* Move runner package outside commands
* Update Makefile (gen-go path)
* Minor prettier fix
* Include new Wire enterprise file into .gitignore
* Update Wire deps
* Update the grabpl version
Co-authored-by: Dan Cech <dcech@grafana.com>
* Add file blacklist to `grafana-cli cue gen-ts` cmd
* Add CI step checking all cuetsification is done
* Add dummy command to make the next one fail
* Generate drone bits
* Check diff output failure
* Echo list of untracked files, for failure locality
* Move git cleanness checking into script
* Blacklist of cue files is complete and correct
* Remove news panel plugin from cuetsify blacklist
* Dummy commit, check that untracked gen still fail
* Tie off remaining errors
* Re-add barchart to blacklist
* Remove file left around by earlier pipeline
* Commit generated news models.gen.ts
* Include eslint as part of cuetsified output gen
* Update pkg/cmd/grafana-cli/commands/cuetsify_command.go
Co-authored-by: Ashley Harrison <ashley.harrison@grafana.com>
* Update scripts/drone/steps/lib.star
Co-authored-by: Maria Alexandra <239999+axelavargas@users.noreply.github.com>
* Update drone.yml
* Last fix on .drone.yml
Co-authored-by: Ashley Harrison <ashley.harrison@grafana.com>
Co-authored-by: Maria Alexandra <239999+axelavargas@users.noreply.github.com>
* Encryption: Add support to encrypt/decrypt sjd
* Add datasources.Service as a proxy to datasources db operations
* Encrypt ds.SecureJsonData before calling SQLStore
* Move ds cache code into ds service
* Fix tlsmanager tests
* Fix pluginproxy tests
* Remove some securejsondata.GetEncryptedJsonData usages
* Add pluginsettings.Service as a proxy for plugin settings db operations
* Add AlertNotificationService as a proxy for alert notification db operations
* Remove some securejsondata.GetEncryptedJsonData usages
* Remove more securejsondata.GetEncryptedJsonData usages
* Fix lint errors
* Minor fixes
* Remove encryption global functions usages from ngalert
* Fix lint errors
* Minor fixes
* Minor fixes
* Remove securejsondata.DecryptedValue usage
* Refactor the refactor
* Remove securejsondata.DecryptedValue usage
* Move securejsondata to migrations package
* Move securejsondata to migrations package
* Minor fix
* Fix integration test
* Fix integration tests
* Undo undesired changes
* Fix tests
* Add context.Context into encryption methods
* Fix tests
* Fix tests
* Fix tests
* Trigger CI
* Fix test
* Add names to params of encryption service interface
* Remove bus from CacheServiceImpl
* Add logging
* Add keys to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Add missing key to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Undo changes in markdown files
* Fix formatting
* Add context to secrets service
* Rename decryptSecureJsonData to decryptSecureJsonDataFn
* Name args in GetDecryptedValueFn
* Add template back to NewAlertmanagerNotifier
* Copy GetDecryptedValueFn to ngalert
* Add logging to pluginsettings
* Fix pluginsettings test
Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Update embeds with new paths, remove deprefixing
* Add dist dashboard scuemata, in CUE
Also add tests that ensure all panel plugins with models.cue are
imported by this file, so that we can't go out of sync.
* Remove comment detritus
* Check error return
* Remove hardcoded path prefix from cli validator
* Re-enable devenv dashboard validation
* Open up dashboard schema composition points
* Introduce composition space at front of scuemata
* Refactor go code to use new composition structure
* Bunch of small cleanups in dashboard.cue
* Enable both base and dist tests of devenv
* Get rid of obsolete CUE loading hacks
* Skip weird failures on these tests
Really don't seem to be testing for what we intend them to be testing
for.
Fixes#30144
Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
* Relocate dashboard scuemata to grafana-schema
* Update assorted tests, vars with dashboard path
* Remove crufty commented var
* Not sure...why that failed
* Remove prefix from base dashboard path var
* Move cue/ui remnants into grafana-schema
* Update import paths in plugin models
* Remove mudball, add package statements
* Remove cuegen.sh
Wooooo we ain't got no codegen
* Revert "Remove mudball, add package statements"
This reverts commit 9bed3098f1.
* Tidy up all the cue files
* Move dashboard scuemata into scuemata/ subdir
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
* Make validate-schema CLI command work vs. disk
Previously, `grafana-cli cue validate-schema` would only validate the
CUE files that had been included at compile time. Now, the command
requires passing a grafana checkout root path, and will actually check
against live files on disk.
* Exempt validateScuemata from sec linter
* Nit: validateScuemata is a simpler, better name
* Try to fix validate-scuemata build step
* Add test for devenv resources
* Refactor validation tests for grokkability
* Devenv dashboards error-tracking script
* Refactor to use cueerrors.Details()
* Further test refinement
* Close major elements of dashboard schema
* Centralize dashboard validation tests
General dashboard validation testing belongs in the load package.
* Better names for error context on glue CUE code
* Fixup validate-resource
Do only one of base or dist, and fix copied docs.
* Skip the devenv test
* Remove test for validateResources
* Fix shellcheck
* Backend linter
Co-authored-by: sam boyer <sdboyer@grafana.com>
* add uninstall flow
* add install flow
* small cleanup
* smaller-footprint solution
* cleanup + make bp start auto
* fix interface contract
* improve naming
* accept version arg
* ensure use of shared logger
* make installer a field
* add plugin decommissioning
* add basic error checking
* fix api docs
* making initialization idempotent
* add mutex
* fix comment
* fix test
* add test for decommission
* improve existing test
* add more test coverage
* more tests
* change test func to use read lock
* refactoring + adding test asserts
* improve purging old install flow
* improve dupe checking
* change log name
* skip over dupe scanned
* make test assertion more flexible
* remove trailing line
* fix pointer receiver name
* update comment
* add context to API
* add config flag
* add base http api test + fix update functionality
* simplify existing check
* clean up test
* refactor tests based on feedback
* add single quotes to errs
* use gcmp in tests + fix logo issue
* make plugin list testing more flexible
* address feedback
* fix API test
* fix linter
* undo preallocate
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* Update docs/sources/administration/configuration.md
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
* fix linting issue in test
* add docs placeholder
* update install notes
* Update docs/sources/plugins/marketplace.md
Co-authored-by: Marcus Olsson <marcus.olsson@hey.com>
* update access wording
* add more placeholder docs
* add link to more info
* PR feedback - improved errors, refactor, lock fix
* improve err details
* propagate plugin version errors
* don't autostart renderer
* add H1
* fix imports
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
Co-authored-by: Marcus Olsson <marcus.olsson@hey.com>
* fall back to any architecture when getting plugin's checksum
* Update pkg/cmd/grafana-cli/commands/install_command.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* remove typo
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* use sha256 checksum instead of md5
* Chore: Rewrite ldap login test to standard library (#29998)
* Chore: Rewrite ldap login test to standard library
* Preserve original ldap enabled setting after test
* Chore: Rewrite models alert test to standard library (#30021)
* Chore: Rewrite models dashboard acl test to standard library (#30022)
* Chore: Rewrite models dashboards test to standard library (#30023)
* Chore: Rewrite login auth test to standard library (#29985)
* Chore: Rewrite login auth test to standard library
* Use assert.Empty when empty string expected
* Chore: Rewrite brute force login protection test to standard library (#29986)
* Update pkg/cmd/grafana-cli/services/api_client.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* Update pkg/cmd/grafana-cli/services/api_client.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* use sha256 checksum instead of md5
* Update pkg/cmd/grafana-cli/services/api_client.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* Update pkg/cmd/grafana-cli/services/api_client.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* grafana-cli: Remove MD5
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Emil Hessman <emil@hessman.se>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* pkg/cmd: Check errors
* pkg/cmd: Make sure server waits on services, even in case of error
* pkg/cmd: Inform of error to show help
* pkg/cmd: Only warn on failure to send systemd notification
* pkg/cmd: Don't log errors stemming from context cancelation
* pkg/cmd: Don't fail if unable to write to systemd
* Feature: Introduce a grafana-cli wrapper
When our users install the *nix packed version of grafana, tendency is to use the services and scripts installed as part of the package for grafana-server. These leverage the default configuration options by specifying the several default paths.
This introduces a similar approach for the grafana-cli binary. We exposed it through a wrapper to ensure a proper configuration is in place. To enable that, we add the .real suffix to the original binary (grafana-cli.real) and then use a bash script named grafana-cli as the wrapper.
* Make the config and homepath flags global
* Introduce `configOverrides` as a global flag
This flag allows us to pass configuration overrides as a string.
The string follows the convention of configuration arguments separated by a space e.g. "cfg:default.paths.data=/dev/nullX cfg:default.paths.logs=/dev/nullX"
Also, it is backwards compatible with similar the previous configuration method through tailing arguments. Tailing arguments take presedence over the configuration options string.
* Only log configuration information in debug mode
* Move the grafana-cli binary to $GRAFANA_HOME/bin
As part of the package install process, we copy all the release files and
directories into the grafana home directory. This includes the /bin folder
from where we copied the binaries into their respective destinations.
After that, the /bin folder gets deleted as we don't want to keep
duplicates of the binaries around.
As part of this commit, we moved the re-creation of /bin within
grafana-home and the copy of the original binary (again) after the
folder gets deleted.
`grafana-cli` uses the third-party library to define the flags and not
the standard library. Using `flag.Parse` conflicts with the defined
flags from our third-party library.
In the case where `flag.Parse` is used, the CLI assumes that definitions
provided are not needed and will not define them; producing errors of
the kind `flag provided but not defined --example-flag`.
Using the context to read any arguments (including flags) is the
recommended approach by the third-party library.
grafana-cli should allow configuration overrides to be received from the command line. e.g.
```
grafana-cli admin reset-password cfg:default.paths.logs=custom/log/directory/
```
Seems like we missed the inclusion of `flag.Parse` as we run the command, to be able to consume them.
Additionally, it'll be useful for the user to know whenever these are being overriden or not - hence the addition of logging the configuration to be used as we run the command.
* Chore: explore possibilities of using makefile
This is an exploratory commit - I wanted to see how
revive/gosec linters could be integrated with makefile and our build scripts.
Looks better then I expected :)
* Chore: make revive happy
Revive execution was not supplied with path, if you restore there is couple
errors that were popping up - so I fixed them
* Chore: make revive happy
Since we do not like some of the default golint rules,
this commit proposes to use https://github.com/mgechev/revive.
And potential revive speed-up should't hurt :).
Right now, presented config (./conf/revive.toml) is permissive,
we might improve it over time however. Fixes for found revive
issues in the code are very limited so it wouldn't be large to review.
Also in this commit:
* Add annotations for makefile commands and declare phony targets
* Rename "gometalinter" script and CI command to "lint"
since we are doing there a bit more then using gometalinter package
* Add Makefile rules to .editorconfig
* Documentation which mentioned "golint" replaced with revive
Fixes#16109
Ref #16160
* refactor: tracing service refactoring
* refactor: sqlstore to instance service
* refactor: sqlstore & registory priority
* refactor: sqlstore refactor wip
* sqlstore: progress on getting tests to work again
* sqlstore: progress on refactoring and getting tests working
* sqlstore: connection string fix
* fix: not sure why this test is not working and required changing expires
* fix: updated grafana-cli
* wip: start on refactoring settings
* settings: progress on settings refactor
* refactor: progress on settings refactoring
* fix: fixed failing test
* settings: moved smtp settings from global to instance
This fixes:
pkg/cmd/grafana-cli/commands/upgrade_all_command.go:56:3: should merge variable declaration with assignment on next line (S1021)
pkg/login/ldap.go:406:4: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:87:2: should merge variable declaration with assignment on next line (S1021)
pkg/services/sqlstore/migrator/dialect.go:165:2: should merge variable declaration with assignment on next line (S1021)
pkg/tsdb/cloudwatch/metric_find_query_test.go:185:2: should merge variable declaration with assignment on next line (S1021)
This fixes:
pkg/cmd/grafana-cli/commands/install_command.go:36:11: should use fmt.Errorf(...) instead of errors.New(fmt.Sprintf(...)) (S1028)
pkg/models/org_user.go:53:11: should use fmt.Errorf(...) instead of errors.New(fmt.Sprintf(...)) (S1028)
pkg/services/notifications/mailer.go:138:16: should use fmt.Errorf(...) instead of errors.New(fmt.Sprintf(...)) (S1028)
This fixes:
build.go:553:6: should omit comparison to bool constant, can be simplified to !strings.Contains(path, ".sha256") (S1002)
pkg/cmd/grafana-cli/commands/ls_command.go:27:5: should omit comparison to bool constant, can be simplified to !pluginDirInfo.IsDir() (S1002)
pkg/components/dynmap/dynmap_test.go:24:5: should omit comparison to bool constant, can be simplified to !value (S1002)
pkg/components/dynmap/dynmap_test.go:122:14: should omit comparison to bool constant, can be simplified to b (S1002)
pkg/components/dynmap/dynmap_test.go:125:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/components/dynmap/dynmap_test.go:128:14: should omit comparison to bool constant, can be simplified to !b (S1002)
pkg/models/org_user.go:51:5: should omit comparison to bool constant, can be simplified to !(*r).IsValid() (S1002)
pkg/plugins/datasource/wrapper/datasource_plugin_wrapper_test.go:77:12: should omit comparison to bool constant, can be simplified to !haveBool (S1002)
pkg/services/alerting/conditions/evaluator.go:23:9: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:48:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/evaluator.go:91:5: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/conditions/query.go:56:6: should omit comparison to bool constant, can be simplified to !reducedValue.Valid (S1002)
pkg/services/alerting/extractor.go:107:20: should omit comparison to bool constant, can be simplified to !enabled.MustBool() (S1002)
pkg/services/alerting/notifiers/telegram.go:222:41: should omit comparison to bool constant, can be simplified to this.UploadImage (S1002)
pkg/services/sqlstore/apikey.go:58:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/apikey.go:72:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:66:33: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard.go:175:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:311:13: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/dashboard.go:444:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:472:12: should omit comparison to bool constant, can be simplified to !exists (S1002)
pkg/services/sqlstore/dashboard.go:554:32: should omit comparison to bool constant, can be simplified to !cmd.Overwrite (S1002)
pkg/services/sqlstore/dashboard_snapshot.go:83:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/plugin_setting.go:39:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:34:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:111:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:136:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/quota.go:213:6: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/temp_user.go:129:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:157:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:182:5: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:191:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:212:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/services/sqlstore/user.go:307:12: should omit comparison to bool constant, can be simplified to !has (S1002)
pkg/social/generic_oauth.go:185:5: should omit comparison to bool constant, can be simplified to !s.extractToken(&data, token) (S1002)
pkg/tsdb/mssql/mssql.go:148:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:212:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mssql/mssql.go:247:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mssql/mssql.go:274:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mssql/mssql.go:282:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:221:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/mysql/mysql.go:256:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/mysql/mysql.go:283:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/mysql/mysql.go:291:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:134:39: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:201:6: should omit comparison to bool constant, can be simplified to !query.Model.Get("fillNull").MustBool(false) (S1002)
pkg/tsdb/postgres/postgres.go:236:56: should omit comparison to bool constant, can be simplified to ok (S1002)
pkg/tsdb/postgres/postgres.go:263:7: should omit comparison to bool constant, can be simplified to !exist (S1002)
pkg/tsdb/postgres/postgres.go:271:8: should omit comparison to bool constant, can be simplified to !exist (S1002)
TLS was not being verified in a number of places:
- connections to grafana.com
- connections to OAuth providers when TLS client authentication was
enabled
- connections to self-hosted Grafana installations when using the CLI
tool
TLS should always be verified unless the user explicitly enables an
option to skip verification.
Removes some instances where `InsecureSkipVerify` is explicitly set to
`false`, the default, to help avoid confusion and make it more difficult
to regress on this fix by accident.
Adds a `--insecure` flag to `grafana-cli` to skip TLS verification.
Adds a `tls_skip_verify_insecure` setting for OAuth.
Adds a `app_tls_skip_verify_insecure` setting under a new `[plugins]`
section.
I'm not super happy with the way the global setting is used by
`pkg/api/app_routes.go` but that seems to be the existing pattern used.
The zip lib is throwing panics sometimes when the response is malformed.
The cli will now try to download the zip file up to three times before
aborting. The cli gives a better error message and informes the user
about retrying.
closes#4651
