* Improve error handling for secrets manager plugin
* Update sever lock log for secret migration to Error
* Move plugin started check to a separate function
* Fix nil pointer exception on HasPluginStarted
* extract errors to errors file
* implement oauth server admin assignment
* add server admin tests
* deduplicate autoAssignOrgRole
* deduplicate strict setting
* deduplicate strict setting
* add support for generic oauth
* add role attribute strict support for generic oauth
* add support for github/gitlab
* assignGrafanaAdmin option is here to stay
* unify similar errors
* add config option
* add okta server admin mapping
* remove never used Company attribute
* unify generic oauth role extract with other methods
* case insensitive role match as in azure
* add ini settings
* add server admin to devenv
* remove duplicate fields
* add documentation to oauth
* fix titlecase test
* implement doc feedback
* Alerting: Sanitize invalid label/annotation names for external alertmanagers
Grafana's built-in Alertmanager supports both Unicode label keys and values; however, if using an external
Prometheus Alertmanager label keys must be compatible with their data model.
This means label keys must only contain ASCII letters, numbers, as well as underscores and match the regex
`[a-zA-Z_][a-zA-Z0-9_]*`.
Any invalid characters will now be removed or replaced by the Grafana alerting engine before being sent to
the external Alertmanager according to the following rules:
- `Whitespace` will be removed.
- `ASCII characters` will be replaced with `_`.
- `All other characters` will be replaced with their lower-case hex representation.
* Prefix hex replacements with `0x`
* Refactor for clarity
* Apply suggestions from code review
Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
* feat: allow jwt role to be set
* chore: update documentation
* fix: cr suggestions
* fix: lint issues
* respect org auto assign and default org ID
* add server admin to devenv
Co-authored-by: jguer <joao.guerreiro@grafana.com>
* Change nav structure when topnav is enable to do initial tests with new information architecture
* Support for nested sections
* Updated
* sentance case
* Progress on plugin challange
* Rewrite to functional component
* Progress
* Updates
* Progress
* Progress on things
* missing file
* Fixing issue with runtime, need to use setter way to set component exposed via runtime
* Move PageLayoutType to grafana/data
* Fixing breadcrumb issue, adding more tests
* reverted backend change
* fix recursive issue with cleanup
* super quick attempt
* feature toggle everything
* only construct alertNav if there are navChildren
* fix toggle name
* plugin landing pages poc
* add apps route + put behind feature toggle
* use toIconName
* rename to NavLandingPage
* feature toggle new routes
* don't modify GetServerAdminNode
* some fairly hacky code to check if the plugin has a root page
* remove trailing slash
* Remove single member team restriction
* Add label when permissions list is empty
* Fix unit tests
* Add co-author.
Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
* Refactor fallback to be isolated to plugin secret store
* Check for error value on replace fallback test helper
* Move ResetPlugin from test_helpers.go to plugin.go
* Add check to GetUnwrappedStoreFromCache
* Add fallback GetAll query to WithFallbackEnabled
* Add mutex lock to WithFallbackEnabled
* Add cache to fallback store
* Fix linter issues
* Fix linter issues
* Fix linter issues
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
This change also updates some of our tests so that they run successfully against postgreSQL - we were using random Int64s, but postgres integers, which our tables use, max out at 2^31-1
* Update GetAlertRulesForScheduling to query for folders (if needed)
* Update scheduler's alertRulesRegistry to cache folder titles along with rules
* Update rule eval loop to take folder title from the
* Extract interface RuleStore
* Pre-fetch the rule keys with the version to detect changes, and query the full table only if there are changes.
* Toggle on the mixed mode option
* Ensure switching to mixed gives existing query prev datasource
* WIP - Populate datasource when switching between mixed and not
* WIP - handle change from mixed
* Remove preimport filter, refine filter to work for queries
* WIP debugging datasource transition
* Ensure creating a new query gets target data source if switching with no matches between
* Add mixed datasource to rich history display
* Cleanup console logs, add relevant comments
* Add feature toggle for mixed datasource
* Fix Wrapper tests
* Fix tests!
* Fix test types and add feature tracking
* Remove unnecessary default, remove explore/mixed workarounds for D2E
* Move display text logic to mixed datasource file
* Add in the default query parameters to a generated empty query
* Condense some code
* Apply suggestions from code review
Co-authored-by: Giordano Ricci <me@giordanoricci.com>
* Add more logic around mixed datasource being off for explore
* Build out logic to handle different datasource scenarios
* Add tests
* Finalize last test
* Fix mixed URL with mixed ds off, and relevant test
* Fix datasource to explore workflow
* Add datasource change function, call import queries if needed
* add logic for changing single query ds
Co-authored-by: Giordano Ricci <me@giordanoricci.com>
* metrics: Fix broken DBStats metrics
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
* Register the sqlstats metrics by default
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
* split out plugin manager
* remove whitespace
* fix tests
* split up tests
* updating naming conventions
* simplify manager
* tidy
* add more fakes
* testing time
* add query verif to int test
* renaming
* add process tests
* tidy up manager tests
* add extra case to int test
* add more coverage to store and process tests
* remove comment
* fix capatilization
* init on provide
* remove addfromsource from API
* Navigation Bar: Remove plugins link under Server Admin
* Modify frontend to handle admin plugins as just plugins
* update assets and documentation mentioned path
* Fix copy to remove redundant text
* refactor apis for consistent outputs
* add dashboardUid validation at API layer
* add check for empty dashboardUid on SavePublicDashboard
* remove public dashboard errors from models package.
Co-authored-by: Ezequiel Victorero <evictorero@gmail.com>
* RBAC: Move metadata to own file
* RBAC: Rename test files
* RBAC: Add wildcard structure and helper function to generate wildcards
from prefix
* RBAC: Refactor filter to use WildcardsFromPrefix
* RBAC: Refactor GetResourceMetadata to use WildcardsFromPrefix
Removes various custom headers logic sprinkled around in the backend.
It should automatically be applied to outgoing HTTP requests via the
CustomHeadersMiddleware.
This also removes decryption of SecureJSONData to populate custom
headers in ngalert which seemed to have caused a ton of CPU usage.
* Search: use SQL search as a fallback when bluge indexing is ongoing
* Search: lint
* Search: feedback fixes - return an empty frame with a special name
* Search: revert readiness check query type
* Search: remove println
* remove sleep, get coffee
* Refactor migrations and tests for secrets kvstore
* Use fake secrets store as a shortcut on tests
* Update wire
* Use global migration logger
* Fix ds proxy tests
* Fix linting issues
* Rename data source test setup function
* RBAC: Test evaluation before attaching mutator
* RBAC: Return error if no resolver is found for scope
* RBAC: Sync changes to evaluation in mock
* RBAC: Check for resolver not found error and just fail the evaluation in that case
* initial cut at migration from plugin
* create new migration from plugin
* only migrate to or from, not both
* remove cfg check from plugin migration itself
* update comments, clean up secret after migration
* add better error handling
* hook up REST API with migrations
* Minor fixes
* fix wire injection issue
* modify migrator to access plugin calls directly. create unit tests
* change pre-migration checks in admin api
* stop plugin after migrating from it
* fix compile issues after merge
* add comment about migration
* fix linting issue
* bleh, fix unit test
* fix another unit test
* update plugin error fatal flag after a migration from the plugin
* add extra logging to migration
* make linter happy
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
* update RouteDeleteAlertRules rules to update as a group
* remove expecter from scheduler mock to support variadic function
* create function to check for provisioning status + tests
Co-authored-by: Alexander Weaver <weaver.alex.d@gmail.com>
* First working version of the TraceQL editor with syntax highlighting and autocomplete
* Add feature flag around the new editor option
* Fix tests and cleanup
* Fix misspelling
* RBAC: Rename interface to Store
* RBAC: Move ranme scopeInjector
* RBAC: Rename files to service
* RBAC: Rename to service
* RBAC: Split up accesscontrol into two components
* RBAC: Add DeclareFixedRoles to AccessControl interface
* Wire: Fix wire bindings
* RBAC: Move resolvers to root
* RBAC: Remove invalid test
* RBAC: Inject access control service
* RBAC: Implement the RoleRegistry interface in fake
* installer -> repo
* add semver format checking
* add plugin callbacks in test
* remove newline
* post install only scans new directories
* remove unused stuff
* everything in own package
* add missing cli params
* make grafana version part of the API
* resolve conflicts
* tidy up logger
* fix cli and tidy log statements
* rename log package
* update struct name
* fix linter issue
* fs -> filestore
* reorder imports
* alias import
* fix test
* fix test
* inline var
* revert jsonc file
* make repo dep of manager
* actually inject the thing
* accept all args for compatability checks
* accept compat from store
* pass os + arch vals
* don't inject fs
* tidy up
* tidy up
* merge with main and tidy fs storage
* fix test
* fix packages
* fix comment + field name
* update fs naming
* fixed wire
* remove unused func
* fix mocks
* fix storage test
* renaming
* fix log line
* fix test
* re-order field
* tidying
* add test for update with same version
* fix wire for CLI
* remove use of ioutil
* don't pass field
* small tidy
* ignore code scanning warn
* fix testdata link
* update lgtm code
* coremodels: Convert plugin-metadata schema to a coremodel
* Newer cuetsy; try quoting field name
* Add slot definitions
* Start sketching out pfs package
* Rerun codegen with fixes, new cuetsy
* Catch up dashboard with new cuetsy
* Update to go1.18
* Use new vmuxers in thema
* Add slot system in Go
* Draft finished implementation of pfs
* Collapse slot pkg into coremodel dir; add PluginInfo
* Add the mux type on top of kernel
* Refactor plugin generator for extensibility
* Change models.cue package, numerous debugs
* Bring new output to parity with old
* Remove old plugin generation logic
* Misc tweaking
* Reintroduce generation of shared schemas
* Drop back to go1.17
* Add globbing to tsconfig exclude
* Introduce pfs test on existing testdata
* Make most existing testdata tests pass with pfs
* coremodels: Convert plugin-metadata schema to a coremodel
* Newer cuetsy; try quoting field name
* Add APIType control concept, regen pluginmeta
* Use proper numeric types for schema fields
* Make pluginmeta schema follow Go type breakdown
* More decomposition into distinct types
* Add test case for no plugin.json file
* Fix missing ref to #Dependencies
* Remove generated TS for pluginmeta
* Update dependencies, rearrange go.mod
* Regenerate without Model prefix
* Use updated thema loader; this is now runnable
* Skip app plugin with weird include
* Make plugin tree extractor reusable
* Split out slot lineage load/validate logic
* Add myriad tests for new plugin validation failures
* Add test for zip fixtures
* One last run of codegen
* Proper delinting
* Ensure validation order is deterministic
* Let there actually be sorting
* Undo reliance on builtIn field (#54009)
* undo builtIn reliance
* fix tests
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* coremodels: Convert plugin-metadata schema to a coremodel
* Newer cuetsy; try quoting field name
* Add APIType control concept, regen pluginmeta
* Use proper numeric types for schema fields
* move saving the state to state manager when scheduler stops
* move saving state to ProcessEvalResults
* add GetRuleKey to State
* add LogContext to AlertRuleKey
* Reverse engineering barchart models.cue from models.gen.ts
* Barchart models.cue and import fixes
* Add models.cue for piechart
* Use single-item syntax to dodge cuetsy bug
* Add some docs
Co-authored-by: sam boyer <sdboyer@grafana.com>
* Rename file to store
* Move resource permission specific database functions to
resourcepermissions package
* Wire: Remove interface bind
* RBAC: Remove injection of resourcepermission Store
* RBAC: Export store constructor
* Tests: Use resource permission package to initiate store used in tests
* RBAC: Remove internal types package and move to resourcepermissions
package
* RBAC: Run database tests as itegration tests
* RBAC: Add orgID to DeleteUserPermissions
* RBAC: Refactor query to delete all permissions in specified org, 0
deletes all permissions
* Delete user permission in org when user is removed
* Remove call to delete permissions in frontend
* Remove user permissions if removed orgs is detected during oauth sync
Co-authored-by: Jo <joao.guerreiro@grafana.com>
Using the prefix "Model" on all generated types was adding more
verbosity to already lengthy generated types. This removed that prefix,
only replacing the base generated struct with the literal "Model" to
avoid a redundant name, e.g. dashboard.Dashboard.
* Relocate dashboards test so it runs
* Cover newer schema versions in stripnulls
* Strip nulls and remove iteration fields
* Fix fiscalYearStartMonth bounds
* Fix up and format dashboard schema
* Update thema, tidy Go dependencies
* Change of sqlstore to use sqlx
* Use sqlx in the playlist store
* Refectory of the interface
* update playlist service
* go mod tidy
* some refectory on interface
* fix kyle
* Remove delete suer from store interface
* Remove get signed in user with cache ctx from store interface
* Support options when setting up access control tests
* Fix broken tests
* Fix lint
* Add user fake to middleware
* Fix middleware tests, remove usertest being initialised twice
* Chore: Add Get User Profile to user and Get User Org List to org service
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
* Remove delete suer from store interface
* Remove get signed in user with cache ctx from store interface
* Support options when setting up access control tests
* Fix broken tests
* Fix lint
* Add user fake to middleware
* Fix middleware tests, remove usertest being initialised twice
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
* Chore: Remove disable user and searchusers methods from store interface
* Remove disable batch user from sqlstore interface
* Remove sqlstore from search store
* Fix lint
* Add disabled column to Org's Users table
* fix typo
* Change column order
* Add test for testing whether GetOrgUsers populates the DTO correctly
* Remove type assertion
* add users-manager command
* add users-manager command
* rename files
* refactor: imports and renaming
* Command: add conflict merge user command
- MergeUser will
- replace all user_ids from conflicting users to the chosen userId
- delete users whose user_ids are not the chosen user
- SameIdentification will
- update chosen user with chosen email,login details
- delete users whose user_ids are not the chosen user
* refactor: clean up
* refactor: create structure for read, validate, ingest
* feat: ls and generate-file for conflicting users
* remove usagestats
* added back pkg/services/login/authinfoservice/database/stats.go
* Revert "added back pkg/services/login/authinfoservice/database/stats.go"
This reverts commit 2ba6e3c4d6.
* Revert "remove usagestats"
This reverts commit 1e3fa97810.
* cherry pick
* Revert "cherry pick"
This reverts commit 461626c306.
* fix test
* make lint
* make test run
* clean up and refactored to align with downstream refactoring
* formatting
* refactor: name list instead of ls
* fix: static lint error use trimprefix
* fix: remove unused functions in sqlstore
* fix: remove unused function
* handling of multiple users and resolve discarded users
* fix tests
* fix: bug that did not exclude the blocks
* ioutil is blacklisted
* should not run tests for mysql
* fix tests
* Correlations: Get Single correlations
* Correlations: Get all correlations for given source ds
* Correlations: Get all correlations
* add tests
* add DB indices
* fix lint errors
* remove skip from tests
* use DatasourceService in test
* Chore: Add user service method SetUsingOrg
* Chore: Add user service method GetSignedInUserWithCacheCtx
* Use method GetSignedInUserWithCacheCtx from user service
* Fix lint after rebase
* Fix lint
* Fix lint error
* roll back some changes
* Roll back changes in api and middleware
* Add xorm tags to SignedInUser ID fields
* add depguard rule for ioutil
* replace ioutil.ReadDir with os.ReadDir
* use legacy option in depguard supported in golangci-lint v1.40
* replace ioutil.ReadDir with os.ReadDir
* return error for file info
* Move license check to secret store plugin
* Use shouldUseRemoteSecretsPlugin for migration
* Encapsulate plugin startup logic in sync.Once
* Remove global logger from startup functions
* Fix issues with wire and secrets plugin check
* Remove todo for plugin fatal error
* Rename fatalErr variable to be less confusing
* Fix merge conflicts
* Fix issue with grafana-cli wire and opentsdb
* Remove duplicated import on remote plugin
* Rename plugin check in favor of error return value
* Remove unnecessary import on grafana-cli wireexts_oss
* Remove unnecessary import on grafana wireexts_oss
* Reset sync.Once during test setup
* Remove unrelated opentsdb change on grafana-cli wire
* Readd opentsdb change on grafana-cli wire
This commit changes the cards in Teams notifications from Office 365
Connector cards to Adaptive Cards to fix an issue where images were not
shown in Teams for desktop and web. Since Office 365 Connector cards
are deprecated, it made sense to move to Adapative Cards and fix this
bug at the same time.
The Adaptive Card messages maintain the design of the Office 365
Connector Card with a number of minor differences:
- In Adaptive Card messages the color of the title is red or green
depending on the status of the alerts, where as with Office 365
connector cards there was a colored border at the top of the title
- In Adaptive Card messages the title is bold to make it easier to read
when the color is red or green
- In Adaptive Card messages the thumbnails for images are medium size
if there are more than two images, otherwise large size
* Wire up to full alert rule struct
* Extract group change detection logic to dedicated file
* GroupDiff -> GroupDelta for consistency
* Calculate deltas and handle backwards compatible requests
* Separate changes and insert/update/delete as needed
* Regenerate files
* Don't touch the DB if there are no changes
* Quota checking, delete unused file
* Mark modified records as provisioned
* Validation + a couple API layer tests
* Address linter errors
* Fix issue with UID assignment and rule creation
* Propagate top level group fields to all rules
* Tests for repeated updates and versioning
* Tests for quota and provenance checks
* Fix linter errors
* Regenerate
* Factor out some shared logic
* Drop unnecessary multiple nilchecks
* Use alternative strategy for rolling UIDs on inserted rules
* Fix tests, add back nilcheck, refresh UIDs during test
* Address feedback
* Add missing nil-check
* Move SignedInUser to user service and RoleType and Roles to org
* Use go naming convention for roles
* Fix some imports and leftovers
* Fix ldap debug test
* Fix lint
* Fix lint 2
* Fix lint 3
* Fix type and not needed conversion
* Clean up messages in api tests
* Clean up api tests 2
* Only SLO user if the user is using SAML
* only one source of truth for auth module info
* ensure SAML is also enabled and not only SLO
* move auth module naming to auth module login package
* use constants in other previously unused spots
* pkg/web: closure-style middlewares
Switches the middleware execution model from web.Handlers in a slice to
web.Middleware.
Middlewares are temporarily kept in a slice to preserve ordering, but
prior to execution they are applied, forming a giant call-stack, giving
granular control over the execution flow.
* pkg/middleware: adapt to web.Middleware
* pkg/middleware/recovery: use c.Req over req
c.Req gets updated by future handlers, while req stays static.
The current recovery implementation needs this newer information
* pkg/web: correct middleware ordering
* pkg/webtest: adapt middleware
* pkg/web/hack: set w and r onto web.Context
By adopting std middlewares, it may happen they invoke next(w,r) without
putting their modified w,r into the web.Context, leading old-style
handlers to operate on outdated fields.
pkg/web now takes care of this
* pkg/middleware: selectively use future context
* pkg/web: accept closure-style on Use()
* webtest: Middleware testing
adds a utility function to web/webtest to obtain a http.ResponseWriter,
http.Request and http.Handler the same as a middleware that runs would receive
* *: cleanup
* pkg/web: don't wrap Middleware from Router
* pkg/web: require chain to write response
* *: remove temp files
* webtest: don't require chain write
* *: cleanup
