* (WIP) Refactor the ImageStore interface to work with our latest alerting repository
* update alerting package
* refactor, new URLExists method in ImageProvider
* tests for the new methods
* fix linter warnings
* use alertingImages as an alias for grafana/alerting/images
* logs about image uris and not found images
* nerf image not found logs
* extract duplicated code to getImageFromURI() method
* refactor getImageFromURI()
* add index on url
* add comment about migration log
* sync generated files
* Moving POC files from #64283 to a new branch
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* Adding missing permission definition
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* Force the service instantiation while client isn't merged
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* Merge conf with main
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* Leave go-sqlite3 version unchanged
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* tidy
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* User SearchUserPermissions instead of SearchUsersPermissions
* Replace DummyKeyService with signingkeys.Service
* Use user🆔<id> as subject
* Fix introspection endpoint issue
* Add X-Grafana-Org-Id to get_resources.bash script
* Regenerate toggles_gen.go
* Fix basic.go
* Add GetExternalService tests
* Add GetPublicKeyScopes tests
* Add GetScopesOnUser tests
* Add GetScopes tests
* Add ParsePublicKeyPem tests
* Add database test for GetByName
* re-add comments
* client tests added
* Add GetExternalServicePublicKey tests
* Add other test case to GetExternalServicePublicKey
* client_credentials grant test
* Add test to jwtbearer grant
* Test Comments
* Add handleKeyOptions tests
* Add RSA key generation test
* Add ECDSA by default to EmbeddedSigningKeysService
* Clean up org id scope and audiences
* Add audiences to the DB
* Fix check on Audience
* Fix double import
* Add AC Store mock and align oauthserver tests
* Fix test after rebase
* Adding missing store function to mock
* Fix double import
* Add CODEOWNER
* Fix some linting errors
* errors don't need type assertion
* Typo codeowners
* use mockery for oauthserver store
* Add feature toggle check
* Fix db tests to handle the feature flag
* Adding call to DeleteExternalServiceRole
* Fix flaky test
* Re-organize routes comments and plan futur work
* Add client_id check to Extended JWT client
* Clean up
* Fix
* Remove background service registry instantiation of the OAuth server
* Comment cleanup
* Remove unused client function
* Update go.mod to use the latest ory/fosite commit
* Remove oauth2_server related configs from defaults.ini
* Add audiences to DTO
* Fix flaky test
* Remove registration endpoint and demo scripts. Document code
* Rename packages
* Remove the OAuthService vs OAuthServer confusion
* fix incorrect import ext_jwt_test
* Comments and order
* Comment basic auth
* Remove unecessary todo
* Clean api
* Moving ParsePublicKeyPem to utils
* re ordering functions in service.go
* Fix comment
* comment on the redirect uri
* Add RBAC actions, not only scopes
* Fix tests
* re-import featuremgmt in migrations
* Fix wire
* Fix scopes in test
* Fix flaky test
* Remove todo, the intersection should always return the minimal set
* Remove unecessary check from intersection code
* Allow env overrides on settings
* remove the term app name
* Remove app keyword for client instead and use Name instead of ExternalServiceName
* LogID remove ExternalService ref
* Use Name instead of ExternalServiceName
* Imports order
* Inline
* Using ExternalService and ExternalServiceDTO
* Remove xorm tags
* comment
* Rename client files
* client -> external service
* comments
* Move test to correct package
* slimmer test
* cachedUser -> cachedExternalService
* Fix aggregate store test
* PluginAuthSession -> AuthSession
* Revert the nil cehcks
* Remove unecessary extra
* Removing custom session
* fix typo in test
* Use constants for tests
* Simplify HandleToken tests
* Refactor the HandleTokenRequest test
* test message
* Review test
* Prevent flacky test on client as well
* go imports
* Revert changes from 526e48ad45
* AuthN: Change the External Service registration form (#68649)
* AuthN: change the External Service registration form
* Gen default permissions
* Change demo script registration form
* Remove unecessary comment
* Nit.
* Reduce cyclomatic complexity
* Remove demo_scripts
* Handle case with no service account
* Comments
* Group key gen
* Nit.
* Check the SaveExternalService test
* Rename cachedUser to cachedClient in test
* One more test case to database test
* Comments
* Remove last org scope
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* Update pkg/services/oauthserver/utils/utils_test.go
* Update pkg/services/sqlstore/migrations/oauthserver/migrations.go
Remove comment
* Update pkg/setting/setting.go
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
---------
Co-authored-by: Mihály Gyöngyösi <mgyongyosi@users.noreply.github.com>
* update to alerting 20230418161049-5f374e58cb32
* rename renamed structs in https://github.com/grafana/alerting/pull/73
* update ValidateContactPoint to use BuildReceiverConfiguration
* update logger factory according to changes
* rewrite integration builder
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* RBAC: Stop reading enabeld from ini file and always set to true
* Migrations: Add a migration for rbac to reset data migrations if rbac
was disabled
* If rbac was disabled we reset the data and data migrations that rbac
has to perform to get it to a correct state
* Migrator: Store migration logs on migrator and add function to clear it from the
in-memory stored logs
* update tests
---------
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
* remove dashboard previews backend
* remove dashboard previews backend
* bring back the migration
* bring back the migration
* bring back the migration
* replace receiver errors with one from alerting
* add the converter to alerting models
* update buildReceiverIntegration to accept GrafanaReceiver
---------
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Add features dependency to SQLBuilder
* Add features dependency to AccessControlDashboardPermissionFilter
* Add test for folder inheritance
* Dashboard permissions: Return recursive query
* Recursive query for inherited folders
* Modify search builder
* Adjust db.SQLBuilder
* Pass flag to SQLbuilder if CTEs are supported
* Add support for mysql < 8.0
* Add benchmarking for search with nested folders
* Set features to AlertStore
* Update pkg/infra/db/sqlbuilder.go
Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
* Set features to LibraryElementService
* SQLBuilder tests with nested folder flag set
* Apply suggestion from code review
Co-authored-by: IevaVasiljeva <ieva.vasiljeva@grafana.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* extract function
* use context logger
* put alert to general folder if folder is missing
* move folderHelper init
* add test
* Update pkg/services/sqlstore/migrations/ualert/ualert.go
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
---------
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
* improvements for starred dashboard search
* fix workflows for the case when no dashboards are starred
* PR feedback (don't query DB if starred dashboards and requested but no starred IDs are found) and linting
* return empty list not null in case of no starred dashboards
* return empty list not null in case of no starred dashboards pt 2
* return empty list not null in case of no starred dashboards pt 3
This commit fixes a serious bug in Grafana 9.4.1 where on upgrade
a migration would pause all existing alert rules and change the
default value of the column to true.
* Mark AM configuration as applied
* add missing checks, make linter happy
* fix deadlock, mark as valid on save and on load
* mark configurations only if needed
* check error after applyConfig()
* code review comments
* code review changes
* more code review changes
* clean HistoricConfigFromAlertConfig function
* Nested folders: Do not skip integration tests
* SQLStore: Fix folder migration
It reduces the length of the title column to be equal with the respective
dashboard column.
* Use suggested value for uid
* update the snapshot
* use __expr__
* replace all -100 with __expr__
* update snapshot
* more changes
* revert redundant change
* Use expr.DatasourceUID where it's possible
* generate files
* Add field in alert_rule model, add state to alert_instance model, and state to eval
* Remove paused state from eval package
* Skip paused alert rules in scheduler
* Add migration to add is_paused field to alert_rule table
* Convert to postable alerts only if not normal, pernding, or paused
* Handle paused eval results in state manager
* Add Paused state to eval package
* Add paused alerts logic in scheduler
* Skip alert on scheduler
* Remove paused status from eval package
* Apply suggestions from code review
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Remove state
* Rethink schedule and manager for paused alerts
* Change return to continue
* Remove unused var
* Rethink alert pausing
* Paused alerts storing annotations
* Only add one state transition
* Revert boolean method renaming refactor
* Revert take image refactor
* Make registry errors public
* Revert method extraction for getting a folder title
* Revert variable renaming refactor
* Undo unnecessary changes
* Revert changes in test
* Remove IsPause check in PatchPartiLAlertRule function
* Use SetNormal to set state
* Fix text by returning to old behaviour on alert rule deletion
* Add test in schedule_unit_test.go to test ticks with paused alerts
* Add coment to clarify usage of context.Background()
* Add comment to clarify resetStateByRuleUID method usage
* Move rule get to a more limited scope
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: George Robinson <george.robinson@grafana.com>
* rum gofmt on pkg/services/ngalert/schedule/schedule.go
* Remove defer cancel for context
* Update pkg/services/ngalert/models/instance_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/models/testing.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/schedule/schedule_unit_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/schedule/schedule_unit_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/models/instance_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* skip scheduler rule state clean up on paused alert rule
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Fix mock in test
* Add (hopefully) final suggestions
* Use error channel from recordAnnotationsSync to cancel context
* Run make gen-cue
* Place pause alert check in channel update after version check
* Reduce branching un update channel select
* Add if for error and move code inside if in state manager ResetStateByRuleUID
* Add reason to logs
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Do not delete alert rule routine, just exit on eval if is paused
* Reduce branching and create-close a channel to avoid deadlocks
* Separate state deletion and state reset (includes history saving)
* Add current pause state in rule route in scheduler
* Split clearState and bring errCh closer to RecordStatesAsync call
* Change rule to ruleMeta in RecordStatesAsync
* copy state to be able to modify it
* Add timeout to context creation
* Shorten the timeout
* Use resetState is rule is paused and deleteState if rule is not paused
* Remove Empty state reason
* Save every rule change in historian
* Add tests for DeleteStateByRuleUID and ResetStateByRuleUID
* Remove useless line
* Remove outdated comment
Co-authored-by: George Robinson <george.robinson@grafana.com>
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>
* Chore: Move team models to models pkg
* Fix ACL tests
* More ACL tests
* Change Id to ID in conflict user command test
* Remove team from models
* Fix ac test lint
* Alerting: Improve legacy migration to include send reminder & frequency
Legacy channel frequency is migrated to the channel's migrated route's
repeat interval if send reminder is true. If send reminder is false, we
pseudo-disable the repeat interval by setting it to a large value (1y).
If there were no default channels, the root notification policy is still
created with the default 4h repeat interval.
* PermissionFilter: Handle all search type and only check one action for dashboards
* PermissionFilter: Still handle multiple action but take short cut when
only one action is required
* Update config store to split between active and history tables
* Migrations to fix up indexes
* Implement migration from old format to new
* Move add migrations call
* Delete duplicated rows
* Explicitly map fields
* Quote the column name because it's a reserved word
* Lift migrations to top
* Use XORM for nearly everything, avoid any non trivial raw SQL
* Touch up indexes and zero out IDs on move
* Drop TODO that's already completed
* Fix assignment of IDs
* Alerting: Prevent short uid collision in legacy migration when db is case-insensitive
Two factors come into play that cause sporadic uid conflicts during legacy alert migration:
- MySQL and MySQL-compatible backends use case-insensitive collation.
- Our short uid generator is not a uniform RNG and generates uids in such a way that generations in quick succession have a higher probability of creating similar uids.
Normally we would be guaranteed unique short uid generation, however if the source alphabet contains
duplicate characters (for example, if we use case-insensitive comparison) this guarantee is void.
Generating even ~1000 uids in quick succession is nearly guaranteed to create a case-insensitive
duplicate.
* introduce alias for json.RawMessage with name RawMessage. This is needed to keep raw JSON and implement a marshaler for YAML, which does not seem to be used but there are tests that fail.
* replace usage of simplejson with RawMessage in NotificationChannelConfig
* remove usage of simplejson in tests
* change migration code to convert simplejson to raw message
* chore: remove unused test helper from sqlstore
TimeNow() is no longer used in any tests in this package.
* chore: move sqlstore.SQLBuilder to the infra/db package
This required some minor refactoring; we need to be a little more explicit about passing around the dialect and engine. On the other hand, that's a few fewer uses of the `dialect` global constant!
* chore: move UserDeletions into the only package using it
* cleanup around moving sqlbuilder
* remove dialect and sqlog global vars
* rename userDeletions to serviceAccountDeletions
* Update config store to split between active and history tables
* Migrations to fix up indexes
* Implement migration from old format to new
* Move add migrations call
* Delete duplicated rows
* Explicitly map fields
* Quote the column name because it's a reserved word
* Lift migrations to top
* introduce Logger interface local to channles + implementaton that wraps the Grafana logger
* make NewFactoryConfig accept LoggerFactory
* add logger field to FactoryConfig
* update usages of log.Logger to internal interface
* move original stats service into a separate package
* add stats service to wire
* move GetAdminStats
* switch to using stats.Service
* add missing package
* fix api tests
* Remove DeleteUser from sqlstore
* Use Delete instead of DeleteUser
* Remove unused method
* Remove DeleteUserSession from sqlstore
* Add fake for DeleteUseraccessControl
* Use user service, add fakes to tests
* Add comments to sqlstore
* Correct typo
* Add quota service initialisation
* Add config to acimpl initialisation
* Add routing to initialisation
* Making user provideStore private
* Use InTransaction instead of session
* Add cfg to userimpl
* Fix lint
* Make ProvideStore public again - enterprise tests
* Fix back ProvideStore to public
* Wrap merge user in transaction
* Delete DeleteUserAccessControl use DeleteUSerPermissions instead
* Add feature mgmt into acimpl
* DeleteUserAccessControl from ac database
* Remove case insensitive clause
This change preallocates slices and maps where the size of the data is known before the object is created.
Co-authored-by: Joe Blubaugh <joe.blubaugh@grafana.com>
* LoginAttemps: Remove from sqlstore mock
* LoginAttemps: Move from models package to service package
* LoginAttemps: Implement functionallity from brute force login in service
* LoginAttemps: Call service
* LoginAttempts: Update name and remove internal functions
* LoginAttempts: Add tests
* LoginAttempt: Add service fake
* LoginAttempt: Register service as a background_services and remove job
from cleanup service
* LoginAttemps: Remove result from command struct
* LoginAttempt: No longer pass pointers
* add db migration at service start time
* make changes for the 3 db
* revert migrator
* fix feature toggle check
Co-authored-by: Serge Zaitsev <serge.zaitsev@grafana.com>
* Remove URL-based alertmanagers from endpoint config
* WIP
* Add migration and alertmanagers from admin_configuration
* Empty comment removed
* set BasicAuth true when user is present in url
* Remove Alertmanagers from GET /admin_config payload
* Remove URL-based alertmanager configuration from UI
* Fix new uid generation in external alertmanagers migration
* Fix tests for URL-based external alertmanagers
* Fix API tests
* Add more tests, move migration code to separate file, and remove possible am duplicate urls
* Fix edge cases in migration
* Fix imports
* Remove useless fields and fix created_at/updated_at retrieval
Co-authored-by: George Robinson <george.robinson@grafana.com>
Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
* clean up and document integration test convention
* clarify integration test conventions
* clean up integration tests that don't follow convention
* mark testIntegration* functions as helpers to avoid confusion
* add: added a repeat migration w. fixed checks for permissions
* add: migration to migrations
* refactor: fix migration instead of making a new one
* fix: removed the old id
* fix: keep old name but change id
* add: migration for patched previous migration
* add: migration from missing file
* Remove methods from sqlstore
* Remove commented out code
* Remove GetUserById from tests
* Adjust fake for get user profile
* Adjust test
* Adjust go mod files
* Try fix test
* Test adjustment
* Adjust test 2
* Remove commented out code
* feat: add new Folder table migration
Add a new folder table to support the Nested Folders feature.
https://github.com/grafana/grafana/issues/56880
* register nested folders feature flag (unused)
* feat: nested folder service (experiment)
This commit adds a NestedFolderSvc interface and stubbed out implementation as an alternative to the existing folder service. This is an experimental feature to try out different methods for backwards compatibility and parallelization, so that Grafana can continue to store folders in the existing (non-nested) manner while also using the new nested folder service.
Eventually the new service will (hopefully) become _the_ service, at which point the legacy service can be deprecated (or remain, with the new service methods replacing the original. whatever makes sense at the time).
* nested folders: don't run the new migration
This commit removes the nested folder migration from the list of active migrations so we can merge this branch and continue development without impacting Grafana instances built off main.
* Chore: Delete unused mocks
* Add back used methods
* Add back used mocks 2
* Add back used mocks 3
* Add back used mocks 4
* Add back the alerting mocks
* Remove mock variables
* Add bacl needed mock var
* chore: add alias for InitTestDB and Session
Adds an alias for the sqlstore InitTestDB and Session, and updates tests using these to reduce dependencies on the sqlstore.Store.
* next pass of removing sqlstore imports
* last little bit
* remove mockstore where possible
* Alerting: Improve notification policies created during migration
Previously, migrated legacy alerts were connected to notification policies through
a `rule_uid` label in a 1:1 fashion. While this correctly mimicked pre-migration routing,
it didn't create a notification policy structure that is easy to view/modify. In addition,
having one policy per migrated alert is, in some ways, counter to the recommended approach of
Unified Alerting.
This change replaces `rule_uid`-based migrated notification policies with a private
label called `__contacts__`. This label stores a list of double quoted strings containing the names of
all contact points an AlertRule should route to (based on legacy notification channels). Finally,
one notification policy is created per contact point with each matching AlertRules via regex on this
`__contacts__` label.
The result is a simpler, clearer, and easier to modify notification policy structure, with the
added benefit that you can see which contact points an AlertRule is being routed to from the
AlertRule creation page.
* RBAC: Change the generate uid function to be deterministic so we can avoid collision
* RBAC: Use fmt.Errorf
* RBAC: Add comment
* RBAC: Export GenerateManagedRoleUID
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
These new transactions are off by default, guarded by the feature toggle "alertingBigTransactions"
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
