* Move alertmanager api silence code to separate files unchanged
* Replace with silence model instead interface
---------
Co-authored-by: Matt Jacobson <matthew.jacobson@grafana.com>
* Alerting: Improve error when receiver used by rule is deleted
* Remove RuleUID from public error and data
* Improve fallback error in am config post
* Refactor to expand to time intervals
* Fix message on unchecked errors to be same as before
* [WIP]: Force lowercase login/email for user CRUD
* warn and remove use of userCaseInsensitiveLogin check
* remove log warning
* reimplementation of the caseinsensitive
* need to decide if we want the conflict check or not
* remvoved the tests for conflict user by getEmail, getLogin
* added tests for user lowercase migration
* wip: emails next
* tests for email lowercasing
* review comments
* optimized login and email lookup before migrating
* Use RoleType from org package
* Move to identity package and re-export from authn
* Replace usage of top level functions for identity
Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
* make cfg private in sqlstore
* fix db init in tests
* fix case
* fix folder test init
* fix imports
* make another Cfg private
* remove another Cfg
* remove unused variable
* use store cfg, it has side-effects
* fix mutated cfg in tests
* Create basic feature toggle
* Rename context to reflect it contains query history and query library
* Update icons and variants
* Rename hooks
* Update tests
* Fix mock
* Add tracking
* Turn button into a toggle
* Make dropdown active as well
This is required to have better UI and an indication of selected state in split view
* Update Query Library icon
This is to make it consistent with the toolbar button
* Hide query history button when query library is available
This is to avoid confusing UX with 2 button triggering the drawer but with slightly different behavior
* Make the drawer bigger for query library
To avoid confusion for current users and test it internally a bit more it's behind a feature toggle. Bigger drawer may obstruct the view and add more friction in the UX.
* Fix tests
The test was failing because queryLibraryAvailable was set to true for tests. This change makes it more explicit what use case is being tested
* Remove active state underline from the dropdown
* Allow closing Query Library drawer from the toolbar
* Simplify dropdown design
* Alerting: Implement SaveAndApplyConfiguration in the forked Alertmanager struct
* call SaveAndApplyConfig on the remote first, log errors for the internal
* add comments explaining why we ignore errors in the internal AM
* restore go.work.sum
This splits the request handlers into two functions, one which is the actual
handler and one which is independent from the Grafana `ReqContext` object. This
is to make it easier to reuse the implementation in other code.
Part of the refactoring changes the functions which get query parameters from
the request to operate on a `url.Values` instead of the request object.
The change also makes the code consistently use `req.Form` instead of a
combination of `req.URL.Query()` and `req.Form`, though I have left
`api_ruler` as-is to avoid this PR growing too large.
* implement SaveAndApplyConfig in the remote Alertmanager struct
* remove ID from CreateGrafanaAlertmanagerConfig call
* decrypt, test that we decrypt, refactor
* fix duplicated declaration in test
* rephrase comment, remove unnecessary conversion to slice of bytes
* fix test
* Alerting: Implement SaveAndApplyDefaultConfig in the forked Alertmanager (remote primary)
* log the error for the internal AM instead of returning it
* Add setting for changing shortlink expiration time
* Add docs, add better language
* put all the numbers in the duration 🤷
* 🙄
* update language to be correct and clear
* Add max limit and more documentation
This test has been skipped for a long time, so it doesn't work anymore. I've
fixed the test so it works again, but left some tests disabled which were
apparently flaky. If we see the other test cases flaking, we'll have to
disable it again.
Fixes:
- Use fake access control for most test cases, and real one for FGAC test cases.
- Check that "file" in API responses the full folder path, not folder title.
* Alerting: Consistently return Prometheus-style responses from rules APIs.
This commit is part refactor and part fix. The /rules API occasionally returns
error responses which are inconsistent with other error responses. This fixes
that, and adds a function to map from Prometheus error type and HTTP code.
* Fix integration tests
* Linter happiness
* Make linter more happy
* Fix up one more place returning non-Prometheus responses
* Alerting: Implement SaveAndApplyDefaultConfig in the remote Alertmanager struct
* send the hash of the encrypted configuration
* tests, default config hash in AM struct
* add missing default config to test
* restore build directory
* go work file...
* fix broken test
* remove unnecessary conversion to []byte
* go work again...
* make things work again with latest main branch changes
* update error messages in tests for decrypting config
Preparing these functions to be used by some other part of the codebase,
which does not have a `contextmodel.ReqContext`, only the normal request
structure (`url.Values`, etc). This is slightly messy because of how
Grafana allows url parameters to be in the URL or in the request body,
so we need to make sure to invoke the form parsing logic in `ReqContext`.
* Alerting: Optimize rule status gathering APIs when a limit is applied.
The frontend very commonly calls the `/rules` API with `limit_alerts=16`. When
there are a very large number of alert instances present, this API is quite
slow to respond, and profiling suggests that a big part of the problem is
sorting the alerts by importance, in order to select the first 16.
This changes the application of the limit to use a more efficient heap-based
top-k algorithm. This maintains a slice of only the highest ranked items whilst
iterating the full set of alert instances, which substantially reduces the
number of comparisons needed. This is particularly effective, as the
`AlertsByImportance` comparison is quite complex.
I've included a benchmark to compare the new TopK function to the existing
Sort/limit strategy. It shows that for small limits, the new approach is
much faster, especially at high numbers of alerts, e.g.
100K alerts / limit 16: 1.91s vs 0.02s (-99%)
For situations where there is no effective limit, sorting is marginally faster,
therefore in the API implementation, if there is either a) no limit or b) no
effective limit, then we just sort the alerts as before. There is also a space
overhead using a heap which would matter for large limits.
* Remove commented test cases
* Make linter happy
* Alerting: Fix simplified routing custom group by override
Custom group by overrides for simplified routing were missing required fields
GroupBy and GroupByAll normally set during upstream Route validation.
This fix ensures those missing fields are applied to the generated routes.
* Inline GroupBy and GroupByAll initialization instead of normalize after
* move run migration to the cloudmigrationimpl layer
* add migration run list logic down a layer
* remove useless comments
* pull cms calls into their own service
* IAM: fix many error messages in access-related code to provide more information
* Remove debug statement
* Refactor resourcepermissions package to use errutil
* Replace a few more errors with errutil and wrap errors found in users and teams services
* Apply diff of openAPI spec
sqlstore: improve recursive CTE support detection
Vitess returns a not supported error, not a parse error
Co-authored-by: Derek Perkins <derek@nozzle.io>
* Alerting: Fix simplified routes '...' groupBy creating invalid routes
There were a few ways to go about this fix:
1. Modifying our copy of upstream validation to allow this
2. Modify our notification settings validation to prevent this
3. Normalize group by on save
4. Normalized group by on generate
Option 4. was chosen as the others have a mix of the following cons:
- Generated routes risk being incompatible with upstream/remote AM
- Awkward FE UX when using '...'
- Rule definition changing after save and potential pitfalls with TF
With option 4. generated routes stay compatible with external/remote AMs, FE
doesn't need to change as we allow mixed '...' and custom label groupBys, and
settings we save to db are the same ones requested.
In addition, it has the slight benefit of allowing us to hide the internal
implementation details of `alertname, grafana_folder` from the user in the
future, since we don't need to send them with every FE or TF request.
* Safer use of DefaultNotificationSettingsGroupBy
* Fix missed API tests
* only users with Grafana Admin role can grant/revoke Grafana Admin role
* check permissions to user amdin endpoints globally
* allow checking global permissions for service accounts
* use a middleware for checking whether the caller is Grafana Admin
* User: remove unused function
* User: Remove UpdatePermissions and support IsGrafanaAdmin flag in Update function instead
* User: Remove Disable function and use Update instead
* add history links for monaco completion provider folder
* add history links for monaco query field folder
* add history links for components folder
* add history links for configuration folder
* add history links for dashboard json folder
* add history links for gcopypaste folder
* add history link for variableMigration
* add history link for querybuilder/components/metrics-modal folder
* add history link for querybuilder/components/promqail folder
* add history links for querybuilder/components folder
* add history links for querybuilder/hooks folder
* add history links for querybuilder/shared folder
* add history links for querybuilder folder
* add history links for querycache folder
* add history links for src folder
* use frontend package and custom auth in module.ts
* remove files and fix import issues
* remove usePrometheusFrontendPackage
* remove extra files
* update betterer
* remove extra files after rebase
* fix betterer for rebase
* fix e2e flakiness
* Add `Service. IsClientEnabled` and `Client.IsEnabled` functions
* Implement `IsEnabled` function for authn clients
* Implement `IsClientEnabled` function for authn services
* AuthN: Add NamespaceID struct. We should replace the usage of encoded namespaceID with this one
* AuthN: Add optional interface that clients can implement to be able to resolve identity for a namespace
* Authn: Implement IdentityResolverClient for api keys
* AuthN: use idenity resolvers
Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
Fixes so that auth middleware trace/span doesn't wrap the next handlers.
Allow tracing service name to be overridden in standalone apiserver.
Change k8s api tracing operation name to KubernetesAPI from
grafana-apiserver (which is the service name)
* Making versioncheck url rely on config instead of being hardcoded
* Update pkg/services/updatechecker/plugins.go
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* making the names a bit more generic and using url.url library
* fixing tests
* fixing linting
---------
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* Authn: Resolve authenticate by and auth id when fethcing signed in user
* Change logout client interface to only take Requester interface
* Session: Fetch external auth info when authenticating sessions
* Use authenticated by from identity
* Move call to get auth-info into session client and use GetAuthenticatedBy in various places
* Alerting: Persist silence state immediately on Create/Delete
Persists the silence state to the kvstore immediately instead of waiting for the
next maintenance run. This is used after Create/Delete to prevent silences from
being lost when a new Alertmanager is started before the state has persisted.
This can happen, for example, in a rolling deployment scenario.
* Fix test that requires real data
* Don't error if silence state persist fails, maintenance will correct
* Revert "Revert "Add FolderUID for library elements" (#83776)"
This reverts commit 0dfdb2ae47.
* Fix bug, dashboard id and library element fodler_id are the corresponding values
Dashboard table hold both dahboards and tables
* extract genericService from RuleService just to reuse it later
* implement silence service
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
* Storage server runs own instrumentation server if its the sole target. Starts adding some sample metrics for now.
* adds metric for failed optimistic locks
* refactors metrics registration to own method on service for testability. Adds tests.
* Register sql storage server metrics from within the service
* fixes test
* troubleshooting drone test failures. Maybe timing when starting instrumentation server?
* Waits until instrumentation server has started. Updates tests.
* defer wont get called unless theres an error. removing.
* wait for instrumentation server to be running
* linter - close res body
* use port 3000 for metrics and removes test metric inc() call
* fixes test - updates port
* refactors module server to provide an instrumentation server module when there is no ALL or CORE target provided and running as single target
* make instrumentation server a dependency of all modules that do not run their own http server
* adds module server test
* adds tests for instrumentation service and removes old tests that aren't needed
* ignore error in test
* uses helper to start and run service
* when running wait on ctx done or http server err
* wait for http server
* removes println
* updates module server test to be integration test
* require no error in goroutine
* skips integration test when GRAFANA_TEST_DB not defined
* move http server start into start, verify returned content
* make test error when run fails
* try waiting longer and see if drone tests pass
* update integration test mysql creds to match drone
* go back to only waiting half second
* debug log drone mysql connection string
* use same db connection config as drone
* try using same hostname as drone
* cant use localhost as mysql hostname in drone tests. Need to parse it from the cfg db connection string
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* support continue at specified resource version
* detect whether list continue pages need to use entity_history, remove BatchRead, expand selectQuery helper
* refactor continue token handling
* fix tests, increase history chunk size
* lint fix
* Alerting: Make retention period configurable for the notification log
* update sample.ini
* fix outdated comment (on disk -> kvstore)
* skip checking cyclomatic complexity for ReadUnifiedAlertingSettings
* Add email and email_verified to id token if identity is a user
* Add endpoint to trigger email verification for user
* Add function to clear stored id tokens and use it when email verification is completed
* E2C: Resources table refactor
* update swagger spec with enums
* use native resource item type, rather than our mock type
* unit tests for resources table
* update spec
* Feature Flags: use FeatureToggles interface where possible
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
* Replace TestFeatureToggles with existing WithFeatures
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
---------
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
* replace sqlstore with db interface in a few packages
* remove from stats
* remove sqlstore in admin test
* remove sqlstore from api plugin tests
* fix another createUser
* remove sqlstore in publicdashboards
* remove sqlstore from orgs
* clean up orguser test
* more clean up in sso
* clean up service accounts
* further cleanup
* more cleanup in accesscontrol
* last cleanup in accesscontrol
* clean up teams
* more removals
* split cfg from db in testenv
* few remaining fixes
* fix test with bus
* pass cfg for testing inside db as an option
* set query retries when no opts provided
* revert golden test data
* rebase and rollback
* Add feature toggle
* Add filters and group by variables by default to all new dashboards
* Nits
* Tests
* Rename feature toggle to newDashboardWithFiltersAndGroupBy
* E2C: Start connecting on-prem to real apis
* actually run the migration
* show resources
* basic dashboards resources
* show dashboard title
* remove console logs
* cleanup 1
* i18n
* disconnect
* i18n
* restore type
* use fixed format
* fix
* Implement run migration endpoint
* Refactor RunMigration method into separate methods
* Save migration runs fix lint
* Minor changes
* Refactor how to use cms endpoint
* fix interface
* complete merge
* add individual items
* adds tracing to getMigration
* linter
* updated swagger definition with the latest changes
* CloudMigrations: Implement core API handlers for cloud migrations and migration runs (#85407)
* implement delete
* add auth token encryption
* implement token validation
* call token validation during migration creation
* implement get migration status
* implement list migration runs
* fix bug
* finish parse domain func
* fix urls
* fix typo
* fix encoding and decoding
* remove double decryption
* add missing slash
* fix id returned by create function
* inject missing services
* finish implementing (as far as I can tell right now) data migration and response handling
* comment out broken test, needs a rewrite
* add a few final touches
* get dashboard migration to work properly
* changed runMigration to a POST
* swagger
* swagger
* swagger
---------
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
Co-authored-by: Michael Mandrus <41969079+mmandrus@users.noreply.github.com>
* reenable ext-jwt-client
* fixup settings struct
* add user and service auth
* lint up
* add user auth to grafana ext
* fixes
* Populate token permissions
Co-authored-by: jguer <joao.guerreiro@grafana.com>
* fix tests
* fix lint
* small prealloc
* small prealloc
* use special namespace for access policies
* fix access policy auth
* fix tests
* fix uncalled settings expander
* add feature toggle
* small feedback fixes
* rename entitlements to permissions
* add authlibn
* allow viewing the signed in user info for non user namespace
* fix invalid namespacedID
* use authlib as verifier for tokens
* Update pkg/services/authn/clients/ext_jwt.go
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
* Update pkg/services/authn/clients/ext_jwt_test.go
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
* fix parameter names
* change asserts to normal package
* add rule for assert
* fix ownerships
* Local diff
* test and lint
* Fix test
* Fix ac test
* Fix pluginproxy test
* Revert testdata changes
* Force revert on test data
---------
Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
* add function to static function to static service
* find email and login claims with jmespath
* rename configuration files
* Replace JWTClaims struct for map
* check for subclaims error
