* Chore: Migrate to use buf for protobuf generation
* fix codeowners
* fixes after review
* docs/tooling for installing buf
* add back codeowners
* Update upgrading-dependencies.md
Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
* sdk v0.229.0
---------
Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
* Remove different constructors and only use NewNamespaceID
* AdminUser: check typed namespace id
* Identity: Add convinient function to parse valid user id when type is either user or service account
* Annotations: Use typed namespace id instead
* added changes for parent teams in team_ids. see https://github.com/grafana/grafana/issues/85916
Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
* added unit test
Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
* addressed review comments to consider case where parent object can be null
Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
* addressed review comment
Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
---------
Signed-off-by: Syed Nihal <syed.nihal@nokia.com>
* Choose mode. Add log field on top level dualwriter
* Add logs
* Turn DualWriter into a full interface. Fix tests
* Lint
* Use struct for dualWriter interface
* Use struct
* Default should be legacyStore for all entities
* Fix test. Get rid of extra concrete type
* Remove comment
* Add comment
* Temp set dualwriter mode 2 for playlists while configs are not in place
* Add modes type + add comment on what each mode does
* Don't require watcher interface for now
* Use storage implementation on mode 2
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Pass log values to the context
* test
* Update pkg/apiserver/rest/dualwriter_mode3.go
Co-authored-by: Dan Cech <dcech@grafana.com>
---------
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* clean up error handling in postDashboard and remove UserDisplayDTO
* replace GetUserUID with GetUID and GetNamespacedUID, enforce namespace constant type
* lint fix
* lint fix
* more lint fixes
* Add OrgRoleMapper
* Address feedback, add more tests
* Prevent resetting assignments when global org mapping fails
* Provide a parsing function instead of recalculating the org_mapping on every mapping
* Introduce strict role parsing/mapping
* Introduce MappingConfiguration
* Handle other edge case
* Add tests
* lint
* Apply documentation update suggestions
* Apply GetDefaultOrgMapping suggestions from code review
* Apply suggestions for cleaning up unnecessary err in ParseOrgMappingSettings
* Apply suggestions from code review
* Address feedback suggestions
* Cleanup
* Reduce cognitive complexity of ParseOrgMappingSettings
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
* Get custom cloud list from grafana-azure-sdk-go for frontend settings
* Remove getAzureClouds() and call CustomClouds() directly
* remove unused type
Storage Server: avoid incorrect parsing of connection string
In storage server, a connection string is generated to connect to the
backing database. In case of Postgres, we were generating the string
like `"user=grafana pass= host=postgres.example port=4567"`.
This triggered an edge case in `pq` (the go postgres driver) to parse
`pass` to be equal to `host=postgres.example`, and host being reset to
default value.
Using single quotes in the connection string fixed this.
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
* Alerting: Implement GetStatus in the remote Alertmanager struct
* update tests
* fix tests, extract AlertmanagerConfig from PostableConfig
* get the remote AM config instead of the Grafana one from the remote AM
* pass grafana AM config in test
* return error in GetStatus instead of logging it (internal AM)
* Add auth checks and test
* Check user is authorized to view rule and add tests
* Change naming
* Update Swagger params
* Update auth test and swagger gen
* Update swagger gen
* Change response to GettableExtendedRuleNode
* openapi3-gen
* Update tests with refactors models pkg
* Fix mode 2 List test
* Set origin timestamp during conversion to k8s resource
* Add instructions for updating a playlist
* Handle partial deletions of a collection in mode 2
* moving folders to slog
* trying to fix the tests
* fix per-logger filters in slog
* allow using slog.Default()
* bring cfg back to keep the pr small
* fix tests
* back to the roots
* - Added migration to add a new collumn UID to both migration and migration_run tables.
- Added migration to set UID for records already existent in the database before adding a new column.
- Added UID field to CloudMigration and CloudMigrationRun dtos (model.go)
* Fix db migration
* Updating store and model
* Updating API to use UID
* fix typo
* update openapi specs and generated endpoints
* fix spec
* update openapi specs and generated endpoints
* Fixing db mapping
* Fix frontend
* fix migration
* remove migration uid from run fetch endpoint
* Revert "merge"
This reverts commit d654e4c530, reversing
changes made to 5fe0b483eb.
* manual merge
* rename some funcs for consistency
* make interfaces consistent
* validate uids
* update generated frontend api
* fix enterprise spec
* manually resolve api
* try again
* try yet again
* once more
---------
Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: joshhunt <josh@trtr.co>
* Add origin keys to List request
* Check origin keys requirement in the sql entity server's List method
* Refactor mode 2 List
* Check origin keys before making Storage List call
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* CloudWatch: Fix metric query with group by not being labelled in alerts
* just use one key for the labels
* not needed
* unused function
* add tests
* pr comments
* fetch dimensions to build labels for MetricQuery type queries
* pr comments
* group cache related tests and use fresh cache for non-cache related tests
* don't cache empty values
* Move alertmanager api silence code to separate files unchanged
* Replace with silence model instead interface
---------
Co-authored-by: Matt Jacobson <matthew.jacobson@grafana.com>
* Chore: Upgrade/fix deps after thema removal
* github.com/deepmap/oapi-codegen v1.14.0
* github.com/deepmap/oapi-codegen v1.15.0
* github.com/deepmap/oapi-codegen v1.16.0
* fix for dep used in enterprise
* github.com/deepmap/oapi-codegen/v2 v2.0.0
* oapi-codegen/v2 v2.1.0, kin-openapi v0.122.0, sdk
* keep kin-openapi at v0.122.0 and allow SDK to use v0.124.0
* remove github.com/deepmap/oapi-codegen v1 dep
* fix owner
* add back github.com/deepmap/oapi-codegen v1 dep
* upgrade github.com/influxdata/influxdb-client-go/v2 to get rid of deepmap/oapi-codegen
* migrate to oapi-codegen/runtime
* sdk
* sdk v0.227.0
* Alerting: Improve error when receiver used by rule is deleted
* Remove RuleUID from public error and data
* Improve fallback error in am config post
* Refactor to expand to time intervals
* Fix message on unchecked errors to be same as before
* [WIP]: Force lowercase login/email for user CRUD
* warn and remove use of userCaseInsensitiveLogin check
* remove log warning
* reimplementation of the caseinsensitive
* need to decide if we want the conflict check or not
* remvoved the tests for conflict user by getEmail, getLogin
* added tests for user lowercase migration
* wip: emails next
* tests for email lowercasing
* review comments
* optimized login and email lookup before migrating
* Use RoleType from org package
* Move to identity package and re-export from authn
* Replace usage of top level functions for identity
Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
* make cfg private in sqlstore
* fix db init in tests
* fix case
* fix folder test init
* fix imports
* make another Cfg private
* remove another Cfg
* remove unused variable
* use store cfg, it has side-effects
* fix mutated cfg in tests
* Create basic feature toggle
* Rename context to reflect it contains query history and query library
* Update icons and variants
* Rename hooks
* Update tests
* Fix mock
* Add tracking
* Turn button into a toggle
* Make dropdown active as well
This is required to have better UI and an indication of selected state in split view
* Update Query Library icon
This is to make it consistent with the toolbar button
* Hide query history button when query library is available
This is to avoid confusing UX with 2 button triggering the drawer but with slightly different behavior
* Make the drawer bigger for query library
To avoid confusion for current users and test it internally a bit more it's behind a feature toggle. Bigger drawer may obstruct the view and add more friction in the UX.
* Fix tests
The test was failing because queryLibraryAvailable was set to true for tests. This change makes it more explicit what use case is being tested
* Remove active state underline from the dropdown
* Allow closing Query Library drawer from the toolbar
* Simplify dropdown design
* Alerting: Implement SaveAndApplyConfiguration in the forked Alertmanager struct
* call SaveAndApplyConfig on the remote first, log errors for the internal
* add comments explaining why we ignore errors in the internal AM
* restore go.work.sum
This splits the request handlers into two functions, one which is the actual
handler and one which is independent from the Grafana `ReqContext` object. This
is to make it easier to reuse the implementation in other code.
Part of the refactoring changes the functions which get query parameters from
the request to operate on a `url.Values` instead of the request object.
The change also makes the code consistently use `req.Form` instead of a
combination of `req.URL.Query()` and `req.Form`, though I have left
`api_ruler` as-is to avoid this PR growing too large.
* implement SaveAndApplyConfig in the remote Alertmanager struct
* remove ID from CreateGrafanaAlertmanagerConfig call
* decrypt, test that we decrypt, refactor
* fix duplicated declaration in test
* rephrase comment, remove unnecessary conversion to slice of bytes
* fix test
* Alerting: Implement SaveAndApplyDefaultConfig in the forked Alertmanager (remote primary)
* log the error for the internal AM instead of returning it
* Add setting for changing shortlink expiration time
* Add docs, add better language
* put all the numbers in the duration 🤷
* 🙄
* update language to be correct and clear
* Add max limit and more documentation
This test has been skipped for a long time, so it doesn't work anymore. I've
fixed the test so it works again, but left some tests disabled which were
apparently flaky. If we see the other test cases flaking, we'll have to
disable it again.
Fixes:
- Use fake access control for most test cases, and real one for FGAC test cases.
- Check that "file" in API responses the full folder path, not folder title.
Currently the grafana cli plugin commands are not reacting to the --config parameter. This PR make it possible to use config to define the plugin endpoints via config as an alternative to providing the --repo flag.
* Simplify proxy dialer creation
- Set new dialer on connector
- Create MSSQL connector in a similar fashion to postgres
* Update test
* Fix lint
* More lint
* Use correct driver name
* Alerting: Consistently return Prometheus-style responses from rules APIs.
This commit is part refactor and part fix. The /rules API occasionally returns
error responses which are inconsistent with other error responses. This fixes
that, and adds a function to map from Prometheus error type and HTTP code.
* Fix integration tests
* Linter happiness
* Make linter more happy
* Fix up one more place returning non-Prometheus responses
* Alerting: Implement SaveAndApplyDefaultConfig in the remote Alertmanager struct
* send the hash of the encrypted configuration
* tests, default config hash in AM struct
* add missing default config to test
* restore build directory
* go work file...
* fix broken test
* remove unnecessary conversion to []byte
* go work again...
* make things work again with latest main branch changes
* update error messages in tests for decrypting config
Preparing these functions to be used by some other part of the codebase,
which does not have a `contextmodel.ReqContext`, only the normal request
structure (`url.Values`, etc). This is slightly messy because of how
Grafana allows url parameters to be in the URL or in the request body,
so we need to make sure to invoke the form parsing logic in `ReqContext`.
* Alerting: Optimize rule status gathering APIs when a limit is applied.
The frontend very commonly calls the `/rules` API with `limit_alerts=16`. When
there are a very large number of alert instances present, this API is quite
slow to respond, and profiling suggests that a big part of the problem is
sorting the alerts by importance, in order to select the first 16.
This changes the application of the limit to use a more efficient heap-based
top-k algorithm. This maintains a slice of only the highest ranked items whilst
iterating the full set of alert instances, which substantially reduces the
number of comparisons needed. This is particularly effective, as the
`AlertsByImportance` comparison is quite complex.
I've included a benchmark to compare the new TopK function to the existing
Sort/limit strategy. It shows that for small limits, the new approach is
much faster, especially at high numbers of alerts, e.g.
100K alerts / limit 16: 1.91s vs 0.02s (-99%)
For situations where there is no effective limit, sorting is marginally faster,
therefore in the API implementation, if there is either a) no limit or b) no
effective limit, then we just sort the alerts as before. There is also a space
overhead using a heap which would matter for large limits.
* Remove commented test cases
* Make linter happy
* Alerting: Fix simplified routing custom group by override
Custom group by overrides for simplified routing were missing required fields
GroupBy and GroupByAll normally set during upstream Route validation.
This fix ensures those missing fields are applied to the generated routes.
* Inline GroupBy and GroupByAll initialization instead of normalize after
* move run migration to the cloudmigrationimpl layer
* add migration run list logic down a layer
* remove useless comments
* pull cms calls into their own service
* Add update methods for the dual writer
* improve errors
* [WIP] add tests for the update method
* Move example package to its own package so it can be used by the rest package. Finish tests
* Add codeowners
* Use Pod as a dummy resource
* :int
* Lint
* [REVIEW] rename var
* [REVIEW] don't rely on legacy storage at all in mode4
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* [REVIEW] improve comment
* Update pkg/apiserver/rest/dualwriter_mode1.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* [REVIEW] improve mode3
* Lint
* Move test files
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Lint
* Update pkg/apiserver/rest/dualwriter_mode4_test.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Fix error
* Lint
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Don't set the flag to true as updatedObj creates an object in case it's not found
* Lint
* Lint
* Add tests on update
* Lint
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* IAM: fix many error messages in access-related code to provide more information
* Remove debug statement
* Refactor resourcepermissions package to use errutil
* Replace a few more errors with errutil and wrap errors found in users and teams services
* Apply diff of openAPI spec
sqlstore: improve recursive CTE support detection
Vitess returns a not supported error, not a parse error
Co-authored-by: Derek Perkins <derek@nozzle.io>
* WIP: add mocks for dual writer
* Test dualwriter mode1
* Re-add non implementation errors
* Use testify assert
* Write tests for mode2.
* Lint
* Remove comment
* Update pkg/apiserver/rest/dualwriter_mode2_test.go
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* [REVIEW improve readability and call fn legacy fn
* [REVIEW] rename mocks and setupFns
* [REVIEW add missing test case
* [REVIEW] add test case with getting object from storage and not legacy
* Add deleteCollection
* Test deleteCollection
* Rename file
---------
Co-authored-by: Arati R. <33031346+suntala@users.noreply.github.com>
* Alerting: Fix simplified routes '...' groupBy creating invalid routes
There were a few ways to go about this fix:
1. Modifying our copy of upstream validation to allow this
2. Modify our notification settings validation to prevent this
3. Normalize group by on save
4. Normalized group by on generate
Option 4. was chosen as the others have a mix of the following cons:
- Generated routes risk being incompatible with upstream/remote AM
- Awkward FE UX when using '...'
- Rule definition changing after save and potential pitfalls with TF
With option 4. generated routes stay compatible with external/remote AMs, FE
doesn't need to change as we allow mixed '...' and custom label groupBys, and
settings we save to db are the same ones requested.
In addition, it has the slight benefit of allowing us to hide the internal
implementation details of `alertname, grafana_folder` from the user in the
future, since we don't need to send them with every FE or TF request.
* Safer use of DefaultNotificationSettingsGroupBy
* Fix missed API tests
* only users with Grafana Admin role can grant/revoke Grafana Admin role
* check permissions to user amdin endpoints globally
* allow checking global permissions for service accounts
* use a middleware for checking whether the caller is Grafana Admin
* User: remove unused function
* User: Remove UpdatePermissions and support IsGrafanaAdmin flag in Update function instead
* User: Remove Disable function and use Update instead
* add history links for monaco completion provider folder
* add history links for monaco query field folder
* add history links for components folder
* add history links for configuration folder
* add history links for dashboard json folder
* add history links for gcopypaste folder
* add history link for variableMigration
* add history link for querybuilder/components/metrics-modal folder
* add history link for querybuilder/components/promqail folder
* add history links for querybuilder/components folder
* add history links for querybuilder/hooks folder
* add history links for querybuilder/shared folder
* add history links for querybuilder folder
* add history links for querycache folder
* add history links for src folder
* use frontend package and custom auth in module.ts
* remove files and fix import issues
* remove usePrometheusFrontendPackage
* remove extra files
* update betterer
* remove extra files after rebase
* fix betterer for rebase
* fix e2e flakiness
* Add `Service. IsClientEnabled` and `Client.IsEnabled` functions
* Implement `IsEnabled` function for authn clients
* Implement `IsClientEnabled` function for authn services
* copied files
* add copy of Pointer
* fix the API
* forward resample to the namespaced code
* moved the aligning-code to the parent
* call namespaced resample directly
* lint fix
* lint fix
* switch to plugin-sdk-go resample
* adjusted import path
* AuthN: Add NamespaceID struct. We should replace the usage of encoded namespaceID with this one
* AuthN: Add optional interface that clients can implement to be able to resolve identity for a namespace
* Authn: Implement IdentityResolverClient for api keys
* AuthN: use idenity resolvers
Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
* added pkg/util/rinq package to handle queueing of notifications
* fix linters
* Fix typo in comment
Co-authored-by: Dan Cech <dcech@grafana.com>
* improve allocation strategy for Enqueue; remove unnecessary clearing of slice
* Update pkg/util/ringq/dyn_chan_bench_test.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* Update pkg/util/ringq/ringq.go
Co-authored-by: Dan Cech <dcech@grafana.com>
* refactor to move stats and shrinking into Ring
* add missing error assertions in tests
* add missing error assertions in tests and linting issues
* simplify controller closed check
* improve encapsulation of internal state in Ring
* use (*Ring).Len for clarity instead of stats
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
Fixes so that auth middleware trace/span doesn't wrap the next handlers.
Allow tracing service name to be overridden in standalone apiserver.
Change k8s api tracing operation name to KubernetesAPI from
grafana-apiserver (which is the service name)
* Making versioncheck url rely on config instead of being hardcoded
* Update pkg/services/updatechecker/plugins.go
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* making the names a bit more generic and using url.url library
* fixing tests
* fixing linting
---------
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
* Authn: Resolve authenticate by and auth id when fethcing signed in user
* Change logout client interface to only take Requester interface
* Session: Fetch external auth info when authenticating sessions
* Use authenticated by from identity
* Move call to get auth-info into session client and use GetAuthenticatedBy in various places
