* preview of work so far
* stylistic improvements
* fix linters
* remove golden tests, they may cause the system to be too rigid to changes
* remove unnecessary code for golden tests
* remove white space mangling in Execute
* also handle output data, improve API, examples and docs
* add helper methods
* fix interface
* Add feature toggles for mode 2 and 3 playlist dual writing
* Make current mode customised based on kind
* Check feature flags when initialising dual writer
* Fix linting
* Refactor NewDualWriter
* Updated headings and to align with release life cycle stages and adding release info
* Update generated files
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
---------
Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
* registers sql metrics for unified storage and removes single quotes from postgres connection string format
* uses db logger
* only register US sql metrics when connecting to another postgres or mysql db
* use engine
* Create recording rule fields in model
* Add migration
* Write to database, support in version table
* extend fingerprint
* Force fields to be empty on validate
* Another storage spot, tests for fingerprint
* Explicitly set defaults in provisioning API
* Tests for main API validation
* Add diff tests even though fields are unpopulated for now
* Use struct tag approach instead of FromDB/ToDB hooks as it better handles nulls when deserializing
* test for deser
* Backout RecordTo for now since it's not decided in the doc
* back out of migration too
* Drop datasourceref for now
* address linter complaints
* Try a single outer struct with all fields embedded
* add action set resolver
* rename variables
* some fixes and some tests
* more tests
* more tests, and put action set storing behind a feature toggle
* undo change from cfg to feature mgmt - will cover it in a separate PR due to the amount of test changes
* fix dependency cycle, update some tests
* add one more test
* fix for feature toggle check not being set on test configs
* linting fixes
* check that action set name can be split nicely
* clean up tests by turning GetActionSetNames into a function
* undo accidental change
* test fix
* more test fixes
* Chore: Migrate to use buf for protobuf generation
* fix codeowners
* fixes after review
* docs/tooling for installing buf
* add back codeowners
* Update upgrading-dependencies.md
Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
* sdk v0.229.0
---------
Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
* Remove different constructors and only use NewNamespaceID
* AdminUser: check typed namespace id
* Identity: Add convinient function to parse valid user id when type is either user or service account
* Annotations: Use typed namespace id instead
* clean up error handling in postDashboard and remove UserDisplayDTO
* replace GetUserUID with GetUID and GetNamespacedUID, enforce namespace constant type
* lint fix
* lint fix
* more lint fixes
* Add OrgRoleMapper
* Address feedback, add more tests
* Prevent resetting assignments when global org mapping fails
* Provide a parsing function instead of recalculating the org_mapping on every mapping
* Introduce strict role parsing/mapping
* Introduce MappingConfiguration
* Handle other edge case
* Add tests
* lint
* Apply documentation update suggestions
* Apply GetDefaultOrgMapping suggestions from code review
* Apply suggestions for cleaning up unnecessary err in ParseOrgMappingSettings
* Apply suggestions from code review
* Address feedback suggestions
* Cleanup
* Reduce cognitive complexity of ParseOrgMappingSettings
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Storage Server: avoid incorrect parsing of connection string
In storage server, a connection string is generated to connect to the
backing database. In case of Postgres, we were generating the string
like `"user=grafana pass= host=postgres.example port=4567"`.
This triggered an edge case in `pq` (the go postgres driver) to parse
`pass` to be equal to `host=postgres.example`, and host being reset to
default value.
Using single quotes in the connection string fixed this.
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
* Alerting: Implement GetStatus in the remote Alertmanager struct
* update tests
* fix tests, extract AlertmanagerConfig from PostableConfig
* get the remote AM config instead of the Grafana one from the remote AM
* pass grafana AM config in test
* return error in GetStatus instead of logging it (internal AM)
* Add auth checks and test
* Check user is authorized to view rule and add tests
* Change naming
* Update Swagger params
* Update auth test and swagger gen
* Update swagger gen
* Change response to GettableExtendedRuleNode
* openapi3-gen
* Update tests with refactors models pkg
* Fix mode 2 List test
* Set origin timestamp during conversion to k8s resource
* Add instructions for updating a playlist
* Handle partial deletions of a collection in mode 2
* moving folders to slog
* trying to fix the tests
* fix per-logger filters in slog
* allow using slog.Default()
* bring cfg back to keep the pr small
* fix tests
* back to the roots
* - Added migration to add a new collumn UID to both migration and migration_run tables.
- Added migration to set UID for records already existent in the database before adding a new column.
- Added UID field to CloudMigration and CloudMigrationRun dtos (model.go)
* Fix db migration
* Updating store and model
* Updating API to use UID
* fix typo
* update openapi specs and generated endpoints
* fix spec
* update openapi specs and generated endpoints
* Fixing db mapping
* Fix frontend
* fix migration
* remove migration uid from run fetch endpoint
* Revert "merge"
This reverts commit d654e4c530, reversing
changes made to 5fe0b483eb.
* manual merge
* rename some funcs for consistency
* make interfaces consistent
* validate uids
* update generated frontend api
* fix enterprise spec
* manually resolve api
* try again
* try yet again
* once more
---------
Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: joshhunt <josh@trtr.co>
* Add origin keys to List request
* Check origin keys requirement in the sql entity server's List method
* Refactor mode 2 List
* Check origin keys before making Storage List call
---------
Co-authored-by: Dan Cech <dcech@grafana.com>
* Move alertmanager api silence code to separate files unchanged
* Replace with silence model instead interface
---------
Co-authored-by: Matt Jacobson <matthew.jacobson@grafana.com>
* Alerting: Improve error when receiver used by rule is deleted
* Remove RuleUID from public error and data
* Improve fallback error in am config post
* Refactor to expand to time intervals
* Fix message on unchecked errors to be same as before
* [WIP]: Force lowercase login/email for user CRUD
* warn and remove use of userCaseInsensitiveLogin check
* remove log warning
* reimplementation of the caseinsensitive
* need to decide if we want the conflict check or not
* remvoved the tests for conflict user by getEmail, getLogin
* added tests for user lowercase migration
* wip: emails next
* tests for email lowercasing
* review comments
* optimized login and email lookup before migrating
* Use RoleType from org package
* Move to identity package and re-export from authn
* Replace usage of top level functions for identity
Co-authored-by: Misi <mgyongyosi@users.noreply.github.com>
* make cfg private in sqlstore
* fix db init in tests
* fix case
* fix folder test init
* fix imports
* make another Cfg private
* remove another Cfg
* remove unused variable
* use store cfg, it has side-effects
* fix mutated cfg in tests
* Create basic feature toggle
* Rename context to reflect it contains query history and query library
* Update icons and variants
* Rename hooks
* Update tests
* Fix mock
* Add tracking
* Turn button into a toggle
* Make dropdown active as well
This is required to have better UI and an indication of selected state in split view
* Update Query Library icon
This is to make it consistent with the toolbar button
* Hide query history button when query library is available
This is to avoid confusing UX with 2 button triggering the drawer but with slightly different behavior
* Make the drawer bigger for query library
To avoid confusion for current users and test it internally a bit more it's behind a feature toggle. Bigger drawer may obstruct the view and add more friction in the UX.
* Fix tests
The test was failing because queryLibraryAvailable was set to true for tests. This change makes it more explicit what use case is being tested
* Remove active state underline from the dropdown
* Allow closing Query Library drawer from the toolbar
* Simplify dropdown design
* Alerting: Implement SaveAndApplyConfiguration in the forked Alertmanager struct
* call SaveAndApplyConfig on the remote first, log errors for the internal
* add comments explaining why we ignore errors in the internal AM
* restore go.work.sum
This splits the request handlers into two functions, one which is the actual
handler and one which is independent from the Grafana `ReqContext` object. This
is to make it easier to reuse the implementation in other code.
Part of the refactoring changes the functions which get query parameters from
the request to operate on a `url.Values` instead of the request object.
The change also makes the code consistently use `req.Form` instead of a
combination of `req.URL.Query()` and `req.Form`, though I have left
`api_ruler` as-is to avoid this PR growing too large.
* implement SaveAndApplyConfig in the remote Alertmanager struct
* remove ID from CreateGrafanaAlertmanagerConfig call
* decrypt, test that we decrypt, refactor
* fix duplicated declaration in test
* rephrase comment, remove unnecessary conversion to slice of bytes
* fix test
* Alerting: Implement SaveAndApplyDefaultConfig in the forked Alertmanager (remote primary)
* log the error for the internal AM instead of returning it
* Add setting for changing shortlink expiration time
* Add docs, add better language
* put all the numbers in the duration 🤷
* 🙄
* update language to be correct and clear
* Add max limit and more documentation
This test has been skipped for a long time, so it doesn't work anymore. I've
fixed the test so it works again, but left some tests disabled which were
apparently flaky. If we see the other test cases flaking, we'll have to
disable it again.
Fixes:
- Use fake access control for most test cases, and real one for FGAC test cases.
- Check that "file" in API responses the full folder path, not folder title.
* Alerting: Consistently return Prometheus-style responses from rules APIs.
This commit is part refactor and part fix. The /rules API occasionally returns
error responses which are inconsistent with other error responses. This fixes
that, and adds a function to map from Prometheus error type and HTTP code.
* Fix integration tests
* Linter happiness
* Make linter more happy
* Fix up one more place returning non-Prometheus responses
* Alerting: Implement SaveAndApplyDefaultConfig in the remote Alertmanager struct
* send the hash of the encrypted configuration
* tests, default config hash in AM struct
* add missing default config to test
* restore build directory
* go work file...
* fix broken test
* remove unnecessary conversion to []byte
* go work again...
* make things work again with latest main branch changes
* update error messages in tests for decrypting config
Preparing these functions to be used by some other part of the codebase,
which does not have a `contextmodel.ReqContext`, only the normal request
structure (`url.Values`, etc). This is slightly messy because of how
Grafana allows url parameters to be in the URL or in the request body,
so we need to make sure to invoke the form parsing logic in `ReqContext`.
* Alerting: Optimize rule status gathering APIs when a limit is applied.
The frontend very commonly calls the `/rules` API with `limit_alerts=16`. When
there are a very large number of alert instances present, this API is quite
slow to respond, and profiling suggests that a big part of the problem is
sorting the alerts by importance, in order to select the first 16.
This changes the application of the limit to use a more efficient heap-based
top-k algorithm. This maintains a slice of only the highest ranked items whilst
iterating the full set of alert instances, which substantially reduces the
number of comparisons needed. This is particularly effective, as the
`AlertsByImportance` comparison is quite complex.
I've included a benchmark to compare the new TopK function to the existing
Sort/limit strategy. It shows that for small limits, the new approach is
much faster, especially at high numbers of alerts, e.g.
100K alerts / limit 16: 1.91s vs 0.02s (-99%)
For situations where there is no effective limit, sorting is marginally faster,
therefore in the API implementation, if there is either a) no limit or b) no
effective limit, then we just sort the alerts as before. There is also a space
overhead using a heap which would matter for large limits.
* Remove commented test cases
* Make linter happy
