* extracted in-proc mode to #93124
* allow insecure conns in dev mode + refactoring
* removed ModeCloud, relying on ModeGrpc and stackID instead to discover if we're running in Cloud
* remove the NamespaceAuthorizer would fail in legacy mode. It will be added back in the future.
* use FlagAppPlatformGrpcClientAuth to enable new behavior, instead of legacy
* extracted authz package changes in #95120
* extracted server side changes in #95086
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
* All objects should have an UID
* Now with a different error message
* Simplify create on DW 2: use the same object to write to both storages
* Run only one test
* Add check for status code
* Add name if it's not present in mode2
* Populate UID in legacy
* Remove logs and commented code
* Change dualwriter1
* Remove commented code
* Fix list test
* remove get on update from dualwriter 2
* Get object before updating. Better var renaming
* Finish rebasing
* Comment test
* Uncomment tests
* Update legacy first. Add preconditions
* Remove preconditions
* Fix update test
* copy RV from unified to legacy objects
* revert changes to playlist xorm store
* Improve logging. Add go routines for mode3
* Add tests for async funcs in mode3
* Lint
* Lint
* Lint. Start to fix tests
* Fix watcher tests
* Fix store tests
* Fiinish fixing watcher tests
* Fix server tests
* add name check
* Update pkg/apiserver/rest/dualwriter_mode1.go
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
* All objects should have an UID
* Now with a different error message
* Simplify create on DW 2: use the same object to write to both storages
* Run only one test
* Add check for status code
* Add name if it's not present in mode2
* Populate UID in legacy
* Remove logs and commented code
* Change dualwriter1
* Remove commented code
* Fix list test
* remove get on update from dualwriter 2
* Get object before updating. Better var renaming
* Finish rebasing
* Comment test
* Uncomment tests
* Fix update test
* revert changes to playlist xorm store
* Improve logging. Add go routines for mode3
* Lint
* Fix watcher tests
* Fiinish fixing watcher tests
* Add mode 5 with etcd test case. Add early check to fail on populated RV in payload
* we can't set RV to the found object when updating
* Lint
* Don't fail on update playlists
* Name should not be different when updating and it should be not empty on creating
* Fix tests
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Lint
* Fix mode 5 tests
* Lint
* Add generateName condition on every mode. Fix tests
* Lint
* Add condition on where name or generate name have to be set
* Fix test
* Lint
* Fix folders test
* We dont need to send name for mode1
* Fail if UID is not present
* Remove change from not running test
* Remove unused line
* Lint
* Update pkg/storage/unified/apistore/store.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Improve error message
* Fix broken watcher test
* Fail on name mismatch on update
* Remove log
* Make sure UIDs match on create in both stores
* Lint
* Write first to unified storage
* Remove uid setting
* Remove RV only in mode2
* Fix test. Remove log line
* test
* No need to asser on RV in mode3
* Remove RV check due to race condition
* Update dualwriter.go
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
* Update pkg/storage/unified/client.go
* remove unused parameter
* log an error for object is missing UID instead of returning an error
* remove obj.SetResourceVersion("")
* log an error for object is missing UID instead of returning an error
* FInalise merge
* Move RV check to where it was
* Remove name check
* Remove server check for backwards compatibility
* Remove unused fn
* Move test checks for another PR
* Dont commit go work sum changes
* Only log error if RV is present for now.
---------
Co-authored-by: Todd Treece <todd.treece@grafana.com>
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
* adds metric for watch latency
* registers storage metrics when creating a new ResourceServer
* defines the latency (in milliseconds) as the diff between now and the RV. Still need to wait until PR for switching RV to millisecond timestamp is rolled out.
* should be micro seconds not milli
* for watch latency, use diff between now and resource version and convert to seconds
* fix typo
* Transforms raw US resource into an intermediate IndexableResource and indexes that. Pulls index mapping code out into different file. For now, we will hardcode which spec fields are indexed, per resource.
* Fixes a few bugs with field casing and timestamps not being formatted right (or not existing).
* adds readme section for using search with US
* extracts to function to transform from search hit to IndexedResource
* get folders when building index
`GuranteedUpdate` method of `apistore.Storage` had a bug, where it would
errorneously conclude that the object is unchanged, in case a
`tryUpdate` function is passed that modifies the existing object itself
(as it is the case in many core types in K8s upstream).
The modified `existingObj` was compared with `updatedObj`, which would
essentially be same and this lead to the update being skipped.
This patch fixes this by always passing a copy of the `existingObj`.
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
* Use epoch with microsecond resolution as RV
* fix backend tests
* Add solution for when the clock goes back
* Add solution for when the clock goes back
* generate mocks
* go lint
* remove comment
* Use Greatest instead of max in msyql and postgres
* update tests
* Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
* cast to bigint
* add additional round trip
* increment the RV using 2 sql round trips instead of 3
* cleanup comments
* cast unix timestamp to integer
* fix postgres query
* remove old increment test data
* remove greatest
* cast unix_timestamp to signed
* Use statement_timestamp instead of clock_timestamp
---------
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
* resource-api: Loosen name validation to match K8s requirements
This patch modifies some of the requirements for name validation of
objects in Resource API to match Kubernetes.
The limit we have on characters in name is 64, but some resources allow
upto 253 characters. Similarly we also include `:` in the regex, as many
objects in default K8s setup use it in the name (the group
`system:masters` for example)
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
* Update the name column length in migrator and update e2e test to verify
---------
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
* adds Filter gRPC and make protobuf
* adds route for querying the filter gRPC
* wires up Filter gRPC call
* [WIP] index from start
* renames gRPC endpoint to "Search"
* adds /apis/search route into k8s routes. Hacky for now.
* updates readme - wrong casing
* adds feature toggle for unified storage search
* hides US search behind feature flag. Clean up print statements.
* removes indexer - will be added in another PR
* Search: Add API Builder
* adds required method
* implementing UpdateAPIGroupInfo (WIP)
* adds groupversion
* commenting out for now
* remove unneeded code from experimenting and update register.go to match interface required
* list resources and load into index
* pass context
* namespaces search route
* lint
* watch
* add todo
* add todo
* merge
* cleanup
* add todo
* gen protobuf
* lint; fix migration issue
* Updates index mapping function to map unified storage object Value
* Changes Index() to pointer receiver - fixes panic
* add delete
* cleanup
* gets search/browse functioning. Results show up as base64 encoded. Still a WIP.
* Doesnt json re-encode gRPC response in search handler
* add kind to SearchRequest proto
* Updates query interface to be more generic. Make proto. Parses query params in api server.
* make protobuf
* removes unused method and imports
* Returns all indexed fields in search results. Adds pagination support (limit + offset).
* remove comment
* remove unused struct
* gets tenant in search k8s api handler
* adds hardcoded spec field mappings - starting with playlists
* adds all spec fields to search results
* moved helper function for field mappings into index
* only includes allowed spec fields in search results
* cleans up error handling
* removes debug log
---------
Co-authored-by: leonorfmartins <leonorfmartins@gmail.com>
Co-authored-by: Todd Treece <todd.treece@grafana.com>
Co-authored-by: Scott Lepper <scott.lepper@gmail.com>
* Add parents field to folder DTO
* Allow subfolder creation when folder flag is enabled
* Update UnstructuredToLegacyFolder
* Include parents field when creating folder
* Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)"
This reverts commit 7c3fc2f261.
* make previous_resource_version nullable
* handle nil case
* Reuse MySQL and Postgres Grafana config instead of the object
- Only reuse the Grafana DB object for SQLite. Support for SQLite will be added in a different PR
- Fail when reusing the Grafana DB object if it is using DB instrumentation
- In the case that we have to reuse a Grafana DB with its instrumentation, fail with an error that describes a workaround
- Add regression tests to reproduce incident 2144
* remove temp file
* fix linter
* fix linter x2
* fix linter x3
* adds Filter gRPC and make protobuf
* adds route for querying the filter gRPC
* wires up Filter gRPC call
* [WIP] index from start
* renames gRPC endpoint to "Search"
* adds /apis/search route into k8s routes. Hacky for now.
* updates readme - wrong casing
* adds feature toggle for unified storage search
* hides US search behind feature flag. Clean up print statements.
* removes indexer - will be added in another PR
* Search: Add API Builder
* adds required method
* implementing UpdateAPIGroupInfo (WIP)
* adds groupversion
* commenting out for now
* remove unneeded code from experimenting and update register.go to match interface required
* namespaces search route
---------
Co-authored-by: leonorfmartins <leonorfmartins@gmail.com>
Co-authored-by: Todd Treece <todd.treece@grafana.com>
* Replace Watch with WatchNext
* remove watchset
* fix previous page and closing the channel
* Remove the broadcaster cache to prevent dupplicated events
* add watch bookmark
* add watch bookmark
* cleanup comments
* disable the tests for bookmarks for now
* Ensure we send previosu events
* lint
* re-introduce the cache
* load from cache
* disabling legacy test
* disabling legacy test
* Update pkg/storage/unified/resource/server.go
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
* Could not read previous events
* add proper migration
* Add previous_resource_version to both history and resource
* First event should have an RV of 2 and not 1
* Test both storage backends
* fix the inital RV for the sql backend
* ensure graceful stop of the stream decoder
* gocyclo
---------
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
* Add authlib gRPC authenticators for in-proc mode
* implement `StaticRequester` signing in the unified resource client
- [x] when the `claims.AuthInfo` value type is `identity.StaticRequester`, and there's no ID token set, create an internal token and sign it with symmetrical key. This is a workaround for `go-jose` not offering the possibility to create an unsigned token.
- [x] update `IDClaimsWrapper` to support the scenario above
- [x] Switch to using `claims.From()` in `dashboardSqlAccess.SaveDashboard()`
---------
Co-authored-by: gamab <gabriel.mabille@grafana.com>
