Commit Graph

60 Commits

Author SHA1 Message Date
Matheus Macabu
1635a3cd67
CloudMigrations: Add support for migration of Library Elements (Panels) resources (#93898)
* CloudMigrations: create snapshots of Library Elements

* CloudMigrations: render library element resource in resources table

* CloudMigrations: create newtype with necessary fields for library element creation
2024-10-03 11:54:54 +02:00
Dana Axinte
1a31abe254
CloudMigrations: Limit frontend query to get latest snapshots (#93639)
* latest param to endpoint and adapt frontend query

* change to sort param

* api

* remove description
2024-10-01 09:28:25 +01:00
Michael Mandrus
1c648fd010
Chore: Fix flaky cloud migration test (#94035)
* attempt to fix flaky test

* remove skip from test
2024-10-01 03:24:47 +01:00
Bruno
6f92fd64ce
Cloud migrations: add more context to errors (#93814)
* Cloud migrations: add more context to errors

* calls to assert.ErrorIs was passing arguments in the wrong order
2024-09-30 09:57:25 -03:00
Bruno
8f66b7fdb0
Cloud migrations: disable flaky test (#93921)
* Cloud migrations: disable flaky test

* Update pkg/services/cloudmigration/cloudmigrationimpl/xorm_store_test.go

Co-authored-by: Kevin Minehart <5140827+kminehart@users.noreply.github.com>

---------

Co-authored-by: Kevin Minehart <5140827+kminehart@users.noreply.github.com>
2024-09-27 16:36:08 +01:00
Michael Mandrus
e672796632
CloudMigrations: Fix OrderBy clause in GetSnapshotList sql handler (#93857)
fix order_by clause in list query
2024-09-26 22:51:01 -04:00
Bruno
1b7cd4c684
Cloud migrations: store resource name in the cloud_migration_resource table (#93612)
* Cloud migrations: store resource name in the cloud_migration_resource table

* remove unused function: convertMigrateDataResponseToDTO

* make swagger-clean && make openapi3-gen

* use DB_Text for cloud_migration_resource.name instead of DB_Varchar
2024-09-26 11:16:52 -03:00
Matheus Macabu
1e720306dd
CloudMigrations: replace slicesext.Chunk with stdlib implementation (#93743) 2024-09-25 16:52:22 +02:00
Matheus Macabu
945484232c
CloudMigrations: remove unused code from sync migration approach (#93430) 2024-09-20 08:58:11 +02:00
Michael Mandrus
f4c5c5c418
Chore: Fix flaky cloud migration test due to ordered steps running in parallel (#93491)
don't run test in parallel
2024-09-19 18:45:02 +03:00
Matheus Macabu
4f21ecf982
CloudMigrations: improve nil handling (#93257)
* CloudMigrations: fail token decryption if session is not found or without a token

* CloudMigrations: do not report event if session is nil
2024-09-17 08:59:47 +02:00
Bruno
90b44fdcbf
Cloud migrations: GetSnapshotList only returns snapshots that belong to a session that exists (#91798)
* Cloud migrations: GetSnapshotList only returns snapshots that belong to a session that exists

* Cloud migrations: test GetSnapshotList

* add one more test case for GetSnapshotList

* fix test

* store encryption key for testing
2024-08-13 09:57:53 -03:00
Michael Mandrus
eee3a75b8b
CloudMigrations: Misc cleanup before codefreeze (#91725)
* fix retry logic

* slight adjustments

* fix disconnect and connect events

* Update pkg/services/cloudmigration/cloudmigrationimpl/snapshot_mgmt.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

---------

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2024-08-09 17:38:08 +03:00
lean.dev
e20c7342a9
CloudMigration: Removes snapshot and resources when deleting a session (#91548) 2024-08-09 16:42:45 +03:00
Michael Mandrus
6cd0971dc6
CloudMigration: Show warning message for successfully migrated non-core data sources (#91545)
* minor performance improvement

* apply a warning to any non-core plugins that successfully migrate

* commit frontend wip while I refactor some stuff

* update api

* repurpose error dialog to be a generic details dialog

* whitespace

* add unit test

* fixes from testing

* fix migration summary

* add comment

* fix localization stuff

* fix backend test

* reduce number of queries to the db

* some PR feedback

* whitespace
2024-08-06 19:35:10 +03:00
Michael Mandrus
3e138449bb
CloudMigrations: Fix bugs found during local testing (#91163)
* send dashboard commands instead of dashboards

* move status updates before goroutine to ensure frontend polls

* fix syncing issues between snapshot state and resources

* make sessionUid a requirement for modifying snapshots

* move the function I meant to move earlier

* remove accidental commit

* another accidental commit

* verify UpdateSnapshot is called with sessionUid

* revert

* pass in session uid everywhere

* forgot to save

* fix unit test

* fix typo

* tiny tweak
2024-07-30 09:02:41 -04:00
lean.dev
d8555019f1
CloudMigrations: Fix migration in docker (#91147) 2024-07-29 15:00:43 -03:00
Michael Mandrus
a6088e4ee4
CloudMigrations: Improvements to backend (#91012)
* E2C: Add stat rollup to MigrationSummary

* fix report event url

* open form in new page

* sort folders by heirarchy

* undo accidental commit

* remove another commit

* make folder sorting dynamic

---------

Co-authored-by: joshhunt <josh@trtr.co>
2024-07-29 17:55:22 +00:00
Michael Mandrus
dc355331a6
CloudMigrations: Store encryption key in unified secrets table (#90908)
* store encryption key in unified secrets table

* fix local dev mode

* make metadata more realistic

* fix tests

* fix sql queries against postgres

* fix stats endpoint
2024-07-25 06:34:25 +03:00
Bruno
8d8f2ba587
Cloud migration: encryption key is a byte slice (#90739)
* Cloud migrations: include metadata returned by CMS in the index file

* Cloud migrations: make EncryptionKey a []byte in structs

* test
2024-07-22 11:25:12 -03:00
Bruno
07685646a3
Cloud migrations: include metadata returned by CMS in the index file (#90658) 2024-07-22 10:11:57 -03:00
Michael Mandrus
ee90cd3031
CloudMigrations: Send local events to gms during the migration process (#90637)
* add gms client function

* add timeout config for endpoint

* report events to gms

* fix lint error

* clean up report calls and make sure reports all have local ids

* extra validation

* improve error logging and fix url
2024-07-20 07:02:31 +03:00
Michael Mandrus
71a97b925a
CloudMigrations: Add and fix logging (#90616)
* add logs so we can see how long async processes take

* fix a bunch of error logs:

* fix logs again
2024-07-18 21:16:56 +03:00
Michael Mandrus
17228e5794
CloudMigrations: Handle cancelation during snapshot building and upload (#90612)
implement cancelation during upload and building
2024-07-18 12:55:27 -04:00
Michael Mandrus
a43a538976
CloudMigrations: Fix issues discovered during end to end testing (#90562)
* improve error handling a retries during async operations

* fix use of contexts

* updates to how we call the folder api

* fix urls for gms

* more progress on the folder issue

* fix folders

* refactor for readability
2024-07-18 18:34:28 +03:00
Michael Mandrus
9b7e9d992b
CloudMigrations: Query GMS for a presigned upload url at upload time (#90505)
query GMS for an upload url at upload time
2024-07-17 11:53:21 -04:00
Michael Mandrus
83b309c724
CloudMigrations: Don't migrate dashboards that are soft-deleted (#90454)
* skip dashboards that are soft-deleted while building snapshots

* add unit test
2024-07-16 22:33:11 +03:00
Michael Mandrus
6585ed16e5
Chore: Fix CloudMigration unit tests (#90489)
remove parallel tests for things that deal with the database
2024-07-16 18:50:22 +03:00
Michael Mandrus
6ff21726b7
CloudMigrations: Query GMS for snapshot status with a results offset (#90453)
* query GMS for status with an offset

* remove unused state
2024-07-16 09:04:21 -04:00
Michael Mandrus
542a1bf3ac
CloudMigrations: Query Grafana Migration Status for status while the snapshot is in the cloud (#90314)
* implement querying gms for snapshot status

* add some documentation

* provide snapshot resources after snapshot is created

* add rate limiting to backend

* fix compilation error

* fix typo

* add unit tests

* finish merge

* lint

* swagger gen

* more testing

* remove duplicate test

* address a couple PR comments

* update switch statement to a map

* add timeouts to gms client through the http client

* remove extra whitespace

* put method back where it was so the PR is less confusing

* fix tests

* add todo

* fix final unit test
2024-07-15 09:22:57 -04:00
Bruno
420ff1d07b
Cloud migration: upload snapshot files using presigned url (#90273)
* Cloud migration: upload snapshot files using presigned url

* log error if index file cannot be closed

* log error if file cannot be closed in uploadUsingPresignedURL
2024-07-11 09:32:02 -03:00
Michael Mandrus
317da43a84
CloudMigration: Provide a stats rollup in the GetSnapshot response (#90252)
* order session list descending

* add snapshot status method to store

* query stats while retrieving snapshot

* return stats in dto

* swagger

* fix tests

* commit results of bingo get

* fix swagger

* minor improvement

* fix typo

* forgot a file
2024-07-10 08:46:38 -04:00
Bruno
d1952bb681
Cloud migrations: create snapshot files (#89693)
* Cloud migrations: create snapshot and store it on disk

* fix merge conflicts

* implement StartSnapshot for gms client

* pass snapshot directory as argument to snapshot builder

* ensure snapshot folder is set

* make swagger-gen

* remove Test_ExecuteAsyncWorkflow

* pass signed in user to buildSnapshot method / use github.com/grafana/grafana-cloud-migration-snapshot to create snapshot files

* fix FakeServiceImpl.CreateSnapshot

* remove new line
2024-07-03 10:38:26 -03:00
Michael Mandrus
6761ed863f
CloudMigrations: Add ordering to snapshot list query (#89916)
add ordering to snapshot list
2024-07-01 11:35:04 -04:00
Michael Mandrus
4d69213829
CloudMigrations: Break snapshot resources out into their own table (#89575)
* create a new table for migration resources

* remove raw result bytes from db

* more snapshot resource management stuff

* integrate new table with snapshots

* pass in result limit and offset as params

* combine create and update

* set up xorm store test

* add unit tests

* save some cpu

* remove unneeded arg

* regen swagger

* fix bug with result processing

* fix update create logic so that uid isn't required for lookup

* change offset to page

* regen swagger

* revert accidental changes to file

* curl command page should be 1 indexed
2024-06-24 23:50:07 -04:00
Michael Mandrus
89337ea01f
CloudMigrations: Add unit tests for snapshot management (#89521)
* add regex support for api tests

* revert dumb thing

* add api tests

* add unit test for core async workflow

* add xorm store unit tests

* fix typo

* remove unnecessary assignment
2024-06-21 16:35:15 +03:00
Michael Mandrus
8a8f97b0e4
CloudMigrations: Implement snapshot management apis (#89296)
* add new apis

* add payloads

* create snapshot status type

* add some impl

* finish implementing update

* start implementing build snapshot func

* add more fake build logic

* add cancel endpoint. do some cleanup

* implement GetSnapshot

* implement upload snapshot

* merge onprem status with gms result

* get it working

* update comment

* rename list endpoint

* add query limit and offset

* add helper method to snapshot

* little bit of cleanup

* work on swagger annotations

* manual merge

* generate swagger specs

* clean up curl commands

* fix bugs found during final testing

* fix linter issue

* fix unit test
2024-06-19 09:20:52 -04:00
Dave Henderson
6262c56132
chore(perf): Pre-allocate where possible (enable prealloc linter) (#88952)
* chore(perf): Pre-allocate where possible (enable prealloc linter)

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* fix TestAlertManagers_buildRedactedAMs

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* prealloc a slice that appeared after rebase

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-06-14 14:16:36 -04:00
Michael Mandrus
9d3a4e236d
CloudMigrations: Refactor API for async work (#89084)
* rename some stuff

* more renaming

* clean up api

* rename more functions

* rename cms -> gms

* update comment

* update swagger gen

* update endpoints

* overzealous

* final touches

* dont modify existing migrations

* break structs into domain and dtos

* add some conversion funcs

* fix build

* update frontend

* try to make swagger happy
2024-06-13 17:58:59 +00:00
lean.dev
330da7916d
Chore: Add unit test for cloudmigration package (#88868) 2024-06-07 13:46:19 -03:00
Bruno
33b9544047
Cloud migrations: create route to delete token (#88297)
* Cloud migrations: create route to delete token

* gcom.DeleteToken returns ErrTokenNotFound instead of a boolean

* remove unnecessary comment

* make openapi3-gen && yarn run rtk-query-codegen-openapi ./scripts/generate-rtk-apis.ts

* gcom stub: implement DeleteToken
2024-05-31 10:03:43 -03:00
Bruno
83543c6b12
Cloud migrations: create route to fetch cloud migration token (#88176)
* Cloud migration: create route to fetch cloud migration token

* implement gcomStub.ListTokens

* fix swagger for POST /cloudmigration/migration

* fix swagger for POST /cloudmigration/migration

* fix swagger for POST /cloudmigration/migration
2024-05-31 09:39:10 -03:00
Bruno
14ccf7f6d1
Cloud migrations: return errutil.Error in http responses (#87448) 2024-05-13 00:22:46 -04:00
Bruno
6a3ab71d77
cms: clean up model file (#87179)
* move CreateAccessTokenResponseDTO to api.go

* remove unused types

* run make swagger-clean && make openapi3-gen
2024-05-13 00:22:19 -04:00
lean.dev
0719f73f35
Cloud Migration: Remove ID migration in favor of UID (#86550)
* - Added migration to add a new collumn UID to both migration and migration_run tables.
- Added migration to set UID for records already existent in the database before adding a new column.
- Added UID field to CloudMigration and CloudMigrationRun dtos (model.go)

* Fix db migration

* Updating store and model

* Updating API to use UID

* fix typo

* update openapi specs and generated endpoints

* fix spec

* update openapi specs and generated endpoints

* Fixing db mapping

* Fix frontend

* fix migration

* remove migration uid from run fetch endpoint

* Revert "merge"

This reverts commit d654e4c530, reversing
changes made to 5fe0b483eb.

* manual merge

* rename some funcs for consistency

* make interfaces consistent

* validate uids

* update generated frontend api

* fix enterprise spec

* manually resolve api

* try again

* try yet again

* once more

---------

Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com>
Co-authored-by: Michael Mandrus <michael.mandrus@grafana.com>
Co-authored-by: joshhunt <josh@trtr.co>
2024-05-01 12:29:25 -04:00
bornohin6
7ce1254913
CloudMigration: Remove details from the migration run list (#87104)
Remove details from the migration run list
2024-04-30 11:44:35 -07:00
Michael Mandrus
45a7f649fe
CMS: Create local implementation of cloud migration for dev use (#86637)
* add developer mode property to config

* create cms stub

* cleanup

* implement and wire up gcom stub

* fix errors

* don't document the flag
2024-04-20 23:51:58 -04:00
Michael Mandrus
df4c8c3cbc
CloudMigrations: Move business logic out of api layer (#86406)
* move run migration to the cloudmigrationimpl layer

* add migration run list logic down a layer

* remove useless comments

* pull cms calls into their own service
2024-04-17 15:43:09 -04:00
lean.dev
c8d185502b
Fixes CMS url for dev environment (#86056) 2024-04-12 14:19:38 +00:00
Bruno
69a92a7a2f
CloudMigrations: include stack id in access policy and token names (#85868)
* CloudMigrations: include stack id in access policy and token names

* CloudMigrations: include slug in the display name

* formatting
2024-04-11 09:43:46 -03:00