Commit Graph

108 Commits

Author SHA1 Message Date
Ryan McKinley
ed1176adc8
Storage: export dashboards + playlists to object store (#57313) 2022-10-19 17:05:51 -07:00
Ryan McKinley
de3737b5de
Store: Add resolver service (#57112) 2022-10-19 07:33:26 -07:00
Kristin Laemmert
05709ce411
chore: remove sqlstore & mockstore dependencies from (most) packages (#57087)
* chore: add alias for InitTestDB and Session

Adds an alias for the sqlstore InitTestDB and Session, and updates tests using these to reduce dependencies on the sqlstore.Store.

* next pass of removing sqlstore imports
* last little bit
* remove mockstore where possible
2022-10-19 09:02:15 -04:00
Ryan McKinley
3b4b528993
Storage: Dashboard summary builder cleanup (#56665) 2022-10-13 09:29:19 -04:00
Ryan McKinley
b24be6c0fc
ObjectStore: add a kind registry (#56507) 2022-10-08 12:05:46 -04:00
Artur Wierzbicki
bf264d2f76
Query library: requiresDevMode dummy backend (#56466)
* query library - dummy backend

* fix tests

* dont explicitly marshall backend dataresponse

* skip integration tests

* null check for tests

* added query library to codeowners

* null check for tests

* lint
2022-10-07 11:31:45 -07:00
Ryan McKinley
609abf00d1
ObjectStore: Write json as json when possible (#56433) 2022-10-06 15:48:53 -04:00
Ryan McKinley
b4e23e5d32
ObjectStore: update dev protobuf definitions (#56428) 2022-10-05 18:46:17 -07:00
Ryan McKinley
97d19830cf
ObjectStore: rename modified to updated (#56420) 2022-10-05 14:58:46 -04:00
Artur Wierzbicki
a94acb7f61
Object store: get user from context (#56346)
* GRPC Server: Add signedInUser to context after auth

* add permissions to signedInUser

* add access control permissions test

* add additional signedInUser checks

* get user from context

* move `UserFromContext` to object/auth.go

Co-authored-by: Todd Treece <todd.treece@grafana.com>
2022-10-05 17:00:34 +00:00
Ryan McKinley
4fc9b9aa35
Storage: Cleanup object history API (#56215) 2022-10-04 14:57:26 -04:00
Ryan McKinley
d5e2713168
Storage: Add HTTP endpoint for object store service (#56214) 2022-10-04 09:40:15 -07:00
Artur Wierzbicki
85b965cbec
Storage: Dummy object server and basic integration tests (#56014)
* object extractors

* update bluge to use summary values

* gosec

* move to store/object package

* references

* references

* references

* same thign but with protobuf

* now the service

* now with summary

* now with summary

* from protobuf

* from protobuf

* cleanup

* remove hand crafted file

* update proto definitions

* update comments

* remove properties

* remove properties

* re-generate

* add batch

* move ref to raw struct

* GRPC test infra

* fix merge

* add delete

* lint

* rename to dummyobjectserver

* update comment

* refactor collection, simplify dummy server

* update

* refactor test structure

* more tests

* more tests

* replace collection with infra/persistentcollection

* skip if not integration test suite

* very important lint fix

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-09-30 15:56:07 -04:00
Ryan McKinley
213cfe5117
Object: use protobuf to declare the raw/summary types (#55646) 2022-09-29 22:18:15 -07:00
Ryan McKinley
e7e7763252
Store/Search: Explore a general interface to extract summary data from a blob (#55598) 2022-09-27 18:08:47 -04:00
Artur Wierzbicki
e19f36649f
Storage: Content root storage (#54929)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-09-12 18:34:46 -04:00
Emil Tullstedt
b287047052
Chore: Upgrade Go to 1.19.1 (#54902)
* WIP

* Set public_suffix to a pre Ruby 2.6 version

* we don't need to install python

* Stretch->Buster

* Bump versions in lib.star

* Manually update linter

Sort of messy, but the .mod-file need to contain all dependencies that
use 1.16+ features, otherwise they're assumed to be compiled with
-lang=go1.16 and cannot access generics et al.

Bingo doesn't seem to understand that, but it's possible to manually
update things to get Bingo happy.

* undo reformatting

* Various lint improvements

* More from the linter

* goimports -w ./pkg/

* Disable gocritic

* Add/modify linter exceptions

* lint + flatten nested list

Go 1.19 doesn't support nested lists, and there wasn't an obvious workaround.
https://go.dev/doc/comment#lists
2022-09-12 12:03:49 +02:00
Artur Wierzbicki
7a340f486b
Storage: add WithContents option to storage.Get() (#53105)
* Storage: add `WithContents` option to `storage.Get()`

* fix tests

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-08-30 15:23:16 +02:00
ying-jeanne
fe062f2eaa
Chore: Use db.DB interface instead of sqlstore (#54358)
* use db.DB interface instead of sqlstore

* make store service depends on db.DB instead of sqlstore
2022-08-26 19:07:58 -05:00
idafurjes
a14621fff6
Chore: Add user service method SetUsingOrg and GetSignedInUserWithCacheCtx (#53343)
* Chore: Add user service method SetUsingOrg

* Chore: Add user service method GetSignedInUserWithCacheCtx

* Use method GetSignedInUserWithCacheCtx from user service

* Fix lint after rebase

* Fix lint

* Fix lint error

* roll back some changes

* Roll back changes in api and middleware

* Add xorm tags to SignedInUser ID fields
2022-08-11 13:28:55 +02:00
Jo
062d255124
Handle ioutil deprecations (#53526)
* replace ioutil.ReadFile -> os.ReadFile

* replace ioutil.ReadAll -> io.ReadAll

* replace ioutil.TempFile -> os.CreateTemp

* replace ioutil.NopCloser -> io.NopCloser

* replace ioutil.WriteFile -> os.WriteFile

* replace ioutil.TempDir -> os.MkdirTemp

* replace ioutil.Discard -> io.Discard
2022-08-10 15:37:51 +02:00
idafurjes
6afad51761
Move SignedInUser to user service and RoleType and Roles to org (#53445)
* Move SignedInUser to user service and RoleType and Roles to org

* Use go naming convention for roles

* Fix some imports and leftovers

* Fix ldap debug test

* Fix lint

* Fix lint 2

* Fix lint 3

* Fix type and not needed conversion

* Clean up messages in api tests

* Clean up api tests 2
2022-08-10 11:56:48 +02:00
Artur Wierzbicki
0d817987ec
Storage: move file quota check to service (#53058)
* Storage: move file quota check to service

* Fix tests
2022-08-01 22:53:44 +04:00
Alexander Emelin
e791a4e576
Search: Move entity events to transaction (#52394) 2022-08-01 18:56:36 +03:00
Artur Wierzbicki
08dabfaffc
Storage: add global resources/system (#53000)
Co-authored-by: nmarrs <nathanielmarrs@gmail.com>
2022-07-29 15:58:47 -04:00
Selene
085ae014cd
Sync: Move ActiveTokenCount to a new service (#52991)
* Move ActiveTokenCount to a new service

* Fixing tests

* fix `RootSystem`

Co-authored-by: Artur Wierzbicki <artur.wierzbicki@grafana.com>
2022-07-29 16:30:46 +02:00
Ryan McKinley
197acd73c0
Storage: support git + github backed roots (#52192) 2022-07-28 23:26:44 -07:00
Artur Wierzbicki
9e841ef21d
Storage: fix delete folder (#52860) 2022-07-28 12:27:26 +03:00
Ryan McKinley
0235fc136c
Storage: allow public access to public files (#52849) 2022-07-26 20:37:35 -07:00
Artur Wierzbicki
3cd43bd7ea
Storage: externalize allow_unsanitized_svg_upload (#52703) 2022-07-25 21:11:17 +04:00
Artur Wierzbicki
d9db155394
Storage: Mime type detection (#52512)
* Storage: implement mime type detection

* lint
2022-07-25 11:30:20 +04:00
Ryan McKinley
3bdec4d138
Storage: remove orgId from sql config (#52426) 2022-07-19 08:13:26 -07:00
Artur Wierzbicki
0300b1c674
Storage: branding access (#52412)
* Storage: add special users for system branding access

* Storage: explicit global org id

* Storage: initialize org storages with global org id

* Storage: add comments

* Storage: simplify - use orgId: 1 for systembranding
2022-07-19 18:47:11 +04:00
Ryan McKinley
824f12a993
Storage: Remove storageLocalUpload flag (#52413) 2022-07-18 10:44:42 -07:00
Artur Wierzbicki
841e5ae8ad
Storage: system/branding storage (#51987)
* Storage: add `system` branding storage root, expose `system/branding` subfolder

* Storage: merge
2022-07-18 19:37:22 +04:00
Artur Wierzbicki
b2736ac1fe
Storage: limit the number of uploaded files (#50796)
* #50608: sql file upload quotas

* rename `files_in_sql` to `file`

* merge conflict
2022-07-18 15:24:39 +04:00
Artur Wierzbicki
67ea2da57e
Storage: fix svg upload (#52395) 2022-07-18 06:36:54 -04:00
Artur Wierzbicki
6188526e1d
Storage: use static access rules (#52334)
* Storage: use static access rules

* Storage: use static access rules

* Storage: add tests
2022-07-17 22:41:54 +04:00
Ryan McKinley
ab6cf9e94d
Storage: refactor readonly support (#52127) 2022-07-13 10:15:25 -07:00
Adela Almasan
2f942c57e8
Storage: Optionally overwrite existing files (#52067)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-07-12 09:13:57 -05:00
Artur Wierzbicki
1d2aa7c69b
Storage: add delete / deleteFolder / createFolder (#51887)
* delete / delete folder / create folder

* add backend tests

* implement force delete

* fix merge

* lint fix

* fix delete root folder

* fix folder name validation

* fix mysql path_hash issue

* Fix returning error
2022-07-08 22:23:16 +04:00
Artur Wierzbicki
e96f67ae2e
Renderer: Add sanitize API (#50936)
* svg fun

* #50597: add proto

* #50597: add sanitizer methods

* #50597: add provider

* #50597: use sanitizer

* #50597: use sanitizer

* update grafana to match new api

* add comments

* add capability check

* add timing

* update sanitize path

* improve log message

* strings.HasPrefix rather than filepath.IsAbs

* filepath.Clean + filepath.ToSlash for windows

* read 404

* remove `path.clean` from `getPathAndScope`

* add resp body close

* remove unneeded prop

* Update pkg/services/rendering/rendering.go

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>

* remove test files

* filepath.ToSlash correct wrapping

* filepath.ToSlash correct wrapping

* filepath.ToSlash comment

* compilation error

* lint fix

* fix error message

* Update pkg/services/rendering/rendering.go

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>

* add `image/svg+xml` mime type

* refactored log

* refactored log

Co-authored-by: Agnès Toulet <35176601+AgnesToulet@users.noreply.github.com>
2022-07-07 15:32:18 +04:00
Artur Wierzbicki
052132be46
Storage: simplify retrieving the file names from the response frame (#51805)
* add util method for storage list frame

* use go:embed

* remove enum
2022-07-07 10:47:05 +04:00
Ryan McKinley
4a00c7ebde
Storage: Add basic file upload management (#50638) 2022-07-05 10:53:41 -07:00
Artur Wierzbicki
cc4473faf3
Storage: validation and sanitization stubs (#50523)
* add `IsPathValidationError` util to fs api

* refactor storage.Upload method

* remove unused struct

* extract `RootUpload` constant

* move file validation outside of the service

* Make UploadErrorToStatusCode exported

* validation/sanitization

* refactor pathValidationError check

* refactor, rename sanitize to transform

* add a todo

* refactor

* transform -> sanitize

* lint fix

* #50608: fix jpg/jpeg

Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-06-15 12:32:29 +04:00
Artur Wierzbicki
9779f684d1
Storage: refactor - decouple the Upload method from HTTP concepts (#50378)
* add `IsPathValidationError` util to fs api

* refactor storage.Upload method

* remove unused struct

* extract `RootUpload` constant

* move file validation outside of the service

* Make UploadErrorToStatusCode exported

* refactor pathValidationError check

* if -> switch

Co-authored-by: Tania B <yalyna.ts@gmail.com>
2022-06-13 21:21:50 +04:00
Kat Yang
bd35e6917a
Chore: Exclude integration tests from running on test-backend step (#50359)
* Chore: Exclude integration tests from running on test-backend step

* Remove -v from go test command

* Add check to skip integration tests before each integration test

* Try to restart pipeline

* Retrying to make pipeline run
2022-06-10 11:46:21 -04:00
Alexander Emelin
d2868a1ce7
SearchV2: instant local updates, folder events (#50001) 2022-06-03 13:11:32 -07:00
Todd Treece
64d93498de
Chore: Update grafana-plugin-sdk-go to v0.136.0 (#50127) 2022-06-03 08:25:52 -04:00
Kat Yang
c63ebc887b
Chore: Run integration tests without grabpl (#49448)
* Chore: Run integration tests without grabpl

* Add new step for integration tests in lib.star

* Remove old integration test step from lib.star

* Change drone signature

* Fix: Edit starlark integration step to not affect enterprise

* Remove all build tags & rename starlark integration test step

* Resync .drone.yml with .drone.star

* Fix lint errors

* Fix lint errors

* Fix lint errors

* Fix more lint errors

* Fix another lint error

* Rename integration test step

* Fix last lint error

* Recomment enterprise step

* Remove comment from Makefile

Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
2022-06-01 14:55:22 -04:00
Kat Yang
50c2b4682a
Chore: Rename integration tests (#49438)
* Chore: Rename integration tests

* Remove one Integration

Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
2022-05-24 11:04:03 +02:00
Artur Wierzbicki
03fe1435a0
Storage: store uploaded files in SQL rather than on the disk (#49034)
* #48259: set up storages per org id

* #48259: migrate to storage_sql
2022-05-21 16:55:11 -07:00
An
900d9bf9a1
FileStorage: Add upload form (#46749)
* move upload to http

* use storage from grafanads

* rever gomod changes

* fix test

* wip

* add upload func

* update upload func

* writing to uploads

* edit response from service

* use dropzone for UI

* modify response struct in service

* better read file

* set content type for svg

* restrict file types upload

* add test and clean up errors

* pass test

* fix backend lint errors

* limit type of files on FE

* add TODO for after merge

* rebase with storage changes

* comment out unused function

* update UI to not have 2 uploads

* only call upload on select

* use utils function to find * in path

* show preview on drag over

* not allowing upload of svg

* add preview to upload tab

* no console.log

* resolve conflicts

* refactor log line

* fix failing BE test

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Artur Wierzbicki <artur.wierzbicki@grafana.com>
2022-04-27 15:12:48 -04:00
Artur Wierzbicki
25e153e4e7
Search: in-memory index (#47709)
* #45498: add entity events table

* #45498: add entity events service

* #45498: hook up entity events service to http server

* #45498: use `dashboards.id` rather than `uid` and `org_id` in grn

* Update pkg/services/entityevents/service.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* #45498: move entityeventsservice to services/store

* #45498: add null check

* #45498: rename

* #45498: fix comment

* #45498: switch grn back to uid

* Search: listen for updates (#47719)

* #45498: wire entity event service with searchv2

* load last event id before building index for org 1

* fix service init in integration tests

* depend on required subset of event store methods

* Update pkg/services/sqlstore/migrations/entity_events_mig.go

Co-authored-by: Alexander Emelin <frvzmb@gmail.com>

* #45498: pointer receiver

* #45498: mockery!

* #45498: add entity events service to background services

* dashboard query pagination, allow queries while re-indexing

* log level cleanups, use rlock, add comments

* fix lint, check feature toggle in search v2 service

* use unix time for event created column

* add missing changes for created column

* fix integration tests init

* log re-index execution times on info level

* #45498: fix entityEventsService tests

* #45498: save events on dashboard delete

* use camel case for log labels

* formatting

* #45498: rename grn to entityid

* #45498: add `IsDisabled` to entityEventsService

* #45498: remove feature flag from migration

* better context usage, fix capacity, comments/cleanups

* replace print with logger

* Revert "#45498: remove feature flag from migration"

This reverts commit ed23968898.

* revert:revert:revert conditional feature flag

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Alexander Emelin <frvzmb@gmail.com>
2022-04-27 12:29:39 +04:00
Artur Wierzbicki
9f0b6a5754
Storage: refactor filtering, improve performance (#47403)
* Storage: refactor filtering, improve performance

* added a comment to `newpathfilter`

* after merge fixes
2022-04-21 23:27:43 +04:00
ying-jeanne
7ddae870e7
fix status code 200 (#47818) 2022-04-15 08:01:58 -04:00
Artur Wierzbicki
712b239d5a
Storage: remove pointer on write path (#47580)
* remove pointer to contents in UpsertFileCommand

* add command to UpsertFileCommand

* fix client after API refactor
2022-04-12 16:58:09 +02:00
Ryan McKinley
1cfb9a4a19
Storage: add basic storage service (#46604) 2022-03-17 10:19:23 -07:00