Commit Graph

177 Commits

Author SHA1 Message Date
Marcus Efraimsson
ac152ca416
Chore: Upgrade grpc-ecosystem/go-grpc-middleware to v2 (#86724)
* Chore: Replace deprecated prometheus grpc middleware

* go mod tidy without enterprise

* with updated sdk branch

* sdk v0.226.0

* remove deprecated opentracing support for outgoing plugin requests

* migrate to github.com/grpc-ecosystem/go-grpc-middleware/v2

* fix
2024-04-25 17:25:48 +02:00
Serge Zaitsev
522a98c126
Chore: Make Cfg field private in SQLStore (#85593)
* make cfg private in sqlstore

* fix db init in tests

* fix case

* fix folder test init

* fix imports

* make another Cfg private

* remove another Cfg

* remove unused variable

* use store cfg, it has side-effects

* fix mutated cfg in tests
2024-04-24 10:38:40 +02:00
owensmallwood
8c8885ef23
Storage Api: Adds traces (#85391)
- adds traces and improved logging to the unified storage server
- add a configurable logger to the gRPC server service
2024-04-16 08:30:51 -06:00
Dan Cech
5dfe4cf407
Storage: Watch tests (#85496)
* basic watch tests working

* refactor to read previous event within poller

* add watch test files

* cleanup

* watch tests passing

* debug cleanup

* special handling for canceled context

* cleanup

* fix wire

* fix sqlite_sequence cleanup issue

* move watch tests to integration, wait for provisioned dashboards

* handle context deadline exceeded and eof errors

* add comment about sleep
2024-04-08 11:42:12 -04:00
owensmallwood
37d39de36d
Storage Api: Add metrics (#85316)
* Storage server runs own instrumentation server if its the sole target. Starts adding some sample metrics for now.

* adds metric for failed optimistic locks

* refactors metrics registration to own method on service for testability. Adds tests.

* Register sql storage server metrics from within the service

* fixes test

* troubleshooting drone test failures. Maybe timing when starting instrumentation server?

* Waits until instrumentation server has started. Updates tests.

* defer wont get called unless theres an error. removing.

* wait for instrumentation server to be running

* linter - close res body

* use port 3000 for metrics and removes test metric inc() call

* fixes test - updates port

* refactors module server to provide an instrumentation server module when there is no ALL or CORE target provided and running as single target

* make instrumentation server a dependency of all modules that do not run their own http server

* adds module server test

* adds tests for instrumentation service and removes old tests that aren't needed

* ignore error in test

* uses helper to start and run service

* when running wait on ctx done or http server err

* wait for http server

* removes println

* updates module server test to be integration test

* require no error in goroutine

* skips integration test when GRAFANA_TEST_DB not defined

* move http server start into start, verify returned content

* make test error when run fails

* try waiting longer and see if drone tests pass

* update integration test mysql creds to match drone

* go back to only waiting half second

* debug log drone mysql connection string

* use same db connection config as drone

* try using same hostname as drone

* cant use localhost as mysql hostname in drone tests. Need to parse it from the cfg db connection string

---------

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-04-08 08:35:01 -06:00
Dan Cech
9c46e71d5a
Storage: Support continue at specified resource version (#84868)
* support continue at specified resource version

* detect whether list continue pages need to use entity_history, remove BatchRead, expand selectQuery helper

* refactor continue token handling

* fix tests, increase history chunk size

* lint fix
2024-04-08 09:57:02 -04:00
Serge Zaitsev
faa1244518
Chore: Replace sqlstore with db interface (#85366)
* replace sqlstore with db interface in a few packages

* remove from stats

* remove sqlstore in admin test

* remove sqlstore from api plugin tests

* fix another createUser

* remove sqlstore in publicdashboards

* remove sqlstore from orgs

* clean up orguser test

* more clean up in sso

* clean up service accounts

* further cleanup

* more cleanup in accesscontrol

* last cleanup in accesscontrol

* clean up teams

* more removals

* split cfg from db in testenv

* few remaining fixes

* fix test with bus

* pass cfg for testing inside db as an option

* set query retries when no opts provided

* revert golden test data

* rebase and rollback
2024-04-04 15:04:47 +02:00
Dan Cech
ef26fe95dc
Storage: GuaranteedUpdate fix & other improvements (#85206)
make GuaranteedUpdate work when ignoring not found errors, increase poll frequency, fix Delete
2024-03-27 10:38:49 -04:00
Sofia Papagiannaki
33b653534e
SQLStore: Enable migration locking by default (#84983)
* Introduce new configuration for migration locking

* Remove feature toggle

* Fix test and turn it into an integration

* Fix docs
2024-03-22 21:22:29 +02:00
Marcus Efraimsson
6c1de260a2
API Server: Standalone observability (#84789)
Adds support for logs (specify level), metrics (enable metrics and Prometheus /metrics endpoint 
and traces (jaeger or otlp) for standalone API server. This will allow any grafana core service 
part of standalone apiserver to use logging, metrics and traces as normal.
2024-03-21 17:06:32 +01:00
Dan Cech
89f3b70e17
Storage: Add support for listing resource history (#84331)
* add support for listing resource history

* make watch handle custom label selectors properly

* fix tests

* Apply suggestions from code review

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* properly handle special characters in json label matcher

* tidy up

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-03-15 19:17:54 -04:00
Dan Cech
0fa0cede75
Storage: streamline context handling (#84319)
streamline context handling
2024-03-14 17:12:20 -04:00
Dan Cech
1ffd1cc8f4
Storage: Support listing deleted entities (#84043)
* support listing deleted entities

* fold listDeleted into List
2024-03-11 13:59:54 -04:00
Dan Cech
7e4badff1d
Storage: Use our own key format and support unnamespaced objects (#83929)
* use our own key format and support unnamespaced objects

* fix tests
2024-03-05 16:31:39 -05:00
Dan Cech
9fa9eaab44
Storage: Support get with resourceversion (#83849)
support getting old resourceversion, return explicit resource version in list
2024-03-05 10:57:32 -05:00
Dan Cech
7b4925ea37
Storage: Watch support (#82282)
* initial naive implementation

* Update pkg/services/store/entity/sqlstash/sql_storage_server.go

Co-authored-by: Igor Suleymanov <radiohead@users.noreply.github.com>

* tidy up

* add action column, batch watch events

* initial implementation of broadcast-based watcher

* fix up watch init

* remove batching, it just adds needless complexity

* use StreamWatcher

* make broadcaster generic

* add circular buffer to replay recent events to new watchers

* loop within poll until all events are read

* add index on entity_history.resource_version to support poller

* increment r.Since when we send events to consumer

* switch broadcaster and cache to use channels instead of mutexes

* cleanup

---------

Co-authored-by: Igor Suleymanov <radiohead@users.noreply.github.com>
2024-03-05 10:14:38 -05:00
Arati R
467302480f
Search: Include collapsed panels in search v2 (#83047)
* Include collapsed panels in searchv2
* Include collapsed row in TestReadSummaries
2024-02-28 16:11:09 +01:00
김은빈
96dfb385ca
Grafana: Replace magic number with a constant variable in response status (#80132)
* Chore: Replace response status with const var

* Apply suggestions from code review

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* Add net/http import

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2024-02-27 18:39:51 +02:00
Dan Cech
790e1feb93
Chore: Update test database initialization (#81673)
* streamline initialization of test databases, support on-disk sqlite test db

* clean up test databases

* introduce testsuite helper

* use testsuite everywhere we use a test db

* update documentation

* improve error handling

* disable entity integration test until we can figure out locking error
2024-02-09 09:35:39 -05:00
Dan Cech
1f1461734c
Storage: Add support for sortBy selector (#80680)
* add support for sortBy field selector

* use label selectors instead of field selectors

* set entity_labels on create & update

* make entity server integration tests work

* test fixes

* be more consistent with handling of empty body, meta or status

* workaround for database is locked errors during migration

* fix double import of sqlite3

* rename functions and tidy up

* refactor update

* disable integration tests until we can fix the database locking issue
2024-02-07 15:05:10 -05:00
Dan Cech
156d7ae194
use in-process grpc client instead of wrapping server interface (#81926)
* use in-process grpc client instead of wrapping server interface

* comment out jwt token checks until we're ready to validate the token
2024-02-07 13:17:02 -05:00
Ryan McKinley
e041055012
K8s/Folders: Rename api group to singular (#81443) 2024-01-30 08:50:37 -08:00
Arati R
77775d548a
Chore: Add some tests for the Create method (#81364)
* Add test for create method
    Co-authored-by: Tania B <10127682+undef1nd@users.noreply.github.com>
* Change structure of entity package to break import cycle
* Update wire file

---------

Co-authored-by: Tania B <10127682+undef1nd@users.noreply.github.com>
2024-01-30 14:15:12 +01:00
Ryan McKinley
1fab107e79
FeatureFlags: Avoid using cfg.IsFeatureToggleEnabled (#81407) 2024-01-28 15:22:45 -08:00
Marcus Efraimsson
6768c6c059
Chore: Remove public vars in setting package (#81018)
Removes the public variable setting.SecretKey plus some other ones. 
Introduces some new functions for creating setting.Cfg.
2024-01-23 12:36:22 +01:00
Sven Grossmann
61ed0ef73f
Tests: Update geomap golden json (#80964)
update geomap golden json
2024-01-22 13:25:51 +02:00
Arati R
ca9d147a44
Give dialects control over how insert and update queries are performed (#79946)
* Refactor insert, update
* Add separate insert, update methods
* Refactor Insert, Update signatures
2024-01-11 19:55:45 +01:00
Arati R
90fb6a0122
Update unified storage readme (#79934)
* Update unified storage readme

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-01-05 23:44:34 +02:00
Dan Cech
b2b4d89501
Storage: Support grafana.app/folder field selector & label selectors (#79816)
* support grafana.app/folder field selector & label selectors

* lint fix
2023-12-21 18:08:01 -05:00
Ryan McKinley
d195c3807e
Storage: Use int64 for resourceVersion (#79735) 2023-12-20 11:26:35 -08:00
Ryan McKinley
67bbdd7c05
K8s/Folders: Add folders api service (with legacy storage) (#79413) 2023-12-20 20:28:56 +02:00
Tania
86ac431097
Chore: Update readme for entity store (#79586)
* Chore: Update readme for entity store

* Update pkg/services/store/entity/README.md

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>

* Apply review feedback

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2023-12-17 13:12:17 +02:00
Dan Cech
d146f4cbc7
Storage: rename uid to name and name to title (#79418)
rename uid to name and name to title
2023-12-12 15:14:55 -05:00
Dan Cech
3482658975
Storage: Add readme for storage server (#79366)
add readme for storage server
2023-12-12 09:24:14 -05:00
Artur Wierzbicki
88fe392f8b
Storage: make it work for Postgres (#79378)
* comment out cockroachdb connection config option that does not work for postgres

* log the error instead
2023-12-12 18:10:29 +04:00
Dan Cech
9f6144059a
Storage: Switch from tenant to namespace & remove GRN (#79250)
* remove GRN and switch tenant to namespace

* clean up remaining references

* simplify and remove inconsistency in With* parameters

* parse listing keys so we can use db index

* bump the schema version

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-12-11 12:26:05 -05:00
Ryan McKinley
deb8faf1e3
Storage: remove git backing for storage (#79181) 2023-12-07 00:04:13 +02:00
Ryan McKinley
9849c954a3
Chore: remove the entity kind registry (#79178) 2023-12-06 14:00:53 -08:00
Dan Cech
c4c9bfaf2e
Storage: Unified Storage based on Entity API (#71977)
* first round of entityapi updates

- quote column names and clean up insert/update queries
- replace grn with guid
- streamline table structure

fixes

streamline entity history

move EntitySummary into proto

remove EntitySummary

add guid to json

fix tests

change DB_Uuid to DB_NVarchar

fix folder test

convert interface to any

more cleanup

start entity store under grafana-apiserver dskit target

CRUD working, kind of

rough cut of wiring entity api to kube-apiserver

fake grafana user in context

add key to entity

list working

revert unnecessary changes

move entity storage files to their own package, clean up

use accessor to read/write grafana annotations

implement separate Create and Update functions

* go mod tidy

* switch from Kind to resource

* basic grpc storage server

* basic support for grpc entity store

* don't connect to database unless it's needed, pass user identity over grpc

* support getting user from k8s context, fix some mysql issues

* assign owner to snowflake dependency

* switch from ulid to uuid for guids

* cleanup, rename Search to List

* remove entityListResult

* EntityAPI: remove extra user abstraction (#79033)

* remove extra user abstraction

* add test stub (but

* move grpc context setup into client wrapper, fix lint issue

* remove unused constants

* remove custom json stuff

* basic list filtering, add todo

* change target to storage-server, allow entityStore flag in prod mode

* fix issue with Update

* EntityAPI: make test work, need to resolve expected differences (#79123)

* make test work, need to resolve expected differences

* remove the fields not supported by legacy

* sanitize out the bits legacy does not support

* sanitize out the bits legacy does not support

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* update feature toggle generated files

* remove unused http headers

* update feature flag strategy

* devmode

* update readme

* spelling

* readme

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-12-06 15:21:21 -05:00
Ryan McKinley
f69fd3726b
FeatureToggles: Add context and and an explicit global check (#78081) 2023-11-14 12:50:27 -08:00
Ryan McKinley
35c1ee9686
EntityStore: Remove http access (can use apiserver now) (#77602) 2023-11-03 08:14:51 -07:00
Ryan McKinley
72ed6434ca
EntityStore: Use protobuf for summary objects (#77600)
use protobuf for summary
2023-11-03 10:49:01 -04:00
Dan Cech
67b2972052
Chore: add/update sqlstore-related helper functions (#77408)
* add/update sqlstore-related helper functions

* add documentation & tests for InsertQuery and UpdateQuery, make generated SQL deterministic by sorting columns

* remove old log line
2023-11-03 10:30:52 -04:00
Drew Slobodnjak
9116043453
Storage: Add maxFiles to list functions (#76414)
* Storage: Add maxFiles to list functions

* Add maxDataPoints argument to listFiles function

* Add maxFiles to ResourceDimensionEditor

* Update pkg/services/store/http.go

* rename First to Limit

---------

Co-authored-by: jennyfana <110450222+jennyfana@users.noreply.github.com>
Co-authored-by: nmarrs <nathanielmarrs@gmail.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2023-11-01 09:42:24 -07:00
Ryan McKinley
29cf60988b
Playlist: Use a different go struct for sql service vs k8s (#76393) 2023-10-12 08:29:06 -07:00
Jo
dcd0c6b11e
Identity: Unfurl OrgID in pkg/services to allow using identity.Requester interface (#76113)
Unfurl OrgID in pkg/services to allow using identity.Requester interface
2023-10-09 10:40:19 +02:00
Andres Martinez Gotor
c70623fb85
Chore: Rename testdata plugin with a fully qualified name (#75104) 2023-09-22 15:00:40 +03:00
Will Browne
e855efb13d
Plugins: Move store and plugin dto to pluginsintegration (#74655)
move store and plugin dto
2023-09-11 13:59:24 +02:00
Serge Zaitsev
bd12ce0cbc
Chore: capitalise log messages for app platform (#74336) 2023-09-04 18:46:08 +02:00
Dan Cech
85a207fceb
Chore: Replace entity GRN with infra/grn GRN (#74198)
replace entity GRN with infra/grn GRN
2023-08-31 15:43:35 -04:00