Commit Graph

213 Commits

Author SHA1 Message Date
Arati R.
47cd8288cd
SearchV2: Fix reading library panels (#92961)
* Fix reading of library panel field
* Add test to check reading of library panel field
2024-09-05 13:42:15 +02:00
Dave Henderson
df3d8915ba
Chore: Bump Go to 1.23.0 (#92105)
* chore: Bump Go to 1.23.0

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* update swagger files

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore: update .bingo/README.md formatting to satisfy prettier

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore(lint): Fix new lint errors found by golangci-lint 1.60.1 and Go 1.23

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* keep golden file

* update openapi

* add name to expected output

* chore(lint): rearrange imports to a sensible order

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-08-21 11:40:42 -04:00
Ryan McKinley
160fe2a3a4
Storage: Remove entity store (#91233) 2024-07-31 09:25:39 +03:00
Ryan McKinley
728150bdbd
Identity: extend k8s user.Info (#90937) 2024-07-30 08:27:23 +03:00
Ryan McKinley
be7b1ce2df
Chore: Replace appcontext.User(ctx) with identity.GetRequester(ctx) (#91030) 2024-07-26 16:39:23 +03:00
Diego Augusto Molina
399d77a0fd
Resource server improvements and fixes (#90715)
* cleanup dependencies and improve list method
* Improve Resource Server API, remove unnecessary dependencies
* Reduce the API footprint of ResourceDBInterface and its implementation
* Improve LifecycleHooks to use context
* Improve testing
* reduce API size and improve code
* sqltemplate: add DialectForDriver func and improve naming
* improve lifecycle API
* many small fixes after adding more tests
2024-07-22 20:08:30 +03:00
Georges Chaudy
08c611c68b
ResourceServer: Resource store sql backend (#90170) 2024-07-18 08:03:18 -07:00
Ryan McKinley
2bb66209ca
Chore: make protobuf (#90282) 2024-07-10 19:19:21 +03:00
Ryan McKinley
079f0715aa
ResourceServer: Add resource server protobuf and wrapper (#90007) 2024-07-09 15:08:13 -07:00
Alexander Zobnin
87d86e81ce
Zanzana: Evaluate permissions alongside with RBAC engine (#90064)
* Zanzana: Evaluate permissions if feature flag enabled

* Fix tests

* adjust logs

* fix spelling

* remove unused

* only evaluate implemented resources

* refactor
2024-07-05 11:31:23 +02:00
Todd Treece
274bd08afc
K8s: Improve key generation and parsing (#90014) 2024-07-04 01:11:45 +03:00
Todd Treece
71270f3203
Storage: Avoid relying on RequestInfo (#89635) 2024-06-25 10:06:03 -04:00
Ryan McKinley
5e95c1bdf8
Storage: Move grpc helper from entity store to resource store (#89490) 2024-06-20 22:32:19 +03:00
Ryan McKinley
27e800768e
K8s: Improve identity mapping setup (#89450) 2024-06-20 17:53:07 +03:00
Ryan McKinley
6834038e91
Storage: Add new resource module and move some utilities (#89443) 2024-06-20 16:11:21 +03:00
Ryan McKinley
9b7f9ae22e
K8s: Rename origin.key to origin.hash (#89337) 2024-06-18 22:27:16 +03:00
Diego Augusto Molina
b9812a0784
Unified Storage: Fix data races and context usage in broadcaster (#88955)
* Fix several broadcaster data races and error handling

- Separate concerns between sender and receiver sides in channel usage
- broadcaster: Fix data race between Subscribe/Unsubscribe and start
- Fix Subscribe error to be io.EOF when broadcaster is terminated
- Fix Watch never unsubscribing
- General cleanup
- fix usage of context
- add a huge amount of documentation about channels
2024-06-15 02:46:14 -03:00
Ryan McKinley
ed400f0bbf
EntityStore: Use standard user identifier rather than custom version (#89080)
user uid string
2024-06-12 19:39:34 +03:00
Dave Henderson
c931a44503
storage: Avoid panic when no previous versions in history (#88880)
Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-06-06 21:33:08 -04:00
Diego Augusto Molina
ae84c5fef9
Unified Storage: Fix Entity Server unconditionally starting (#88857)
fix Entity Server unconditionally starting
2024-06-06 12:17:46 -03:00
Diego Augusto Molina
5fc580b401
Unified Storage: Testing For Fix Create, Update and Delete wrt Resource Versions (#88568)
Add testing harness
2024-06-05 15:18:33 -03:00
Diego Augusto Molina
6fcd7d9e03
Unified Storage: Fix Create, Update and Delete wrt Resource Versions (#88183)
* add sqltemplate utilities, improve tests and documentation

* bunch of things

* remove unnecessary message

* add queries

* add queries

* add queries

* add folders support

* fix diff

* fix linters

* fix diff

* fix linters

* fix linters

* fix typo

* fix linters

* fix linters

* fix linters

* several fixes

* several fixes

* temporarily disable k8s integration tests for Entity Server

* postpone some tests

* postpone documentation changes

* Fix bug in create

* improve error reporting

* fix PostgeSQL parameters

* fix MySQL sqlmode

* fix MySQL-5.7

* reduce but document the number of database connection options

* remove unused code and improve docs
2024-06-05 14:23:32 -03:00
Marcus Efraimsson
0b1aec6767
Tracing: Various improvements (#88308) 2024-05-27 14:21:40 +02:00
Diego Augusto Molina
8b02b6b76a
Unified Storage: create kind_version table migration, add SQL and fix db (#87977)
* fix database interfaces

* add queries

* fix queries

* fix linters

* add owner to imported go library

* remove unused funcs

* run go work sync

* improve critical section fix in data race fix

* fix linters

* remove sync

* fix typo

* improve data embedding

* fix linters

* fix migration

* remove unnecessary comments

* fix linters

* improve SQL templates readability

* remove group_version from kind_version for consistency in History method

* add created_at and updated_at columns to kind_version table
2024-05-22 11:59:40 -03:00
owensmallwood
557c3a9cdd
Unified Storage: Adds metrics for database query counts (#87781) 2024-05-14 10:17:38 -06:00
Diego Augusto Molina
cbcd945251
Unified Storage: in SQL template, also handle output data, improve API, examples and docs (#87560)
* preview of work so far

* stylistic improvements

* fix linters

* remove golden tests, they may cause the system to be too rigid to changes

* remove unnecessary code for golden tests

* remove white space mangling in Execute

* also handle output data, improve API, examples and docs

* add helper methods

* fix interface
2024-05-14 12:32:21 -03:00
owensmallwood
3bf39d6d9a
Unified Storage: Fixes bug with postgres connection string and adds tests (#87656) 2024-05-13 10:16:26 -06:00
owensmallwood
77686da969
Unified Storage: Adds sql metrics (#87651)
* registers sql metrics for unified storage and removes single quotes from postgres connection string format

* uses db logger

* only register US sql metrics when connecting to another postgres or mysql db

* use engine
2024-05-13 08:09:25 -06:00
Diego Augusto Molina
acf17c7fb1
Unified Storage: Add SQL template package (#87524)
* added sqltemplate package

* addded example

* fix linting issues

* improve code readability

* fix documentation
2024-05-08 17:58:18 -03:00
Marcus Efraimsson
41b29ff93c
Chore: Migrate to use buf for protobuf generation (#87407)
* Chore: Migrate to use buf for protobuf generation

* fix codeowners

* fixes after review

* docs/tooling for installing buf

* add back codeowners

* Update upgrading-dependencies.md

Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>

* sdk v0.229.0

---------

Co-authored-by: Giuseppe Guerra <giuseppe.guerra@grafana.com>
2024-05-08 15:42:40 +02:00
Prem Saraswat
196134b0b4
Storage Server: Avoid incorrect parsing of connection string (#85293)
Storage Server: avoid incorrect parsing of connection string

In storage server, a connection string is generated to connect to the
backing database. In case of Postgres, we were generating the string
like `"user=grafana pass= host=postgres.example port=4567"`.

This triggered an edge case in `pq` (the go postgres driver) to parse
`pass` to be equal to `host=postgres.example`, and host being reset to
default value.

Using single quotes in the connection string fixed this.

Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
2024-05-03 10:54:39 -04:00
Georges Chaudy
dd0e646e92
k8s: remove dependency on app_mode development (#85474)
* k8s: remove dependency on app_mode development

* remove dev conditions on storage server

* remove logic to drop entity tables
2024-05-03 11:28:15 +03:00
Arati R
9e6de035c0
Storage: Add mode 2 dual writing improvements (#87204)
* Fix mode 2 List test
* Set origin timestamp during conversion to k8s resource
* Add instructions for updating a playlist
* Handle partial deletions of a collection in mode 2
2024-05-02 16:06:51 +02:00
owensmallwood
40558899e4
Unified Storage: Adds health check service for entity server (#86471) 2024-05-01 11:36:44 -06:00
Arati R
61f3d08c3f
Storage: Add List support for pagination (#87093)
* Add origin keys to List request
* Check origin keys requirement in the sql entity server's List method
* Refactor mode 2 List
* Check origin keys before making Storage List call

---------

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-05-01 08:42:47 +02:00
Arati R
ec6f59a678
Chore: Update protoc-gen-go (#87116)
Update protoc-gen-go
2024-04-30 12:14:33 +02:00
Marcus Efraimsson
ac152ca416
Chore: Upgrade grpc-ecosystem/go-grpc-middleware to v2 (#86724)
* Chore: Replace deprecated prometheus grpc middleware

* go mod tidy without enterprise

* with updated sdk branch

* sdk v0.226.0

* remove deprecated opentracing support for outgoing plugin requests

* migrate to github.com/grpc-ecosystem/go-grpc-middleware/v2

* fix
2024-04-25 17:25:48 +02:00
Serge Zaitsev
522a98c126
Chore: Make Cfg field private in SQLStore (#85593)
* make cfg private in sqlstore

* fix db init in tests

* fix case

* fix folder test init

* fix imports

* make another Cfg private

* remove another Cfg

* remove unused variable

* use store cfg, it has side-effects

* fix mutated cfg in tests
2024-04-24 10:38:40 +02:00
owensmallwood
8c8885ef23
Storage Api: Adds traces (#85391)
- adds traces and improved logging to the unified storage server
- add a configurable logger to the gRPC server service
2024-04-16 08:30:51 -06:00
Dan Cech
5dfe4cf407
Storage: Watch tests (#85496)
* basic watch tests working

* refactor to read previous event within poller

* add watch test files

* cleanup

* watch tests passing

* debug cleanup

* special handling for canceled context

* cleanup

* fix wire

* fix sqlite_sequence cleanup issue

* move watch tests to integration, wait for provisioned dashboards

* handle context deadline exceeded and eof errors

* add comment about sleep
2024-04-08 11:42:12 -04:00
owensmallwood
37d39de36d
Storage Api: Add metrics (#85316)
* Storage server runs own instrumentation server if its the sole target. Starts adding some sample metrics for now.

* adds metric for failed optimistic locks

* refactors metrics registration to own method on service for testability. Adds tests.

* Register sql storage server metrics from within the service

* fixes test

* troubleshooting drone test failures. Maybe timing when starting instrumentation server?

* Waits until instrumentation server has started. Updates tests.

* defer wont get called unless theres an error. removing.

* wait for instrumentation server to be running

* linter - close res body

* use port 3000 for metrics and removes test metric inc() call

* fixes test - updates port

* refactors module server to provide an instrumentation server module when there is no ALL or CORE target provided and running as single target

* make instrumentation server a dependency of all modules that do not run their own http server

* adds module server test

* adds tests for instrumentation service and removes old tests that aren't needed

* ignore error in test

* uses helper to start and run service

* when running wait on ctx done or http server err

* wait for http server

* removes println

* updates module server test to be integration test

* require no error in goroutine

* skips integration test when GRAFANA_TEST_DB not defined

* move http server start into start, verify returned content

* make test error when run fails

* try waiting longer and see if drone tests pass

* update integration test mysql creds to match drone

* go back to only waiting half second

* debug log drone mysql connection string

* use same db connection config as drone

* try using same hostname as drone

* cant use localhost as mysql hostname in drone tests. Need to parse it from the cfg db connection string

---------

Co-authored-by: Dan Cech <dcech@grafana.com>
2024-04-08 08:35:01 -06:00
Dan Cech
9c46e71d5a
Storage: Support continue at specified resource version (#84868)
* support continue at specified resource version

* detect whether list continue pages need to use entity_history, remove BatchRead, expand selectQuery helper

* refactor continue token handling

* fix tests, increase history chunk size

* lint fix
2024-04-08 09:57:02 -04:00
Serge Zaitsev
faa1244518
Chore: Replace sqlstore with db interface (#85366)
* replace sqlstore with db interface in a few packages

* remove from stats

* remove sqlstore in admin test

* remove sqlstore from api plugin tests

* fix another createUser

* remove sqlstore in publicdashboards

* remove sqlstore from orgs

* clean up orguser test

* more clean up in sso

* clean up service accounts

* further cleanup

* more cleanup in accesscontrol

* last cleanup in accesscontrol

* clean up teams

* more removals

* split cfg from db in testenv

* few remaining fixes

* fix test with bus

* pass cfg for testing inside db as an option

* set query retries when no opts provided

* revert golden test data

* rebase and rollback
2024-04-04 15:04:47 +02:00
Dan Cech
ef26fe95dc
Storage: GuaranteedUpdate fix & other improvements (#85206)
make GuaranteedUpdate work when ignoring not found errors, increase poll frequency, fix Delete
2024-03-27 10:38:49 -04:00
Sofia Papagiannaki
33b653534e
SQLStore: Enable migration locking by default (#84983)
* Introduce new configuration for migration locking

* Remove feature toggle

* Fix test and turn it into an integration

* Fix docs
2024-03-22 21:22:29 +02:00
Marcus Efraimsson
6c1de260a2
API Server: Standalone observability (#84789)
Adds support for logs (specify level), metrics (enable metrics and Prometheus /metrics endpoint 
and traces (jaeger or otlp) for standalone API server. This will allow any grafana core service 
part of standalone apiserver to use logging, metrics and traces as normal.
2024-03-21 17:06:32 +01:00
Dan Cech
89f3b70e17
Storage: Add support for listing resource history (#84331)
* add support for listing resource history

* make watch handle custom label selectors properly

* fix tests

* Apply suggestions from code review

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* properly handle special characters in json label matcher

* tidy up

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-03-15 19:17:54 -04:00
Dan Cech
0fa0cede75
Storage: streamline context handling (#84319)
streamline context handling
2024-03-14 17:12:20 -04:00
Dan Cech
1ffd1cc8f4
Storage: Support listing deleted entities (#84043)
* support listing deleted entities

* fold listDeleted into List
2024-03-11 13:59:54 -04:00
Dan Cech
7e4badff1d
Storage: Use our own key format and support unnamespaced objects (#83929)
* use our own key format and support unnamespaced objects

* fix tests
2024-03-05 16:31:39 -05:00