Gabriel MABILLE
aa2b4751a0
AuthZ: Launch service within IAM app ( #96421 )
2024-11-20 11:13:33 +01:00
Ryan McKinley
7985fa573e
UnifiedStorage: Add ResourceTable format ( #96506 )
2024-11-19 15:47:59 +02:00
Georges Chaudy
e270412dbf
unistore: wire the authz client ( #96632 )
...
* unistore: wire the authz client
* rename dashboards.grafana.app into dashboard.grafana.app
* wire the authz client
* wire the authz client
* resuse the Standalone constructor
* configure default migration for resource folder
* add tests
* cleanup
* add logging
2024-11-19 15:13:30 +02:00
Stephanie Hingtgen
6571451a57
K8s: dashboards: fix conversion ( #96648 )
2024-11-19 05:32:05 -07:00
Ryan McKinley
cc6d057a18
Provisioning: Rename k8s origin metadata to repo ( #96524 )
2024-11-15 17:26:14 +03:00
owensmallwood
9240cf1930
Unified storage Indexer: Add kind to hardcoded resource types ( #96149 )
...
* add kind to hardcoded resource types for indexer
* call ListOptions in Watch
* Use GaugeVec instead of CounterVec
* index folders before dashboards
2024-11-14 14:28:52 -05:00
Ryan McKinley
d9b87ef987
Storage: Add authz access client stubs ( #95548 )
...
authz stubs
authz stubs
lint fix
Add folder
move check.Name on creat
fix watch
2024-11-13 13:17:15 +01:00
Ryan McKinley
e9956f2345
Storage: Always set a UID in the apiserver ( #96086 )
2024-11-12 18:58:32 +02:00
Georges Chaudy
f4e91da6b1
Unistore: fix fodler migration ( #96299 )
...
configure default migration for resource folder
2024-11-12 16:06:42 +02:00
Georges Chaudy
c3571752b6
Unitstore: Fix the Folder migration ( #96078 )
...
* UniStore: Add folder to the SQL backend
This reverts commit aaa4f4a23e
2024-11-12 12:52:04 +01:00
Ryan McKinley
c0de407fee
K8s/Dashboards: Delegate large objects to blob store ( #94943 )
2024-11-09 08:09:46 +03:00
Scott Lepper
f93fd77c9c
search: list and filter tags ( #96130 )
...
search: list and filter tags on the dashboard list
2024-11-08 16:39:08 -05:00
owensmallwood
d023cb5931
Unified Storage: Adds metric to count indexed docs by kind ( #96056 )
...
Adds metric to count indexed docs by kind
2024-11-07 16:12:13 -03:00
Arati R.
7fa4a979e3
K8s/Folders: Add write access hook before creating index server ( #96030 )
...
Change order of when to add write access hook
2024-11-07 16:20:07 +02:00
Scott Lepper
3dbeacfebc
search: dashboard list sorting ( #95983 )
...
search: dashboard list sorting
2024-11-06 15:25:44 -05:00
owensmallwood
b6596db75e
Unified Storage Indexer: Build tenant indexes concurrently ( #95795 )
...
* WIP - build tenant indexes concurrently
* adds local dev seeders and readme for indexer
* update logging and adds locking in getShard()
* update logs
* Adds Namespaces func. Initializes index after ResourceServer is created.
* fixes Count() and updates test lint issues
* Initialize index separately. Don't do it when creating the resource server. Makes testing really awkward.
* fix lint error
* handles error when getting namespaces
* updates test and index helper funcs
2024-11-06 12:58:07 -06:00
Leonor Oliveira
aaa4f4a23e
Revert "UniStore: Add folder to the SQL backend" ( #95966 )
...
Revert "UniStore: Add folder to the SQL backend (#95671 )"
This reverts commit f075662696
2024-11-06 17:04:57 +02:00
Scott Lepper
1feaf6df99
search: add OR filter for kinds ( #95915 )
...
search: add OR filter for kinds
2024-11-05 17:02:13 -05:00
Scott Lepper
ef29ce5b85
[search] - folder tests; search by ids ( #95797 )
...
[search] - folder tests; search by ids
2024-11-05 09:34:54 -05:00
Georges Chaudy
f075662696
UniStore: Add folder to the SQL backend ( #95671 )
...
* Add folder to the SQL backend
* remove unused key
* fix tests
* Return folders from watch
2024-11-05 10:37:23 +01:00
Zoltán Bedi
85c696c4ad
SQL: Add macro support in select case ( #88514 )
...
* Feat: timeGroup macro handling in VQB
* Add tests
* Add functions to SQL ds
* Fix lint errors
* Add feature toggle
* Add rendering based on object
* Fix lint
* Fix CI failures
* Fix tests
* Address review comments
* Add docs
* Fix JSX runtime warnings
* Remove docs part that mentions suggest more macros
* Update docs/sources/shared/datasources/sql-query-builder-macros.md
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
* Add smoke test for this feature
* lint
* Add supported macros to influx
* Add setupTests.ts to include in tsconfig.json
* Import jest-dom instead of setupTests.ts
---------
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
2024-11-04 17:13:35 +01:00
owensmallwood
0eb7b755e2
Unified Storage Indexer: Add integration tests ( #95687 )
...
* indexer integration tests WIP
* make protobuf
* Adds a few more integration test cases to cover the basics. Use Limit instead of Size param from SearchRequest.
* skip if testing.Short()
* adds test comments
2024-11-01 07:58:10 -06:00
Scott Lepper
f8a5813573
[search] grouping - support indexing dashboard tags, filtering by tag, listing tags ( #95670 )
...
[search] grouping
2024-10-31 13:09:17 -04:00
Georges Chaudy
76b43267c8
Unified Storage: Use ssl_mode instead of sslmode ( #95658 )
...
change unistore to use ssl_mode instead of sslmode
2024-10-31 16:10:47 +01:00
Ryan McKinley
4e1f0dadbd
UnifiedStorage: Default to running unified-grpc in integration tests ( #93492 )
2024-10-31 16:29:32 +03:00
Scott Lepper
7619d9cb11
[search] minimal dashboard indexing ( #95631 )
...
[search] minimal dashboard indexing
2024-10-31 07:46:00 -04:00
owensmallwood
2b0a439ad3
Unified Storage Indexer: Dont create temp dir for indexDir ( #95626 )
...
* dont create temp dir for indexDir
* uses logger from trace ctx so traceID is on logs
2024-10-30 18:20:42 +00:00
owensmallwood
50a6069532
Unified Storage Indexer: Uses in-memory index ( #95576 )
...
uses in-memory index
2024-10-29 21:40:25 +00:00
owensmallwood
995128d1db
Search PoC: Improves initial indexing speed. Makes params configurable. ( #95439 )
...
* Improves initial indexing speed. Makes params configurable.
* fix linter errors
* removes kind param
* updates index test
* remove println from test
* removes error check in test
* adds log for high index latency ands updates max goroutine var with workers config var
* fix test timing out - set worker limit
* set the batch size
---------
Co-authored-by: Scott Lepper <scott.lepper@gmail.com>
2024-10-29 12:24:31 -06:00
Scott Lepper
337f2a0a8b
[search] add index; increase page size ( #95515 )
...
[search] add index; increase page size
2024-10-28 15:48:49 -04:00
Gabriel MABILLE
5a0ef46280
Add tracing to the gRPC Authentication flow ( #94466 )
...
commit ad4df4b3f63bdf3e16423ac8c3fdb1a7fae5582e
Author: gamab <gabriel.mabille@grafana.com>
Date: Thu Oct 24 10:24:04 2024 +0200
nit
commit eb8b9cf2f3e27cae258b3ae310f1584da5ba36b5
Author: gamab <gabriel.mabille@grafana.com>
Date: Thu Oct 24 10:23:25 2024 +0200
miss
commit aab1aed204a5dedcc6dd187b2f636995bbe2c5c6
Merge: 5aafdec9233 7fe710b141e9e2b11ba2e9e2b11ba2b5209dba64562d499e85562d499e85addc6aaca47c6d80f6aa64a5e55d619dc2ccdbfd64a5e55d614fe2c03457c7e36759cdf5c2c79981742295c89aa45998c8d3ffdc3017184a03ed7d7da2c30f5328ead390f6082f3c539d9bead390f608😁
commit 3948048880c7a0eb2360a35b0cc9f3686f2edfef
Author: gamab <gabriel.mabille@grafana.com>
Date: Wed Oct 9 14:02:41 2024 +0200
Add traces to NamespaceAuthorizer
commit cc695bb77c37a097174556303721fbc48b9464a0
Author: gamab <gabriel.mabille@grafana.com>
Date: Wed Oct 9 13:56:48 2024 +0200
Add traces to authentication flow
commit 8686c46be508c3d237dc4a3ce6619308c3d237dc33fd104cfd84d580179d33fd104cfd68af25fbc338f57d270a68af25fbc34fba5c9b3286867a14cac591631135c80c46ca6ae37b43117bc80c46ca6a3acada9d474224d059343acada9d47914ca237e271c33dcbe352f248eebb920d79680d52f248eebba6c977ba4dfb7bbf743b8da1d78c92fb7bbf743ba28440c40b79d9969aa8a8b07b0c8179d9969aa836b37524908ce354bb06b89f3f81158ce354bb06bdf79f3b2f8f4df8973d8eb7e55f8f8f4df8973d2441cd8d539338e40dc32441cd8d532904074a2f2bbce8a7f72904074a2f125cb3c834499a31df53f5d383644d755485751ed09e14c26a21220c2cca6cf1efdcc44b73a938836032ab3ae1601beb5327a1b64081270d70225c1a1128c417d80d70225c1a62f165f6f9bb5ee88d4f84866a8a5114a102160584f8c9be947fe8d623047c2353ae2552b7cf855014ddfbd8fb884c4a8c240fd1988beda1190b165b0fd1988bedb766bfb24fe0950a1283b766bfb24f6993f108a268751ed3106993f108a25f073b04d0f1ba609b345f073b04d00620891d45ac5ebe6e4d0620891d456a272e8e2a15f2b08f006a272e8e2a31c7b030ba0fdd2ff802763961210cf384759ad1763961210cc46b42a59592aba937a90145b0fe7092aba937a9
2024-10-28 14:35:30 +02:00
owensmallwood
5e3e7cca0e
Search PoC: Adds traces to indexer ( #95372 )
...
* adds traces to indexer
* fix linting errors
2024-10-25 00:05:33 +03:00
owensmallwood
ecebaf1bdf
Unified Storage: Add traces to poller ( #95369 )
...
add traces to poller
2024-10-24 22:49:38 +03:00
Diego Augusto Molina
257aca5340
Unistore Chore: Fix span name ( #94965 )
...
fix span name
2024-10-24 12:12:20 -03:00
Claudiu Dragalina-Paraipan
830600dab0
AuthN: Optionally use tokens for unified storage client authentication ( #91665 )
...
* extracted in-proc mode to #93124
* allow insecure conns in dev mode + refactoring
* removed ModeCloud, relying on ModeGrpc and stackID instead to discover if we're running in Cloud
* remove the NamespaceAuthorizer would fail in legacy mode. It will be added back in the future.
* use FlagAppPlatformGrpcClientAuth to enable new behavior, instead of legacy
* extracted authz package changes in #95120
* extracted server side changes in #95086
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
2024-10-24 09:12:37 +02:00
owensmallwood
2ffb88b0ee
Unified Storage: Add index for poller query ( #95293 )
...
add index for poller query
2024-10-23 15:40:56 -06:00
Gabriel MABILLE
b68b69c2b4
AuthN: Use tokens for unified storage server authentication ( #95086 )
...
* Extract server code
---------
Co-authored-by: Claudiu Dragalina-Paraipan <drclau@users.noreply.github.com>
2024-10-23 15:04:15 +02:00
Leonor Oliveira
a03652494c
Dual Writer simplification ( #93852 )
...
* All objects should have an UID
* Now with a different error message
* Simplify create on DW 2: use the same object to write to both storages
* Run only one test
* Add check for status code
* Add name if it's not present in mode2
* Populate UID in legacy
* Remove logs and commented code
* Change dualwriter1
* Remove commented code
* Fix list test
* remove get on update from dualwriter 2
* Get object before updating. Better var renaming
* Finish rebasing
* Comment test
* Uncomment tests
* Update legacy first. Add preconditions
* Remove preconditions
* Fix update test
* copy RV from unified to legacy objects
* revert changes to playlist xorm store
* Improve logging. Add go routines for mode3
* Add tests for async funcs in mode3
* Lint
* Lint
* Lint. Start to fix tests
* Fix watcher tests
* Fix store tests
* Fiinish fixing watcher tests
* Fix server tests
* add name check
* Update pkg/apiserver/rest/dualwriter_mode1.go
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
* All objects should have an UID
* Now with a different error message
* Simplify create on DW 2: use the same object to write to both storages
* Run only one test
* Add check for status code
* Add name if it's not present in mode2
* Populate UID in legacy
* Remove logs and commented code
* Change dualwriter1
* Remove commented code
* Fix list test
* remove get on update from dualwriter 2
* Get object before updating. Better var renaming
* Finish rebasing
* Comment test
* Uncomment tests
* Fix update test
* revert changes to playlist xorm store
* Improve logging. Add go routines for mode3
* Lint
* Fix watcher tests
* Fiinish fixing watcher tests
* Add mode 5 with etcd test case. Add early check to fail on populated RV in payload
* we can't set RV to the found object when updating
* Lint
* Don't fail on update playlists
* Name should not be different when updating and it should be not empty on creating
* Fix tests
* Update pkg/apiserver/rest/dualwriter_mode2.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Lint
* Fix mode 5 tests
* Lint
* Add generateName condition on every mode. Fix tests
* Lint
* Add condition on where name or generate name have to be set
* Fix test
* Lint
* Fix folders test
* We dont need to send name for mode1
* Fail if UID is not present
* Remove change from not running test
* Remove unused line
* Lint
* Update pkg/storage/unified/apistore/store.go
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
* Improve error message
* Fix broken watcher test
* Fail on name mismatch on update
* Remove log
* Make sure UIDs match on create in both stores
* Lint
* Write first to unified storage
* Remove uid setting
* Remove RV only in mode2
* Fix test. Remove log line
* test
* No need to asser on RV in mode3
* Remove RV check due to race condition
* Update dualwriter.go
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
* Update pkg/storage/unified/client.go
* remove unused parameter
* log an error for object is missing UID instead of returning an error
* remove obj.SetResourceVersion("")
* log an error for object is missing UID instead of returning an error
* FInalise merge
* Move RV check to where it was
* Remove name check
* Remove server check for backwards compatibility
* Remove unused fn
* Move test checks for another PR
* Dont commit go work sum changes
* Only log error if RV is present for now.
---------
Co-authored-by: Todd Treece <todd.treece@grafana.com>
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2024-10-23 10:29:41 +02:00
owensmallwood
d9bc4f7395
Search poc: Add debug log for metric and update metric buckets ( #95217 )
...
* adds debug log for metric
* updates buckets
2024-10-23 09:35:17 +02:00
owensmallwood
9763199b53
Search Poc: Add metrics ( #95111 )
2024-10-22 12:25:08 -06:00
Andrej Ocenas
5b1d99b8c4
Sidecar: Remove extension helpers and use the sidecar service directly ( #94979 )
2024-10-22 11:46:29 +02:00
owensmallwood
bda27ec8c6
Unified Storage: Add watch latency metric ( #93509 )
...
* adds metric for watch latency
* registers storage metrics when creating a new ResourceServer
* defines the latency (in milliseconds) as the diff between now and the RV. Still need to wait until PR for switching RV to millisecond timestamp is rolled out.
* should be micro seconds not milli
* for watch latency, use diff between now and resource version and convert to seconds
* fix typo
2024-10-21 17:15:11 -06:00
owensmallwood
fb9cfd0d1b
Search PoC: Indexmapping per resource and transform to intermediate IndexedResource ( #94906 )
...
* Transforms raw US resource into an intermediate IndexableResource and indexes that. Pulls index mapping code out into different file. For now, we will hardcode which spec fields are indexed, per resource.
* Fixes a few bugs with field casing and timestamps not being formatted right (or not existing).
* adds readme section for using search with US
* extracts to function to transform from search hit to IndexedResource
* get folders when building index
2024-10-21 09:34:41 -06:00
Scott Lepper
5700b4b13a
[search] - restart watch ( #94981 )
...
undo go sum change
2024-10-18 15:39:51 -04:00
Diego Augusto Molina
3e1f5559a6
Unistore Chore: Add database-level observability ( #92266 )
...
* add testing harness
* fix mockery and linters
* WIP
* wip
* fix transactions
* fix transaction tracing; add tracing by default
* rename package
* move WithTx to simplify logic of DB implementations
* fix potential issue with context deadline
* add db instrumentation to dbutil
* add otel tests
* improve naming
* minor fix in semantics and add comprehensive OTel testing
* fix naming
* instrument resourceVersionAtomicInc
* provide a default testing tracer
* fix docs
* fix typo in docs
* add semconv for k8s
2024-10-18 00:32:08 -03:00
Diego Augusto Molina
cf08f6762d
Unistore Chore: Add OTEL testing harness ( #94835 )
...
* add testing harness
* fix mockery and linters
* fix data race in tests
* fix data race in tests
* reduce cardinality of data
2024-10-17 08:41:06 -03:00
Ryan McKinley
3457f219be
Storage: Add blob storage interfaces ( #90932 )
...
Co-authored-by: Jean-Philippe Quémémer <jeanphilippe.quemener@grafana.com>
2024-10-17 12:18:29 +02:00
Diego Augusto Molina
44282134da
Unistore Chore: Make it easier to implement DB interface ( #94680 )
...
make it easier to implement DB interface
2024-10-16 13:40:35 -03:00
Prem Saraswat
e2672021bc
[unified-storage/apistore] Fix GuranteedUpdate skipping updates when tryUpdate is passed ( #94557 )
...
`GuranteedUpdate` method of `apistore.Storage` had a bug, where it would
errorneously conclude that the object is unchanged, in case a
`tryUpdate` function is passed that modifies the existing object itself
(as it is the case in many core types in K8s upstream).
The modified `existingObj` was compared with `updatedObj`, which would
essentially be same and this lead to the update being skipped.
This patch fixes this by always passing a copy of the `existingObj`.
Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
2024-10-11 17:15:43 +05:30
Georges Chaudy
d999b415df
UniStore: Use epoch with microsecond resolution as RV ( #92638 )
...
* Use epoch with microsecond resolution as RV
* fix backend tests
* Add solution for when the clock goes back
* Add solution for when the clock goes back
* generate mocks
* go lint
* remove comment
* Use Greatest instead of max in msyql and postgres
* update tests
* Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
* cast to bigint
* add additional round trip
* increment the RV using 2 sql round trips instead of 3
* cleanup comments
* cast unix timestamp to integer
* fix postgres query
* remove old increment test data
* remove greatest
* cast unix_timestamp to signed
* Use statement_timestamp instead of clock_timestamp
---------
Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-10-11 12:11:33 +03:00
