IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2026-07-29 15:59:50 -05:00
Code Issues Packages Projects Releases Wiki Activity

Grafana app platform: an aggregator cmd and package (#79948)

Browse Source
This commit is contained in:
Charandas
2024-01-08 22:33:42 +02:00
committed by GitHub
parent 26f54a2fc7
commit 48612063dd
57 changed files with 5915 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/cmd/grafana/apiserver/README.md
-89
View File
@@ -1,89 +0,0 @@
# grafana apiserver (standalone)
The example-apiserver closely resembles the
[sample-apiserver](https://github.com/kubernetes/sample-apiserver/tree/master) project in code and thus
allows the same
[CLI flags](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) as kube-apiserver.
It is currently used for testing our deployment pipelines for aggregated servers. You can optionally omit the
aggregation path altogether and just run this example apiserver as a standalone process.
## Standalone Mode
### Usage
```shell
go run ./pkg/cmd/grafana apiserver example.grafana.app \
--secure-port 8443
```
### Verify that all works
```shell
export KUBECONFIG=./example-apiserver/kubeconfig
kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
dummy example.grafana.app/v0alpha1 true DummyResource
runtime example.grafana.app/v0alpha1 false RuntimeInfo
```
## Aggregated Mode
### Prerequisites:
1. kind: you will need kind (or another local K8s setup) if you want to test aggregation.
```
go install sigs.k8s.io/kind@v0.20.0 && kind create cluster
```
### Usage
You can start the example-apiserver with an invocation as shown below. The Authn / Authz flags are set up so that the kind cluster
can be used as a root server for this example-apiserver (in aggregated mode). Here, it's assumed that you have a local
kind cluster and that you can provide its kubeconfig in the parameters to the example-apiserver.
```shell
go run ./pkg/cmd/grafana apiserver example.grafana.app \
--authentication-kubeconfig ~/.kube/config \
--authorization-kubeconfig ~/.kube/config \
--kubeconfig ~/.kube/config \
--secure-port 8443
```
Once, the `example-apiserver` is running, you can configure aggregation against your kind cluster
by applying a `APIService` and it's corresponding `Service` object. Sample kustomizations are provided
for local development on [Linux](./deploy/linux/kustomization.yaml) and [macOS](./deploy/darwin/kustomization.yaml).
```shell
kubectl deploy -k ./deploy/darwin # or /linux
```
### Verify that all works
With kubectl configured against `kind-kind` context, you can run the following:
```shell
kubectl get --raw /apis/example.grafana.app/v0alpha1 | jq -r
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "example.grafana.app/v0alpha1",
"resources": [
{
"name": "runtime",
"singularName": "runtime",
"namespaced": false,
"kind": "RuntimeInfo",
"verbs": [
"list"
]
}
]
}
```
```shell
kubectl get apiservice v0alpha1.example.grafana.app
NAME SERVICE AVAILABLE AGE
v0alpha1.example.grafana.app grafana/example-apiserver True 4h1m
```
pkg/cmd/grafana/apiserver/aggregator.md
+51
View File
@@ -0,0 +1,51 @@
# grafana aggregator
The `aggregator` command in this binary is our equivalent of what kube-apiserver does for aggregation using
the `kube-aggregator` pkg. Here, we enable only select controllers that are useful for aggregation in a Grafana
cloud context. In future, Grafana microservices (and even plugins) will run as separate API servers
hosting each their own APIs (with specific Group/Versions). The `aggregator` component here shall act similar to what
`kube-apiserver` does: doing healthchecks for `APIService` objects registered against it and acting as a proxy for
the specified `GroupVersion` therein.
## How to get started
1. Generate the PKI using `openssl` (for development purposes, we will use the CN of `system:masters`):
```shell
./hack/make-aggregator-pki.sh
```
2. Start the aggregator:
```shell
# This will generate the kubeconfig which you can use in the extension apiservers for
# enforcing delegate authnz under $PWD/data/grafana-apiserver/aggregator.kubeconfig
go run ./pkg/cmd/grafana aggregator --secure-port 8443 \
--proxy-client-cert-file $PWD/data/grafana-aggregator/client.crt \
--proxy-client-key-file $PWD/data/grafana-aggregator/client.key
```
3. Apply the manifests:
```shell
export KUBECONFIG=$PWD/data/grafana-apiserver/aggregator.kubeconfig
kubectl apply -k ./pkg/cmd/grafana/apiserver/deploy/aggregator-test
# SAMPLE OUTPUT
# apiservice.apiregistration.k8s.io/v0alpha1.example.grafana.app created
# externalname.service.grafana.app/example-apiserver created
kubectl get apiservice
# SAMPLE OUTPUT
# NAME SERVICE AVAILABLE AGE
# v0alpha1.example.grafana.app grafana/example-apiserver False (FailedDiscoveryCheck) 29m
```
4. In another tab, start the example microservice that will be aggregated by the parent apiserver:
```shell
go run ./pkg/cmd/grafana apiserver example.grafana.app \
--kubeconfig $PWD/data/grafana-aggregator/aggregator.kubeconfig \
--secure-port 7443 \
--client-ca-file=$PWD/data/grafana-aggregator/ca.crt
```
5. Check `APIService` again:
```shell
export KUBECONFIG=$PWD/data/grafana-apiserver/aggregator.kubeconfig
kubectl get apiservice
# SAMPLE OUTPUT
# NAME SERVICE AVAILABLE AGE
# v0alpha1.example.grafana.app grafana/example-apiserver True 30m
```
pkg/cmd/grafana/apiserver/apiserver.md
+29
View File
@@ -0,0 +1,29 @@
# grafana apiserver (standalone)
The example-apiserver closely resembles the
[sample-apiserver](https://github.com/kubernetes/sample-apiserver/tree/master) project in code and thus
allows the same
[CLI flags](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) as kube-apiserver.
It is currently used for testing our deployment pipelines for aggregated servers. You can optionally omit the
aggregation path altogether and just run this example apiserver as a standalone process.
## Standalone Mode
### Usage
```shell
go run ./pkg/cmd/grafana apiserver example.grafana.app \
--secure-port 7443
```
### Verify that all works
```shell
export KUBECONFIG=./example-apiserver/kubeconfig
kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
dummy example.grafana.app/v0alpha1 true DummyResource
runtime example.grafana.app/v0alpha1 false RuntimeInfo
```
pkg/cmd/grafana/apiserver/cmd.go
+104 -1
View File
@@ -2,15 +2,27 @@ package apiserver
import (
"os"
"path"
"github.com/grafana/grafana/pkg/aggregator"
"github.com/grafana/grafana/pkg/services/grafana-apiserver/utils"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/klog/v2"
"github.com/spf13/cobra"
genericapiserver "k8s.io/apiserver/pkg/server"
"k8s.io/apiserver/pkg/server/options"
"k8s.io/component-base/cli"
aggregatorscheme "k8s.io/kube-aggregator/pkg/apiserver/scheme"
)
const (
aggregatorDataPath = "data/grafana-aggregator"
defaultAggregatorEtcdPathPrefix = "/registry/grafana.aggregator"
)
func newCommandStartExampleAPIServer(o *APIServerOptions, stopCh <-chan struct{}) *cobra.Command {
devAcknowledgementNotice := "The apiserver command is in heavy development. The entire setup is subject to change without notice"
devAcknowledgementNotice := "The apiserver command is in heavy development. The entire setup is subject to change without notice"
cmd := &cobra.Command{
Use: "apiserver [api group(s)]",
@@ -59,3 +71,94 @@ func RunCLI() int {
return cli.Run(cmd)
}
func newCommandStartAggregator(o *aggregator.AggregatorServerOptions) *cobra.Command {
devAcknowledgementNotice := "The aggregator command is in heavy development. The entire setup is subject to change without notice"
cmd := &cobra.Command{
Use: "aggregator",
Short: "Run the grafana aggregator",
Long: "Run a standalone kubernetes based aggregator server. " +
devAcknowledgementNotice,
Example: "grafana aggregator",
RunE: func(c *cobra.Command, args []string) error {
return run(o)
},
}
return cmd
}
func run(serverOptions *aggregator.AggregatorServerOptions) error {
if err := serverOptions.LoadAPIGroupBuilders(); err != nil {
klog.Errorf("Error loading prerequisite APIs: %s", err)
return err
}
serverOptions.RecommendedOptions.SecureServing.BindPort = 8443
delegationTarget := genericapiserver.NewEmptyDelegate()
config, err := serverOptions.CreateAggregatorConfig()
if err != nil {
klog.Errorf("Error creating aggregator config: %s", err)
return err
}
aggregator, err := serverOptions.CreateAggregatorServer(config, delegationTarget)
if err != nil {
klog.Errorf("Error creating aggregator server: %s", err)
return err
}
// Install the API Group+version
for _, b := range serverOptions.Builders {
g, err := b.GetAPIGroupInfo(Scheme, Codecs, config.GenericConfig.RESTOptionsGetter)
if err != nil {
klog.Errorf("Error getting group info for prerequisite API group: %s", err)
return err
}
if g == nil || len(g.PrioritizedVersions) < 1 {
continue
}
err = aggregator.GenericAPIServer.InstallAPIGroup(g)
if err != nil {
klog.Errorf("Error installing prerequisite API groups for aggregator: %s", err)
return err
}
}
if err := clientcmd.WriteToFile(
utils.FormatKubeConfig(aggregator.GenericAPIServer.LoopbackClientConfig),
path.Join(aggregatorDataPath, "aggregator.kubeconfig"),
); err != nil {
klog.Errorf("Error persisting aggregator.kubeconfig: %s", err)
return err
}
// Finish the config (a noop for now)
prepared, err := aggregator.PrepareRun()
if err != nil {
return err
}
stopCh := genericapiserver.SetupSignalHandler()
if err := prepared.Run(stopCh); err != nil {
return err
}
return nil
}
func RunCobraWrapper() int {
serverOptions := aggregator.NewAggregatorServerOptions(os.Stdout, os.Stderr)
// Register standard k8s flags with the command line
serverOptions.RecommendedOptions = options.NewRecommendedOptions(
defaultAggregatorEtcdPathPrefix,
aggregatorscheme.Codecs.LegacyCodec(), // codec is passed to etcd and hence not used
)
cmd := newCommandStartAggregator(serverOptions)
serverOptions.AddFlags(cmd.Flags())
return cli.Run(cmd)
}
pkg/cmd/grafana/apiserver/deploy/base/apiservice.yaml → pkg/cmd/grafana/apiserver/deploy/aggregator-test/apiservice.yaml
+2 -1
View File
@@ -1,3 +1,4 @@
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
@@ -11,4 +12,4 @@ spec:
service:
name: example-apiserver
namespace: grafana
port: 8443
port: 7443
pkg/cmd/grafana/apiserver/deploy/aggregator-test/externalname.yaml
+8
View File
@@ -0,0 +1,8 @@
apiVersion: service.grafana.app/v0alpha1
kind: ExternalName
metadata:
name: example-apiserver
namespace: grafana
spec:
host: localhost
pkg/cmd/grafana/apiserver/deploy/base/kustomization.yaml → pkg/cmd/grafana/apiserver/deploy/aggregator-test/kustomization.yaml
+1 -1
View File
@@ -1,3 +1,3 @@
resources:
- namespace.yaml
- apiservice.yaml
- externalname.yaml
pkg/cmd/grafana/apiserver/deploy/base/namespace.yaml
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: grafana
pkg/cmd/grafana/apiserver/deploy/darwin/kustomization.yaml
-5
View File
@@ -1,5 +0,0 @@
namespace: grafana
resources:
- ../base
- service.yaml
pkg/cmd/grafana/apiserver/deploy/darwin/service.yaml
-10
View File
@@ -1,10 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: example-apiserver
spec:
type: ExternalName
externalName: host.docker.internal
ports:
- port: 8443
name: https
pkg/cmd/grafana/apiserver/deploy/linux/kustomization.yaml
-5
View File
@@ -1,5 +0,0 @@
namespace: grafana
resources:
- ../base
- service.yaml
pkg/cmd/grafana/apiserver/deploy/linux/service.yaml
-23
View File
@@ -1,23 +0,0 @@
---
apiVersion: v1
kind: Endpoints
metadata:
name: example-apiserver
subsets:
- addresses:
- ip: 172.17.0.1 # this is the gateway IP in the "bridge" docker network
ports:
- appProtocol: https
port: 8443
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: example-apiserver
spec:
ports:
- protocol: TCP
appProtocol: https
port: 8443
targetPort: 8443
pkg/cmd/grafana/apiserver/server.go
+31 -17
View File
@@ -32,9 +32,11 @@ const (
var (
Scheme = runtime.NewScheme()
Codecs = serializer.NewCodecFactory(Scheme)
)
unversionedVersion = schema.GroupVersion{Group: "", Version: "v1"}
unversionedTypes = []runtime.Object{
func init() {
unversionedVersion := schema.GroupVersion{Group: "", Version: "v1"}
unversionedTypes := []runtime.Object{
&metav1.Status{},
&metav1.WatchEvent{},
&metav1.APIVersions{},
@@ -42,9 +44,6 @@ var (
&metav1.APIGroup{},
&metav1.APIResourceList{},
}
)
func init() {
// we need to add the options to empty v1
metav1.AddToGroupVersion(Scheme, schema.GroupVersion{Group: "", Version: "v1"})
Scheme.AddUnversionedTypes(unversionedVersion, unversionedTypes...)
@@ -116,9 +115,18 @@ func (o *APIServerOptions) ModifiedApplyTo(config *genericapiserver.RecommendedC
if err := o.RecommendedOptions.Audit.ApplyTo(&config.Config); err != nil {
return err
}
//if err := o.RecommendedOptions.Features.ApplyTo(&config.Config); err != nil {
// return err
//}
// TODO: determine whether we need flow control (API priority and fairness)
// We can't assume that a shared informers config was provided in standalone mode and will need a guard
// when enabling below
/* kubeClient, err := kubernetes.NewForConfig(config.ClientConfig)
if err != nil {
return err
}
if err := o.RecommendedOptions.Features.ApplyTo(&config.Config, kubeClient, config.SharedInformerFactory); err != nil {
return err
} */
if err := o.RecommendedOptions.CoreAPI.ApplyTo(config); err != nil {
return err
@@ -139,7 +147,11 @@ func (o *APIServerOptions) Config() (*genericapiserver.RecommendedConfig, error)
}
o.RecommendedOptions.Authentication.RemoteKubeConfigFileOptional = true
o.RecommendedOptions.Authorization.RemoteKubeConfigFileOptional = true
// TODO: determine authorization, currently insecure because Authorization provided by recommended options doesn't work
// reason: an aggregated server won't be able to post subjectaccessreviews (Grafana doesn't have this kind)
// exact error: the server could not find the requested resource (post subjectaccessreviews.authorization.k8s.io)
o.RecommendedOptions.Authorization = nil
o.RecommendedOptions.Admission = nil
o.RecommendedOptions.Etcd = nil
@@ -160,6 +172,9 @@ func (o *APIServerOptions) Config() (*genericapiserver.RecommendedConfig, error)
}
}
serverConfig.DisabledPostStartHooks = serverConfig.DisabledPostStartHooks.Insert("generic-apiserver-start-informers")
serverConfig.DisabledPostStartHooks = serverConfig.DisabledPostStartHooks.Insert("priority-and-fairness-config-consumer")
// Add OpenAPI specs for each group+version
defsGetter := grafanaAPIServer.GetOpenAPIDefinitions(o.builders)
serverConfig.OpenAPIConfig = genericapiserver.DefaultOpenAPIConfig(
@@ -190,6 +205,7 @@ func (o *APIServerOptions) Complete() error {
func (o *APIServerOptions) RunAPIServer(config *genericapiserver.RecommendedConfig, stopCh <-chan struct{}) error {
delegationTarget := genericapiserver.NewEmptyDelegate()
completedConfig := config.Complete()
server, err := completedConfig.New("example-apiserver", delegationTarget)
if err != nil {
return err
@@ -210,14 +226,12 @@ func (o *APIServerOptions) RunAPIServer(config *genericapiserver.RecommendedConf
}
}
// in standalone mode, write the local config to disk
if o.RecommendedOptions.CoreAPI == nil {
if err = clientcmd.WriteToFile(
utils.FormatKubeConfig(server.LoopbackClientConfig),
path.Join(dataPath, "grafana.kubeconfig"),
); err != nil {
return err
}
// write the local config to disk
if err = clientcmd.WriteToFile(
utils.FormatKubeConfig(server.LoopbackClientConfig),
path.Join(dataPath, "apiserver.kubeconfig"),
); err != nil {
return err
}
return server.PrepareRun().Run(stopCh)
pkg/cmd/grafana/main.go
+13
View File
@@ -45,6 +45,19 @@ func main() {
return nil
},
},
gsrv.ServerCommand(version, commit, enterpriseCommit, buildBranch, buildstamp),
{
// The kube-aggregator inspired grafana aggregator
Name: "aggregator",
Usage: "run grafana aggregator (experimental)",
// Skip parsing flags because the command line is actually managed by cobra
SkipFlagParsing: true,
Action: func(context *cli.Context) error {
// exit here because apiserver handles its own error output
os.Exit(apiserver.RunCobraWrapper())
return nil
},
},
},
CommandNotFound: cmdNotFound,
EnableBashCompletion: true,