mirror of
https://github.com/grafana/grafana.git
synced 2025-02-20 11:48:34 -06:00
0fcc864bbb
* Docs: Add Schema maturity docs * Remove 'which category' section from maturity.md Co-authored-by: sam boyer <sdboyer@grafana.com> * Remove Maturity process guides from maturity.md Co-authored-by: sam boyer <sdboyer@grafana.com> * Remove Defaults and Field optionality from maturity.md Co-authored-by: sam boyer <sdboyer@grafana.com> * Remove MaybeRemove section from maturity.md Co-authored-by: sam boyer <sdboyer@grafana.com> * Remove Milestone 3 and 4 from maturity.md Co-authored-by: sam boyer <sdboyer@grafana.com> * Rearrange schema maturity docs * Regenerate schema docs after merge with main * Update Maturity docs headers, keep single h1 * Update wording in Schema docs Co-authored-by: Tania <yalyna.ts@gmail.com> * Regenerate docs after merge with main --------- Co-authored-by: sam boyer <sdboyer@grafana.com> Co-authored-by: Tania <yalyna.ts@gmail.com>
293 lines
17 KiB
Markdown
293 lines
17 KiB
Markdown
---
|
||
keywords:
|
||
- grafana
|
||
- schema
|
||
- maturity
|
||
title: Grafana Kinds - From Zero to Maturity
|
||
weight: 300
|
||
---
|
||
|
||
# Grafana Kinds - From Zero to Maturity
|
||
|
||
> Grafana’s schema, Kind, and related codegen systems are under intense development.
|
||
|
||
Fear of unknown impacts leads to defensive coding, slow PRs, circular arguments, and an overall hesitance to engage.
|
||
That friction alone is sufficient to sink a large-scale project. This guide seeks to counteract this friction by
|
||
defining an end goal for all schemas: “mature.” This is the word we’re using to refer to the commonsense notion of “this
|
||
software reached 1.0.”
|
||
|
||
In general, 1.0/mature suggests: “we’ve thought about this thing, done the necessary experimenting, know what it is, and
|
||
feel confident about presenting it to the world.” In the context of schemas intended to act as a single source of truth
|
||
driving many use cases, we can intuitively phrase maturity as:
|
||
|
||
- The schema follows general best practices (e.g. good comments, follows field type rules), and the team owning the
|
||
schema believes that the fields described in the schema are accurate.
|
||
- Automation propagates the schema as source of truth to every relevant
|
||
[domain](https://docs.google.com/document/d/13Rv395_T8WTLBgdL-2rbXKu0fx_TW-Q9yz9x6oBjm6g/edit#heading=h.67pop2k2f8fq)
|
||
(for example: types in frontend, backend, as-code; plugins SDK; docs; APIs and storage; search indexing)
|
||
|
||
This intuitive definition gets us pointed in the right direction. But we can’t just jump straight there - we have to
|
||
approach it methodically. To that end, this doc outlines four (ok five, but really, four) basic maturity milestones that
|
||
we expect Kinds and their schemas to progress through:
|
||
|
||
- *(Planned - Put a Kind name on the official TODO list)*
|
||
- **Merged** - Get an initial schema written down. Not final. Not perfect.
|
||
- **Experimental** - Kind schemas are the source of truth for basic working code.
|
||
- **Stable** - Kind schemas are the source of truth for all target domains.
|
||
- **Mature** - The operational transition path for the Kind is battle-tested and reliable.
|
||
|
||
These milestones have functional definitions, tied to code and enforced in CI. A Kind having reached a particular
|
||
milestone corresponds to properties of the code that are enforced in CI; advancing to the next milestone likely has a
|
||
direct impact on code generation and runtime behavior.
|
||
|
||
Finally, the above definitions imply that maturity for *individual Kinds/schemas* depends on *the Kind system* being
|
||
mature, as well. This is by design: **Grafana Labs does not intend to publicize any single schema as mature until
|
||
[certain schema system milestones are met](https://github.com/orgs/grafana/projects/133/views/8).**
|
||
|
||
## Schema Maturity Milestones
|
||
|
||
Maturity milestones are a linear progression. Each milestone implies that the conditions of its predecessors continue to
|
||
be met.
|
||
|
||
Reaching a particular milestone implies that the properties of all prior milestones are still met.
|
||
|
||
### (Milestone 0 - Planned) {#planned}
|
||
|
||
| **Goal** | Put a Kind name on the official TODO list: [Kind Schematization Progress Tracker](https://docs.google.com/spreadsheets/d/1DL6nZHyX42X013QraWYbKsMmHozLrtXDj8teLKvwYMY/edit#gid=0) |
|
||
|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||
| **Reached when** | The planned Kind is listed in the relevant sheet of the progress tracker with a link to track / be able to see when exactly it is planned and who is responsible for doing it |
|
||
| **Common hurdles** | Existing definitions may not correspond clearly to an object boundary - e.g. playlists are currently in denormalized SQL tables playlist and playlist_item |
|
||
| **Public-facing guarantees** | None |
|
||
| **customer-facing stage** | None |
|
||
|
||
### Milestone 1 - Merged {#merged}
|
||
|
||
| **Goal** | Get an initial schema written down. Not final. Not perfect. |
|
||
|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||
| **Reached when** | A PR introducing the initial version of a schema has been merged. |
|
||
| **Common hurdles** | Getting comfortable with Thema and CUE<br/>Figuring out where all the existing definitions of the Kind are<br/>Knowing whether it’s safe to omit possibly-crufty fields from the existing definitions when writing the schema |
|
||
| **Public-facing guarantees** | None |
|
||
| **User-facing stage** | None |
|
||
|
||
### Milestone 2 - Experimental {#experimental}
|
||
|
||
| **Goal** | Schemas are the source of truth for basic working code. |
|
||
|------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||
| **Reached when** | Go and TypeScript types generated from schema are used in all relevant production code, having replaced handwritten type definitions (if any). |
|
||
| **Common hurdles** | Compromises on field definitions that seemed fine to reach “committed” start to feel unacceptable<br/>Ergonomics of generated code may start to bite<br/>Aligning with the look and feel of related schemas |
|
||
| **Public-facing guarantees** | Kinds are available for as-code usage in [grok](https://github.com/grafana/grok), and in tools downstream of grok, following all of grok’s standard patterns. |
|
||
| **Stage comms** | Internal users:- Start using the schema and give feedback internally to help move to the next stage.External users:- Align with the [experimental](https://docs.google.com/document/d/1lqp0hALax2PT7jSObsX52EbQmIDFnLFMqIbBrJ4EYCE/edit#heading=h.ehl5iy7pcjvq) stage in the release definition document. - Experimental schemas will be discoverable, and from a customer PoV should never be used in production, but they can be explored and we are more than happy to receive feedback |
|
||
|
||
|
||
## Schema-writing guidelines
|
||
|
||
### Avoid anonymous nested structs
|
||
|
||
***Always name your sub-objects.***
|
||
|
||
In CUE, nesting structs is like nesting objects in JSON, and just as easy:
|
||
~~~ json
|
||
one: {
|
||
two: {
|
||
three: {
|
||
}
|
||
}
|
||
~~~
|
||
|
||
While these can be accurately represented in other languages, they aren’t especially friendly to work with:
|
||
|
||
~~~ typescript
|
||
// TypeScript
|
||
export interface One {
|
||
two: {
|
||
three: string;
|
||
};
|
||
}
|
||
~~~
|
||
|
||
~~~ go
|
||
// Go
|
||
type One struct {
|
||
Two struct {
|
||
Three string `json:"three"`
|
||
} `json:"two"`
|
||
}
|
||
~~~
|
||
|
||
Instead, within your schema, prefer to make root-level definitions with the appropriate attributes:
|
||
|
||
~~~ cue
|
||
// Cue
|
||
one: {
|
||
two: #Two
|
||
#Two: {
|
||
three: string
|
||
} @cuetsy(kind="interface")
|
||
}
|
||
~~~
|
||
|
||
~~~ Typescript
|
||
// TypeScript
|
||
export interface Two {
|
||
three: string;
|
||
}
|
||
export interface One {
|
||
two: Two;
|
||
}
|
||
~~~
|
||
|
||
~~~ Go
|
||
// Go
|
||
type One struct {
|
||
Two Two `json:"two"`
|
||
}
|
||
type Two struct {
|
||
Three string `json:"three"`
|
||
}
|
||
~~~
|
||
|
||
### Use precise numeric types
|
||
|
||
***Use precise numeric types like `float64` or `uint32`. Never use `number`.***
|
||
|
||
Never use `number` for a numeric type in a schema.
|
||
|
||
Instead, use a specific, sized type like `int64` or `float32`. This makes your intent precisely clear.
|
||
TypeScript will still represent these fields with `number`, but other languages (e.g. Go, Protobuf) can be more precise.
|
||
|
||
Unlike in Go, int and uint are not your friends. These correspond to `math/big` types. Use a sized type,
|
||
like `uint32` or `int32`, unless the use case specifically requires a huge numeric space.
|
||
|
||
### No explicit `null`
|
||
|
||
***Do not use `null` as a type in any schema.***
|
||
|
||
This one is tricky to think about, and requires some background.
|
||
|
||
Historically, Grafana’s dashboard JSON has often contained fields with the explicit value `null`.
|
||
This was problematic, because explicit `null` introduces an ambiguity: is a JSON field being present
|
||
with value null meaningfully different from the field being absent? That is, should a program behave differently
|
||
if it encounters a null vs. an absent field?
|
||
|
||
In almost all cases, the answer is “no.” Thus, the ambiguity: if both explicit null and absence are *accepted*
|
||
by a system, it pushes responsibility onto anyone writing code in that system to decide, case-by-case,
|
||
whether the two are *intended to be meaningfully different*, and therefore whether behavior should be different.
|
||
|
||
CUE does have a `null` type, and only accepts data containing `nulls` as valid if the schema explicitly allows a `null`.
|
||
That means, by default, using CUE for schemas removes the possibility of ambiguity in code that receives data validated
|
||
by those schemas, even if the language they’re writing in still allows for ambiguity. (Javascript does, Go doesn’t.)
|
||
|
||
As a schema author, this means you’re being unambiguous by default - no `nulls`. That’s good! The only question is
|
||
whether it’s worth explicitly allowing a `null` for some particular case:
|
||
~~~ Cue
|
||
someField: int32 | null
|
||
~~~
|
||
|
||
The *only* time this *may* be a good idea is if your field needs to be able to represent a value
|
||
that is not otherwise acceptable within the value space - for example, if `someField` needs to be able to contain
|
||
[Infinity](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/POSITIVE_INFINITY).
|
||
When such values are serialized to null by default, it can be convenient to accept null in the schema - but even then,
|
||
explicit null is unlikely to be the best way to represent such values, because it is so subtle and falsey.
|
||
|
||
**Above all, DO NOT accept `null` in a schema simply because current behavior sometimes unintentionally produces a `null`.**
|
||
Schematization is an opportunity to get rid of this ambiguity. Fix the accidental null-producing behavior, instead.
|
||
|
||
### Issues
|
||
|
||
- If a schema has a "kind" field and its set as enum, it generates a Kind alias that conflicts with the generated
|
||
Kind struct.
|
||
- Byte fields are existing in Go but not in TS, so the generator fails.
|
||
- **omitempty** is useful when we return things like json.RawMessage (alias of []byte) because Postgres saves this
|
||
information as `nil`, when MySQL and SQLite save it as `{}`. If we found it in the rest of the cases, it isn't necessary
|
||
to set `?` in the field in the schema.
|
||
|
||
|
||
## Schema Attributes
|
||
|
||
Grafana’s schema system relies on [CUE attributes](https://cuelang.org/docs/references/spec/#attributes)declared on
|
||
properties within schemas to control some aspects of code generation behavior.
|
||
In a schema, an attribute is the whole of `@cuetsy(kind=”type”)`:
|
||
|
||
~~~ Cue
|
||
field: string @cuetsy(kind="type")
|
||
~~~
|
||
|
||
CUE attributes are purely informational - they cannot influence CUE evaluation behavior, including the types being
|
||
expressed in a Thema schema.
|
||
|
||
CUE attributes have three parts. In `@cuetsy(kind=”type”)`, those are:
|
||
- name - `@cuetsy`
|
||
- arg - `kind`
|
||
- argval - `“type”`
|
||
|
||
Any given attribute may consist of `{name}`, `{name,arg}`, or `{name,arg,argval}`. These three levels form a tree
|
||
(meaning of any argval is specific to its arg, which is specific to its name). The following documentation represents
|
||
this tree using a header hierarchy.
|
||
|
||
### @cuetsy
|
||
|
||
These attributes control the behavior of the [cuetsy code generator](https://github.com/grafana/cuetsy), which converts
|
||
CUE to TypeScript. We include only the kind arg here for brevity; cuetsy’s README has the canonical documentation on all
|
||
supported args and argvals, and their intended usage.
|
||
|
||
Notes:
|
||
- Only top-level fields in a Thema schema are scanned for `@cuetsy` attributes.
|
||
- Grafana’s code generators hardcode that an interface (`@cuetsy(kind=”interface”)`) is generated to represent the root
|
||
schema object, unless it is known to be a [grouped lineage](https://docs.google.com/document/d/13Rv395_T8WTLBgdL-2rbXKu0fx_TW-Q9yz9x6oBjm6g/edit#heading=h.vx7stzpxtw4t).
|
||
|
||
#### kind
|
||
|
||
Indicates the kind of TypeScript symbol that should be generated for that schema field.
|
||
|
||
#### interface
|
||
|
||
Generate the schema field as a TS interface. Field must be struct-kinded.
|
||
|
||
#### enum
|
||
|
||
Generate the schema field as a TS enum. Field must be either int-kinded (numeric enums) or string-kinded (string enums).
|
||
|
||
#### type
|
||
|
||
Generate the schema field as a TS type alias.
|
||
|
||
### @grafana
|
||
|
||
These attributes control code generation behaviors that are specific to Grafana core. Some may also be supported
|
||
in plugin code generators.
|
||
|
||
#### TSVeneer
|
||
|
||
Applying a TSVeneer arg to a field in a schema indicates that the schema author wants to enrich the generated type
|
||
(for example by adding generic type parameters), so code generation should expect a handwritten
|
||
[veneer](https://docs.google.com/document/d/13Rv395_T8WTLBgdL-2rbXKu0fx_TW-Q9yz9x6oBjm6g/edit#heading=h.bmtjq0bb1yxp).
|
||
|
||
TSVeneer requires at least one argval, each of which impacts TypeScript code generation in its own way.
|
||
Multiple argvals may be given, separated by `|`.
|
||
|
||
A TSVeneer arg has no effect if it is applied to a field that is not exported as a standalone TypeScript type
|
||
(which usually means a CUE field that also has an `@cuetsy(kind=)` attribute).
|
||
|
||
#### type
|
||
|
||
A handwritten veneer is needed to refine the raw generated TypeScript type, for example by adding generics.
|
||
See [the dashboard types veneer](https://github.com/grafana/grafana/blob/5f93e67419e9587363d1fc1e6f1f4a8044eb54d0/packages/grafana-schema/src/veneer/dashboard.types.ts)
|
||
for an example, and [some](https://github.com/grafana/grafana/blob/5f93e67419e9587363d1fc1e6f1f4a8044eb54d0/kinds/dashboard/dashboard_kind.cue#L12)
|
||
[corresponding](https://github.com/grafana/grafana/blob/5f93e67419e9587363d1fc1e6f1f4a8044eb54d0/kinds/dashboard/dashboard_kind.cue#L143)
|
||
CUE attributes.
|
||
|
||
### @grafanamaturity
|
||
|
||
These attributes are used to support iterative development of a schema towards maturity.
|
||
|
||
Grafana code generators and CI enforce that schemas marked as mature MUST NOT have any `@grafanamaturity` attributes.
|
||
|
||
#### NeedsExpertReview
|
||
|
||
Indicates that a non-expert on that schema wrote the field, and was not fully confident in its type and/or docs.
|
||
|
||
Primarily useful on very large schemas, like the dashboard schema, for getting *something* written down for a given
|
||
field that at least makes validation tests pass, but making clear that the field isn’t necessarily properly correct.
|
||
|
||
No argval is accepted. (Use a `//` comment to say more about the attention that’s needed.)
|