grafana/docs/sources/developers/kinds/maturity.md

keywords	title	weight
grafana schema maturity	Grafana Kinds - From Zero to Maturity	300

Grafana Kinds - From Zero to Maturity

Grafana’s schema, Kind, and related codegen systems are under intense development.

Fear of unknown impacts leads to defensive coding, slow PRs, circular arguments, and an overall hesitance to engage. That friction alone is sufficient to sink a large-scale project. This guide seeks to counteract this friction by defining an end goal for all schemas: “mature.” This is the word we’re using to refer to the commonsense notion of “this software reached 1.0.”

In general, 1.0/mature suggests: “we’ve thought about this thing, done the necessary experimenting, know what it is, and feel confident about presenting it to the world.” In the context of schemas intended to act as a single source of truth driving many use cases, we can intuitively phrase maturity as:

• The schema follows general best practices (e.g. good comments, follows field type rules), and the team owning the schema believes that the fields described in the schema are accurate.
• Automation propagates the schema as source of truth to every relevant domain (for example: types in frontend, backend, as-code; plugins SDK; docs; APIs and storage; search indexing)

This intuitive definition gets us pointed in the right direction. But we can’t just jump straight there - we have to approach it methodically. To that end, this doc outlines four (ok five, but really, four) basic maturity milestones that we expect Kinds and their schemas to progress through:

• (Planned - Put a Kind name on the official TODO list)
• Merged - Get an initial schema written down. Not final. Not perfect.
• Experimental - Kind schemas are the source of truth for basic working code.
• Stable - Kind schemas are the source of truth for all target domains.
• Mature - The operational transition path for the Kind is battle-tested and reliable.

These milestones have functional definitions, tied to code and enforced in CI. A Kind having reached a particular milestone corresponds to properties of the code that are enforced in CI; advancing to the next milestone likely has a direct impact on code generation and runtime behavior.

Finally, the above definitions imply that maturity for individual Kinds/schemas depends on the Kind system being mature, as well. This is by design: Grafana Labs does not intend to publicize any single schema as mature until certain schema system milestones are met.

Schema Maturity Milestones

Maturity milestones are a linear progression. Each milestone implies that the conditions of its predecessors continue to be met.

Reaching a particular milestone implies that the properties of all prior milestones are still met.

(Milestone 0 - Planned)

Goal	Put a Kind name on the official TODO list: Kind Schematization Progress Tracker
Reached when	The planned Kind is listed in the relevant sheet of the progress tracker with a link to track / be able to see when exactly it is planned and who is responsible for doing it
Common hurdles	Existing definitions may not correspond clearly to an object boundary - e.g. playlists are currently in denormalized SQL tables playlist and playlist_item
Public-facing guarantees	None
customer-facing stage	None

Milestone 1 - Merged

Goal	Get an initial schema written down. Not final. Not perfect.
Reached when	A PR introducing the initial version of a schema has been merged.
Common hurdles	Getting comfortable with Thema and CUE Figuring out where all the existing definitions of the Kind are Knowing whether it’s safe to omit possibly-crufty fields from the existing definitions when writing the schema
Public-facing guarantees	None
User-facing stage	None

Milestone 2 - Experimental

Goal	Schemas are the source of truth for basic working code.
Reached when	Go and TypeScript types generated from schema are used in all relevant production code, having replaced handwritten type definitions (if any).
Common hurdles	Compromises on field definitions that seemed fine to reach “committed” start to feel unacceptable Ergonomics of generated code may start to bite Aligning with the look and feel of related schemas
Public-facing guarantees	Kinds are available for as-code usage in grok, and in tools downstream of grok, following all of grok’s standard patterns.
Stage comms	Internal users:- Start using the schema and give feedback internally to help move to the next stage.External users:- Align with the experimental stage in the release definition document.  - Experimental schemas will be discoverable, and from a customer PoV should never be used in production, but they can be explored and we are more than happy to receive feedback

Schema-writing guidelines

Avoid anonymous nested structs

Always name your sub-objects.

In CUE, nesting structs is like nesting objects in JSON, and just as easy:

one: {
  two: {
    three: {
  }
}

While these can be accurately represented in other languages, they aren’t especially friendly to work with:

// TypeScript
export interface One {
  two: {
    three: string;
  };
}

// Go
type One struct {
  Two struct {
    Three string `json:"three"`
  } `json:"two"`
}

Instead, within your schema, prefer to make root-level definitions with the appropriate attributes:

// Cue
one: {
  two: #Two
  #Two: {
    three: string
  } @cuetsy(kind="interface")
}

// TypeScript
export interface Two {
  three: string;
}
export interface One {
  two: Two;
}

// Go
type One struct {
  Two Two `json:"two"`
}
type Two struct {
  Three string `json:"three"`
}

Use precise numeric types

Use precise numeric types like float64 or uint32. Never use number.

Never use number for a numeric type in a schema.

Instead, use a specific, sized type like int64 or float32. This makes your intent precisely clear. TypeScript will still represent these fields with number, but other languages (e.g. Go, Protobuf) can be more precise.

Unlike in Go, int and uint are not your friends. These correspond to math/big types. Use a sized type, like uint32 or int32, unless the use case specifically requires a huge numeric space.

No explicit null

Do not use null as a type in any schema.

This one is tricky to think about, and requires some background.

Historically, Grafana’s dashboard JSON has often contained fields with the explicit value null. This was problematic, because explicit null introduces an ambiguity: is a JSON field being present with value null meaningfully different from the field being absent? That is, should a program behave differently if it encounters a null vs. an absent field?

In almost all cases, the answer is “no.” Thus, the ambiguity: if both explicit null and absence are accepted by a system, it pushes responsibility onto anyone writing code in that system to decide, case-by-case, whether the two are intended to be meaningfully different, and therefore whether behavior should be different.

CUE does have a null type, and only accepts data containing nulls as valid if the schema explicitly allows a null. That means, by default, using CUE for schemas removes the possibility of ambiguity in code that receives data validated by those schemas, even if the language they’re writing in still allows for ambiguity. (Javascript does, Go doesn’t.)

As a schema author, this means you’re being unambiguous by default - no nulls. That’s good! The only question is whether it’s worth explicitly allowing a null for some particular case:

someField: int32 | null

The only time this may be a good idea is if your field needs to be able to represent a value that is not otherwise acceptable within the value space - for example, if someField needs to be able to contain Infinity. When such values are serialized to null by default, it can be convenient to accept null in the schema - but even then, explicit null is unlikely to be the best way to represent such values, because it is so subtle and falsey.

Above all, DO NOT accept null in a schema simply because current behavior sometimes unintentionally produces a null. Schematization is an opportunity to get rid of this ambiguity. Fix the accidental null-producing behavior, instead.

Issues

• If a schema has a "kind" field and its set as enum, it generates a Kind alias that conflicts with the generated Kind struct.
• Byte fields are existing in Go but not in TS, so the generator fails.
• omitempty is useful when we return things like json.RawMessage (alias of []byte) because Postgres saves this information as nil, when MySQL and SQLite save it as {}. If we found it in the rest of the cases, it isn't necessary to set ? in the field in the schema.

Schema Attributes

Grafana’s schema system relies on CUE attributesdeclared on properties within schemas to control some aspects of code generation behavior. In a schema, an attribute is the whole of @cuetsy(kind=”type”):

field: string @cuetsy(kind="type")

CUE attributes are purely informational - they cannot influence CUE evaluation behavior, including the types being expressed in a Thema schema.

CUE attributes have three parts. In @cuetsy(kind=”type”), those are:

• name - @cuetsy
• arg - kind
• argval - “type”

Any given attribute may consist of {name}, {name,arg}, or {name,arg,argval}. These three levels form a tree (meaning of any argval is specific to its arg, which is specific to its name). The following documentation represents this tree using a header hierarchy.

@cuetsy

These attributes control the behavior of the cuetsy code generator, which converts CUE to TypeScript. We include only the kind arg here for brevity; cuetsy’s README has the canonical documentation on all supported args and argvals, and their intended usage.

Notes:

• Only top-level fields in a Thema schema are scanned for @cuetsy attributes.
• Grafana’s code generators hardcode that an interface (@cuetsy(kind=”interface”)) is generated to represent the root schema object, unless it is known to be a grouped lineage.

kind

Indicates the kind of TypeScript symbol that should be generated for that schema field.

interface

Generate the schema field as a TS interface. Field must be struct-kinded.

enum

Generate the schema field as a TS enum. Field must be either int-kinded (numeric enums) or string-kinded (string enums).

type

Generate the schema field as a TS type alias.

@grafana

These attributes control code generation behaviors that are specific to Grafana core. Some may also be supported in plugin code generators.

TSVeneer

Applying a TSVeneer arg to a field in a schema indicates that the schema author wants to enrich the generated type (for example by adding generic type parameters), so code generation should expect a handwritten veneer.

TSVeneer requires at least one argval, each of which impacts TypeScript code generation in its own way. Multiple argvals may be given, separated by |.

A TSVeneer arg has no effect if it is applied to a field that is not exported as a standalone TypeScript type (which usually means a CUE field that also has an @cuetsy(kind=) attribute).

type

A handwritten veneer is needed to refine the raw generated TypeScript type, for example by adding generics. See the dashboard types veneer for an example, and some corresponding CUE attributes.

@grafanamaturity

These attributes are used to support iterative development of a schema towards maturity.

Grafana code generators and CI enforce that schemas marked as mature MUST NOT have any @grafanamaturity attributes.

NeedsExpertReview

Indicates that a non-expert on that schema wrote the field, and was not fully confident in its type and/or docs.

Primarily useful on very large schemas, like the dashboard schema, for getting something written down for a given field that at least makes validation tests pass, but making clear that the field isn’t necessarily properly correct.

No argval is accepted. (Use a // comment to say more about the attention that’s needed.)