IntenseFinance/zitadel
3
0
Fork 0
mirror of https://github.com/zitadel/zitadel.git synced 2026-08-17 16:35:14 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs(guide): Using Prometheus to scrape Zitadel metrics (#11255)

Browse Source 
Resolves #6062
This commit is contained in:
Matías Racedo
2026-01-02 12:58:18 -03:00
committed by GitHub
parent 9a29eebd72
commit ce85d8e8ed
3 changed files with 403 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/self-hosting/manage/metrics/overview.mdx
+71
View File
@@ -0,0 +1,71 @@
---
title: Metrics
description: Overview of ZITADEL metrics, exposure endpoint, and compatibility with monitoring systems.
---
# Metrics
ZITADEL exposes operational and performance metrics to support monitoring, alerting, and capacity planning.
Metrics are **enabled by default** in standard deployments and are exposed via an HTTP endpoint that can be consumed by Prometheus and other monitoring systems that understand the Prometheus exposition format or use an OpenTelemetry pipeline.
## Metrics endpoint
ZITADEL exposes metrics at the following endpoint:
`/debug/metrics`
This endpoint returns a snapshot of the current metrics in **Prometheus-compatible exposition format**.
If you changed ports, paths, or network exposure as part of your deployment (for example via Helm values, reverse proxies, or ingress), adjust the endpoint accordingly.
## Instrumentation model
Internally, ZITADEL instruments metrics using **OpenTelemetry (OTel)**.
The exposed metrics include:
- **Process and runtime metrics**
(for example Go runtime, memory usage, goroutines)
- **HTTP and gRPC server metrics**
(request counts, latencies, error rates)
- **ZITADEL-specific metrics**
(server behavior and internal components available in your version)
The exact set of metrics and labels may evolve between ZITADEL releases.
## Collection and compatibility
The `/debug/metrics` endpoint can be consumed in multiple ways:
- **Prometheus** can scrape the endpoint directly.
- **OpenTelemetry Collector** can scrape the endpoint using a Prometheus receiver and forward metrics to other backends.
- **Managed monitoring platforms** (for example Grafana Cloud, Amazon Managed Service for Prometheus, Datadog, New Relic) can ingest the same endpoint via Prometheus-compatible agents or collectors.
ZITADEL does **not** require a specific metrics backend. Any system capable of scraping or ingesting Prometheus-format metrics can be used.
## Configuration
Metrics are enabled by default in standard ZITADEL deployments.
If metrics were explicitly disabled in your configuration, re-enable them before proceeding. The default configuration is defined in the project's `defaults.yaml` file:
- https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml
Refer to the configuration documentation for details on enabling or adjusting metrics behavior.
## Next steps
Choose how you want to collect and store metrics:
- **Prometheus** — scrape metrics directly from ZITADEL
- **OpenTelemetry Collector** — aggregate and forward metrics to another backend
- **Managed observability platforms** — use their Prometheus-compatible agents
The following sections describe concrete setups, starting with **Prometheus**.
docs/docs/self-hosting/manage/metrics/prometheus.md
+322
View File
@@ -0,0 +1,322 @@
---
title: Prometheus
description: Configure Prometheus to scrape ZITADEL metrics from /debug/metrics on Kubernetes and Docker.
---
# Prometheus
This page shows platform-agnostic ways to scrape ZITADEL metrics using:
- Kubernetes **annotation-based autodiscovery** (no Operator required)
- Kubernetes with **Prometheus Operator** using **ServiceMonitor**
- Standalone Prometheus on a VM or using **Docker**
- How to verify scraping, plus a minimal alert example
> ZITADEL’s endpoint path is `/debug/metrics`. If you changed ports or paths via your deployment tooling, adjust the examples accordingly.
## Before you begin
- You have a running ZITADEL instance reachable from your monitoring stack.
- You know the base URL and port to reach ZITADEL (e.g., `http://zitadel.zitadel.svc:8080` in Kubernetes or `http://localhost:8080` locally).
- Metrics are enabled in your runtime [configuration](https://zitadel.com/docs/self-hosting/manage/configure) (they are enabled by default in standard setups). If you explicitly disabled metrics in your configuration, re-enable them before proceeding. The (default) configuration is located in the [defaults.yaml](https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml).
## Option A — Kubernetes (annotation-based autodiscovery)
This approach is common when you **don’t use the Prometheus Operator**. Vanilla Prometheus can **auto-discover** scrape targets by reading standard annotations on Pods/Services:
```yaml
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/debug/metrics"
prometheus.io/port: "8080"
```
When your Prometheus server is configured with **Kubernetes service discovery** and **relabeling rules** that honor these annotations, it will automatically discover and scrape ZITADEL without any per-target `scrape_configs`.
### 1) Ensure Prometheus has Kubernetes discovery and relabeling
Your Prometheus configuration (often installed via Helm) should include jobs like the following. These are canonical examples that keep annotated Pods and map the annotated path/port to the actual metrics endpoint:
```yaml
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Only keep pods with prometheus.io/scrape: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
# Use prometheus.io/path for the metrics path
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
# Replace the address with <pod_ip>:<prometheus.io/port>
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
```
> Most Prometheus Helm charts already ship with similar discovery jobs and relabeling rules. If you installed Prometheus via Helm, you likely already have these in place.
### 2) Annotate ZITADEL Pods (or the Service)
If you deploy ZITADEL via Helm and the chart emits scrape annotations on the Deployment/Pods, no extra work is needed. Otherwise, add the annotations yourself (via values override or a strategic patch):
```yaml
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/debug/metrics"
prometheus.io/port: "8080"
```
If you prefer annotating the Service and your Prometheus config uses `role: service` discovery, add the same keys to the Service metadata.
### 3) RBAC
Prometheus must have permission to **list/watch** Pods/Endpoints in the target namespaces. Ensure its ServiceAccount has the standard ClusterRole/ClusterRoleBinding for discovery. Missing RBAC typically shows up as discovery errors in the Prometheus logs.
## Option B — Kubernetes with Prometheus Operator (ServiceMonitor)
If you run **kube-prometheus-stack** or the **Prometheus Operator**, use a **ServiceMonitor** (or **PodMonitor**). ZITADEL’s Helm chart provides **out-of-the-box** [ServiceMonitor](https://github.com/zitadel/zitadel-charts/blob/main/charts/zitadel/templates/servicemonitor.yaml) support that you can enable via values—no manual YAML is required.
### Enable the built-in ServiceMonitor (recommended)
```bash
helm repo add zitadel https://charts.zitadel.com
helm upgrade --install zitadel zitadel/zitadel \
--namespace zitadel --create-namespace \
--set metrics.enabled=true \
--set metrics.serviceMonitor.enabled=true
```
**Optional but common settings:**
* Make the ServiceMonitor discoverable by your Prometheus (Operator) instance (many stacks match a label like `release=kube-prometheus-stack`):
```bash
--set metrics.serviceMonitor.additionalLabels.release=kube-prometheus-stack
```
* Place the ServiceMonitor in a central monitoring namespace (default is the ZITADEL release namespace):
```bash
--set metrics.serviceMonitor.namespace=monitoring
```
* Tune scraping:
```bash
--set metrics.serviceMonitor.scrapeInterval=15s \
--set metrics.serviceMonitor.scrapeTimeout=10s
```
* Network/TLS customization (use only if you need them):
```bash
--set metrics.serviceMonitor.scheme=https \
--set metrics.serviceMonitor.tlsConfig.insecureSkipVerify=true
```
* Relabeling:
```bash
--set metrics.serviceMonitor.relabellings[0].action=replace \
--set metrics.serviceMonitor.metricRelabellings[0].action=drop
```
### What the chart generates (explained)
The chart renders a `ServiceMonitor` roughly equivalent to:
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: <release-name> # {{ include "zitadel.fullname" . }}
# namespace: <metrics.serviceMonitor.namespace> # only if you set it
labels:
# Standard chart labels + any you add:
# {{- include "zitadel.start.labels" . | nindent 4 }}
# {{- toYaml .Values.metrics.serviceMonitor.additionalLabels | nindent 4 }}
spec:
jobLabel: <release-name> # {{ include "zitadel.fullname" . }}
namespaceSelector:
matchNames:
- "<release-namespace>" # defaults to the Helm release namespace
selector:
matchLabels:
# Matches the ZITADEL Service created by the chart
# {{- include "zitadel.service.selectorLabels" . | nindent 6 }}
endpoints:
- port: "<protocol>-server" # e.g., "http-server" or "https-server"
path: /debug/metrics
# Optional tunables below are included only if set:
interval: <metrics.serviceMonitor.scrapeInterval>
scrapeTimeout: <metrics.serviceMonitor.scrapeTimeout>
scheme: <metrics.serviceMonitor.scheme> # http|https
tlsConfig: # metrics.serviceMonitor.tlsConfig
# ...
proxyUrl: <metrics.serviceMonitor.proxyUrl>
honorLabels: <metrics.serviceMonitor.honorLabels>
honorTimestamps: <metrics.serviceMonitor.honorTimestamps>
relabelings: # metrics.serviceMonitor.relabellings
# ...
metricRelabelings: # metrics.serviceMonitor.metricRelabellings
# ...
```
**Details that matter:**
* **Port name**: The chart uses
`port: {{ regexReplaceAll "\\W+" .Values.service.protocol "-" }}-server`
which resolves to **`http-server`** when `service.protocol=http` (default) or **`https-server`** when `service.protocol=https`. You do **not** need to edit this—just make sure you didn’t rename the ZITADEL Service port.
* **Namespace selection**: By default, the ServiceMonitor targets the **ZITADEL release namespace** via:
```yaml
namespaceSelector:
matchNames:
- "<release-namespace>"
```
Set `metrics.serviceMonitor.namespace` if you want the ServiceMonitor object itself to live elsewhere (e.g., `monitoring`). The `selector.matchLabels` still points to the ZITADEL **Service** labels.
* **Labels for discovery**: If your Prometheus (Operator) instance selects ServiceMonitors by label (common in kube-prometheus-stack), add those under `metrics.serviceMonitor.additionalLabels`—for example:
```yaml
metrics:
serviceMonitor:
additionalLabels:
release: kube-prometheus-stack
```
* **Path**: The chart fixes the metrics path to **`/debug/metrics`** (matches ZITADEL’s endpoint).
### Manual ServiceMonitor (only if you don’t want Helm to render it)
If you prefer to manage the `ServiceMonitor` yourself, keep it aligned with the chart’s conventions:
* Target the ZITADEL **Service** (not pods) using the same selector labels the chart adds.
* Use the correct **port name** (`http-server` or `https-server`) and **path** (`/debug/metrics`).
* Ensure your Prometheus (Operator) selects this ServiceMonitor by label/namespace.
> **Tip**
> If you already run the **Prometheus Operator**, prefer this ServiceMonitor approach. If you run **vanilla Prometheus without the Operator**, consider the **annotation-based discovery** method instead (Option A).
## Option C — Standalone Prometheus (VM / Docker)
If you run Prometheus outside of Kubernetes, add a static job pointing at ZITADEL’s metrics endpoint:
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "zitadel"
metrics_path: "/debug/metrics"
scheme: "http" # use https if TLS is enabled for Zitadel
static_configs:
- targets: ["<ZITADEL_HOST>:8080"] # e.g., "localhost:8080", "zitadel.internal:8080" or "host.docker.internal:8080"
```
In this snippet, replace `<ZITADEL_HOST>:8080` with the appropriate address. This could be localhost:8080 for local deployments, or a DNS name / IP of the server or Kubernetes service where ZITADEL is running. If ZITADEL is behind a reverse proxy or ingress, ensure that `/debug/metrics` is reachable (you might expose it internally only). The metrics_path is set to /debug/metrics to match ZITADEL’s endpoint. We use http scheme assuming an internal/non-TLS endpoint; if you have enabled TLS on ZITADEL, use https and the appropriate port (e.g., `443`) and adjust any hostname (like zitadel.yourdomain.com).
> When running **Prometheus in Docker** on your workstation:
>
> - **macOS/Windows**: if ZITADEL runs on your host, use `host.docker.internal:8080`.
> - **Linux**: either add `--add-host=host.docker.internal:host-gateway` to `docker run`, attach Prometheus to the same Docker network as ZITADEL and use the service name (e.g., `zitadel:8080`), or run Prometheus with `--network host` (Linux only).
## Verify scraping
Use Prometheus’s built-in UI to confirm your target is up:
1. Open the UI (default `http://localhost:9090`).
2. Go to **Status → Targets** and find the `zitadel` target. It should be **UP**.
3. Go to **Graph** and run:
```promql
up{job="zitadel"}
```
Value **1** indicates successful scrapes.
To explore ZITADEL metrics, type `zitadel` in the expression box and pick from auto-complete. Common families include:
- **Go runtime / process** (e.g., `go_goroutines`, `process_resident_memory_bytes`)
- **HTTP / gRPC** request counters/latencies (names vary by build and exporters)
- **ZITADEL server / DB** metrics (e.g., connection pool counts or other instrumentation available in your version)
> Metric names and labels can change between releases. Use the **Graph → Insert metric at cursor** dropdown to discover what your instance exposes.
## Minimal alerting example
Create a basic rule to alert when ZITADEL stops scraping:
```yaml
# /etc/prometheus/alerts/zitadel.yml
groups:
- name: zitadel-basic
rules:
- alert: ZitadelTargetDown
expr: up{job="zitadel"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "ZITADEL metrics target is down"
description: "Prometheus has not scraped ZITADEL successfully for 5 minutes."
```
Reference the rule file in your Prometheus config:
```yaml
rule_files:
- "/etc/prometheus/alerts/*.yml"
```
(Configure Alertmanager routing according to your environment.)
## Troubleshooting
**Target is DOWN / connection refused**
- In Docker: remember **`localhost` inside the Prometheus container is the container itself**, not your host. Use `host.docker.internal:8080` (plus `--add-host=host.docker.internal:host-gateway` on Linux), or join Prometheus to the **same Docker network** as ZITADEL and use the **service name** (e.g., `zitadel:8080`), or run with `--network host` on Linux.
- In Kubernetes: verify the Service port name (for example, `http-server` or `https-server`) and path (`/debug/metrics`) match your ServiceMonitor (or annotations). Check that Prometheus has RBAC to list Pods/Endpoints.
**Metrics path mismatch**
- ZITADEL uses **`/debug/metrics`**. If you see 404s, confirm your Prometheus job or annotations aren’t still using `/metrics`.
**No targets discovered (Kubernetes)**
- If using annotations, make sure your Prometheus config has **Kubernetes discovery & relabeling** rules that honor `prometheus.io/*` annotations and that the Pods/Service are annotated.
- If using ServiceMonitor, ensure your Prometheus Operator instance **selects** the ServiceMonitor by label/namespace.
**Nothing shows up in the Graph dropdown**
- First confirm `up{job="zitadel"}` returns **1**. If yes, metrics are being scraped—start typing generic prefixes like `go_` or `process_` to explore. ZITADEL’s exported metric set can evolve; check the raw output at `/debug/metrics` to see exactly what is exposed by your version.
## Alternatives that understand OTEL/Prometheus
While Prometheus is the most common choice, other collectors and services can ingest the same endpoint:
- **Amazon Managed Service for Prometheus (AMP)** — managed, Prometheus-compatible backend on AWS.
- **AWS CloudWatch via ADOT/OTel Collector** — scrape with the OpenTelemetry Collector and export to CloudWatch Metrics.
- **Grafana Cloud / VictoriaMetrics / Thanos** — remote-write targets or managed TSDBs for Prometheus data.
- **Datadog / New Relic / Splunk Observability** — agents or OTel pipelines can ingest Prometheus-format metrics.
If you already operate one of these platforms, you can point their agents/collectors at `/debug/metrics` or use an OTel Collector with a **Prometheus receiver** and the appropriate exporter.
docs/sidebars.js
+10
View File
@@ -1165,6 +1165,16 @@ module.exports = {
"self-hosting/manage/database/database",
"self-hosting/manage/cache",
"self-hosting/manage/service_ping",
{
type: "category",
label: "Metrics",
collapsed: false,
link: {
type: "doc",
id: "self-hosting/manage/metrics/overview",
},
items: ["self-hosting/manage/metrics/prometheus"],
},
"self-hosting/manage/updating_scaling",
"self-hosting/manage/usage_control",
{