Users deploying ZITADEL against a managed PostgreSQL service (RDS, Cloud
SQL, Azure Database, etc.) often do not have superuser access and cannot
provide `Admin.*` credentials. The documented workaround — provisioning
the user and database manually and then running `start-from-setup` —
silently skips schema bootstrapping, causing `relation
"eventstore.events" does not exist` errors with no clear recovery path.
The root cause is that `zitadel init` conflates two steps that require
different privileges without exposing them separately:
- **Provisioning step** (`CREATE ROLE`, `CREATE DATABASE`, `GRANT`) —
requires superuser.
- **Schema bootstrapping step** (create
`eventstore`/`projections`/`system` schemas and base tables) — requires
only DB owner.
Users who handle the provisioning step externally have no supported way
to run schema bootstrapping alone.
## Changes
- **`cmd/initialise/verify_schema.go`** (renamed from
`verify_zitadel.go`): Rename `newZitadel()` → `newSchema()` (internal);
rename the `init zitadel` sub-command to `zitadel init schema`
(backwards-compatible alias kept) with a clear description that it
bootstraps the ZITADEL database schema without admin/superuser
privileges. Fix stale error log message to reference `init schema`.
- **`cmd/initialise/verify_schema_test.go`** (renamed from
`verify_zitadel_test.go`): Test file renamed to match source file.
- **`cmd/initialise/init.go`**: Update call site to `newSchema()`. Add
guidance in the `init` command's Long description about using `zitadel
init schema` for users without admin credentials.
- **`cmd/initialise/verify_database.go`**: Add a `pg_database` catalog
pre-check before attempting `CREATE DATABASE`, so `zitadel init` with
`ADMIN=service_user` no longer fails with `permission denied to create
database` when the database was already provisioned externally.
- **`cmd/initialise/verify_database_test.go`**: Add test cases covering
the new catalog-check skip path, the existing error-skip path, and the
error-propagation path when the `pg_database` query itself fails.
- **`apps/docs/content/self-hosting/manage/database/index.mdx`**: Inline
the `_postgres.mdx` partial (now only PostgreSQL is supported), add a
top-level callout, and add a **Managed PostgreSQL / No Admin Access**
section with explicit 3-step instructions and security guidance (strong
passwords, SSL, TLS). Additional improvements: add inline `# Use
'require' or 'verify-full' for production` comments on `Mode: disable`
lines in the YAML example; add clarifying comment to the redundant
`GRANT` in the SQL snippet; replace admonition syntax with proper
Fumadocs `<Callout>` components; add full database connection env vars
to the `start-from-setup` example.
- **`apps/docs/content/self-hosting/manage/updating_scaling.mdx`**:
Rewrite the init phase description to clearly distinguish the
provisioning and schema bootstrapping steps, and document `zitadel init
schema` as the path for manual provisioning. Replace admonition syntax
with proper Fumadocs `<Callout>` components.
- **`apps/docs/content/self-hosting/manage/database/_postgres.mdx`**:
Deleted (content merged into `index.mdx`).
## Problem
This relates to https://github.com/zitadel/zitadel/discussions/9363
I think we can improve our UX in cases where a user wants to use an
external DB and/or does not want to share too broad permissions with
zitadel
## Related problems
* https://github.com/zitadel/zitadel/issues/10432
* https://github.com/zitadel/zitadel/discussions/8583
* https://github.com/zitadel/zitadel/issues/7903
* https://github.com/zitadel/zitadel/issues/9718
* https://github.com/zitadel/zitadel/issues/8012
* https://github.com/zitadel/zitadel/issues/8558
## Related PRs
https://github.com/zitadel/zitadel/pull/11021
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
This pull request addresses an issue where ZITADEL's initialization
process could fail in environments with restricted database permissions,
such as managed database services (e.g., Google Cloud SQL, Amazon RDS,
Azure PostgreSQL).
## Which problem is solved
Previously, `zitadel init` (and its sub commands) and `zitadel
start-from-init` commands would attempt to execute `CREATE DATABASE`,
`CREATE USER`, and `GRANT` statements. These commands often fail if the
provided database user lacks superuser privileges, which is a common
security practice in hosted database environments.
## How the problem is solved
With this change, Zitadel is now smarter during initialization. It
checks if the database and user specified in your configuration already
exist and are accessible. If they are, ZITADEL will skip the creation
and grant commands, allowing for a seamless setup even with a
less-privileged database user.
## Additional information
closes#10730
closes https://github.com/zitadel/zitadel/issues/8198
### How to leverage this feature (skip admin commands)
To have ZITADEL skip the database administration commands, you need to
configure it so that the initialization process can use the
already-existing database and user. This is achieved by pointing the
admin access in your ZITADEL configuration to the user and database that
ZITADEL will use for its operations.
In your ZITADEL configuration file (e.g., zitadel.yaml), configure the
Database section as follows:
```yaml
Database:
Postgres:
Database: existing_database
# Admin credentials should be the same as the application user credentials
Admin:
User: zitadel-user
ExistingDatabase: existing_database # this is used to connect the admin to the previously created database
# Application user credentials
User:
User: zitadel-user
```
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
Zitadel currently uses 3 database pool, 1 for queries, 1 for pushing
events and 1 for scheduled projection updates. This defeats the purpose
of a connection pool which already handles multiple connections.
During load tests we found that the current structure of connection
pools consumes a lot of database resources. The resource usage dropped
after we reduced the amount of database pools to 1 because existing
connections can be used more efficiently.
# How the Problems Are Solved
Removed logic to handle multiple connection pools and use a single one.
# Additional Changes
none
# Additional Context
part of https://github.com/zitadel/zitadel/issues/8352
# Which Problems Are Solved
Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.
The API accepts the generation of the following key types and
parameters:
- RSA keys with 2048, 3072 or 4096 bit in size and:
- Signing with SHA-256 (RS256)
- Signing with SHA-384 (RS384)
- Signing with SHA-512 (RS512)
- ECDSA keys with
- P256 curve
- P384 curve
- P512 curve
- ED25519 keys
# How the Problems Are Solved
Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.
Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:
1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted
# Additional Changes
- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files
# Additional Context
- Closes#8030
- Part of #7809
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
The v2beta services are stable but not GA.
# How the Problems Are Solved
The v2beta services are copied to v2. The corresponding v1 and v2beta
services are deprecated.
# Additional Context
Closes#7236
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
Adds the possibility to mirror an existing database to a new one.
For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.
Sub commands:
* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
* `zitadel mirror projections`: runs all projections
* `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.
The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances
The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table
# Additional Changes
A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.
It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.
# Additional info
It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.
It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.
closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486
### Definition of Ready
- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
chore(fmt): run gci on complete project
Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.
Co-authored-by: Elio Bischof <elio@zitadel.com>
* chore: use pgx v5
* chore: update go version
* remove direct pq dependency
* remove unnecessary type
* scan test
* map scanner
* converter
* uint8 number array
* duration
* most unit tests work
* unit tests work
* chore: coverage
* go 1.21
* linting
* int64 gopfertammi
* retry go 1.22
* retry go 1.22
* revert to go v1.21.5
* update go toolchain to 1.21.8
* go 1.21.8
* remove test flag
* go 1.21.5
* linting
* update toolchain
* use correct array
* use correct array
* add byte array
* correct value
* correct error message
* go 1.21 compatible
* fix(db): add additional connection pool for projection spooling
* use correct connection pool for projections
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
Florian Forster
Tim MöhlmannandSilvan
Tim ZookandSilvan
Elio Bischof
