mirror of
https://github.com/zitadel/zitadel.git
synced 2026-08-19 01:14:48 -05:00
561da54ff71741e95ef185cb96800e4f8a8686da
298
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tim Möhlmann
|
e94d4c3986 |
feat(crypto): FIPS 140-3 compliant build and runtime checks (#12233)
# Which Problems Are Solved Enable FIPS 140-3 compliant build. # How the Problems Are Solved - Add runtime config validation, if the FIPS flag is enabled fail the application when a non-compliant hasher is used, or throw a warning when a legacy verifier is used - Add a build matrix for FIPS certified build: - Go binary is built with `GOFIPS140=certified` - Login container uses a separate base: [ubi9](https://catalog.redhat.com/en/software/containers/ubi9/ubi/615bcf606feffc5384e8452e) from redhat which provides a FIPS certified OpenSSL (used by NodeJS TLS stack) - Non-FIPS images where already pushed to both Github Container Registry and Google Artifact Repository (GAR). Fips images are only pushed to the GAR. - Tag versions are suffixed `-fips`. So on release the following images will be additionally available: ``` europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel-login:vX.Y.Z-fips europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel:vX.Y.Z-fips-debug europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel:vX.Y.Z-fips ``` # Other changes - Bumb Go toolchain. At least v1.25.10 is required for a GOFIPS140=certified setting. # Additional Context - Closes https://github.com/zitadel/zitadel/issues/4335 - Build [test run](https://github.com/zitadel/zitadel/actions/runs/27253916052) pushing FIPS and non-FIPS images |
||
  |
25e263394e |
chore: update passwap v0.12.1 and align hash validation defaults/errors (#12179)
# Which Problems Are Solved - Upgrading to `zitadel/passwap` v0.12.1 introduced new encoded-hash validation paths that still had review feedback open. - Secret hasher defaults were internally inconsistent (`Hasher.Cost: 4` vs `Limits.Bcrypt.MinCost: 10`), which could reject hashes created by the configured hasher. - New validation error IDs/messages and test coverage needed to be aligned with project conventions and expected behavior branches. # How the Problems Are Solved - Kept the dependency upgrade to `zitadel/passwap` v0.12.1 and completed the validation integration. - Updated `ValidateEncodedHash` error handling in `internal/crypto/passwap.go` to: - use unique random-style error IDs, - return `Errors.Hash.NotSupported` for no-verifier cases, - keep invalid-hash branches mapped to invalid argument errors. - Expanded `TestHasher_ValidateEncodedHash` in `internal/crypto/passwap_test.go` to cover and assert: - bounds error branch, - no-verifier branch, - generic invalid-hash branch, - expected ZITADEL error IDs/messages. - Restored lost inline verifier-context comments for argon2 and md5plain verifier entries. # Additional Changes - Added the missing explanatory `Limits` comment for `SecretHasher` in `cmd/defaults.yaml`. - Corrected `SecretHasher.Limits.Bcrypt.MinCost` from `10` to `4` to match the configured default bcrypt cost and avoid configuration footguns. # Additional Context - Follow-up for PR review feedback in https://github.com/zitadel/zitadel/pull/12179#pullrequestreview-4313121965 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Livio Spring <9405495+livio-a@users.noreply.github.com> |
||
 Marco A.
|
c73e58489b |
chore(deps): Update go-ntlmssp (#12193)
# Which Problems Are Solved Dependabot signalling that go-ntlmssp requires an update # How the Problems Are Solved `go get -u` |
||
|
Livio Spring
|
cc74a36b65 |
chore(deps): update go dependencies (#12094)
This PR updates all (possible) go dependencies and requires a backport to v4.x |
||
 |
01fe34a526 |
fix(oidc): use authenticated encryption for opaque tokens (#12017)
# Which Problems Are Solved Opaque tokens now use authenticated encryption. # How the Problems Are Solved - Upgrade zitadel/oidc to v3.47 - Copy crypto implementation for refresh and session tokens (internal to zitadel) - Added config that allows validating old tokens for gradual roll-out # Additional Changes - Set NX cache for `integration-test-build` to `false`, working on a seperate fix. # Additional Context - closes #11315 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com> |
||
 Mridang Agarwalla
|
330548e13c | feat: support standard OTEL env vars via autoexport (#11864) | ||
|
Livio Spring
|
f94e4065c2 |
chore: update go dependencies (#11883)
Updates all dependencies to latests versions (apart from the ones where there are already issues to solve their updates). Some updates required minor changes. |
||
 
|
997aa607c4 |
feat(telemetry): unify request details (#11509)
# Which Problems Are Solved
In the "new" structured logging, request details were added in different
middlewares, such as request, instance and user IDs. This meant the the
upstream request logging middleware did not have access to metadata that
got added later, resulting in incomplete logs. Furthermore it was not
possible to correlate an API error response to log output.
# How the Problems Are Solved
A mutable request details object is added to the context early on. When
the api authz function run, the instance and user IDs are added to this
object as they become available. Every logline emitted after this
(time-wise) will then all contain these details under the `request` log
group.
<details>
<summary>example output in JSON</summary>
```json
{
"time": "2026-03-13T19:21:20.890428806Z",
"level": "INFO",
"source": {
"function": "github.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.LogHandler.func1.1",
"file": "/workspaces/zitadel/internal/api/grpc/server/connect_middleware/log_interceptor.go",
"line": 34
},
"msg": "request served",
"request": {
"id": "d6q67c04vtjmi77cbbpg",
"instance_host": "localhost:8080",
"instance_id": "362349751439458307",
"user_id": "362349751440048131"
},
"TraceID": "a9e0fee3522224f3583bbdcda737f4b4",
"SpanID": "a563056eee36920a",
"stream": "request",
"version": "2026-03-13T19:20:59Z",
"protocol": "connect",
"service": "zitadel.user.v2.UserService",
"http_method": "POST",
"path": "/zitadel.user.v2.UserService/ListUsers",
"code": "code_0",
"duration": 12254350
}
```
</details>
Request IDs are now also returned with a response header or metadata.
Depending on the protocol:
- HTTP calls always return the request ID as header, regardless of
status
- gRPC calls always return the request ID as header, even if there was
an error
- connect RPC calls returns the request ID as header on success, trailer
in case of error. This is because header must be set on the response
object, which is nil in case of error. When there is an error, metadata
can be added which are then sent as trailers.
# Additional Changes
- Use the existing call duration middleware for both request ID and
logging for a consistent request start timestamp in all layers.
- Upgrade sloggcp for some fixes (notably TraceID)
- Modify the NoCache middleware so it uses `SetHeaders` instead of
`SendHeaders`. The latter prevented any other handler from setting
headers, including the new request ID middleware.
# Additional Context
Follow up on demo of:
- https://github.com/zitadel/zitadel/pull/11159
- https://github.com/zitadel/zitadel/pull/11435
- backport to v4
---------
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Marco A. <marco@zitadel.com>
|
||
 
|
5df8a97f9d |
feat(repository): implement user repository (#10967)
## Summary This PR introduces the relational user repository for backend v3, including schema, repository APIs, reducers/projections, and tests for both human and machine users. ## What changed * Added user-focused relational migrations (users, verifications, metadata, PATs, machine keys, passkeys, identity provider links) and related FK wiring. * Implemented/expanded repository domain contracts and storage implementations for human and machine user flows. * Added support for machine secret handling and aligned access token type modeling. * Extended user domain models (verifications, password/email/phone/TOTP state, passkeys, metadata, invite handling, identity provider links). * Updated projection/reduction coverage and integration tests for key user lifecycle events. * Regenerated and updated mocks/enums plus supporting database change/column utilities. ## Notes for reviewers * This PR is large because it establishes the foundational relational user storage layer and related test scaffolding. Sorry for that. ## Closes Closes #10218 Closes #10224 Closes #10225 Closes #10209 Closes #10802 --------- Co-authored-by: Iraq Jaber <iraq+github@zitadel.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Marco A. <marco@zitadel.com> Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> |
||
|
Tim MöhlmannandSilvan
|
11dbb1b277 |
feat(logging): add streams (#11435)
# Which Problems Are Solved Streams allow differentiating logs produced by different components of Zitadel. # How the Problems Are Solved The `backend/v3/instrumentation/logging` package now exposes convenience function for setting and getting a logger from the context. As well as high-level functions to emit log records at various levels. When constructing a new logger a "stream" needs to be specified: - **runtime**: General runtime logs, such as startup and shutdown messages. Default for logs that do not belong to the other categories. - **request**: Logs for incoming API and HTTP requests. - **event_handler**: Logs for event handling in projections. - **queue**: Logs for the job queue processing. - **event_pusher**: Logs for event pushing to the database. Disabled by default, contains sensitive information. Each line from the returned logger contains a `stream` field as well as a `version` field with the current Zitadel version. ## Runtime config Streams can be enabled by passing an array of stream names in the runtime config. Because some log streams may contain sensitive data (especially events), it is now also possible to mask values by their key. # Additional Changes - Wrap `slogctx` in the `logging` package. (Except API error converter packages, because of import cycle) - Add some docs to `logging` package so other devs understand how to add logging to Zitadel - Add `logging.OnError` and `logging.WithError` helper functions with `Panic()` and `Fatal()` methods, to preserve current calls in the `cmd` packages. - Add instance context extractor. - Only output request details in the request info log. Request ID remains propagated through context. - Moved middleware functionality into protocol specific packages. - Removed setting of URI to context in metric middleware. There were ony setters and no getters. (Unused value) - Reuse a single statusWriter in the middleware package for middlewares that need to know the response status. # Additional Context - Closes #11333 - Closes #11331 - Partly #11330 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com> |
||
|
Tim MöhlmannandCopilot
|
34799389b7 |
feat(logging): gcp error reporting (#11355)
# Which Problems Are Solved Zitadel Errors (`zerrors` package) are logged by default. This change add more verbose error reporting through log output, including compatibility with GCP error reporting if enabled. # How the Problems Are Solved Errors can now carry their "report location" and a stack trace. "Report location" is enabled by default, while stack-traces are disabled. Our new [`zitadel/sloggcp`](https://github.com/zitadel/sloggcp) package provides the GCP error reporting handler for `slog`. The following section displays the different formats for the same error (User not found). Newlines and formatting of output is applied for readability and is not part of the actual log output. <details> <summary>Default text output</summary> ``` time=2026-01-15T14:27:20.625+01:00 level=WARN source=/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/gerrors/zitadel_errors.go:57 msg="User could not be found (COMMAND-bd4ir1mblj)" domain.instance_host=localhost:8080 domain.protocol=http protocol=connect service=zitadel.user.v2.UserService http_method=POST path=/zitadel.user.v2.UserService/DeleteUser request_id=d5keme0ednc50mm74bfg duration=66.761569ms TraceID=319374baa56acc2dbd1d0179f1bbec9f SpanID=541621b0abea7ff1 err.kind=NotFound err.message="User could not be found" err.id=COMMAND-bd4ir1mblj err.reportLocation.filePath=/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go err.reportLocation.lineNumber=140 err.reportLocation.functionName=github.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2 ``` </details> <details> <summary>Default JSON output</summary> ```json { "time": "2026-01-15T14:30:59.280497754+01:00", "level": "WARN", "source": { "function": "github.com/zitadel/zitadel/internal/api/grpc/gerrors.ZITADELToConnectError", "file": "/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/gerrors/zitadel_errors.go", "line": 57 }, "msg": "User could not be found (COMMAND-bd4ir1mblj)", "domain": { "instance_host": "localhost:8080", "protocol": "http" }, "protocol": "connect", "service": "zitadel.user.v2.UserService", "http_method": "POST", "path": "/zitadel.user.v2.UserService/DeleteUser", "request_id": "d5keo4oednc6klmo3vbg", "duration": 54229055, "TraceID": "daa21183bd8da5da56969b0ea45f4388", "SpanID": "70da728e3086c104", "err": { "kind": "NotFound", "message": "User could not be found", "id": "COMMAND-bd4ir1mblj", "reportLocation": { "filePath": "/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go", "lineNumber": 140, "functionName": "github.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2" } } } ``` </details> When stack trace is enabled, using a standard logger, it is logged under the `stackTrace` key. <details> <summary>Standard JSON format with stack trace</summary> ```json { "time": "2026-01-15T14:36:56.522692368+01:00", "level": "WARN", "source": { "function": "github.com/zitadel/zitadel/internal/api/grpc/gerrors.ZITADELToConnectError", "file": "/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/gerrors/zitadel_errors.go", "line": 57 }, "msg": "User could not be found (COMMAND-bd4ir1mblj)", "domain": { "instance_host": "localhost:8080", "protocol": "http" }, "protocol": "connect", "service": "zitadel.user.v2.UserService", "http_method": "POST", "path": "/zitadel.user.v2.UserService/DeleteUser", "request_id": "d5kequ0ednc7lphvgfn0", "duration": 57820793, "TraceID": "1b05d2b01306f2a64feb1a24b65a89c5", "SpanID": "c5ff35a27a917764", "err": { "kind": "NotFound", "message": "User could not be found", "id": "COMMAND-bd4ir1mblj", "reportLocation": { "filePath": "/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go", "lineNumber": 140, "functionName": "github.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2" }, "stackTrace": "goroutine 673 [running]:\nruntime/debug.Stack()\n\t/usr/lib/go/src/runtime/debug/stack.go:26 +0x8e\ngithub.com/zitadel/zitadel/internal/zerrors.newZitadelError(0x5, {0x0, 0x0}, {0x6af7719, 0x12}, {0x6aff994, 0x14})\n\t/home/tim/Repositories/zitadel/zitadel/internal/zerrors/zerror.go:139 +0x1df\ngithub.com/zitadel/zitadel/internal/zerrors.ThrowNotFound({0x0, 0x0}, {0x6af7719, 0x12}, {0x6aff994, 0x14})\n\t/home/tim/Repositories/zitadel/zitadel/internal/zerrors/not_found.go:6 +0x70\ngithub.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2(0xc0035a4008, {0xa3c1410, 0xc003ded5c0}, {0xc0029693a0, 0x3}, {0x0, 0x0}, {0xd550a60, 0x0, 0x0}, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go:140 +0x711\ngithub.com/zitadel/zitadel/internal/api/grpc/user/v2.(*Server).DeleteUser(0xc00272d080, {0xa3c1410, 0xc003ded5c0}, 0xc002ebf180)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/user/v2/user.go:191 +0x3f1\nconnectrpc.com/connect.NewUnaryHandler[...].func1({0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:51 +0x17e\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded5c0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ActivityInterceptor.func1.1({0xa3c1410, 0xc003ded5c0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/activity_interceptor.go:20 +0x1b1\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ServiceHandler.func1.1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/service_interceptor.go:22 +0x283\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.validate({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180}, 0xc0031bff60)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/validation_interceptor.go:35 +0x1bd\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ValidationHandler.func1.1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/validation_interceptor.go:15 +0x85\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ExecutionHandler.func1.1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/execution_interceptor.go:39 +0x376\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.QuotaExhaustedInterceptor.func1.1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/quota_interceptor.go:25 +0x42d\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.TranslationHandler.func1.1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/translation_interceptor.go:18 +0x94\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003ded4d0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.authorize({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180}, 0xc00321c040, {0xa3f0a70, 0xc0004ebc70}, {{0xc000d2fc80, 0x9, 0x9}}, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/auth_interceptor.go:42 +0x894\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.AuthorizationInterceptor.func1.1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/auth_interceptor.go:17 +0x151\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.LimitsInterceptor.func1.1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/limits_interceptor.go:31 +0x3b7\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.toConnectError({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180}, 0xc00321c060)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/error_interceptor.go:21 +0x8a\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ErrorHandler.func1.1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/error_interceptor.go:15 +0x85\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.AccessStorageInterceptor.func1.1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/access_interceptor.go:21 +0x814\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003dec000}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.addInstanceByRequestedHost({0xa3c1410, 0xc003db1110}, {0xa405e70, 0xc002ebf180}, 0xc00321c090, {0xa3a63e8, 0xc0002f4a00}, 0xc002dee9b0, {0xc0012106a0, 0x9})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:107 +0xbc3\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.setInstance({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180}, 0xc00321c090, {0xa3a63e8, 0xc0002f4a00}, {0xc0012106a0, 0x9}, 0xc002dee9b0, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:61 +0xb05\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.InstanceInterceptor.func1.1({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:23 +0x147\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.NoCacheInterceptor.func1.1({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/cache_interceptor.go:21 +0x351\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/backend/v3/instrumentation/logging.NewConnectInterceptor.func1({0xa3c1410, 0xc003db10e0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/backend/v3/instrumentation/logging/connect_handler.go:26 +0x27b\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003db1050}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.RegisterMetrics({0xa3c1410, 0xc003db1050}, {0xa405e70, 0xc002ebf180}, 0xc00321c0d0, {0xcf754f8, 0x3, 0x3}, {0xc0008b8840, 0x4, ...})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/metrics_interceptor.go:47 +0x3f3\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.MetricsHandler.func1.1({0xa3c1410, 0xc003db1050}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/metrics_interceptor.go:31 +0x111\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003db1050}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\nconnectrpc.com/otelconnect.(*Interceptor).WrapUnary.func1({0xa3c1410, 0xc003db1050}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/otelconnect@v0.8.0/interceptor.go:145 +0x16d7\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1410, 0xc003db0ff0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.CallDurationHandler.func1.1({0xa3c1410, 0xc003db0ff0}, {0xa405e70, 0xc002ebf180})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/call_interceptor.go:15 +0xab\nconnectrpc.com/connect.NewUnaryHandler[...].func2({0x7f11083fe700, 0xc000503a40})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:78 +0x4e2\nconnectrpc.com/connect.(*Handler).ServeHTTP(0xc0004dd420, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:333 +0x870\ngithub.com/zitadel/zitadel/pkg/grpc/user/v2/userconnect.NewUserServiceHandler.func1({0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/home/tim/Repositories/zitadel/zitadel/pkg/grpc/user/v2/userconnect/user_service.connect.go:1951 +0x8e5\nnet/http.HandlerFunc.ServeHTTP(0xc002d7a000, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/rs/cors.(*Cors).Handler.func1({0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/home/tim/go/pkg/mod/github.com/rs/cors@v1.11.1/cors.go:289 +0x2fd\nnet/http.HandlerFunc.ServeHTTP(0xc000191aa0, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/zitadel/zitadel/internal/api/http/middleware.RobotsTagHandler.func1({0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/http/middleware/robots_tag_interceptor.go:12 +0x88\nnet/http.HandlerFunc.ServeHTTP(0xc002e14798, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b2c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/zitadel/zitadel/internal/api/http/middleware.WithOrigin.func1.1({0xa3bb420, 0xc0006b22a0}, 0xc002e3b180)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/http/middleware/origin_interceptor.go:27 +0x42a\nnet/http.HandlerFunc.ServeHTTP(0xc003d89a40, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b180)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc002dfa840, {0xa3bb420, 0xc0006b22a0}, 0xc002e3b180)\n\t/home/tim/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x2bb\ngolang.org/x/net/http2.(*serverConn).runHandler(0xc003e143c0, 0xc0006b22a0, 0xc0002f4000, 0xc002d9ae88)\n\t/home/tim/go/pkg/mod/golang.org/x/net@v0.47.0/http2/server.go:2424 +0x1b9\ncreated by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 607\n\t/home/tim/go/pkg/mod/golang.org/x/net@v0.47.0/http2/server.go:2359 +0x1df\n" } } ``` </details> When the GCP error reporting handler is enabled with stack trace, as per [formatting requirements](https://docs.cloud.google.com/error-reporting/docs/formatting-error-messages): - A `@type` field is provided - Some fields are renamed according to Google Cloud Logging specs. (severity, message etc) - The stack strace is appended to the `message` field. - The report location is moved into the top-level `reportLocation` field. The original error is still logged in the `error` field. This is not part of the error reporting API but may still be useful for logging. <details> <summary>GCP error report with stack trace</summary> ```json { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "SpanID": "4fdeae9817ade4f5", "TraceID": "78e31366d4c36e0718a2dcb6ca73d2f3", "domain": { "instance_host": "localhost:8080", "protocol": "http" }, "duration": "82.464896ms", "error": { "id": "COMMAND-bd4ir1mblj", "kind": "NotFound", "message": "User could not be found" }, "http_method": "POST", "logging.googleapis.com/sourceLocation": { "function": "github.com/zitadel/zitadel/internal/api/grpc/gerrors.ZITADELToConnectError", "file": "/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/gerrors/zitadel_errors.go", "line": 57 }, "message": "ID=COMMAND-bd4ir1mblj Message=User could not be found\ngoroutine 1165 [running]:\nruntime/debug.Stack()\n\t/usr/lib/go/src/runtime/debug/stack.go:26 +0x8e\ngithub.com/zitadel/zitadel/internal/zerrors.newZitadelError(0x5, {0x0, 0x0}, {0x6af7719, 0x12}, {0x6aff994, 0x14})\n\t/home/tim/Repositories/zitadel/zitadel/internal/zerrors/zerror.go:139 +0x1df\ngithub.com/zitadel/zitadel/internal/zerrors.ThrowNotFound({0x0, 0x0}, {0x6af7719, 0x12}, {0x6aff994, 0x14})\n\t/home/tim/Repositories/zitadel/zitadel/internal/zerrors/not_found.go:6 +0x70\ngithub.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2(0xc000498588, {0xa3c1450, 0xc002cba540}, {0xc003030910, 0xb}, {0x0, 0x0}, {0xd550a80, 0x0, 0x0}, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go:140 +0x711\ngithub.com/zitadel/zitadel/internal/api/grpc/user/v2.(*Server).DeleteUser(0xc003101500, {0xa3c1450, 0xc002cba540}, 0xc00216ec80)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/user/v2/user.go:191 +0x3f1\nconnectrpc.com/connect.NewUnaryHandler[...].func1({0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:51 +0x17e\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba540}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ActivityInterceptor.func1.1({0xa3c1450, 0xc002cba540}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/activity_interceptor.go:20 +0x1b1\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ServiceHandler.func1.1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/service_interceptor.go:22 +0x283\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.validate({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80}, 0xc003033180)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/validation_interceptor.go:35 +0x1bd\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ValidationHandler.func1.1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/validation_interceptor.go:15 +0x85\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ExecutionHandler.func1.1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/execution_interceptor.go:39 +0x376\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.QuotaExhaustedInterceptor.func1.1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/quota_interceptor.go:25 +0x42d\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.TranslationHandler.func1.1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/translation_interceptor.go:18 +0x94\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc002cba330}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.authorize({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80}, 0xc0030331e0, {0xa3f0ab0, 0xc000203f10}, {{0xc000865980, 0x9, 0x9}}, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/auth_interceptor.go:42 +0x894\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.AuthorizationInterceptor.func1.1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/auth_interceptor.go:17 +0x151\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.LimitsInterceptor.func1.1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/limits_interceptor.go:31 +0x3b7\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.toConnectError({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80}, 0xc003033200)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/error_interceptor.go:21 +0x8a\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.ErrorHandler.func1.1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/error_interceptor.go:15 +0x85\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.AccessStorageInterceptor.func1.1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/access_interceptor.go:21 +0x814\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0015827b0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.addInstanceByRequestedHost({0xa3c1450, 0xc001636630}, {0xa405eb0, 0xc00216ec80}, 0xc003033230, {0xa3a6428, 0xc00069f040}, 0xc002076910, {0xc002cef510, 0x9})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:107 +0xbc3\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.setInstance({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80}, 0xc003033230, {0xa3a6428, 0xc00069f040}, {0xc002cef510, 0x9}, 0xc002076910, ...)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:61 +0xb05\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.InstanceInterceptor.func1.1({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/instance_interceptor.go:23 +0x147\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.NoCacheInterceptor.func1.1({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/cache_interceptor.go:21 +0x351\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/backend/v3/instrumentation/logging.NewConnectInterceptor.func1({0xa3c1450, 0xc0016365a0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/backend/v3/instrumentation/logging/connect_handler.go:26 +0x27b\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0016364e0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.RegisterMetrics({0xa3c1450, 0xc0016364e0}, {0xa405eb0, 0xc00216ec80}, 0xc003033270, {0xcf754f8, 0x3, 0x3}, {0xc000530f60, 0x4, ...})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/metrics_interceptor.go:47 +0x3f3\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.MetricsHandler.func1.1({0xa3c1450, 0xc0016364e0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/metrics_interceptor.go:31 +0x111\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc0016364e0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\nconnectrpc.com/otelconnect.(*Interceptor).WrapUnary.func1({0xa3c1450, 0xc0016364e0}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/otelconnect@v0.8.0/interceptor.go:145 +0x16d7\nconnectrpc.com/connect.unaryThunk.func1({0xa3c1450, 0xc001636450}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/interceptor.go:120 +0xd9\ngithub.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.CallDurationHandler.func1.1({0xa3c1450, 0xc001636450}, {0xa405eb0, 0xc00216ec80})\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/server/connect_middleware/call_interceptor.go:15 +0xab\nconnectrpc.com/connect.NewUnaryHandler[...].func2({0x7fb6100a8e60, 0xc002fa5680})\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:78 +0x4e2\nconnectrpc.com/connect.(*Handler).ServeHTTP(0xc0040768c0, {0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/home/tim/go/pkg/mod/connectrpc.com/connect@v1.19.1/handler.go:333 +0x870\ngithub.com/zitadel/zitadel/pkg/grpc/user/v2/userconnect.NewUserServiceHandler.func1({0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/home/tim/Repositories/zitadel/zitadel/pkg/grpc/user/v2/userconnect/user_service.connect.go:1951 +0x8e5\nnet/http.HandlerFunc.ServeHTTP(0xc00314a200, {0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/rs/cors.(*Cors).Handler.func1({0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/home/tim/go/pkg/mod/github.com/rs/cors@v1.11.1/cors.go:289 +0x2fd\nnet/http.HandlerFunc.ServeHTTP(0xc0022cbdc0, {0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/zitadel/zitadel/internal/api/http/middleware.RobotsTagHandler.func1({0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/http/middleware/robots_tag_interceptor.go:12 +0x88\nnet/http.HandlerFunc.ServeHTTP(0xc001c8fae8, {0xa3bb460, 0xc0025bf360}, 0xc0024c23c0)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/zitadel/zitadel/internal/api/http/middleware.WithOrigin.func1.1({0xa3bb460, 0xc0025bf360}, 0xc0024c2140)\n\t/home/tim/Repositories/zitadel/zitadel/internal/api/http/middleware/origin_interceptor.go:27 +0x42a\nnet/http.HandlerFunc.ServeHTTP(0xc00048e540, {0xa3bb460, 0xc0025bf360}, 0xc0024c2140)\n\t/usr/lib/go/src/net/http/server.go:2322 +0x33\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc00207c540, {0xa3bb460, 0xc0025bf360}, 0xc0024c2140)\n\t/home/tim/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x2bb\ngolang.org/x/net/http2.(*serverConn).runHandler(0xc0025545a0, 0xc0025bf360, 0xc0023fd180, 0xc001703488)\n\t/home/tim/go/pkg/mod/golang.org/x/net@v0.47.0/http2/server.go:2424 +0x1b9\ncreated by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 620\n\t/home/tim/go/pkg/mod/golang.org/x/net@v0.47.0/http2/server.go:2359 +0x1df\n", "path": "/zitadel.user.v2.UserService/DeleteUser", "protocol": "connect", "reportLocation": { "filePath": "/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go", "lineNumber": 140, "functionName": "github.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2" }, "request_id": "d5kgsu8edncaon6puvag", "service": "zitadel.user.v2.UserService", "severity": "WARNING", "time": "2026-01-15T16:57:45.341000592+01:00" } ``` </details> When the GCP error reporting handler is enabled without stack trace, the `message` field contains the error string, also as per formatting requirements. <details> <summary>GCP error report without stack trace</summary> ```json { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "SpanID": "2b4157f874eddbf4", "TraceID": "c34f7404fc0d021e1868ccc687bf2995", "domain": { "instance_host": "localhost:8080", "protocol": "http" }, "duration": "52.061597ms", "error": { "id": "COMMAND-bd4ir1mblj", "kind": "NotFound", "message": "User could not be found" }, "http_method": "POST", "logging.googleapis.com/sourceLocation": { "function": "github.com/zitadel/zitadel/internal/api/grpc/gerrors.ZITADELToConnectError", "file": "/home/tim/Repositories/zitadel/zitadel/internal/api/grpc/gerrors/zitadel_errors.go", "line": 57 }, "message": "ID=COMMAND-bd4ir1mblj Message=User could not be found", "path": "/zitadel.user.v2.UserService/DeleteUser", "protocol": "connect", "reportLocation": { "filePath": "/home/tim/Repositories/zitadel/zitadel/internal/command/user_v2.go", "lineNumber": 140, "functionName": "github.com/zitadel/zitadel/internal/command.(*Commands).RemoveUserV2" }, "request_id": "d5kf5tgedncaiajvls9g", "service": "zitadel.user.v2.UserService", "severity": "WARNING", "time": "2026-01-15T15:00:22.910899273+01:00" } ``` </details> # Additional Changes - Private constructor for zitadel errors so that the stack-depth is consistent for different ways of error creation. (Throw and Create variants) - Request logs are now always at info level. - Bump Go version 1.25, required for `sloggcp` # Additional Context - closes #11329 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Tim Möhlmann
|
eb22b58756 |
feat(telemetry): improved instrumentation for observability (#11159)
# Which Problems Are Solved Zitadel did not provide easy correlation between errors, logs, traces and metrics. The configuration for those instrumentations was also not consistent, with some supporting different exporters then others. Implementation and parsing of config was also spaghettified over multiple packages, with awkward parsing and inconsistent naming of options. # How the Problems Are Solved All telemetry is now merged under the name "instrumentation". Why? 1. We thought it was a good idea in the past to call the milestone exporter `Telemtry` in the runtime config. Calling this `TelemetryV2` looks weird. 2. Not everything is a meter and not everything is sent (tele...). 3. It's also [defined](https://opentelemetry.io/docs/concepts/instrumentation/) as such by the OTEL documentation. ## New features - Adds structured, context based logging with trace-ID awareness - Static log fields are added to the context, such as service and request path - Static log fields are injected in each logline emitted by the application - Structured logs can also be send to an otel exporter - Structured logs can be printed to StdErr in text and JSON format - Error sinks make sure every error is logged at the correct level: - Warnings for client side errors (HTTP 400 range, Invalid request etc) - Error for server side errors (Internal server errors) - Metrics can now also be send to a OTEL collector. (previously they could only be scraped from `/debug/metrics` with prometheus) ## Exporters This change adds all the exporters supported by OTEL upstream and some google specific exporters for our cloud deployment. - StdOut / StdErr: all instrumentations - OTEL gRPC / HTTP: all instrumentations - Google: all instrumentations except logging - Prometheus (pull-based): only metrics The exception is profiling, which only supports the google exporting due to lack of support by OTEL upstream. ## Configuration and structure - All instrumentation is moved into the new `backend/v3/instrumentation` package. It reuses configuration types, so both code and runtime configuration are easier to understand. - The `internal/telemetry` packages are removed. - Instrumentation is started with a single function and a proper shutdown function is now provided. - Legacy configuration is still parsed from the runtime config, as long as the new configuration is disabled. This allows backporting this feature to v4 without breaking existing configurations. # Additional Changes - Devcontainer: set `$PATH` variable so installed go binaries can be run individually, without NX. - NX: install GCI tool to fix imports # Additional Context - Closes https://github.com/zitadel/zitadel/issues/8408 - Closes https://github.com/zitadel/zitadel/issues/6664 - Backport to v4 |
||
 Livio Spring
|
5f3f53dca1 |
fix(api): add tracing interceptor on connectRPC services (#11147)
# Which Problems Are Solved ConnectRPC based services (v2 API) was missing the root span of all requests. Rest calls through grpc-gateway more or less worked properly. This was due to a missing server interceptor for connectRPC services itself. Rest call would work as the trace would start on the gateway already. # How the Problems Are Solved Initiate an connectRPC OTEL interceptor and pass it to all services. # Additional Changes None # Additional Context - noticed internally - requires backport to v4.x |
||
|
Livio Spring
|
6051ce591d |
fix(deps): update saml and all other dependencies (#11067)
Updates saml and all other direct dependencies (which are possible) to the latest version to include fixes and solve requested features: - https://github.com/zitadel/saml/issues/103 - https://github.com/zitadel/saml/issues/104 (requires backport to v.4x at least for saml) |
||
|
Livio Spring
|
61964f92be |
chore: update dependencies (#10923)
Updates all direct dependencies to latests versions (apart from the one where there are already issues to solve their updates) |
||
 Stefan BenzandLivio Spring
|
d5cfbc7b00 |
chore: update crewjam/saml to v0.5.1 (#10652)
# Which Problems Are Solved No usage of the current version of crewjam/saml. # How the Problems Are Solved Update dependency to v0.5.1. # Additional Changes None # Additional Context Closes #9783 --------- Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
|
Tim MöhlmannandMarco A.
|
4440579f0a |
chore(deps): update to latest 1.24 go toolchain (#10667)
# Which Problems Are Solved Update the Go toolchain to the latest bugfix release, so we include latest security fixes in the standard library. # How the Problems Are Solved Set the toolchain directive to 1.24.7 # Additional Changes - go mod tidy # Additional Context - https://go.dev/doc/devel/release#go1.24.0 Co-authored-by: Marco A. <marco@zitadel.com> |
||
|
61cab8878e |
feat(backend): state persisted objects (#9870)
This PR initiates the rework of Zitadel's backend to state-persisted objects. This change is a step towards a more scalable and maintainable architecture. ## Changes * **New `/backend/v3` package**: A new package structure has been introduced to house the reworked backend logic. This includes: * `domain`: Contains the core business logic, commands, and repository interfaces. * `storage`: Implements the repository interfaces for database interactions with new transactional tables. * `telemetry`: Provides logging and tracing capabilities. * **Transactional Tables**: New database tables have been defined for `instances`, `instance_domains`, `organizations`, and `org_domains`. * **Projections**: New projections have been created to populate the new relational tables from the existing event store, ensuring data consistency during the migration. * **Repositories**: New repositories provide an abstraction layer for accessing and manipulating the data in the new tables. * **Setup**: A new setup step for `TransactionalTables` has been added to manage the database migrations for the new tables. This PR lays the foundation for future work to fully transition to state-persisted objects for these components, which will improve performance and simplify data access patterns. This PR initiates the rework of ZITADEL's backend to state-persisted objects. This is a foundational step towards a new architecture that will improve performance and maintainability. The following objects are migrated from event-sourced aggregates to state-persisted objects: * Instances * incl. Domains * Orgs * incl. Domains The structure of the new backend implementation follows the software architecture defined in this [wiki page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel). This PR includes: * The initial implementation of the new transactional repositories for the objects listed above. * Projections to populate the new relational tables from the existing event store. * Adjustments to the build and test process to accommodate the new backend structure. This is a work in progress and further changes will be made to complete the migration. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Iraq Jaber <iraq+github@zitadel.com> Co-authored-by: Iraq <66622793+kkrime@users.noreply.github.com> Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> |
||
|
Marco A.
|
8cc79c1376 |
chore(dep): update zitadel SAML library (#10640)
# Which Problems Are Solved The change partially addresses https://github.com/zitadel/zitadel/issues/5630 , specifically the part about "Order of SAML attributes are non-compliant". # How the Problems Are Solved - Upgrade of the zitadel SAML library # Additional Context - Partially addresses #5630 - Depends on https://github.com/zitadel/saml/pull/106 |
||
|
Tim MöhlmannandStefan Benz
|
a9ebc06c77 |
perf(actionsv2): execution target router (#10564)
# Which Problems Are Solved The event execution system currently uses a projection handler that subscribes to and processes all events for all instances. This creates a high static cost because the system over-fetches event data, handling many events that are not needed by most instances. This inefficiency is also reflected in high "rows returned" metrics in the database. # How the Problems Are Solved Eliminate the use of a project handler. Instead, events for which "execution targets" are defined, are directly pushed to the queue by the eventstore. A Router is populated in the Instance object in the authz middleware. - By joining the execution targets to the instance, no additional queries are needed anymore. - As part of the instance object, execution targets are now cached as well. - Events are queued within the same transaction, giving transactional guarantees on delivery. - Uses the "insert many fast` variant of River. Multiple jobs are queued in a single round-trip to the database. - Fix compatibility with PostgreSQL 15 # Additional Changes - The signing key was stored as plain-text in the river job payload in the DB. This violated our [Secrets Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage) principle. This change removed the field and only uses the encrypted version of the signing key. - Fixed the target ordering from descending to ascending. - Some minor linter warnings on the use of `io.WriteString()`. # Additional Context - Introduced in https://github.com/zitadel/zitadel/pull/9249 - Closes https://github.com/zitadel/zitadel/issues/10553 - Closes https://github.com/zitadel/zitadel/issues/9832 - Closes https://github.com/zitadel/zitadel/issues/10372 - Closes https://github.com/zitadel/zitadel/issues/10492 --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com> |
||
|
Livio SpringandMarco Ardizzone
|
5d2d1d6da6 |
feat(OIDC): handle logout hint on end_session_endpoint (#10039)
# Which Problems Are Solved The OIDC session endpoint allows to pass a `id_token_hint` to identify the session to terminate. In case the application is not able to pass that, e.g. Console currently allows multiple sessions to be open, but will only store the id_token of the current session, allowing to pass the `logout_hint` to identify the user adds some new possibilities. # How the Problems Are Solved In case the end_session_endpoint is called with no `id_token_hint`, but a `logout_hint` and the v2 login UI is configured, the information is passed to the login UI also as `login_hint` parameter to allow the login UI to determine the session to be terminated, resp. let the user decide. # Additional Changes Also added the `ui_locales` as parameter to handle and pass to the V2 login UI. # Dependencies ⚠️ ~These changes depend on https://github.com/zitadel/oidc/pull/774~ # Additional Context closes #9847 --------- Co-authored-by: Marco Ardizzone <marco@zitadel.com> |
||
 Livio SpringandElio Bischof
|
9ebf2316c6 |
feat: exchange gRPC server implementation to connectRPC (#10145)
# Which Problems Are Solved The current maintained gRPC server in combination with a REST (grpc) gateway is getting harder and harder to maintain. Additionally, there have been and still are issues with supporting / displaying `oneOf`s correctly. We therefore decided to exchange the server implementation to connectRPC, which apart from supporting connect as protocol, also also "standard" gRCP clients as well as HTTP/1.1 / rest like clients, e.g. curl directly call the server without any additional gateway. # How the Problems Are Solved - All v2 services are moved to connectRPC implementation. (v1 services are still served as pure grpc servers) - All gRPC server interceptors were migrated / copied to a corresponding connectRPC interceptor. - API.ListGrpcServices and API. ListGrpcMethods were changed to include the connect services and endpoints. - gRPC server reflection was changed to a `StaticReflector` using the `ListGrpcServices` list. - The `grpc.Server` interfaces was split into different combinations to be able to handle the different cases (grpc server and prefixed gateway, connect server with grpc gateway, connect server only, ...) - Docs of services serving connectRPC only with no additional gateway (instance, webkey, project, app, org v2 beta) are changed to expose that - since the plugin is not yet available on buf, we download it using `postinstall` hook of the docs # Additional Changes - WebKey service is added as v2 service (in addition to the current v2beta) # Additional Context closes #9483 --------- Co-authored-by: Elio Bischof <elio@zitadel.com> |
||
|
Tim Möhlmann
|
26ec29a513 |
chore(deps): upgrade oidc and chi for dependabot alert (#10160)
# Which Problems Are Solved Solve dependabot alerts for Go packages. # How the Problems Are Solved - Upgrade to latest github.com/zitadel/oidc, which already pulls the fixed version of chi. - Upgrade mapstructure # Additional Changes - none # Additional Context - https://github.com/zitadel/zitadel/security/dependabot/323 - https://github.com/zitadel/zitadel/security/dependabot/324 |
||
|
Livio Spring
|
f93a35c7a8 |
feat: implement service ping (#10080)
This PR is still WIP and needs changes to at least the tests. # Which Problems Are Solved To be able to report analytical / telemetry data from deployed Zitadel systems back to a central endpoint, we designed a "service ping" functionality. See also https://github.com/zitadel/zitadel/issues/9706. This PR adds the first implementation to allow collection base data as well as report amount of resources such as organizations, users per organization and more. # How the Problems Are Solved - Added a worker to handle the different `ReportType` variations. - Schedule a periodic job to start a `ServicePingReport` - Configuration added to allow customization of what data will be reported - Setup step to generate and store a `systemID` # Additional Changes None # Additional Context relates to #9869 |
||
|
Marco A.
|
28f7218ea1 |
feat: Hosted login translation API (#10011)
# Which Problems Are Solved This PR implements https://github.com/zitadel/zitadel/issues/9850 # How the Problems Are Solved - New protobuf definition - Implementation of retrieval of system translations - Implementation of retrieval and persistence of organization and instance level translations # Additional Context - Closes #9850 # TODO - [x] Integration tests for Get and Set hosted login translation endpoints - [x] DB migration test - [x] Command function tests - [x] Command util functions tests - [x] Query function test - [x] Query util functions tests |
||
 Abhinav SethiandSilvan
|
83839fc2ef |
fix: enable opentelemetry metrics for river queue (#10044)
# Which Problems Are Solved
Right now we have no visibility into river queue's job processing times
and queue sizes. This makes it difficult to reliably know if
notifications are actually being published in a reasonable time and
current queue size.
# How the Problems Are Solved
Integrates River's OpenTelemetry middleware with Zitadel's metrics
system by adding the otelriver middleware to the queue configuration.
# Additional Changes
- Updated dependencies to include required `otelriver` package
# Additional Context
Example output from `/debug/metrics`
<details>
<summary>output</summary>
# HELP failed_deliveries_json_total Failed JSON message deliveries
# TYPE failed_deliveries_json_total counter
failed_deliveries_json_total{otel_scope_name="",otel_scope_version="",triggering_event_type="user.human.phone.code.added"}
2
# HELP go_gc_duration_seconds A summary of the wall-time pause
(stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.8e-05
go_gc_duration_seconds{quantile="0.25"} 6.3916e-05
go_gc_duration_seconds{quantile="0.5"} 7.5584e-05
go_gc_duration_seconds{quantile="0.75"} 9.2584e-05
go_gc_duration_seconds{quantile="1"} 0.000204292
go_gc_duration_seconds_sum 0.003028502
go_gc_duration_seconds_count 34
# HELP go_gc_gogc_percent Heap size target percentage configured by the
user, otherwise 100. This value is set by the GOGC environment variable,
and the runtime/debug.SetGCPercent function. Sourced from
/gc/gogc:percent
# TYPE go_gc_gogc_percent gauge
go_gc_gogc_percent 100
# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the
user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT
environment variable, and the runtime/debug.SetMemoryLimit function.
Sourced from /gc/gomemlimit:bytes
# TYPE go_gc_gomemlimit_bytes gauge
go_gc_gomemlimit_bytes 9.223372036854776e+18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 231
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.24.3"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated in heap and
currently in use. Equals to /memory/classes/heap/objects:bytes.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 7.7565832e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated in
heap until now, even if released already. Equals to
/gc/heap/allocs:bytes.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 7.3319844e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the
profiling bucket hash table. Equals to
/memory/classes/profiling/buckets:bytes.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.63816e+06
# HELP go_memstats_frees_total Total number of heap objects frees.
Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.1496925e+07
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage
collection system metadata. Equals to
/memory/classes/metadata/other:bytes.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.182776e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and
currently in use, same as go_memstats_alloc_bytes. Equals to
/memory/classes/heap/objects:bytes.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 7.7565832e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be
used. Equals to /memory/classes/heap/released:bytes +
/memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.8179584e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in
use. Equals to /memory/classes/heap/objects:bytes +
/memory/classes/heap/unused:bytes
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 8.5868544e+07
# HELP go_memstats_heap_objects Number of currently allocated objects.
Equals to /gc/heap/objects:objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 573723
# HELP go_memstats_heap_released_bytes Number of heap bytes released to
OS. Equals to /memory/classes/heap/released:bytes.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 7.20896e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from
system. Equals to /memory/classes/heap/objects:bytes +
/memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes
+ /memory/classes/heap/free:bytes.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.44048128e+08
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of
last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.749491558214289e+09
# HELP go_memstats_mallocs_total Total number of heap objects allocated,
both live and gc-ed. Semantically a counter version for
go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects +
/gc/heap/tiny/allocs:objects.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.2070648e+07
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache
structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 16912
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache
structures obtained from system. Equals to
/memory/classes/metadata/mcache/inuse:bytes +
/memory/classes/metadata/mcache/free:bytes.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31408
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan
structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 1.3496e+06
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan
structures obtained from system. Equals to
/memory/classes/metadata/mspan/inuse:bytes +
/memory/classes/metadata/mspan/free:bytes.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 2.18688e+06
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage
collection will take place. Equals to /gc/heap/goal:bytes.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.34730994e+08
# HELP go_memstats_other_sys_bytes Number of bytes used for other system
allocations. Equals to /memory/classes/other:bytes.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 3.125168e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes obtained from
system for stack allocator in non-CGO environments. Equals to
/memory/classes/heap/stacks:bytes.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 2.752512e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system
for stack allocator. Equals to /memory/classes/heap/stacks:bytes +
/memory/classes/os-stacks:bytes.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 2.752512e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
Equals to /memory/classes/total:byte.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.58965032e+08
# HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS
setting, or the number of operating system threads that can execute
user-level Go code simultaneously. Sourced from
/sched/gomaxprocs:threads
# TYPE go_sched_gomaxprocs_threads gauge
go_sched_gomaxprocs_threads 14
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 25
# HELP grpc_server_grpc_status_code_total Grpc status code counter
# TYPE grpc_server_grpc_status_code_total counter
grpc_server_grpc_status_code_total{grpc_method="/zitadel.management.v1.ManagementService/ListUserChanges",otel_scope_name="",otel_scope_version="",return_code="200"}
1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.management.v1.ManagementService/ListUserMetadata",otel_scope_name="",otel_scope_version="",return_code="200"}
2
grpc_server_grpc_status_code_total{grpc_method="/zitadel.management.v1.ManagementService/ResendHumanPhoneVerification",otel_scope_name="",otel_scope_version="",return_code="200"}
1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.user.v2.UserService/GetUserByID",otel_scope_name="",otel_scope_version="",return_code="200"}
1
# HELP grpc_server_request_counter_total Grpc request counter
# TYPE grpc_server_request_counter_total counter
grpc_server_request_counter_total{grpc_method="/zitadel.management.v1.ManagementService/ListUserChanges",otel_scope_name="",otel_scope_version=""}
1
grpc_server_request_counter_total{grpc_method="/zitadel.management.v1.ManagementService/ListUserMetadata",otel_scope_name="",otel_scope_version=""}
2
grpc_server_request_counter_total{grpc_method="/zitadel.management.v1.ManagementService/ResendHumanPhoneVerification",otel_scope_name="",otel_scope_version=""}
1
grpc_server_request_counter_total{grpc_method="/zitadel.user.v2.UserService/GetUserByID",otel_scope_name="",otel_scope_version=""}
1
# HELP grpc_server_total_request_counter_total Total grpc request
counter
# TYPE grpc_server_total_request_counter_total counter
grpc_server_total_request_counter_total{otel_scope_name="",otel_scope_version=""}
5
# HELP otel_scope_info Instrumentation Scope metadata
# TYPE otel_scope_info gauge
otel_scope_info{otel_scope_name="",otel_scope_version=""} 1
otel_scope_info{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version=""}
1
# HELP projection_events_processed_total Number of events reduced to
process projection updates
# TYPE projection_events_processed_total counter
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",success="true"}
1
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.instance_features2",success="true"}
0
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.login_names3",success="true"}
0
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.notifications",success="true"}
1
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.orgs1",success="true"}
0
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.user_metadata5",success="true"}
0
projection_events_processed_total{otel_scope_name="",otel_scope_version="",projection="projections.users14",success="true"}
0
# HELP projection_handle_timer_seconds Time taken to process a
projection update
# TYPE projection_handle_timer_seconds histogram
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.005"}
0
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.01"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.05"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.1"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="1"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="5"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="10"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="30"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="60"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="120"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="+Inf"}
1
projection_handle_timer_seconds_sum{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler"}
0.007344541
projection_handle_timer_seconds_count{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.005"}
0
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.01"}
0
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.05"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.1"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="1"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="5"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="10"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="30"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="60"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="120"}
1
projection_handle_timer_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="+Inf"}
1
projection_handle_timer_seconds_sum{otel_scope_name="",otel_scope_version="",projection="projections.notifications"}
0.014258458
projection_handle_timer_seconds_count{otel_scope_name="",otel_scope_version="",projection="projections.notifications"}
1
# HELP projection_state_latency_seconds When finishing processing a
batch of events, this track the age of the last events seen from current
time
# TYPE projection_state_latency_seconds histogram
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.1"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="0.5"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="1"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="5"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="10"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="30"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="60"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="300"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="600"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="1800"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler",le="+Inf"}
1
projection_state_latency_seconds_sum{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler"}
0.012979
projection_state_latency_seconds_count{otel_scope_name="",otel_scope_version="",projection="projections.execution_handler"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.1"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="0.5"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="1"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="5"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="10"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="30"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="60"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="300"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="600"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="1800"}
1
projection_state_latency_seconds_bucket{otel_scope_name="",otel_scope_version="",projection="projections.notifications",le="+Inf"}
1
projection_state_latency_seconds_sum{otel_scope_name="",otel_scope_version="",projection="projections.notifications"}
0.0199
projection_state_latency_seconds_count{otel_scope_name="",otel_scope_version="",projection="projections.notifications"}
1
# HELP promhttp_metric_handler_requests_in_flight Current number of
scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by
HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP river_insert_count_total Number of jobs inserted
# TYPE river_insert_count_total counter
river_insert_count_total{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok"}
1
# HELP river_insert_many_count_total Number of job batches inserted (all
jobs are inserted in a batch, but batches may be one job)
# TYPE river_insert_many_count_total counter
river_insert_many_count_total{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok"}
1
# HELP river_insert_many_duration_histogram_seconds Duration of job
batch insertion (histogram)
# TYPE river_insert_many_duration_histogram_seconds histogram
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="0"}
0
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="5"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="10"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="25"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="50"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="75"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="100"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="250"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="500"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="750"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="1000"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="2500"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="5000"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="7500"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="10000"}
1
river_insert_many_duration_histogram_seconds_bucket{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok",le="+Inf"}
1
river_insert_many_duration_histogram_seconds_sum{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok"}
0.002905666
river_insert_many_duration_histogram_seconds_count{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok"}
1
# HELP river_insert_many_duration_seconds Duration of job batch
insertion
# TYPE river_insert_many_duration_seconds gauge
river_insert_many_duration_seconds{otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",status="ok"}
0.002905666
# HELP river_work_count_total Number of jobs worked
# TYPE river_work_count_total counter
river_work_count_total{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
1
river_work_count_total{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
1
# HELP river_work_duration_histogram_seconds Duration of job being
worked (histogram)
# TYPE river_work_duration_histogram_seconds histogram
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="0"}
0
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="5"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="10"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="25"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="50"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="75"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="100"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="250"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="500"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="750"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="1000"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="2500"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="5000"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="7500"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="10000"}
1
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="+Inf"}
1
river_work_duration_histogram_seconds_sum{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
0.029241083
river_work_duration_histogram_seconds_count{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="0"}
0
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="5"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="10"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="25"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="50"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="75"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="100"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="250"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="500"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="750"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="1000"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="2500"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="5000"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="7500"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="10000"}
1
river_work_duration_histogram_seconds_bucket{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]",le="+Inf"}
1
river_work_duration_histogram_seconds_sum{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
0.0408745
river_work_duration_histogram_seconds_count{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
1
# HELP river_work_duration_seconds Duration of job being worked
# TYPE river_work_duration_seconds gauge
river_work_duration_seconds{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
0.029241083
river_work_duration_seconds{attempt="2",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="error",tag="[]"}
0.0408745
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_name="ZITADEL",service_version="2025-06-09T13:52:29-04:00",telemetry_sdk_language="go",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.35.0"}
1
</details>
Example grafana dashboard:
- Closes #10043
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
|
||
|
|
8fc11a7366 |
feat: user api requests to resource API (#9794)
# Which Problems Are Solved This pull request addresses a significant gap in the user service v2 API, which currently lacks methods for managing machine users. # How the Problems Are Solved This PR adds new API endpoints to the user service v2 to manage machine users including their secret, keys and personal access tokens. Additionally, there's now a CreateUser and UpdateUser endpoints which allow to create either a human or machine user and update them. The existing `CreateHumanUser` endpoint has been deprecated along the corresponding management service endpoints. For details check the additional context section. # Additional Context - Closes https://github.com/zitadel/zitadel/issues/9349 ## More details - API changes: https://github.com/zitadel/zitadel/pull/9680 - Implementation: https://github.com/zitadel/zitadel/pull/9763 - Tests: https://github.com/zitadel/zitadel/pull/9771 ## Follow-ups - Metadata: support managing user metadata using resource API https://github.com/zitadel/zitadel/pull/10005 - Machine token type: support managing the machine token type (migrate to new enum with zero value unspecified?) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
|
SilvanandLivio Spring
|
131f70db34 |
fix(eventstore): use decimal, correct mirror (#9914)
# Eventstore fixes - `event.Position` used float64 before which can lead to [precision loss](https://github.com/golang/go/issues/47300). The type got replaced by [a type without precision loss](https://github.com/jackc/pgx-shopspring-decimal) - the handler reported the wrong error if the current state was updated and therefore took longer to retry failed events. # Mirror fixes - max age of auth requests can be configured to speed up copying data from `auth.auth_requests` table. Auth requests last updated before the set age will be ignored. Default is 1 month - notification projections are skipped because notifications should be sent by the source system. The projections are set to the latest position - ensure that mirror can be executed multiple times --------- Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
|
Livio Spring
|
968d91a3e0 |
chore: update dependencies (#9784)
# Which Problems Are Solved Some dependencies are out of date and published new version including (unaffected) vulnerability fixes. # How the Problems Are Solved - Updated at least all direct dependencies apart from i18n, webauthn (existing issues), - crewjam (https://github.com/zitadel/zitadel/issues/9783) and - github.com/gorilla/csrf (https://github.com/gorilla/csrf/issues/190, https://github.com/gorilla/csrf/issues/189, https://github.com/gorilla/csrf/issues/188, https://github.com/gorilla/csrf/issues/187, https://github.com/gorilla/csrf/issues/186) - noteworthy: https://github.com/golang/go/issues/73626 - Some dependencies require Go 1.24, which triggered an update for zitadel to go 1.24 as well. # Additional Changes None # Additional Context None |
||
 Juriaan Kennedy
|
38013d0e84 |
feat(crypto): support for SHA2 and PHPass password hashes (#9809)
# Which Problems Are Solved - Allow users to use SHA-256 and SHA-512 hashing algorithms. These algorithms are used by Linux's crypt(3) function. - Allow users to import passwords using the PHPass algorithm. This algorithm is used by older PHP systems, WordPress in particular. # How the Problems Are Solved - Upgrade passwap to [v0.9.0](https://github.com/zitadel/passwap/releases/tag/v0.9.0) - Add sha2 and phpass as a new verifier option in defaults.yaml # Additional Changes - Updated docs to explain the two algorithms # Additional Context Implements the changes in the passwap library from https://github.com/zitadel/passwap/pull/59 and https://github.com/zitadel/passwap/pull/60 |
||
+5     
|
07ce3b6905 |
chore!: Introduce ZITADEL v3 (#9645)
This PR summarizes multiple changes specifically only available with ZITADEL v3: - feat: Web Keys management (https://github.com/zitadel/zitadel/pull/9526) - fix(cmd): ensure proper working of mirror (https://github.com/zitadel/zitadel/pull/9509) - feat(Authz): system user support for permission check v2 (https://github.com/zitadel/zitadel/pull/9640) - chore(license): change from Apache to AGPL (https://github.com/zitadel/zitadel/pull/9597) - feat(console): list v2 sessions (https://github.com/zitadel/zitadel/pull/9539) - fix(console): add loginV2 feature flag (https://github.com/zitadel/zitadel/pull/9682) - fix(feature flags): allow reading "own" flags (https://github.com/zitadel/zitadel/pull/9649) - feat(console): add Actions V2 UI (https://github.com/zitadel/zitadel/pull/9591) BREAKING CHANGE - feat(webkey): migrate to v2beta API (https://github.com/zitadel/zitadel/pull/9445) - chore!: remove CockroachDB Support (https://github.com/zitadel/zitadel/pull/9444) - feat(actions): migrate to v2beta API (https://github.com/zitadel/zitadel/pull/9489) --------- Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com> Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com> Co-authored-by: Ramon <mail@conblem.me> Co-authored-by: Elio Bischof <elio@zitadel.com> Co-authored-by: Kenta Yamaguchi <56732734+KEY60228@users.noreply.github.com> Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com> Co-authored-by: Livio Spring <livio@zitadel.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Iraq <66622793+kkrime@users.noreply.github.com> Co-authored-by: Florian Forster <florian@zitadel.com> Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Max Peintner <peintnerm@gmail.com> |
||
|
Stefan Benz
|
0e10ed0e0b |
fix: SAML and OIDC issuer (in proxied use cases) (#9638)
# Which Problems Are Solved When using implicit flow through the session API and a login UI on a custom domain (proxy), the tokens were signed by the API domain of the instance, rather than the public (proxy) domain. The SAML response had the same issue. Additionally, the saml library had an issue and lost the issuer context. This prevented also a successful login through the hosted login UI. # How the Problems Are Solved - The issuer of the SAML and Auth request is persisted to provide the information when signing the responses and tokens. - The SAML library is updated to the latest version. # Additional Changes None # Additional Context None |
||
|
Livio Spring
|
c2c02492b9 |
chore: update dependencies (#9614)
# Which Problems Are Solved Multiple dependencies used by the Zitadel backend are outdated. # How the Problems Are Solved Update all direct dependencies (apart from the following two) to the latest version: - github.com/go-webauthn/webauthn: see https://github.com/zitadel/zitadel/issues/9615 - github.com/nicksnyder/go-i18n/v2 (v2.5.0 / 2.5.1 prevent the use of reserved keys such as `description`: https://github.com/nicksnyder/go-i18n/releases/tag/v2.5.0) # Additional Changes None # Additional Context None |
||
|
Tim MöhlmannandLivio Spring
|
c3258ecf78 |
chore(deps): upgrade go mods from zitadel organization (#9601)
# Which Problems Are Solved Outdated dependencies foir packages developed by Zitadel. Some of them included important security updates from sub-dependencies. # How the Problems Are Solved Upgrade all packages under `github.com/zitadel/*` to the latest released version. # Additional Changes - The `github.com/superseriousbusiness/exifremove` was removed from Github. We copied the cached go mod code to https://github.com/zitadel/exifremove and use this module now. # Additional Context - Related to https://github.com/zitadel/zitadel/issues/9422 - Closes https://github.com/zitadel/zitadel/issues/9443 --------- Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
|
RamonandLivio Spring
|
8b1b9cbb98 |
fix: allow native applications to use https:// on loopback redirect addresses (#9073)
# Which Problems Are Solved - The current validation for native redirect URIs does not allow HTTPS loopback addresses. # How the Problems Are Solved - Enhanced the validation logic to permit HTTPS loopback addresses, ensuring that developers can use these addresses without encountering validation errors. - Updated zitadel/oidc to latest version # Additional Context - Closes #4091 - This pr need to be closed first in our OIDC lib: https://github.com/zitadel/oidc/pull/691 --------- Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
 Miro Trisc
|
e4c12864e5 |
feat(crypto): support md5 salted for imported password hashes (#9596)
# Which Problems Are Solved Allow verification of imported salted passwords hashed with plain md5. # How the Problems Are Solved - Upgrade passwap to [v0.7.0](https://github.com/zitadel/passwap/releases/tag/v0.7.0) - Add md5salted as a new verifier option in `defaults.yaml` # Additional Changes - go version and libraries updated (required by passkey v0.7.0) - secrets.md verifiers updated - configuration verifiers updated - added MD5salted and missing MD5Plain to test cases |
||
|
Stefan BenzandSilvan
|
0c87a96e2c |
feat: actions v2 for functions (#9420)
# Which Problems Are Solved Actions v2 are not executed in different functions, as provided by the actions v1. # How the Problems Are Solved Add functionality to call actions v2 through OIDC and SAML logic to complement tokens and SAMLResponses. # Additional Changes - Corrected testing for retrieved intent information - Added testing for IDP types - Corrected handling of context for issuer in SAML logic # Additional Context - Closes #7247 - Dependent on https://github.com/zitadel/saml/pull/97 - docs for migration are done in separate issue: https://github.com/zitadel/zitadel/issues/9456 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com> |
||
|
Silvan
|
415bc32ed6 |
feat: add task queue (#9321)
# Which Problems Are Solved To integrate river as a task queue we need to ensure the migrations of river are executed. # How the Problems Are Solved - A new schema was added to the Zitadel database called "queue" - Added a repeatable setup step to Zitadel which executes the [migrations of river](https://riverqueue.com/docs/migrations#go-migration-api). # Additional Changes - Added more hooks to the databases to properly set the schema for the task queue # Additional Context - Closes https://github.com/zitadel/zitadel/issues/9280 |
||
 Lars
|
1915d35605 |
feat: list users scim v2 endpoint (#9187)
# Which Problems Are Solved
- Adds support for the list users SCIM v2 endpoint
# How the Problems Are Solved
- Adds support for the list users SCIM v2 endpoints under `GET
/scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search`
# Additional Changes
- adds a new function `SearchUserMetadataForUsers` to the query layer to
query a metadata keyset for given user ids
- adds a new function `NewUserMetadataExistsQuery` to the query layer to
query a given metadata key value pair exists
- adds a new function `CountUsers` to the query layer to count users
without reading any rows
- handle `ErrorAlreadyExists` as scim errors `uniqueness`
- adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison
methods
- adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query
comparison methods
# Additional Context
Part of #8140
Supported fields for scim filters:
* `meta.created`
* `meta.lastModified`
* `id`
* `username`
* `name.familyName`
* `name.givenName`
* `emails` and `emails.value`
* `active` only eq and ne
* `externalId` only eq and ne
|
||
|
Tim Möhlmann
|
94cbf97534 |
fix(permissions_v2): add membership fields migration (#9199)
# Which Problems Are Solved Memberships did not have a fields table fill migration. # How the Problems Are Solved Add filling of membership fields to the repeatable steps. # Additional Changes - Use the same repeatable step for multiple fill fields handlers. - Fix an error for PostgreSQL 15 where a subquery in a `FROM` clause needs an alias ing the `permitted_orgs` function. # Additional Context - Part of https://github.com/zitadel/zitadel/issues/9188 - Introduced in https://github.com/zitadel/zitadel/pull/9152 |
||
|
Livio Spring
|
d9a89d9fcf |
chore: update dependencies (#9087)
Updates most urgent dependencies |
||
|
Stefan BenzandLivio Spring
|
c3b97a91a2 |
feat: add saml request to link to sessions (#9001)
# Which Problems Are Solved It is currently not possible to use SAML with the Session API. # How the Problems Are Solved Add SAML service, to get and resolve SAML requests. Add SAML session and SAML request aggregate, which can be linked to the Session to get back a SAMLResponse from the API directly. # Additional Changes Update of dependency zitadel/saml to provide all functionality for handling of SAML requests and responses. # Additional Context Closes #6053 --------- Co-authored-by: Livio Spring <livio.a@gmail.com> |
||
 Silvan
|
77cd430b3a |
refactor(handler): cache active instances (#9008)
# Which Problems Are Solved Scheduled handlers use `eventstore.InstanceIDs` to get the all active instances within a given timeframe. This function scrapes through all events written within that time frame which can cause heavy load on the database. # How the Problems Are Solved A new query cache `activeInstances` is introduced which caches the ids of all instances queried by id or host within the configured timeframe. # Additional Changes - Changed `default.yaml` - Removed `HandleActiveInstances` from custom handler configs - Added `MaxActiveInstances` to define the maximal amount of cached instance ids - fixed start-from-init and start-from-setup to start auth and admin projections twice - fixed org cache invalidation to use correct index # Additional Context - part of #8999 |
||
|
Tim Möhlmann
|
3b7b0c69e6 |
feat(cache): redis circuit breaker (#8890)
# Which Problems Are Solved If a redis cache has connection issues or any other type of permament error, it tanks the responsiveness of ZITADEL. We currently do not support things like Redis cluster or sentinel. So adding a simple redis cache improves performance but introduces a single point of failure. # How the Problems Are Solved Implement a [circuit breaker](https://learn.microsoft.com/en-us/previous-versions/msp-n-p/dn589784(v=pandp.10)?redirectedfrom=MSDN) as [`redis.Limiter`](https://pkg.go.dev/github.com/redis/go-redis/v9#Limiter) by wrapping sony's [gobreaker](https://github.com/sony/gobreaker) package. This package is picked as it seems well maintained and we already use their `sonyflake` package # Additional Changes - The unit tests constructed an unused `redis.Client` and didn't cleanup the connector. This is now fixed. # Additional Context Closes #8864 |
||
|
Tim Möhlmann
|
250f2344c8 |
feat(cache): redis cache (#8822)
# Which Problems Are Solved Add a cache implementation using Redis single mode. This does not add support for Redis Cluster or sentinel. # How the Problems Are Solved Added the `internal/cache/redis` package. All operations occur atomically, including setting of secondary indexes, using LUA scripts where needed. The [`miniredis`](https://github.com/alicebob/miniredis) package is used to run unit tests. # Additional Changes - Move connector code to `internal/cache/connector/...` and remove duplicate code from `query` and `command` packages. - Fix a missed invalidation on the restrictions projection # Additional Context Closes #8130 |
||
 Livio SpringandHidde Wieringa
|
041af26917 |
feat(OIDC): add back channel logout (#8837)
# Which Problems Are Solved Currently ZITADEL supports RP-initiated logout for clients. Back-channel logout ensures that user sessions are terminated across all connected applications, even if the user closes their browser or loses connectivity providing a more secure alternative for certain use cases. # How the Problems Are Solved If the feature is activated and the client used for the authentication has a back_channel_logout_uri configured, a `session_logout.back_channel` will be registered. Once a user terminates their session, a (notification) handler will send a SET (form POST) to the registered uri containing a logout_token (with the user's ID and session ID). - A new feature "back_channel_logout" is added on system and instance level - A `back_channel_logout_uri` can be managed on OIDC applications - Added a `session_logout` aggregate to register and inform about sent `back_channel` notifications - Added a `SecurityEventToken` channel and `Form`message type in the notification handlers - Added `TriggeredAtOrigin` fields to `HumanSignedOut` and `TerminateSession` events for notification handling - Exported various functions and types in the `oidc` package to be able to reuse for token signing in the back_channel notifier. - To prevent that current existing session termination events will be handled, a setup step is added to set the `current_states` for the `projections.notifications_back_channel_logout` to the current position - [x] requires https://github.com/zitadel/oidc/pull/671 # Additional Changes - Updated all OTEL dependencies to v1.29.0, since OIDC already updated some of them to that version. - Single Session Termination feature is correctly checked (fixed feature mapping) # Additional Context - closes https://github.com/zitadel/zitadel/issues/8467 - TODO: - Documentation - UI to be done: https://github.com/zitadel/zitadel/issues/8469 --------- Co-authored-by: Hidde Wieringa <hidde@hiddewieringa.nl> |
||
|
Tim MöhlmannandSilvan
|
25dc7bfe72 |
perf(cache): pgx pool connector (#8703)
# Which Problems Are Solved Cache implementation using a PGX connection pool. # How the Problems Are Solved Defines a new schema `cache` in the zitadel database. A table for string keys and a table for objects is defined. For postgreSQL, tables are unlogged and partitioned by cache name for performance. Cockroach does not have unlogged tables and partitioning is an enterprise feature that uses alternative syntax combined with sharding. Regular tables are used here. # Additional Changes - `postgres.Config` can return a pxg pool. See following discussion # Additional Context - Part of https://github.com/zitadel/zitadel/issues/8648 - Closes https://github.com/zitadel/zitadel/issues/8647 --------- Co-authored-by: Silvan <silvan.reusser@gmail.com> |
||
|
Livio Spring
|
bee0744d46 |
fix: update logging to allow slog field overwrite (#8720)
# Which Problems Are Solved When using slog (e.g. in OIDC) the logs field name can not be overwritten. This is necessary for example to change log level to severity. # How the Problems Are Solved - Update logging library # Additional Changes None # Additional Context None |
||
 
|
14e2aba1bc |
feat: Add Twilio Verification Service (#8678)
# Which Problems Are Solved Twilio supports a robust, multi-channel verification service that notably supports multi-region SMS sender numbers required for our use case. Currently, Zitadel does much of the work of the Twilio Verify (eg. localization, code generation, messaging) but doesn't support the pool of sender numbers that Twilio Verify does. # How the Problems Are Solved To support this API, we need to be able to store the Twilio Service ID and send that in a verification request where appropriate: phone number verification and SMS 2FA code paths. This PR does the following: - Adds the ability to use Twilio Verify of standard messaging through Twilio - Adds support for international numbers and more reliable verification messages sent from multiple numbers - Adds a new Twilio configuration option to support Twilio Verify in the admin console - Sends verification SMS messages through Twilio Verify - Implements Twilio Verification Checks for codes generated through the same # Additional Changes # Additional Context - base was implemented by @zhirschtritt in https://github.com/zitadel/zitadel/pull/8268 ❤️ - closes https://github.com/zitadel/zitadel/issues/8581 --------- Co-authored-by: Zachary Hirschtritt <zachary.hirschtritt@klaviyo.com> Co-authored-by: Joey Biscoglia <joey.biscoglia@klaviyo.com> |
||
|
Tim Möhlmann
|
aeb379e7de | fix(eventstore): revert precise decimal (#8527) (#8679) |
