# Which Problems Are Solved
The username input to login with LDAP was taken as is without escaping it, which could potentially allow manipulating the filter used to check if the user exists (LDAP injection).
# How the Problems Are Solved
Escape the username when used in the filter.
# Additional Changes
Removed unnecessary parsing of the returned user DN.
# Additional Context
None
(cherry picked from commit 08007da70e)
(cherry picked from commit a56d4bf0bc)
# Which Problems Are Solved
- Organization scopes not being enforced in device authorization flow
- No check whether orgs exist in an authorization request
# How the Problems Are Solved
- Logic to get ID/Domain of an organisation is moved to the `Request` interface.
- Added assertion for the org ID/Domain when creating the authorization request.
- Added checks when authorizing the auth / device request with a session to check if a specific organization is enforced.
- Added checks to prevent using a session of a user from another organization if enforces by scope
# Additional Changes
# Additional Context
- Related Issue: #11160
---------
Co-authored-by: Livio Spring <livio@zitadel.com>
Co-authored-by: Max Peintner <peintnerm@gmail.com>
(cherry picked from commit c3a6bdb0bb)
(cherry picked from commit d90285929c)
# Which Problems Are Solved
When an expired invite code was used for webauthn (passkeys), the change date got updated by the failed event. This change date was used to test for expiry, meaning failed events would reset the expiry timeout.
# How the Problems Are Solved
This fix adds a Code Creation Date to the writemodel which gets set by the first event. This can be the added or requested event.
# Additional Changes
- Expiry renamed to CodeExpiry so it's consistent with similar write models using secret codes.
- humanVerifyPasswordlessInitCode takes an algorithm instead of the complete generator, so the method can be unit tested easier
- Added tests that reproduced the original issue
(cherry picked from commit 64ee480f84)
# Which Problems Are Solved
URL encoded paths were not correctly checked in the auth handler and computing the necessary path prefix would break.
# How the Problems Are Solved
- Pass the path prefix instead of computing it and fully rely on the path template in case the exact path was not found in the check.
# Additional Changes
None
# Additional Context
- requires backport to v4.x and v3.x
(cherry picked from commit 4a8a5e2fef)
# Which Problems Are Solved
Some endpoints in the management API verified the user's permission but did not honor the organization the permission was granted in the internal process. This led to cross-tenant information disclosure.
The following endpoints were affected:
* `GetProjectByID`
* `GetGrantedProjectByID`
* `GetProjectGrantByID`
* `GetAppByID`
* `ListApps`
* `ListHumanAuthFactors`
* `ListHumanPasswordless`
# How the Problems Are Solved
Correctly treat the requested context (organization) where the permission was granted and use that to query the resource.
# Additional Changes
None
# Additional Context
- requires backport to v4.x and v3.x
(cherry picked from commit ab51fd9563)
(cherry picked from commit da227a458a)
# Conflicts:
# internal/query/app.go
# Which Problems Are Solved
User are allowed to change their own data including email and phone. However, they were unintentionally able to also set them to verified without according permission.
# How the Problems Are Solved
Fix self-management of email and phone and disallow setting them to verified without according permission.
# Additional Changes
None
# Additional Context
- requires backport to v4.x and v3.x
(cherry picked from commit 0261536243)
(cherry picked from commit 288f064e3c)
# Which Problems Are Solved
Opaque OIDC access tokens in v2 format, truncated to 80 characters are still considered valid.
Zitadel uses a symmetric AES encryption for opaque tokens. The cleartext payload is a concatenation of a couple of identifiers, such as a token ID and user ID. Internally Zitadel has 2 different versions of token payloads. v1 tokens are no longer created, but are still verified as to not invalidate existing session after upgrade.
The cleartext payload has a format of `<token_id>:<user_id>`. v2 tokens distinguished further where the `token_id` is of the format `v2_<oidc_session_id>-at_<access_token_id>`. This is an example of such a cleartext: `V2_354201447279099906-at_354201447279165442:354201364702363650`
## Impact
V1 token authZ/N session data is retrieved from the database using the (simple) `token_id` value and `user_id` value. The `user_id` (called `subject` in some parts of our code) was used as being the trusted user iD.
V2 token authZ/N session data is retrieved from the database using the `oidc_session_id` and `access_token_id` and in this case the `user_id` from the token is ignored and taken from the session data in the DB.
By truncating the token to 80 chars, the user_id is now missing from the cleartext of the v2 token: `V2_354201447279099906-at_354201447279165442:` The back-end still accepts this for above reasons.
This issue is not considered exploitable, but may look awkward when reproduced.
# How the Problems Are Solved
Compare the token subject with the stored user ID. If they don't match, consider the token invalid, as some type of tampering must have happened.
(cherry picked from commit feab8e1fa3)
(cherry picked from commit d2a6af8e07)
# Which Problems Are Solved
Similar to https://github.com/zitadel/zitadel/pull/11241, there are
additional unit tests where certificates expired.
# How the Problems Are Solved
The tests use dedicated combinations of SAML metadata, certificates and
responses and also already use the `TimeNow` function from the
`crewjam/saml` package to overwrite necessary time validations. This PR
now additionally sets the `Clock` variable in the same package, which is
used in XML validations like the signing.
# Additional Changes
None
# Additional Context
- relates to #11241
- requires backport to at least v4.x
(cherry picked from commit 565ac73475)
# Which Problems Are Solved
Some SAML integration tests used a hardcoded certificate, which expired
on the 25th December 2025.
# How the Problems Are Solved
Added helper functions to the integration package to create and store a
private key and certificate for SAML, which can be used in SAML SP/IDP
integration tests.
# Additional Changes
none
# Additional Context
- noted on #11196
- requires backport to at least v4.x
(cherry picked from commit 72c8f0192c)
# Which Problems Are Solved
Using public endpoints in the login UI which accept a userID as a query parameter like the password reset endpoint, it was possible to check if a userID exists, since the response would be different depending if the id exists or not.
# How the Problems Are Solved
- Return generic error if either user does not exists or codes are wrong
- Use the default organization's policy instead of the failing when the user's policy can't be loaded
# Additional Changes
None
# Additional Context
requires backport to v4.x and v3.x
(cherry picked from commit 9039540395)
(cherry picked from commit b85ab69e46)
The change was introduced in
https://github.com/zitadel/zitadel/pull/11178.
The fix is to prevent wiping the memberships because the projection did
not init yet.
### Changes
- Introduces a check to determine if the member role synchronization
should be executed based on the existence of a specific database table
(`projections.instance_members4`).
- Ensures that the synchronization process only runs if the required
table is present in the database.
(cherry picked from commit 1b54a9eb05)
# Which Problems Are Solved
Zitadel v4.7.2 fixed a security issue by switching to the permission v2
framework for user APIs. It appears that systems that are running since
before v2.68 that were affected by a precision bug in the eventstore,
which was fixed in that version. The precision bug results in certain
events being "skipped" while being projected into the fields table, used
by the new permission system. This caused certain membership roles to be
missing, resulting in empty user lists when executed by the affected
member. The permission system basically finds no matching memberships
and therefore returns no users at all.
# How the Problems Are Solved
After research we concluded that the legacy membership projections are
projected correctly. This PR synchronizes the projected state into the
fields table. As the membership roles are not marked unique, all rows
are first deleted and then the correct membership roles are then
inserted. The operation happens in a single transaction, during which
the fields table will remain locked for modifications. This to prevent
possible concurrent modifications to membership states.
# Additional Changes
- none
# Additional Context
- Introduced in
https://github.com/zitadel/zitadel/commit/0e17d0005a98ccbf92139961ef702ef03208ffd3
- Released in
[v4.7.2](https://github.com/zitadel/zitadel/releases/tag/v4.7.2)
- Related: https://github.com/zitadel/zitadel/issues/8863
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 58612a6ef7)
# Which Problems Are Solved
When the feature flag for enabling permission checks v2 is disabled, a user without permission could list users across instances and get the total number of users available.
# How the Problems Are Solved
Disregard the state of the feature flag and always enforce permission checks v2 on v2 APIs.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 826039c620)
(cherry picked from commit 0e17d0005a)
# Which Problems Are Solved
When auto-linking was enabled on an IdP, there was no check if linking to the found user is allowed, i.e. if the corresponding IdP is active in the user's organization or if external authentication in general was allowed.
# How the Problems Are Solved
- (Re)Check the login policy of the user's organization before linking the external identity.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Max Peintner <peintnerm@gmail.com>
(cherry picked from commit 33c51deb20)
# Which Problems Are Solved
The current cache interface implementation for postgres is not
compatible with Postgres18, since we rely on partitioned unlogged
tables, which are no longer supported.
# How the Problems Are Solved
Use postgres 17 and update compatibility in the docs.
# Additional Changes
None
# Additional Context
- requires backport to v3.x, v4.x
(cherry picked from commit f7fbd0cdfd)
# Which Problems Are Solved
Zitadel enforces MFA if required by the organization's policy, but did not ensure in all cases, if a user voluntarily set it up.
# How the Problems Are Solved
Ensure 2FA/MFA is required in any call to Zitadel if set up by user even if policy does not require.
# Additional Changes
None
# Additional Context
- requires backports
(cherry picked from commit b284f8474e)
(cherry picked from commit f7309f8295)
# Which Problems Are Solved
While the lockout policy was correctly applied on the session API and other authentication and management endpoints , it had no effect on the user service v2 endpoints.
# How the Problems Are Solved
- Correctly apply lockout policy on the user service v2 endpoints.
- Added tar pitting to auth factor checks (authentication and management API) to prevent brute-force attacks or denial of service because of user lockouts.
- Tar pitting is not active if `IgnoreUnknownUsername` option is active to prevent leaking information whether a user exists or not.
# Additional Changes
None
# Additional Context
- requires backports
* cleanup
(cherry picked from commit b8db8cdf9c)
(cherry picked from commit d3713dfaed)
# Which Problems Are Solved
Host headers used to identify the instance and further used in public responses such as OIDC discovery endpoints, email links and more were not correctly handled. While they were matched against existing instances, they were not properly sanitized.
# How the Problems Are Solved
Sanitize host header including port validation (if provided).
# Additional Changes
None
# Additional Context
- requires backports
(cherry picked from commit 72a5c33e6a)
(cherry picked from commit 7520450e11)
# Which Problems Are Solved
The TestServer_Limits_AuditLogRetention is too reliant on time
constraints when checking that a limit is correctly applied. IN case it
takes to long to do all the preparation, there won't be any events to
read and the test will fail.
# How the Problems Are Solved
Don't require any events to be returned.
# Additional Changes
None
# Additional Context
- Noted a lot of pipeline to fail on this step.
- requires backport to at least v4.x
(cherry picked from commit 8574d6fbab)
# Which Problems Are Solved
I noticed that a failure in the projections handlers `reduce` function
(e.g. creating the statement or checking preconditions for the
statement) would not update the `failed_events2` table.
This was due to a wrong error handling, where as long as the
`maxFailureCount` was not reached, the error was returned after updating
the `failed_events2` table, which causes the transaction to be rolled
back and thus losing the update.
# How the Problems Are Solved
Wrap the error into an `executionError`, so the transaction is not
rolled back.
# Additional Changes
none
# Additional Context
- noticed internally
- requires backport to v3.x and v4.x
(cherry picked from commit ee92560f32)
This PR fixes a bug where projections could skip events if they were
written within the same microsecond, which can occur during high load on
different transactions.
## Problem
The event query ordering was not fully deterministic. Events created at
the exact same time (same `position`) and in the same transaction
(`in_tx_order`) were not guaranteed to be returned in the same order on
subsequent queries. This could lead to some events being skipped by the
projection logic.
## Solution
To solve this, the `ORDER BY` clause for event queries has been extended
to include `instance_id`, `aggregate_type`, and `aggregate_id`. This
ensures a stable and deterministic ordering for all events, even if they
share the same timestamp.
## Additionally changes:
* Replaced a manual slice search with the more idiomatic
`slices.Contains` to skip already projected instances.
* Changed the handling of already locked projections to log a debug
message and skip execution instead of returning an error.
* Ensures the database transaction is explicitly committed.
(cherry picked from commit 25ab6b2397)
A timing issue (a race condition) was identified in our event processing
system. Under specific circumstances, it was possible for the system to
skip processing certain events, leading to potential data
inconsistencies.
## Which problems are solved
The system tracks its progress through the event log using timestamps.
The issue occurred because we were using the timestamp from the start of
a database transaction. If a query to read new events began after the
transaction started but before the new event was committed, the query
would not see the new event and would fail to process it.
## How the problems are solved
The fix is to change which timestamp is used for tracking. We now use
the precise timestamp of when the event is actually written to the
database. This ensures that the event's timestamp is always correctly
ordered, closing the timing gap and preventing the race condition.
This change enhances the reliability and integrity of our event
processing pipeline. It guarantees that all events are processed in the
correct order and eliminates the risk of skipped events, ensuring data
is always consistent across the system.
## Additional information
original fix: https://github.com/zitadel/zitadel/pull/10560
(cherry picked from commit 136363deda)
This PR overhauls our event projection system to make it more robust and
prevent skipped events under high load. The core change replaces our
custom, transaction-based locking with standard PostgreSQL advisory
locks. We also introduce a worker pool to manage concurrency and prevent
database connection exhaustion.
### Key Changes
* **Advisory Locks for Projections:** Replaces exclusive row locks and
inspection of `pg_stat_activity` with PostgreSQL advisory locks for
managing projection state. This is a more reliable and standard approach
to distributed locking.
* **Simplified Await Logic:** Removes the complex logic for awaiting
open transactions, simplifying it to a more straightforward time-based
filtering of events.
* **Projection Worker Pool:** Implements a worker pool to limit
concurrent projection triggers, preventing connection exhaustion and
improving stability under load. A new `MaxParallelTriggers`
configuration option is introduced.
### Problem Solved
Under high throughput, a race condition could cause projections to miss
events from the eventstore. This led to inconsistent data in projection
tables (e.g., a user grant might be missing). This PR fixes the
underlying locking and concurrency issues to ensure all events are
processed reliably.
### How it Works
1. **Event Writing:** When writing events, a *shared* advisory lock is
taken. This signals that a write is in progress.
2. **Event Handling (Projections):**
* A projection worker attempts to acquire an *exclusive* advisory lock
for that specific projection. If the lock is already held, it means
another worker is on the job, so the current one backs off.
* Once the lock is acquired, the worker briefly acquires and releases
the same *shared* lock used by event writers. This acts as a barrier,
ensuring it waits for any in-flight writes to complete.
* Finally, it processes all events that occurred before its transaction
began.
### Additional Information
* ZITADEL no longer modifies the `application_name` PostgreSQL variable
during event writes.
* The lock on the `current_states` table is now `FOR NO KEY UPDATE`.
* Fixes https://github.com/zitadel/zitadel/issues/8509
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit 0575f67e94)
# Which Problems Are Solved
It should not be possible to start 2 projections with the same name.
If this happens, it can cause issues with the event store such as events
being skipped/unprocessed and can be very hard/time-consuming to
diagnose.
# How the Problems Are Solved
A check was added to make sure no 2 projections have the same table
Closes https://github.com/zitadel/zitadel/issues/10453
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 10bd747105)
# Which Problems Are Solved
There was an left-behind index introduced to optimize the old and
removed event execution handler. The index confuses prostgres and it
sometimes picks this index in favor of the projection specific index.
This sometimes leads to bad query performance in the projectio handlers.
# How the Problems Are Solved
Drop the index
# Additional Changes
- none
# Additional Context
- Forgotten in https://github.com/zitadel/zitadel/pull/10564
(cherry picked from commit 54554b8fb9)
# Which Problems Are Solved
Invalid id_tokens used as `id_token_hint` on the authorization endpoints
currently return an error, resp. get display on the endpoint itself.
# How the Problems Are Solved
Ignore invalid id_token_hint errors and just log them.
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10673
- backport to v4.x
(cherry picked from commit e158f9447e)
# Which Problems Are Solved
It was noticed that on actions v2 when subscribing to events, the
webhook would always receive an empty `event_payload`:
```
{
"aggregateID": "336494809936035843",
"aggregateType": "user",
"resourceOwner": "336392597046099971",
"instanceID": "336392597046034435",
"version": "v2",
"sequence": 1,
"event_type": "user.human.added",
"created_at": "2025-09-05T08:55:36.156333Z",
"userID": "336392597046755331",
"event_payload":
{}
}
```
The problem was due to using `json.Marshal` on the `Event` interface,
where the underlying `BaseEvent` prevents the data to be marshalled:
https://github.com/zitadel/zitadel/blob/131f70db3423b80da1b038a822da59c908e4ffa6/internal/eventstore/event_base.go#L38
# How the Problems Are Solved
The `Event`s `Unmarshal` function is used with a `json.RawMessage`.
# Additional Changes
none
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10650
- requires backport to v4.x
(cherry picked from commit 79809d0199)
# Which Problems Are Solved
The
[otelriver](https://github.com/riverqueue/rivercontrib/tree/master/otelriver)
package uses default otel histogram buckets that are designed for
millisecond measurements. OTEL docs also suggest standardizing on using
seconds as the measurement unit. However, the default buckets from
opentelemetry-go are more or less useless when used with seconds as the
smallest measurement is 5 seconds and the largest is nearly 3 hours.
Example:
```
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="0"} 0
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="5"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="10"} 1144
<...more buckets here...>
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="7500"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="10000"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="+Inf"} 1144
```
# How the Problems Are Solved
Change the default unit to "ms" from "s" as supported by the middleware
API:
https://riverqueue.com/docs/open-telemetry#list-of-middleware-options
# Additional Changes
None
# Additional Context
None
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit fcdc598320)
# Which Problems Are Solved
I noticed some outdated / misleading logs when starting zitadel:
- The `init-projections` were no longer in beta for a long time.
- The LRU auth request cache is disabled by default, which results in
the following message, which has caused confusion by customers:
```level=info msg="auth request cache disabled" error="must provide a positive size"```
# How the Problems Are Solved
- Removed the beta info
- Disable cache initialization if possible
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
(cherry picked from commit a1ad87387d)
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit a9ebc06c77)
Removes manual schema and application name setup via raw SQL and
switches to using River’s built-in schema configuration.
# Which Problems Are Solved
River provides a configuration flag to set the schema of the queue.
Zitadel sets the schema through database statements which is not needed
anymore.
# How the Problems Are Solved
Set the schema in the river configuration and removed old code
(cherry picked from commit b5f97d64b0)
# Which Problems Are Solved
This PR fixes an issue where all features where patched, instead of a
single one. This led to instance overrides which were not intended.
With this change, an update is executed whenever a toggle is hit, only
containing the respective feature, not all.
# How the Problems Are Solved
The console application was overriding the feature settings as an entire
request. A toggle change is now only changing the desired and targeted
feature using partial patches.
# Additional Context
Closes#10459
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
(cherry picked from commit d8518d48f2)
# Conflicts:
# console/src/app/components/features/features.component.ts
# e2e/package.json
# Which Problems Are Solved
When the webkey feature flag was not enabled before an upgrade to v4,
all JWT tokens became invalid.
This created a couple of issues:
- All users with JWT access tokens are logged-out
- Clients that are unable to refresh keys based on key ID break
- id_token_hint could no longer be validated.
# How the Problems Are Solved
Force-enable the webkey feature on the v3 version, so that the upgrade
path is cleaner. Sessions now have time to role-over to the new keys
before initiating the upgrade to v4.
# Additional Changes
- none
# Additional Context
- Related https://github.com/zitadel/zitadel/issues/10673
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Metadata encoding and decoding on the organization detail page was
broken due to use of the old, generated gRPC client.
# How the Problems Are Solved
The metadata values are now correctly base64 decoded and encoded on the
organization detail page.
# Additional Changes
Refactored parts of the code to remove the dependency on the buffer npm
package, replacing it with the browser-native TextEncoder and
TextDecoder APIs.
# Additional Context
- This is a backport of #10024
# Which Problems Are Solved
We are preparing to roll-out and stabilize webkeys in the next version
of Zitadel. Before removing legacy signing-key code, we must ensure all
existing instances have their webkeys generated.
# How the Problems Are Solved
Add a setup step which generate 2 webkeys for each existing instance
that didn't have webkeys yet.
# Additional Changes
Return an error from the config type-switch, when the type is unknown.
# Additional Context
- Part 1/2 of https://github.com/zitadel/zitadel/issues/10029
- Should be back-ported to v3
(cherry picked from commit fa9de9a0f1)
# Which Problems Are Solved
User enumeration was possible on the select account page by passing any userID as part of the form POST. Existing users could be selected even if they never authenticated on the same user agent (browser).
# How the Problems Are Solved
A check for an existing session on the same user agent was added to the select user function, resp. only required for the account selection page, since in other cases there doesn't have to be an existing session and the user agent integrity is already checked.
# Additional Changes
None
# Additional Context
None
(cherry picked from commit 7abe759c95)
# Which Problems Are Solved
While investigating a support ticket, it was discovered that some
queries using equals or not equals without case matching were not
correctly escaping the value to compare. If a value contained a
backslash (`\`) the row would not match.
# How the Problems Are Solved
- Fixed the escaping for backslash for `like` operations.
- Changed equals and not equals comparison without case matching to `=`
instead of `like`.
# Additional Changes
None
# Additional Context
- related to a support request
- requires backport to v.3 and v4.x
(cherry picked from commit 6c8d027e72)
The `/introspect` endpoint showed poor performance during v4 load
testing due to an inefficient database query in
`internal/query/introspection_client_by_id.sql`. This PR optimizes the
query structure to significantly improve performance.
## Query Optimizations
**UNION → UNION ALL**: Changed expensive `UNION` to `UNION ALL` since
`client_id` is unique across both API and OIDC config tables,
eliminating unnecessary deduplication overhead (30-50% improvement
expected).
**Simplified Keys CTE**: Optimized the keys lookup logic by using
`$2::text as client_id` instead of `identifier as client_id` with `group
by`, and added explicit `$3 = true` condition to the LEFT JOIN for
better query planning.
**Enhanced Readability**: Added consistent table aliases (c, a, p, o, k)
for better maintainability.
## Benefits
- **Zero-downtime deployment**: Uses existing database indexes, no
schema changes required
- **Secondary performance gains**: Other similar queries
(`oidc_client_by_id.sql`, `userinfo_client_by_id.sql`) will also benefit
from the optimizations
- **Minimal code changes**: Only 13 lines added, 9 lines removed in the
SQL query
- **Backward compatible**: Same result set and API behavior
The optimized query maintains the same functionality while providing
significant performance improvements for the introspection endpoint
under high concurrent load.
Fixes#10389.
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 Share your feedback on Copilot coding agent for the chance to win a
$200 gift card! Click
[here](https://survey.alchemer.com/s3/8343779/Copilot-Coding-agent) to
start the survey.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit a28950661c)
# Which Problems Are Solved
When searching for an existing external userID from an IdP response, the
comparison is case sensitive. This can lead to issues esp. when using
SAML, since the `NameID`'s value case could change. The existing user
would not be found and the login would try to create a new one, but fail
since the uniqueness check of IdP ID and external userID is not case
insensitive.
# How the Problems Are Solved
Search case insensitive for external useriDs.
# Additional Changes
None
# Additional Context
- closes#10457, #10387
- backport to v3.x
(cherry picked from commit 4630b53313)
# Which Problems Are Solved
Currently, the prometheus endpoint metrics contain otel specific labels
that increase the overall metric size to the point that the exemplar
implementation in the underlying prom exporter library throws an error,
see https://github.com/zitadel/zitadel/issues/10047. The MaxRuneSize for
metric refs in exemplars is 128 and many of metrics cross this because
of `otel_scope_name`.
# How the Problems Are Solved
This change drops those otel specific labels on the prometheus exporter:
`otel_scope_name` and `otel_scope_version`
Current metrics example:
```
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="0"} 0
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="5"} 100
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="10"} 100
...
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListIAMMemberRoles",otel_scope_name="",otel_scope_version="",return_code="200"} 3
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListIAMMembers",otel_scope_name="",otel_scope_version="",return_code="200"} 3
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListMilestones",otel_scope_name="",otel_scope_version="",return_code="200"} 1
```
New example:
```
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="10"} 8
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="25"} 8
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="50"} 9
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="75"} 9
...
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/GetSupportedLanguages",return_code="200"} 1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListMilestones",return_code="200"} 1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.auth.v1.AuthService/GetMyLabelPolicy",return_code="200"} 3
```
# Additional Changes
None
# Additional Context
From my understanding, this change is fully spec compliant with
Prometheus and Otel:
*
https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#instrumentation-scope
However, these tags were originally added as optional labels to
disambiguate metrics. But I'm not sure we need to care about that right
now? My gut feeling is that exemplar support (the ability for traces to
reference metrics) would be a preferable tradeoff to this label
standard.
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Livio Spring
Livio Spring
Tim Möhlmann
Livio Spring
Silvan
Marco A
kkrime
Zach Hirschtritt
Max Peintner
Ramon
Copilot