# Which Problems Are Solved
When an expired invite code was used for webauthn (passkeys), the change date got updated by the failed event. This change date was used to test for expiry, meaning failed events would reset the expiry timeout.
# How the Problems Are Solved
This fix adds a Code Creation Date to the writemodel which gets set by the first event. This can be the added or requested event.
# Additional Changes
- Expiry renamed to CodeExpiry so it's consistent with similar write models using secret codes.
- humanVerifyPasswordlessInitCode takes an algorithm instead of the complete generator, so the method can be unit tested easier
- Added tests that reproduced the original issue
(cherry picked from commit 64ee480f84)
# Which Problems Are Solved
Opaque OIDC access tokens in v2 format, truncated to 80 characters are still considered valid.
Zitadel uses a symmetric AES encryption for opaque tokens. The cleartext payload is a concatenation of a couple of identifiers, such as a token ID and user ID. Internally Zitadel has 2 different versions of token payloads. v1 tokens are no longer created, but are still verified as to not invalidate existing session after upgrade.
The cleartext payload has a format of `<token_id>:<user_id>`. v2 tokens distinguished further where the `token_id` is of the format `v2_<oidc_session_id>-at_<access_token_id>`. This is an example of such a cleartext: `V2_354201447279099906-at_354201447279165442:354201364702363650`
## Impact
V1 token authZ/N session data is retrieved from the database using the (simple) `token_id` value and `user_id` value. The `user_id` (called `subject` in some parts of our code) was used as being the trusted user iD.
V2 token authZ/N session data is retrieved from the database using the `oidc_session_id` and `access_token_id` and in this case the `user_id` from the token is ignored and taken from the session data in the DB.
By truncating the token to 80 chars, the user_id is now missing from the cleartext of the v2 token: `V2_354201447279099906-at_354201447279165442:` The back-end still accepts this for above reasons.
This issue is not considered exploitable, but may look awkward when reproduced.
# How the Problems Are Solved
Compare the token subject with the stored user ID. If they don't match, consider the token invalid, as some type of tampering must have happened.
(cherry picked from commit feab8e1fa3)
(cherry picked from commit d2a6af8e07)
# Which Problems Are Solved
Zitadel v4.7.2 fixed a security issue by switching to the permission v2
framework for user APIs. It appears that systems that are running since
before v2.68 that were affected by a precision bug in the eventstore,
which was fixed in that version. The precision bug results in certain
events being "skipped" while being projected into the fields table, used
by the new permission system. This caused certain membership roles to be
missing, resulting in empty user lists when executed by the affected
member. The permission system basically finds no matching memberships
and therefore returns no users at all.
# How the Problems Are Solved
After research we concluded that the legacy membership projections are
projected correctly. This PR synchronizes the projected state into the
fields table. As the membership roles are not marked unique, all rows
are first deleted and then the correct membership roles are then
inserted. The operation happens in a single transaction, during which
the fields table will remain locked for modifications. This to prevent
possible concurrent modifications to membership states.
# Additional Changes
- none
# Additional Context
- Introduced in
https://github.com/zitadel/zitadel/commit/0e17d0005a98ccbf92139961ef702ef03208ffd3
- Released in
[v4.7.2](https://github.com/zitadel/zitadel/releases/tag/v4.7.2)
- Related: https://github.com/zitadel/zitadel/issues/8863
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 58612a6ef7)
# Which Problems Are Solved
There was an left-behind index introduced to optimize the old and
removed event execution handler. The index confuses prostgres and it
sometimes picks this index in favor of the projection specific index.
This sometimes leads to bad query performance in the projectio handlers.
# How the Problems Are Solved
Drop the index
# Additional Changes
- none
# Additional Context
- Forgotten in https://github.com/zitadel/zitadel/pull/10564
(cherry picked from commit 54554b8fb9)
# Which Problems Are Solved
When the webkey feature flag was not enabled before an upgrade to v4,
all JWT tokens became invalid.
This created a couple of issues:
- All users with JWT access tokens are logged-out
- Clients that are unable to refresh keys based on key ID break
- id_token_hint could no longer be validated.
# How the Problems Are Solved
Force-enable the webkey feature on the v3 version, so that the upgrade
path is cleaner. Sessions now have time to role-over to the new keys
before initiating the upgrade to v4.
# Additional Changes
- none
# Additional Context
- Related https://github.com/zitadel/zitadel/issues/10673
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
We are preparing to roll-out and stabilize webkeys in the next version
of Zitadel. Before removing legacy signing-key code, we must ensure all
existing instances have their webkeys generated.
# How the Problems Are Solved
Add a setup step which generate 2 webkeys for each existing instance
that didn't have webkeys yet.
# Additional Changes
Return an error from the config type-switch, when the type is unknown.
# Additional Context
- Part 1/2 of https://github.com/zitadel/zitadel/issues/10029
- Should be back-ported to v3
(cherry picked from commit fa9de9a0f1)
# Which Problems Are Solved
Some projection queries took a long time to run. It seems that 1 or more
queries couldn't make proper use of the `es_projection` index. This
might be because of a specific complexity aggregate_type and event_type
arguments, making the index unfeasible for postgres.
# How the Problems Are Solved
Following the index recommendation, add and index that covers just
instance_id and position.
# Additional Changes
- none
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/9832
(cherry picked from commit bb56b362a7)
# Which Problems Are Solved
Instance that had improved performance flags set, got event errors when
getting instance features. This is because the improved performance
flags were marshalled using the enumerated integers, but now needed to
be unmashalled using the added UnmarshallText method.
# How the Problems Are Solved
- Remove emnumer generation
# Additional Changes
- none
# Additional Context
- reported on QA
- Backport to next-rc / v3
(cherry picked from commit 0465d5093e)
# Which Problems Are Solved
Webkeys were not generated with new instances when the webkey feature
flag was enabled for instance defaults. This would cause a redirect loop
with console for new instances on QA / coud.
# How the Problems Are Solved
- uncomment the webkeys section on defaults.yaml
- Fix field naming of webkey config
# Additional Changes
- Add all available features as comments.
- Make the improved performance type enum parsable from the config,
untill now they were just ints.
- Running of the enumer command created missing enum entries for feature
keys.
# Additional Context
- Needs to be back-ported to v3 / next-rc
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 91bc71db74)
# Which Problems Are Solved
Outdated dependencies foir packages developed by Zitadel.
Some of them included important security updates from sub-dependencies.
# How the Problems Are Solved
Upgrade all packages under `github.com/zitadel/*` to the latest released
version.
# Additional Changes
- The `github.com/superseriousbusiness/exifremove` was removed from
Github. We copied the cached go mod code to
https://github.com/zitadel/exifremove and use this module now.
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/9422
- Closes https://github.com/zitadel/zitadel/issues/9443
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Setup fails to push all role permission events when running Zitadel with
CockroachDB. `TransactionRetryError`s were visible in logs which finally
times out the setup job with `timeout: context deadline exceeded`
# How the Problems Are Solved
As suggested in the [Cockroach documentation](timeout: context deadline
exceeded), _"break down larger transactions"_. The commands to be pushed
for the role permissions are chunked in 50 events per push. This
chunking is only done with CockroachDB.
# Additional Changes
- gci run fixed some unrelated imports
- access to `command.Commands` for the setup job, so we can reuse the
sync logic.
# Additional Context
Closes#9293
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Systems running with PostgreSQL before Zitadel v2.39 are likely to have
a wrong type for the `in_tx_order` column in the `eventstore.event2`
table. The migration at the time used the `event_sequence` as default
value without typecast, which results in a `bigint` type for that
column. However, when creating the table from scratch, we explicitly
specify the type to be `integer`.
Starting from Zitadel v2.67 we use a Pl/PgSQL function to push events.
The function requires the types from `eventstore.events2` to the same as
the `select` destinations used in the function. In the function
`in_tx_order` is also expected to by of `integer` type.
CochroachDB systems are not affected because `bigint` is an alias to the
`int` type. In other words, CockroachDB uses `int8` when specifying type
`int`. Therefore the types already match.
# How the Problems Are Solved
Retrieve the actual column type currently in use. A template is used to
assign the type to the `ordinality` column returned as `in_tx_order`.
# Additional Changes
- Detailed logging on migration failure
# Additional Context
- Closes#9180
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit bcc6a689fa)
# Which Problems Are Solved
Systems running with PostgreSQL before Zitadel v2.39 are likely to have
a wrong type for the `in_tx_order` column in the `eventstore.event2`
table. The migration at the time used the `event_sequence` as default
value without typecast, which results in a `bigint` type for that
column. However, when creating the table from scratch, we explicitly
specify the type to be `integer`.
Starting from Zitadel v2.67 we use a Pl/PgSQL function to push events.
The function requires the types from `eventstore.events2` to the same as
the `select` destinations used in the function. In the function
`in_tx_order` is also expected to by of `integer` type.
CochroachDB systems are not affected because `bigint` is an alias to the
`int` type. In other words, CockroachDB uses `int8` when specifying type
`int`. Therefore the types already match.
# How the Problems Are Solved
Retrieve the actual column type currently in use. A template is used to
assign the type to the `ordinality` column returned as `in_tx_order`.
# Additional Changes
- Detailed logging on migration failure
# Additional Context
- Closes#9180
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.
# How the Problems Are Solved
Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.
# Additional Changes
None
# Additional Context
Introduced in https://github.com/zitadel/zitadel/pull/9199
(cherry picked from commit ec5f18c168)
# Which Problems Are Solved
The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.
# How the Problems Are Solved
Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.
# Additional Changes
None
# Additional Context
Introduced in https://github.com/zitadel/zitadel/pull/9199
# Which Problems Are Solved
Memberships did not have a fields table fill migration.
# How the Problems Are Solved
Add filling of membership fields to the repeatable steps.
# Additional Changes
- Use the same repeatable step for multiple fill fields handlers.
- Fix an error for PostgreSQL 15 where a subquery in a `FROM` clause
needs an alias ing the `permitted_orgs` function.
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/9188
- Introduced in https://github.com/zitadel/zitadel/pull/9152
# Which Problems Are Solved
It was possible to set a diffent algorithm for the legacy signer. This
is not supported howerver and breaks the token endpoint.
# How the Problems Are Solved
Remove the OIDC.SigningKeyAlgorithm config option and hard-code RS256
for the legacy signer.
# Additional Changes
- none
# Additional Context
Only RS256 is supported by the legacy signer. It was mentioned in the
comment of the config not to use it and use the webkeys resource
instead.
- closes#9121
(cherry picked from commit db8d794794)
# Which Problems Are Solved
Currently ZITADEL defines organization and instance member roles and
permissions in defaults.yaml. The permission check is done on API call
level. For example: "is this user allowed to make this call on this
org". This makes sense on the V1 API where the API is permission-level
shaped. For example, a search for users always happens in the context of
the organization. (Either the organization the calling user belongs to,
or through member ship and the x-zitadel-orgid header.
However, for resource based APIs we must be able to resolve permissions
by object. For example, an IAM_OWNER listing users should be able to get
all users in an instance based on the query filters. Alternatively a
user may have user.read permissions on one or more orgs. They should be
able to read just those users.
# How the Problems Are Solved
## Role permission mapping
The role permission mappings defined from `defaults.yaml` or local
config override are synchronized to the database on every run of
`zitadel setup`:
- A single query per **aggregate** builds a list of `add` and `remove`
actions needed to reach the desired state or role permission mappings
from the config.
- The required events based on the actions are pushed to the event
store.
- Events define search fields so that permission checking can use the
indices and is strongly consistent for both query and command sides.
The migration is split in the following aggregates:
- System aggregate for for roles prefixed with `SYSTEM`
- Each instance for roles not prefixed with `SYSTEM`. This is in
anticipation of instance level management over the API.
## Membership
Current instance / org / project membership events now have field table
definitions. Like the role permissions this ensures strong consistency
while still being able to use the indices of the fields table. A
migration is provided to fill the membership fields.
## Permission check
I aimed keeping the mental overhead to the developer to a minimal. The
provided implementation only provides a permission check for list
queries for org level resources, for example users. In the `query`
package there is a simple helper function `wherePermittedOrgs` which
makes sure the underlying database function is called as part of the
`SELECT` query and the permitted organizations are part of the `WHERE`
clause. This makes sure results from non-permitted organizations are
omitted. Under the hood:
- A Pg/PlSQL function searches for a list of organization IDs the passed
user has the passed permission.
- When the user has the permission on instance level, it returns early
with all organizations.
- The functions uses a number of views. The views help mapping the
fields entries into relational data and simplify the code use for the
function. The views provide some pre-filters which allow proper index
usage once the final `WHERE` clauses are set by the function.
# Additional Changes
# Additional Context
Closes#9032
Closes https://github.com/zitadel/zitadel/issues/9014https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for
the new permission framework based on this concept.
# Which Problems Are Solved
We were seeing high query costs in a the lateral join executed in the
commands_to_events procedural function in the database. The high cost
resulted in incremental CPU usage as a load test continued and less
req/sec handled, sarting at 836 and ending at 130 req/sec.
# How the Problems Are Solved
1. Set `PARALLEL SAFE`. I noticed that this option defaults to `UNSAFE`.
But it's actually safe if the function doesn't `INSERT`
2. Set the returned `ROWS 10` parameter.
3. Function is re-written in Pl/PgSQL so that we eliminate expensive
joins.
4. Introduced an intermediate state that does `SELECT DISTINCT` for the
aggregate so that we don't have to do an expensive lateral join.
# Additional Changes
Use a `COALESCE` to get the owner from the last event, instead of a
`CASE` switch.
# Additional Context
- Function was introduced in
https://github.com/zitadel/zitadel/pull/8816
- Closes https://github.com/zitadel/zitadel/issues/8352
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
We were seeing high query costs in a the lateral join executed in the
commands_to_events procedural function in the database. The high cost
resulted in incremental CPU usage as a load test continued and less
req/sec handled, sarting at 836 and ending at 130 req/sec.
# How the Problems Are Solved
1. Set `PARALLEL SAFE`. I noticed that this option defaults to `UNSAFE`.
But it's actually safe if the function doesn't `INSERT`
2. Set the returned `ROWS 10` parameter.
3. Function is re-written in Pl/PgSQL so that we eliminate expensive
joins.
4. Introduced an intermediate state that does `SELECT DISTINCT` for the
aggregate so that we don't have to do an expensive lateral join.
# Additional Changes
Use a `COALESCE` to get the owner from the last event, instead of a
`CASE` switch.
# Additional Context
- Function was introduced in
https://github.com/zitadel/zitadel/pull/8816
- Closes https://github.com/zitadel/zitadel/issues/8352
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
It was possible to set a diffent algorithm for the legacy signer. This
is not supported howerver and breaks the token endpoint.
# How the Problems Are Solved
Remove the OIDC.SigningKeyAlgorithm config option and hard-code RS256
for the legacy signer.
# Additional Changes
- none
# Additional Context
Only RS256 is supported by the legacy signer. It was mentioned in the
comment of the config not to use it and use the webkeys resource
instead.
- closes#9121
# Which Problems Are Solved
When `LastUseAge` was configured properly, the Redis LUA script uses
manual cleanup for `MaxAge` based expiry. The expiry obtained from Redis
apears to be a string and was compared to an int, resulting in a script
error.
# How the Problems Are Solved
Convert expiry to number.
# Additional Changes
- none
# Additional Context
- Introduced in #8822
- LastUseAge was fixed in #9097
- closes https://github.com/zitadel/zitadel/issues/9140
(cherry picked from commit 56427cca50)
# Which Problems Are Solved
When `LastUseAge` was configured properly, the Redis LUA script uses
manual cleanup for `MaxAge` based expiry. The expiry obtained from Redis
apears to be a string and was compared to an int, resulting in a script
error.
# How the Problems Are Solved
Convert expiry to number.
# Additional Changes
- none
# Additional Context
- Introduced in #8822
- LastUseAge was fixed in #9097
- closes https://github.com/zitadel/zitadel/issues/9140
# Which Problems Are Solved
When downgrading zitadel and upgrading it again, it might be that orgs
deleted in this period still have stale entries in the fields table.
# How the Problems Are Solved
- Make the cleanup repeatable
- Scope the query by instance so that an index is used.
(cherry picked from commit da706a8b30)
# Which Problems Are Solved
When downgrading zitadel and upgrading it again, it might be that orgs
deleted in this period still have stale entries in the fields table.
# How the Problems Are Solved
- Make the cleanup repeatable
- Scope the query by instance so that an index is used.
# Which Problems Are Solved
Migration step 39 is supposed to cleanup stale organization entries in
the eventstore.fields table. In order to do this it used the projection
to check which orgs still exist.
During initial setup of ZITADEL the first instance with the organization
is created. Howevet, the projections are filled after all migrations are
done. With the organization projection empty, the fields of the first
org would be deleted.
This was discovered during development of a new field type. The
accosiated events did not yet have any projection based filled assigned.
It seems fields with a pre-fill projection are somehow restored.
Therefore a restoration migration isn't required IMO.
# How the Problems Are Solved
Query the event store for `org.removed` events instead. This has the
drawback of using a sequential scan on the eventstore, making the
migration more expensive.
# Additional Changes
- none
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/8946
# Which Problems Are Solved
Integration guide with event API examples used outdated
`user.token.added` events which are no longer produced by ZITADEL.
# How the Problems Are Solved
Modify the example to use events from the `oidc_session` aggregate.
# Additional Changes
- Add a TODO for related SAML events.
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8983
# Which Problems Are Solved
When Zitadel starts the first time with a configured Redis cache, the
circuit break would open on the first requests, with no explanatory
error and only log-lines explaining the state of the Circuit breaker.
Using a debugger, `NOSCRIPT No matching script. Please use EVAL.` was
found the be passed to `Limiter.ReportResult`. This error is actually
retried by go-redis after a
[`Script.Run`](https://pkg.go.dev/github.com/redis/go-redis/v9@v9.7.0#Script.Run):
> Run optimistically uses EVALSHA to run the script. If script does not
exist it is retried using EVAL.
# How the Problems Are Solved
Add the `NOSCRIPT` error prefix to the whitelist.
# Additional Changes
- none
# Additional Context
- Introduced in: https://github.com/zitadel/zitadel/pull/8890
- Workaround for: https://github.com/redis/go-redis/issues/3203
# Which Problems Are Solved
In eventstore queries with aggregate ID exclusion filters, filters on
events creation date where not passed to the sub-query. This results in
a high amount of returned rows from the sub-query and high overall query
cost.
# How the Problems Are Solved
When CreatedAfter and CreatedBefore are used on the global search query,
copy those filters to the sub-query. We already did this for the
position column filter.
# Additional Changes
- none
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/8940
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Integration tests are flaky due to eventual consistency.
# How the Problems Are Solved
Remove t.Parallel so that less concurrent requests on multiple instance
happen. This allows the projections to catch up more easily.
# Additional Changes
- none
# Additional Context
- none
# Which Problems Are Solved
When an org is removed, the corresponding fields are not deleted. This
creates issues, such as recreating a new org with the same verified
domain.
# How the Problems Are Solved
Remove the search fields by the org aggregate, instead of just setting
the removed state.
# Additional Changes
- Cleanup migration script that removed current stale fields.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8943
- Related to https://github.com/zitadel/zitadel/pull/8790
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
# Which Problems Are Solved
For truly event-based notification handler, we need to be able to filter
out events of aggregates which are already handled. For example when an
event like `notify.success` or `notify.failed` was created on an
aggregate, we no longer require events from that aggregate ID.
# How the Problems Are Solved
Extend the query builder to use a `NOT IN` clause which excludes
aggregate IDs when they have certain events for a certain aggregate
type. For optimization and proper index usages, certain filters are
inherited from the parent query, such as:
- Instance ID
- Instance IDs
- Position offset
This is a prettified query as used by the unit tests:
```sql
SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM eventstore.events2
WHERE instance_id = $1
AND aggregate_type = $2
AND event_type = $3
AND "position" > $4
AND aggregate_id NOT IN (
SELECT aggregate_id
FROM eventstore.events2
WHERE aggregate_type = $5
AND event_type = ANY($6)
AND instance_id = $7
AND "position" > $8
)
ORDER BY "position" DESC, in_tx_order DESC
LIMIT $9
```
I used this query to run it against the `oidc_session` aggregate looking
for added events, excluding aggregates where a token was revoked,
against a recent position. It fully used index scans:
<details>
```json
[
{
"Plan": {
"Node Type": "Index Scan",
"Parallel Aware": false,
"Async Capable": false,
"Scan Direction": "Forward",
"Index Name": "es_projection",
"Relation Name": "events2",
"Alias": "events2",
"Actual Rows": 2,
"Actual Loops": 1,
"Index Cond": "((instance_id = '286399006995644420'::text) AND (aggregate_type = 'oidc_session'::text) AND (event_type = 'oidc_session.added'::text) AND (\"position\" > 1731582100.784168))",
"Rows Removed by Index Recheck": 0,
"Filter": "(NOT (hashed SubPlan 1))",
"Rows Removed by Filter": 1,
"Plans": [
{
"Node Type": "Index Scan",
"Parent Relationship": "SubPlan",
"Subplan Name": "SubPlan 1",
"Parallel Aware": false,
"Async Capable": false,
"Scan Direction": "Forward",
"Index Name": "es_projection",
"Relation Name": "events2",
"Alias": "events2_1",
"Actual Rows": 1,
"Actual Loops": 1,
"Index Cond": "((instance_id = '286399006995644420'::text) AND (aggregate_type = 'oidc_session'::text) AND (event_type = 'oidc_session.access_token.revoked'::text) AND (\"position\" > 1731582100.784168))",
"Rows Removed by Index Recheck": 0
}
]
},
"Triggers": [
]
}
]
```
</details>
# Additional Changes
- None
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8931
---------
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
# Which Problems Are Solved
We need a reliable way to lock events that are being processed as part
of a job queue. For example in the notification handlers.
# How the Problems Are Solved
Allow setting `FOR UPDATE [ NOWAIT | SKIP LOCKED ]` to the eventstore
query builder using an open transaction.
- NOWAIT returns an errors if the lock cannot be obtained
- SKIP LOCKED only returns row which are not locked.
- Default is to wait for the lock to be released.
# Additional Changes
- none
# Additional Context
- [Locking
docs](https://www.postgresql.org/docs/17/sql-select.html#SQL-FOR-UPDATE-SHARE)
- Related to https://github.com/zitadel/zitadel/issues/8931
# Which Problems Are Solved
Organizations are ofter searched for by ID or primary domain. This
results in many redundant queries, resulting in a performance impact.
# How the Problems Are Solved
Cache Organizaion objects by ID and primary domain.
# Additional Changes
- Adjust integration test config to use all types of cache.
- Adjust integration test lifetimes so the pruner has something to do
while the tests run.
# Additional Context
- Closes#8865
- After #8902
# Which Problems Are Solved
Explain the usage of the new cache mechanisms.
# How the Problems Are Solved
Provide a dedicated page on caches with reference to `defaults.yaml`.
# Additional Changes
- Fix a broken link tag in token exchange docs.
# Additional Context
- Closes#8855
# Which Problems Are Solved
Load-test requires single endpoint to be used for each test type.
# How the Problems Are Solved
Remove userinfo call from machine tests.
# Additional Changes
- Add load-test/.env to gitignore.
# Additional Context
- Related to #4424
# Which Problems Are Solved
By having default entries in the `Username` and `ClientName` fields, it
was not possible to unset there parameters. Unsetting them is required
for GCP connections
# How the Problems Are Solved
Set the fields to empty strings.
# Additional Changes
- none
# Additional Context
- none
(cherry picked from commit b77901cb4b)
# Which Problems Are Solved
Do not return an access token for implicit flow from v1 login, if the
`response_type` is `id_token`
# How the Problems Are Solved
Do not create the access token event if if the `response_type` is
`id_token`.
# Additional Changes
Token endpoint calls without auth request, such as machine users, token
exchange and refresh token, do not have a `response_type`. For such
calls the `OIDCResponseTypeUnspecified` enum is added at a `-1` offset,
in order not to break existing client configs.
# Additional Context
- https://discord.com/channels/927474939156643850/1294001717725237298
- Fixes https://github.com/zitadel/zitadel/issues/8776
(cherry picked from commit 778b4041ca)
# Which Problems Are Solved
By having default entries in the `Username` and `ClientName` fields, it
was not possible to unset there parameters. Unsetting them is required
for GCP connections
# How the Problems Are Solved
Set the fields to empty strings.
# Additional Changes
- none
# Additional Context
- none
# Which Problems Are Solved
If a redis cache has connection issues or any other type of permament
error,
it tanks the responsiveness of ZITADEL.
We currently do not support things like Redis cluster or sentinel. So
adding a simple redis cache improves performance but introduces a single
point of failure.
# How the Problems Are Solved
Implement a [circuit
breaker](https://learn.microsoft.com/en-us/previous-versions/msp-n-p/dn589784(v=pandp.10)?redirectedfrom=MSDN)
as
[`redis.Limiter`](https://pkg.go.dev/github.com/redis/go-redis/v9#Limiter)
by wrapping sony's [gobreaker](https://github.com/sony/gobreaker)
package. This package is picked as it seems well maintained and we
already use their `sonyflake` package
# Additional Changes
- The unit tests constructed an unused `redis.Client` and didn't cleanup
the connector. This is now fixed.
# Additional Context
Closes#8864
# Which Problems Are Solved
Do not return an access token for implicit flow from v1 login, if the
`response_type` is `id_token`
# How the Problems Are Solved
Do not create the access token event if if the `response_type` is
`id_token`.
# Additional Changes
Token endpoint calls without auth request, such as machine users, token
exchange and refresh token, do not have a `response_type`. For such
calls the `OIDCResponseTypeUnspecified` enum is added at a `-1` offset,
in order not to break existing client configs.
# Additional Context
- https://discord.com/channels/927474939156643850/1294001717725237298
- Fixes https://github.com/zitadel/zitadel/issues/8776
# Which Problems Are Solved
Add a cache implementation using Redis single mode. This does not add
support for Redis Cluster or sentinel.
# How the Problems Are Solved
Added the `internal/cache/redis` package. All operations occur
atomically, including setting of secondary indexes, using LUA scripts
where needed.
The [`miniredis`](https://github.com/alicebob/miniredis) package is used
to run unit tests.
# Additional Changes
- Move connector code to `internal/cache/connector/...` and remove
duplicate code from `query` and `command` packages.
- Fix a missed invalidation on the restrictions projection
# Additional Context
Closes#8130
# Which Problems Are Solved
Migration of milestones failed on our QA due to the new milestone Type
enum being 0-indexed. The valid range was 0 till 5, inclusive. While on
the previous zitadel version this was 1 till 6, inclusive.
# How the Problems Are Solved
Offset the first constant with `1`.
# Additional Changes
- none
# Additional Context
Introduced in https://github.com/zitadel/zitadel/pull/8788
# Which Problems Are Solved
Milestones used existing events from a number of aggregates. OIDC
session is one of them. We noticed in load-tests that the reduction of
the oidc_session.added event into the milestone projection is a costly
business with payload based conditionals. A milestone is reached once,
but even then we remain subscribed to the OIDC events. This requires the
projections.current_states to be updated continuously.
# How the Problems Are Solved
The milestone creation is refactored to use dedicated events instead.
The command side decides when a milestone is reached and creates the
reached event once for each milestone when required.
# Additional Changes
In order to prevent reached milestones being created twice, a migration
script is provided. When the old `projections.milestones` table exist,
the state is read from there and `v2` milestone aggregate events are
created, with the original reached and pushed dates.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8800
# Which Problems Are Solved
Since the optiimzation of the token endpoint, we longer push the
`user.token.added` event. However, the milestone projection keeps
quering for it, including a payload query.
This incured a static waste of DB resources.
# How the Problems Are Solved
Remove the `user.token.added` event reducer from the milestone
projection
# Additional Changes
- none
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8742. Other
changes ommitted so this PR can be backported to stable.
# Which Problems Are Solved
Optimize the query that checks for terminated sessions in the access
token verifier. The verifier is used in auth middleware, userinfo and
introspection.
# How the Problems Are Solved
The previous implementation built a query for certain events and then
appended a single `PositionAfter` clause. This caused the postgreSQL
planner to use indexes only for the instance ID, aggregate IDs,
aggregate types and event types. Followed by an expensive sequential
scan for the position. This resulting in internal over-fetching of rows
before the final filter was applied.
Furthermore, the query was searching for events which are not always
applicable. For example, there was always a session ID search and if
there was a user ID, we would also search for a browser fingerprint in
event payload (expensive). Even if those argument string would be empty.
This PR changes:
1. Nest the position query, so that a full `instance_id, aggregate_id,
aggregate_type, event_type, "position"` index can be matched.
2. Redefine the `es_wm` index to include the `position` column.
3. Only search for events for the IDs that actually have a value. Do not
search (noop) if none of session ID, user ID or fingerpint ID are set.
New query plan:
# Additional Changes
- cleanup how we load multi-statement migrations and make that a bit
more reusable.
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/7639
# Which Problems Are Solved
Cache implementation using a PGX connection pool.
# How the Problems Are Solved
Defines a new schema `cache` in the zitadel database.
A table for string keys and a table for objects is defined.
For postgreSQL, tables are unlogged and partitioned by cache name for
performance.
Cockroach does not have unlogged tables and partitioning is an
enterprise feature that uses alternative syntax combined with sharding.
Regular tables are used here.
# Additional Changes
- `postgres.Config` can return a pxg pool. See following discussion
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/8648
- Closes https://github.com/zitadel/zitadel/issues/8647
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
# Which Problems Are Solved
When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.
At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.
However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.
# How the Problems Are Solved
Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.
# Additional Changes
- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`
# Additional Context
- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check
Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:
Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 63d733b3a2)
# Which Problems Are Solved
When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.
At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.
However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.
# How the Problems Are Solved
Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.
# Additional Changes
- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`
# Additional Context
- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check
Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:
Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
We identified the need of caching.
Currently we have a number of places where we use different ways of
caching, like go maps or LRU.
We might also want shared chaches in the future, like Redis-based or in
special SQL tables.
# How the Problems Are Solved
Define a generic Cache interface which allows different implementations.
- A noop implementation is provided and enabled as.
- An implementation using go maps is provided
- disabled in defaults.yaml
- enabled in integration tests
- Authz middleware instance objects are cached using the interface.
# Additional Changes
- Enabled integration test command raceflag
- Fix a race condition in the limits integration test client
- Fix a number of flaky integration tests. (Because zitadel is super
fast now!) 🎸🚀
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/8648
# Which Problems Are Solved
Reduce the chance for projection dead-locks. Increasing or disabling the
projection transaction duration solved dead-locks in all reported cases.
# How the Problems Are Solved
Increase the default transaction duration to 1 minute.
Due to the high value it is functionally similar to disabling,
however it still provides a safety net for transaction that do freeze,
perhaps due to connection issues with the database.
# Additional Changes
- Integration test uses default.
- Technical advisory
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8517
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
# Which Problems Are Solved
When the `openid` scope was not requested, as is possible in machine
authentication, we didn't set the `sub` (subject) claim to tokens and
possibly also userInfo and introspection.
This fix always sets the `sub` claim for all cases.
# How the Problems Are Solved
Set the `Subject` field to regardless of passed scopes.
# Additional Changes
- none
# Additional Context
According to standards:
- [RFC9068 - JSON Web Token (JWT) Profile for OAuth 2.0 Access
Tokens](https://datatracker.ietf.org/doc/html/rfc9068#name-data-structure)
this claim is **required**.
- [RFC7667 - OAuth 2.0 Token
Introspection](https://datatracker.ietf.org/doc/html/rfc7662#section-2.2)
the claim is optional, however there is no correlation to the `openid`
or OpenID Connect. Therefore it doesn't harm to always return this
claim.
- [OpenID connect, User Info
Response](https://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse):
"The sub (subject) Claim **MUST** always be returned in the UserInfo
Response."
Closes https://github.com/zitadel/zitadel/issues/8591
# Which Problems Are Solved
Upload the integration test server logs as artifacts, even if the tests
fail.
Before this change logs where printed through the Makefile.
However if a test would fail, the logs wouldn't get printed.
# How the Problems Are Solved
- Add an extra build step that pushes `tmp/zitadel.log` and
`tmp/race.log.$pid` to artificats storage.
- Logs are no longer printed in the `core_integration_reports` Makefile
recipe.
# Additional Changes
Do not remove coverage data when generating the coverage report in
`core_integration_reports`. This is to prevent future "File not found"
erros when running the command repeatedly.
# Additional Context
Reported as internal feedback
# Which Problems Are Solved
In actions/v3 there was no check for the feature flag on any of the:
- ListExecutionFunctions
- ListExecutionMethods
- ListExecutionServices
In the integration tests `ensureFeatureEnabled` relies on
`ListExecutionMethods` to return an error if the feature is not enabled.
This fix makes the test wait untill the feature is fully projected.
# How the Problems Are Solved
Add the feature check to all of the above methods.
# Additional Changes
- none
# Additional Context
Flaky introduced in https://github.com/zitadel/zitadel/pull/8407
# Which Problems Are Solved
Improve performance by removing a GetUserByID call. The call also
executed a Trigger on projections, which significantly impacted
concurrent requests.
# How the Problems Are Solved
Token creation needs information from the user, such as the resource
owner and access token type.
For client credentials this is solved in a single search. By getting the
user by username (`client_id`), the user details and secret were
obtained in a single query. After that verification and token creation
can proceed. For JWT profile it is a bit more complex. We didn't know
anything about the user until after JWT verification.
The verification did a query for the AuthN key and after that we did a
GetUserByID to get remaining details.
This change uses a joined query when the OIDC library calls the
`GetKeyByIDAndClientID` method on the token storage. The found user
details are set to the verifieer object and returned after verification
is completed.
It is safe because the `jwtProfileKeyStorage` is a single-use object as
a wrapper around `query.Queries`.
This way getting the public key and user details are obtained in a
single query.
# Additional Changes
- Correctly set the `client_id` field with machine's username.
# Additional Context
- Related to: https://github.com/zitadel/zitadel/issues/8352
# Which Problems Are Solved
Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.
# How the Problems Are Solved
- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
- Add
- Change
- Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger
# Additional Changes
- none
# Additional Context
- Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
# Which Problems Are Solved
defaults.yaml only specifies defaults for cockroach. Therefore, options
omitted for postgresql are actually set to `0`.
This means that the connections timeouts are set to `0` and connections
were not reused, resulting in a performance penalty while running the
integration tests.
# How the Problems Are Solved
Set MaxConnLifeTime and MaxConnIdleTime options in postgres
# Additional Changes
- none
# Additional Context
- none
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Return the user's project roles when the
`urn:zitadel:iam:org:projects:roles` scope is requested.
We alreayd returned it for access tokens, now also ID tokens.
# How the Problems Are Solved
Set `idTokenRoleAssertion` to `true` when calling
`accessTokenResponseFromSession` for service users. This parameter is
normally set to the client config. However, service user authentication
does not have a client.
# Additional Changes
- none
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/8046
- Closes https://github.com/zitadel/zitadel/issues/8107
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Deal with eventual consistency in the webkey integration tests.
# How the Problems Are Solved
Use an eventual with T for the list state check.
# Additional Changes
- none
# Additional Context
- none
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Use a single server instance for API integration tests. This optimizes
the time taken for the integration test pipeline,
because it allows running tests on multiple packages in parallel. Also,
it saves time by not start and stopping a zitadel server for every
package.
# How the Problems Are Solved
- Build a binary with `go build -race -cover ....`
- Integration tests only construct clients. The server remains running
in the background.
- The integration package and tested packages now fully utilize the API.
No more direct database access trough `query` and `command` packages.
- Use Makefile recipes to setup, start and stop the server in the
background.
- The binary has the race detector enabled
- Init and setup jobs are configured to halt immediately on race
condition
- Because the server runs in the background, races are only logged. When
the server is stopped and race logs exist, the Makefile recipe will
throw an error and print the logs.
- Makefile recipes include logic to print logs and convert coverage
reports after the server is stopped.
- Some tests need a downstream HTTP server to make requests, like quota
and milestones. A new `integration/sink` package creates an HTTP server
and uses websockets to forward HTTP request back to the test packages.
The package API uses Go channels for abstraction and easy usage.
# Additional Changes
- Integration test files already used the `//go:build integration`
directive. In order to properly split integration from unit tests,
integration test files need to be in a `integration_test` subdirectory
of their package.
- `UseIsolatedInstance` used to overwrite the `Tester.Client` for each
instance. Now a `Instance` object is returned with a gRPC client that is
connected to the isolated instance's hostname.
- The `Tester` type is now `Instance`. The object is created for the
first instance, used by default in any test. Isolated instances are also
`Instance` objects and therefore benefit from the same methods and
values. The first instance and any other us capable of creating an
isolated instance over the system API.
- All test packages run in an Isolated instance by calling
`NewInstance()`
- Individual tests that use an isolated instance use `t.Parallel()`
# Additional Context
- Closes#6684
- https://go.dev/doc/articles/race_detector
- https://go.dev/doc/build-cover
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Do not push secret succeeded and failed events for API and OIDC clients
on the introspection endpoint.
On instances where introspection was fequently called, the pushed events
created issues on duplicate primary keys, due to collisions on the
`sequence` column in the eventstore. As the event pusher retries on this
collision and we pushed above mentioned events async, it would create a
backpressure of concurrent pushers and effectively cripple an instance.
We considered that pushing these events have little value with regards
to the audit trail, as we do not push similar events when client
assertion is used. Also, before #7657 the events were defined, but not
pushed.
# How the Problems Are Solved
- Removed API secret check succeeded and faild event definitions
- Removed OIDC secret check succeeded and faild event definitions
- Push only Hash Updated event when needed
# Additional Changes
- None
# Additional Context
- Fixes https://github.com/zitadel/zitadel/issues/8479
- Closes https://github.com/zitadel/zitadel/issues/8430
- Intoduced in https://github.com/zitadel/zitadel/pull/7657
# Which Problems Are Solved
Use web keys, managed by the `resources/v3alpha/web_keys` API, for OIDC
token signing and verification,
as well as serving the public web keys on the jwks / keys endpoint.
Response header on the keys endpoint now allows caching of the response.
This is now "safe" to do since keys can be created ahead of time and
caches have sufficient time to pickup the change before keys get
enabled.
# How the Problems Are Solved
- The web key format is used in the `getSignerOnce` function in the
`api/oidc` package.
- The public key cache is changed to get and store web keys.
- The jwks / keys endpoint returns the combined set of valid "legacy"
public keys and all available web keys.
- Cache-Control max-age default to 5 minutes and is configured in
`defaults.yaml`.
When the web keys feature is enabled, fallback mechanisms are in place
to obtain and convert "legacy" `query.PublicKey` as web keys when
needed. This allows transitioning to the feature without invalidating
existing tokens. A small performance overhead may be noticed on the keys
endpoint, because 2 queries need to be run sequentially. This will
disappear once the feature is stable and the legacy code gets cleaned
up.
# Additional Changes
- Extend legacy key lifetimes so that tests can be run on an existing
database with more than 6 hours apart.
- Discovery endpoint returns all supported algorithms when the Web Key
feature is enabled.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8031
- Part of https://github.com/zitadel/zitadel/issues/7809
- After https://github.com/zitadel/oidc/pull/637
- After https://github.com/zitadel/oidc/pull/638
# Which Problems Are Solved
Currently the OIDC API of ZITADEL only prints parent errors to the logs.
Where 4xx status are typically warn level and 5xx error level. This
makes it hard to debug certain errors for client in multi-instance
environments like ZITADEL cloud, where there is no direct access to
logs. In case of support requests we often can't correlate past
log-lines to the error that was reported.
This change adds the possibility to return the parent error in the
response to the OIDC client. For the moment this only applies to JSON
body responses, not error redirects to the RP.
# How the Problems Are Solved
- New instance-level feature flag: `debug_oidc_parent_error`
- Use the new `WithReturnParentToClient()` function from the oidc lib
introduced in https://github.com/zitadel/oidc/pull/629 for all cases
where `WithParent` was already used and the request context is
available.
# Additional Changes
none
# Additional Context
- Depends on: https://github.com/zitadel/oidc/pull/629
- Related to: https://github.com/zitadel/zitadel/issues/8362
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Sometimes integrations tests are failing with an error `http: no
location header in response`. The underlying cause was hidden, as in
some tests we assumed a 3xx range response but got a 4xx response
instead. No assertion on the status code was made, resulting in the
above error message on calling `resp.Location()`.
The underlying issue, the application not found in the projection, is
also fixed.
# How the Problems Are Solved
This change adds a check for the status code and returns the response
body if the response is not in the 3xx status code range.
Helper function that create applications now do an additional
`GetAppByID` in a retry loop to ensure consitency in the projection
before proceeding with tests.
# Additional Changes
- none
# Additional Context
- Pipline failures were observed, no issue was created
- Cherry-picked form WIP #8407
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
When Web keys with a large RSA Bitsize were generated, the new web key
did not get projected.
Zitadel logs printed the following errors:
```
time="2024-08-15T09:18:47Z" level=debug msg="execution of statements failed" caller="/home/tim/Repositories/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:518" error="statement failed: ID=CRDB-pKtsr Message=exec failed Parent=(ERROR: index row size 3400 exceeds btree version 4 maximum 2704 for index \"web_keys_web_key_state_idx\" (SQLSTATE 54000))" lastProcessedIndex=-1 projection=projections.web_keys
```
# How the Problems Are Solved
Remove the the `WithInclude` option with the Private key from the
`web_keys` projection index and bump the projection version.
# Additional Changes
- Correct RPC summaries for documentation
- Fix docs slug in sidebar.js
# Additional Context
- Found during release review.
# Which Problems Are Solved
Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.
The API accepts the generation of the following key types and
parameters:
- RSA keys with 2048, 3072 or 4096 bit in size and:
- Signing with SHA-256 (RS256)
- Signing with SHA-384 (RS384)
- Signing with SHA-512 (RS512)
- ECDSA keys with
- P256 curve
- P384 curve
- P512 curve
- ED25519 keys
# How the Problems Are Solved
Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.
Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:
1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted
# Additional Changes
- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files
# Additional Context
- Closes#8030
- Part of #7809
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
Log some details when a secret generator is not found.
This should help us debugging such issue.
# How the Problems Are Solved
When a secret generator by type query fails,
we log the generator type and instance id for which
the generator was requested.
# Additional Changes
- none
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8379
- Also encountered in https://github.com/zitadel/zitadel/pull/8407
# Which Problems Are Solved
We noticed logging where 500: Internal Server errors were returned from
the token endpoint, mostly for the `refresh_token` grant. The error was
thrown by the database as it received non-UTF8 strings for token IDs
Zitadel uses symmetric encryption for opaque tokens, including refresh
tokens. Encrypted values are base64 encoded. It appeared to be possible
to send garbage base64 to the token endpoint, which will pass decryption
and string-splitting. In those cases the resulting ID is not a valid
UTF-8 string.
Invalid non-UTF8 strings are now rejected during token decryption.
# How the Problems Are Solved
- `AESCrypto.DecryptString()` checks if the decrypted bytes only contain
valid UTF-8 characters before converting them into a string.
- `AESCrypto.Decrypt()` is unmodified and still allows decryption on
non-UTF8 byte strings.
- `FromRefreshToken` now uses `DecryptString` instead of `Decrypt`
# Additional Changes
- Unit tests added for `FromRefreshToken` and
`AESCrypto.DecryptString()`.
- Fuzz tests added for `FromRefreshToken` and
`AESCrypto.DecryptString()`. This was to pinpoint the problem
- Testdata with values that resulted in invalid strings are committed.
In the pipeline this results in the Fuzz tests to execute as regular
unit-test cases. As we don't use the `-fuzz` flag in the pipeline no
further fuzzing is performed.
# Additional Context
- Closes#7765
- https://go.dev/doc/tutorial/fuzz
# Which Problems Are Solved
We noticed logging where 500: Internal Server errors were returned from
the token endpoint, mostly for the `refresh_token` grant. The error was
thrown by the database as it received non-UTF8 strings for token IDs
Zitadel uses symmetric encryption for opaque tokens, including refresh
tokens. Encrypted values are base64 encoded. It appeared to be possible
to send garbage base64 to the token endpoint, which will pass decryption
and string-splitting. In those cases the resulting ID is not a valid
UTF-8 string.
Invalid non-UTF8 strings are now rejected during token decryption.
# How the Problems Are Solved
- `AESCrypto.DecryptString()` checks if the decrypted bytes only contain
valid UTF-8 characters before converting them into a string.
- `AESCrypto.Decrypt()` is unmodified and still allows decryption on
non-UTF8 byte strings.
- `FromRefreshToken` now uses `DecryptString` instead of `Decrypt`
# Additional Changes
- Unit tests added for `FromRefreshToken` and
`AESCrypto.DecryptString()`.
- Fuzz tests added for `FromRefreshToken` and
`AESCrypto.DecryptString()`. This was to pinpoint the problem
- Testdata with values that resulted in invalid strings are committed.
In the pipeline this results in the Fuzz tests to execute as regular
unit-test cases. As we don't use the `-fuzz` flag in the pipeline no
further fuzzing is performed.
# Additional Context
- Closes#7765
- https://go.dev/doc/tutorial/fuzz
# Which Problems Are Solved
Improve the performance of human imports by optimizing the query that
finds domains claimed by other organizations.
# How the Problems Are Solved
Use the fields search table introduced in
https://github.com/zitadel/zitadel/pull/8191 by storing each
organization domain as Object ID and the verified status as field value.
# Additional Changes
- Feature flag for this optimization
# Additional Context
- Performance improvements for import are evaluated and acted upon
internally at the moment
---------
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
# Which Problems Are Solved
Imporve the performance of user grant addition, especially for import.
# How the Problems Are Solved
Use the search table to query for the project grant state.
This could easily be done by making the search used in
`checkProjectGrantPreCondition` reusable.
# Additional Changes
Chanded event declerations to `const` in the
`internal/repository/project` package.
# Additional Context
- Performance improvements for import are evaluated and acted upon
internally at the moment
---------
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Allow verification of imported passwords hashed with plain md5, without
salt. These are password digests typically created by one of:
- `printf "password" | md5sum` on most linux systems.
- PHP's `md5("password")`
- Python3's `hashlib.md5(b"password").hexdigest()`
# How the Problems Are Solved
- Upgrade passwap to
[v0.6.0](https://github.com/zitadel/passwap/releases/tag/v0.6.0)
- Add md5plain as a new verfier option in `defaults.yaml`
# Additional Changes
- Updated documentation to explain difference between `md5` (crypt) and
`md5plain` verifiers.
# Additional Context
- Requested by customer for import case
# Which Problems Are Solved
Improve the performance of the `admin/v1/import` API endpoint.
Specifaclly the import of large amount of project grants.
# How the Problems Are Solved
`AddProjectGrantWithID` and `AddProjectGrantMember` methods of
`Commands` used to get the current state of the Writemodel to check if
the current GrantID or the combination of GrantID & UserID wasn't
already used. However, the Added events already have protection against
duplication by the `UniqueConstaint` methods.
The queries become very slow when there is a great amount of project
grants. Because all the events are pushed to the aggregate ID of the
project, we had to obtain all related project events, including events
of grantIDs we do not care about. This O(n) duration for bached import
jobs adding many organization granted to a single project.
This change removes the unnecesary state query to improve performance.
# Additional Changes
- Add integration tests for import
# Additional Context
- reported internally
# Which Problems Are Solved
This fix adds tracing spans to all V1 API import related functions. This
is to troubleshoot import related performance issues reported to us.
# How the Problems Are Solved
Add a tracing span to `api/grpc/admin/import.go` and all related
functions that are called in the `command` package.
# Additional Changes
- none
# Additional Context
- Reported by internal communication
# Which Problems Are Solved
Zitadel never stored or returned the requested `response_mode` in oidc
Auth Requests. This caused the oidc library to fallback to the default
based on the response_type.
# How the Problems Are Solved
- Store the `response_mode` in the Auth request repo
- Store the `response_mode` in the Auth request v2 events
- Return the `resonse_mode` from the Auth Request v1 and v2
`ResponseMode()` methods. (Was hard-coded to an empty string)
# Additional Changes
- Populate the `response_modes_supported` to the oidc Discovery
Configuration. When it was empty, the standard specifies the default of
`query` and `fragment`. However, our oidc library also supports
`form_post` and by this fix, zitadel now also supports this.
# Additional Context
- Closes#6586
- Reported
https://discord.com/channels/927474939156643850/1151508313717084220
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
An admin / application might want to be able to reduce the amount of
roles returned in the token, for example if a user is granted to many
organizations or for specific cases where the application want to narrow
down the access for that token to a specific organization or multiple.
This can now be achieved by providing a scope with the id of the
organization, resp. multiple scopes for every organization, which should
be included.
```
urn:zitadel:iam:org:roles🆔{orgID}
```
**Note:** the new scope does not work when Introspection / Userinfo are
set to legacy mode.
# How the Problems Are Solved
The user info query now has two variants:
1. Variant that returns all organization authorization grants if the new
scope wasn't provided for backward compatibility.
2. Variant that filters the organizations based on the IDs passed in one
or more of the above scopes and returns only those authorization grants.
The query is defined as a `text/template` and both variants are rendered
once in package `init()`.
# Additional Changes
- In the integration tests `assertProjectRoleClaims` now also checks the
org IDs in the roles.
# Additional Context
- Closes#7996
# Which Problems Are Solved
After https://github.com/zitadel/zitadel/pull/7822 was merged we
discovered that
v2 tokens that where obtained through an IDP using the v1 login, can't
be used for
zitadel API calls.
- Because we used to store the AMR claim on the auth request, but
internally use the domain.UserAuthMethod type. AMR has no notion of an
IDP login, so that "factor" was lost
during conversion. Rendering those v2 tokens invalid on the zitadel API.
- A wrong check on machine user tokens falsly allowed some tokens to be
valid
- The client ID was set to tokens from client credentials and JWT
profile, which made client queries fail in the validation middleware.
The middleware expects client ID unset for machine users.
# How the Problems Are Solved
Store the domain.AuthMethods directly in the auth requests and session,
instead of using AMR claims with lossy conversion.
- IDPs have seperate auth method, which is not an AMR claim
- Machine users are treated specialy, eg auth methods are not required.
- Do not set the client ID for client credentials and JWT profile
# Additional Changes
Cleaned up mostly unused `oidc.getInfoFromRequest()`.
# Additional Context
- Bugs were introduced in https://github.com/zitadel/zitadel/pull/7822
and not yet part of a release.
- Reported internally.
# Which Problems Are Solved
The session update requires the current session token as argument.
Since this adds extra complexity but no real additional security and
prevents case like magic links, we want to remove this requirement.
We still require the session token on other resouces / endpoints, e.g.
for finalizing the auth request or on idp intents.
# How the Problems Are Solved
- Removed the session token verifier in the Update Session GRPc call.
- Removed the session token from login UI examples session update calls
# Additional Changes
- none
# Additional Context
- Closes#7883
* implement code exchange
* port tokenexchange to v2 tokens
* implement refresh token
* implement client credentials
* implement jwt profile
* implement device token
* cleanup unused code
* fix current unit tests
* add user agent unit test
* unit test domain package
* need refresh token as argument
* test commands create oidc session
* test commands device auth
* fix device auth build error
* implicit for oidc session API
* implement authorize callback handler for legacy implicit mode
* upgrade oidc module to working draft
* add missing auth methods and time
* handle all errors in defer
* do not fail auth request on error
the oauth2 Go client automagically retries on any error. If we fail the auth request on the first error, the next attempt will always fail with the Errors.AuthRequest.NoCode, because the auth request state is already set to failed.
The original error is then already lost and the oauth2 library does not return the original error.
Therefore we should not fail the auth request.
Might be worth discussing and perhaps send a bug report to Oauth2?
* fix code flow tests by explicitly setting code exchanged
* fix unit tests in command package
* return allowed scope from client credential client
* add device auth done reducer
* carry nonce thru session into ID token
* fix token exchange integration tests
* allow project role scope prefix in client credentials client
* gci formatting
* do not return refresh token in client credentials and jwt profile
* check org scope
* solve linting issue on authorize callback error
* end session based on v2 session ID
* use preferred language and user agent ID for v2 access tokens
* pin oidc v3.23.2
* add integration test for jwt profile and client credentials with org scopes
* refresh token v1 to v2
* add user token v2 audit event
* add activity trigger
* cleanup and set panics for unused methods
* use the encrypted code for v1 auth request get by code
* add missing event translation
* fix pipeline errors (hopefully)
* fix another test
* revert pointer usage of preferred language
* solve browser info panic in device auth
* remove duplicate entries in AMRToAuthMethodTypes to prevent future `mfa` claim
* revoke v1 refresh token to prevent reuse
* fix terminate oidc session
* always return a new refresh toke in refresh token grant
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
We are trying to reproduce a few 500 responses we observe on zitadel cloud's token endpoint.
As in the past these were caused by wrongly encoded or encrypted refresh tokens, I created a integration test which tries to reproduce 500 errors by sending invalid refresh tokens.
The added test does not reproduce 500s, all returned errors are in the 400 range as they should. However, as the test is already written, we might as well include them.
Related to #7765
Co-authored-by: Livio Spring <livio.a@gmail.com>
* fix(oidc): roles in userinfo for client credentials token
When tokens were obtained using the client credentials grant,
with audience and role scopes, userinfo would not return the role claims. This had multiple causes:
1. There is no auth request flow, so for legacy userinfo project data was never attached to the token
2. For optimized userinfo, there is no client ID that maps to an application. The client ID for client credentials is the machine user's name. There we can't obtain a project ID. When the project ID remained empty, we always ignored the roleAudience.
This PR fixes situation 2, by always taking the roleAudience into account, even when the projectID is empty. The code responsible for the bug is also refactored to be more readable and understandable, including additional godoc.
The fix only applies to the optimized userinfo code introduced in #7706 and released in v2.50 (currently in RC). Therefore it can't be back-ported to earlier versions.
Fixes#6662
* chore(deps): update all go deps (#7764)
This change updates all go modules, including oidc, a major version of go-jose and the go 1.22 release.
* Revert "chore(deps): update all go deps" (#7772)
Revert "chore(deps): update all go deps (#7764)"
This reverts commit 6893e7d060.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 9ccbbe05bc)
* fix(oidc): roles in userinfo for client credentials token
When tokens were obtained using the client credentials grant,
with audience and role scopes, userinfo would not return the role claims. This had multiple causes:
1. There is no auth request flow, so for legacy userinfo project data was never attached to the token
2. For optimized userinfo, there is no client ID that maps to an application. The client ID for client credentials is the machine user's name. There we can't obtain a project ID. When the project ID remained empty, we always ignored the roleAudience.
This PR fixes situation 2, by always taking the roleAudience into account, even when the projectID is empty. The code responsible for the bug is also refactored to be more readable and understandable, including additional godoc.
The fix only applies to the optimized userinfo code introduced in #7706 and released in v2.50 (currently in RC). Therefore it can't be back-ported to earlier versions.
Fixes#6662
* chore(deps): update all go deps (#7764)
This change updates all go modules, including oidc, a major version of go-jose and the go 1.22 release.
* Revert "chore(deps): update all go deps" (#7772)
Revert "chore(deps): update all go deps (#7764)"
This reverts commit 6893e7d060.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
This fixes the projection of events that have a null audience or scope.
As audience was added in v2.50, legacy events do not have an audience, this made replay of the old events not possible after an upgrade.
(cherry picked from commit be00e3861a)
This fixes the projection of events that have a null audience or scope.
As audience was added in v2.50, legacy events do not have an audience, this made replay of the old events not possible after an upgrade.
* feat(oidc): optimize the userinfo endpoint
* store project ID in the access token
* query for projectID if not in token
* add scope based tests
* Revert "store project ID in the access token"
This reverts commit 5f0262f239.
* query project role assertion
* use project role assertion setting to return roles
* workaround eventual consistency and handle PAT
* do not append empty project id
Tim Möhlmann
