mirror of
https://github.com/mattermost/mattermost.git
synced 2026-11-04 13:16:45 -06:00
85b7d85b57
* MM: Add PropertySyncer API for LDAP/SAML sync of text, select and multiselect attributes Introduces the shared sync engine the enterprise LDAP and SAML packages will call instead of carrying their own text-only logic: - model: PropertyFieldType.SupportsExternalSync, sync source constants and helpers, PropertySync result types, 500-option cap constant. - app: PropertySyncer (NewPropertySyncer, SyncUser, PruneOrphanedOptions) that matches source values to options by exact name, provisions missing options with optimistic concurrency, clears values for absent/empty attributes, and prunes options no live value references. - properties: UpdatePropertyFieldsOpts.ExpectedUpdateAts extends optimistic concurrency to the caller's own read; validation hook now keeps ldap/saml on select/multiselect, accepts empty options on synced fields, locks the option set of a synced field to its sync (colors stay editable), and grandfathers managed=admin so the sync can provision options; read access gives the owning sync the full option list. - store: PropertyValueStore.GetReferencedOptionIDs plus a partial index on PropertyValues(GroupID, FieldID) WHERE DeleteAt = 0. * Address lint findings in PropertySyncer Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Drop the option re-provisioning retry from PropertySyncer value writes The batch upsert already falls back to one write per field so a single bad value cannot block the rest. The extra path that re-read the field and re-provisioned options when a concurrent sync pruned one between provisioning and the write covered a narrow race, was untested, and the next sync run or login corrects it anyway. Removing it drops ~80 lines and an i18n key. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Give linked fields their template's sync source, and only user fields createPropertyField copied the template's ldap/saml attrs onto every field linking to it, whatever its object type, and kept a sync source the caller sent when the template had none. A channel or post field of an attribute synced from AD/LDAP or SAML then carried the link, so the sync lock refused every write to its values -- values no sync ever writes, since an identity source reports attributes of users. A linked field's sync source is now exactly its template's, and only a user field takes it. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Refuse property values for a deleted field UpsertPropertyValues checked that each value's field exists and has the requested object type, but a soft-deleted field passed both checks: GetMany returns deleted rows. A writer holding a field it read before the deletion -- the AD/LDAP sync keeps its fields for a whole run -- would write live values onto it. A deleted field is now reported as not found, like a missing one. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Sync custom profile attributes through their Global Attribute definition Custom profile attributes are now Global Attributes: a template defines the attribute -- its type, its options and the AD/LDAP or SAML attribute it is synced from -- and user, channel and post fields link to it, serving its options. The syncer followed the old shape, a user field holding its own options, and would have provisioned options onto linked fields (which cannot own them through a field write), pruned options a channel's value still references, and missed a sync link an admin added to the template after the user field existed. The syncer now resolves each user field to its definition -- the template it links to, or the field itself -- and takes the external attribute from it. Options are created on and pruned from the definition through the per-option API, which compare-and-swaps every change in the property service; a sync that loses the race re-reads the option list and retries, so the ExpectedUpdateAts extension to UpdatePropertyFields is no longer needed and is removed. An option is orphaned only when no live value of the definition or any field linking to it references it, so GetReferencedOptionIDs takes a set of fields. Also: - A definition naming both sources is synced from AD/LDAP only, the only sync the value lock admits for it; SAML used to fail on every login. - Fields and definitions are read from the master: the LDAP job keeps them for its whole run. - A field deleted during a run is dropped after its first failed write rather than failing for every remaining user. - The legacy custom profile attribute event withholds values of restricted fields, like every other broadcast. - The syncer logs each field outcome itself, at the LDAP levels for the AD/LDAP source, so the enterprise adapters no longer duplicate that logging. - A value_type skip no longer quotes the value, and prune results no longer carry option names; both are attribute values. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Enforce the synced-options rule on templates and the options endpoints Only an attribute's own sync may add, rename or remove its options. The rule was checked on field writes only, and on the field carrying the options -- which for a Global Attribute is the template, where Attribute Management edits them. It now also runs as the PreChangePropertyFieldOptions hook, so the per-option endpoints are no longer a way around it. That hook sees the field but not the change, so it refuses every change but the sync's, a recolor included; recoloring stays possible through the field's option list. The sync writes options through those endpoints now, so it no longer updates fields and the managed=admin exemption added for it is reverted. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Extract i18n strings for the enterprise sync adapters The AD/LDAP and SAML adapters no longer build their own attribute-to-field maps and pass the LDAP entry straight to the shared syncer, so their CPA field-listing and empty-entry errors are gone. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Say what a prune race leaves behind for channel values A value written between the prune's read of references and its removal keeps naming a deleted option. A user's is rewritten by the next sync; a channel's has no sync and stays unset until someone sets it again. Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com> * Drop retired CPA sync strings from the locale catalogs The enterprise AD/LDAP and SAML adapters no longer raise ent.ldap.cpa_field_mapping.list_error, ent.ldap.update_cpa.empty_attribute or ent.saml.cpa_field_mapping.list_error, so en.json dropped them. The 21 locale catalogs still carried their translations, which make i18n-verify rejects as ids en.json no longer has. * Attribute Management: sync select and multiselect attributes from AD/LDAP and SAML The server now syncs text, select and multiselect attributes through the shared PropertySyncer, but the System Console still treated sync as text-only: linking a source converted the attribute to Text, switching to any other type cleared the links, and the link modal and the AD/LDAP/SAML settings section warned that everything else would be converted. - supportsExternalSync mirrors PropertyFieldType.SupportsExternalSync. Linking keeps a type it accepts (phone, URL and email included, which are text on the server); only a type it rejects is converted to Text, and only a switch to one clears the links. - A synced select or multiselect shows its options read-only, with a note naming the source that manages them, and no longer requires an option before Save, since the sync provisions them. - Saving an attribute that was already synced leaves its options out of the patch: the server refuses anyone but the sync adding, renaming or removing one, and a list loaded before the sync added an option would read as removing it. Linking for the first time still saves the options the admin seeded. - Adding a link is no longer locked by an Applies-to resource when the type would not change. - The picker tells a Type switch from a chip removal by whether the type changed, not by the type being Text. - The AD/LDAP/SAML "User attributes sync" section patches only the link for a syncable type instead of converting it to Text and resending every attr, keeps linked select/multiselect fields editable, and links to Attribute Management. * Add OpenID Connect as a user attribute sync source A user attribute can now be linked to an OpenID Connect claim through attrs.openid, next to attrs.ldap and attrs.saml. The openid source has its own sync caller (system:openid_sync) and the same sync lock: only that sync may write the attribute's values. The sources are one ordered list (PropertySyncSources), which the sync source helpers, the sync lock, the owner check, the linked-field copy and the validation hook now iterate instead of naming ldap and saml, so the next source is a one-line change. Behaviour for AD/LDAP and SAML is unchanged. An OpenID Connect link is exclusive: a definition naming openid may not also name ldap or saml. OpenID Connect users never go through the AD/LDAP or SAML sync, so under the AD/LDAP-first precedence such a field would be locked against the OpenID Connect sync and never written for them. FeatureFlags.OpenIdAttributeSync (default on) is a kill switch: while it is off, a new or changed openid link is refused, and an existing one rides along untouched. sanitizeAndValidateFieldAttrs now takes the existing field instead of three values derived from it, so the rules that compare against the previous state read it the same way. * Sync user attributes from OpenID Connect claims at sign-in AuthorizeOAuthUser now returns an OAuthAuthorization carrying the raw ID token alongside the userinfo response, which used to be dropped once the provider had built a user from it. CompleteOAuth reads the userinfo response once, and after the sign-in, sign-up or switch to OpenID Connect has resolved the user, and before the session exists, syncs the user attributes linked to OpenID Connect claims through PropertySyncer. A switch from OpenID Connect to email does not sync. The sync runs only for the generic OpenID Connect service, with the OpenIdAttributeSync flag on and a license with the OpenId feature at Enterprise tier or above. The claims come from a new optional provider interface, einterfaces.OAuthClaimsProvider; the provider verifies the ID token and merges it with the userinfo response. A sync that cannot run, or whose claims cannot be trusted, is logged and never fails the sign-in or changes stored values. OpenIdSettings.AdditionalScopes (through SSOSettings) adds scopes to the authorization request, for providers that release some claims only under them. It is validated as an RFC 6749 scope list of at most 1024 characters. * Add an end-to-end spec for OpenID Connect attribute sync The mattermost-openid client of both Keycloak realm exports (dev and Playwright) now releases three user attributes as claims: department as the nested org.department, clearances as an array, and employeeId as employee_id in the ID token only. The Playwright library can give a Keycloak user attributes, replace them (setKeycloakUserAttributes), and configure OpenID Connect from the discovery document (ensureKeycloakOpenId({discovery: true})), which the server needs to verify ID tokens. With the realm frontend URL set, Keycloak names the browser-facing URL as issuer and authorization endpoint and the requesting alias for the token, userinfo and JWKS endpoints, so the split the explicit endpoints set up still holds. The alias is added to the allowed untrusted internal connections, since the server fetches the document and the keys from it. The spec signs a user in through Keycloak, checks that the nested, array and ID-token-only claims filled their attributes, that a membership policy on the synced department admits the user, and that removing the department at Keycloak clears it at the next sign-in and the policy then removes the user. * Document syncing user attributes from OpenID Connect claims The OpenID Connect guide gains a section on linking user attributes to claims: discovery is required for ID token claims, how claim names and dot paths resolve, how values are stored, that a missing claim clears the attribute at the next sign-in, and provider notes (Keycloak user-editable attributes, Entra ID group overage, scoped claims). The authentication settings reference documents Additional scopes, and the user attributes page points to the new section. * OpenID Connect attribute sync: link user attributes to OpenID Connect claims An admin can now link a text, select or multiselect user attribute to an OpenID Connect claim (attrs.openid), next to AD/LDAP and SAML. - Attribute Management: the external-source helpers move from positional ldap/saml arguments to a map keyed by source, shared with PROPERTY_SYNC_SOURCES in property_utils. The link picker offers OpenID Connect with its own modal copy. It is listed but disabled, with the reason inline and in a tooltip beside the entry, when the OpenIdAttributeSync flag is off, the license lacks OpenID Connect, or OpenID Connect sign-in is off. Existing links keep their chips. - OpenID Connect is exclusive with AD/LDAP and SAML in both directions, matching app.property_field.sync_source_exclusive.app_error on the server; AD/LDAP plus SAML stays allowed. - Saves send attrs.openid like the other sources, null to unlink, and cascade it onto the linked Users field. - The Source column lists every linked source instead of a fixed "AD/LDAP, SAML" label; Managed by, the system user detail chips and read-only state, isPropertyFieldSourceManaged, the ABAC editor's hasControlledAttributeValues and the attribute selector read openid too. - The OpenID Connect settings page gets the shared "User attributes sync" section and an Additional Scopes setting (OpenIdSettings.AdditionalScopes). The section now reads the disabled prop its settings pages pass, so a read-only admin can no longer edit it. - The profile editor and the hierarchical attribute editor lock an attribute synced from the source the user signs in with, openid included. - The retired table source labels are dropped from en.json and, as check_icu.mjs requires, from the 21 locale catalogs. Test fixtures that set attrs.ldap to a boolean now use a mapping string, the shape the server stores. * System user detail: name OpenID Connect and hide chip removal on locked multiselects The authentication method of an OpenID Connect user read "Openid", and of a GitLab user "Gitlab", from title-casing the auth service. They now read "OpenID Connect" and "GitLab". A multiselect attribute the admin can't edit (synced from an identity source, or otherwise locked) still showed a remove button on each value, which did nothing. A disabled select now leaves them out. Both found in a browser pass against a server syncing attributes from Keycloak over OpenID Connect. * Profile settings: name OpenID Connect as the login of an OpenID Connect user The collapsed Email row names the login provider and address for GitLab, Google, Entra ID, AD/LDAP and SAML users, but had no case for OpenID Connect, so it was empty. It now reads "Login done through OpenID Connect (email)", like the others. Found in a browser pass with a Keycloak OpenID Connect user. * List the new OpenID Connect settings in config fixtures The support packet golden config and the Playwright default config spell out every setting, so they need SSOSettings.AdditionalScopes (GitLab, Google and OpenID Connect settings) and FeatureFlags.OpenIdAttributeSync (the golden only). CI failed TestSupportPacketMarshalGolden and the Playwright type check without them. * Attribute Management: hold Save on a synced select until it applies to Users The sync provisions a select or multiselect's options from the values it writes to users, and the server lets nothing but the sync change the options of a synced attribute. A synced select that applies only to Channels or Posts was therefore left with an option list nobody could add to. Save now waits, with an inline note, until the attribute applies to Users or the link is removed, including after Users is removed from an attribute that was already synced. The locked Type control's accessible name also announces the type the attribute keeps, instead of always saying Text, now that linking a select or multiselect no longer converts it. * Update the E2E specs for select and multiselect sync The AD/LDAP and SAML settings page and the Attribute Management form specs still expected linking a select to convert it to Text. They now check that a linked select keeps its type and saves only its link, that a type sync cannot populate (Ranked) is still disabled or converted with a warning, and that a synced select holds Save until it applies to Users. * Docs: document select and multiselect sync in Attribute Management The Attribute Management page still said linking an external source converts an attribute to Text. Describe what linking does from v12.1: Select and Multiselect keep their type and get their options from the sync, which needs the attribute to apply to Users; a type kept by the link can be linked while the attribute applies to a resource; and a value the identity provider stops sending is removed at the next sync. * E2E: retry the Keycloak realm frontend URL update a parallel worker refused Tests that point the server at Keycloak each set the realm's frontend URL on first use. When several run at once against a fresh realm, Keycloak refuses all but one of the concurrent realm updates with 409 and the losing tests failed. A refused update is now retried against a fresh read, which finds the URL the winner set. * E2E: expect the disabled OpenID Connect source in Attribute Management With OpenID Connect sync, the external source menu lists OpenID Connect, disabled with "Can't be combined with ..." while AD/LDAP or SAML is linked. That label names AD/LDAP, so the unanchored /AD\/LDAP/ locator matched it, and once both AD/LDAP and SAML are linked the trigger stays to show it. Anchor the locators and check that only the disabled OpenID Connect item is left. * Give the external-source Type lock label a new message id Its text gained the {value} placeholder, which every existing translation of the old id lacks, so the translation check rejected them all. The old id is dropped from the locale catalogs, where it would otherwise be an extra key. * Name OpenID Connect in the external-source Type lock label's description * Fix a race in TestCheckLdapUserPasswordConcurrency's LDAP mock Each goroutine registered the DoLogin expectation itself. testify adds an expectation with On() and sets its return values in Return(), so a goroutine calling DoLogin in between matched an expectation with no return values and panicked, failing the Postgres shard that ran it. The expectation is now registered once, before the attempts start. The check also no longer dereferences an attempt's error before checking it is set. * Docs: resolve the OpenID Connect link in Additional scopes through useBaseUrl The docs build rejects a site-absolute href in raw HTML, since Docusaurus does not apply baseUrl to it. * Let a field's own sync clear it when required, and keep a sync off unsynced options A synced user field marked required (possible through the API) refused the sync's clearing write, so the user kept a value the directory no longer has. The field's own sync may now clear it: the source decides the value, including that there is none. A user field linked to a template it has no copy of the link of is judged by the template's link. The options endpoints let any caller change the options of a field that is not synced, so a sync job that loaded the field before an admin unlinked it could still add or remove its options. A sync now changes only the options of a field synced from its own source. * Clear a synced value the source reports but that cannot be stored When the sync could not store a user's new value (too long, past the option cap, or its option could not be created), it skipped the field and kept the old value, so someone who moved out of Finance could still pass a Finance-only policy. The stored value is now cleared instead, as for an attribute the source no longer reports; the outcome still says skipped or error, and why. A failure caused by the attribute being unlinked while the syncer ran is the exception: the field is the admin's again, so its value is left alone and the syncer stops syncing it. For the same reason the prune now skips a definition that is no longer synced from its source, instead of deleting options the admin added after unlinking it. * Regenerate the Playwright default feature flags for OpenIdAttributeSync The file is generated from the server's feature flag defaults since #39091, and the OpenIdAttributeSync flag this branch adds was missing from it, failing the generated files check. * Fix the OpenID Connect attribute sync note and the linking sentence in docs The note on syncing attributes from OpenID Connect claims rendered as one paragraph with a literal "-", since its list had been collapsed onto one line. Restore the four bullets. Attribute Management also said AD/LDAP and SAML can be linked together without saying OpenID Connect can't join them. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ibrahim Serdar Acikgoz <isacikgoz@users.noreply.github.com>