f54f3f765d
* ci(e2e): triage red Playwright and Cypress runs against test history Each PR lane's status is published by the toolkit's e2e-status action, and a red run is triaged by e2e-triage: flaky or broken on master, the same failure on other PRs, environment errors, or caused by this PR. - e2e-tests-lane-result.yml: one reusable workflow for both Cypress and Playwright that publishes the lane status, triages, refreshes the e2e-test/triage summary, posts the webhook and runs the pass/fail gate. - .github/e2e-status-lanes.json: the one list of PR lanes (enterprise, fips, subpath) that feed e2e-test/triage, read from the workflow's own commit. Master and release contexts carry a suffix and are triaged per lane. - The verified label and the override cover every listed lane. - use-staging picks the TSIO instance, like the TSIO actions. - The subpath lane has its own concurrency key, so it never cancels the enterprise lane. - test_e2e_status_workflows.py checks this wiring. Toolkit pinned to bdce26e. * ci(e2e): pin the toolkit to 24bf208; use the renamed e2e-test-* actions The toolkit renamed e2e-triage to e2e-test-triage and e2e-status to e2e-test-status; inputs are unchanged. Triage now holds a failure repeated on this PR's last commits and never on the base branch as out of date (merge the base branch and re-run) instead of clearing it, and the status line names each kind: flaky, broken on main, caused by PR, out of date, needs investigation. * ci(e2e): pin the toolkit to main (419b0a9) Toolkit #5 merged; this pins its merge on main instead of the PR branch. * ci(e2e): pin the toolkit to main (b3ca677) Toolkit #6 merged: a run flagged as a whole now says why once ("mostly environment errors, re-run" or "many failures in one run") instead of repeating "investigation required". * ci(e2e): re-trigger CI on the update-branch merge head The merge commit created by /update-branch (7040758, authored by Mattermost Build) never received the external Enterprise CI/branch or Enterprise CI/docker-image statuses, so Argo never dispatched e2e-tests-ci.yml for this head. An empty commit re-fires the push/synchronize triggers without changing any file. Co-authored-by: yasser khan <attitude3cena.yf@gmail.com> * ci(e2e): pin the toolkit to e2e-triage-judge-flagged-runs (f1f5d54, toolkit#8) * ci(e2e): pin the toolkit to e2e-triage-judge-flagged-runs (77c3048, toolkit#8) * ci(e2e): retry image pulls before starting the E2E server A registry outage (quay.io answering 503 for keycloak) failed three Cypress shards at create, before any test ran. Pull with the existing retry helper first: 5 tries, 30 seconds apart. * ci(e2e): use the toolkit's main while triage is observed toolkit#8 merged (b7757e9). Following review, consumers track main so toolkit changes need no bump here; pin a hash again once stable. * ci(e2e): re-trigger CI; Enterprise CI did not build 54a0e2f's image * ci(e2e): re-trigger CI again; Enterprise CI silent since54a0e2fHeads54a0e2fand80b876creceived no cla/mattermost, Enterprise CI/branch or Enterprise CI/docker-image statuses, so Argo never dispatched e2e-tests-ci.yml for them. Enterprise CI is posting normally for other PRs. Empty commit to re-fire the push triggers; no file changes. Co-authored-by: yasser khan <attitude3cena.yf@gmail.com> --------- Co-authored-by: Mattermost Build <build@mattermost.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
34 KiB
E2E Test Pipelines
Three automated E2E test pipelines cover different stages of the development lifecycle.
Pipelines
| Pipeline | Trigger | Editions Tested | Image Source |
|---|---|---|---|
PR (e2e-tests-ci.yml) |
Argo Events on Enterprise CI/docker-image status |
enterprise | mattermostdevelopment/** |
Merge to master/release (e2e-tests-on-merge.yml) |
Platform delivery after docker build (delivery-platform/.github/workflows/mattermost-platform-delivery.yaml) |
enterprise, fips | mattermostdevelopment/** |
Release cut (e2e-tests-on-release.yml) |
Platform release after docker build (delivery-platform/.github/workflows/release-mattermost-platform.yml) |
enterprise, fips, team (future) | mattermost/** |
All pipelines follow the smoke-then-full pattern: smoke tests run first, full tests only run if smoke passes.
Workflow Files
.github/workflows/
├── e2e-tests-ci.yml # PR orchestrator
├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches)
├── e2e-tests-on-release.yml # Release cut orchestrator
├── e2e-tests-cypress.yml # Shared wrapper: calls the cypress template
├── e2e-tests-playwright.yml # Shared wrapper: calls the playwright template
├── e2e-tests-playwright-rolling-upgrades.yml # resolves the from-version matrix, calls the template per version
├── e2e-tests-cypress-template.yml # cypress + test-system-io dispatch
├── e2e-tests-playwright-template.yml # playwright + test-system-io dispatch
├── e2e-tests-lane-result.yml # status, triage, webhook and gate for one lane (called last by both templates)
└── e2e-tests-playwright-rolling-upgrades-template.yml # one from-version: upgrade-from/to, then dispatch on the upgraded server
Call hierarchy
e2e-tests-ci.yml ─────────────────┐
e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ────► e2e-tests-cypress-template.yml
e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ─► e2e-tests-playwright-template.yml
│ └► e2e-tests-playwright-rolling-upgrades.yml
│ └► e2e-tests-playwright-rolling-upgrades-template.yml
│ (once per resolved from-version)
e2e-tests-playwright-rolling-upgrades.yml is also directly runnable from the Actions UI, so it has no caller in that case.
Workflow Architecture
The template splits into five jobs — prepare-run, prep-deps, dispatch-begin, workers (matrix), and report — and pushes spec-level execution to Test System IO so workers stay thin and identical.
┌──────────────────────────────────────────────────────────────────────────┐
│ Template: e2e-tests-{cypress,playwright}-template.yml │
│ │
│ ┌───────────────────┐ ┌──────────────────────────────┐ │
│ │ prepare-run │ │ prep-deps │ │
│ │ (1 runner) │ parallel │ (1 runner) │ │
│ │ │ ◄────────────► │ │ │
│ │ • build workers │ │ Cypress: │ │
│ │ matrix [1..N] │ │ • cypress/node_modules │ │
│ │ • compute commit │ │ • ~/.cache/Cypress (binary)│ │
│ │ status context │ │ │ │
│ │ • emit composite │ │ Playwright: │ │
│ │ identity │ │ • webapp/platform/{client, │ │
│ │ │ │ types}/{lib,node_mod} │ │
│ │ │ │ • playwright/node_modules │ │
│ │ │ │ • playwright/lib/dist │ │
│ │ │ │ • ~/.cache/ms-playwright │ │
│ │ │ │ (chromium only) │ │
│ │ │ │ │ │
│ │ │ │ → saved to actions/cache │ │
│ └─────────┬─────────┘ └───────────────┬──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────────────────────┐ │
│ │ │ dispatch-begin │ │
│ │ │ • register run with │ │
│ │ │ Test System IO │ │
│ │ │ • runs immediately before │ │
│ │ │ workers to minimise the │ │
│ │ │ inactivity-timeout window │ │
│ │ └───────────────┬──────────────┘ │
│ │ │ │
│ └────────────────────┬─────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ workers (matrix, fail-fast: false) │ │
│ │ Cypress full: N=40 | Playwright full: N=10 │ │
│ │ │ │
│ │ each worker, in parallel: │ │
│ │ 1. sparse-checkout actions + full checkout-repo │ │
│ │ 2. setup-node │ │
│ │ 3. restore caches ◄─── actions/cache (from prep-deps) │ │
│ │ (fail-on-cache-miss: true) │ │
│ │ 4. cloud-init + start-server (docker compose stack) │ │
│ │ 5. prepare-cypress | prepare-playwright (run setup project) │ │
│ │ 6. dispatch-run ──────────────────────────────────┐ │ │
│ │ (pulls specs from Test System IO, runs locally, │ │ │
│ │ posts result, loops until queue is empty) │ │ │
│ │ 7. cloud-teardown │ │ │
│ └────────────────────┬────────────────────────────────────┼───────┘ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────┐ │ │
│ │ report │ │ │
│ │ • pull aggregated results from Test System IO │ ◄────┘ │
│ │ • post commit status │ │
│ │ • send webhook notification │ │
│ └─────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┬─┘
│
┌───────────────────────────▼───┐
│ Test System IO (external) │
│ • spec-level dispatch │
│ • result aggregation │
│ • retry orchestration │
└───────────────────────────────┘
Key properties
- Spec-level vs. job-level parallelism. The matrix sizes the runner pool; Test System IO does the spec assignment. Slow specs don't block a worker — fast workers keep pulling the next spec from the queue.
- Cache-only workers.
prep-depsinstalls once per workflow run and saves toactions/cache. Every worker restores withfail-on-cache-miss: trueand runs zeronpm ci. Eliminates the 40-wayEEXIST/ENOENTrace in npm's shared cacache writer. - dispatch-begin runs late. It depends on
prep-depsso the gap between Test System IO run registration and the first worker callingdispatch-runis just per-worker setup (~3–5 min). Registering earlier risks the run timing out before any worker checks in, bulk-failing every spec. - Playwright slim slice. Playwright only consumes
@mattermost/clientand@mattermost/typesfrom webapp, so prep-deps caches just those two packages' builtlib/andnode_modules(~10–30 MB) instead of the fullwebapp/node_modulestree (~1–2 GB). - Browser/binary caches. Cypress caches
~/.cache/Cypress(cypress binary lives outside node_modules); playwright caches~/.cache/ms-playwright(chromium only). Both keyed on the framework's lockfile so they invalidate on version bumps. - No retry plumbing in the template. Test System IO handles per-spec retries; the workflow only sees aggregated results.
Playwright rolling upgrades (separate from full suite)
Two workflows, split on the question each answers:
e2e-tests-playwright-rolling-upgrades.yml— which versions to upgrade from. Resolves the matrix viascript/resolve_upgrade_matrix.mjs(last 3 minors + active ESR; ESR entries carryisESR: trueand acontextLabelending in-esr), then calls the template once per entry in astrategy.matrixof reusable-workflow invocations.e2e-tests-playwright-rolling-upgrades-template.yml— how to test one version, given two images:upgrade_from_server_image(that entry'sdockerTag) andSERVER_IMAGE(the to-image). Same job shape ase2e-tests-playwright-template.yml—prepare-run→prep-deps→dispatch-begin→workers→report— with the harness prepended to each worker.
Per worker, within one from-version:
upgrade-from→upgrade-todispatch-runagainst the just-upgraded server
So the upgraded server does not stop at the harness — it goes on to run the normal suite, which is the coverage this pipeline exists for. Nothing sits between the two: no --project=setup, no re-patching, no restart, since an upgrade that needed the suite's setup re-run before the server was usable would not be a passing upgrade. The upgrade specs never enter the queue themselves: upgrade-specs/ sits outside the Playwright testDir, so dispatch-begin cannot enqueue them.
Statuses match the matrix. A 4-entry matrix posts 4 contexts, e2e-test/playwright-full/{edition}/upgrade-from-{contextLabel}, each covering that version's harness and its post-upgrade suite — posted pending by its dispatch-begin and resolved by its report. There is no aggregate context. An empty matrix is the exception: the workflow posts a single upgrade-from-none context and never invokes the template.
Each worker upgrades its own server (a server cannot be shared across runners), so the harness runs workers times per from-version and the pipeline costs matrix size × workers runners. workers defaults to 20, matching the full suite, and is an input; workers get 60m (vs the full suite's 30m) to cover the harness's two image pulls and swap on top of their share of the suite.
There is deliberately no testcontainers:down first — the runner is fresh, so upgrade-from's clean-slate guard has nothing to trip over. PW_TESTCONTAINERS_REUSE=true is what lets upgrade-to adopt the stack, survive the swap, and keep every per-spec dispatch on the same upgraded server. Step 2 is skipped when the harness fails; that worker then never reports, the run closes short of total-reports-expected, and the version's context goes red.
Commit status description — written by the Test System IO summary, same shape as the full suite (image_tag: is the target/to image). The from-version needs no description field of its own: it is already in the context name, one context per version.
When it runs. e2e-tests-ci.yml's check-changes turns it on for a PR whose diff touches e2e-tests/playwright/upgrade-specs/, e2e-tests/playwright/script/resolve_upgrade_matrix.mjs, either e2e-tests-playwright-rolling-upgrades* workflow, or a server migration (server/channels/db/migrations/, server/config/migrations/, server/channels/app/migrations.go, server/channels/app/permissions_migrations.go, server/public/model/version.go). Everything else — including e2e-tests/playwright/lib/ and playwright.config.ts — is unmatched and needs the manual dispatch's opt-in checkbox to run within this workflow. That decision is the only gate: unlike the full suite, this pipeline does not also consult should_run, because should_run matches ^e2e-tests/.*\.(ts|tsx|js|jsx)$ and would veto a run whose only change is the .mjs matrix resolver. Consequently there is no "skipped" status — when it is not requested, nothing is posted.
e2e-tests-on-merge.yml mirrors this same gate on merges to master/release-*: its own check-changes job diffs the merge commit against its parent (git diff HEAD~1 HEAD, since there is no PR base to diff against) and applies the identical harness/migration conditions above (it has no manual opt-in checkbox, unlike the PR pipeline). Release cut (e2e-tests-on-release.yml) stays unconditional — no gate, always true.
Failed rolling-upgrade contexts are included when applying E2E Tests/verified or the override-status workflow (discovered by pattern e2e-test/playwright-full/{edition}/upgrade-from-*, same principle as full-suite contexts).
This pipeline is invoked from e2e-tests-playwright.yml when run_rolling_upgrades: "true". It is not embedded in e2e-tests-playwright-template.yml.
| Trigger | run_rolling_upgrades |
|---|---|
| PR (Argo / automated) | false (default) |
PR (workflow_dispatch, manual) |
Opt-in: Run rolling upgrades on e2e-tests-ci.yml |
Merge to master / release-* |
Auto — same file-change gate as PR (touches harness/migrations), evaluated against the merge commit's parent |
| Release cut | true |
| Ad-hoc | n/a — Run workflow directly on e2e-tests-playwright-rolling-upgrades.yml |
e2e-tests-playwright-rolling-upgrades.yml carries its own workflow_dispatch so the coverage can be run from the Actions UI without a PR: commit_sha defaults to the selected ref's HEAD, server_image_tag to master, and the license / webhook secrets fall back to the repo secrets the calling workflows would have forwarded.
When resolve_upgrade_matrix.mjs returns [] (no supported from-versions), the entry workflow posts success on e2e-test/playwright-full/{edition}/upgrade-from-none and never invokes the template.
Pipeline 1: PR (e2e-tests-ci.yml)
Runs E2E tests for every PR commit after the enterprise docker image is built. Fails if the commit is not associated with an open PR.
Trigger chain:
PR commit ─► Enterprise CI builds docker image
─► Argo Events detects "Enterprise CI/docker-image" status
─► dispatches e2e-tests-ci.yml
For PRs from forks, body.branches may be empty so the workflow falls back to master for workflow files (trusted code), while commit_sha still points to the fork's commit.
Jobs: 2 (cypress + playwright), each does smoke -> full
Commit statuses (4 total):
| Context | Description (pending) | Description (result) |
|---|---|---|
e2e-test/cypress-smoke|enterprise |
tests running, image_tag:abc1234 |
100% passed (1313), 440 specs, image_tag:abc1234 |
e2e-test/cypress-full|enterprise |
tests running, image_tag:abc1234 |
100% passed (1313), 440 specs, image_tag:abc1234 |
e2e-test/playwright-smoke|enterprise |
tests running, image_tag:abc1234 |
100% passed (200), 50 specs, image_tag:abc1234 |
e2e-test/playwright-full|enterprise |
tests running, image_tag:abc1234 |
99.5% passed (199/200), 1 failed, 50 specs, image_tag:abc1234 |
Manual trigger (CLI):
gh workflow run e2e-tests-ci.yml \
--repo mattermost/mattermost \
--field pr_number="35171"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (smoke-then-full)
- Click Run workflow
- Fill in
pr_number(e.g.,35171) - Click Run workflow
On-demand testing
For on-demand E2E testing, the existing triggers still work:
- Comment triggers:
/e2e-test,/e2e-test fips, or withMM_ENVparameters - Label trigger:
E2E/Run
These are separate from the automated workflow and can be used for custom test configurations or re-runs.
Pipeline 2: Merge (e2e-tests-on-merge.yml)
Runs E2E tests after every push/merge to master or release-* branches.
Trigger chain:
Push to master/release-*
─► Argo Events (mattermost-platform-package sensor)
─► delivery-platform/.github/workflows/mattermost-platform-delivery.yaml
─► builds docker images (enterprise + fips)
─► trigger-e2e-tests job dispatches e2e-tests-on-merge.yml
Jobs: 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only, plus a check-changes job that conditionally forwards run_rolling_upgrades to the two playwright jobs (see Playwright rolling upgrades above).
Commit statuses (4 total, +1 per rolling-upgrade from-version when triggered):
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:abc1234_def5678 |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:abc1234_def5678 |
e2e-test/playwright-full|enterprise |
100% passed (200), 50 specs, image_tag:abc1234_def5678 |
e2e-test/playwright-full|fips |
100% passed (200), 50 specs, image_tag:abc1234_def5678 |
Manual trigger (CLI):
# For master
gh workflow run e2e-tests-on-merge.yml \
--repo mattermost/mattermost \
--field branch="master" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="<image_tag>"
# For release branch
gh workflow run e2e-tests-on-merge.yml \
--repo mattermost/mattermost \
--field branch="release-11.4" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="<image_tag>"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (master/release - merge)
- Click Run workflow
- Fill in:
branch:masterorrelease-11.4commit_sha: full 40-char SHAserver_image_tag: e.g.,abc1234_def5678
- Click Run workflow
Pipeline 3: Release Cut (e2e-tests-on-release.yml)
Runs E2E tests after a release cut against the published release images.
Trigger chain:
Manual release cut
─► delivery-platform/.github/workflows/release-mattermost-platform.yml
─► builds and publishes release docker images
─► trigger-e2e-tests job dispatches e2e-tests-on-release.yml
Jobs: 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only. Team edition planned for future.
Commit statuses (4 total, 6 when team is enabled):
Descriptions include alias tags showing which rolling docker tags point to the same image.
RC example (11.4.0-rc3):
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
e2e-test/cypress-full|team (future) |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
Stable example (11.4.0) — includes MAJOR.MINOR alias:
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
e2e-test/cypress-full|team (future) |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
Manual trigger (CLI):
gh workflow run e2e-tests-on-release.yml \
--repo mattermost/mattermost \
--field branch="release-11.4" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="11.4.0" \
--field server_image_aliases="release-11.4, release-11, 11.4"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (release cut)
- Click Run workflow
- Fill in:
branch:release-11.4commit_sha: full 40-char SHAserver_image_tag: e.g.,11.4.0or11.4.0-rc3server_image_aliases: e.g.,release-11.4, release-11, 11.4(optional)
- Click Run workflow
Commit Status Format
Context name: e2e-test/<phase>|<edition>
Where <phase> is cypress-smoke, cypress-full, playwright-smoke, or playwright-full.
Description format:
-
All passed:
100% passed (<count>), <specs> specs, image_tag:<tag>[ (<aliases>)] -
With failures:
<rate>% passed (<passed>/<total>), <failed> failed, <specs> specs, image_tag:<tag>[ (<aliases>)] -
Pending:
tests running, image_tag:<tag>[ (<aliases>)] -
Rolling upgrades: identical shape — the from-version lives in the context (
…/upgrade-from-release-11.9), one per resolved version, not in the description -
Pass rate:
100%if all pass, otherwise one decimal (e.g.,99.5%) -
Aliases only present for release cuts
-
image_tag:is the target/to server image. Rolling upgrades put the from-version in the context name only, not in the description.
Failure behavior
- Smoke test fails: Full tests are skipped, only smoke commit status shows failure
- Full test fails: Full commit status shows failure with pass rate
- Both pass: Both smoke and full commit statuses show success
- No PR found (PR pipeline only): Workflow fails immediately
Smoke-then-Full Pattern
Each wrapper (Cypress/Playwright) follows this flow:
generate-build-variables (branch, build_id, server_image)
─► smoke tests (1 worker, minimal docker services)
─► if smoke passes ─► full tests (20 workers cypress / 1 worker playwright, all docker services)
─► report (aggregate results)
─► lane result (commit status, triage, webhook, pass/fail gate; e2e-tests-lane-result.yml)
Test filtering
| Framework | Smoke | Full |
|---|---|---|
| Cypress | --stage=@prod --group=@smoke |
--stage="@prod" --excludeGroup="@te_only,@cloud_only,@high_availability" --sortFirst=... --sortLast=... |
| Playwright | --grep @smoke |
--grep-invert "@smoke|@visual" |
Worker configuration
| Framework | Smoke Workers | Full Workers |
|---|---|---|
| Cypress | 1 | 20 |
| Playwright | 1 | 1 (uses internal parallelism via PW_WORKERS) |
Docker services
| Test Phase | Docker Services |
|---|---|
| Smoke | postgres inbucket |
| Full | postgres inbucket minio openldap elasticsearch keycloak |
Tagging Smoke Tests
Cypress
Add @smoke to the Group comment at the top of spec files:
// Stage: @prod
// Group: @channels @messaging @smoke
Playwright
Add @smoke to the test tag option:
test('critical login flow', {tag: ['@smoke', '@login']}, async ({pw}) => {
// ...
});
Shared Wrapper Inputs
The wrappers (e2e-tests-cypress.yml, e2e-tests-playwright.yml) accept these inputs:
| Input | Default | Description |
|---|---|---|
server_edition |
enterprise |
Edition: enterprise, fips, or team |
server_image_repo |
mattermostdevelopment |
Docker namespace: mattermostdevelopment or mattermost |
server_image_tag |
derived from commit_sha |
Docker image tag |
server_image_aliases |
(empty) | Alias tags shown in commit status description |
ref_branch |
(empty) | Source branch name for webhook messages (e.g., master or release-11.4) |
The automation dashboard branch name is derived from context:
- PR:
pr-<pr_number>(e.g.,pr-35205) - Master merge:
master - Release merge:
release-<version>(e.g.,release-11.7) - Fallback:
commit-<short_sha>(e.g.,commit-abc1234)
Master and release runs use the real branch verbatim so the
/reports/{repo}/{branch} view aggregates every build on that branch
into one history. PR runs use a pr-<n> token because there's no real
branch to use; the commit-only fallback uses a commit-<short_sha>
token (defensive — no current caller hits this path).
The test type suffix (-smoke or -full) is appended by the template.
The server image is derived as:
{server_image_repo}/{edition_image_name}:{server_image_tag}
Where edition_image_name maps to:
enterprise->mattermost-enterprise-editionfips->mattermost-enterprise-fips-editionteam->mattermost-team-edition
Webhook Message Format
After full tests complete, a webhook notification is sent to the configured REPORT_WEBHOOK_URL. The results line uses the same commit_status_message as the GitHub commit status. The source line varies by pipeline using report_type and ref_branch.
Report types: PR, MASTER, RELEASE, RELEASE_CUT
PR
:open-pull-request: mattermost-pr-35205
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234
100% passed (1313), 440 specs | full report
Merge to master
:git_merge: abc1234 on master
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
100% passed (1313), 440 specs | full report
Merge to release branch
:git_merge: abc1234 on release-11.4
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
100% passed (1313), 440 specs | full report
Release cut
:github_round: abc1234 on release-11.4
:docker: mattermost/mattermost-enterprise-edition:11.4.0-rc3
100% passed (1313), 440 specs | full report
The commit short SHA links to the commit on GitHub. The PR number links to the pull request.
Related Files
e2e-tests/cypress/- Cypress test suitee2e-tests/playwright/- Playwright test suitee2e-tests/.ci/- CI configuration and environment filese2e-tests/Makefile- Makefile with targets for running tests, generating cycles, and reporting
Status ownership across dispatch paths
Cypress and Playwright serialize each lane by resolved PR, suite, edition and
report type, independent of the workflow dispatch ref. Status publishers also
share e2e-status-publish-<full tested SHA> with cancel-in-progress: false and
queue: max, because statuses and the cross-lane summary belong to a commit.
The toolkit e2e-status action claims pending before preparation and guards all
raw results, skips and triage verdicts by run ID and publishing attempt. The
TSIO actions collect diagnostics with their status side effects disabled.
Cancelled runs retain diagnostic summaries/artifacts and cannot publish.
Missing current reports remain red, and a later pending/raw result refreshes
the aggregate instead of leaving an old all-green snapshot.
The orchestration summary can pass before workers finish uploading reports. Before publishing success, the owned writer checks the exact run/attempt's complete, nonempty report group on the same TSIO environment as dispatch. Preparation, dispatch and all workers must also succeed. Failed execution or missing/unreadable reports stay red even if triage returns SUCCESS; triage runs in report-only mode for these cases. The final job assertion uses these same gates, a successful triage step with a confirmed status write, and triage's effective mode after applying the PR's report-only label.
Lane result and the lane list
e2e-tests-lane-result.yml is the one place a lane publishes its result. The Cypress and
Playwright templates call it after report, with the lane's identity, execution_success
(preparation, dispatch and every worker succeeded), the summary outcome and the report URL. It
holds the publication lock, publishes the owned status, triages a red run, refreshes the
aggregate, posts the webhook, and then asserts the gate. The gate is unchanged: a missing or
incomplete report is always red, and triage can only clear a failure in enforce mode.
The PR lanes that feed the e2e-test/triage aggregate are listed once, in
.github/e2e-status-lanes.json. Every writer checks out .github at the workflow's own ref (github.sha) into
.e2e-trusted and runs .e2e-trusted/.github/actions/e2e-status-lanes, which reads
the list beside it. The list therefore never comes from the tested commit or the
PR head, and a malformed list fails the job. The toolkit
refreshes the aggregate only for a context in that list.
| Context | Published by | Feeds the aggregate and triage |
|---|---|---|
e2e-test/{cypress,playwright}-full/enterprise (PR) |
template claim and lane result; skip path | yes |
e2e-test/playwright-full/enterprise/subpath (PR) |
playwright template with server_deploy: subpath |
yes, when should_run_subpath |
e2e-test/{cypress,playwright}-full/fips (PR) |
template claim and lane result; skip path | yes, when should_run_fips |
e2e-test/playwright-full/{edition}/upgrade-from-<label> |
rolling-upgrades template through TSIO dispatch-begin and summary, with no lock and no owner check | no. Labels come from the upgrade matrix, so a static list cannot cover them; override-status finds them by pattern |
…/master, …/release, …/release-cut and their /subpath variants (…/master/subpath) |
on-merge and on-release callers | no. Each lane gets its own triage check and writes only its own status |
A suffixed context never equals a listed lane, so it never claims or refreshes the aggregate. A
listed lane that did not run is ignored by the summary. override-status and E2E Tests/verified
take the same list from the lane reader (the one source, subpath included) and write failed statuses
for every PR lane in it, plus the rolling-upgrade contexts found by pattern. A failing subpath lane
is therefore verifiable, and the verified refresh of the aggregate can clear it. The lane reader
runs before the writer step in both workflows.
Regression checks:
python3 .github/scripts/test_e2e_status_workflows.py # requires PyYAML
# In mattermost-test-automation-toolkit:
node --test scripts/e2e-*.test.js
For workflow lint, actionlint 1.7.11 and 1.7.12 predate GitHub's queue: max property.
Until the installed linter supports it, exclude only its unexpected key "queue" for "concurrency" section diagnostic and run the structural test above;
keep all other workflow checks enabled.
Rollout requires these workflow changes on both dispatch source refs:
automated dispatch uses the selected PR ref, while manual dispatch can use
master. e2e-tests-check.yml also dispatches without --ref, so its automatic
E2E-only trigger uses the default branch (master). The existing toolkit PR supplies the pinned owned writer; its immutable
commit is usable by this branch before toolkit main is updated. Merging the
Mattermost workflow changes into master is still required for manual/master
dispatches (and other branches need the updated workflows). The manual override and verified-label workflows must also contain the SHA
publication lock. Running workflows
keep their original YAML and pinned actions, so drain older unguarded runs
before relying on ownership. Branch tests and a push do not repair production
or update already-running reporters. No TSIO service deployment is required.