chore(ci): warm Go module and build cache daily, restore read-only in CI (#38285)

* chore: extract install-<tool> Makefile targets

Pull each `go install` of a build tool into its own `install-<tool>`
target so the installs are declared once and reused as prerequisites,
and add `install-ci-tools` plus a `go-cache` target that pre-warms the
Go module and build caches.

Also collapse the duplicate `vet-api` target declarations into one.

* chore(ci): add restore-go-cache composite action

* chore(ci): mount warmed Go cache into run-in-buildenv

* chore(ci): add daily Go module/build cache warm workflow

* chore(ci): restore warmed Go cache read-only across server CI jobs

* Simplify Go cache key to glob-match all go.mod and go.sum files

* make GOBIN explicit

* Pass Go cache paths as step outputs, not GITHUB_ENV

restore-go-cache wrote GOMODCACHE/GOCACHE/GOBIN to GITHUB_ENV so later
steps would pick them up. zizmor flags any GITHUB_ENV write as a possible
code-execution vector, since an attacker-controlled value there can set
LD_PRELOAD or NODE_OPTIONS for every subsequent step.

Nothing untrusted reaches these particular values, but the write is also
unnecessary: expose them as action outputs and map them into the one step
that actually needs them, matching what 8f6d17d875 did for the buildenv
actions.

Only server-test-template's shard split consumes them outside the action.
Everything else either runs in the buildenv container, which gets the
container paths via explicit -e, or reads RUNNER_TEMP directly. That step
runs `go work` and shard-split.js's `go test -list` on the host, so it
takes GOMODCACHE and GOCACHE via env; GOBIN is omitted because the host
never invokes the tools in server/bin.

Also drops the `${{ github.workspace }}` interpolation in favour of the
runner's GITHUB_WORKSPACE, so no expression is expanded into the script.

zizmor now reports no findings for the action, with no ignore comment or
config entry needed.

* Note that server/bin is only a warm start

The cache key covers the Go version and go.mod/go.sum, so it doesn't
invalidate on a tools/mattermost-govet source edit or a Makefile version
pin bump. That reads like a staleness bug until you notice every consumer
re-runs its install-<tool> prerequisite, and `go install` rebuilds on
content, so a stale binary is overwritten before anything uses it.

Say so where the key is described, since it came up in review.

.github/actions/restore-go-cache/action.yml

@@ -0,0 +1,49 @@
+name: Restore Go Cache
+description: >
+  Read-only restore of the Go module cache, build cache, and CI tool binaries
+  warmed by server-ci-cache-warm. Module/build caches live under RUNNER_TEMP
+  (outside the checkout); tool binaries in server/bin.
+
+outputs:
+  gomodcache:
+    description: "Go module cache path, for GOMODCACHE"
+    value: ${{ steps.paths.outputs.gomodcache }}
+  gocache:
+    description: "Go build cache path, for GOCACHE"
+    value: ${{ steps.paths.outputs.gocache }}
+  gobin:
+    description: "CI tool binary path, for GOBIN"
+    value: ${{ steps.paths.outputs.gobin }}
+
+runs:
+  using: composite
+  steps:
+    - name: Resolve Go cache paths
+      id: paths
+      shell: bash
+      run: |
+        {
+          echo "gomodcache=${RUNNER_TEMP}/go-cache/mod"
+          echo "gocache=${RUNNER_TEMP}/go-cache/build"
+          echo "gobin=${GITHUB_WORKSPACE}/server/bin"
+        } >> "${GITHUB_OUTPUT}"
+    - name: Create Go cache directories
+      shell: bash
+      env:
+        GOMODCACHE: ${{ steps.paths.outputs.gomodcache }}
+        GOCACHE: ${{ steps.paths.outputs.gocache }}
+        GOBIN: ${{ steps.paths.outputs.gobin }}
+      run: mkdir -p "${GOMODCACHE}" "${GOCACHE}" "${GOBIN}"
+    - name: Restore Go module and build cache
+      uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+      with:
+        path: |
+          ${{ runner.temp }}/go-cache/mod
+          ${{ runner.temp }}/go-cache/build
+          server/bin
+        key: gomod-v1-${{ hashFiles('server/.go-version') }}-${{ hashFiles('**/go.mod', '**/go.sum') }}
+        # On a go.mod change, prefix-match the newest cache for this Go version
+        # and download only the delta. The Go-version segment is never dropped:
+        # build cache and tool binaries can't be reused across toolchains.
+        restore-keys: |
+          gomod-v1-${{ hashFiles('server/.go-version') }}-

.github/actions/run-in-buildenv/action.yml

@@ -60,9 +60,15 @@ runs:
         SCRIPT=$(mktemp)
         printf 'git config --global --add safe.directory /mattermost\n' > "${SCRIPT}"
         printf '%s\n' "${CMD}" >> "${SCRIPT}"
+        # Reuse the Go caches warmed by server-ci-cache-warm and restored by
+        # restore-go-cache (under RUNNER_TEMP, outside the checkout).
         docker run --rm \
           --network host \
           -e HEAD_REF -e REF_NAME -e RUN_ID \
+          -e GOMODCACHE=/gocache/mod \
+          -e GOCACHE=/gocache/build \
+          -e GOBIN=/mattermost/server/bin \
+          -v "${RUNNER_TEMP}/go-cache:/gocache" \
           -v "$PWD:/mattermost" \
           -v "${SCRIPT}:/run-script.sh" \
           -w "${WORKING_DIR}" \

.github/workflows/mmctl-test-template.yml

@@ -66,6 +66,8 @@ jobs:
         with:
           build-server-tag: ${{ inputs.build-server-tag }}
           fips-enabled: ${{ inputs.fips-enabled }}
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - name: Setup log artifact name
         id: log-artifact
         run: |
@@ -107,6 +109,10 @@ jobs:
             --env MM_SQLSETTINGS_DATASOURCE="$INPUT_DATASOURCE" \
             --env MMCTL_TESTFLAGS="$TESTFLAGS" \
             --env FIPS_ENABLED="$INPUT_FIPS_ENABLED" \
+            --env GOMODCACHE=/gocache/mod \
+            --env GOCACHE=/gocache/build \
+            --env GOBIN=/mattermost/server/bin \
+            -v "${RUNNER_TEMP}/go-cache:/gocache" \
             -v $PWD:/mattermost \
             -w /mattermost/server \
             $BUILD_IMAGE \

.github/workflows/server-ci-cache-warm.yml

@@ -0,0 +1,65 @@
+# Warms the Go module cache, build cache, and CI tool binaries that server CI
+# jobs restore read-only (via restore-go-cache). Runs daily on master and on
+# demand, keyed on the Go version and all go.mod/go.sum files.
+# server/bin is only a warm start: every consumer re-runs its install-<tool>
+# prerequisite, so a binary the key doesn't cover still gets rebuilt before use.
+name: Server CI Cache Warm
+
+on:
+  schedule:
+    - cron: "0 2 * * *" # Daily 2am UTC — before the nightly server CI runs
+  workflow_dispatch:
+
+# Serialize scheduled and manual runs so they don't race to save the same key.
+concurrency:
+  group: server-ci-cache-warm
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  actions: write # required to save the Actions cache
+
+jobs:
+  warm:
+    name: Warm Go Module and Build Cache
+    runs-on: ubuntu-22.04
+    # Don't let a hung run hold the concurrency group for the default 6 hours.
+    timeout-minutes: 60
+    steps:
+      - name: Checkout mattermost project
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Check if Go cache already warm
+        id: go-cache-check
+        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: |
+            ${{ runner.temp }}/go-cache/mod
+            ${{ runner.temp }}/go-cache/build
+            server/bin
+          key: gomod-v1-${{ hashFiles('server/.go-version') }}-${{ hashFiles('**/go.mod', '**/go.sum') }}
+          lookup-only: true
+
+      - name: Warm cache inside build container
+        if: steps.go-cache-check.outputs.cache-hit != 'true'
+        # Populates GOMODCACHE/GOCACHE (mounted from RUNNER_TEMP) and installs
+        # tools to server/bin — the paths saved below.
+        uses: ./.github/actions/run-in-buildenv
+        with:
+          run: make go-cache
+
+      - name: Fix file ownership (Docker writes files as root)
+        if: steps.go-cache-check.outputs.cache-hit != 'true'
+        run: sudo chown -R "$USER:$USER" "${RUNNER_TEMP}/go-cache" server/bin
+
+      - name: Save Go cache
+        if: steps.go-cache-check.outputs.cache-hit != 'true'
+        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: |
+            ${{ runner.temp }}/go-cache/mod
+            ${{ runner.temp }}/go-cache/build
+            server/bin
+          key: gomod-v1-${{ hashFiles('server/.go-version') }}-${{ hashFiles('**/go.mod', '**/go.sum') }}

.github/workflows/server-ci.yml

@@ -104,6 +104,8 @@ jobs:
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - uses: ./.github/actions/run-in-buildenv
         with:
           run: |
@@ -129,6 +131,8 @@ jobs:
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - name: Check for renumbered or renamed migrations
         uses: ./.github/actions/run-in-buildenv
         with:
@@ -147,6 +151,8 @@ jobs:
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - uses: ./.github/actions/run-in-buildenv
         with:
           run: make check-style
@@ -160,6 +166,8 @@ jobs:
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - uses: ./.github/actions/run-in-buildenv
         with:
           run: make vet-api
@@ -336,6 +344,8 @@ jobs:
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
         with:
           persist-credentials: false
+      - name: Restore Go cache
+        uses: ./.github/actions/restore-go-cache
       - name: Build
         uses: ./.github/actions/run-in-buildenv
         with:

.github/workflows/server-test-template.yml

@@ -117,6 +117,10 @@ jobs:
           build-server-tag: ${{ inputs.build-server-tag }}
           fips-enabled: ${{ inputs.fips-enabled }}
 
+      - name: Restore Go cache
+        id: go-cache
+        uses: ./.github/actions/restore-go-cache
+
       - name: Restore test timing data
         if: inputs.shard-total > 1
         id: timing-cache
@@ -189,6 +193,10 @@ jobs:
         env:
           SHARD_INDEX: ${{ inputs.shard-index }}
           SHARD_TOTAL: ${{ inputs.shard-total }}
+          # `go work` and shard-split.js's `go test -list` run on the host, so
+          # they need the restored caches that the container gets via -e.
+          GOMODCACHE: ${{ steps.go-cache.outputs.gomodcache }}
+          GOCACHE: ${{ steps.go-cache.outputs.gocache }}
         run: |
           set -euo pipefail
 
@@ -261,6 +269,10 @@ jobs:
             --env RACE_MODE \
             --env TEST_TARGET \
             --env BUILD_NUMBER \
+            --env GOMODCACHE=/gocache/mod \
+            --env GOCACHE=/gocache/build \
+            --env GOBIN=/mattermost/server/bin \
+            -v "${RUNNER_TEMP}/go-cache:/gocache" \
             -v $PWD:/mattermost \
             -w /mattermost/server \
             $BUILD_IMAGE \