mirror of https://github.com/mattermost/mattermost.git synced 2026-08-27 05:37:15 -05:00

Files

T

Doug Lauder 7778bb79d1 MM-69792: Recover shared channel sync after a brief remote outage (#37499 )

* MM-69792: Recover shared channel sync after a brief remote outage

When a remote went offline for less than the 5-minute IsOnline()
threshold, sync recovery never ran. IsOnline() never flipped, so no
connection-state-change event fired and ForceSyncForRemote was never
triggered. Posts created during the outage stayed unsynced until the
next organic change in the channel.

The root cause was that send failures are reported asynchronously
through a callback that sendSyncMsgToRemote discarded. On a failed
send syncForRemote still returned nil, so processTask saw success:
the retry loop and the failure-notification hook were effectively
dead code for delivery failures.

Changes:

- sendSyncMsgToRemote now propagates the delivery error reported via
  the callback instead of swallowing it. A failed send is retried
  rather than silently dropped. This applies to every sync type
  (posts, users, memberships, reactions, statuses); the success path
  is unchanged.

- remotecluster tracks a per-remote "sync failed since last ping"
  flag (NotifySyncFailed). On the next successful ping, PingNow fires
  a connection-state-change event even when IsOnline() never flipped,
  driving a single ForceSyncForRemote on recovery.

- Sync retries are now spaced by SyncRetryDelay (15s) instead of
  retrying immediately, giving a short outage time to recover and
  guaranteeing a failed ping before retries exhaust.

Also adds a post/offline-recovery case to the sharedchannel-test
integration tool (stop and restart Server B mid-run) and disables
metrics on the two test servers so they do not collide on the
metrics port when the loaded config has metrics enabled.

* fix linter error

* Fix Rabbit nit

* fix CI

* Delete pr37499-fix-findings-1-and-4.md

* address review comments

* MM-69792: Keep a per-remote retry task when a shared channel sync fans out

A sync task with no specific remote (remoteID == "") fans out to every
remote sharing the channel. On failure each remote's retry was a copy of
the original task that still carried the original, remote-less task id.
Because addTask merges on task id, all of those per-remote retries
collapsed into a single entry, so only one failed remote was ever
retried and the rest were silently dropped. Delivery failures were
partly masked by the ping-driven ForceSyncForRemote recovery, but
non-delivery errors for the other remotes had no path back.

Give each per-remote retry a remote-specific id (task.id + remoteId) when
the originating task had no remote, so every failed remote keeps its own
retry task. Single-remote tasks already have a remote-specific id and are
left untouched, since recomputing would grow the id on each retry.

Also harden the tests: assert the retry is scheduled the full
SyncRetryDelay out (no tolerance), assert the result callback is not
invoked on delivery failures or unconfirmed responses (so the cursor
stays un-advanced), and add a regression test covering fan-out to
multiple failing remotes.

* fix one more pre-existing edge case.

* fix linter

2026-07-15 09:47:55 -04:00

mattermost-govet

chore: bump Go version to 1.26.4 (#37287 )

2026-07-02 16:58:40 +03:00

mmgotool

Improve license error message for wrong service environment (#37198 )

2026-07-07 23:18:00 +02:00

sharedchannel-test

MM-69792: Recover shared channel sync after a brief remote outage (#37499 )

2026-07-15 09:47:55 -04:00

README.md

Inline mattermost-govet into the monorepo (#35869 )

2026-04-01 13:24:22 +00:00

README.md

Tools

This directory aims to provide a set of tools that simplify and enhance various development tasks. This README file serves as a guide to help you understand the directory, features of these tools, and how to get started using it. This is a collection of utilities and scripts designed to streamline common development tasks for Mattermost. These tools aim to help automate repetitive tasks and improve productivity.

Included tools

mattermost-govet: custom Go vet analyzers enforcing Mattermost-specific code conventions (structured logging, error handling, SQL safety, etc.). Used by make vet in the server.
mmgotool: is a CLI to help with i18n related checks for the mattermost/server development.
sharedchannel-test: integration test tool that validates shared channel synchronization (posts, reactions, membership) between two real Mattermost server instances.

Installation & Usage

mmgotool

To install mmgotool, simply run the following command: go install github.com/mattermost/mattermost/tools/mmgotool

Make sure you have the necessary prerequisites such as Go compiler.

mmgotool i18n has following subcommands described below:

check: Check translations
check-empty-src: Check for empty translation source strings
clean-empty: Clean empty translations
extract: Extract translations

sharedchannel-test

Stands up two Mattermost Enterprise instances, creates a remote cluster connection, and runs integration tests for shared channel synchronization (membership, posts, reactions).

Prerequisites:

make start-docker (Postgres and friends running)
Enterprise repo present at ../../enterprise
An enterprise license file

Usage:

# Managed mode (builds server, starts/stops both instances automatically)
cd tools/sharedchannel-test
go run . --license /path/to/license.mattermost-license --server-dir ../../server

# External mode (connect to already-running instances)
go run . --license /path/to/license.mattermost-license --manage=false \
  --server-a http://localhost:9065 --server-b http://localhost:9066