fix: git sync on multiple replicas deployments

CLAUDE.md

@@ -966,9 +966,9 @@ Everything else follows from that:
   database's. Imported content is rendered with **no script or style permission** whoever ran the
   import, since the file need not have been written by them.
 - **An import is never written back to the target it reads from.** What it adopts is saved like any
-  edit and so copied to every target holding it, and `git` and `sftp` serialize every operation per
-  target — with the import itself being one, a copy back to the source waits on the import that is
-  waiting on it. `importTree` runs under `storage.importingFrom(target)`, an `AsyncLocalStorage`
+  edit and so copied to every target holding it, and `sftp` serializes every operation per target —
+  with the import itself being one, a copy back to the source waits on the import that is waiting on
+  it. For `git` the copy back would be a queued change re-publishing what was just pulled. `importTree` runs under `storage.importingFrom(target)`, an `AsyncLocalStorage`
   scope that `putAsset`, `eachPageTarget` and `eachAssetTarget` skip that target in (counting it,
   for `putAsset`, as somewhere the file is held). Anything else a module does to the wiki while
   holding its own queue — git's removals in `applyIncoming` — needs the same scope.
@@ -1002,15 +1002,25 @@ misconfigured one reads healthy until something is actually asked of it.
 
 **The git target is the disk target plus history and a remote.** Same tree, same helpers, and then:
 
-- **Commits are local and immediate; the network is batched.** A page save writes, commits and
-  returns — `prepareRepo` deliberately never contacts a remote, so an unreachable one cannot make the
-  wiki slow to edit or fail an upload. `ensureRemote` is the half that does, and only `sync` calls it.
+- **The working copy is a cache; the database is what every instance agrees on.** An HA set has a
+  working copy per pod, on no shared volume and perhaps on no volume at all, so nothing is kept only
+  there. A page save or an upload queues a row in `storageOutbox` — the file as written, who wrote it,
+  when — and that is all it does: nothing on the request path touches git or the network. The commit
+  the wiki and the remote last agreed on is `storageSyncState`, recorded against the remote URL and
+  branch it belongs to. A sync, on whichever instance runs it, `checkoutAt`s that commit (wiping
+  whatever its own copy held, an interrupted rebase and a stale `index.lock` included), `replay`s the
+  queue as one commit per change with the original author and date, settles with the remote, pushes,
+  applies what came back, and only then drops the published rows and records the new commit, in one
+  transaction. Rows are dropped by id, never by range, since ids are not handed out in commit order.
+- **One sync per target across the whole set.** `storage.exclusivelyFor` holds a session advisory lock
+  (`pg_try_advisory_lock(TARGET_LOCK_CLASS, hashtext(targetId))`) for the length of it; an instance
+  that finds it taken reports so and does nothing, since the holder is doing the same job. `withRepo`
+  serializes within the process on top of that.
 - **`sync` runs on a schedule.** `tasks/simple/sync-storage-targets.ts`, on a `* * * * *`
   `jobSchedule` row seeded by `jobs.init()`, walks every enabled target whose module declares a `sync`
   handler (`storage.syncableTargets`) and runs it through `executeAction`, so a failure lands on the
-  target's Status card. The **Force Sync** action is the same call on demand. In an HA set the
-  scheduler hands the job to one instance, which is the one whose working copy syncs — each instance
-  keeps its own.
+  target's Status card. The **Force Sync** action is the same call on demand. Which instance gets a
+  tick does not matter.
 - **How often is `storage.syncInterval`, per site** (the **Configuration** tab, `syncIntervalFor`,
   default `5m`). Since it is per site, one cron row cannot express it: the tick is every minute — as
   fine as the shortest interval anyone can set — and the task skips the sites whose turn it is not.
@@ -1024,38 +1034,41 @@ misconfigured one reads healthy until something is actually asked of it.
   has to come off the tree rather than the file: the stem is looked up first and only counts as the
   page if `pages.storageFileNameOf` matches the name that went, so a deleted `readme.pdf` never takes
   the page `readme` with it.
-- **A conflict is settled, never left open.** The pull is `--rebase --autostash`, and a rebase that
-  conflicts stops and waits for a human that this working copy does not have — with the index
-  unmerged, which git then refuses every write against, so it is not one failed sync but every commit
-  after it (`ensureRemote`'s checkout is the first thing to fail, and what such a target reports).
-  `pullRebase` therefore rolls a conflicted rebase back and pulls again with `-X ours`, which during a
-  rebase is the *remote*, consistent with a pull being authoritative. What `-X` cannot settle is a
-  file one side changed and the other deleted; that fails the sync, but leaves the working copy usable.
-  `abortInterrupted` also runs at the top of every sync and in `prepareRepo`, so a copy somebody left
-  mid-rebase by hand heals itself instead of needing Purge.
-- **A working copy started again from nothing re-attaches itself.** It is a directory in a container
-  while the content is a database elsewhere, so an upgrade that replaces the container without a volume
-  for `data/repo` takes it with it — and `prepareRepo` then inits an empty one, the next page save
-  commits into it, and the wiki has the remote's own files under a second root commit. `sync` asks
-  `sharesHistoryWith` before it pulls, and with no commit in common `reattach` merges the remote's
-  history in with `--allow-unrelated-histories -X ours` instead: the remote's history and every file
-  the wiki has not re-saved come back, the working copy's version wins where the two overlap (these
-  files were written from the database *since* the copy was created, so the remote's copy is the older
-  one), and the push that follows puts the two in step. It runs in **every** mode — a `push` mode force
-  push from a working copy created ten minutes ago would otherwise replace the whole repository with
-  it. The wiki itself is not written to and `applyIncoming` is skipped for that sync, since the diff is
-  the entire repository; Import Everything is what takes in content the repository has and the wiki
-  does not.
+- **A conflict is settled, never left open.** The queue's commits go onto the remote with `rebase
+  --onto origin/<branch> <agreed commit>`, and a rebase that conflicts stops and waits for a human
+  this working copy does not have. `rebaseOnto` therefore rolls a conflicted rebase back and runs it
+  again with `-X ours`, which during a rebase is the *remote*, consistent with a pull being
+  authoritative; what `-X` cannot settle — a file one side changed and the other deleted — is finished
+  by `settleRebase`, taking stage 2 (the remote's side) of every unmerged path. A sync that stopped
+  there would fail every sync after it, since the queue it could not publish is the next one's too.
+- **Changed in the wiki while a sync ran** means queued after it read the queue: the remote's change
+  to that path is not applied over it, and the next sync publishes the wiki's. In **Pull only** the
+  wiki queues nothing at all (`acceptsWrites`, and `canStore` declines assets), and a row left from
+  before the switch is dropped once the remote changes its path.
+- **The first sync with a remote applies nothing.** With no agreed commit — a new target, one pointed
+  at another URL or branch, an installation upgraded from before the queue existed, or a remote whose
+  history was rewritten past the agreed commit — the remote's branch is taken up as it stands and the
+  queue committed on top; there is nothing to diff it against. Import Everything is what takes in
+  content the repository has and the wiki does not, and it reads the agreed commit's tree out of git
+  (`ls-tree`, `cat-file`) rather than off any working tree.
+- **An asset is read out of git, not off the working tree** (`readBlob`), which on any given instance
+  is wherever its last sync left it. Not yet published, its bytes are the queue's — followed back
+  through a queued move — and after that they are the agreed commit's, fetched first by an instance
+  that has never seen it. The fetch is the only network a read can cost, and only on a cold instance.
 - **The diff, not the tree.** Incoming changes come from `git diff --name-status -M -z` between the
-  commit the branch was on and the one it is on now — a sync runs every few minutes and cannot read
+  commit the branch was on after the replay and the one it is on after the rebase — a sync runs every few minutes and cannot read
   every file each time. `-z` because a path may contain anything, newlines included.
-- **Every operation is serialized per target** by `withRepo`: git locks its index for the length of a
-  write, so two concurrent uploads would otherwise have one fail on `index.lock`.
 - **An empty commit is never made.** Most page saves do not change the stored form of the page (a
   re-publish, a tag reordered into the same order), and `commitPaths` checks `diff --cached` before
   committing so the history says what actually changed.
 - Operator git config is inherited on purpose — including `commit.gpgsign`, which will fail every
-  commit if it is on without a usable key. The failure surfaces as the target's recorded `error`.
+  commit if it is on without a usable key. The failure surfaces as the target's recorded `error`, and
+  the queue is kept for the sync after it. simple-git strips every ambient `GIT_*` variable before
+  spawning git, so `GIT_CONFIG_GLOBAL` cannot switch that off for a test instance; a `HOME` of its own
+  can.
+- **Without a Repository URI the repository is one instance's own**: the queue is committed into
+  whichever working copy syncs, and nothing else holds those commits. That needs a single instance on
+  a persistent volume; with a remote, neither is needed.
 
 **A change carries who made it, and only git cares.** `StorageAssetRef` and `StoragePageRef` carry an
 optional `actorId`, which is the user id the models already had in hand at each dispatch site;

backend/core/maintenance.ts

@@ -55,6 +55,7 @@ export default {
     await WIKI.models.assets.purgeCache()
     await WIKI.models.icons.purgeCache()
     await WIKI.models.blocks.purgeCache()
+    WIKI.models.storage.targetCache.clear()
 
     await WIKI.models.locales.reloadCache()
     await WIKI.models.sites.reloadCache()
@@ -94,6 +95,15 @@ export default {
     WIKI.events.inbound.on('reloadSites', async () => {
       await WIKI.models.sites.reloadCache()
     })
+    // -> Storage targets are cached per instance too, and decide where every upload is written. Dropped
+    //    rather than reloaded: the next request that needs them reads them back.
+    WIKI.events.inbound.on('reloadStorageTargets', ({ data: siteId }) => {
+      if (typeof siteId === 'string') {
+        WIKI.models.storage.targetCache.delete(siteId)
+      } else {
+        WIKI.models.storage.targetCache.clear()
+      }
+    })
     WIKI.events.inbound.on('reloadBlocks', async () => {
       await WIKI.models.blocks.refreshCustomIndex()
       await WIKI.models.blocks.sweepCache()

backend/db/migrations/20261012072149_storage-outbox/migration.sql

@@ -0,0 +1,26 @@
+CREATE TABLE "storageOutbox" (
+	"id" bigint PRIMARY KEY GENERATED ALWAYS AS IDENTITY (sequence name "storageOutbox_id_seq" INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START WITH 1 CACHE 1),
+	"targetId" uuid NOT NULL,
+	"kind" varchar(16) NOT NULL,
+	"op" varchar(16) NOT NULL,
+	"path" text NOT NULL,
+	"previousPath" text,
+	"label" text NOT NULL,
+	"previousLabel" text,
+	"data" bytea,
+	"actorId" uuid,
+	"createdAt" timestamp DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "storageSyncState" (
+	"targetId" uuid PRIMARY KEY,
+	"remote" text NOT NULL,
+	"branch" varchar(255) NOT NULL,
+	"commit" varchar(64) NOT NULL,
+	"updatedAt" timestamp DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE INDEX "storageOutbox_targetId_idx" ON "storageOutbox" ("targetId","id");--> statement-breakpoint
+CREATE INDEX "storageOutbox_path_idx" ON "storageOutbox" ("targetId","path");--> statement-breakpoint
+ALTER TABLE "storageOutbox" ADD CONSTRAINT "storageOutbox_targetId_storage_id_fkey" FOREIGN KEY ("targetId") REFERENCES "storage"("id") ON DELETE CASCADE;--> statement-breakpoint
+ALTER TABLE "storageSyncState" ADD CONSTRAINT "storageSyncState_targetId_storage_id_fkey" FOREIGN KEY ("targetId") REFERENCES "storage"("id") ON DELETE CASCADE;

backend/db/schema.ts

@@ -1411,6 +1411,70 @@ export const storage = pgTable(
   (table) => [uniqueIndex('storage_composite_idx').on(table.siteId, table.module)]
 )
 
+/**
+ * Changes a target has accepted and not yet published, oldest first.
+ *
+ * For a target whose own copy of the content is not somewhere every instance can see — the git
+ * target, whose working copy is a directory on whichever pod made the change. A change is recorded
+ * here instead, by whichever instance made it, and the target's sync replays the queue as commits on
+ * whichever instance runs it. So the queue is what every instance of an HA set agrees on, and a
+ * working copy is a cache any of them can rebuild.
+ *
+ * `data` is a snapshot of the file as it was written, not a pointer to the page or asset: forty saves
+ * between two syncs are forty commits, each with what that save wrote, and an asset whose only home is
+ * this target has its bytes nowhere else until the sync has pushed them.
+ */
+export const storageOutbox = pgTable(
+  'storageOutbox',
+  {
+    // -> Replay order. Bigint because every page save on a busy wiki takes one.
+    id: bigint({ mode: 'number' }).primaryKey().generatedAlwaysAsIdentity(),
+    targetId: uuid()
+      .notNull()
+      .references(() => storage.id, { onDelete: 'cascade' }),
+    // -> Which decides the commit message
+    kind: varchar({ length: 16 }).$type<'page' | 'asset'>().notNull(),
+    op: varchar({ length: 16 }).$type<'put' | 'delete' | 'move'>().notNull(),
+    // -> Relative to the target's root, as the layout files it. Where a move ends up.
+    path: text().notNull(),
+    // -> Where a move started
+    previousPath: text(),
+    // -> How a commit message names the content, worked out when the change was made
+    label: text().notNull(),
+    previousLabel: text(),
+    // -> The file as written, for a `put`
+    data: bytea(),
+    // -> Not a foreign key: a commit made after the account is gone still happened, and falls back to
+    //    the target's default author
+    actorId: uuid(),
+    createdAt: timestamp().notNull().defaultNow()
+  },
+  (table) => [
+    index('storageOutbox_targetId_idx').on(table.targetId, table.id),
+    // -> A read of a file this target has accepted and not yet published
+    index('storageOutbox_path_idx').on(table.targetId, table.path)
+  ]
+)
+
+/**
+ * Where a target that syncs with a remote last left off, as every instance sees it.
+ *
+ * `commit` is the remote's commit the wiki and the remote last agreed on: everything the outbox held up
+ * to then was pushed in it, and everything the remote had up to then was applied to the wiki. It is
+ * only meaningful for the remote and branch it was recorded against, which is why those are kept with
+ * it — a target pointed somewhere else starts again from that remote rather than diffing against a
+ * commit the remote has never had.
+ */
+export const storageSyncState = pgTable('storageSyncState', {
+  targetId: uuid()
+    .primaryKey()
+    .references(() => storage.id, { onDelete: 'cascade' }),
+  remote: text().notNull(),
+  branch: varchar({ length: 255 }).notNull(),
+  commit: varchar({ length: 64 }).notNull(),
+  updatedAt: timestamp().notNull().defaultNow()
+})
+
 // TAGS --------------------------------
 export const tags = pgTable(
   'tags',

backend/helpers/storageFiles.ts

@@ -53,6 +53,11 @@ export const ROOT_PAGE_NAME = 'index'
  */
 const IGNORED_SEGMENT = /^\.|\.tmp$/
 
+/** Whether a stored path is one of those, and so never content. */
+export function isIgnoredPath(segments: string[]): boolean {
+  return segments.some((segment) => IGNORED_SEGMENT.test(segment))
+}
+
 /**
  * A configured root as an absolute path.
  *
@@ -352,7 +357,7 @@ export async function walkStored(root: string): Promise<StoredFile[] | null> {
     }
     const filePath = path.join(entry.parentPath, entry.name)
     const segments = path.relative(root, filePath).split(path.sep)
-    if (segments.some((segment) => IGNORED_SEGMENT.test(segment))) {
+    if (isIgnoredPath(segments)) {
       continue
     }
     files.push({ filePath, segments })

backend/models/storage.ts

@@ -2,7 +2,7 @@ import { AsyncLocalStorage } from 'node:async_hooks'
 import fs from 'node:fs/promises'
 import path from 'node:path'
 import { load } from 'js-yaml'
-import { and, eq, inArray } from 'drizzle-orm'
+import { and, asc, desc, eq, getTableColumns, inArray, lt } from 'drizzle-orm'
 import {
   CustomError,
   dataPathRoot,
@@ -11,7 +11,12 @@ import {
   parseModuleProps,
   resolveLocalPath
 } from '../helpers/common.ts'
-import { sites as sitesTable, storage as storageTable } from '../db/schema.ts'
+import {
+  sites as sitesTable,
+  storage as storageTable,
+  storageOutbox as storageOutboxTable,
+  storageSyncState as storageSyncStateTable
+} from '../db/schema.ts'
 import type { ModuleProp } from '../helpers/common.ts'
 import type { AssetKind } from './assets.ts'
 
@@ -29,6 +34,12 @@ const DB_MODULE = 'db'
 /** The target an import in progress is reading from. See `Storage.importingFrom`. */
 const importSource = new AsyncLocalStorage<string>()
 
+/**
+ * The first half of every per-target advisory lock (`exclusivelyFor`), the target's id hashed being
+ * the second. The two-key form is a keyspace of its own, apart from the single-key locks elsewhere.
+ */
+const TARGET_LOCK_CLASS = 4210003
+
 /** Which content type an asset falls into when it is not large enough to count as a large file. */
 const CONTENT_TYPE_BY_KIND: Record<AssetKind, ContentType> = {
   image: 'images',
@@ -96,10 +107,9 @@ const SIZE_UNIT_BYTES: Record<string, number> = {
 /**
  * How long the configured targets of a site are trusted before they are read again.
  *
- * The backstop rather than the mechanism: saving the admin form drops the entry on the instance that
- * handled the request, but a second instance has no way to hear about it — so every entry expires on
- * its own as well. Short enough that a change made elsewhere takes effect promptly, long enough that
- * a burst of uploads resolves its target once rather than once per file.
+ * The backstop rather than the mechanism: a saved change drops the entry on every instance at once
+ * (`invalidateTargets`), and this only bounds how long an instance that missed the notification — its
+ * listening connection was down at the time — goes on answering from the old configuration.
  */
 const TARGET_CACHE_TTL_MS = 30_000
 
@@ -415,6 +425,34 @@ export interface StoragePageContent {
   content: string
 }
 
+/** A change as a target queues it. See `storageOutbox` in the schema. */
+export interface StorageOutboxChange {
+  kind: 'page' | 'asset'
+  op: 'put' | 'delete' | 'move'
+  path: string
+  previousPath?: string | null
+  label: string
+  previousLabel?: string | null
+  data?: Buffer | null
+  actorId?: string | null
+}
+
+/** A queued change as it is read back, without its bytes. */
+export interface StorageQueuedChange extends Omit<StorageOutboxChange, 'data'> {
+  id: number
+  previousPath: string | null
+  previousLabel: string | null
+  actorId: string | null
+  createdAt: Date
+}
+
+/** Where a target that syncs last agreed with its remote. See `storageSyncState` in the schema. */
+export interface StorageSyncState {
+  remote: string
+  branch: string
+  commit: string
+}
+
 /**
  * What a storage module implementation exports.
  *
@@ -755,6 +793,20 @@ class Storage {
     return targets
   }
 
+  /**
+   * Drop a site's cached targets here and on every other instance.
+   *
+   * A target's configuration decides where an upload is written and whether a page save is queued for
+   * git at all, so an instance answering from the old one for the length of `TARGET_CACHE_TTL_MS`
+   * would write into a target that was just switched off — or, for a git target just set to Pull
+   * only, queue changes nothing will ever publish. Not emitted from `syncSite`, which every instance
+   * runs for itself at boot.
+   */
+  invalidateTargets(siteId: string): void {
+    this.targetCache.delete(siteId)
+    WIKI.events.outbound.emit('reloadStorageTargets', siteId)
+  }
+
   /**
    * The same list, answered from memory where it can be.
    *
@@ -1072,7 +1124,7 @@ class Storage {
       .update(storageTable)
       .set(values)
       .where(and(eq(storageTable.siteId, siteId), eq(storageTable.id, target.id)))
-    this.targetCache.delete(siteId)
+    this.invalidateTargets(siteId)
     return (result.rowCount ?? 0) > 0
   }
 
@@ -1237,7 +1289,7 @@ class Storage {
     }
     const updated = await WIKI.models.sites.updateSite(siteId, { config: { storage: config } })
     // -> A target's content type depends on the threshold, so the resolved list is now stale
-    this.targetCache.delete(siteId)
+    this.invalidateTargets(siteId)
     return updated
   }
 
@@ -1779,6 +1831,147 @@ class Storage {
     return actor
   }
 
+  /**
+   * Record a change a target has accepted, for its sync to publish.
+   *
+   * What a target does instead of writing to its own copy when that copy is not somewhere every
+   * instance can see. See `storageOutbox` in the schema.
+   */
+  async queueChange(target: StorageTarget, change: StorageOutboxChange): Promise<void> {
+    await WIKI.db.insert(storageOutboxTable).values({ ...change, targetId: target.id })
+  }
+
+  /**
+   * Every change queued for a target, oldest first — without the bytes, which `queuedData` reads one
+   * at a time. An offload can queue every file a site has, and holding all of them at once would be
+   * the whole asset table in memory.
+   */
+  async queuedChanges(target: StorageTarget): Promise<StorageQueuedChange[]> {
+    const { data: _data, ...columns } = getTableColumns(storageOutboxTable)
+    return WIKI.db
+      .select(columns)
+      .from(storageOutboxTable)
+      .where(eq(storageOutboxTable.targetId, target.id))
+      .orderBy(asc(storageOutboxTable.id))
+  }
+
+  /** The file a queued `put` wrote. */
+  async queuedData(id: number): Promise<Buffer | null> {
+    const [row] = await WIKI.db
+      .select({ data: storageOutboxTable.data })
+      .from(storageOutboxTable)
+      .where(eq(storageOutboxTable.id, id))
+    return row?.data ?? null
+  }
+
+  /**
+   * The latest queued change that left something at this path, or took it away.
+   *
+   * @param beforeId Only changes queued before this one — for following a move back to where the
+   *   file was before it
+   */
+  async latestQueuedAt(
+    target: StorageTarget,
+    relPath: string,
+    beforeId?: number
+  ): Promise<StorageQueuedChange | null> {
+    const { data: _data, ...columns } = getTableColumns(storageOutboxTable)
+    const [row] = await WIKI.db
+      .select(columns)
+      .from(storageOutboxTable)
+      .where(
+        and(
+          eq(storageOutboxTable.targetId, target.id),
+          eq(storageOutboxTable.path, relPath),
+          beforeId === undefined ? undefined : lt(storageOutboxTable.id, beforeId)
+        )
+      )
+      .orderBy(desc(storageOutboxTable.id))
+      .limit(1)
+    return row ?? null
+  }
+
+  /** Where a target that syncs last agreed with its remote, if it has. See `storageSyncState`. */
+  async syncStateFor(target: StorageTarget): Promise<StorageSyncState | null> {
+    const [row] = await WIKI.db
+      .select({
+        remote: storageSyncStateTable.remote,
+        branch: storageSyncStateTable.branch,
+        commit: storageSyncStateTable.commit
+      })
+      .from(storageSyncStateTable)
+      .where(eq(storageSyncStateTable.targetId, target.id))
+    return row ?? null
+  }
+
+  /**
+   * Finish a sync: drop the changes it published and record where it left off, both or neither.
+   *
+   * By id rather than by "everything up to the last one": ids are handed out when a row is inserted
+   * and become visible when it commits, which is not the same order, so a change saved on another
+   * instance while the sync ran can carry a lower id than one the sync already read.
+   */
+  async settleSync(
+    target: StorageTarget,
+    { published, state }: { published: number[]; state: StorageSyncState | null }
+  ): Promise<void> {
+    await WIKI.db.transaction(async (trx) => {
+      if (published.length > 0) {
+        await trx.delete(storageOutboxTable).where(inArray(storageOutboxTable.id, published))
+      }
+      if (state) {
+        await trx
+          .insert(storageSyncStateTable)
+          .values({ targetId: target.id, ...state })
+          .onConflictDoUpdate({
+            target: storageSyncStateTable.targetId,
+            set: { ...state, updatedAt: new Date() }
+          })
+      }
+    })
+  }
+
+  /**
+   * Run something no other instance is running for this target at the same time.
+   *
+   * A session-level advisory lock on a connection held for the length of the work, rather than a
+   * transaction-level one, because the work is a sync — minutes of network and git — and a
+   * transaction held open across it would pin rows as well as a connection. Postgres drops the lock
+   * with the connection, so an instance that dies mid-sync does not leave it held.
+   *
+   * @returns `{ busy: true }` without running anything when another instance holds it. A sync is
+   *   periodic: the instance that has it is doing the job, and waiting to do it again straight after is
+   *   no use to anybody.
+   */
+  async exclusivelyFor<T>(
+    target: StorageTarget,
+    work: () => Promise<T>
+  ): Promise<{ busy: true } | { busy: false; result: T }> {
+    const client = await WIKI.dbManager.pool!.connect()
+    let broken = false
+    try {
+      const { rows } = await client.query(
+        'SELECT pg_try_advisory_lock($1, hashtext($2)) AS locked',
+        [TARGET_LOCK_CLASS, target.id]
+      )
+      if (!rows[0]?.locked) {
+        return { busy: true }
+      }
+      try {
+        return { busy: false, result: await work() }
+      } finally {
+        await client
+          .query('SELECT pg_advisory_unlock($1, hashtext($2))', [TARGET_LOCK_CLASS, target.id])
+          .catch(() => {
+            // -> A connection that cannot unlock is thrown away, which unlocks it
+            broken = true
+          })
+      }
+    } finally {
+      client.release(broken)
+    }
+  }
+
   /**
    * Ensure a module's implementation is loaded
    *

backend/modules/storage/git/definition.yml

@@ -2,7 +2,7 @@ key: git
 title: Git
 icon: '/_assets/icons/ultraviolet-git.svg'
 banner: '/_assets/storage/git.jpg'
-description: Keep this site's content in a Git repository, committed as it changes and synchronized with a remote. Every page and file is an ordinary versioned file, so the wiki's history is also the repository's.
+description: Keep this site's content in a Git repository, committed per change and synchronized with a remote. Every page and file is an ordinary versioned file, so the wiki's history is also the repository's. Changes are committed and pushed by each sync, from any instance, so it needs no persistent or shared volume.
 assetDelivery:
   isDirectAccessSupported: false
 contentTypes:
@@ -22,21 +22,21 @@ props:
   repoUrl:
     type: String
     title: Repository URI
-    hint: Git-compliant URI (e.g. git@server.com:org/repo.git for SSH, https://server.com/org/repo.git for basic). Leave empty to keep a purely local repository, committed but never pushed.
+    hint: Git-compliant URI (e.g. git@server.com:org/repo.git for SSH, https://server.com/org/repo.git for basic). Leave empty to keep a purely local repository, committed but never pushed - which lives on one instance's disk only, so needs a persistent volume and a single instance.
     icon: dns
     order: 2
   branch:
     type: String
     default: 'main'
     title: Branch
-    hint: The branch to use during pull / push. It must already exist on the remote.
+    hint: The branch to use during pull / push. The first push creates it on a remote that has no branches yet.
     icon: code-fork
     order: 3
   syncMode:
     type: String
     default: 'sync'
     title: Sync Direction
-    hint: Sync pulls and then pushes. Push force-pushes the wiki's commits and never takes anything back, so the wiki always wins. Pull only takes changes in, and is what makes the remote the authority.
+    hint: Sync pulls and then pushes. Push force-pushes the wiki's commits and never takes anything back, so the wiki always wins. Pull only takes changes in and never writes the wiki's own to the repository, which makes the remote the authority.
     icon: synchronize
     enum:
       - sync|Sync
@@ -127,7 +127,7 @@ props:
     type: String
     title: Local Repository Path
     default: './data/repo'
-    hint: Where the working copy is kept. Give each site its own path unless you turn on Add Site ID Prefix under Configuration, since two sites sharing a repository would otherwise write over each other. Relative paths are resolved from the Wiki.js install directory.
+    hint: Where each instance keeps its working copy, which is a cache it rebuilds from the remote whenever it needs to. Give each site its own path unless you turn on Add Site ID Prefix under Configuration, since two sites sharing a repository would otherwise write over each other. Relative paths are resolved from the Wiki.js install directory.
     icon: symlink-directory
     order: 32
     localPath: data
@@ -146,15 +146,15 @@ actions:
     icon: synchronize
   syncUntracked:
     label: Add Untracked Changes
-    hint: Write every page and file this target is configured to hold into the repository and commit whatever is missing. Content created before Git was enabled - or while it was turned off - is untracked until this is run.
+    hint: Write every page and file this target is configured to hold into the repository, commit whatever is missing and sync. Content created before Git was enabled - or while it was turned off - is untracked until this is run.
     icon: database-daily-export
   importAll:
     label: Import Everything
-    hint: Take everything currently in the local repository into the wiki, whatever the last commit did. For picking up content from a remote repository that existed before Git was enabled here.
-    warn: The repository wins every collision. A page it replaces keeps its previous version in its history, but a file has none - the bytes it replaces are gone from every storage target holding them.
+    hint: Take everything the repository held at the last sync into the wiki, whatever the last commit did. For picking up content from a remote repository that existed before Git was enabled here, which the first sync does not take in.
+    warn: The repository wins every collision, except with changes the wiki has not published yet. A page it replaces keeps its previous version in its history, but a file has none - the bytes it replaces are gone from every storage target holding them.
     icon: database-daily-import
   purge:
     label: Purge Local Repository
-    hint: Empty the local working copy and clone it again from the remote. A working copy that was replaced re-attaches itself on the next sync, so this is for one git can no longer make sense of at all. The remote is not touched and nothing is committed.
-    warn: Any commit that exists only in the local repository and has never been pushed is lost. Run a Force Sync first if you are not sure.
+    hint: Delete the working copy of the instance that answers this request. The next sync there takes it again from the remote, with the changes the wiki has queued on top, so nothing is lost. The remote is not touched and nothing is committed.
+    warn: With no Repository URI configured, the working copy is the whole repository, and its history is deleted with it.
     icon: trash

backend/tasks/simple/sync-storage-targets.ts

@@ -21,9 +21,11 @@ import type { TaskContext } from '../../core/scheduler.ts'
  * must not stop the rest of them syncing. `executeAction` records the outcome on the target itself,
  * so a failure shows up on its Status card in the admin area rather than only in this log.
  *
- * **One instance at a time.** The scheduler hands a job to a single instance, which is the one whose
- * working copy is synced. Every instance in a high-availability set keeps its own, so they each fall
- * in step at their own turn rather than fighting over one repository.
+ * **Any instance will do.** The scheduler hands each tick to whichever instance claims it, and a
+ * target's working copy on that instance is only a cache: what the wiki has not yet published is
+ * queued in the database, and so is the commit it last agreed with the remote on, so a sync rebuilds
+ * the working copy from those wherever it runs. Two instances never sync the same target at once — one
+ * that finds another already at it reports so and leaves it be.
  */
 export async function task(_payload: unknown, { signal }: TaskContext): Promise<void> {
   const syncable = await WIKI.models.storage.syncableTargets()

dev/chart/README.md

@@ -107,8 +107,11 @@ included, so no sticky sessions are needed.
 
 The data directory (`/wiki/data`) is per replica unless `persistence` points every replica at one
 ReadWriteMany claim. That does not matter for the caches it normally holds, which are rebuilt from
-the database. It does matter for a **Local Disk** or **Git** storage target left at its default
-location under `data/`, which would otherwise be a separate copy on each replica.
+the database, nor for a **Git** storage target with a remote: each replica's working copy under
+`data/repo` is a cache too, rebuilt from the remote and from the changes queued in the database, so
+any replica can sync and none of them needs a volume. It does matter for a **Local Disk** storage
+target left at its default location under `data/`, and for a Git target with no remote configured,
+either of which would otherwise be a separate copy on each replica.
 
 With one replica on a ReadWriteOnce claim, set `strategy.type: Recreate`. Otherwise a rolling
 update's new pod may land on a node the volume cannot be attached to.

dev/chart/values.yaml

@@ -184,16 +184,17 @@ volumeMounts: []
 #    subPath: known_hosts
 #    readOnly: true
 
-# The wiki's data directory, /wiki/data. Holds the caches (icons, served files, blocks), which rebuild
-# themselves from the database, plus the default locations of the Local Disk (`data/content`) and Git
-# (`data/repo`) storage targets. An emptyDir is enough unless one of those two is in use.
+# The wiki's data directory, /wiki/data. Holds the caches (icons, served files, blocks, and a Git
+# storage target's working copy), which rebuild themselves from the database and the remote, plus the
+# default location of the Local Disk storage target (`data/content`). An emptyDir is enough unless
+# that target is in use, or a Git target with no remote configured.
 persistence:
   enabled: false
   # Use an existing claim instead of creating one.
   existingClaim: ''
   # `-` for the cluster's default class, empty to leave it unset.
   storageClass: ''
-  # ReadWriteMany is what lets several replicas share one Local Disk or Git storage target.
+  # ReadWriteMany is what lets several replicas share one Local Disk storage target.
   accessModes:
     - ReadWriteOnce
   size: 10Gi