feat: add prerendering for unautheticated requests

CLAUDE.md

@@ -86,8 +86,10 @@ path in silence.
   `WIKI` global (config + logger + lazy `ensureDb()`) and dynamically imports the task.
 - `base.yml` — system defaults for every config key. Do not edit as a user-facing config; it defines
   the shape merged with `config.yml` and the db `settings` table.
-- `helpers/` — small pure utilities (`common.ts`, `config.ts`), plus `storageFiles.ts`, which is the
-  file-tree half of the storage modules that address content by path (see [Storage targets](#storage-targets)).
+- `helpers/` — small pure utilities (`common.ts`, `config.ts`), plus two that are not: `storageFiles.ts`,
+  the file-tree half of the storage modules that address content by path (see
+  [Storage targets](#storage-targets)), and `appShell.ts`, which describes a page in the HTML document
+  served for it (see [The app shell and SEO](#the-app-shell-and-seo)).
 - `types/` — ambient declarations: `global.d.ts` (the `WIKI` global) and `fastify.d.ts` (session +
   route-permission augmentations).
 - `locales/` — `en.json` source strings (CrowdIn-managed) + `metadata.js` language table (the one
@@ -785,6 +787,89 @@ store; no SVG is ever written into content.
 - Picking an icon calls `POST /_api/icons/materialize`, which is what guarantees the wiki can serve it
   afterwards without the Iconify API.
 
+### The app shell and SEO
+
+The compiled SPA is one document for every path on the wiki, served by the `setNotFoundHandler` in
+`index.ts` — which is the fallback rather than a route because a page lives at any path a user cares
+to give it, and the frontend's router is what resolves one.
+
+That document says nothing about the page at its URL until a browser has run it — a `<title>` reading
+`Wiki.js` for every page of every wiki — and **plenty of clients never run it**: a chat client
+building an unfurl card, an AI crawler (GPTBot, ClaudeBot, PerplexityBot and friends fetch raw HTML
+and render nothing), a search engine that does not render, a reader with JavaScript off.
+`helpers/appShell.ts` is what puts something in the document, and `renderAppShell` is the whole of
+the decision.
+
+**There is no server-side rendering here, and none is wanted.** A page's HTML is already a string in
+`pages.render`, produced once in the editor's browser at save time — so describing a page is reading a
+row and two string insertions, not running Vue, a component tree or a second build. Anything that
+proposes rendering on the server is solving a problem this schema does not have.
+
+**Every request gets a head describing the page at its URL.** What separates the two kinds of
+document is whether the client will run the app, and each has its own function:
+
+- **`fragmentsForCrawler`** — for a client that will not. The PUBLIC's view of the page
+  (`pages.describePageForPublic`, which is `listForSitemap`'s question asked of one page): a head, the
+  page's markup appended, and 404 where the public may read nothing there. What goes in is what the
+  GUESTS group may read and nothing else, which is what makes it the same document for whoever asked
+  — and therefore the only half that is cached and handed on.
+- **`fragmentsForBrowser`** — for a browser that will. The head alone, describing the page as THAT
+  requester may see it (`pages.describePageForRequest`), which makes exactly the cut the page route
+  makes: `read:pages` per path, and any signed-in session sees an unpublished page while an API key is
+  answered as the public is. Never cached, because the answer is one reader's. It carries no body (the
+  app is about to draw the page properly, so a copy is bytes on every hard navigation for content the
+  browser discards), never 404s (whether a path this reader may create is empty is the app's own flow
+  to present) and adds no page-specific `noindex`.
+
+  It matters even though the app sets the title itself a moment later: the document's own title is
+  what the tab reads while the bundle loads, and what a bookmark or a history entry made before boot
+  finishes keeps for ever.
+- **The head is columns, not prose**: `<title>` composed exactly as `MainLayout.vue` composes it so
+  the tab does not jump when the app boots, `description`, a canonical link, `hreflang` alternates
+  from the page's locale group, and the `og:`/`twitter:` pair — which is what a link pasted into
+  Slack or Discord reads, and the surface this was most visibly missing.
+- **The body is the stored render with everything that would *run* taken out** — `<script>`,
+  `<style>`, inline handlers (`stripActiveMarkup`). A page whose author holds `write:scripts` keeps
+  those in its render and the app runs them when it draws the page; a copy of the same markup in the
+  document as the browser parses it would run them a second time, before the app exists. Not a
+  sanitizer — the render was sanitized at save time — just the same markup with less in it.
+- **It lands in `#wiki-prerender`, which both sides know about.** `frontend/index.html` hides it and
+  restyles it inside `<noscript>`, so a reader with JavaScript off gets the page as prose; `main.js`
+  removes the element before Vue mounts. The `<noscript>` typography puts back the browser defaults
+  Tailwind's preflight took away and is deliberately not a copy of the content styles, which belong
+  to elements the app draws.
+- **An anonymous request to a page path with nothing public at it answers 404**, where the shell used
+  to answer 200 everywhere — which is what taught a crawler that every URL on the wiki exists. No
+  page, an unpublished one and one the guests group's rules refuse are one answer, since the
+  difference is not something to tell whoever is asking. **The site root is the exception and always
+  answers 200**: there is something to show there whatever the database says — the welcome screen of a
+  wiki with no home page yet, a login form on a wiki that is not public — and a root answering 404
+  would report a working instance as broken to every uptime check pointed at it. Note the consequence
+  on a **private wiki**, where the guests group denies everything by default: every page path answers
+  404 to an anonymous client, which is the truthful answer and matches the empty sitemap such a site
+  already serves.
+- **Indexing is one header and never a tag.** `robotsTagFor` folds the site's **General → SEO**
+  settings together with the document's own say — a page marked not searchable, a path with nothing
+  public at it, and every URL that is not a page at all (`/_admin`, `/_search`: an interface, not
+  content) are all `noindex`. `X-Robots-Tag` rather than `<meta name="robots">` because it reaches
+  the same clients and cannot fall out of step with itself.
+- **The fragments are cached and the shell is not.** `WIKI.cache` holds the head and body per
+  origin-and-path for ten minutes — the sitemap's figure, for the sitemap's reason — while the shell
+  is re-read per request so that `npm run build` in `frontend/` takes effect immediately. The key
+  carries the request's own host, because a canonical link does; hence the crude
+  `SHELL_CACHE_MAX_ENTRIES` ceiling, without which anybody could grow the cache by inventing
+  hostnames. `invalidateAppShellCache()` is called from every page mutation and from the two places
+  `invalidateSitemaps()` is. Nothing a particular requester holds ever reaches it: the key has no
+  session dimension because the only thing cached has no requester in it.
+- **`groups.actorForPublic()` is the public as an actor**, and is deliberately separate from
+  `actorForRequest`: it is the guests group with no group-wide permissions, asked without a request in
+  hand. Both things that get cached and handed on — the sitemap and the public document — are built
+  from it, and neither may be built from anything one requester happens to hold.
+
+What already existed and is unchanged: `controllers/rootFiles.ts` serves `robots.txt` and
+`sitemap.xml` (with `hreflang` alternates), so **discovery** was never the missing half — the
+document was.
+
 ### Audit log
 
 Every action a **person** takes is one row in `auditLog` — `userId`, `clientIP`, `ts`, `kind`

backend/controllers/rootFiles.ts

@@ -1,6 +1,7 @@
 import { chunk } from 'es-toolkit/array'
-import type { FastifyInstance, FastifyReply, FastifyRequest } from 'fastify'
+import type { FastifyInstance, FastifyReply } from 'fastify'
 import type { SitemapPage } from '../models/pages.ts'
+import { originOf } from '../helpers/common.ts'
 
 /**
  * The sitemap protocol's own ceiling: 50,000 URLs, and 50MB uncompressed, per file. Past it the
@@ -29,19 +30,14 @@ function xmlEscape(value: string): string {
     .replaceAll("'", ''')
 }
 
-/**
- * The origin a `<loc>` — or robots.txt's `Sitemap:` line — is written against.
- *
- * The requester's own, and deliberately not the site's configured hostname: a sitemap may only list
- * URLs on the host it was itself fetched from — a crawler discards the rest as a cross-submission —
- * so the host in the request IS the answer, whether the site is bound to it or is the catch-all `*`.
- * It comes off a header and is therefore whatever the client said, which is why it is escaped before
- * it reaches the XML. It cannot carry a line break into robots.txt: a header value holding one is
- * rejected by the HTTP parser long before this.
- */
-function originOf(req: FastifyRequest): string {
-  return `${req.protocol}://${req.host}`
-}
+/*
+  `originOf` (helpers/common.ts) is the origin every `<loc>` — and robots.txt's `Sitemap:` line — is
+  written against: a sitemap may only list URLs on the host it was itself fetched from, since a
+  crawler discards the rest as a cross-submission, so the host in the request IS the answer whether
+  the site is bound to it or is the catch-all `*`. It comes off a header and is therefore whatever the
+  client said, which is why it is escaped before it reaches the XML. It cannot carry a line break into
+  robots.txt: a header value holding one is rejected by the HTTP parser long before this.
+*/
 
 /**
  * Where a page is, absolute and ready to be written into an element.

backend/helpers/appShell.ts

@@ -0,0 +1,487 @@
+import * as cheerio from 'cheerio'
+import type { FastifyRequest } from 'fastify'
+import type { PageDescription } from '../models/pages.ts'
+import { htmlEscape, isPageUrl, normalizePagePath, originOf, splitLocalePath } from './common.ts'
+
+/**
+ * What the app shell is enriched with before it is handed to a client that will not run it.
+ *
+ * The compiled SPA is one document for every path on the wiki: a `<title>` reading `Wiki.js`, no
+ * description, and an empty `<div id="app">` that only means something once a browser has run the
+ * bundle. Anything that does not — a chat client building an unfurl card, an AI crawler, a search
+ * engine that does not render, a reader with JavaScript off — sees exactly that, for every page.
+ *
+ * This is the cheap half of the fix, and it is cheap because there is nothing to render: a page's
+ * HTML is already a string in `pages.render`, produced once in the editor at save time. So a document
+ * for a crawler is the shell plus that string plus a handful of meta tags, and the server never runs
+ * a renderer, a component tree or a second build to produce one. See the note on caching below for
+ * what a scrape actually costs.
+ *
+ * There are two kinds of document, and which one a request gets turns on whether it will run the app:
+ *
+ *  - **A client that will not** — `fragmentsForCrawler` — gets the public's view of the page: a head
+ *    describing it, its markup appended, and 404 where the public may read nothing there. What goes
+ *    in is what the GUESTS group may read and nothing else, which is what makes it the same document
+ *    for whoever asked, and therefore the half that is cached and handed on.
+ *  - **A browser that will** — `fragmentsForBrowser` — gets the head alone, describing the page as
+ *    THAT reader may see it. It still matters that the title is right: the document's own title is
+ *    what the tab reads while the bundle loads and what a bookmark made before it finishes keeps.
+ *
+ * Two further rules hold across both:
+ *
+ *  - **Nothing is rendered, and nothing runs.** The injected copy is the stored render with its
+ *    scripts and styles taken out — see `stripActiveMarkup`.
+ *  - **Nothing a requester holds reaches the cache.** Only the public half is cached, keyed by origin
+ *    and path with no session dimension, because there is nothing in it that varies by requester.
+ */
+
+/** Namespaced so the whole lot can be dropped without knowing which hosts or paths are in it. */
+const SHELL_CACHE_PREFIX = 'appShell:'
+
+/**
+ * How long an assembled fragment set is held, in seconds.
+ *
+ * The same figure as the sitemap's, for the same reason: a title or a description a few minutes out
+ * of date misleads nobody, and the alternative is reading a page out of the database for every scrape
+ * of it. What waits for the TTL is an edit showing up in an unfurl card — the wiki itself never shows
+ * a stale page, since everyone who can edit one is logged in and is served the live app.
+ */
+const SHELL_CACHE_TTL = 600
+
+/**
+ * How many fragment sets are held at once.
+ *
+ * A crude ceiling rather than an eviction order: at the limit the whole namespace goes and fills
+ * again. Nothing here is worth the bookkeeping a least-recently-used cache would need — a wiki whose
+ * public traffic fits in this many pages gets a perfect hit rate, and one being crawled end to end
+ * gets little from caching either way, since a crawler fetches each page once.
+ *
+ * Having a ceiling at all is the point, and it is not about a wiki's size: the key carries the
+ * request's own host (see `cacheKeyFor`), so without one anybody could grow this without limit by
+ * asking for the same page under a made-up hostname.
+ */
+const SHELL_CACHE_MAX_ENTRIES = 500
+
+/** The site root, as the path below a locale prefix comes back — `/fr` alone is `/` in French. */
+const ROOT_PATHS = new Set(['', '/'])
+
+/** The page a site's root addresses. Mirrors `normalizePath` in the frontend's page store. */
+const HOME_PATH = 'home'
+
+/** The element the injected copy is wrapped in. `frontend/index.html` styles it; `main.js` removes it. */
+const PRERENDER_ID = 'wiki-prerender'
+
+/** What the shell is enriched with, before it is put into the document. */
+interface ShellFragments {
+  /** Replaces the shell's `<title>` and is appended to its `<head>`. */
+  head: string
+  /** Appended to the shell's `<body>`. Empty where there is no page body to show. */
+  body: string
+  /** 200, or 404 for a page path with nothing a reader without a session may read at it. */
+  status: number
+  /** The `X-Robots-Tag` to answer with, or null where a crawler's default is what the site wants. */
+  robots: string | null
+}
+
+export interface AppShellDocument {
+  html: string
+  status: number
+  robots: string | null
+}
+
+/**
+ * Drop every enriched fragment set.
+ *
+ * Called wherever something that shapes a public document changes: a page (its title, its body, its
+ * indexability, where it sits), a site's settings (its own title and description, whether it wants to
+ * be indexed, how it brackets URLs by locale) or a group's rules (what the public may read at all).
+ * The last of those is a permissions question rather than a freshness one, which is why it cannot be
+ * left to the TTL — the same reasoning as `invalidateSitemaps`, and the same call sites.
+ */
+export function invalidateAppShellCache(): void {
+  WIKI.cache.del(WIKI.cache.keys().filter((key) => key.startsWith(SHELL_CACHE_PREFIX)))
+}
+
+/**
+ * Whether this request will boot the app, or is something that will only ever read the document.
+ *
+ * A verified API key counts as a session for this, as it does everywhere else: a caller holding one
+ * has an identity, and enriching its document with the public's view of a page would be answering a
+ * different question from the one it asked.
+ */
+function isAnonymous(req: FastifyRequest): boolean {
+  return !req.session?.authenticated && !req.apiKey
+}
+
+/**
+ * The cache identity of a document.
+ *
+ * The host is part of it because the document is: a canonical link and an unfurl card carry absolute
+ * URLs, and those are built against the host the request arrived on (`originOf`). A site bound to the
+ * catch-all `*` answers on any number of hostnames, so two requests for the same page are not
+ * necessarily the same document. Hence also `SHELL_CACHE_MAX_ENTRIES`.
+ */
+function cacheKeyFor(origin: string, urlPath: string): string {
+  return `${SHELL_CACHE_PREFIX}${origin}${urlPath}`
+}
+
+/**
+ * What a site tells a crawler about a document it has already fetched, as an `X-Robots-Tag` value.
+ *
+ * The other half of the **General → SEO** settings, and the half that carries what robots.txt cannot:
+ * that file can only say whether to CRAWL a path (see `controllers/rootFiles.ts`), while `noindex`
+ * and `nofollow` are instructions about a document in hand. A search engine reads this header exactly
+ * as it reads a `<meta name="robots">` tag, and — unlike a tag the frontend would set once it booted —
+ * it is there for a crawler that does not run the page's JavaScript. It is the header and not a tag
+ * for that reason, and it is only the header so that the two can never disagree.
+ *
+ * `noindex` is also the setting's only thorough form. `Disallow: /` keeps a crawler off the page, but
+ * a page nobody fetched can still be listed from its inbound links alone; this is what says not to
+ * list it.
+ *
+ * @param indexable False for a document there is no sense in listing whatever the site says: a page
+ *   marked as not searchable, one with nothing public at it, and every URL that is not a page at all —
+ *   the app's own screens, which are an interface and not content.
+ *
+ * @returns Null when the site wants both and the document allows it, which is every crawler's default
+ *   anyway: no header says the same thing as `index, follow`, and a wiki that wants to be found should
+ *   not have to repeat it on every response.
+ */
+function robotsTagFor(siteId: string | undefined, indexable: boolean): string | null {
+  // -> A host matching no site at all is still handed the app shell, and is told not to index it:
+  //    there is no site here whose settings could say otherwise
+  const robots = siteId ? WIKI.sites[siteId]?.config?.robots : undefined
+  const index = Boolean(robots?.index) && indexable
+  const follow = Boolean(robots?.follow)
+  if (index && follow) {
+    return null
+  }
+  return `${index ? 'index' : 'noindex'}, ${follow ? 'follow' : 'nofollow'}`
+}
+
+/**
+ * Which page a URL addresses, as the database holds it.
+ *
+ * The same reading the frontend router gives it, which is what keeps the injected document and the
+ * app that replaces it about the same page: `splitLocalePath` takes off a locale prefix if the first
+ * segment names one of the site's locales, and what is left is a page path. A path arriving without a
+ * prefix is in the site's primary locale — the request hook in `index.ts` is what redirects one that
+ * should have had a prefix, and it runs before this.
+ */
+function resolvePagePath(
+  siteId: string | undefined,
+  urlPath: string
+): { locale: string; path: string; isRoot: boolean } {
+  const locales = siteId ? WIKI.sites[siteId]?.config?.locales : undefined
+  const split = splitLocalePath(urlPath, WIKI.models.locales.urlPrefixesFor(locales?.active))
+  const below = split?.path ?? urlPath
+  return {
+    locale: split?.locale ?? locales?.primary ?? 'en',
+    // -> The site root is the `home` page rather than an empty path, the same as everywhere else
+    path: normalizePagePath(below) || HOME_PATH,
+    isRoot: ROOT_PATHS.has(below)
+  }
+}
+
+/**
+ * The stored render with everything that would *run* taken out.
+ *
+ * A page whose author holds `write:scripts` or `write:styles` keeps its `<script>` and `<style>` in
+ * the render, and the app runs them when it displays the page. The same markup sitting in the document
+ * as the browser parses it would run them a second time, before the app exists and outside anything
+ * that knows about the page — so what goes into the shell is the prose and nothing else. Inline
+ * handlers, `<template>` and `<link>` go for the same reason.
+ *
+ * Not a security boundary, and not a second sanitizer: the render was sanitized at save time against
+ * what its author was allowed to embed (`models/rendering.ts`), and this is that same markup with
+ * less in it. What this prevents is one page being executed twice.
+ */
+function stripActiveMarkup(html: string): string {
+  const $ = cheerio.load(html, null, false)
+  $('script, style, link, template').remove()
+  for (const el of $('*')) {
+    // -> `$('*')` only ever yields elements, but its element type is the union every node could be
+    for (const attr of Object.keys((el as { attribs?: Record<string, string> }).attribs ?? {})) {
+      if (attr.toLowerCase().startsWith('on')) {
+        $(el).removeAttr(attr)
+      }
+    }
+  }
+  return $.html()
+}
+
+/** One `<meta>`, or nothing at all for a value the site or the page has not filled in. */
+function metaTag(kind: 'name' | 'property', key: string, value: string | null | undefined): string {
+  return value ? `<meta ${kind}="${key}" content="${htmlEscape(value)}">` : ''
+}
+
+/**
+ * The absolute URL of a page, which is what a canonical link and an unfurl card name it by.
+ *
+ * `urlFor` decides the path, the same as for every link the wiki makes of its own pages and for every
+ * `<loc>` in the sitemap — so the two documents a crawler reads agree about where a page is, locale
+ * prefix and all. A page path is held to `[a-zA-Z0-9-_/]` when it is saved, but a locale's short code
+ * is an administrator's to alias, so the result is percent-encoded before it is escaped.
+ */
+function urlOf(origin: string, siteId: string, locale: string, path: string): string {
+  return `${origin}${encodeURI(WIKI.models.pages.urlFor(siteId, locale, path))}`
+}
+
+/**
+ * The head of a document describing one page.
+ *
+ * What a crawler and an unfurl card actually read, and all of it is a column: the title, the
+ * description, when the page last changed, and the locales it exists in. `og:` and `twitter:` are
+ * here because no chat client reads anything else — a link pasted into Slack or Discord is the most
+ * common way a wiki page is shared, and it is the surface this was most visibly missing.
+ *
+ * The title is composed exactly as `MainLayout.vue` composes it, so the tab does not change under a
+ * reader the moment the app boots.
+ *
+ * There is deliberately no `<meta name="robots">`: that is the `X-Robots-Tag` above, which reaches
+ * the same clients and cannot fall out of step with itself.
+ */
+function headForPage(
+  origin: string,
+  siteId: string,
+  siteConfig: any,
+  page: PageDescription
+): string {
+  const siteTitle: string = siteConfig?.title || 'Wiki.js'
+  const url = urlOf(origin, siteId, page.locale, page.path)
+  const description = page.description || siteConfig?.description || ''
+  // -> Only a logo an administrator actually put there: the route falls back to the Wiki.js mark for a
+  //    site with none, and a card carrying that says less than a card carrying no image
+  const image =
+    siteConfig?.logoUrl || (siteConfig?.assets?.logo ? `${origin}/_site/current/logo` : '')
+
+  return [
+    `<title>${htmlEscape(`${page.title} - ${siteTitle}`)}</title>`,
+    metaTag('name', 'description', description),
+    `<link rel="canonical" href="${htmlEscape(url)}">`,
+    ...page.alternates.map(
+      (alt) =>
+        `<link rel="alternate" hreflang="${htmlEscape(alt.locale)}" href="${htmlEscape(urlOf(origin, siteId, alt.locale, alt.path))}">`
+    ),
+    metaTag('property', 'og:type', 'article'),
+    metaTag('property', 'og:site_name', siteTitle),
+    metaTag('property', 'og:title', page.title),
+    metaTag('property', 'og:description', description),
+    metaTag('property', 'og:url', url),
+    metaTag('property', 'og:image', image),
+    metaTag(
+      'property',
+      'article:modified_time',
+      page.updatedAt.toTemporalInstant().toString({
+        smallestUnit: 'second'
+      })
+    ),
+    metaTag('name', 'twitter:card', image ? 'summary' : null)
+  ]
+    .filter(Boolean)
+    .join('\n    ')
+}
+
+/**
+ * The head of a document that is not a page: the site itself, and nothing more specific.
+ *
+ * What a URL with no page at it gets, so that a 404 and a wiki whose home page has not been written
+ * yet still carry the site's own name rather than `Wiki.js`.
+ */
+function headForSite(siteConfig: any): string {
+  const siteTitle: string = siteConfig?.title || 'Wiki.js'
+  return [
+    `<title>${htmlEscape(siteTitle)}</title>`,
+    metaTag('name', 'description', siteConfig?.description),
+    metaTag('property', 'og:type', 'website'),
+    metaTag('property', 'og:site_name', siteTitle),
+    metaTag('property', 'og:title', siteTitle)
+  ]
+    .filter(Boolean)
+    .join('\n    ')
+}
+
+/**
+ * The page's own markup, for a client that will not run the app.
+ *
+ * Wrapped in one element with a known id, which is what lets both sides deal with it: the stylesheet
+ * in `frontend/index.html` keeps it out of sight of anyone whose browser is about to draw the real
+ * thing and shows it to anyone whose browser is not, and `main.js` takes it out of the document
+ * before Vue mounts, so nothing the app does has to know it was ever there.
+ *
+ * The title is deliberately not repeated above the body as a heading. A page written the ordinary way
+ * opens with its own `# Title`, so a synthesized one is the same words twice — and where a page has
+ * no heading of its own, the title is still in `<title>` and `og:title`, which is where every client
+ * that cares about it looks.
+ */
+function bodyForPage(page: PageDescription): string {
+  if (!page.render) {
+    return ''
+  }
+  return `<div id="${PRERENDER_ID}">${stripActiveMarkup(page.render)}</div>`
+}
+
+/**
+ * The fragments for a client that will never run the app — a crawler, an unfurl card, a reader with
+ * JavaScript off.
+ *
+ * The public's view of the page and nobody else's, which is what makes this the one that is cached
+ * and handed on: it is the same document whoever fetched it. It is also the only kind that carries a
+ * body and the only kind that ever answers 404.
+ */
+async function fragmentsForCrawler(
+  req: FastifyRequest,
+  siteId: string | undefined,
+  urlPath: string
+): Promise<ShellFragments> {
+  const siteConfig = siteId ? WIKI.sites[siteId]?.config : undefined
+
+  /*
+    Everything that is not a page path is the app's own interface — `/_admin`, `/_profile`, `/_edit`,
+    the search screen — and there is nothing in any of it for a search engine to list. Told so
+    explicitly rather than left to a crawler's judgement, and given no description of its own: these
+    screens have no content until the app has run, and a site that wants to be indexed should not have
+    its admin area competing with its pages for the result.
+  */
+  if (!siteId || !isPageUrl(urlPath)) {
+    return {
+      head: headForSite(siteConfig),
+      body: '',
+      status: 200,
+      robots: robotsTagFor(siteId, false)
+    }
+  }
+
+  const { locale, path, isRoot } = resolvePagePath(siteId, urlPath)
+  const page = await WIKI.models.pages.describePageForPublic(siteId, { locale, path })
+
+  if (!page) {
+    /*
+      Nothing here that the public may read — no page, an unpublished one, or one the guests group's
+      rules refuse. All three are one answer, and the answer is 404: the shell answering 200 for every
+      path is what tells a crawler that a mistyped URL is a page, and a wiki can be enumerated
+      indefinitely on the strength of it.
+
+      The site root is the exception and always answers 200. There is something to show there whatever
+      the database says — the welcome screen of a wiki whose home page has not been written yet, or a
+      login form on a wiki that is not public at all — and a root that answered 404 would report a
+      working instance as broken to every uptime check pointed at it.
+    */
+    return {
+      head: headForSite(siteConfig),
+      body: '',
+      status: isRoot ? 200 : 404,
+      robots: robotsTagFor(siteId, false)
+    }
+  }
+
+  return {
+    head: headForPage(originOf(req), siteId, siteConfig, page),
+    body: bodyForPage(page),
+    status: 200,
+    robots: robotsTagFor(siteId, page.isIndexable)
+  }
+}
+
+/**
+ * The fragments for a browser that is about to boot the app.
+ *
+ * The head alone, describing the page as THIS requester may see it — an unpublished page included, a
+ * page the public may not read included, and nothing they could not have opened themselves.
+ * `describePageForRequest` makes exactly the cut the page route makes, so the title in the document
+ * and the page the app then draws can never be about different things.
+ *
+ * It matters even though the app will set the title itself a moment later: the document's own title is
+ * what the tab reads while the bundle loads, what a bookmark or a history entry made before it
+ * finishes boot keeps for ever, and what a `Ctrl+D` on a slow connection saves. `Wiki.js` for every
+ * page of every wiki is simply wrong there.
+ *
+ * Three things it deliberately does not do. No body: the app is about to draw the page properly, so a
+ * copy is bytes on every hard navigation for content the browser will discard. No 404: the difference
+ * between a path with no page and one this reader may create is the app's own flow to present, and it
+ * needs the document to say 200. And no page-specific `noindex` — nothing will index a document
+ * fetched with a session, and a header invented here would be describing that session rather than the
+ * wiki.
+ *
+ * Never cached. The answer depends on who asked, which is exactly what the cached public document may
+ * not do; one read of one row per hard navigation is the price, and the app is about to fetch the same
+ * page over the API regardless.
+ */
+async function fragmentsForBrowser(
+  req: FastifyRequest,
+  siteId: string | undefined,
+  urlPath: string
+): Promise<ShellFragments> {
+  const siteConfig = siteId ? WIKI.sites[siteId]?.config : undefined
+  const robots = robotsTagFor(siteId, true)
+
+  if (!siteId || !isPageUrl(urlPath)) {
+    return { head: headForSite(siteConfig), body: '', status: 200, robots }
+  }
+
+  const { locale, path } = resolvePagePath(siteId, urlPath)
+  const page = await WIKI.models.pages.describePageForRequest(siteId, { locale, path }, req)
+
+  return {
+    head: page ? headForPage(originOf(req), siteId, siteConfig, page) : headForSite(siteConfig),
+    body: '',
+    status: 200,
+    robots
+  }
+}
+
+/**
+ * The document to answer a request for the app shell with.
+ *
+ * Every request gets a head describing the page at its URL; which page that is, and what else travels
+ * with it, is what `fragmentsForCrawler` and `fragmentsForBrowser` differ about.
+ *
+ * Only the public half is cached, and the shell is never cached: the shell is re-read per request so
+ * that `npm run build` in `frontend/` takes effect immediately, which a cached whole document would
+ * have delayed by the TTL, and the two string insertions that combine them are nothing next to a
+ * database read. Both insertions use a replacer function rather than a replacement string — a page
+ * containing `$&` would otherwise rewrite itself as it was inserted.
+ *
+ * @param shell The compiled `assets/index.html`, as read for this request
+ */
+export async function renderAppShell(
+  req: FastifyRequest,
+  siteId: string | undefined,
+  shell: string
+): Promise<AppShellDocument> {
+  const urlPath = req.raw.url!.split('?')[0]!
+  const fragments = isAnonymous(req)
+    ? await publicFragments(req, siteId, urlPath)
+    : await fragmentsForBrowser(req, siteId, urlPath)
+
+  /*
+    The head replaces the shell's own `<title>` where there is one and is appended to its `<head>`
+    where there is not, so that a shell built without one is enriched rather than silently skipped.
+  */
+  const withoutTitle = shell.replace(/[ \t]*<title>[\s\S]*?<\/title>\n?/i, '')
+  const html = withoutTitle
+    .replace('</head>', () => `  ${fragments.head}\n  </head>`)
+    .replace('</body>', () => `${fragments.body}</body>`)
+
+  return { html, status: fragments.status, robots: fragments.robots }
+}
+
+/** The cached half: one URL's public fragments, keeping the namespace under its ceiling. */
+async function publicFragments(
+  req: FastifyRequest,
+  siteId: string | undefined,
+  urlPath: string
+): Promise<ShellFragments> {
+  const key = cacheKeyFor(originOf(req), urlPath)
+  const cached = WIKI.cache.get<ShellFragments>(key)
+  if (cached) {
+    return cached
+  }
+  const fragments = await fragmentsForCrawler(req, siteId, urlPath)
+  const held = WIKI.cache.keys().filter((k) => k.startsWith(SHELL_CACHE_PREFIX)).length
+  if (held >= SHELL_CACHE_MAX_ENTRIES) {
+    invalidateAppShellCache()
+  }
+  WIKI.cache.set(key, fragments, SHELL_CACHE_TTL)
+  return fragments
+}

backend/helpers/common.ts

@@ -3,7 +3,7 @@ import { startCase } from 'es-toolkit/string'
 import crypto from 'node:crypto'
 import mime from 'mime'
 import fs from 'node:fs'
-import type { FastifyReply } from 'fastify'
+import type { FastifyReply, FastifyRequest } from 'fastify'
 
 export interface Deferred<T = void> {
   resolve: (value: T) => void
@@ -73,6 +73,49 @@ export function createDeferred<T = void>(): Deferred<T> {
  */
 export const RESERVED_ROOT_FILES = new Set(['favicon.ico', 'robots.txt', 'sitemap.xml'])
 
+/**
+ * Whether a URL addresses the page tree rather than the server itself.
+ *
+ * Everything the server mounts sits under a leading-underscore segment — `/_api`, `/_assets`,
+ * `/_files`, and the rest registered in `initHTTPServer` — which is what makes the distinction a
+ * prefix test rather than a list to keep in step with the routes.
+ *
+ * Note that the answer is about the URL and not about what is there: a page path with no page at it
+ * is still a page path, which is what lets the app shell answer 404 for one.
+ */
+export function isPageUrl(urlPath: string): boolean {
+  const firstSegment = urlPath.split('/')[1] ?? ''
+  return !firstSegment.startsWith('_') && !RESERVED_ROOT_FILES.has(firstSegment.toLowerCase())
+}
+
+/**
+ * The origin a URL this server writes into a document is built against.
+ *
+ * The requester's own, and deliberately not the site's configured hostname: a site may be bound to
+ * the catch-all `*` and have none, and every document this produces — a sitemap, a canonical link, an
+ * unfurl card — has to name the host it was itself fetched from or be discarded as pointing somewhere
+ * else. It comes off a header and is therefore whatever the client said, which is why everything
+ * built from it is escaped before it reaches a document.
+ */
+export function originOf(req: FastifyRequest): string {
+  return `${req.protocol}://${req.host}`
+}
+
+/**
+ * Escape a string for use as HTML text or inside a double-quoted attribute.
+ *
+ * Both at once, which is why the set is all five: an attribute needs the quotes and text needs the
+ * angle brackets, and a value that is safe in both is one fewer thing to get right per call site.
+ */
+export function htmlEscape(value: string): string {
+  return value
+    .replaceAll('&', '&amp;')
+    .replaceAll('<', '&lt;')
+    .replaceAll('>', '&gt;')
+    .replaceAll('"', '&quot;')
+    .replaceAll("'", '&#39;')
+}
+
 /**
  * Decode a tree path
  *

backend/index.ts

@@ -34,7 +34,13 @@ import configSvc from './core/config.ts'
 import dbManager from './core/db.ts'
 import logger from './core/logger.ts'
 import scheduler from './core/scheduler.ts'
-import { RESERVED_ROOT_FILES, splitLocalePath, stripPageExtension } from './helpers/common.ts'
+import { renderAppShell } from './helpers/appShell.ts'
+import {
+  isPageUrl,
+  RESERVED_ROOT_FILES,
+  splitLocalePath,
+  stripPageExtension
+} from './helpers/common.ts'
 import { corsOrigin, parseCspDirectives } from './helpers/security.ts'
 
 const nanoid = customAlphabet('1234567890abcdef', 10)
@@ -85,65 +91,6 @@ function isServerUrl(urlPath: string): boolean {
   return SERVER_ROUTE_SEGMENTS.has(segments[1] ?? '')
 }
 
-/**
- * Whether a URL addresses the page tree rather than the server itself.
- *
- * Everything the server mounts sits under a leading-underscore segment — `/_api`, `/_assets`,
- * `/_files`, and the rest registered in `initHTTPServer` — which is what makes the distinction a
- * prefix test rather than a list to keep in step with the routes.
- */
-function isPageUrl(urlPath: string): boolean {
-  const firstSegment = urlPath.split('/')[1] ?? ''
-  return !firstSegment.startsWith('_') && !RESERVED_ROOT_FILES.has(firstSegment.toLowerCase())
-}
-
-/**
- * What a site tells a crawler about a document it has already fetched, as an `X-Robots-Tag` value.
- *
- * The other half of the **General → SEO** settings, and the half that carries what robots.txt cannot:
- * that file can only say whether to CRAWL a path (see `controllers/rootFiles.ts`), while `noindex`
- * and `nofollow` are instructions about a document in hand. A search engine reads this header exactly
- * as it reads a `<meta name="robots">` tag, and — unlike a tag the frontend would set once it booted —
- * it is there for a crawler that does not run the page's JavaScript.
- *
- * `noindex` is also the setting's only thorough form. `Disallow: /` keeps a crawler off the page, but
- * a page nobody fetched can still be listed from its inbound links alone; this is what says not to
- * list it.
- *
- * Null when both settings are on, which is every crawler's default anyway: no header says the same
- * thing as `index, follow`, and a wiki that wants to be found should not have to repeat it on every
- * response.
- */
-function robotsTagFor(siteId: string | undefined): string | null {
-  // -> A host matching no site at all is still handed the app shell, and is told not to index it:
-  //    there is no site here whose settings could say otherwise
-  const robots = siteId ? WIKI.sites[siteId]?.config?.robots : undefined
-  if (robots?.index && robots.follow) {
-    return null
-  }
-  return `${robots?.index ? 'index' : 'noindex'}, ${robots?.follow ? 'follow' : 'nofollow'}`
-}
-
-/**
- * The segments a site's locale-prefixed URLs may start with, mapped to the locale each names.
- *
- * Every code a locale answers to, not only the short one it is addressed by now: an alias an
- * administrator changed leaves the links people have already saved pointing at the old segment, and
- * a wiki that answers 404 to them has broken them. `localeForShortCode` is what knows the set.
- */
-function localePrefixesFor(activeCodes?: string[] | null): Map<string, string> {
-  const prefixes = new Map<string, string>()
-  for (const code of activeCodes ?? []) {
-    const locale = WIKI.cache?.get(`locale:${code}`) as any
-    for (const segment of [locale?.displayCode, locale?.derivedCode, code]) {
-      if (segment) {
-        prefixes.set(segment, code)
-      }
-    }
-  }
-  return prefixes
-}
-
 if (!semver.satisfies(process.version, '>=26')) {
   console.error('ERROR: Node.js 26.x or later required!')
   process.exit(1)
@@ -733,7 +680,7 @@ async function initHTTPServer() {
       */
       const siteLocales = WIKI.sites[siteId]?.config?.locales
       if (siteLocales?.forcePrefix) {
-        const prefixes = localePrefixesFor(siteLocales.active)
+        const prefixes = WIKI.models.locales.urlPrefixesFor(siteLocales.active)
         if (!splitLocalePath(trimmed, prefixes)) {
           const primary = WIKI.models.locales.shortCodeFor(siteLocales.primary)
           reply.redirect(withQuery(`/${primary}${trimmed === '/' ? '' : trimmed}`), 302)
@@ -826,20 +773,9 @@ async function initHTTPServer() {
     if (!isReadRequest || isSystemPath || isReservedRootFile) {
       return reply.notFound()
     }
+    let shell: string
     try {
-      const shell = await readFile(appShellPath, 'utf8')
-      /*
-        Every HTML document this wiki serves leaves through here — a page and an app route alike — so
-        this is the one place the site's indexing settings can be attached to all of them.
-
-        Straight off the site caches for the same reason the SEO hook above reads them that way: both
-        lookups are what `getSiteByHostname` would do, minus its optional reload.
-      */
-      const robotsTag = robotsTagFor(WIKI.sitesMappings[req.hostname] || WIKI.sitesMappings['*'])
-      if (robotsTag) {
-        reply.header('X-Robots-Tag', robotsTag)
-      }
-      return reply.header('Cache-Control', 'no-store').type('text/html; charset=utf-8').send(shell)
+      shell = await readFile(appShellPath, 'utf8')
     } catch (err: any) {
       // -> Nothing to serve means the frontend was never built, which is a setup step rather than a
       //    fault of this request: say which one, since a bare 500 sends people looking in the server
@@ -849,6 +785,38 @@ async function initHTTPServer() {
         .type('text/plain; charset=utf-8')
         .send('The frontend has not been built yet. Run `npm run build` in frontend/.\n')
     }
+
+    /*
+      Every HTML document this wiki serves leaves through here — a page and an app route alike — so this
+      is the one place the site's indexing settings can be attached to all of them, and the one place a
+      document can be made to say something about the page at its URL before the app has run.
+      `renderAppShell` is that: what it does, what it costs and who it does it for are all set out in
+      `helpers/appShell.ts`.
+
+      The site is read straight off the caches for the same reason the SEO hook above reads it that way:
+      both lookups are what `getSiteByHostname` would do, minus its optional reload.
+
+      A failure there is logged and the plain shell goes out instead. Enriching a document is for the
+      benefit of clients that will not run it, and a database that cannot answer for one is no reason to
+      stop serving the app to a reader whose browser would have rendered the page anyway.
+    */
+    const siteId = WIKI.sitesMappings[req.hostname] || WIKI.sitesMappings['*']
+    let doc = { html: shell, status: 200, robots: null as string | null }
+    try {
+      doc = await renderAppShell(req, siteId, shell)
+    } catch (err: any) {
+      WIKI.logger.warn(
+        `Cannot describe ${urlPath} for a client that will not render it: ${err.message}`
+      )
+    }
+    if (doc.robots) {
+      reply.header('X-Robots-Tag', doc.robots)
+    }
+    return reply
+      .code(doc.status)
+      .header('Cache-Control', 'no-store')
+      .type('text/html; charset=utf-8')
+      .send(doc.html)
   })
 
   // ----------------------------------------

backend/models/groups.ts

@@ -1,6 +1,7 @@
 import { v4 as uuid } from 'uuid'
 import { and, count, eq, ilike, ne, or, sql } from 'drizzle-orm'
 import { groups as groupsTable, userGroups, users as usersTable } from '../db/schema.ts'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
 import { CustomError } from '../helpers/common.ts'
 import { resolvePageRule, type RulePageRef } from '../helpers/pageRules.ts'
 import type { SystemIds } from './types.ts'
@@ -158,11 +159,13 @@ class Groups {
     for (const row of rows) {
       rulesCache[row.id] = (row.rules ?? []) as GroupRule[]
     }
-    // -> The sitemap is a list of what the GUESTS group may read, held for minutes at a time. Every
-    //    other consumer of these rules asks per request and is correct the moment this returns; that
-    //    one would go on publishing paths a rule had just taken away, which is a permission waiting
-    //    for a timer rather than a document being a little out of date
+    // -> Both of these are what the GUESTS group may read, held for minutes at a time: an index of
+    //    paths, and the page descriptions served to a client that will not run the app. Every other
+    //    consumer of these rules asks per request and is correct the moment this returns; those two
+    //    would go on publishing what a rule had just taken away, which is a permission waiting for a
+    //    timer rather than a document being a little out of date
     WIKI.models.pages.invalidateSitemaps()
+    invalidateAppShellCache()
     WIKI.logger.info(`Loaded page rules for ${rows.length} groups [ OK ]`)
   }
 
@@ -195,6 +198,23 @@ class Groups {
     }
   }
 
+  /**
+   * The actor the PUBLIC speaks for, which is not the same as a request that happens to be anonymous.
+   *
+   * The guests group and nothing else, asked without a request in hand — which is what "whatever
+   * anybody may read" means, and what makes an answer built from it identical for whoever fetched it.
+   * That is the whole reason it exists separately: the sitemap and the app shell's public document are
+   * both cached and handed to the next requester, so neither may be built from anything a particular
+   * requester holds.
+   *
+   * No group-wide permissions rather than the guests group's own: `checkAccess` reads that list only
+   * for `manage:system`, and a wiki that had somehow handed the public an administrator's permission
+   * should not also publish every page it has as a consequence.
+   */
+  actorForPublic(): AccessActor {
+    return { groupIds: [WIKI.data.systemIds.guestsGroupId], permissions: [] }
+  }
+
   /**
    * Whether this caller may do this to this page.
    *

backend/models/locales.ts

@@ -469,6 +469,29 @@ class Locales {
     return match?.code ?? segment
   }
 
+  /**
+   * The segments a site's locale-prefixed URLs may start with, mapped to the locale each names.
+   *
+   * Every code a locale answers to, not only the short one it is addressed by now: an alias an
+   * administrator changed leaves the links people have already saved pointing at the old segment, and
+   * a wiki that answers 404 to them has broken them — the same reasoning as `localeForShortCode`,
+   * built as a map because the request hooks that read a page URL do it on every request.
+   *
+   * @param activeCodes The locales the site offers, i.e. `config.locales.active`
+   */
+  urlPrefixesFor(activeCodes?: string[] | null): Map<string, string> {
+    const prefixes = new Map<string, string>()
+    for (const code of activeCodes ?? []) {
+      const locale = WIKI.cache?.get(`locale:${code}`) as any
+      for (const segment of [locale?.displayCode, locale?.derivedCode, code]) {
+        if (segment) {
+          prefixes.set(segment, code)
+        }
+      }
+    }
+    return prefixes
+  }
+
   async getLocales({ cache = true }: { cache?: boolean } = {}): Promise<any[]> {
     if (!WIKI.cache.has('locales') || !cache) {
       const locales = await WIKI.db

backend/models/pages.ts

@@ -6,6 +6,9 @@ import {
   normalizePagePath,
   timingSafeCompare
 } from '../helpers/common.ts'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
+import type { AccessActor } from './groups.ts'
+import type { FastifyRequest } from 'fastify'
 import type { RenderPermissions, TocNode } from './rendering.ts'
 import type { DeletedEntry } from './tree.ts'
 import type { StoragePageContent, StoragePageRef } from './storage.ts'
@@ -266,6 +269,40 @@ export interface SitemapPage {
   localeGroupId: string | null
 }
 
+/**
+ * A page as the HTML document served for it describes itself — see `describePageForPublic` and
+ * `describePageForRequest`.
+ *
+ * Everything here is scoped to whoever it was asked for: a page they may not read is not described at
+ * all, and neither are the translations of one they may not reach.
+ */
+export interface PageDescription {
+  locale: string
+  path: string
+  title: string
+  description: string | null
+  /**
+   * The stored render, or null where this page has no body to show a reader who has not asked for
+   * one: a password-protected page, whose body is exactly what the password covers, and a redirection,
+   * which has nowhere to keep one.
+   *
+   * Only wanted by the caller building a document for a client that will never run the app; a browser
+   * about to boot the SPA is served the head alone.
+   */
+  render: string | null
+  updatedAt: Date
+  /**
+   * Whether a search engine should list it — `isSearchable`, and off as well for the two cases with
+   * nothing to list. The same question the sitemap asks, answered for one page.
+   */
+  isIndexable: boolean
+  /**
+   * Every locale this page exists in and this requester may read, this one included. Empty for a page
+   * that is not part of a translation set, since an alternates list naming one document says nothing.
+   */
+  alternates: { locale: string; path: string }[]
+}
+
 export interface PageActor {
   id: string
   permissions: string[]
@@ -928,10 +965,7 @@ class Pages {
       //    next fetch as it did on the one that handed the crawler those numbers
       .orderBy(pagesTable.locale, pagesTable.path)
 
-    // -> No group-wide permissions rather than the guests group's own: `checkAccess` reads that list
-    //    only for `manage:system`, and a wiki that had somehow handed the public an administrator's
-    //    permission should not also publish an index of every page it has as a consequence
-    const guests = { groupIds: [WIKI.data.systemIds.guestsGroupId], permissions: [] }
+    const guests = WIKI.models.groups.actorForPublic()
     const pages = rows
       .filter((row) => WIKI.models.groups.checkAccess(guests, 'read:pages', row))
       .map(({ locale, path, updatedAt, localeGroupId }) => ({
@@ -945,6 +979,148 @@ class Pages {
     return pages
   }
 
+  /**
+   * What the PUBLIC may see at a path, for the HTML document served to a client that will not run the
+   * app.
+   *
+   * The question `listForSitemap` asks of a whole site, asked of one page: the guests group's rules
+   * decide and nothing about the requester does, which is what makes the answer the same for whoever
+   * fetched it and therefore safe to cache and hand on.
+   *
+   * Published only, as it is everywhere the public is being answered — which is also what lets the app
+   * shell say 404 rather than 200 at a path with nothing published at it.
+   */
+  async describePageForPublic(
+    siteId: string,
+    ref: { locale: string; path: string }
+  ): Promise<PageDescription | null> {
+    return this.describePage(siteId, ref, WIKI.models.groups.actorForPublic(), true)
+  }
+
+  /**
+   * What THIS REQUESTER may see at a path, for the document a browser is about to boot the app from.
+   *
+   * Deliberately the same cut the page route itself makes (`GET /sites/:siteId/pages/:pageIdOrHash`),
+   * so that the title in the document and the page the app then draws can never be about different
+   * things: its `read:pages` check per path, and its `publicOnly: !actorFrom(req)` — which is to say
+   * any signed-in session sees an unpublished page, and an API key is answered as the public is, since
+   * a draft belongs to the people working on it rather than to whatever holds a token.
+   *
+   * Not for caching across requesters: the answer is one reader's.
+   */
+  async describePageForRequest(
+    siteId: string,
+    ref: { locale: string; path: string },
+    req: FastifyRequest
+  ): Promise<PageDescription | null> {
+    const isSession = Boolean(req.session?.authenticated && req.session.user?.id)
+    return this.describePage(siteId, ref, WIKI.models.groups.actorForRequest(req), !isSession)
+  }
+
+  /**
+   * The read behind both, which is the only thing that should call it.
+   *
+   * `actor` and `publicOnly` are paired by the two methods above rather than left to a caller, because
+   * the wrong pairing — the public actor with unpublished pages included — would describe a draft to
+   * whoever asked.
+   */
+  private async describePage(
+    siteId: string,
+    { locale, path }: { locale: string; path: string },
+    actor: AccessActor,
+    publicOnly: boolean
+  ): Promise<PageDescription | null> {
+    const conditions = [
+      eq(pagesTable.siteId, siteId),
+      eq(pagesTable.locale, locale),
+      // -> By path, and not by the hash `getPage` addresses a page with: that hash is 53 bits and is
+      //    never checked against the path it stands for, which is a collision a document served to
+      //    whoever asked simply need not have — nothing here was handed a hash to look up
+      eq(pagesTable.path, path)
+    ]
+    if (publicOnly) {
+      conditions.push(eq(pagesTable.publishState, 'published'))
+    }
+
+    const rows = await WIKI.db
+      .select({
+        locale: pagesTable.locale,
+        path: pagesTable.path,
+        title: pagesTable.title,
+        description: pagesTable.description,
+        render: pagesTable.render,
+        updatedAt: pagesTable.updatedAt,
+        tags: pagesTable.tags,
+        editor: pagesTable.editor,
+        isSearchable: pagesTable.isSearchable,
+        password: pagesTable.password,
+        localeGroupId: pagesTable.localeGroupId
+      })
+      .from(pagesTable)
+      .where(and(...conditions))
+      .limit(1)
+
+    const row = rows[0]
+    if (!row) {
+      return null
+    }
+    if (!WIKI.models.groups.checkAccess(actor, 'read:pages', row)) {
+      return null
+    }
+
+    // -> A page and its lock are two things: the title and the description are not what a password
+    //    covers (the search index treats them the same way), so they travel and the body does not.
+    //    Held against the column and not against this requester's unlocks — the public description of
+    //    a page is cached and shared, so a body any one requester had earned must not get into it
+    const isProtected = Boolean(row.password)
+    const isRedirect = row.editor === REDIRECT_EDITOR
+    return {
+      locale: row.locale,
+      path: row.path,
+      title: row.title,
+      description: row.description,
+      render: isProtected || isRedirect ? null : row.render,
+      updatedAt: row.updatedAt,
+      isIndexable: row.isSearchable && !isProtected && !isRedirect,
+      alternates: await this.readableAlternates(siteId, row.localeGroupId, actor)
+    }
+  }
+
+  /**
+   * The locales a page can be read in, for the alternates a document names.
+   *
+   * Published and readable by this actor, and including the page itself, which is what an `hreflang`
+   * set calls for. A translation the asker may not read is not a URL to point them at — for the public
+   * that is the same cut the sitemap makes, which is why the two say the same thing about a page.
+   *
+   * Published regardless of who is asking, unlike the page itself: a draft translation is not an
+   * alternate version of a document, it is one that does not exist yet.
+   */
+  private async readableAlternates(
+    siteId: string,
+    localeGroupId: string | null,
+    actor: AccessActor
+  ): Promise<{ locale: string; path: string }[]> {
+    if (!localeGroupId) {
+      return []
+    }
+    const rows = await WIKI.db
+      .select({ locale: pagesTable.locale, path: pagesTable.path, tags: pagesTable.tags })
+      .from(pagesTable)
+      .where(
+        and(
+          eq(pagesTable.siteId, siteId),
+          eq(pagesTable.localeGroupId, localeGroupId),
+          eq(pagesTable.publishState, 'published')
+        )
+      )
+      .orderBy(pagesTable.locale)
+    const readable = rows.filter((row) => WIKI.models.groups.checkAccess(actor, 'read:pages', row))
+    // -> One document is not a set of alternates: a page whose only readable version is itself has
+    //    nothing for an annotation to point at
+    return readable.length > 1 ? readable.map(({ locale, path }) => ({ locale, path })) : []
+  }
+
   /**
    * Where a page lives, as a path.
    *
@@ -1244,6 +1420,15 @@ class Pages {
       metadata: { title: page.title, description: page.description, editor }
     })
 
+    /*
+      Everything a document served to a client that will not run the app says about a page comes out
+      of the row this just wrote — its title, its description, its body, whether it is published and
+      searchable, and where it sits. Dropping the lot is cheaper than working out which URLs a change
+      reached: a rename moves a page, and joining or leaving a translation set moves its siblings'
+      alternates with it.
+    */
+    invalidateAppShellCache()
+
     return { page: (await this.getPage({ siteId, id: page.id })) as Page, versionId }
   }
 
@@ -1414,6 +1599,7 @@ class Pages {
       authorId: actor.id,
       metadata: { title: updated.title, description: updated.description }
     })
+    invalidateAppShellCache()
 
     return { page: updated, versionId }
   }
@@ -1547,6 +1733,7 @@ class Pages {
       siteId,
       authorId: actor.id
     })
+    invalidateAppShellCache()
     return { page: moved, versionId }
   }
 
@@ -1590,6 +1777,7 @@ class Pages {
       siteId,
       authorId: actor.id
     })
+    invalidateAppShellCache()
     return { page, versionId }
   }
 
@@ -1691,6 +1879,7 @@ class Pages {
         authorId: actor.id
       })
     }
+    invalidateAppShellCache()
     WIKI.logger.debug(`Deleted ${entries.length} page(s) that went with a deleted folder.`)
   }
 
@@ -2066,6 +2255,8 @@ class Pages {
     // -> Nothing was updated when the page went while it sat in the queue
     if (updated[0]) {
       await WIKI.models.search.indexPage(id, updated[0].locale)
+      // -> This is the column the injected copy of a page IS, so a re-render changes it
+      invalidateAppShellCache()
     }
   }
 

backend/models/sites.ts

@@ -7,6 +7,7 @@ import {
   storage as storageTable
 } from '../db/schema.ts'
 import { and, eq } from 'drizzle-orm'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
 import { detectImageMime, detectSvg, normalizeImage, svgMimeType } from '../helpers/images.ts'
 import type { ImageNormalization } from '../helpers/images.ts'
 import type { SystemIds } from './types.ts'
@@ -84,13 +85,16 @@ class Sites {
       WIKI.sitesMappings[site.hostname] = site.id
     }
     /*
-      Sitemap lists are held per site for minutes at a time, and `WIKI.cache` has no expiry sweeper —
-      an entry is only dropped when its own key is next read. So a site that was deleted, or whose
-      sitemap was just switched off, would hold its last list for the life of the process: nothing
-      will ever ask for that key again. This is the one place every create, update and delete passes
-      through, which makes it the place to let them go.
+      Sitemap lists and app shell fragments are held per site for minutes at a time, and `WIKI.cache`
+      has no expiry sweeper — an entry is only dropped when its own key is next read. So a site that
+      was deleted, or whose sitemap was just switched off, would hold its last list for the life of the
+      process: nothing will ever ask for that key again. This is also the one place every create,
+      update and delete of a site's settings passes through, and those settings are in both — the
+      site's own title and description, whether it wants to be indexed, how it brackets URLs by
+      locale. So it is the place to let them go.
     */
     WIKI.models.pages.invalidateSitemaps()
+    invalidateAppShellCache()
     WIKI.logger.info(`Loaded ${sites.length} site configurations [ OK ]`)
   }
 

frontend/index.html

@@ -45,6 +45,19 @@
         animation: initspinner .6s linear infinite;
         z-index: 2000000000;
       }
+
+      /*
+        The copy of the page the server puts into this document for a client that will never run the
+        app -- a crawler, a chat client building an unfurl card, a reader with JavaScript off. See
+        `backend/helpers/appShell.ts` for what goes in it and why.
+
+        Out of sight of anyone whose browser is about to draw the real page, and shown (below) to
+        anyone whose browser is not. `main.js` takes the element out of the document altogether
+        before Vue mounts, so nothing the app does has to know it was ever there.
+      */
+      #wiki-prerender {
+        display: none;
+      }
     </style>
 
     <noscript>
@@ -64,6 +77,74 @@
         .scroll.relative-position > .absolute {
           position: relative !important;
         }
+
+        /* With no app coming, the server's copy of the page is the page. */
+        #wiki-prerender {
+          display: block;
+          max-width: 60rem;
+          margin: 0 auto;
+          padding: 2rem 1.5rem;
+          line-height: 1.6;
+        }
+
+        /*
+          The browser defaults Tailwind's preflight takes away, put back. Deliberately not a copy of
+          the content styles in `css/tailwind.css`: those belong to elements the app draws, which is
+          the one thing that is not going to happen here, and a second copy of them would only drift.
+          What is wanted is prose that reads as prose.
+        */
+        #wiki-prerender h1 { font-size: 2em; }
+        #wiki-prerender h2 { font-size: 1.5em; }
+        #wiki-prerender h3 { font-size: 1.17em; }
+        #wiki-prerender h1,
+        #wiki-prerender h2,
+        #wiki-prerender h3,
+        #wiki-prerender h4,
+        #wiki-prerender h5,
+        #wiki-prerender h6 {
+          font-weight: bold;
+          margin: 1.5em 0 .5em;
+          line-height: 1.25;
+        }
+        #wiki-prerender p,
+        #wiki-prerender ul,
+        #wiki-prerender ol,
+        #wiki-prerender pre,
+        #wiki-prerender blockquote,
+        #wiki-prerender table {
+          margin: 0 0 1em;
+        }
+        #wiki-prerender ul,
+        #wiki-prerender ol {
+          padding-left: 2em;
+          list-style: revert;
+        }
+        #wiki-prerender pre,
+        #wiki-prerender code {
+          font-family: monospace;
+        }
+        #wiki-prerender pre {
+          overflow-x: auto;
+          padding: .75em 1em;
+          background: #f4f4f4;
+        }
+        #wiki-prerender blockquote {
+          padding-left: 1em;
+          border-left: 4px solid #ccc;
+          color: #555;
+        }
+        #wiki-prerender img {
+          max-width: 100%;
+          height: auto;
+        }
+        #wiki-prerender table {
+          border-collapse: collapse;
+        }
+        #wiki-prerender th,
+        #wiki-prerender td {
+          padding: .35em .75em;
+          border: 1px solid #ccc;
+        }
       </style>
     </noscript>
   </head>

frontend/src/main.js

@@ -32,4 +32,8 @@ initializeEventBus()
 initializeIconify()
 initializeExternals(router, store)
 initializeI18n(app, store)
+// The server's copy of the page, for clients that never get this far -- see
+// `backend/helpers/appShell.ts`. It has done its job by now, and Vue is about to draw the real thing.
+document.getElementById('wiki-prerender')?.remove()
+
 app.mount('#app')