The numa=par boot option now activates the parallel memory-test
scheduler built by the previous commits: at every stage start,
numa_run_active is set when NUMA_PAR is selected and the mapping model
validated with enough selected CPU-backed memory domains, and a NUMA_PAR
unavailable with the current selection falls back to the
topology-agnostic legacy paths (never NUMA_ON chunking), so no memory is
omitted. The dummy calibration applies the same window-0 policy as the
real run.
AArch64 and LoongArch64 are enabled behind their mapping validation:
the LoongArch64 NUMA address transform is checked for transformed-span
overlap (smp_numa_transform_valid(), always true without the
transform), and vmem_prepare_execution_contexts() downgrades NUMA_PAR
when it fails. The mapping models of AArch64 (identity) and
LoongArch64 (direct addressing) are shared, so no per-context roots
are prepared there; numa_parallel_available() no longer gates on
x86-64.
The tests call do_tick() once per SPIN_SIZE block, so every CPU of a
team must cover the same number of blocks: a last chunk pushed across
a block boundary would execute one extra do_tick()/barrier generation
and hang the team at the phase boundary. equalize_block_counts()
rebalances the chunk size so the chunk and the last chunk stay in the
same block bucket, and is applied to the legacy NUMA_ON, the legacy
topology-agnostic and the new NUMA_PAR chunking paths (the latter
splits each per-context segment by the dense per-context chunk index
assigned at wave binding). Legacy NUMA_ON chunking is only used in
NUMA_ON mode; a requested-but-inactive NUMA_PAR uses the
topology-agnostic path.
The NUMA_PAR scheduling state and the multi-wave team machinery, with
unchanged behavior in legacy modes and in the dummy run (the NUMA_PAR
flag is only set by the final switch-on commit):
- build_numa_domain_list() materializes the BSP-first CPU-backed memory
domain list after the per-run CPU selection snapshot; memory-only
domains are owned by the BSP context in wave 0, CPU-only domains are
parked. num_execution_waves follows from the domain count.
- bind_execution_wave() assigns distinct contexts to the wave's domains,
publishes cpu_execution_context[]/cpu_is_test_participant[] and the
dense per-context chunk indexes, configures the per-context barriers
exactly once, and re-points the bound contexts' translation roots at
the current image (new vmem_rebase_execution_context()), so contexts
never walk tables in the other, test-overwritten image copy.
- test_context()/test_context_index()/execution_context_for_cpu()/
test_run_barrier() return the calling CPU's wave binding; parked CPUs
(including later waves) execute only global control flow.
- Window enumeration is per context via the context barrier: the master
computes window/map, the barrier publishes them, every participant
maps and tests. The end of the enumeration is a sticky per-binding
flag published by the barrier, so every participant takes the same
exit decision without re-reading the master's mutable window state.
- setup_vm_map()'s NUMA span loop stops at the window range's end, so a
window whose last span belongs to another team terminates cleanly
instead of spinning on a zero-length query, and a defensive progress
check guards against malformed topology data. setup_vm_map() splits each span by proximity-domain
ownership in NUMA_PAR and records terminal context statuses (the
atomic status helpers are shared with the cancellation machinery that
follows).
- Window 0 is executed whenever the selected map has memory below
LOW_LOAD_LIMIT: the decision is purely map-based, so selected memory
in an uncovered SRAT gap still reaches setup_vm_map() and triggers
the topology-failure fallback instead of being silently omitted.
- The legacy NUMA_ON chunk indexes are only assigned in NUMA_ON mode,
and the per-context paging roots are rebased in startup64.S after
every relocation.
Lionel Debroux