diff --git a/docs/superpowers/specs/2026-06-20-perf-comparison-suite-design.md b/docs/superpowers/specs/2026-06-20-perf-comparison-suite-design.md new file mode 100644 index 00000000..db2b29c0 --- /dev/null +++ b/docs/superpowers/specs/2026-06-20-perf-comparison-suite-design.md @@ -0,0 +1,285 @@ +# Performance comparison suite (shell vs Go distrobox) + +## Goal + +Build a developer-local performance test suite that measures wall time, peak +memory, and CPU counters of a `distrobox` executable across a fixed set of +scenarios. The suite takes the executable path as an argument so the same +scenarios can be run against either the shell implementation (`main` branch's +`./distrobox`) or the Go rewrite (`next` branch's `./bin/distrobox`), producing +result directories that a separate comparison tool diffs into a markdown +report. + +Out of scope: functional/behavioral testing (covered by `tests/compare.sh`), +CI integration, multi-host comparison, flamegraph rendering, Go pprof +profiling. + +## Constraints / decisions already made + +- **Scope:** pure-overhead scenarios AND end-to-end container lifecycle scenarios. +- **Profiling depth:** black-box (hyperfine, `/usr/bin/time -v`, strace optional) + plus `perf stat` / `perf record` on both implementations. No language-specific + profilers (no pprof, no `bash -x`). +- **Container engine:** podman only, pinned via `DBX_CONTAINER_MANAGER=podman`. +- **Test image:** `docker.io/library/alpine:latest`. Pre-pulled once, never + removed by the suite. +- **Harness language:** POSIX shell (`/bin/sh`), mirroring the existing + `tests/compare.sh` style. Tools assumed present: `hyperfine`, `jq`, + `/usr/bin/time` (GNU), `perf`, `podman`. + +## Repository layout + +``` +bench/ +├── run.sh # entry point: ./bench/run.sh [opts] +├── compare.sh # ./bench/compare.sh +├── lib/ +│ ├── hyperfine.sh # hyperfine wrapper, JSON export +│ ├── time.sh # /usr/bin/time -v wrapper + parser to JSON +│ ├── perf.sh # perf stat + perf record wrappers +│ ├── container.sh # podman setup/cleanup, tracker file ops +│ └── common.sh # logging, scenario loader, preflight +├── scenarios/ +│ ├── 01-startup-help.sh +│ ├── 02-startup-version.sh +│ ├── 03-subcommand-help.sh +│ ├── 04-assemble-parse.sh # may be a no-op if --dry-run not supported; see §Open +│ ├── 10-list-empty.sh +│ ├── 11-list-many.sh +│ ├── 20-create-rm.sh +│ ├── 21-enter-exec.sh +│ └── 22-ephemeral-true.sh +├── fixtures/ +│ └── assemble.ini +└── results/ + ├──