changes

docs/v2.0/libreqos-backend-architecture.md

@@ -248,6 +248,7 @@ Practical effect:
 |---|---|---|---|
 | Circuit IP-only change | Yes | No | Mapping update can often be applied without tree rebuild |
 | Circuit/site speed change (subset) | Yes | Sometimes | Depends on structural impact and available class handles |
+| TreeGuard runtime node virtualization (supported subtree/top-level rebalance path) | Yes | No | Bakery now applies it as a live runtime plan: non-top-level nodes use reparent/prune/restore, and supported top-level nodes use a rebalance + cross-queue migration path |
 | Bulk all-circuit changes | Sometimes | Often | Scale and transaction/cardinality limits |
 | Topology re-parent/restructure | Rarely | Yes | HTB subtree mutation constraints |
 | Add/remove circuits | Yes (small/moderate) | Sometimes | Handle availability and diff correctness boundaries |
@@ -275,6 +276,15 @@ flowchart TD
 3. Expect higher risk when many circuits and many structure-affecting changes happen together.
 4. Build operations cadence around incremental-safe updates by default.
 
+### 7.4 Full-reload safety guards
+
+Current Bakery full reloads apply two conservative safety checks before and during large queue rebuilds:
+
+1. A qdisc preflight estimates planned qdiscs per interface and also separates infrastructure, `cake`, and `fq_codel` leaf qdiscs.
+2. That same preflight applies a conservative memory forecast and hard-blocks clearly unsafe full reloads before `tc -batch` starts.
+3. During chunked full reload apply, Bakery re-checks host memory at chunk boundaries and aborts the remaining apply if available memory drops below its safety floor.
+4. These guards are intentionally biased toward false positives on large reloads so the system fails early with diagnostics instead of spiraling into an OOM event.
+
 ## 8) Design Boundaries for Operators
 
 ### 8.1 Observability boundaries

docs/v2.0/node-manager-ui.md

@@ -9,6 +9,7 @@ This page documents key WebUI (Node Manager) views and operational behavior in t
 - Dashboard content can vary by version and enabled features.
 - Executive Summary provides a compact operational view for large networks, with drilldown pages for executive heatmaps and leaderboards.
 - Bakery provides dedicated status for queue apply state, recent Bakery events, qdisc safety/preflight results, and the existing circuit-activity graph.
+- Bakery qdisc preflight now presents per-interface planned qdiscs as compact usage bars with qdisc-kind mix badges, plus a separate limits/memory table for safe budget, kernel limit, estimated qdisc memory, current available memory, and safety-floor headroom.
 - Some charts may take a short time to populate after first opening a tab, especially on busy systems or immediately after a service restart.
 - During a Bakery full reload, queue-count cards keep showing the last known HTB/CAKE/fq-codel values and mark them as `Reloading` instead of briefly zeroing them.
 

docs/v2.0/treeguard.md

@@ -32,11 +32,19 @@ TreeGuard evaluates utilization, RTT freshness, CPU guardrails, and optional QoO
 High-level behavior:
 
 1. For sustained low-load conditions, TreeGuard may switch a direction from `cake` to `fq_codel`.
-2. If utilization rises, RTT/QoO guardrails are unsafe, or revert conditions are met, TreeGuard switches back to `cake`.
+2. If utilization rises, QoO guardrails are unsafe, or revert conditions are met, TreeGuard switches back toward the circuit's base SQM policy.
 3. Decisions can be independent per direction when `independent_directions = true`.
 
 This gives a dynamic profile where loaded directions favor `cake diffserv4`, while low-load directions can use `fq_codel` when conditions are safe.
 
+The base SQM policy comes from operator intent, not from TreeGuard defaults. In practice, TreeGuard starts from the effective configured policy for each circuit and only persists its own temporary overlay when it needs to differ from that base.
+
+Important base-policy rule:
+
+1. If a direction's base SQM policy is `cake`, TreeGuard may temporarily switch that direction to `fq_codel` and later return it to base.
+2. If a direction's base SQM policy is `fq_codel`, TreeGuard does not circuit-switch that direction to `cake`.
+3. Link virtualization remains available regardless of the circuit SQM base policy.
+
 ## Configuration (`/etc/lqos.conf`)
 
 TreeGuard config lives under `[treeguard]` and sub-sections:
@@ -80,20 +88,38 @@ enabled = true
 
 ## Overrides and Operational Notes
 
-When enabled and not dry-run, TreeGuard may persist decisions to:
+When enabled and not dry-run, TreeGuard may persist circuit SQM decisions to:
 
 - `lqos_overrides.treeguard.json`
 
 TreeGuard is designed to avoid fighting operator-owned overrides. If operator overrides exist for enrolled entities, TreeGuard skips those entities and reports warnings.
 
-TreeGuard virtual-node decisions are runtime overrides. They are not materialized back into the
-base `network.json` by the scheduler, so disabling or clearing TreeGuard does not permanently
-rewrite operator-authored topology.
+TreeGuard virtual-node decisions are runtime-only Bakery operations. They are not materialized back
+into the base `network.json` by the scheduler, and they are not persisted as TreeGuard-owned
+`set_node_virtual` entries in the effective shaping input. In v1 they are ephemeral: a daemon
+restart returns the physical tree to the base operator-defined topology until TreeGuard decides
+again.
 
-TreeGuard also refuses to manage nodes that are already marked `"virtual": true` in the base `network.json`. If stale TreeGuard-owned node-virtualization overrides exist for those nodes, TreeGuard clears its own override layer and falls back to the base topology definition.
+TreeGuard circuit SQM decisions are also runtime overrides. The scheduler does not materialize TreeGuard-owned SQM changes back into the base `ShapedDevices.csv`, so clearing TreeGuard does not permanently rewrite operator-authored circuit SQM policy.
+
+TreeGuard also refuses to manage nodes that are already marked `"virtual": true` in the base `network.json`. If stale legacy TreeGuard-owned node-virtualization overrides exist for those nodes, TreeGuard clears that legacy override state and falls back to the base topology definition.
 
 For circuit SQM management, TreeGuard treats duplicate `device_id` values as unsafe identity collisions. If the same `device_id` appears in more than one circuit in `ShapedDevices.csv`, TreeGuard skips those affected circuits and clears any TreeGuard-owned SQM overrides for those duplicate device IDs.
 
+If RTT telemetry is temporarily unavailable after a restart, TreeGuard does not treat missing RTT alone as evidence that it should revert `fq_codel` directions. Other guardrails such as utilization, QoO, and CPU pressure still apply.
+
+TreeGuard also applies a conservative global per-tick circuit SQM change budget. On very large enrolled populations, excess circuit SQM changes are deferred to later ticks instead of stampeding Bakery in one pass.
+
+For scale, TreeGuard no longer rebuilds circuit membership from `ShapedDevices.csv` on every circuit tick. It now keeps a cached per-circuit inventory derived from `ShapedDevices.csv`, reads per-circuit live telemetry from the shared once-per-second circuit rollup snapshot, and spreads large `all_circuits = true` SQM evaluations across multiple ticks instead of rescanning every enrolled circuit every second.
+
+In practice, this means:
+
+1. Link virtualization still follows the normal TreeGuard tick cadence.
+2. Circuit SQM evaluation for small enrollments still completes quickly.
+3. Very large `all_circuits` enrollments are swept incrementally over multiple ticks, with a target full sweep around 15 seconds instead of attempting a full per-second scan.
+4. TreeGuard node virtualization now goes through Bakery live runtime planning/apply paths instead of forcing a LibreQoS reload or Bakery full reload.
+5. Supported top-level runtime virtualization now uses a Bakery-side rebalance/migration plan that can promote child sites and direct circuits across queue roots while preserving the logical hierarchy for reporting.
+
 Recent TreeGuard activity is available in two places:
 
 - The WebUI TreeGuard status/activity views.

docs/v2.0/troubleshooting.md

@@ -48,13 +48,16 @@ And include:
 
 ### User password not working
 
-Delete the lqusers file:
+Current builds will:
+- migrate older auth files automatically
+- redirect `/login.html` to `/first-run.html` when no users exist
+
+If the correct username/password still fails, first restart `lqosd` and try again:
+```bash
+sudo systemctl restart lqosd
 ```
-sudo rm /opt/libreqos/src/lqusers.toml
-sudo systemctl restart lqosd lqos_scheduler
-```
-Then visit: BOX_IP:9123/index.html
-This will allow you to set up the user again from scratch using the WebUI.
+
+Only remove `lqusers.toml` if you are intentionally resetting access or if the file is corrupt and cannot be repaired. After removing it, restart `lqosd` and open `BOX_IP:9123/login.html`; the WebUI should redirect you to first-run setup automatically.
 
 ### No WebUI at x.x.x.x:9123
 
@@ -236,7 +239,8 @@ WebUI urgent issues include machine-readable codes. Use them to triage quickly.
 |---|---|---|---|
 | `MAPPED_CIRCUIT_LIMIT` | Bakery is enforcing a mapped-circuit limit. | Insight license status, `journalctl -u lqosd` for requested/allowed/dropped counts. | Reduce mapped circuits immediately or update license/limits. |
 | `TC_U16_OVERFLOW` | Queue/class minor IDs exceeded the Linux tc u16 range on a CPU queue. | `journalctl -u lqos_scheduler -u lqosd`, topology depth/queue distribution. | Increase queue count and/or simplify/rebalance hierarchy (for example with integration strategy or root promotion changes). |
-| `TC_QDISC_CAPACITY` | Planned auto-allocated qdiscs exceed the per-interface safe budget before apply. | Estimated per-interface qdisc counts in the urgent issue context, `journalctl -u lqos_scheduler -u lqosd`, `on_a_stick` and `monitor_only` config. | Reduce the planned qdisc load for this run (for example fewer circuits/devices in the test shape) before retrying; do not trust partial apply. |
+| `TC_QDISC_CAPACITY` | Planned auto-allocated qdiscs exceed the per-interface safe budget or Bakery's conservative memory-safety preflight before apply. | Estimated per-interface qdisc counts, qdisc-kind breakdown, and memory fields in the urgent issue context, `journalctl -u lqos_scheduler -u lqosd`, `on_a_stick` and `monitor_only` config. | Reduce the planned qdisc load for this run (for example fewer circuits/devices in the test shape) before retrying; do not trust partial apply. |
+| `BAKERY_MEMORY_GUARD` | A chunked Bakery full reload was stopped mid-apply because available host memory fell below the safety floor. | `journalctl -u lqosd`, available/total memory in the urgent issue context, and recent Bakery apply progress. | Treat the run as failed, reduce memory pressure or queue footprint, and retry only after the host is stable. |
 | `XDP_IP_MAPPING_CAPACITY` | Required IP mappings exceed the current XDP kernel map capacity. | `ShapedDevices.csv` row shape, IPv4/IPv6 mix, one-device-vs-many-device assumptions, `journalctl -u lqos_scheduler -u lqosd`. | Reduce required mappings immediately (for example fewer devices or IPv4-only test shape), or raise kernel map capacity in a coordinated change. |
 | `XDP_IP_MAPPING_APPLY_FAILED` | One or more IP mapping inserts failed during apply. | `journalctl -u lqos_scheduler -u lqosd` for summarized failure examples and counts. | Fix the underlying mapping failure, then rerun; do not trust partial shaping. |