LibreQoS/docs/v2.0/troubleshooting.md

Troubleshooting

Start Here: Symptom Triage

Use this table to jump to the first checks quickly.

Need definitions for licensing/scheduler terms? See the Glossary.

Symptom	First check	WebUI location	Next section
Cannot access WebUI	systemctl status lqosd caddy	N/A (UI unavailable)	No WebUI at x.x.x.x:9123 or HTTPS URL
Traffic is not shaping	verify to_internet / to_network, service state	WebUI Dashboard	LibreQoS Is Running, But Traffic Not Shaping
Scheduler appears unhealthy	check lqosd and lqos_scheduler logs	WebUI -> Scheduler Status	Scheduler status in WebUI looks unhealthy
Topology/flow views blank	confirm recent traffic and lqosd health	WebUI -> Flow Globe / Tree / ASN Analysis	Flow Globe / Tree Overview / ASN Analysis appears blank
Urgent issue code appears	open issue details and map code	WebUI -> Urgent Issues	Urgent issue codes and first actions
Mapped circuit cap events	validate license state and mapped counts	Insight UI + WebUI -> Urgent Issues	Mapped circuit limit reached

Common Issues

Where in WebUI

• Service/health overview: WebUI -> Dashboard
• Scheduler readiness: WebUI -> Scheduler Status
• High-priority alerts: WebUI -> Urgent Issues
• Topology/traffic visualization: WebUI -> Network Tree Overview and Flow Globe
• Shaped records review: WebUI -> Shaped Devices Editor

Before asking in chat: collect this evidence

Collect these first to reduce back-and-forth:

sudo systemctl status lqosd lqos_scheduler
journalctl -u lqosd --since "30 minutes ago"
journalctl -u lqos_scheduler --since "30 minutes ago"

If integration-related, also include:

ls -lh /opt/libreqos/src/topology_import.json /opt/libreqos/src/shaping_inputs.json

If you run a manual or custom-file deployment instead of a built-in integration, include:

ls -lh /opt/libreqos/src/network.json /opt/libreqos/src/ShapedDevices.csv

And include:

• current version/build
• integration type and strategy (if used)
• exact symptom and when it started

User password not working

Current builds will:

• migrate older auth files automatically
• redirect /login.html to /first-run.html when no users exist

If the correct username/password still fails, first restart lqosd and try again:

sudo systemctl restart lqosd

Only remove lqusers.toml if you are intentionally resetting access or if the file is corrupt and cannot be repaired. After removing it, restart lqosd and open BOX_IP:9123/login.html if SSL is disabled; the WebUI should redirect you to first-run setup automatically.

No WebUI at x.x.x.x:9123 or HTTPS URL

The WebUI is controlled by lqosd. If optional HTTPS with Caddy is enabled, caddy also has to be healthy.

Start by checking:

sudo systemctl status lqosd caddy

Then:

• If SSL is disabled, test http://your-management-ip:9123/
• If SSL is enabled with a hostname, test https://your-hostname/
• If SSL is enabled without a hostname, test https://your-management-ip/
• If browsers warn in local-certificate mode, trust /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt on the operator workstation

Then follow the full workflow in Service lqosd is not running or failed to start below.

LibreQoS Is Running, But Traffic Not Shaping

In /etc/lqos.conf, ensure that to_internet and to_network are set correctly. If not, simply swap the interfaces between those and restart lqosd and the scheduler.

sudo systemctl restart lqosd lqos_scheduler

Make sure your services are running properly

sudo systemctl status lqosd lqos_scheduler

The service lqos_scheduler is dependent on the lqosd service being in a healthy, running state.

On-a-stick shaping looks wrong or one direction is weak

On-a-stick mode depends on queue splitting per direction. If TX queue discovery is wrong or override_available_queues is mis-set, directional mapping can be degraded.

Check:

sudo systemctl status lqosd
journalctl -u lqosd --since "10 minutes ago"

Then verify queue-related config in /etc/lqos.conf and restart:

sudo systemctl restart lqosd lqos_scheduler

Service lqosd is not running or failed to start

Check to see the state of the lqosd service:

sudo systemctl status lqosd

If the status is 'failed', examine why using journalctl, which shows the full status of the service:

journalctl -u lqosd --since "10 minutes ago"

Press the End key on the keyboard to take you to the bottom of the log to see the latest updates to that log.

Lqosd will provide specific reasons it failed, such as an interface not being up, an interface lacking multi-queue, or other concerns.

If the log shows LibreQoS failed to attach the XDP/TC kernel or Unable to load the XDP/TC kernel, treat lqosd startup as failed. The WebUI and local bus will not start until the kernel program loads and attaches successfully. The load error includes the raw return value, errno number, and errno code, for example raw=-11, errno=11, code=EAGAIN. Check for an existing XDP program, busy TC hook, missing driver support, or stale pinned maps before restarting lqosd.

If journalctl -u lqosd shows lqosd host memory pressure or lqosd process memory critical, the daemon detected high memory usage and logged diagnostic context. The watchdog does not restart lqosd; it records available memory, total memory, lqosd RSS/swap, thread count, flow count, and timing counters that help diagnose the source of memory growth. Host memory pressure is logged when available memory is below 10% of installed RAM. Process memory is logged as critical when lqosd RSS plus swap reaches 90% of installed RAM.

You can disable these diagnostics with a systemd environment override during short troubleshooting windows:

sudo systemctl edit lqosd

Set LQOSD_MEMORY_WATCHDOG_DISABLED=1 only when you are actively watching memory pressure through another tool.

High CPU usage at peak traffic (unstable TSC clocksource)

If the CPU pegs at 100% during peak hours and throughput/latency suffer, the Linux kernel may have marked the CPU's TSC as an unstable clocksource and fallen back to a slower one (usually HPET). LibreQoS operators have observed this on some AMD Ryzen hosts with CPU C-states enabled, sometimes only after a reboot.

LibreQoS warns at startup when HPET or the ACPI PM timer is active. If you see this lqosd warning, check the clocksource directly:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

If the output is hpet or acpi_pm on a physical LibreQoS host, fix it by either:

• adding tsc=reliable (or tsc=nowatchdog) to the kernel command line in /etc/default/grub (GRUB_CMDLINE_LINUX_DEFAULT) and running sudo update-grub, then rebooting, or
• disabling CPU C-states in the BIOS/UEFI.

On hosts where the stability watchdog demotes the TSC, adding only clocksource=tsc does not prevent the demotion.

Advanced lqosd debug

At the command-line, run:

sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd

which runs lqosd in debug mode, and will provide specifics regarding why it failed to start.

Service lqos_scheduler shows errors

If sudo systemctl status lqosd lqos_scheduler reveals that the lqos_scheduler service has issues, you can export a comprehensive log of the errors to a file:

sudo journalctl -u lqos_scheduler --since "1 day ago" --no-pager > lqos_sched_log.txt

This exports a log file to lqos_sched_log.txt. You can review this file to see what caused the scheduler to error out.

If the scheduler fails immediately after a restart with a message like Socket (typically /run/lqos/bus) not found, that indicates lqosd had not finished binding the local bus yet. Current builds wait briefly for bus readiness at scheduler startup instead of crashing immediately, so repeated startup panics after restart should no longer be expected.

If a host upgrades to a newer CPython 3.x minor release than the machine used to build LibreQoS, current packages should no longer require an exact Python minor match for liblqos_python.so. The extension is now built in PyO3 abi3 mode with a Python 3.10 floor. If you still see interpreter crashes or import failures after such an upgrade, treat that as a bug and capture:

python3 --version
python3 - <<'PY'
import sysconfig
print(sysconfig.get_config_var("SOABI"))
PY
file /opt/libreqos/src/liblqos_python.so
ldd /opt/libreqos/src/liblqos_python.so

Routine package upgrades now keep lqosd in charge of the main WebUI when /etc/lqos.conf already exists. Current packages no longer start the dedicated lqos_setup web service during a normal upgrade just because newer first-run checks are incomplete. If the upgraded host still needs a first admin user or a topology source, finish that work in the normal WebUI (first-run.html or Complete Setup) instead of expecting lqos_setup to take over port 9123.

If startup shaping begins before topology runtime has finished publishing the current generation of shaping_inputs.json, current builds keep the scheduler in a startup wait state and retry the initial shaping pass every few seconds. A short still building outputs for the current source generation message right after restart now usually means LibreQoS is still finishing the import/runtime publish cycle, not that shaping is stuck until the next 30-minute refresh.

If a scheduled integration refresh lands while topology runtime is still publishing outputs for the new source generation, current builds keep the scheduler in a waiting state for that generation and retry the scheduled shaping refresh automatically as soon as topology runtime finishes. Treat Scheduled shaping refresh deferred as a transient wait only when the message says topology runtime is still building outputs for the current generation. If the message instead says topology runtime failed for the current generation, investigate that runtime failure directly.

If scheduler startup stays in that wait state for an unusually long time, or degrades with a message that topology runtime failed for the current generation, inspect:

cat /opt/libreqos/state/topology/topology_runtime_status.json
ls -lh /opt/libreqos/state/topology/topology_effective_state.json /opt/libreqos/state/topology/network.effective.json /opt/libreqos/state/shaping/shaping_inputs.json
journalctl -u lqos_scheduler --since "30 minutes ago"
journalctl -u lqosd --since "30 minutes ago"

If Topology Manager changes or imports seem stuck on older data, check whether LibreQoS set older snapshots aside under /opt/libreqos/src/.topology_stale/, then review recent scheduler and lqosd logs before retrying.

If Insight topology looks wrong, review the current troubleshooting snapshot that lqosd is preparing for Insight:

cat /opt/libreqos/src/network.insight.debug.json

Treat network.insight.debug.json as a troubleshooting snapshot only; do not edit it.

If journalctl -u lqosd shows repeated BeginIngest queue full, IngestChunk queue full, or EndIngest queue full warnings during startup or immediately after a topology import, older builds were dropping Insight ingest frames because the node's outbound control-channel queue was too small for burst uploads. Current builds apply backpressure on the Insight socket for ingest batches instead, so those warnings should no longer be expected during short import/startup bursts. If they still appear after upgrading, inspect recent lqosd CPU pressure and control-channel connectivity before assuming shaping itself is unhealthy.

If specific APs or switches appear multiple times with suffixed names such as ... [AP deadbeef], check whether UISP is returning duplicate rows for the same device ID. Current builds defensively deduplicate raw UISP devices by identification.id before topology graph construction, and skip any residual duplicate device IDs during graph assembly.

If an integration subprocess fails, current builds keep the scheduler alive, publish a shortened output preview to the scheduler status/error surfaces, and save the full captured output to a timestamped file under /tmp such as lqos_scheduler_uisp_integration_YYYYMMDD_HHMMSS.log. If shaping can continue from the last-known-good topology, the scheduler may still report ready, but the latest integration failure remains visible in scheduler status until the next successful integration run.

If the scheduler cannot read lqos_overrides.json or its materialized override layers because another process holds the overrides lock, current builds retry briefly and then block that reload. The previous topology remains in use, and the scheduler error includes lock-holder details such as PID, process name, operation, and lock creation time when available.

Scheduler status in WebUI looks unhealthy

Recent builds expose scheduler readiness/state in the WebUI (Node Manager). If the scheduler is still starting, the sidebar now reports the current startup phase and a coarse progress ring rather than only a spinner. Current builds also treat scheduler progress, output, and error bus messages as proof that the scheduler is alive, so the sidebar should not stay stuck on Scheduler available: false while the scheduler is actively reporting work. If the scheduler modal says scheduler details timed out, current builds keep showing the last good scheduler snapshot with its age instead of turning that transport problem into a scheduler error. Treat that warning as a WebUI or lqosd communication issue first, then confirm scheduler health in the service logs before assuming shaping failed.

If scheduler status appears down/stale:

1. Verify both services:

sudo systemctl status lqosd lqos_scheduler

2. Check recent scheduler logs:

journalctl -u lqos_scheduler --since "30 minutes ago"

3. Check lqosd bus/log state for scheduler-ready or scheduler-error messages:

journalctl -u lqosd --since "30 minutes ago"

4. If config/integration changes were recent, restart services cleanly:

sudo systemctl restart lqosd lqos_scheduler

If status repeatedly oscillates between ready/error, collect both logs and confirm integration credentials/timeouts in /etc/lqos.conf.

RTNETLINK answers: Invalid argument

This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the recommended NICs.

Python dependency or virtual environment errors

Packaged installs keep LibreQoS Python dependencies in /opt/libreqos/venv. The services still run as root, but Python packages do not mix with apt-managed system packages. If the scheduler reports missing Python modules, or package configuration was interrupted while installing Python dependencies, rebuild the virtual environment:

sudo /opt/libreqos/src/bin/rebuild_python_venv.sh
sudo dpkg --configure -a
sudo systemctl restart lqosd lqos_scheduler

Git-based installs should use ./build_rust.sh after pulling updates. It rebuilds the virtual environment before refreshing service files or restarting services. If systemd reports status=203/EXEC on /opt/libreqos/venv/bin/python, or a failed scheduler pre-start check, rebuild the virtual environment with the command above and restart lqos_scheduler.

For manual shaping tests, use the same interpreter as the service:

sudo systemctl stop lqos_scheduler
sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py
sudo systemctl start lqos_scheduler

Older installs that predate the virtual environment may show ModuleNotFoundError and suggest system pip commands. Do not repair current installs with system pip or --break-system-packages; those packages are not used by the venv-backed scheduler service. Upgrade to a package that creates /opt/libreqos/venv, then use the repair command above.

All customer IPs are listed under Unknown IPs, rather than Shaped Devices in GUI

cd /opt/libreqos/src
sudo systemctl stop lqos_scheduler
sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py

The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json Once you have identified the error and fixed ShapedDevices.csv and/or Network.json, please then run

sudo systemctl start lqos_scheduler

Flow Globe / Tree Overview / ASN Analysis appears blank

Some views require enough recent data to render meaningfully. If pages look empty:

1. Confirm lqosd is healthy.
2. Wait for traffic/data to accumulate.
3. Reload the page after 1-2 minutes.
4. Check logs for websocket or ticker warnings:

journalctl -u lqosd --since "10 minutes ago"

If still blank under normal traffic, collect recent logs and open an issue.

Circuits show Generated_PN instead of network.json node names

If a DIY/manual deployment uses network.json and ShapedDevices.csv, but the Circuit page shows parents such as Generated_PN_1, check the runtime shaping inputs:

jq '.circuits[] | {circuit_id, logical_parent_node_name, effective_parent_node_name, resolution_source}' /opt/libreqos/state/shaping/shaping_inputs.json

If resolution_source is flat_bucket, LibreQoS is in flat topology mode. That mode intentionally assigns circuits to generated CPU bucket queues instead of shaping them under the named Parent Node hierarchy.

To shape circuits under the node names from network.json, set:

[topology]
compile_mode = "full"

Then reload or wait for the scheduler to regenerate network.effective.json and shaping_inputs.json.

Site Map appears blank or slow

Site Map has one extra dependency beyond normal WebUI data feeds: current builds fetch bbox/bootstrap data and raster tiles from https://insight.libreqos.com.

If Site Map alone is blank or slow:

1. Confirm lqosd is healthy.
2. Confirm the box can reach insight.libreqos.com from its management network.
3. Confirm runtime topology still carries coordinates for mapped Sites/APs in network.effective.json.
4. Wait briefly and reload the page; the map page retries tile requests automatically while cold tiles are being populated upstream.
5. Check recent lqosd logs:

journalctl -u lqosd --since "10 minutes ago"

If the rest of WebUI is healthy but Site Map continues to fail, treat it as a map/tile dependency issue rather than a general scheduler or shaping failure.

Virtual node promotion collision (network.json)

If LibreQoS.py fails with an error like Virtual node promotion collision: 'AP_A' already exists at this level., you have a "virtual": true node whose children get promoted into a parent level where a node with the same name already exists.

Rename one of the colliding nodes (names must be unique among siblings after virtual-node promotion), or restructure the hierarchy so promoted children won’t collide. For a visual of the logical-to-physical promotion flow and CPU placement, see Advanced Configuration Reference.

Mapped circuit limit reached

If logs mention messages like:

• Mapped circuit limit reached
• Bakery mapped circuit cap enforced

ShapedDevices.csv can contain unlimited entries, but without a valid Insight or Local license/grant state LibreQoS admits only the first 1000 valid mapped circuits into active shaping state.

The default 1000 mapped-circuit limit applies when license/grant state is:

• missing
• expired
• otherwise invalid
• operating with offline-invalid local grant state

Typical operator-visible symptoms:

• prominent mapped-circuit-limit warning in WebUI
• left-hand navigation usage indicator showing approach to or exhaustion of the 1000 limit
• journalctl -u lqosd messages showing requested/allowed/dropped mapped counts
• partial shaping, with circuits beyond the active limit left outside shaping state

Recommended checks:

1. Confirm license status in the License & Services page.
2. Review lqosd logs for requested/allowed/dropped counts.
3. Reduce mapped circuit count (short term) or update licensing/limits (long term).

The Node API follows the same count starting with LibreQoS 2.2. At 1,000 or fewer mapped circuits, confirm that a named local API key (or compatible legacy/license key) is configured and that lqos_api is running. New and revoked named keys may take up to 30 seconds to take effect. Above 1,000, also confirm that the effective API or Insight entitlement is valid.

Urgent issue codes and first actions

WebUI urgent issues include machine-readable codes. Use them to triage quickly.

Code	Meaning	First checks	Typical fix path
MAPPED_CIRCUIT_LIMIT	Bakery is enforcing a mapped-circuit limit.	Insight license status, journalctl -u lqosd for requested/allowed/dropped counts.	Reduce mapped circuits immediately or update license/limits.
TC_U16_OVERFLOW	Queue/class minor IDs exceeded the Linux tc u16 range on a CPU queue.	journalctl -u lqos_scheduler -u lqosd, topology depth/queue distribution.	Increase queue count and/or simplify/rebalance hierarchy (for example with integration strategy or root promotion changes).
TC_QDISC_CAPACITY	Planned auto-allocated qdiscs exceed the per-interface safe budget or Bakery's conservative memory-safety preflight before apply.	Estimated per-interface qdisc counts, qdisc-kind breakdown, and memory fields in the urgent issue context, journalctl -u lqos_scheduler -u lqosd, on_a_stick and queue_mode config.	Reduce the planned qdisc load for this run (for example fewer circuits/devices in the test shape) before retrying; do not trust partial apply.
BAKERY_MEMORY_GUARD	A chunked Bakery full reload was stopped mid-apply because available host memory fell below the scaled safety floor.	journalctl -u lqosd, available/total memory in the urgent issue context, and recent Bakery apply progress.	Treat the run as failed, reduce memory pressure or queue footprint, and retry only after the host is stable.
XDP_IP_MAPPING_CAPACITY	Required IP mappings exceed the current XDP kernel map capacity.	ShapedDevices.csv row shape, IPv4/IPv6 mix, one-device-vs-many-device assumptions, journalctl -u lqos_scheduler -u lqosd.	Reduce required mappings immediately (for example fewer devices or IPv4-only test shape), or raise kernel map capacity in a coordinated change.
XDP_IP_MAPPING_APPLY_FAILED	One or more IP mapping inserts failed during apply.	journalctl -u lqos_scheduler -u lqosd for summarized failure examples and counts.	Fix the underlying mapping failure, then rerun; do not trust partial shaping.

Operational pattern:

1. Open urgent issue details in WebUI (code/message/context).
2. Pull matching logs from lqosd and lqos_scheduler.
3. Apply the immediate mitigation.
4. Acknowledge/clear the issue in UI once stable.

Related Pages

• Quickstart
• Configure LibreQoS
• CRM/NMS Integrations
• Scale Planning and Topology Design
• Performance Tuning