diff --git a/docs-es/v2.0/Quickstart/services-and-run.md b/docs-es/v2.0/Quickstart/services-and-run.md index 1e4c6864..b05fc9af 100644 --- a/docs-es/v2.0/Quickstart/services-and-run.md +++ b/docs-es/v2.0/Quickstart/services-and-run.md @@ -48,19 +48,19 @@ sudo systemctl stop lqos_scheduler For one-time runs of LibreQoS.py, use ```shell -sudo ./LibreQoS.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py ``` - To use the debug mode with more verbose output, use: ```shell -sudo ./LibreQoS.py --debug +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py --debug ``` To confirm that lqos_scheduler (scheduler.py) is able to work correctly, run: ```shell -sudo python3 scheduler.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/scheduler.py ``` Once you have any errors eliminated, restart lqos_scheduler with diff --git a/docs-es/v2.0/TechnicalDocs/git-install.md b/docs-es/v2.0/TechnicalDocs/git-install.md index 154e2127..1b3a4405 100644 --- a/docs-es/v2.0/TechnicalDocs/git-install.md +++ b/docs-es/v2.0/TechnicalDocs/git-install.md @@ -21,15 +21,16 @@ By specifying `libreqos` at the end, git will ensure the folder name is lowercas You need to have a few packages from `apt` installed: ```shell -sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev +sudo apt-get install -y python3-pip python3-venv clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev ``` Then you need to install some Python dependencies: ```shell cd /opt/libreqos -PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt -sudo PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt +sudo python3 -m venv /opt/libreqos/venv +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade pip +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade -r requirements.txt ``` ## Install the Rust development system diff --git a/docs-es/v2.0/TechnicalDocs/troubleshooting.md b/docs-es/v2.0/TechnicalDocs/troubleshooting.md index 5bd88b72..e91e2600 100644 --- a/docs-es/v2.0/TechnicalDocs/troubleshooting.md +++ b/docs-es/v2.0/TechnicalDocs/troubleshooting.md @@ -57,7 +57,19 @@ At the command-line, run ```sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd``` wh This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](../../SystemRequirements/Compute.md#network-interface-requirements). -### Python ModuleNotFoundError in Ubuntu 24.04 +### Python dependency or virtual environment errors + +Packaged installs keep LibreQoS Python dependencies in `/opt/libreqos/venv`. The services still run as root, but Python packages do not mix with apt-managed system packages. If the scheduler reports missing Python modules, or package configuration was interrupted while installing Python dependencies, rebuild the virtual environment: + +```bash +sudo /opt/libreqos/src/bin/rebuild_python_venv.sh +sudo dpkg --configure -a +sudo systemctl restart lqosd lqos_scheduler +``` + +Git-based installs should use `./build_rust.sh` after pulling updates. It rebuilds the virtual environment before refreshing service files or restarting services. If systemd reports `status=203/EXEC`, rebuild the virtual environment with the command above and restart `lqos_scheduler`. + +### Python ModuleNotFoundError in older Ubuntu 24.04 installs ``` pip uninstall binpacking --break-system-packages --yes sudo pip uninstall binpacking --break-system-packages --yes @@ -73,7 +85,7 @@ sudo pip install deepdiff --break-system-packages ``` cd /opt/libreqos/src sudo systemctl stop lqos_scheduler -sudo python3 LibreQoS.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py ``` The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json diff --git a/docs-es/v2.0/components-es.md b/docs-es/v2.0/components-es.md index 0eff075d..7dcbed04 100644 --- a/docs-es/v2.0/components-es.md +++ b/docs-es/v2.0/components-es.md @@ -51,15 +51,15 @@ sudo journalctl -u lqos_scheduler -b ### Depuración de lqos_scheduler -`lqos_scheduler` ejecuta `scheduler.py`, que a su vez invoca `LibreQoS.py`. +`lqos_scheduler` ejecuta `scheduler.py`, que a su vez invoca `LibreQoS.py`. En instalaciones empaquetadas, Python se ejecuta desde el entorno virtual propiedad de root en `/opt/libreqos/venv`. Flujo de depuración recomendado: ```bash sudo systemctl stop lqos_scheduler cd /opt/libreqos/src -sudo ./LibreQoS.py --debug -sudo python3 scheduler.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py --debug +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/scheduler.py sudo systemctl start lqos_scheduler ``` diff --git a/docs-es/v2.0/git-install-es.md b/docs-es/v2.0/git-install-es.md index 9f155ef8..e1670714 100644 --- a/docs-es/v2.0/git-install-es.md +++ b/docs-es/v2.0/git-install-es.md @@ -20,15 +20,16 @@ Al especificar `libreqos` al final, Git asegurará que el nombre del directorio Es necesario instalar ciertos paquetes mediante `apt`: ```shell -sudo apt-get install -y python3-pip clang mold esbuild gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev +sudo apt-get install -y python3-pip python3-venv clang mold esbuild gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev ``` Posteriormente, debe instalar algunas dependencias de Python: ```shell cd /opt/libreqos -PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt -sudo PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt +sudo python3 -m venv /opt/libreqos/venv +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade pip +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade -r requirements.txt ``` ## Instalar el entorno de desarrollo de Rust diff --git a/docs-es/v2.0/troubleshooting-es.md b/docs-es/v2.0/troubleshooting-es.md index db4a348d..c9883eec 100644 --- a/docs-es/v2.0/troubleshooting-es.md +++ b/docs-es/v2.0/troubleshooting-es.md @@ -112,6 +112,16 @@ sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd sudo journalctl -u lqos_scheduler --since "1 day ago" --no-pager > lqos_sched_log.txt ``` +Las instalaciones empaquetadas mantienen las dependencias Python de LibreQoS en `/opt/libreqos/venv`. Los servicios siguen ejecutándose como root, pero los paquetes Python no se mezclan con los paquetes administrados por apt. Si el scheduler informa módulos faltantes, o si la configuración del paquete se interrumpió al instalar dependencias Python, reconstruya el entorno virtual: + +```bash +sudo /opt/libreqos/src/bin/rebuild_python_venv.sh +sudo dpkg --configure -a +sudo systemctl restart lqosd lqos_scheduler +``` + +Las instalaciones basadas en git deben usar `./build_rust.sh` después de actualizar. Ese script reconstruye el entorno virtual antes de actualizar los archivos de servicio o reiniciar servicios. Si systemd informa `status=203/EXEC`, reconstruya el entorno virtual con el comando anterior y reinicie `lqos_scheduler`. + Si el scheduler falla inmediatamente después de un reinicio con un mensaje como `Socket (typically /run/lqos/bus) not found`, eso indica que `lqosd` todavía no había terminado de enlazar el bus local. Los builds actuales esperan brevemente la disponibilidad del bus al arrancar el scheduler en lugar de abortar de inmediato, por lo que ya no deberían aparecer panics repetidos de arranque tras un reinicio. ### El estado del scheduler en WebUI aparece no saludable @@ -174,7 +184,7 @@ sudo pip install deepdiff --break-system-packages ```bash cd /opt/libreqos/src sudo systemctl stop lqos_scheduler -sudo python3 LibreQoS.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py ``` Corrija errores en `ShapedDevices.csv` y/o `network.json`, luego: diff --git a/docs/v2.0/components.md b/docs/v2.0/components.md index bc4bdf7b..a43fec72 100644 --- a/docs/v2.0/components.md +++ b/docs/v2.0/components.md @@ -31,6 +31,7 @@ flowchart LR ### lqos_scheduler - lqos_scheduler performs continuous refreshes of LibreQoS' shapers, including pulling from any enabled CRM Integrations (UISP, Splynx, Netzur). +- Packaged installs run scheduler Python from the root-owned virtual environment at `/opt/libreqos/venv`. - Actions: - On start: Run a full setup of queues - Current builds wait briefly for `lqosd` to finish binding the local bus before the first scheduler run. @@ -89,19 +90,19 @@ sudo systemctl stop lqos_scheduler For one-time runs of LibreQoS.py, use ```shell -sudo ./LibreQoS.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py ``` - To use the debug mode with more verbose output, use: ```shell -sudo ./LibreQoS.py --debug +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py --debug ``` To confirm that lqos_scheduler (scheduler.py) is able to work correctly, run: ```shell -sudo python3 scheduler.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/scheduler.py ``` If an integration fails during a scheduler run, current builds keep the scheduler alive, surface a shortened output preview in scheduler status/error reporting, and save the full captured output to a timestamped `/tmp/lqos_scheduler__*.log` file. diff --git a/docs/v2.0/git-install.md b/docs/v2.0/git-install.md index 4c4ad0ba..0c5d2287 100644 --- a/docs/v2.0/git-install.md +++ b/docs/v2.0/git-install.md @@ -20,15 +20,16 @@ By specifying `libreqos` at the end, git will ensure the folder name is lowercas You need to have a few packages from `apt` installed: ```shell -sudo apt-get install -y python3-pip clang mold esbuild gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev +sudo apt-get install -y python3-pip python3-venv clang mold esbuild gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev ``` Then you need to install some Python dependencies: ```shell cd /opt/libreqos -PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt -sudo PIP_BREAK_SYSTEM_PACKAGES=1 pip install -r requirements.txt +sudo python3 -m venv /opt/libreqos/venv +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade pip +sudo /opt/libreqos/venv/bin/python -m pip install --upgrade -r requirements.txt ``` ## Install the Rust development system diff --git a/docs/v2.0/troubleshooting.md b/docs/v2.0/troubleshooting.md index 609287fe..e4da9fc3 100644 --- a/docs/v2.0/troubleshooting.md +++ b/docs/v2.0/troubleshooting.md @@ -214,7 +214,29 @@ If status repeatedly oscillates between ready/error, collect both logs and confi This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](requirements.md). -### Python ModuleNotFoundError in Ubuntu 24.04 +### Python dependency or virtual environment errors + +Packaged installs keep LibreQoS Python dependencies in `/opt/libreqos/venv`. The services still run as root, but Python packages do not mix with apt-managed system packages. If the scheduler reports missing Python modules, or package configuration was interrupted while installing Python dependencies, rebuild the virtual environment: + +```bash +sudo /opt/libreqos/src/bin/rebuild_python_venv.sh +sudo dpkg --configure -a +sudo systemctl restart lqosd lqos_scheduler +``` + +Git-based installs should use `./build_rust.sh` after pulling updates. It rebuilds the virtual environment before refreshing service files or restarting services. If systemd reports `status=203/EXEC`, rebuild the virtual environment with the command above and restart `lqos_scheduler`. + +For manual shaping tests, use the same interpreter as the service: + +```bash +sudo systemctl stop lqos_scheduler +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py +sudo systemctl start lqos_scheduler +``` + +Older installs that predate the virtual environment may show `ModuleNotFoundError` and suggest system `pip` commands. Upgrade to a package that creates `/opt/libreqos/venv`, then use the repair command above. + +### Python ModuleNotFoundError in older Ubuntu 24.04 installs ``` pip uninstall binpacking --break-system-packages --yes sudo pip uninstall binpacking --break-system-packages --yes @@ -230,7 +252,7 @@ sudo pip install deepdiff --break-system-packages ``` cd /opt/libreqos/src sudo systemctl stop lqos_scheduler -sudo python3 LibreQoS.py +sudo /opt/libreqos/venv/bin/python /opt/libreqos/src/LibreQoS.py ``` The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json diff --git a/src/bin/lqos_scheduler.service.example b/src/bin/lqos_scheduler.service.example index c1e766dc..78a42d74 100644 --- a/src/bin/lqos_scheduler.service.example +++ b/src/bin/lqos_scheduler.service.example @@ -5,7 +5,7 @@ Requires=lqosd.service [Service] WorkingDirectory=/opt/libreqos/src -ExecStart=/usr/bin/python3 /opt/libreqos/src/scheduler.py +ExecStart=/opt/libreqos/venv/bin/python /opt/libreqos/src/scheduler.py Restart=always [Install] diff --git a/src/bin/rebuild_python_venv.sh b/src/bin/rebuild_python_venv.sh new file mode 100755 index 00000000..b3909d11 --- /dev/null +++ b/src/bin/rebuild_python_venv.sh @@ -0,0 +1,42 @@ +#!/bin/bash +set -euo pipefail + +VENV_DIR="/opt/libreqos/venv" +REQUIREMENTS="" +CONSTRAINTS="" + +if [ "$(id -u)" -ne 0 ]; then + echo "rebuild_python_venv.sh must be run as root." >&2 + exit 1 +fi + +for candidate in /opt/libreqos/src/requirements.txt /opt/libreqos/requirements.txt; do + if [ -f "$candidate" ]; then + REQUIREMENTS="$candidate" + break + fi +done + +if [ -z "$REQUIREMENTS" ]; then + echo "LibreQoS Python requirements file not found in /opt/libreqos/src or /opt/libreqos." >&2 + exit 1 +fi + +for candidate in /opt/libreqos/src/deb-requirements-constraints.txt /opt/libreqos/deb-requirements-constraints.txt; do + if [ -s "$candidate" ]; then + CONSTRAINTS="$candidate" + break + fi +done + +python3 -m venv --clear "$VENV_DIR" +"$VENV_DIR/bin/python" -m pip install --upgrade pip + +if [ -n "$CONSTRAINTS" ]; then + "$VENV_DIR/bin/python" -m pip install --upgrade -c "$CONSTRAINTS" -r "$REQUIREMENTS" +else + "$VENV_DIR/bin/python" -m pip install --upgrade -r "$REQUIREMENTS" +fi + +chown -R root:root "$VENV_DIR" +chmod -R go-w "$VENV_DIR" diff --git a/src/build_dpkg.sh b/src/build_dpkg.sh index 66388a1d..45e2c74c 100755 --- a/src/build_dpkg.sh +++ b/src/build_dpkg.sh @@ -12,7 +12,7 @@ PACKAGE=libreqos VERSION=$(cat ./VERSION_STRING).$BUILD_DATE PKGVERSION="${PACKAGE}_${VERSION}" DPKG_DIR=dist/$PKGVERSION-1_amd64 -APT_DEPENDENCIES="python3-pip, nano, curl, ca-certificates" +APT_DEPENDENCIES="python3-pip, python3-venv, nano, curl, ca-certificates" DEBIAN_DIR=$DPKG_DIR/DEBIAN LQOS_DIR=$DPKG_DIR/opt/libreqos/src LQOS_STATE_DIR=$DPKG_DIR/opt/libreqos/state @@ -53,6 +53,7 @@ LQOS_BIN_FILES=( lqosd.service.example lqos_api.service.example lqos_setup.service.example + rebuild_python_venv.sh ) RUSTPROGS=( @@ -140,16 +141,10 @@ else fi } -# Install Python Dependencies -pushd /opt/libreqos > /dev/null -# - Setup Python dependencies as a post-install task. Use --ignore-installed so -# pip does not try to uninstall Debian-managed packages that do not ship a -# pip RECORD file (for example blinker on Ubuntu 24.04). -if [ -s src/deb-requirements-constraints.txt ]; then -PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --ignore-installed -c src/deb-requirements-constraints.txt -r src/requirements.txt -else -PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --ignore-installed -r src/requirements.txt -fi +# - Setup Python dependencies in a root-owned virtual environment. LibreQoS +# services still run as root, but Python packages no longer mix with +# apt-managed system site-packages. +/opt/libreqos/src/bin/rebuild_python_venv.sh # Ensure folder permissions are correct post-install set_libreqos_operator_permissions @@ -191,7 +186,6 @@ HOTFIX exit 1 ;; esac -popd > /dev/null EOF # Uninstall Script @@ -235,6 +229,8 @@ for file in "${LQOS_BIN_FILES[@]}"; do cp "bin/$file" "$LQOS_DIR/bin" done +chmod a+x "$LQOS_DIR/bin/rebuild_python_venv.sh" + # Copy the remove pinned maps cp rust/remove_pinned_maps.sh "$LQOS_DIR"/rust diff --git a/src/build_rust.sh b/src/build_rust.sh index d61c9407..36f15f02 100755 --- a/src/build_rust.sh +++ b/src/build_rust.sh @@ -22,8 +22,10 @@ do esac done +VENV_PYTHON="/opt/libreqos/venv/bin/python" + # Check Pre-Requisites -sudo apt install python3-pip clang gcc gcc-multilib llvm libelf-dev git nano curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev curl +sudo apt install python3-pip python3-venv clang gcc gcc-multilib llvm libelf-dev git nano curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev curl if ! rustup -V &> /dev/null then @@ -214,7 +216,30 @@ clear_pinned_maps_before_lqosd_restart() { fi } +rebuild_python_venv() { + local script_path="./bin/rebuild_python_venv.sh" + if [ ! -x "$script_path" ]; then + echo "Expected $script_path to exist and be executable before refreshing Python services." + exit 1 + fi + + echo "Rebuilding /opt/libreqos/venv before restarting Python services." + if ! sudo "$script_path"; then + echo "Failed to rebuild /opt/libreqos/venv. Skipping service refresh/restarts." + exit 1 + fi + + if [ ! -x "$VENV_PYTHON" ]; then + echo "Expected $VENV_PYTHON to exist and be executable after rebuilding the Python venv." + echo "Skipping service refresh/restarts." + exit 1 + fi +} + SERVICE_UNITS_UPDATED=0 + +rebuild_python_venv + refresh_service_unit lqosd refresh_service_unit lqos_scheduler refresh_service_unit lqos_api @@ -250,8 +275,9 @@ else echo "lqosd is active as a service. Restarting it. You may need to enter your sudo password." sudo systemctl restart lqosd fi - if service_is_active lqos_scheduler; then - echo "lqos_scheduler is active as a service. Restarting it. You may need to enter your sudo password." + if service_is_active lqos_scheduler || service_is_failed lqos_scheduler || service_is_enabled lqos_scheduler; then + echo "lqos_scheduler is installed as an active, failed, or enabled service. Restarting it. You may need to enter your sudo password." + sudo systemctl reset-failed lqos_scheduler || true sudo systemctl restart lqos_scheduler fi if service_is_active lqos_api || service_is_failed lqos_api || service_is_enabled lqos_api; then @@ -267,6 +293,7 @@ fi echo "-----------------------------------------------------------------" echo "Don't forget to setup /etc/lqos.conf!" echo "Template .service files can be found in bin/" +echo "Debian package installs create/update /opt/libreqos/venv for Python dependencies." echo "If src/deb-requirements-constraints.txt exists, Debian package installs use it to constrain Python dependencies." echo "Use ./systemd_hotfix.sh to evaluate or install the Ubuntu 24.04 networkd hotfix from the LibreQoS APT repo at https://repo.libreqos.com." echo "The hotfix installer now offers to schedule a reboot after it finishes."