LibreQoS v1.3.1

LibreQoS v1.3.1 Installation & Usage Guide - Physical Server and Ubuntu 22.04

Notes for upgrading from v1.2 or prior

Custom CRM Integrations

If you use a custom CRM integration, please ensure your integration uses a unique circuit identifier for the ‘Circuit ID’ field in ShapedDevices.csv. This is now required in v1.3 in order to make partial reloading possible. A good choice for this ID would be internet service plan unique ID, or the subscriber site ID your CRM provides for customer service locations. Multiple devices within the same circuit would use the same ‘Circuit ID’, but aside from that, all Circuit IDs should be distinct. The built-in Splynx and UISP integrations for v1.3 handle this automatically.

Network Design Assumptions

Officially supported configuration:

Edge and Core routers with MTU 1500 on links between them
If you use MPLS, you would terminate MPLS traffic at the core router. LibreQoS cannot decapsulate MPLS on its own.
OSPF primary link (low cost) through the server running LibreQoS
OSPF backup link

Is it possible to use LibreQoS in-line without a core router, but that setup requires depending on STP instead of OSPF, which can cause issues. Such configurations are not officially supported.

Network Interface Card

LibreQoS requires a NIC with 2 or more RX/TX queues and XDP support. While many cards theoretically meet these requirements, less commonly used cards tend to have unreported driver bugs which impede XDP functionality and make them unusable for our purposes. At this time we can only recommend Intel x520, Intel x710, and Nvidia (ConnectX-5 or newer) NICs.

Server Setup

Disable hyperthreading on the BIOS/UEFI of your host system. Hyperthreaading is also known as Simultaneous Multi Threading (SMT) on AMD systems. Disabling this is very important for optimal performance of the XDP cpumap filtering and, in turn, throughput and latency.

Boot, pressing the appropriate key to enter the BIOS settings
For AMD systems, you will have to navigate the settings to find the “SMT Control” setting. Usually it is under something like Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control Once you find it, switch to “Disabled” or “Off”
For Intel systems, you will also have to navigate the settings to find the “hyperthrading” toggle option. On HP servers it’s under System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.
Save changes and reboot

Install Ubuntu

Download Ubuntu Server 22.04 from https://ubuntu.com/download/server.

Boot Ubuntu Server from USB.
Follow the steps to install Ubuntu Server.
If you use a Mellanox network card, the Ubuntu Server installer will ask you whether to install the mellanox/intel NIC drivers. Check the box to confirm. This extra driver is important.
On the Networking settings step, it is recommended to assign a static IP address to the management NIC.
Ensure SSH server is enabled so you can more easily log into the server later.
You can use scp or sftp to access files from your LibreQoS server for easier file editing. Here’s how to access via scp or sftp using an Ubuntu or Windows machine.

Use Installer Script (For Sponsors - Skip If Not Applicable)

Sponsors can use the LibreQoS-Installer script. This script does the following:

Disables IRQbalance
Disables required offloading types using service
Creates a bridge between two interfaces - applied by the above service at each boot
Installs LibreQoS and cpumap-pping

Once complete - skip to this section of the guide.

Setup

Disable IRQbalance

sudo systemctl stop irqbalance
sudo systemctl disable irqbalance

Disable Offloading

We need to disable certain hardware offloading features, as they break XDP, used by XDP-CPUMAP-TC to send traffic to appropriate CPUs. You can create a bash script to disabled these offload features upon boot.

sudo nano /usr/local/sbin/offloadOff.sh

Enter the following

#!/bin/sh
ethtool --offload eth1 gso off tso off lro off sg off gro off
ethtool --offload eth2 gso off tso off lro off sg off gro off

Replace eth1 and eth2 with your two shaper interfaces (order doesn’t matter). Then create

sudo nano /etc/systemd/system/offloadOff.service

With the following

[Unit]
After=network.service

[Service]
ExecStart=/usr/local/sbin/offloadOff.sh

[Install]
WantedBy=default.target

Then change permissions and enable the service with

sudo chmod 664 /etc/systemd/system/offloadOff.service
sudo chmod 744 /usr/local/sbin/offloadOff.sh
sudo systemctl daemon-reload
sudo systemctl enable offloadOff.service
sudo reboot

Add a bridge between edge/core interfaces

From the Ubuntu VM, create a linux interface bridge - br0 - with the two shaping interfaces. Find your existing .yaml file in /etc/netplan/ with

cd /etc/netplan/
ls

Then edit the .yaml file there with

sudo nano XX-cloud-init.yaml

with XX corresponding to the name of the existing file.

Editing the .yaml file, we need to define the shaping interfaces (here, ens19 and ens20) and add the bridge with those two interfaces. Assuming your interfaces are ens18, ens19, and ens20, here is what your file might look like:

# This is the network config written by 'subiquity'
network:
  ethernets:
    ens18:
      addresses:
      - 10.0.0.12/24
      routes:
      - to: default
        via: 10.0.0.1
      nameservers:
        addresses:
        - 1.1.1.1
        - 8.8.8.8
        search: []
    ens19:
      dhcp4: no
    ens20:
      dhcp4: no
  version: 2
  bridges:
    br0:
      interfaces:
        - ens19
        - ens20

Make sure to replace 10.0.0.12/24 with your LibreQoS VM’s address and subnet, and to replace the default gateway 10.0.0.1 with whatever your default gateway is.

Then run

sudo netplan apply

Install LibreQoS and dependencies

Cd to your preferred directory and download the latest release

cd home/$USER/
sudo apt update
sudo apt install python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
git clone https://github.com/rchac/LibreQoS.git
git checkout v1.3.1

Install and compile cpumap-pping

cd home/$USER/LibreQoS/src
git submodule update --init
cd cpumap-pping/
git submodule update --init
cd src/
make

Install InfluxDB for Graphing

To install InfluxDB 2.x., follow the steps at https://portal.influxdata.com/downloads/.

For high throughput networks (5+ Gbps) you will likely want to install InfluxDB to a separate machine or VM from that of the LibreQoS server to avoid CPU load.

Restart your system that is running InfluxDB

sudo reboot

Check to ensure InfluxDB is running properly. This command should show “Active: active” with green dot.

sudo service influxdb status

Check that Web UI is running:

http://SERVER_IP_ADDRESS:8086

Create Bucket

Data > Buckets > Create Bucket

Call the bucket “libreqos” (all lowercase, without quotes).
Have it store as many days of data as you prefer. 7 days is standard.<> Import Dashboard

Boards > Create Dashboard > Import Dashboard

Then upload the file influxDBdashboardTemplate.json to InfluxDB.

Generate an InfluxDB Token. It will be added to ispConfig.py in the following steps.

Modify ispConfig.py

Copy ispConfig.example.py to ispConfig.py and edit as needed

cd /home/$USER/LibreQoS/src/
cp ispConfig.example.py ispConfig.py
nano ispConfig.py

Set upstreamBandwidthCapacityDownloadMbps and upstreamBandwidthCapacityUploadMbps to match the bandwidth in Mbps of your network’s upstream / WAN internet connection. The same can be done for generatedPNDownloadMbps and generatedPNUploadMbps.
Set interfaceA to the interface facing your core router (or bridged internal network if your network is bridged)
Set interfaceB to the interface facing your edge router
Set enableActualShellCommands = True to allow the program to actually run the commands.

Integrations

Integrations now share a common framework thanks to this pull. This also allows for graphing the network topology with graphviz.

UISP Integration

To run the UISP Integration, use

python3 integrationUISP.py

On the first successful run, it will create a network.json and ShapedDevices.csv file. If a network.json file exists, it will not be overwritten. You can modify the network.json file to more accurately reflect bandwidth limits. ShapedDevices.csv will be overwritten every time the UISP integration is run. You have the option to run integrationUISP.py automatically on boot and every 30 minutes, which is recommended. This can be enabled by setting automaticImportUISP = True in ispConfig.py

Network.json

Network.json allows ISP operators to define a Hierarchical Network Topology, or Flat Network Topology.

For networks with no Parent Nodes (no strictly defined Access Points or Sites) edit the network.json to use a Flat Network Topology with nano network.json setting the following file content:

{}

If you plan to use the built-in UISP or Splynx integrations, you do not need to create a network.json file quite yet.

If you plan to use the built-in UISP integration, it will create this automatically on its first run (assuming network.json is not already present). You can then modify the network.json to more accurately reflect your topology.

If you will not be using an integration, you can manually define the network.json following the template file - network.example.json

+-----------------------------------------------------------------------+
| Entire Network                                                        |
+-----------------------+-----------------------+-----------------------+
| Parent Node A         | Parent Node B         | Parent Node C         |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 |       |                       |       | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+

Manual Editing

Modify the network.json file using your preferred JSON editor (Geany for example)
Parent node name must match that used for clients in ShapedDevices.csv

ShapedDevices.csv

If you are using an integration, this file will be automatically generated. If you are not using an integration, you can manually edit the file.

Modify the ShapedDevices.csv file using your preferred spreadsheet editor (LibreOffice Calc, Excel, etc), following the template file - ShapedDevices.example.csv
An IPv4 address or IPv6 address is required for each entry.
The Access Point or Site name should be set in the Parent Node field. Parent Node can be left blank for flat networks.
The ShapedDevices.csv file allows you to set minimum guaranteed, and maximum allowed bandwidth per subscriber.
The minimum allowed plan rates for Circuits are 2Mbit. Bandwidth min and max should both be above that threshold.
Recommendation: set the min bandwidth to something like 25/10 and max to 1.15X advertised plan rate by using bandwidthOverheadFactor = 1.15
- This way, when an AP hits its ceiling, users have any remaining AP capacity fairly distributed between them.
- Ensure a reasonable minimum bandwidth minimum for every subscriber, allowing them to utilize up to the maximum provided when AP utilization is below 100%.

Note regarding SLAs: For customers with SLA contracts that guarantee them a minimum bandwidth, set their plan rate as the minimum bandwidth. That way when an AP approaches its ceiling, SLA customers will always get that amount.

How to run LibreQoS

One-Time / Debug Runs

One-time runs show the response from the terminal for each filter rule applied, and can be very helpful for debugging and to make sure it is correctly configured.

Modify setting parameters in ispConfig.py to suit your environment
For one-time runs, use

sudo ./LibreQoS.py

To use the debug mode with more verbose output, use:

sudo ./LibreQoS.py --debug

Running as a service

To run as a service, we create a systemd service to run scheduler.py. scheduler.py does the following:

On start: Run a full setup of queues
Every 30 minutes: Update queues, pulling new configuration from CRM integration if enabled

On Linux distributions that use systemd, such as Ubuntu, we create

sudo nano /etc/systemd/system/LibreQoS.service

Then paste the text below, replacing “/home/YOUR_USERNAME/LibreQoS” with wherever you downloaded LibreQoS to. Be sure to replace YOUR_USERNAME with your actual username, because otherwise when the root user executes it, it will look in the wrong directory.

[Unit]
After=network.service

[Service]
WorkingDirectory=/home/YOUR_USERNAME/LibreQoS/src
ExecStart=/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/scheduler.py
ExecStopPost=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
ExecStop=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
Restart=always

[Install]
WantedBy=default.target

Then run

sudo chmod 664 /etc/systemd/system/LibreQoS.service
sudo systemctl daemon-reload
sudo systemctl enable LibreQoS.service

You can start the service using

sudo systemctl start LibreQoS.service

You can check the status of the service using

sudo systemctl status LibreQoS.service

You can restart the service to refresh any changes you’ve made to the ShapedDevices.csv file by doing

sudo systemctl restart LibreQoS.service

You can also stop the service to remove all queues and IP rules by doing

sudo systemctl stop LibreQoS.service

Crontab

At 4AM: Runs a full reload of all queues to make sure they perfectly match queueStructure.py and that any changes to network.json can be applied.

First, check to make sure the cron job does not already exist.

sudo crontab -l | grep -q 'LibreQoS'  && echo 'entry exists' || echo 'entry does not exist'

The above should output “entry does not exist”. If so, proceed to add it with:

(sudo crontab -l 2>/dev/null; echo "0 4 * * * /bin/systemctl try-restart LibreQoS") | sudo crontab -
sudo /etc/init.d/cron start

Common Issues

Program Running, But Traffic Not Shaping

In ispConfig.py, make sure the edge and core interfaces correspond to correctly to the edge and core. Try swapping the interfaces to see if shaping starts to work.

RTNETLINK answers: Invalid argument

This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the recommended NICs.

Performance Tuning

OSPF

It is recommended to set the OSPF timers of both OSPF neighbors (core and edge router) to minimize downtime upon a reboot of the LibreQoS server.

hello interval
dead