LibreQoS/docs/Legacy/v1.3.1.md

# LibreQoS v1.3.1

## LibreQoS v1.3.1 Installation & Usage Guide  - Physical Server and Ubuntu 22.04

## Notes for upgrading from v1.2 or prior

### Custom CRM Integrations

If you use a custom CRM integration, please ensure your integration uses a unique circuit identifier for the 'Circuit ID' field in ShapedDevices.csv. This is now required in v1.3 in order to make partial reloading possible. A good choice for this ID would be internet service plan unique ID, or the subscriber site ID your CRM provides for customer service locations. Multiple devices within the same circuit would use the same 'Circuit ID', but aside from that, all Circuit IDs should be distinct. The built-in Splynx and UISP integrations for v1.3 handle this automatically.

## Network Design Assumptions

Officially supported configuration:

- Edge and Core routers with MTU 1500 on links between them
- If you use MPLS, you would terminate MPLS traffic at the core router.
LibreQoS cannot decapsulate MPLS on its own.
- OSPF primary link (low cost) through the server running LibreQoS
- OSPF backup link

![](https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/design.png)

Is it possible to use LibreQoS in-line without a core router, but that setup requires depending on STP instead of OSPF, which can cause issues. Such configurations are not officially supported.

## Network Interface Card

LibreQoS requires a NIC with 2 or more RX/TX queues and XDP support. While many cards theoretically meet these requirements, less commonly used cards tend to have unreported driver bugs which impede XDP functionality and make them unusable for our purposes. At this time we can only recommend Intel x520, Intel x710, and Nvidia (ConnectX-5 or newer) NICs.

## Server Setup

Disable hyperthreading on the BIOS/UEFI of your host system. Hyperthreaading is also known as Simultaneous Multi Threading (SMT) on AMD systems. Disabling this is very important for optimal performance of the XDP cpumap filtering and, in turn, throughput and latency.

- Boot, pressing the appropriate key to enter the BIOS settings
- For AMD systems, you will have to navigate the settings to find the "SMT Control" setting. Usually it is under something like ```Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control``` Once you find it, switch to "Disabled" or "Off"
- For Intel systems, you will also have to navigate the settings to find the "hyperthrading" toggle option. On HP servers it's under ```System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.```
- Save changes and reboot

## Install Ubuntu

Download Ubuntu Server 22.04 from <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.

1. Boot Ubuntu Server from USB.
2. Follow the steps to install Ubuntu Server.
3. If you use a Mellanox network card, the Ubuntu Server installer will ask you whether to install the mellanox/intel NIC drivers. Check the box to confirm. This extra driver is important.
4. On the Networking settings step, it is recommended to assign a static IP address to the management NIC.
5. Ensure SSH server is enabled so you can more easily log into the server later.
6. You can use scp or sftp to access files from your LibreQoS server for easier file editing. Here's how to access via scp or sftp using an [Ubuntu](https://www.addictivetips.com/ubuntu-linux-tips/sftp-server-ubuntu/) or [Windows](https://winscp.net/eng/index.php) machine.

## Use Installer Script (For Sponsors - Skip If Not Applicable)

Sponsors can use the LibreQoS-Installer script. This script does the following:

- Disables IRQbalance
- Disables required offloading types using service
- Creates a bridge between two interfaces - applied by the above service at each boot
- Installs LibreQoS and cpumap-pping

Once complete - skip to [this section](https://github.com/LibreQoE/LibreQoS/wiki/LibreQoS-v1.3-Installation-&-Usage-Guide-Physical-Server-and-Ubuntu-22.04#install-influxdb-for-graphing) of the guide.

## Setup

### Disable IRQbalance

```shell
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance
```

### Disable Offloading

We need to disable certain hardware offloading features, as they break XDP, used by XDP-CPUMAP-TC to send traffic to appropriate CPUs.
You can create a bash script to disabled these offload features upon boot.

```shell
sudo nano /usr/local/sbin/offloadOff.sh
```

Enter the following

```shell
#!/bin/sh
ethtool --offload eth1 gso off tso off lro off sg off gro off
ethtool --offload eth2 gso off tso off lro off sg off gro off
```

Replace eth1 and eth2 with your two shaper interfaces (order doesn't matter).
Then create

```shell
sudo nano /etc/systemd/system/offloadOff.service
```

With the following

```text
[Unit]
After=network.service

[Service]
ExecStart=/usr/local/sbin/offloadOff.sh

[Install]
WantedBy=default.target
```

Then change permissions and enable the service with

```shell
sudo chmod 664 /etc/systemd/system/offloadOff.service
sudo chmod 744 /usr/local/sbin/offloadOff.sh
sudo systemctl daemon-reload
sudo systemctl enable offloadOff.service
sudo reboot
```

### Add a bridge between edge/core interfaces

From the Ubuntu VM, create a linux interface bridge - br0 - with the two shaping interfaces.
Find your existing .yaml file in /etc/netplan/ with

```shell
cd /etc/netplan/
ls
```

Then edit the .yaml file there with

```shell
sudo nano XX-cloud-init.yaml
```

with XX corresponding to the name of the existing file.

Editing the .yaml file, we need to define the shaping interfaces (here, ens19 and ens20) and add the bridge with those two interfaces. Assuming your interfaces are ens18, ens19, and ens20, here is what your file might look like:

```yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    ens18:
      addresses:
      - 10.0.0.12/24
      routes:
      - to: default
        via: 10.0.0.1
      nameservers:
        addresses:
        - 1.1.1.1
        - 8.8.8.8
        search: []
    ens19:
      dhcp4: no
    ens20:
      dhcp4: no
  version: 2
  bridges:
    br0:
      interfaces:
        - ens19
        - ens20
```

Make sure to replace 10.0.0.12/24 with your LibreQoS VM's address and subnet, and to replace the default gateway 10.0.0.1 with whatever your default gateway is.

Then run

```shell
sudo netplan apply
```

### Install LibreQoS and dependencies

Cd to your preferred directory and download the latest release

```shell
cd home/$USER/
sudo apt update
sudo apt install python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
git clone https://github.com/rchac/LibreQoS.git
git checkout v1.3.1
```

### Install and compile cpumap-pping

```shell
cd home/$USER/LibreQoS/src
git submodule update --init
cd cpumap-pping/
git submodule update --init
cd src/
make
```

### Install InfluxDB for Graphing

To install InfluxDB 2.x., follow the steps at [https://portal.influxdata.com/downloads/](https://portal.influxdata.com/downloads/).

For high throughput networks (5+ Gbps) you will likely want to install InfluxDB to a separate machine or VM from that of the LibreQoS server to avoid CPU load.

Restart your system that is running InfluxDB

```shell
sudo reboot
```

Check to ensure InfluxDB is running properly. This command should show "Active: active" with green dot.

```shell
sudo service influxdb status
```

Check that Web UI is running:<br>

```shell
http://SERVER_IP_ADDRESS:8086
```

Create Bucket

- Data > Buckets > Create Bucket

Call the bucket "libreqos" (all lowercase, without quotes).<br>
Have it store as many days of data as you prefer. 7 days is standard.<>
Import Dashboard

- Boards > Create Dashboard > Import Dashboard

Then upload the file [influxDBdashboardTemplate.json](https://github.com/rchac/LibreQoS/blob/main/src/influxDBdashboardTemplate.json) to InfluxDB.

[Generate an InfluxDB Token](https://docs.influxdata.com/influxdb/cloud/security/tokens/create-token/). It will be added to ispConfig.py in the following steps.

### Modify ispConfig.py

Copy ispConfig.example.py to ispConfig.py and edit as needed

```shell
cd /home/$USER/LibreQoS/src/
cp ispConfig.example.py ispConfig.py
nano ispConfig.py
```

- Set upstreamBandwidthCapacityDownloadMbps and upstreamBandwidthCapacityUploadMbps to match the bandwidth in Mbps of your network's upstream / WAN internet connection. The same can be done for generatedPNDownloadMbps and generatedPNUploadMbps.
- Set interfaceA to the interface facing your core router (or bridged internal network if your network is bridged)
- Set interfaceB to the interface facing your edge router
- Set ```enableActualShellCommands = True``` to allow the program to actually run the commands.

### Integrations

Integrations now share a common framework thanks to [this pull](https://github.com/rchac/LibreQoS/pull/145). This also allows for graphing the network topology with graphviz.

#### UISP Integration

To run the UISP Integration, use

```shell
python3 integrationUISP.py
```
On the first successful run, it will create a network.json and ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the UISP integration is run.
You have the option to run integrationUISP.py automatically on boot and every 30 minutes, which is recommended. This can be enabled by setting ```automaticImportUISP = True``` in ispConfig.py

### Network.json

Network.json allows ISP operators to define a Hierarchical Network Topology, or Flat Network Topology.

For networks with no Parent Nodes (no strictly defined Access Points or Sites) edit the network.json to use a Flat Network Topology with
```nano network.json```
setting the following file content:

```json
{}
```

If you plan to use the built-in UISP or Splynx integrations, you do not need to create a network.json file quite yet.

If you plan to use the built-in UISP integration, it will create this automatically on its first run (assuming network.json is not already present). You can then modify the network.json to more accurately reflect your topology.

If you will not be using an integration, you can manually define the network.json following the template file - network.example.json

```text
+-----------------------------------------------------------------------+
| Entire Network                                                        |
+-----------------------+-----------------------+-----------------------+
| Parent Node A         | Parent Node B         | Parent Node C         |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 |       |                       |       | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+
```

#### Manual Editing

- Modify the network.json file using your preferred JSON editor (Geany for example)
- Parent node name must match that used for clients in ShapedDevices.csv

### ShapedDevices.csv

If you are using an integration, this file will be automatically generated. If you are not using an integration, you can manually edit the file.

- Modify the ShapedDevices.csv file using your preferred spreadsheet editor (LibreOffice Calc, Excel, etc), following the template file - ShapedDevices.example.csv
- An IPv4 address or IPv6 address is required for each entry.
- The Access Point or Site name should be set in the Parent Node field. Parent Node can be left blank for flat networks.
- The ShapedDevices.csv file allows you to set minimum guaranteed, and maximum allowed bandwidth per subscriber.
- The minimum allowed plan rates for Circuits are 2Mbit. Bandwidth min and max should both be above that threshold.
- Recommendation: set the min bandwidth to something like 25/10 and max to 1.15X advertised plan rate by using bandwidthOverheadFactor = 1.15
  - This way, when an AP hits its ceiling, users have any remaining AP capacity fairly distributed between them.
  - Ensure a reasonable minimum bandwidth minimum for every subscriber, allowing them to utilize up to the maximum provided when AP utilization is below 100%.

Note regarding SLAs: For customers with SLA contracts that guarantee them a minimum bandwidth, set their plan rate as the minimum bandwidth. That way when an AP approaches its ceiling, SLA customers will always get that amount.

![image](https://user-images.githubusercontent.com/22501920/200134960-28709d0f-48fe-4129-b4fd-70b204cade2c.png)

## How to run LibreQoS

### One-Time / Debug Runs

One-time runs show the response from the terminal for each filter rule applied, and can be very helpful for debugging and to make sure it is correctly configured.

- Modify setting parameters in ispConfig.py to suit your environment
- For one-time runs, use

```shell
sudo ./LibreQoS.py
```

- To use the debug mode with more verbose output, use:

```shell
sudo ./LibreQoS.py --debug
```

### Running as a service

To run as a service, we create a systemd service to run scheduler.py.
scheduler.py does the following:

- On start: Run a full setup of queues
- Every 30 minutes: Update queues, pulling new configuration from CRM integration if enabled

On Linux distributions that use systemd, such as Ubuntu, we create

```shell
sudo nano /etc/systemd/system/LibreQoS.service
```

Then paste the text below, replacing "/home/YOUR_USERNAME/LibreQoS" with wherever you downloaded LibreQoS to. Be sure to replace YOUR_USERNAME with your actual username, because otherwise when the root user executes it, it will look in the wrong directory.

```text
[Unit]
After=network.service

[Service]
WorkingDirectory=/home/YOUR_USERNAME/LibreQoS/src
ExecStart=/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/scheduler.py
ExecStopPost=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
ExecStop=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
Restart=always

[Install]
WantedBy=default.target
```

Then run

```shell
sudo chmod 664 /etc/systemd/system/LibreQoS.service
sudo systemctl daemon-reload
sudo systemctl enable LibreQoS.service
```

You can start the service using

```shell
sudo systemctl start LibreQoS.service
```

You can check the status of the service using

```shell
sudo systemctl status LibreQoS.service
```

You can restart the service to refresh any changes you've made to the ShapedDevices.csv file by doing

```shell
sudo systemctl restart LibreQoS.service
```

You can also stop the service to remove all queues and IP rules by doing

```shell
sudo systemctl stop LibreQoS.service
```

### Crontab

- At 4AM: Runs a full reload of all queues to make sure they perfectly match queueStructure.py and that any changes to network.json can be applied.

First, check to make sure the cron job does not already exist.

```shell
sudo crontab -l | grep -q 'LibreQoS'  && echo 'entry exists' || echo 'entry does not exist'
```

The above should output "entry does not exist". If so, proceed to add it with:

```shell
(sudo crontab -l 2>/dev/null; echo "0 4 * * * /bin/systemctl try-restart LibreQoS") | sudo crontab -
sudo /etc/init.d/cron start
```

## Common Issues

### Program Running, But Traffic Not Shaping

In ispConfig.py, make sure the edge and core interfaces correspond to correctly to the edge and core. Try swapping the interfaces to see if shaping starts to work.

### RTNETLINK answers: Invalid argument

This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](#network-interface-card).

## Performance Tuning

### OSPF

It is recommended to set the OSPF timers of both OSPF neighbors (core and edge router) to minimize downtime upon a reboot of the LibreQoS server.

- hello interval
- dead