IntenseWebs/LibreQoS
3
0 Fork 0
mirror of https://github.com/LibreQoE/LibreQoS.git synced 2026-10-26 06:40:05 -05:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #825 from LibreQoE/doc-cleaning

Browse Source 
Documentation cleanup
This commit is contained in:
Robert Chacón authored and GitHub committed 2025-10-02 09:38:38 -06:00
2 parents 9fe34b9324 d367be43e3
commit 09d406d4ed
58 files changed
-8209
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs-es/ChangeNotes/v1.5.md
-5
View File
@@ -1,5 +0,0 @@
# Resumen de cambios en LibreQoS v1.4 a v1.5
## Configuración Unificada
Toda la configuración se ha trasladado a `/etc/lqos.conf`.
docs-es/Legacy/v1.3.1.md
-434
View File
@@ -1,434 +0,0 @@
# LibreQoS v1.3.1
## LibreQoS v1.3.1 Installation & Usage Guide - Physical Server and Ubuntu 22.04
## Notes for upgrading from v1.2 or prior
### Custom CRM Integrations
If you use a custom CRM integration, please ensure your integration uses a unique circuit identifier for the 'Circuit ID' field in ShapedDevices.csv. This is now required in v1.3 in order to make partial reloading possible. A good choice for this ID would be internet service plan unique ID, or the subscriber site ID your CRM provides for customer service locations. Multiple devices within the same circuit would use the same 'Circuit ID', but aside from that, all Circuit IDs should be distinct. The built-in Splynx and UISP integrations for v1.3 handle this automatically.
## Network Design Assumptions
Officially supported configuration:
- Edge and Core routers with MTU 1500 on links between them
- If you use MPLS, you would terminate MPLS traffic at the core router.
LibreQoS cannot decapsulate MPLS on its own.
- OSPF primary link (low cost) through the server running LibreQoS
- OSPF backup link
![](https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/design.png)
Is it possible to use LibreQoS in-line without a core router, but that setup requires depending on STP instead of OSPF, which can cause issues. Such configurations are not officially supported.
## Network Interface Card
LibreQoS requires a NIC with 2 or more RX/TX queues and XDP support. While many cards theoretically meet these requirements, less commonly used cards tend to have unreported driver bugs which impede XDP functionality and make them unusable for our purposes. At this time we can only recommend Intel x520, Intel x710, and Nvidia (ConnectX-5 or newer) NICs.
## Server Setup
Disable hyperthreading on the BIOS/UEFI of your host system. Hyperthreaading is also known as Simultaneous Multi Threading (SMT) on AMD systems. Disabling this is very important for optimal performance of the XDP cpumap filtering and, in turn, throughput and latency.
- Boot, pressing the appropriate key to enter the BIOS settings
- For AMD systems, you will have to navigate the settings to find the "SMT Control" setting. Usually it is under something like ```Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control``` Once you find it, switch to "Disabled" or "Off"
- For Intel systems, you will also have to navigate the settings to find the "hyperthrading" toggle option. On HP servers it's under ```System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.```
- Save changes and reboot
## Install Ubuntu
Download Ubuntu Server 22.04 from <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.
1. Boot Ubuntu Server from USB.
2. Follow the steps to install Ubuntu Server.
3. If you use a Mellanox network card, the Ubuntu Server installer will ask you whether to install the mellanox/intel NIC drivers. Check the box to confirm. This extra driver is important.
4. On the Networking settings step, it is recommended to assign a static IP address to the management NIC.
5. Ensure SSH server is enabled so you can more easily log into the server later.
6. You can use scp or sftp to access files from your LibreQoS server for easier file editing. Here's how to access via scp or sftp using an [Ubuntu](https://www.addictivetips.com/ubuntu-linux-tips/sftp-server-ubuntu/) or [Windows](https://winscp.net/eng/index.php) machine.
## Use Installer Script (For Sponsors - Skip If Not Applicable)
Sponsors can use the LibreQoS-Installer script. This script does the following:
- Disables IRQbalance
- Disables required offloading types using service
- Creates a bridge between two interfaces - applied by the above service at each boot
- Installs LibreQoS and cpumap-pping
Once complete - skip to [this section](https://github.com/LibreQoE/LibreQoS/wiki/LibreQoS-v1.3-Installation-&-Usage-Guide-Physical-Server-and-Ubuntu-22.04#install-influxdb-for-graphing) of the guide.
## Setup
### Disable IRQbalance
```shell
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance
```
### Disable Offloading
We need to disable certain hardware offloading features, as they break XDP, used by XDP-CPUMAP-TC to send traffic to appropriate CPUs.
You can create a bash script to disabled these offload features upon boot.
```shell
sudo nano /usr/local/sbin/offloadOff.sh
```
Enter the following
```shell
#!/bin/sh
ethtool --offload eth1 gso off tso off lro off sg off gro off
ethtool --offload eth2 gso off tso off lro off sg off gro off
```
Replace eth1 and eth2 with your two shaper interfaces (order doesn't matter).
Then create
```shell
sudo nano /etc/systemd/system/offloadOff.service
```
With the following
```text
[Unit]
After=network.service
[Service]
ExecStart=/usr/local/sbin/offloadOff.sh
[Install]
WantedBy=default.target
```
Then change permissions and enable the service with
```shell
sudo chmod 664 /etc/systemd/system/offloadOff.service
sudo chmod 744 /usr/local/sbin/offloadOff.sh
sudo systemctl daemon-reload
sudo systemctl enable offloadOff.service
sudo reboot
```
### Add a bridge between edge/core interfaces
From the Ubuntu VM, create a linux interface bridge - br0 - with the two shaping interfaces.
Find your existing .yaml file in /etc/netplan/ with
```shell
cd /etc/netplan/
ls
```
Then edit the .yaml file there with
```shell
sudo nano XX-cloud-init.yaml
```
with XX corresponding to the name of the existing file.
Editing the .yaml file, we need to define the shaping interfaces (here, ens19 and ens20) and add the bridge with those two interfaces. Assuming your interfaces are ens18, ens19, and ens20, here is what your file might look like:
```yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens18:
addresses:
- 10.0.0.12/24
routes:
- to: default
via: 10.0.0.1
nameservers:
addresses:
- 1.1.1.1
- 8.8.8.8
search: []
ens19:
dhcp4: no
ens20:
dhcp4: no
version: 2
bridges:
br0:
interfaces:
- ens19
- ens20
```
Make sure to replace 10.0.0.12/24 with your LibreQoS VM's address and subnet, and to replace the default gateway 10.0.0.1 with whatever your default gateway is.
Then run
```shell
sudo netplan apply
```
### Install LibreQoS and dependencies
Cd to your preferred directory and download the latest release
```shell
cd home/$USER/
sudo apt update
sudo apt install python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
git clone https://github.com/rchac/LibreQoS.git
git checkout v1.3.1
```
### Install and compile cpumap-pping
```shell
cd home/$USER/LibreQoS/src
git submodule update --init
cd cpumap-pping/
git submodule update --init
cd src/
make
```
### Install InfluxDB for Graphing
To install InfluxDB 2.x., follow the steps at [https://portal.influxdata.com/downloads/](https://portal.influxdata.com/downloads/).
For high throughput networks (5+ Gbps) you will likely want to install InfluxDB to a separate machine or VM from that of the LibreQoS server to avoid CPU load.
Restart your system that is running InfluxDB
```shell
sudo reboot
```
Check to ensure InfluxDB is running properly. This command should show "Active: active" with green dot.
```shell
sudo service influxdb status
```
Check that Web UI is running:<br>
```shell
http://SERVER_IP_ADDRESS:8086
```
Create Bucket
- Data > Buckets > Create Bucket
Call the bucket "libreqos" (all lowercase, without quotes).<br>
Have it store as many days of data as you prefer. 7 days is standard.<>
Import Dashboard
- Boards > Create Dashboard > Import Dashboard
Then upload the file [influxDBdashboardTemplate.json](https://github.com/rchac/LibreQoS/blob/main/src/influxDBdashboardTemplate.json) to InfluxDB.
[Generate an InfluxDB Token](https://docs.influxdata.com/influxdb/cloud/security/tokens/create-token/). It will be added to ispConfig.py in the following steps.
### Modify ispConfig.py
Copy ispConfig.example.py to ispConfig.py and edit as needed
```shell
cd /home/$USER/LibreQoS/src/
cp ispConfig.example.py ispConfig.py
nano ispConfig.py
```
- Set upstreamBandwidthCapacityDownloadMbps and upstreamBandwidthCapacityUploadMbps to match the bandwidth in Mbps of your network's upstream / WAN internet connection. The same can be done for generatedPNDownloadMbps and generatedPNUploadMbps.
- Set interfaceA to the interface facing your core router (or bridged internal network if your network is bridged)
- Set interfaceB to the interface facing your edge router
- Set ```enableActualShellCommands = True``` to allow the program to actually run the commands.
### Integrations
Integrations now share a common framework thanks to [this pull](https://github.com/rchac/LibreQoS/pull/145). This also allows for graphing the network topology with graphviz.
#### UISP Integration
To run the UISP Integration, use
```shell
python3 integrationUISP.py
```
On the first successful run, it will create a network.json and ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the UISP integration is run.
You have the option to run integrationUISP.py automatically on boot and every 30 minutes, which is recommended. This can be enabled by setting ```automaticImportUISP = True``` in ispConfig.py
### Network.json
Network.json allows ISP operators to define a Hierarchical Network Topology, or Flat Network Topology.
For networks with no Parent Nodes (no strictly defined Access Points or Sites) edit the network.json to use a Flat Network Topology with
```nano network.json```
setting the following file content:
```json
{}
```
If you plan to use the built-in UISP or Splynx integrations, you do not need to create a network.json file quite yet.
If you plan to use the built-in UISP integration, it will create this automatically on its first run (assuming network.json is not already present). You can then modify the network.json to more accurately reflect your topology.
If you will not be using an integration, you can manually define the network.json following the template file - network.example.json
```text
+-----------------------------------------------------------------------+
| Entire Network |
+-----------------------+-----------------------+-----------------------+
| Parent Node A | Parent Node B | Parent Node C |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 | | | | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+
```
#### Manual Editing
- Modify the network.json file using your preferred JSON editor (Geany for example)
- Parent node name must match that used for clients in ShapedDevices.csv
### ShapedDevices.csv
If you are using an integration, this file will be automatically generated. If you are not using an integration, you can manually edit the file.
- Modify the ShapedDevices.csv file using your preferred spreadsheet editor (LibreOffice Calc, Excel, etc), following the template file - ShapedDevices.example.csv
- An IPv4 address or IPv6 address is required for each entry.
- The Access Point or Site name should be set in the Parent Node field. Parent Node can be left blank for flat networks.
- The ShapedDevices.csv file allows you to set minimum guaranteed, and maximum allowed bandwidth per subscriber.
- The minimum allowed plan rates for Circuits are 2Mbit. Bandwidth min and max should both be above that threshold.
- Recommendation: set the min bandwidth to something like 25/10 and max to 1.15X advertised plan rate by using bandwidthOverheadFactor = 1.15
- This way, when an AP hits its ceiling, users have any remaining AP capacity fairly distributed between them.
- Ensure a reasonable minimum bandwidth minimum for every subscriber, allowing them to utilize up to the maximum provided when AP utilization is below 100%.
Note regarding SLAs: For customers with SLA contracts that guarantee them a minimum bandwidth, set their plan rate as the minimum bandwidth. That way when an AP approaches its ceiling, SLA customers will always get that amount.
![image](https://user-images.githubusercontent.com/22501920/200134960-28709d0f-48fe-4129-b4fd-70b204cade2c.png)
## How to run LibreQoS
### One-Time / Debug Runs
One-time runs show the response from the terminal for each filter rule applied, and can be very helpful for debugging and to make sure it is correctly configured.
- Modify setting parameters in ispConfig.py to suit your environment
- For one-time runs, use
```shell
sudo ./LibreQoS.py
```
- To use the debug mode with more verbose output, use:
```shell
sudo ./LibreQoS.py --debug
```
### Running as a service
To run as a service, we create a systemd service to run scheduler.py.
scheduler.py does the following:
- On start: Run a full setup of queues
- Every 30 minutes: Update queues, pulling new configuration from CRM integration if enabled
On Linux distributions that use systemd, such as Ubuntu, we create
```shell
sudo nano /etc/systemd/system/LibreQoS.service
```
Then paste the text below, replacing "/home/YOUR_USERNAME/LibreQoS" with wherever you downloaded LibreQoS to. Be sure to replace YOUR_USERNAME with your actual username, because otherwise when the root user executes it, it will look in the wrong directory.
```text
[Unit]
After=network.service
[Service]
WorkingDirectory=/home/YOUR_USERNAME/LibreQoS/src
ExecStart=/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/scheduler.py
ExecStopPost=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
ExecStop=/bin/bash -c '/usr/bin/python3 /home/YOUR_USERNAME/LibreQoS/src/LibreQoS.py --clearrules'
Restart=always
[Install]
WantedBy=default.target
```
Then run
```shell
sudo chmod 664 /etc/systemd/system/LibreQoS.service
sudo systemctl daemon-reload
sudo systemctl enable LibreQoS.service
```
You can start the service using
```shell
sudo systemctl start LibreQoS.service
```
You can check the status of the service using
```shell
sudo systemctl status LibreQoS.service
```
You can restart the service to refresh any changes you've made to the ShapedDevices.csv file by doing
```shell
sudo systemctl restart LibreQoS.service
```
You can also stop the service to remove all queues and IP rules by doing
```shell
sudo systemctl stop LibreQoS.service
```
### Crontab
- At 4AM: Runs a full reload of all queues to make sure they perfectly match queueStructure.py and that any changes to network.json can be applied.
First, check to make sure the cron job does not already exist.
```shell
sudo crontab -l | grep -q 'LibreQoS' && echo 'entry exists' || echo 'entry does not exist'
```
The above should output "entry does not exist". If so, proceed to add it with:
```shell
(sudo crontab -l 2>/dev/null; echo "0 4 * * * /bin/systemctl try-restart LibreQoS") | sudo crontab -
sudo /etc/init.d/cron start
```
## Common Issues
### Program Running, But Traffic Not Shaping
In ispConfig.py, make sure the edge and core interfaces correspond to correctly to the edge and core. Try swapping the interfaces to see if shaping starts to work.
### RTNETLINK answers: Invalid argument
This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](#network-interface-card).
## Performance Tuning
### OSPF
It is recommended to set the OSPF timers of both OSPF neighbors (core and edge router) to minimize downtime upon a reboot of the LibreQoS server.
- hello interval
- dead
docs-es/Quickstart/configuration.md
-103
View File
@@ -1,103 +0,0 @@
# Configure LibreQoS
## Configuración de lqos.conf
Copie la configuración archivo del demonio lqosd en `/etc`:
```shell
cd /opt/libreqos/src
sudo cp lqos.example /etc/lqos.conf
```
Ahora edite el archivo para igualar su configuración con
```shell
sudo nano /etc/lqos.conf
```
Cambie `enp1s0f1` y `enp1s0f2` para coincidir con sus interfaces de red. No importa cuál sea cuál. Tenga en cuenta que está emparejando las interfaces, por lo que cuando ingresa por primera vez enps0f<ins>**1**</ins> en la primera línea, el parámetro es enp1s0f<ins>**2**</ins> (reemplace con los nombres de interfaz reales).
- Primera Línea: `name = "enp1s0f1", redirect_to = "enp1s0f2"`
- Segunda Línea: `name = "enp1s0f2", redirect_to = "enp1s0f1"`
Luego, si estará usando Bifrost/XDP, configure `use_xdp_bridge = true` bajo la misma sección `[bridge]`.
## Configuración de ispConfig.py
Copie ispConfig.example.py a ispConfig.py y edite según necesidad
```shell
cd /opt/libreqos/src/
cp ispConfig.example.py ispConfig.py
nano ispConfig.py
```
- Configure upstreamBandwidthCapacityDownloadMbps y upstreamBandwidthCapacityUploadMbps para que coincidan con el ancho de banda de subida de su red/WAN o conexión a Internet. Lo mismo puede hacerse para generatedPNDownloadMbps y generatedPNUploadMbps.
- Estableza la interfaceA en la interfaz orientada a su router principal (o red interna en puente si su red está en esa modalidad)
- Estableza la interfaceB en la interfaz orientada hacia su enrutador de borde.
- Establezca ```enableActualShellCommands = True``` para permitir que el programa ejecute los comandos propiamente tal.
## Network.json
Network.json permite a los operadores de ISP definir una Topología de Red Jerárquica, o una Topología de Red Plana.
Para redes sin Nodos Principales (sin Puntos de Acceso ni Sitios definidos estrictamentens) edite el archivo network.json para usar una Topología de Red Plana con
```nano network.json```
configurando el siguiente contenido de archivo:
```json
{}
```
Si planea usar las integraciones UISP o Splynx incorporadas, todavía no es necesario crear un archivo network.json.
Si planea usar las integraciones UISP incorporadas, este se creará automáticamente en su primera ejecución (asumiendo que network.json no esté ya presente). Después puede modificar network.json para que refleje más ajustadamente su topología.
Si no usará una integración, puede definirlo manualmente en network.json siguiendo el archiv plantilla - network.example.json
```text
+-----------------------------------------------------------------------+
| Entire Network |
+-----------------------+-----------------------+-----------------------+
| Parent Node A | Parent Node B | Parent Node C |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 | | | | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+
```
## Configuración Manual
Usted puede usar
```shell
python3 csvToNetworkJSON.py
```
Para convertir manualNetwork.csv como un archivo network.json.
manualNetwork.csv puede ser copiado desde un archivo plantilla, manualNetwork.template.csv
Nota: El nombre del nodo principal debe coincidir con el utilizado para los clientes en ShapedDevices.csv
## ShapedDevices.csv
Si está realizando una integración, este archivo será generado automáticamente. Si no está haciendo una integración, puede editar manualmente el archivo.
### Edición Manual
- Modifique el archivo ShapedDevices.csv su editor de planillas electrónicas preferido (LibreOffice Calc, Excel, etc), siguiendo el archivo de plantilla - ShapedDevices.example.csv
- Se requiere ID Circuit. Este debe ser una cadena de algún tipo (un entero es suficiente, se analiza como cadena). NO debe incluir símbolos numéricos (#). Cada circuito necesita un CircuitID único, el que no se puede reutilizar. Aquí, un circuito significa básicamente la ubicación del cliente. Si un cliente tiene múltiples ubicaciones en diferentes partes de la red, utilice un ID de circuito único para cada una de ellas.
- Al menos una dirección IPv4 o IPv6 se requiere para cada entrada.
- El nombre del Punto de Acceso o Sitio debe configurarse en el campo de "Nodo Principal". Este campo puede quedar en blanco para redes planas.
- El archivo ShapedDevices.csv permite establecer un ancho de banda mínimo garantizado y un máximo permitido por suscriptor.
- La velocidad mínima permitida para los circuitos es de 2 Mbit. El ancho de banda mínimo de ser superior a ese umbral.
- Recomendación: establezca el ancho de banda mínimo en algo así como 25/10 y un máximo de 1.15X en base a la tarifa del plan publicitado utilizando bandwidthOverheadFactor = 1.15
- De esta manera, cuando un AP alcanza su límite, los usuarios tienen toda la capacidad de AP restante distribuida equitativamente entre ellos.
- Garantice un ancho de banda mínimo para cada suscriptor, permitiéndoles utilizar el máximo proporcionado cuando la utilización del AP sea inferior al 100%.
Nota acerca de los SLAs: Para los clientes que tienen contratos de SLA que les garanticen un ancho de banda mínimo, establezcan la tarifa de su plan como el ancho de banda mínimo. De esta manera, cuando un punto de acceso se acerque a su límite, los clientes con SLA siempre recibirán esa cantidad.
![image](https://user-images.githubusercontent.com/22501920/200134960-28709d0f-48fe-4129-b4fd-70b204cade2c.png)
Una vez completada la configuración, estará listo para ejecutar la aplicación e iniciar [Deamons](./services-and-run.md)
docs-es/Quickstart/networkdesignassumptions.md
-46
View File
@@ -1,46 +0,0 @@
# Supuestos de Diseño de Red
## Configuraciones oficialmente soportadas
- LibreQoS se ubica dentro de una red, generalmente entre un enrutador de borde (NAT, firewall) y un enrutador central (distribución a sitios a través de la red).
- Si usa NAT/CG-NAT, coloque LibreQoS en lpínea al sur de donde se aplica NAT, debido a que LibreQoS necesita dar formar a direcciones internas (100.64.0.0/12), no a IP públicas posteriores a NAT.
- - Los enrutadores de borde y de núcleo deben tener 1500 MTU en los enlaces entre ellos
- Si usa MPLS, debería terminar el tráfico MPLS en el enrutador principal. LibreQoS no puede desencapsular MPLS por sí solo.
- Enlace principal OSPF (bajo costo) a través del servidor que ejecuta LibreQoS
- Enlace de respaldo OSPF (alto costo, quizás 200 por ejemplo)
![Configuración oficial](https://raw.githubusercontent.com/LibreQoE/LibreQoS/main/docs/design.png)
## Configuración del banco de pruebas
Cuando pruebe LibreQoS por primera vez, recomendamos implementar un banco de pruebas a pequeña escala para verlo en acción.
![image](https://github.com/user-attachments/assets/6174bd29-112d-4b00-bea8-41314983d37a)
### Tarjeta de Interfaz de Red
```{note}
Debe tener algouno de estos:
- Una sola NIC con dos interfaces,
- dos NIC con una sola interfaz,
- Interfaz 2x VLANs interface (utilizando una o dos NIC).
```
LibreQoS requiere que las tarjetas de red (NIC) tengan dos o más colas RX/TX y compatibilidad con XDP. Si bien muchas tarjetas cumplen teóricamente estos requisitos, las tarjetas menos utilizadas suelen presentar errores de controlador no reportados que impiden la funcionalidad XDP y las inutilizan para nuestros propósitos. Actualmente, recomendamos las tarjetas de red (NIC) Intel x520, Intel x710 y Nvidia (ConnectX-5 o posterior). No podemos garantizar la compatibilidad con otras tarjetas.
## Configuración alternativa (no compatible oficialmente)
Esta configuración alternativa utiliza el Protocolo de árbol de expansión (STP) para modificar la ruta de datos en caso de que el dispositivo LibreQoS esté fuera de línea por mantenimiento u otro problema.
```{note}
La mayoría de estas consideraciones se aplican a la configuración alternativa y a la configuración admitida oficialmente.```
- LibreQoS se coloca en línea en la red, generalmente entre un enrutador de borde (NAT, firewall) y un enrutador central (distribución a sitios a través de la red).
- Si usa NAT/CG-NAT, coloque LibreQoS en línea al sur de donde se aplica NAT, ya que LibreQoS necesita dar forma a direcciones internas (100.64.0.0/12), no a IP públicas posteriores a NAT.
- El enrutador de borde y el conmutador central deben tener 1500 MTU en los enlaces entre ellos
- Si usa MPLS, debería terminar el tráfico MPLS en algún lugar al sur del conmutador central/distribuidor. LibreQoS no puede desencapsular MPLS por sí solo.
- Enlace principal de Spanning Tree (bajo costo) a través del servidor que ejecuta LibreQoS
- Enlace de respaldo de Spanning Tree (alto costo, quizás 80 por ejemplo)
Tenga en cuenta que si utiliza enlaces con diferentes anchos de banda, por ejemplo, 10 Gbps a través de LibreQoS y 1 Gbps entre el conmutador central y el enrutador de borde, es posible que deba ser más intencional con los costos de STP.
![Configuración Alternativa](../stp-diagram.png)
docs-es/Quickstart/quickstart-libreqos-1.4.md
-37
View File
@@ -1,37 +0,0 @@
# Instalar LibreQoS 1.4
## Actualización desde la versión v1.3
### Eliminar offloadOff.service
```shell
sudo systemctl disable offloadOff.service
sudo rm /usr/local/sbin/offloadOff.sh /etc/systemd/system/offloadOff.service
```
### Eliminar las tareas cron de la versión v1.3
Ejecute ```sudo crontab -e``` y elimine cualquier entrada relacionada con LibreQoS desde v1.3.
## Instalación sencilla desde el paquete .Deb (recomendada)
Utilice el paquete .deb de la [última versión v1.4 ](https://github.com/LibreQoE/LibreQoS/releases/).
```shell
sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" | sudo tee -a /etc/apt/sources.list.d/libreqos.list
sudo wget -O - -q http://stats.libreqos.io/repo.asc | sudo apt-key add -
apt-get update
apt-get install libreqos
```
Se le realizarán algunas preguntas sobre su configuración, y el demonio de administración y el servidor web se iniciarán automáticamente. Vaya a http://<your_ip>:9123/ para finalizar la instalación.
## Instalación compleja (no recomendada)
```{note}
Use esta instalación si quiere implementar constantemente desde la rama principal en Github. ¡Solo para usuarios experimentados!
```
[Instalación Compleja](../TechnicalDocs/complex-install.md)
Ahora está listo para [Configurar](./configuration.md) LibreQoS!
docs-es/Quickstart/quickstart-prereq.md
-87
View File
@@ -1,87 +0,0 @@
# Configuración del servidor - Requisitos previos
Desactive el hyperthreading en la BIOS/UEFI de su sistema host. El hyperthreading también se conoce como multihilo simultáneo (SMT) en sistemas AMD. Desactivarlo es fundamental para optimizar el filtrado de CPUmap de XDP y, por consiguiente, el rendimiento y la latencia.
- Arranque, presionando la tecla apropiada para ingresar a la configuración del BIOS
- En sistemas AMD, deberá navegar por la configuración para encontrar la opción "Control SMT". Normalmente se encuentra en ```Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control``` Una vez que lo encuentres, cambia a "Disabled" o "Off"
- Para los sistemas Intel, también deberá navegar por la configuración para encontrar la opción de "hyperthreading". En los servidores HP, se encuentra en ```System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.```
- Guardar cambios y reiniciar
## Instalar en servidor Ubuntu
Recomendamos Ubuntu Server porque su versión de kernel suele coincidir estrechamente con las versiones principales de Linux. Nuestra documentación actual asume Ubuntu Server. Para ejecutar LibreQoS v1.4, se requiere el kernel de Linux 5.11 o superior, ya que incluye algunos parches importantes para XDP. Ubuntu Server 22.04 usa el kernel 5.13, que cumple con este requisito.
Puede descargar Ubuntu Server 22.04 desde <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.
1. Arrancar el servidor Ubuntu desde USB.
2. Siga los pasos para instalar Ubuntu Server.
3. Si usa una tarjeta de red Mellanox, el instalador de Ubuntu Server le preguntará si desea instalar los controladores de NIC Mellanox/Intel. Marque la casilla para confirmar. Este controlador adicional es importante.
4. En el paso de configuración de red, se recomienda asignar una dirección IP estática a la NIC de administración.
5. Asegúrese de que el servidor SSH esté habilitado para que pueda iniciar sesión en el servidor más fácilmente más tarde.
6. Puedes usar SCP o SFTP para acceder a archivos desde tu servidor LibreQoS y facilitar su edición. Aquí te explicamos cómo acceder mediante SCP o SFTP usando una máquina[Ubuntu](https://www.addictivetips.com/ubuntu-linux-tips/sftp-server-ubuntu/) o [Windows](https://winscp.net/eng/index.php).
### Elija el tipo de puente
Hay dos opciones para que el puente pase datos a través de sus dos interfaces:
- Puente acelerado Bifrost XDP
- Puente Linux regular
Se recomienda el puente Bifrost para NIC Intel con soporte XDP, como X520 y X710.Se recomienda el puente Linux normal para las NIC Nvidia/Mellanox como la serie ConnectX-5 (que tienen un rendimiento de puente superior) y configuraciones de VM que usan NIC virtualizadas.
Para utilizar el puente Bifrost, omita la sección del puente de Linux normal a continuación y asegúrese de habilitar Bifrost/XDP en lqos.conf algunas secciones a continuación.
### Agregar un puente Linux normal (si no se utiliza el puente Bifrost XDP)
Desde la máquina virtual Ubuntu, cree un puente de interfaz Linux (br0) con las dos interfaces de modelado.
Busque su archivo .yaml existente en /etc/netplan/ con
```shell
cd /etc/netplan/
ls
```
Luego edite el archivo .yaml allí con
```shell
sudo nano XX-cloud-init.yaml
```
Con XX correspondiente al nombre del archivo existente.
229 / 5.000
Al editar el archivo .yaml, necesitamos definir las interfaces de modelado (en este caso, ens19 y ens20) y agregar el puente con estas dos interfaces. Suponiendo que las interfaces sean ens18, ens19 y ens20, el archivo podría verse así:
```yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens18:
addresses:
- 10.0.0.12/24
routes:
- to: default
via: 10.0.0.1
nameservers:
addresses:
- 1.1.1.1
- 8.8.8.8
search: []
ens19:
dhcp4: no
ens20:
dhcp4: no
version: 2
bridges:
br0:
interfaces:
- ens19
- ens20
```
Asegúrese de reemplazar 10.0.0.12/24 con la dirección y subred de su VM LibreQoS, y de reemplazar la puerta de enlace predeterminada 10.0.0.1 con la que sea su puerta de enlace predeterminada.
Entonces ejecute
```shell
sudo netplan apply
```
docs-es/Quickstart/services-and-run.md
-71
View File
@@ -1,71 +0,0 @@
# Demonios de LibreQoS
lqosd
- Gestiona el código XDP. Construido en Rust.
lqos_node_manager
- Ejecuta la GUI disponible en http://a.b.c.d:9123
lqos_scheduler
- lqos_scheduler maneja estadísticas y realiza actualizaciones continuas de los modeladores de LibreQoS, incluida la extracción de cualquier integración de CRM habilitada (UISP, Splynx).
- Al iniciar: ejecuta una configuración completa de colas
- Cada 10 segundos: Grafica estadísticas de ancho de banda y latencia
- Cada 30 segundos: Actualiza colas, extrayendo nueva configuración de la integración de CRM si está habilitada
## Ejecutar daemons con systemd
Puede configurar `lqosd`, `lqos_node_manager`, y `lqos_scheduler` como servicios systemd.
```shell
sudo cp /opt/libreqos/src/bin/lqos_node_manager.service.example /etc/systemd/system/lqos_node_manager.service
sudo cp /opt/libreqos/src/bin/lqosd.service.example /etc/systemd/system/lqosd.service
sudo cp /opt/libreqos/src/bin/lqos_scheduler.service.example /etc/systemd/system/lqos_scheduler.service
```
Finalmente, correr
```shell
sudo systemctl daemon-reload
sudo systemctl enable lqosd lqos_node_manager lqos_scheduler
```
Ahora puede apuntar un navegador web a `http://a.b.c.d:9123` (reemplazar `a.b.c.d` con la dirección IP de administración de su servidor de modelado) y disfrute de una vista en tiempo real de su red.
## Depuración lqos_scheduler
En el fondo, lqos_scheduler corre scheduler.py, que a su vez corre LibreQoS.py
Las ejecuciones únicas de estos componentes individuales pueden ser muy útiles para la depuración y para asegurarse de que todo esté configurado correctamente.
Primero, detener lqos_scheduler
```shell
sudo systemctl stop lqos_scheduler
```
Para ejecuciones iniciales de of LibreQoS.py, utilice
```shell
sudo ./LibreQoS.py
```
- Para utilizar el modo de depuración con una salida más detallada, utilice:
```shell
sudo ./LibreQoS.py --debug
```
Para confirmar que lqos_scheduler (scheduler.py) puede funcionar correctamente, ejecute:
```shell
sudo python3 scheduler.py
```
Una vez que haya eliminado todos los errores, reinicie lqos_scheduler con
```shell
sudo systemctl start lqos_scheduler
```
docs-es/Quickstart/share.md
-32
View File
@@ -1,32 +0,0 @@
# Comparte tu antes y después
Le pedimos que comparta una captura de pantalla anónima de su implementación de LibreQoS antes (modo de solo monitor) y después (cola habilitada) en el [Chat LibreQoS ](https://chat.libreqos.io/join/fvu3cerayyaumo377xwvpev6/). T
Esto nos ayuda a evaluar el impacto de nuestro software. Y también nos alegra.
1. Habilitar el modo de solo monitor
2. Modo Klingon (Redactar información del cliente)
3. Captura de pantalla
4. Reanudar las colas regulares
5. Captura de pantalla
## Habilitar el modo de solo monitor
```shell
sudo systemctl stop lqos_scheduler
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
```
## Modo Klingon
Vaya a la interfaz web y haga clic en Configuración. Active la opción "Redactar información del cliente" (modo de captura de pantalla) y luego "Aplicar cambios".
## Reanudar la cola normal
```shell
sudo systemctl start lqos_scheduler
```
## Captura de pantalla
Para generar una captura de pantalla, acceda a la interfaz web y haga clic en Configuración. Active "Ocultar información del cliente" (modo de captura de pantalla), "Aplicar cambios" y, a continuación, vuelva al panel de control para realizar una captura de pantalla.
docs-es/SystemRequirements/Compute.md
-90
View File
@@ -1,90 +0,0 @@
## Requerimientos de Sistema
### Servidor físico
* LibreQoS requiere un servidor físico x86_64 de manera dedicada.
* Aunque es técnicamente posible correr en una VM, no está oficialmente soportado, y conlleva una pérdida de rendimiento significativa del 30 % (incluso al usar la transferencia de NIC). En las máquinas virtuales, la transferencia de NIC es necesaria para un rendimiento superior a 1 Gbps (XDP frente a XDP genérico).
### CPU
* 2 o más cores de CPU
* Una CPU con un rendimiento sólido [single-thread performance](https://www.cpubenchmark.net/singleThread.html#server-thread) que se ajuste a su presupuesto. El encolado consume muchos recursos de la CPU y requiere un alto rendimiento de un solo hilo.
El rendimiento de la CPU de un solo subproceso determinará la capacidad máxima de un solo HTB (núcleo de CPU) y, a su vez, la capacidad máxima de cualquier nodo de nivel superior en la jerarquía de la red (por ejemplo, los sitios de nivel superior de su red). Esto también afecta el plan de velocidad máxima que puede ofrecer a sus clientes dentro de márgenes seguros.
| Máx nodos nivel | Puntuación de un hilo |
| --------------------| ------------------------ |
| 1 Gbps | 1000 |
| 2 Gbps | 1500 |
| 3 Gbps | 2000 |
| 5 Gbps | 4000 |
| Plan máximo cliente | Puntuación de un hilo |
| --------------------| ------------------------ |
| 100 Mbps | 1000 |
| 250 Mbps | 1250 |
| 500 Mbps | 1500 |
| 1 Gbps | 1750 |
| 2.5 Gbps | 2000 |
| 5 Gbps | 4000 |
A continuación se muestra una tabla de capacidad agregada aproximada, suponiendo una CPU con una [hilo único](https://www.cpubenchmark.net/singleThread.html#server-thread) desempeño de 1000 / 2000 / 4000:
| Núcleos CPU | Puntuación de un solo hilo = 1000 | Puntuación de un solo hilo = 2000 | Puntuación de un solo hilo = 4000 |
|-----------|----------------------------|----------------------------|----------------------------|
| 2 | 1 Gbps | 3 Gbps | 7 Gbps |
| 4 | 3 Gbps | 5 Gbps | 13 Gbps |
| 6 | 4 Gbps | 8 Gbps | 20 Gbps |
| 8 | 5 Gbps | 10 Gbps | 27 Gbps |
| 16 | 10 Gbps | 21 Gbps | 54 Gbps |
| 32 | 21 Gbps | 42 Gbps | 108 Gbps |
| 64 | 42 Gbps | 83 Gbps | 216 Gbps |
### Memoria
* RAM Recomendada:
| RAM (usando CAKE)| Máx Suscriptores|
| ---------------- | --------------- |
| 8 GB | 1,000 |
| 16 GB | 2,000 |
| 32 GB | 5,000 |
| 64 GB | 10,000 |
| 128 GB | 20,000 |
| 256 GB | 40,000 |
### Recomendaciones de servidor
Aquí hay algunas opciones de servidor listas para usar y convenientes para considerar:
| **Rendimiento** | 2.5 Gbps | 10 Gbps | 10 Gbps | 10 Gbps | 25 Gbps |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Por Nodo / Por núcleo CPU** | 1 Gbps | 3 Gbps | 5 Gbps | 5 Gbps | 3 Gbps |
| **Modelo** | [Supermicro SYS-E102-13R-E](https://store.supermicro.com/us_en/compact-embedded-iot-i5-1350pe-sys-e102-13r-e.html) | [Supermicro AS-1115S-FWTRT](https://store.supermicro.com/us_en/1u-amd-epyc-8004-compact-server-as-1115s-fwtrt.html) | [Supermicro SYS-511R-M](https://store.supermicro.com/us_en/mainstream-1u-sys-511r-m.html) | [Dell PowerEdge R260](https://www.dell.com/en-us/shop/dell-poweredge-servers/new-poweredge-r260-rack-server/spd/poweredge-r260/pe_r260_tm_vi_vp_sb?configurationid=2cd33e43-57a3-4f82-aa72-9d5f45c9e24c) | [Supermicro AS-1115S-FWTRT](https://store.supermicro.com/us_en/1u-amd-epyc-8004-compact-server-as-1115s-fwtrt.html) |
| **Opción CPU** | Default | 8124P | E-2488 | E-2456 | 8534P |
| **Opción RAM** | 2x8GB | 2x16GB | 2x32GB | 2x32GB | 4x16GB |
| **Opción NIC** | Incorporado | Mellanox (2 x SFP28) | 10-Gigabit X710-BM2 (2 x SFP+) | Intel X710-T2L (2 x 10G RJ45) | Mellanox (2 x SFP28) |
| **Extras** | [USB-C RJ45](https://www.amazon.com/Anker-Ethernet-PowerExpand-Aluminum-Portable/dp/B08CK9X9Z8/) | | | | |
| **Rango de Temperatura** | 0°C ~ 40°C | 0°C ~ 40°C | 0°C ~ 40°C | 5–40°C | 0°C ~ 40°C |
| **Rango de Temperatura** | (32°F ~ 104°F) | (32°F ~ 104°F) | (32°F ~ 104°F) | (41–104°F) | (32°F ~ 104°F) |
### Requisitos de la interfaz de red
* Una interfaz de red de administración completamente independiente de las interfaces de modelado de tráfico. Normalmente, esta sería la interfaz Ethernet integrada en la placa base.
* Una tarjeta de interfaz de red dedicada para dos interfaces de modelado
A continuación se enumeran las tarjetas de interfaz de red compatibles oficialmente para las dos interfaces de modelado.:
| Controlador NIC | Velocidad Puerto | Modelos Sugeridos | Problemas conocidos |
|------------------------|------------------|-----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
| Intel X520 | 10 Gbps | | Compatibilidad de módulos* |
| Intel X710 | 10 Gbps | [X710-BM2 10G]( https://www.fs.com/products/75600.html?now_cid=4253) | Compatibilidad de módulos* |
| Intel XXV710 | 10 / 25 Gbps | [XXV710 25G](https://www.fs.com/products/75604.html?attribute=67774&id=1709896) | Módulo de compatibilidad* |
| Intel XL710 | 10 / 40 Gbps | [XL710-BM2 40G](https://www.fs.com/products/75604.html?attribute=67774&id=1709896 ) | Compatibilidad de módulos* |
| Mellanox ConnectX-4 Lx | 10/25/40/50 Gbps | | No hay problemas conocidos. |
| Mellanox ConnectX-5 | 100 Gbps | [MCX516A-CCAT 100G](https://www.fs.com/products/119647.html?attribute=67743&id=3746410) | Calor extremo con carga alta (más de 50 Gbps). Utilice el kit de refrigeración líquida para CPU en el chip para evitar el sobrecalentamiento. |
| Mellanox ConnectX-6 | 10/25 Gbps | [MCX631102AN-ADAT](https://www.fs.com/products/212177.html?now_cid=4014) | No hay problemas conocidos. |
| Mellanox ConnectX-6 | 100 Gbps | [MCX623106AN-CDAT 100G](https://www.fs.com/products/119646.html?now_cid=4014) | No hay problemas conocidos. |
| Mellanox ConnectX-7 | 200 Gbps | [MCX755106AS-HEAT 200G](https://www.fs.com/products/242589.html?now_cid=4014) | No hay problemas conocidos. |
(*) Intel suele restringir la compatibilidad de los módulos SFP+. Verifique la compatibilidad del módulo antes de comprarlo. Mellanox no presenta este problema.
**SÓLO brindaremos soporte para sistemas que utilicen una NIC mencionada anteriormente**. Algunas otras tarjetas de red *podrían* funcionar, pero no serán compatibles oficialmente con LibreQoS. Si desea *probar* la compatibilidad de otra tarjeta, tenga en cuenta estos requisitos fundamentales de la tarjeta de red:
* La NIC debe tener varias colas de transmisión TX/RX, mayores o iguales que la cantidad de núcleos de CPU. [Aquí se explica cómo comprobarlo desde la línea de comandos](https://serverfault.com/questions/772380/how-to-tell-if-nic-has-multiqueue-enabled).
* NIC debe tener [compatibilidad controlador XDP](https://github.com/xdp-project/xdp-project/blob/master/areas/drivers/README.org) para alto rendimiento (10 Gbps+).
Si usted descubre que una tarjeta que no está listada en la tabla de arriba es compatible, por favor, háganoslo saber enviando un correo electrónico a support [at] libreqos.io.
docs-es/TechnicalDocs/complex-install.md
-56
View File
@@ -1,56 +0,0 @@
# Instalación compleja (no recomendada)
## Clonar el repositorio
La ubicación de instalación recomendada es `/opt/libreqos`
Vaya a la ubicación de instalación y clone el repositorio.:
```shell
cd /opt/
git clone https://github.com/LibreQoE/LibreQoS.git libreqos
sudo chown -R YOUR_USER /opt/libreqos
```
Al especificar `libreqos` al final, git se asegurará que el nombre de la carpeta esté en minúsculas.
## Instalar dependencias desde apt y pip
Necesitas tener algunos paquetes de `apt` instalados:
```shell
sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev
```
Luego necesitas instalar algunas dependencias de Python:
```shell
cd /opt/libreqos
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
```
## Instalar el sistema de desarrollo Rust
Ve a [RustUp](https://rustup.rs) y sigue las instrucciones. Básicamente, ejecuta lo siguiente:
```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
Cuando Rust termine de instalarse, te pedirá que ejecutes un comando para instalar las herramientas de compilación de Rust en tu ruta. Debes ejecutar este comando o cerrar sesión y volver a iniciarla.
Una vez hecho esto, por favor, ejecute:
```shell
cd /opt/libreqos/src/
./build_rust.sh
```
La primera vez, esto llevará un tiempo, pero pondrá todo en el lugar correcto.
Ahora, para construir cajas de rust, corre:
```shell
cd rust
cargo build --all
```
docs-es/TechnicalDocs/extras.md
-12
View File
@@ -1,12 +0,0 @@
# Extras
## Flamegraph
```shell
git clone https://github.com/brendangregg/FlameGraph.git
cd FlameGraph
sudo perf record -F 99 -a -g -- sleep 60
perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl --title LibreQoS --width 7200 out.folded > libreqos.svg
```
docs-es/TechnicalDocs/integrations.md
-65
View File
@@ -1,65 +0,0 @@
# Integraciones
## Integración UISP
Primero, configure los parámetros relevantes para UISP (uispAuthToken, UISPbaseURL, etc.) en `/etc/lqos.conf`.
Para probar la integración de UISP, utilice
```shell
python3 integrationUISP.py
```
En la primera ejecución exitosa, se crearán los archivos network.json y ShapedDevices.csv.
Si existe un archivo network.json, no se sobrescribirá.
Puede modificar el archivo network.json para que refleje con mayor precisión los límites de ancho de banda.
ShapedDevices.csv se sobrescribirá cada vez que se ejecute la integración UISP.
Puede ejecutar IntegrationUISP.py automáticamente al arrancar y cada 10 minutos, lo cual es recomendable. Esto se puede habilitar configurando ```enable_uisp = true``` en `/etc/lqos.conf`
## Integración con Powercode
Primero, configure los parámetros relevantes para Powercode (powercode_api_key, powercode_api_url, etc.) en`/etc/lqos.conf`.
Para probar la integración de Powercode, utilice
```shell
python3 integrationPowercode.py
```
En la primera ejecución exitosa, se creará un archivo ShapedDevices.csv.
Puede modificar manualmente el archivo network.json para reflejar los límites de ancho de banda del sitio/AP.
ShapedDevices.csv se sobrescribirá cada vez que se ejecute la integración de Powercode.
Tiene la opción de ejecutar IntegrationPowercode.py automáticamente al arrancar y cada 10 minutos, lo cual es recomendable. Esto se puede habilitar configurando ```enable_powercode = true``` en `/etc/lqos.conf`
## Integración con Sonar
En primer lugar, configure los parámetros relevantes para Sonar (sonar_api_key, sonar_api_url, etc.) en `/etc/lqos.conf`.
Para probar la integración del sonar, utilice
```shell
python3 integrationSonar.py
```
En la primera ejecución exitosa, se creará un archivo ShapedDevices.csv.
Si existe un archivo network.json, no se sobrescribirá.
Puede modificar el archivo network.json para que refleje con mayor precisión los límites de ancho de banda.
ShapedDevices.csv se sobrescribirá cada vez que se ejecute la integración de Sonar.
Puede ejecutar IntegrationSonar.py automáticamente al arrancar y cada 10 minutos, lo cual es recomendable. Esto se puede habilitar configurando ```enable_sonar = true``` en `/etc/lqos.conf`
## Integración con Splynx
Primero, configure los parámetros relevantes para Splynx (splynx_api_key, splynx_api_secret, etc.) en`/etc/lqos.conf`.
La integración de Splynx utiliza autenticación básica. Para usar este tipo de autenticación, asegúrese de habilitar [Acceso no seguro](https://splynx.docs.apiary.io/#introduction/authentication) en la configuración de su clave API de Splynx. Además, la clave API de Splynx debe tener acceso a los permisos necesarios.
Para probar la integración de Splynx, utilice
```shell
python3 integrationSplynx.py
```
En la primera ejecución exitosa, se creará un archivo ShapedDevices.csv.
Puedes crear manualmente el archivo network.json para reflejar con mayor precisión los límites de ancho de banda.
ShapedDevices.csv se sobrescribirá cada vez que se ejecute la integración de Splynx.
Tiene la opción de ejecutar IntegrationSplynx.py automáticamente al arrancar y cada 10 minutos, lo cual es recomendable. Esto se puede habilitar configurando ```enable_spylnx = true``` en `/etc/lqos.conf`.
docs-es/TechnicalDocs/troubleshooting.md
-42
View File
@@ -1,42 +0,0 @@
# Solución de Problemas
## Problemas Comunes
### LibreQoS se está ejecutando, pero no está regulando el tráfico
En ispConfig.py, asegúrese de que las interfaces edge y core correspondan correctamente a la interfaz de borde (edge) y núcleo (core). Pruebe intercambiando las interfaces para ver si el tráfico empieza a regularse correctamente.
Asegúrese de que sus servicios se estén ejecutando correctamente:
- `lqosd.service`
- `lqos_node_manager`
- `lqos_scheduler`
Node manager y scheduler dependen de que el servicio `lqos.service` esté en buen estado y en ejecución.
Por ejemplo, para verificar el estado de lqosd, ejecute:
```sudo systemctl status lqosd```
### lqosd no se está ejecutando o falló al iniciar
En la terminal, ejecute ```sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd```. Esto proporcionará detalles sobre por qué falló al iniciar.
### RTNETLINK answers: Invalid argument
Este error suele aparecer cuando el "MQ qdisc" no puede agregarse correctamente a la interfaz NIC. Esto sugiere que la NIC tiene un número insuficiente de filas RX/TX. Por favor asegúrese de estar utilizando las [NICs recomendadas](../SystemRequirements/Networking.md).
### InfluxDB "Failed to update bandwidth graphs"
El scheduler (scheduler.py) ejecuta la integración con InfluxDB dentro de una instrucción try/except.
Si falla al actualizar InfluxDB, mostrará "Failed to update bandwidth graphs".
Para encontrar la causa exacta del error, ejecute: ```python3 graphInfluxDB.py``` lo cual mostrará errores más específicos.
### Todas las IPs de clientes aparecen bajo Unknown IPs en lugar de Shaped Devices en la Interfaz Gráfica
```
cd /opt/libreqos/src
sudo systemctl stop lqos_scheduler
sudo python3 LibreQoS.py
```
La salida de consola al ejecutar LibreQoS.py directamente proporciona errores más específicos relacionados con ShapedDevices.csv y network.json. Una vez que haya identificado y corregido el error en ShapedDevices.csv y/o network.json, ejecute:
```sudo systemctl start lqos_scheduler```
docs-es/Updates/update.md
-24
View File
@@ -1,24 +0,0 @@
# Actualizar 1.4 a la última versión
```{advertencia}
Si utiliza el puente XDP, el tráfico dejará de pasar a través del puente durante la actualización (el puente XDP solo funciona mientras lqosd se esta ejecutando).
```
## Si instalaste con Git
1. Cambie a su directorio LibreQoS (ejemplo: cd /opt/LibreQoS)
2. Actualize desde Git: `git pull`
3. Recompile: `./build-rust.sh`
4. `sudo rust/remove_pinned_maps.sh`
Ejecute los siguientes comandos para reiniciar los servicios de LibreQoS:
```shell
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
sudo systemctl restart lqos_scheduler
```
## Si instalaste mediante el repositorio APT
En este caso, lo único que necesita hacer es ejecutar `sudo apt update && sudo apt upgrade` y LibreQoS debería instalar el nuevo paquete.
docs/Quickstart/configuration.md
-99
View File
@@ -1,99 +0,0 @@
# Configure LibreQoS
## Configure lqos.conf
Copy the lqosd daemon configuration file to `/etc`:
```shell
cd /opt/libreqos/src
sudo cp lqos.example /etc/lqos.conf
```
Now edit the file to match your setup with
```shell
sudo nano /etc/lqos.conf
```
Change `eth0` and `eth1` to match your network interfaces. The interface facing the Internet should be specified in `to_internet`. The interfacing facing your ISP network should be in `to_network`.
Then, if using Bifrost/XDP set `use_xdp_bridge = true` under that same `[bridge]` section.
For example:
```toml
[bridge]
use_xdp_bridge = true
to_internet = "eth0"
to_network = "eth1"
```
## Configure Your Network Settings
- Set `uplink_bandwidth_mbps` and `downlink_bandwidth_mbps` to match the bandwidth in Mbps of your network's upstream / WAN internet connection. The same can be done for `generated_pn_download_mbps` and `generated_pn_upload_mbps`.
- Set ```dry_run = false``` to allow the program to actually run the commands.
## Network.json
Network.json allows ISP operators to define a Hierarchical Network Topology, or Flat Network Topology.
For networks with no Parent Nodes (no strictly defined Access Points or Sites) edit the network.json to use a Flat Network Topology with
```nano network.json```
setting the following file content:
```json
{}
```
If you plan to use the built-in UISP or Splynx integrations, you do not need to create a network.json file quite yet.
If you plan to use the built-in UISP integration, it will create this automatically on its first run (assuming network.json is not already present). You can then modify the network.json to more accurately reflect your topology.
If you will not be using an integration, you can manually define the network.json following the template file - network.example.json
```text
+-----------------------------------------------------------------------+
| Entire Network |
+-----------------------+-----------------------+-----------------------+
| Parent Node A | Parent Node B | Parent Node C |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 | | | | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+
```
## Manual Setup
You can use
```shell
python3 csvToNetworkJSON.py
```
to convert manualNetwork.csv to a network.json file.
manualNetwork.csv can be copied from the template file, manualNetwork.template.csv
Note: The parent node name must match that used for clients in ShapedDevices.csv
## ShapedDevices.csv
If you are using an integration, this file will be automatically generated. If you are not using an integration, you can manually edit the file.
### Manual Editing
- Modify the ShapedDevices.csv file using your preferred spreadsheet editor (LibreOffice Calc, Excel, etc), following the template file - ShapedDevices.example.csv
- Circuit ID is required. Must be a string of some sort (int is fine, gets parsed as string). Must NOT include any number symbols (#). Every circuit needs a unique CircuitID - they cannot be reused. Here, circuit essentially means customer location. If a customer has multiple locations on different parts of your network, use a unique CircuitID for each of those locations.
- At least one IPv4 address or IPv6 address is required for each entry.
- The Access Point or Site name should be set in the Parent Node field. Parent Node can be left blank for flat networks.
- The ShapedDevices.csv file allows you to set minimum guaranteed, and maximum allowed bandwidth per subscriber.
- The minimum allowed plan rates for Circuits are 2Mbit. Bandwidth min and max should both be above that threshold.
- Recommendation: set the min bandwidth to something like 25/10 and max to 1.15X advertised plan rate by using bandwidthOverheadFactor = 1.15
- This way, when an AP hits its ceiling, users have any remaining AP capacity fairly distributed between them.
- Ensure a reasonable minimum bandwidth minimum for every subscriber, allowing them to utilize up to the maximum provided when AP utilization is below 100%.
Note regarding SLAs: For customers with SLA contracts that guarantee them a minimum bandwidth, set their plan rate as the minimum bandwidth. That way when an AP approaches its ceiling, SLA customers will always get that amount.
![image](https://user-images.githubusercontent.com/22501920/200134960-28709d0f-48fe-4129-b4fd-70b204cade2c.png)
Once your configuration is complete. You're ready to run the application and start the [Deamons](./services-and-run.md)
docs/Quickstart/quickstart-libreqos-1.4.md
-37
View File
@@ -1,37 +0,0 @@
# Install LibreQoS 1.4
## Updating from v1.3
### Remove offloadOff.service
```shell
sudo systemctl disable offloadOff.service
sudo rm /usr/local/sbin/offloadOff.sh /etc/systemd/system/offloadOff.service
```
### Remove cron tasks from v1.3
Run ```sudo crontab -e``` and remove any entries pertaining to LibreQoS from v1.3.
## Simple install via .Deb package (Recommended)
Use the deb package from the [latest v1.4 release](https://github.com/LibreQoE/LibreQoS/releases/).
```shell
sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" | sudo tee -a /etc/apt/sources.list.d/libreqos.list
sudo wget -O - -q http://stats.libreqos.io/repo.asc | sudo apt-key add -
apt-get update
apt-get install libreqos
```
You will be asked some questions about your configuration, and the management daemon and webserver will automatically start. Go to http://<your_ip>:9123/ to finish installation.
## Complex Install (Not Reccomended)
```{note}
Use this install if you'd like to constantly deploy from the main branch on Github. For experienced users only!
```
[Complex Installation](../TechnicalDocs/complex-install.md)
You are now ready to [Configure](./configuration.md) LibreQoS!
docs/Quickstart/quickstart-prereq.md
-87
View File
@@ -1,87 +0,0 @@
# Server Setup - Pre-requisites
Disable hyperthreading on the BIOS/UEFI of your host system. Hyperthreaading is also known as Simultaneous Multi Threading (SMT) on AMD systems. Disabling this is very important for optimal performance of the XDP cpumap filtering and, in turn, throughput and latency.
- Boot, pressing the appropriate key to enter the BIOS settings
- For AMD systems, you will have to navigate the settings to find the "SMT Control" setting. Usually it is under something like ```Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control``` Once you find it, switch to "Disabled" or "Off"
- For Intel systems, you will also have to navigate the settings to find the "hyperthrading" toggle option. On HP servers it's under ```System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.```
- Save changes and reboot
## Install Ubuntu Server
We recommend Ubuntu Server because its kernel version tends to track closely with the mainline Linux releases. Our current documentation assumes Ubuntu Server. To run LibreQoS v1.4, Linux kernel 5.11 or greater is required, as 5.11 includes some important XDP patches. Ubuntu Server 22.04 uses kernel 5.13, which meets that requirement.
You can download Ubuntu Server 22.04 from <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.
1. Boot Ubuntu Server from USB.
2. Follow the steps to install Ubuntu Server.
3. If you use a Mellanox network card, the Ubuntu Server installer will ask you whether to install the mellanox/intel NIC drivers. Check the box to confirm. This extra driver is important.
4. On the Networking settings step, it is recommended to assign a static IP address to the management NIC.
5. Ensure SSH server is enabled so you can more easily log into the server later.
6. You can use scp or sftp to access files from your LibreQoS server for easier file editing. Here's how to access via scp or sftp using an [Ubuntu](https://www.addictivetips.com/ubuntu-linux-tips/sftp-server-ubuntu/) or [Windows](https://winscp.net/eng/index.php) machine.
### Choose Bridge Type
There are two options for the bridge to pass data through your two interfaces:
- Bifrost XDP-Accelerated Bridge
- Regular Linux Bridge
The Bifrost Bridge is recommended for Intel NICs with XDP support, such as the X520 and X710.
The regular Linux bridge is recommended for Nvidea/Mellanox NICs such as the ConnectX-5 series (which have superior bridge performance), and VM setups using virtualized NICs.
To use the Bifrost bridge, skip the regular Linux bridge section below, and be sure to enable Bifrost/XDP in lqos.conf a few sections below.
### Adding a regular Linux bridge (if not using Bifrost XDP bridge)
From the Ubuntu VM, create a linux interface bridge - br0 - with the two shaping interfaces.
Find your existing .yaml file in /etc/netplan/ with
```shell
cd /etc/netplan/
ls
```
Then edit the .yaml file there with
```shell
sudo nano XX-cloud-init.yaml
```
With XX corresponding to the name of the existing file.
Editing the .yaml file, we need to define the shaping interfaces (here, ens19 and ens20) and add the bridge with those two interfaces. Assuming your interfaces are ens18, ens19, and ens20, here is what your file might look like:
```yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens18:
addresses:
- 10.0.0.12/24
routes:
- to: default
via: 10.0.0.1
nameservers:
addresses:
- 1.1.1.1
- 8.8.8.8
search: []
ens19:
dhcp4: no
ens20:
dhcp4: no
version: 2
bridges:
br0:
interfaces:
- ens19
- ens20
```
Make sure to replace 10.0.0.12/24 with your LibreQoS VM's address and subnet, and to replace the default gateway 10.0.0.1 with whatever your default gateway is.
Then run
```shell
sudo netplan apply
```
docs/Quickstart/services-and-run.md
-71
View File
@@ -1,71 +0,0 @@
# LibreQoS daemons
lqosd
- Manages actual XDP code. Build with Rust.
lqos_node_manager
- Runs the GUI available at http://a.b.c.d:9123
lqos_scheduler
- lqos_scheduler handles statistics and performs continuous refreshes of LibreQoS' shapers, including pulling from any enabled CRM Integrations (UISP, Splynx).
- On start: Run a full setup of queues
- Every 10 seconds: Graph bandwidth and latency stats
- Every 30 minutes: Update queues, pulling new configuration from CRM integration if enabled
## Run daemons with systemd
You can setup `lqosd`, `lqos_node_manager`, and `lqos_scheduler` as systemd services.
```shell
sudo cp /opt/libreqos/src/bin/lqos_node_manager.service.example /etc/systemd/system/lqos_node_manager.service
sudo cp /opt/libreqos/src/bin/lqosd.service.example /etc/systemd/system/lqosd.service
sudo cp /opt/libreqos/src/bin/lqos_scheduler.service.example /etc/systemd/system/lqos_scheduler.service
```
Finally, run
```shell
sudo systemctl daemon-reload
sudo systemctl enable lqosd lqos_node_manager lqos_scheduler
```
You can now point a web browser at `http://a.b.c.d:9123` (replace `a.b.c.d` with the management IP address of your shaping server) and enjoy a real-time view of your network.
## Debugging lqos_scheduler
In the background, lqos_scheduler runs scheduler.py, which in turn runs LibreQoS.py
One-time runs of these individual components can be very helpful for debugging and to make sure everything is correctly configured.
First, stop lqos_scheduler
```shell
sudo systemctl stop lqos_scheduler
```
For one-time runs of LibreQoS.py, use
```shell
sudo ./LibreQoS.py
```
- To use the debug mode with more verbose output, use:
```shell
sudo ./LibreQoS.py --debug
```
To confirm that lqos_scheduler (scheduler.py) is able to work correctly, run:
```shell
sudo python3 scheduler.py
```
Once you have any errors eliminated, restart lqos_scheduler with
```shell
sudo systemctl start lqos_scheduler
```
docs/Quickstart/share.md
-31
View File
@@ -1,31 +0,0 @@
# Share your before and after
We ask that you please share an anonymized screenshot of your LibreQoS deployment before (monitor only mode) and after (queuing enabled) to the [LibreQoS Chat](https://chat.libreqos.io/join/fvu3cerayyaumo377xwvpev6/). This helps us gauge the impact of our software. It also makes us smile.
1. Enable monitor only mode
2. Klingon mode (Redact customer info)
3. Screenshot
4. Resume regular queuing
5. Screenshot
## Enable monitor only mode
```shell
sudo systemctl stop lqos_scheduler
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
```
## Klingon mode
Please go to the Web UI and click Configuration. Toggle Redact Customer Information (screenshot mode) and then Apply Changes.
## Resume regular queuing
```shell
sudo systemctl start lqos_scheduler
```
## Screenshot
To generate a screenshot - please go to the Web UI and click Configuration. Toggle Redact Customer Information (screenshot mode), Apply Changes, and then return to the dashboard to take a screenshot.
docs/TechnicalDocs/complex-install.md
-56
View File
@@ -1,56 +0,0 @@
# Complex install (Not Recommended)
## Clone the repo
The recommended install location is `/opt/libreqos`
Go to the install location, and clone the repo:
```shell
cd /opt/
git clone https://github.com/LibreQoE/LibreQoS.git libreqos
sudo chown -R YOUR_USER /opt/libreqos
```
By specifying `libreqos` at the end, git will ensure the folder name is lowercase.
## Install Dependencies from apt and pip
You need to have a few packages from `apt` installed:
```shell
sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev
```
Then you need to install some Python dependencies:
```shell
cd /opt/libreqos
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
```
## Install the Rust development system
Go to [RustUp](https://rustup.rs) and follow the instructions. Basically, run the following:
```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
When Rust finishes installing, it will tell you to execute a command to place the Rust build tools into your path. You need to either execute this command or logout and back in again.
Once that's done, please run:
```shell
cd /opt/libreqos/src/
./build_rust.sh
```
This will take a while the first time, but it puts everything in the right place.
Now, to build rust crates, run:
```shell
cd rust
cargo build --all
```
docs/TechnicalDocs/extras.md
-12
View File
@@ -1,12 +0,0 @@
# Extras
## Flamegraph
```shell
git clone https://github.com/brendangregg/FlameGraph.git
cd FlameGraph
sudo perf record -F 99 -a -g -- sleep 60
perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl --title LibreQoS --width 7200 out.folded > libreqos.svg
```
docs/TechnicalDocs/integrations.md
-65
View File
@@ -1,65 +0,0 @@
# Integrations
## UISP Integration
First, set the relevant parameters for UISP (uispAuthToken, UISPbaseURL, etc.) in `/etc/lqos.conf`.
To test the UISP Integration, use
```shell
python3 integrationUISP.py
```
On the first successful run, it will create a network.json and ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the UISP integration is run.
You have the option to run integrationUISP.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```enable_uisp = true``` in `/etc/lqos.conf`
## Powercode Integration
First, set the relevant parameters for Powercode (powercode_api_key, powercode_api_url, etc.) in `/etc/lqos.conf`.
To test the Powercode Integration, use
```shell
python3 integrationPowercode.py
```
On the first successful run, it will create a ShapedDevices.csv file.
You can modify the network.json file manually to reflect Site/AP bandwidth limits.
ShapedDevices.csv will be overwritten every time the Powercode integration is run.
You have the option to run integrationPowercode.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```enable_powercode = true``` in `/etc/lqos.conf`
## Sonar Integration
First, set the relevant parameters for Sonar (sonar_api_key, sonar_api_url, etc.) in `/etc/lqos.conf`.
To test the Sonar Integration, use
```shell
python3 integrationSonar.py
```
On the first successful run, it will create a ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the Sonar integration is run.
You have the option to run integrationSonar.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```enable_sonar = true``` in `/etc/lqos.conf`
## Splynx Integration
First, set the relevant parameters for Splynx (splynx_api_key, splynx_api_secret, etc.) in `/etc/lqos.conf`.
The Splynx Integration uses Basic authentication. For using this type of authentication, please make sure you enable [Unsecure access](https://splynx.docs.apiary.io/#introduction/authentication) in your Splynx API key settings. Also the Splynx API key should be granted access to the necessary permissions.
To test the Splynx Integration, use
```shell
python3 integrationSplynx.py
```
On the first successful run, it will create a ShapedDevices.csv file.
You can manually create your network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the Splynx integration is run.
You have the option to run integrationSplynx.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```enable_spylnx = true``` in `/etc/lqos.conf`.
docs/TechnicalDocs/troubleshooting.md
-42
View File
@@ -1,42 +0,0 @@
# Troubleshooting
## Common Issues
### LibreQoS Is Running, But Traffic Not Shaping
In ispConfig.py, make sure the edge and core interfaces correspond to correctly to the edge and core. Try swapping the interfaces to see if shaping starts to work.
Make sure your services are running properly
- `lqosd.service`
- `lqos_node_manager`
- `lqos_scheduler`
Node manager and scheduler are dependent on the `lqos.service` being in a healthy, running state.
For example to check the status of lqosd, run:
```sudo systemctl status lqosd```
### lqosd not running or failed to start
At the command-line, type ```sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd``` which will provide specifics regarding why it failed to start.
### RTNETLINK answers: Invalid argument
This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](../SystemRequirements/Compute.md#network-interface-requirements).
### InfluxDB "Failed to update bandwidth graphs"
The scheduler (scheduler.py) runs the InfluxDB integration within a try/except statement. If it fails to update InfluxDB, it will report "Failed to update bandwidth graphs".
To find the exact cause of the failure, please run ```python3 graphInfluxDB.py``` which will provde more specific errors.
### All customer IPs are listed under Unknown IPs, rather than Shaped Devices in GUI
```
cd /opt/libreqos/src
sudo systemctl stop lqos_scheduler
sudo python3 LibreQoS.py
```
The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json
Once you have identified the error and fixed ShapedDevices.csv and/or Network.json, please then run
```sudo systemctl start lqos_scheduler```
docs/Updates/update.md
-24
View File
@@ -1,24 +0,0 @@
# Updating 1.4 To Latest Version
```{warning}
If you use the XDP bridge, traffic will stop passing through the bridge during the update (XDP bridge is only operating while lqosd runs).
```
## If you installed with Git
1. Change to your `LibreQoS` directory (e.g. `cd /opt/LibreQoS`)
2. Update from Git: `git pull`
3. Recompile: `./build-rust.sh`
4. `sudo rust/remove_pinned_maps.sh`
Run the following commands to reload the LibreQoS services.
```shell
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
sudo systemctl restart lqos_scheduler
```
## If you installed through the APT repository
All you should have to do in this case is run `sudo apt update && sudo apt upgrade` and LibreQoS should install the new package.
docs/v1.4/ChangeNotes/v1.4.md
-120
View File
@@ -1,120 +0,0 @@
# LibreQoS v1.3.1 to v1.4 Change Summary
Version 1.4 is a huge milestone. A whole new back-end, new GUI, 30%+ performance improvements, support for single-interface mode.
## Some Statistics
- **564** Commits since 1.3.1
- **28,399** Lines of Code
- **10,142** lines of Rust
- **5,448** lines of HTML & JavaScript
- **3,126** lines of Python
- **2,023** lines of C
## Peak Performance (So Far)
- Tested single-stream performance of just under 10 gbit/s on a 16-core Xeon Gold (single interface architecture, using 8 cores for each direction). The flow was shaped with Cake, and retained good (<10 ms RTT latency) performance.
- Tested 25 gbit/s total throughput on the same system. CPU was not saturated---we didn't have a bigger network to test!
- Running live at ISPs with 11 gbit/s of real customer performance and plenty of room to grow.
## New Architecture
- Rust-based back-end provides:
- `lqosd` - a daemon that manages:
- Loading/setup/unloading eBPF programs.
- Gathers statistics directly from eBPF.
- Provides a local "bus" for transporting data between components.
- Sets "tunables", replacing the need for a separate offloading service.
- `lqtop` - a console-based utility for viewing current activity.
- `lqos_node_manager` - a web-based GUI that:
- Monitors current activity.
- Monitors system status.
- Provides "best/worst" summaries of RTT.
- Provides visibility into the working of queues.
- Categorizes traffic to match your network hierarchy, letting you quickly find the bottlenecks.
- Let's you browse and search your shaped devices.
- Lists "unknown IP addresses" that are passing through the shaper but do not have a rule associated.
- Allows you to view and edit the LibreQoS configuration.
- `lqos_python` - provides Python access to the bus system.
- `lqos_setup` - builds enough configuration files to get you started.
- `lqos_users` - authentication for the GUIs.
- High-performance Python script:
- Batches TC commands for fast execution.
- Batches bus-transactions to associate IP subnets with users for fast execution.
- Improved scheduler for InfluxDB graphing.
## High Performance Bridge (Bifrost)
- Optionally replace the Linux bridge system with an XDP-based bridge accelerator.
- Throughput is 30% higher in this mode.
## Packet and Flow Analysis (Heimdall)
- Viewing a circuit in the web UI displays a summary of IP traffic flows for that circuit.
- A "capture" button will capture packet headers, and allow nanosecond-level analysis of traffic data.
- You can download the packet captures in `libpcap` format, for analysis in Wireshark and similar tools.
- Configure the capture delay in `/etc/lqos.conf`
## Single-interface Mode
- Operate with a single network interface and VLANs for "in" and "out".
## Graphs
- Graph current throughput, shaped and unshaped.
- Graph CPU and RAM performance.
- Graph individual Cake shaper tins, backlog, delays.
- TCP "round trip time" histogram showing overall network latency performance.
- Per-network node traffic graph.
- Per-network node RTT latency histogram, to let you zero-in on troublespots.
## Miscellaneous
- `build_rust.sh` builds the entire package from a Git update, with minimal (<1 second) downtime.
- `build_dpkg.sh` assembles the entire system into an Ubuntu/Debian `.deb` installer.
- Sample `.service` files for `systemd` integration.
- Real-time adjustment to tunables.
- Redact text into Klingon to allow screenshots without sharing customer data.
- Preliminary support for reading IP data inside MPLS packets, as long as they are ordered "VLAN->MPLS->VPLS" and not the other way around.
- Automatically trim network trees that exceed 9 levels deep.
- Very accurate timing functions for better statistics.
- Greatly improved documentation.
- Improved rejection of TCP round-trip-time outliers (from long-polled connections).
- Improved Spylnx and UISP integrations.
## Better Distribution
> This is in alpha testing. It has worked on some test setups, but needs production testing.
Installation via `apt-get` and LibreQoS's own repo. Add the `libreqos` repo, and you can use `apt-get` to install/update the traffic shaper. This doesn't get you the development toolchain.
```sh
sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" > /etc/apt/sources.list.d/libreqos.list
wget -O - -q http://stats.libreqos.io/repo.asc | apt-key add -
apt-get update
apt-get install libreqos
```
You will be asked some questions about your configuration, and the management daemon and webserver will automatically start. Go to `http://<your_ip>:9123/` to finish installation.
## Gallery
### Node Manager - Dashboard
![image](https://user-images.githubusercontent.com/14896751/227727398-ff9e0321-cae3-4daf-a9f7-59fafaf30061.png)
*The node manager displays current activity on your network*
### Node Manager - Circuit View
![image](https://user-images.githubusercontent.com/14896751/227727646-c6eaaece-f2f0-4667-9b66-5cb08a7f4eb8.png)
*Find out exactly what's going on inside each circuit, monitoring all of the queue stats - you can even view the details of each category tin*
### Node Manager - Flow Analysis
![image](https://user-images.githubusercontent.com/14896751/227727691-e82bc021-4da4-436a-93af-0f7cd13824be.png)
*Analyze what's going on for a specific client, viewing real-time traffic flow data. No need to run `torch` or equivalent on their router. Ideal for finding connectivity problems.*
### Node Manager - Packet Capture
![image](https://user-images.githubusercontent.com/14896751/227727755-e8c9bce6-58b2-4c50-a466-2e68b29a4b18.png)
*Capture traffic and analyze per-packet in an intuitive, zoomable traffic analysis system. You can view down to nanosecond granularity to find performance problems, and see what's really going on with your traffic. Click "Download PCAP Dump" to analyze the same data in Wireshark.*
docs/v1.4/Quickstart/configuration.md
-103
View File
@@ -1,103 +0,0 @@
# Configure LibreQoS
## Configure lqos.conf
Copy the lqosd daemon configuration file to `/etc`:
```shell
cd /opt/libreqos/src
sudo cp lqos.example /etc/lqos.conf
```
Now edit the file to match your setup with
```shell
sudo nano /etc/lqos.conf
```
Change `enp1s0f1` and `enp1s0f2` to match your network interfaces. It doesn't matter which one is which. Notice, it's paring the interfaces, so when you first enter enps0f<ins>**1**</ins> in the first line, the `redirect_to` parameter is enp1s0f<ins>**2**</ins> (replacing with your actual interface names).
- First Line: `name = "enp1s0f1", redirect_to = "enp1s0f2"`
- Second Line: `name = "enp1s0f2", redirect_to = "enp1s0f1"`
Then, if using Bifrost/XDP set `use_xdp_bridge = true` under that same `[bridge]` section.
## Configure ispConfig.py
Copy ispConfig.example.py to ispConfig.py and edit as needed
```shell
cd /opt/libreqos/src/
cp ispConfig.example.py ispConfig.py
nano ispConfig.py
```
- Set upstreamBandwidthCapacityDownloadMbps and upstreamBandwidthCapacityUploadMbps to match the bandwidth in Mbps of your network's upstream / WAN internet connection. The same can be done for generatedPNDownloadMbps and generatedPNUploadMbps.
- Set interfaceA to the interface facing your core router (or bridged internal network if your network is bridged)
- Set interfaceB to the interface facing your edge router
- Set ```enableActualShellCommands = True``` to allow the program to actually run the commands.
## Network.json
Network.json allows ISP operators to define a Hierarchical Network Topology, or Flat Network Topology.
For networks with no Parent Nodes (no strictly defined Access Points or Sites) edit the network.json to use a Flat Network Topology with
```nano network.json```
setting the following file content:
```json
{}
```
If you plan to use the built-in UISP or Splynx integrations, you do not need to create a network.json file quite yet.
If you plan to use the built-in UISP integration, it will create this automatically on its first run (assuming network.json is not already present). You can then modify the network.json to more accurately reflect your topology.
If you will not be using an integration, you can manually define the network.json following the template file - network.example.json
```text
+-----------------------------------------------------------------------+
| Entire Network |
+-----------------------+-----------------------+-----------------------+
| Parent Node A | Parent Node B | Parent Node C |
+-----------------------+-------+-------+-------+-----------------------+
| Parent Node D | Sub 3 | Sub 4 | Sub 5 | Sub 6 | Sub 7 | Parent Node F |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Sub 1 | Sub 2 | | | | Sub 8 | Sub 9 |
+-------+-------+-------+-----------------------+-------+-------+-------+
```
## Manual Setup
You can use
```shell
python3 csvToNetworkJSON.py
```
to convert manualNetwork.csv to a network.json file.
manualNetwork.csv can be copied from the template file, manualNetwork.template.csv
Note: The parent node name must match that used for clients in ShapedDevices.csv
## ShapedDevices.csv
If you are using an integration, this file will be automatically generated. If you are not using an integration, you can manually edit the file.
### Manual Editing
- Modify the ShapedDevices.csv file using your preferred spreadsheet editor (LibreOffice Calc, Excel, etc), following the template file - ShapedDevices.example.csv
- Circuit ID is required. Must be a string of some sort (int is fine, gets parsed as string). Must NOT include any number symbols (#). Every circuit needs a unique CircuitID - they cannot be reused. Here, circuit essentially means customer location. If a customer has multiple locations on different parts of your network, use a unique CircuitID for each of those locations.
- At least one IPv4 address or IPv6 address is required for each entry.
- The Access Point or Site name should be set in the Parent Node field. Parent Node can be left blank for flat networks.
- The ShapedDevices.csv file allows you to set minimum guaranteed, and maximum allowed bandwidth per subscriber.
- The minimum allowed plan rates for Circuits are 2Mbit. Bandwidth min and max should both be above that threshold.
- Recommendation: set the min bandwidth to something like 25/10 and max to 1.15X advertised plan rate by using bandwidthOverheadFactor = 1.15
- This way, when an AP hits its ceiling, users have any remaining AP capacity fairly distributed between them.
- Ensure a reasonable minimum bandwidth minimum for every subscriber, allowing them to utilize up to the maximum provided when AP utilization is below 100%.
Note regarding SLAs: For customers with SLA contracts that guarantee them a minimum bandwidth, set their plan rate as the minimum bandwidth. That way when an AP approaches its ceiling, SLA customers will always get that amount.
![image](https://user-images.githubusercontent.com/22501920/200134960-28709d0f-48fe-4129-b4fd-70b204cade2c.png)
Once your configuration is complete. You're ready to run the application and start the [Deamons](./services-and-run.md)
docs/v1.4/Quickstart/networkdesignassumptions.md
-42
View File
@@ -1,42 +0,0 @@
# Network Design Assumptions
## Officially supported configuration
- LibreQoS placed inline in network, usually between an edge router (NAT, firewall) and core router (distribution to sites across network).
- If you use NAT/CG-NAT, place LibreQoS inline south of where NAT is applied, as LibreQoS needs to shape internal addresses (100.64.0.0/12) not public post-NAT IPs.
- Edge and Core routers should have 1500 MTU on links between them
- If you use MPLS, you would terminate MPLS traffic at the core router. LibreQoS cannot decapsulate MPLS on its own.
- OSPF primary link (low cost) through the server running LibreQoS
- OSPF backup link (high cost, maybe 200 for example)
![Offical Configuration](https://raw.githubusercontent.com/LibreQoE/LibreQoS/main/docs/design.png)
### Network Interface Card
```{note}
You must have one of these:
- single NIC with two interfaces,
- two NICs with single interface,
- 2x VLANs interface (using one or two NICs).
```
LibreQoS requires NICs to have 2 or more RX/TX queues and XDP support. While many cards theoretically meet these requirements, less commonly used cards tend to have unreported driver bugs which impede XDP functionality and make them unusable for our purposes. At this time we recommend the Intel x520, Intel x710, and Nvidia (ConnectX-5 or newer) NICs. We cannot guarantee compatibility with other cards.
## Alternate configuration (Not officially supported)
This alternate configuration uses Spanning Tree Protocol (STP) to modify the data path in the event the LibreQoS device is offline for maintenance or another problem.
```{note}
Most of the same considerations apply to the alternate configuration as they do to the officially supported configuation
```
- LibreQoS placed inline in network, usually between an edge router (NAT, firewall) and core router (distribution to sites across network).
- If you use NAT/CG-NAT, place LibreQoS inline south of where NAT is applied, as LibreQoS needs to shape internal addresses (100.64.0.0/12) not public post-NAT IPs.
- Edge router and Core switch should have 1500 MTU on links between them
- If you use MPLS, you would terminate MPLS traffic somewhere south of the core/distribution switch. LibreQoS cannot decapsulate MPLS on its own.
- Spanning Tree primary link (low cost) through the server running LibreQoS
- Spanning Tree backup link (high cost, maybe 80 for example)
Keep in mind that if you use different bandwidth links, for example, 10 Gbps through LibreQoS, and 1 Gbps between core switch and edge router, you may need to be more intentional with your STP costs.
![Alternate Configuration](../stp-diagram.png)
docs/v1.4/Quickstart/quickstart-libreqos-1.4.md
-37
View File
@@ -1,37 +0,0 @@
# Install LibreQoS 1.4
## Updating from v1.3
### Remove offloadOff.service
```shell
sudo systemctl disable offloadOff.service
sudo rm /usr/local/sbin/offloadOff.sh /etc/systemd/system/offloadOff.service
```
### Remove cron tasks from v1.3
Run ```sudo crontab -e``` and remove any entries pertaining to LibreQoS from v1.3.
## Simple install via .Deb package (Recommended)
Use the deb package from the [latest v1.4 release](https://github.com/LibreQoE/LibreQoS/releases/).
```shell
sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" | sudo tee -a /etc/apt/sources.list.d/libreqos.list
sudo wget -O - -q http://stats.libreqos.io/repo.asc | sudo apt-key add -
apt-get update
apt-get install libreqos
```
You will be asked some questions about your configuration, and the management daemon and webserver will automatically start. Go to http://<your_ip>:9123/ to finish installation.
## Complex Install (Not Reccomended)
```{note}
Use this install if you'd like to constantly deploy from the main branch on Github. For experienced users only!
```
[Complex Installation](../TechnicalDocs/complex-install.md)
You are now ready to [Configure](./configuration.md) LibreQoS!
docs/v1.4/Quickstart/quickstart-prereq.md
-87
View File
@@ -1,87 +0,0 @@
# Server Setup - Pre-requisites
Disable hyperthreading on the BIOS/UEFI of your host system. Hyperthreaading is also known as Simultaneous Multi Threading (SMT) on AMD systems. Disabling this is very important for optimal performance of the XDP cpumap filtering and, in turn, throughput and latency.
- Boot, pressing the appropriate key to enter the BIOS settings
- For AMD systems, you will have to navigate the settings to find the "SMT Control" setting. Usually it is under something like ```Advanced -> AMD CBS -> CPU Common Options -> Thread Enablement -> SMT Control``` Once you find it, switch to "Disabled" or "Off"
- For Intel systems, you will also have to navigate the settings to find the "hyperthrading" toggle option. On HP servers it's under ```System Configuration > BIOS/Platform Configuration (RBSU) > Processor Options > Intel (R) Hyperthreading Options.```
- Save changes and reboot
## Install Ubuntu Server
We recommend Ubuntu Server because its kernel version tends to track closely with the mainline Linux releases. Our current documentation assumes Ubuntu Server. To run LibreQoS v1.4, Linux kernel 5.11 or greater is required, as 5.11 includes some important XDP patches. Ubuntu Server 22.04 uses kernel 5.13, which meets that requirement.
You can download Ubuntu Server 22.04 from <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.
1. Boot Ubuntu Server from USB.
2. Follow the steps to install Ubuntu Server.
3. If you use a Mellanox network card, the Ubuntu Server installer will ask you whether to install the mellanox/intel NIC drivers. Check the box to confirm. This extra driver is important.
4. On the Networking settings step, it is recommended to assign a static IP address to the management NIC.
5. Ensure SSH server is enabled so you can more easily log into the server later.
6. You can use scp or sftp to access files from your LibreQoS server for easier file editing. Here's how to access via scp or sftp using an [Ubuntu](https://www.addictivetips.com/ubuntu-linux-tips/sftp-server-ubuntu/) or [Windows](https://winscp.net/eng/index.php) machine.
### Choose Bridge Type
There are two options for the bridge to pass data through your two interfaces:
- Bifrost XDP-Accelerated Bridge
- Regular Linux Bridge
The Bifrost Bridge is recommended for Intel NICs with XDP support, such as the X520 and X710.
The regular Linux bridge is recommended for Nvidea/Mellanox NICs such as the ConnectX-5 series (which have superior bridge performance), and VM setups using virtualized NICs.
To use the Bifrost bridge, skip the regular Linux bridge section below, and be sure to enable Bifrost/XDP in lqos.conf a few sections below.
### Adding a regular Linux bridge (if not using Bifrost XDP bridge)
From the Ubuntu VM, create a linux interface bridge - br0 - with the two shaping interfaces.
Find your existing .yaml file in /etc/netplan/ with
```shell
cd /etc/netplan/
ls
```
Then edit the .yaml file there with
```shell
sudo nano XX-cloud-init.yaml
```
With XX corresponding to the name of the existing file.
Editing the .yaml file, we need to define the shaping interfaces (here, ens19 and ens20) and add the bridge with those two interfaces. Assuming your interfaces are ens18, ens19, and ens20, here is what your file might look like:
```yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens18:
addresses:
- 10.0.0.12/24
routes:
- to: default
via: 10.0.0.1
nameservers:
addresses:
- 1.1.1.1
- 8.8.8.8
search: []
ens19:
dhcp4: no
ens20:
dhcp4: no
version: 2
bridges:
br0:
interfaces:
- ens19
- ens20
```
Make sure to replace 10.0.0.12/24 with your LibreQoS VM's address and subnet, and to replace the default gateway 10.0.0.1 with whatever your default gateway is.
Then run
```shell
sudo netplan apply
```
docs/v1.4/Quickstart/services-and-run.md
-71
View File
@@ -1,71 +0,0 @@
# LibreQoS daemons
lqosd
- Manages actual XDP code. Build with Rust.
lqos_node_manager
- Runs the GUI available at http://a.b.c.d:9123
lqos_scheduler
- lqos_scheduler handles statistics and performs continuous refreshes of LibreQoS' shapers, including pulling from any enabled CRM Integrations (UISP, Splynx).
- On start: Run a full setup of queues
- Every 10 seconds: Graph bandwidth and latency stats
- Every 30 minutes: Update queues, pulling new configuration from CRM integration if enabled
## Run daemons with systemd
You can setup `lqosd`, `lqos_node_manager`, and `lqos_scheduler` as systemd services.
```shell
sudo cp /opt/libreqos/src/bin/lqos_node_manager.service.example /etc/systemd/system/lqos_node_manager.service
sudo cp /opt/libreqos/src/bin/lqosd.service.example /etc/systemd/system/lqosd.service
sudo cp /opt/libreqos/src/bin/lqos_scheduler.service.example /etc/systemd/system/lqos_scheduler.service
```
Finally, run
```shell
sudo systemctl daemon-reload
sudo systemctl enable lqosd lqos_node_manager lqos_scheduler
```
You can now point a web browser at `http://a.b.c.d:9123` (replace `a.b.c.d` with the management IP address of your shaping server) and enjoy a real-time view of your network.
## Debugging lqos_scheduler
In the background, lqos_scheduler runs scheduler.py, which in turn runs LibreQoS.py
One-time runs of these individual components can be very helpful for debugging and to make sure everything is correctly configured.
First, stop lqos_scheduler
```shell
sudo systemctl stop lqos_scheduler
```
For one-time runs of LibreQoS.py, use
```shell
sudo ./LibreQoS.py
```
- To use the debug mode with more verbose output, use:
```shell
sudo ./LibreQoS.py --debug
```
To confirm that lqos_scheduler (scheduler.py) is able to work correctly, run:
```shell
sudo python3 scheduler.py
```
Once you have any errors eliminated, restart lqos_scheduler with
```shell
sudo systemctl start lqos_scheduler
```
docs/v1.4/Quickstart/share.md
-31
View File
@@ -1,31 +0,0 @@
# Share your before and after
We ask that you please share an anonymized screenshot of your LibreQoS deployment before (monitor only mode) and after (queuing enabled) to the [LibreQoS Chat](https://chat.libreqos.io/join/fvu3cerayyaumo377xwvpev6/). This helps us gauge the impact of our software. It also makes us smile.
1. Enable monitor only mode
2. Klingon mode (Redact customer info)
3. Screenshot
4. Resume regular queuing
5. Screenshot
## Enable monitor only mode
```shell
sudo systemctl stop lqos_scheduler
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
```
## Klingon mode
Please go to the Web UI and click Configuration. Toggle Redact Customer Information (screenshot mode) and then Apply Changes.
## Resume regular queuing
```shell
sudo systemctl start lqos_scheduler
```
## Screenshot
To generate a screenshot - please go to the Web UI and click Configuration. Toggle Redact Customer Information (screenshot mode), Apply Changes, and then return to the dashboard to take a screenshot.
docs/v1.4/TechnicalDocs/complex-install.md
-56
View File
@@ -1,56 +0,0 @@
# Complex install (Not Recommended)
## Clone the repo
The recommended install location is `/opt/libreqos`
Go to the install location, and clone the repo:
```shell
cd /opt/
git clone https://github.com/LibreQoE/LibreQoS.git libreqos
sudo chown -R YOUR_USER /opt/libreqos
```
By specifying `libreqos` at the end, git will ensure the folder name is lowercase.
## Install Dependencies from apt and pip
You need to have a few packages from `apt` installed:
```shell
sudo apt-get install -y python3-pip clang gcc gcc-multilib llvm libelf-dev git nano graphviz curl screen llvm pkg-config linux-tools-common linux-tools-`uname -r` libbpf-dev libssl-dev
```
Then you need to install some Python dependencies:
```shell
cd /opt/libreqos
python3 -m pip install -r requirements.txt
sudo python3 -m pip install -r requirements.txt
```
## Install the Rust development system
Go to [RustUp](https://rustup.rs) and follow the instructions. Basically, run the following:
```shell
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
When Rust finishes installing, it will tell you to execute a command to place the Rust build tools into your path. You need to either execute this command or logout and back in again.
Once that's done, please run:
```shell
cd /opt/libreqos/src/
./build_rust.sh
```
This will take a while the first time, but it puts everything in the right place.
Now, to build rust crates, run:
```shell
cd rust
cargo build --all
```
docs/v1.4/TechnicalDocs/extras.md
-12
View File
@@ -1,12 +0,0 @@
# Extras
## Flamegraph
```shell
git clone https://github.com/brendangregg/FlameGraph.git
cd FlameGraph
sudo perf record -F 99 -a -g -- sleep 60
perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl --title LibreQoS --width 7200 out.folded > libreqos.svg
```
docs/v1.4/TechnicalDocs/integrations.md
-65
View File
@@ -1,65 +0,0 @@
# Integrations
## UISP Integration
First, set the relevant parameters for UISP (uispAuthToken, UISPbaseURL, etc.) in ispConfig.py.
To test the UISP Integration, use
```shell
python3 integrationUISP.py
```
On the first successful run, it will create a network.json and ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the UISP integration is run.
You have the option to run integrationUISP.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```automaticImportUISP = True``` in ispConfig.py
## Powercode Integration
First, set the relevant parameters for Powercode (powercode_api_key, powercode_api_url, etc.) in ispConfig.py.
To test the Powercode Integration, use
```shell
python3 integrationPowercode.py
```
On the first successful run, it will create a ShapedDevices.csv file.
You can modify the network.json file manually to reflect Site/AP bandwidth limits.
ShapedDevices.csv will be overwritten every time the Powercode integration is run.
You have the option to run integrationPowercode.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```automaticImportPowercode = True``` in ispConfig.py
## Sonar Integration
First, set the relevant parameters for Sonar (sonar_api_key, sonar_api_url, etc.) in ispConfig.py.
To test the Sonar Integration, use
```shell
python3 integrationSonar.py
```
On the first successful run, it will create a ShapedDevices.csv file.
If a network.json file exists, it will not be overwritten.
You can modify the network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the Sonar integration is run.
You have the option to run integrationSonar.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```automaticImportSonar = True``` in ispConfig.py
## Splynx Integration
First, set the relevant parameters for Splynx (splynx_api_key, splynx_api_secret, etc.) in ispConfig.py.
The Splynx Integration uses Basic authentication. For using this type of authentication, please make sure you enable [Unsecure access](https://splynx.docs.apiary.io/#introduction/authentication) in your Splynx API key settings. Also the Splynx API key should be granted access to the necessary permissions.
To test the Splynx Integration, use
```shell
python3 integrationSplynx.py
```
On the first successful run, it will create a ShapedDevices.csv file.
You can manually create your network.json file to more accurately reflect bandwidth limits.
ShapedDevices.csv will be overwritten every time the Splynx integration is run.
You have the option to run integrationSplynx.py automatically on boot and every 10 minutes, which is recommended. This can be enabled by setting ```automaticImportSplynx = True``` in ispConfig.py
docs/v1.4/TechnicalDocs/performance-tuning.md
-28
View File
@@ -1,28 +0,0 @@
# Performance Tuning
## Ubuntu Starts Slowly (~2 minutes)
### List all services which requires network
```shell
systemctl show -p WantedBy network-online.target
```
### For Ubuntu 22.04 this command can help
```shell
systemctl disable cloud-config iscsid cloud-final
```
### Set proper governor for CPU (baremetal/hypervisior host)
```shell
cpupower frequency-set --governor performance
```
### OSPF
It is recommended to set the OSPF timers of both OSPF neighbors (core and edge router) to minimize downtime upon a reboot of the LibreQoS server.
* hello interval
* dead
docs/v1.4/TechnicalDocs/troubleshooting.md
-42
View File
@@ -1,42 +0,0 @@
# Troubleshooting
## Common Issues
### LibreQoS Is Running, But Traffic Not Shaping
In ispConfig.py, make sure the edge and core interfaces correspond to correctly to the edge and core. Try swapping the interfaces to see if shaping starts to work.
Make sure your services are running properly
- `lqosd.service`
- `lqos_node_manager`
- `lqos_scheduler`
Node manager and scheduler are dependent on the `lqos.service` being in a healthy, running state.
For example to check the status of lqosd, run:
```sudo systemctl status lqosd```
### lqosd not running or failed to start
At the command-line, type ```sudo RUST_LOG=info /opt/libreqos/src/bin/lqosd``` which will provide specifics regarding why it failed to start.
### RTNETLINK answers: Invalid argument
This tends to show up when the MQ qdisc cannot be added correctly to the NIC interface. This would suggest the NIC has insufficient RX/TX queues. Please make sure you are using the [recommended NICs](../SystemRequirements/Networking.md).
### InfluxDB "Failed to update bandwidth graphs"
The scheduler (scheduler.py) runs the InfluxDB integration within a try/except statement. If it fails to update InfluxDB, it will report "Failed to update bandwidth graphs".
To find the exact cause of the failure, please run ```python3 graphInfluxDB.py``` which will provde more specific errors.
### All customer IPs are listed under Unknown IPs, rather than Shaped Devices in GUI
```
cd /opt/libreqos/src
sudo systemctl stop lqos_scheduler
sudo python3 LibreQoS.py
```
The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json
Once you have identified the error and fixed ShapedDevices.csv and/or Network.json, please then run
```sudo systemctl start lqos_scheduler```
docs/v1.4/Updates/update.md
-24
View File
@@ -1,24 +0,0 @@
# Updating 1.4 To Latest Version
```{warning}
If you use the XDP bridge, traffic will stop passing through the bridge during the update (XDP bridge is only operating while lqosd runs).
```
## If you installed with Git
1. Change to your `LibreQoS` directory (e.g. `cd /opt/LibreQoS`)
2. Update from Git: `git pull`
3. Recompile: `./build-rust.sh`
4. `sudo rust/remove_pinned_maps.sh`
Run the following commands to reload the LibreQoS services.
```shell
sudo systemctl restart lqosd
sudo systemctl restart lqos_node_manager
sudo systemctl restart lqos_scheduler
```
## If you installed through the APT repository
All you should have to do in this case is run `sudo apt update && sudo apt upgrade` and LibreQoS should install the new package.
docs/v1.4/test.txt
-1
View File
@@ -1 +0,0 @@
old/v1.3/INTEGRATION_HOWTO.md
-149
View File
@@ -1,149 +0,0 @@
# LibreQoS Integrations
If you need to create an integration for your network, we've tried to give you the tools you need. We currently ship integrations for UISP and Spylnx. We'd love to include more.
### Overall Concept
LibreQoS enforces customer bandwidth limits, and applies CAKE-based optimizations at several levels:
* Per-user Cake flows are created. These require the maximum bandwidth permitted per customer.
* Customers can have more than one device that share a pool of bandwidth. Customers are grouped into "circuits"
* *Optional* Access points can have a speed limit/queue, applied to all customers associated with the access point.
* *Optional* Sites can contain access points, and apply a speed limit/queue to all access points (and associated circuits).
* *Optional* Sites can be nested beneath other sites and access point, providing for a queue hierarchy that represents physical limitations of backhaul connections.
Additionally, you might grow to have more than one shaper - and need to express your network topology from the perspective of different parts of your network. (For example, if *Site A* and *Site B* both have Internet connections - you want to generate an efficient topology for both sites. It's helpful if you can derive this from the same overall topology)
LibreQoS's network modeling accomplishes this by modeling your network as a *graph*: a series of interconnected nodes, each featuring a "parent". Any "node" (entry) in the graph can be turned into a "root" node, allowing you to generate the `network.json` and `ShapedDevices.csv` files required to manage your customers from the perspective of that root node.
### Flat Shaping
The simplest form of integration produces a "flat" network. This is the highest performance model in terms of raw throughput, but lacks the ability to provide shaping at the access point or site level: every customer site is parented directly off the root.
> For an integration, it's recommended that you fetch the customer/device data from your management system rather than type them all in Python.
A flat integration is relatively simple. Start by importing the common API:
```python
from integrationCommon import isIpv4Permitted, fixSubnet, NetworkGraph, NetworkNode, NodeType
```
Then create an empty network graph (it will grow to represent your network):
```python
net = NetworkGraph()
```
Once you have your `NetworkGraph` object, you start adding customers and devices. Customers may have any number of devices. You can add a single customer with one device as follows:
```python
# Add the customer
customer = NetworkNode(
id="Unique Customer ID",
displayName="The Doe Family",
type=NodeType.client,
download=100, # Download is in Mbit/second
upload=20, # Upload is in Mbit/second
address="1 My Road, My City, My State")
net.addRawNode(customer) # Insert the customer ID
# Give them a device
device = NetworkNode(
id="Unique Device ID",
displayName="Doe Family CPE",
parentId="Unique Customer ID", # must match the customer's ID
type=NodeType.device,
ipv4=["100.64.1.5/32"], # As many as you need, express networks as the network ID - e.g. 192.168.100.0/24
ipv6=["feed:beef::12/64"], # Same again. May be [] for none.
mac="00:00:5e:00:53:af"
)
net.addRawNode(device)
```
If the customer has multiple devices, you can add as many as you want - with `ParentId` continuing to match the parent customer's `id`.
Once you have entered all of your customers, you can finish the integration:
```python
net.prepareTree() # This is required, and builds parent-child relationships.
net.createNetworkJson() # Create `network.json`
net.createShapedDevices() # Create the `ShapedDevices.csv` file.
```
### Detailed Hierarchies
Creating a full hierarchy (with as many levels as you want) uses a similar strategy to flat networks---we recommend that you start by reading the "flat shaping" section above.
Start by importing the common API:
```python
from integrationCommon import isIpv4Permitted, fixSubnet, NetworkGraph, NetworkNode, NodeType
```
Then create an empty network graph (it will grow to represent your network):
```python
net = NetworkGraph()
```
Now you can start to insert sites and access points. Sites and access points are inserted like customer or device nodes: they have a unique ID, and a `ParentId`. Customers can then use a `ParentId` of the site or access point beneath which they should be located.
For example, let's create `Site_1` and `Site_2` - at the top of the tree:
```python
net.addRawNode(NetworkNode(id="Site_1", displayName="Site_1", parentId="", type=NodeType.site, download=1000, upload=1000))
net.addRawNode(NetworkNode(id="Site_2", displayName="Site_2", parentId="", type=NodeType.site, download=500, upload=500))
```
Let's attach some access points and point-of-presence sites:
```python
net.addRawNode(NetworkNode(id="AP_A", displayName="AP_A", parentId="Site_1", type=NodeType.ap, download=500, upload=500))
net.addRawNode(NetworkNode(id="Site_3", displayName="Site_3", parentId="Site_1", type=NodeType.site, download=500, upload=500))
net.addRawNode(NetworkNode(id="PoP_5", displayName="PoP_5", parentId="Site_3", type=NodeType.site, download=200, upload=200))
net.addRawNode(NetworkNode(id="AP_9", displayName="AP_9", parentId="PoP_5", type=NodeType.ap, download=120, upload=120))
net.addRawNode(NetworkNode(id="PoP_6", displayName="PoP_6", parentId="PoP_5", type=NodeType.site, download=60, upload=60))
net.addRawNode(NetworkNode(id="AP_11", displayName="AP_11", parentId="PoP_6", type=NodeType.ap, download=30, upload=30))
net.addRawNode(NetworkNode(id="PoP_1", displayName="PoP_1", parentId="Site_2", type=NodeType.site, download=200, upload=200))
net.addRawNode(NetworkNode(id="AP_7", displayName="AP_7", parentId="PoP_1", type=NodeType.ap, download=100, upload=100))
net.addRawNode(NetworkNode(id="AP_1", displayName="AP_1", parentId="Site_2", type=NodeType.ap, download=150, upload=150))
```
When you attach a customer, you can specify a tree entry (e.g. `PoP_5`) as a parent:
```python
# Add the customer
customer = NetworkNode(
id="Unique Customer ID",
displayName="The Doe Family",
parentId="PoP_5",
type=NodeType.client,
download=100, # Download is in Mbit/second
upload=20, # Upload is in Mbit/second
address="1 My Road, My City, My State")
net.addRawNode(customer) # Insert the customer ID
# Give them a device
device = NetworkNode(
id="Unique Device ID",
displayName="Doe Family CPE",
parentId="Unique Customer ID", # must match the customer's ID
type=NodeType.device,
ipv4=["100.64.1.5/32"], # As many as you need, express networks as the network ID - e.g. 192.168.100.0/24
ipv6=["feed:beef::12/64"], # Same again. May be [] for none.
mac="00:00:5e:00:53:af"
)
net.addRawNode(device)
```
Once you have entered all of your network topology and customers, you can finish the integration:
```python
net.prepareTree() # This is required, and builds parent-child relationships.
net.createNetworkJson() # Create `network.json`
net.createShapedDevices() # Create the `ShapedDevices.csv` file.
```
You can also add a call to `net.plotNetworkGraph(False)` (use `True` to also include every customer; this can make for a HUGE file) to create a PDF file (currently named `network.pdf.pdf`) displaying your topology. The example shown here looks like this:
![](testdata/sample_layout.png)
old/v1.3/LibreQoS.py
-1255
View File
File diff suppressed because it is too large. Load diff
old/v1.3/README.md
-46
View File
@@ -1,46 +0,0 @@
# v1.3 (IPv4 + IPv6)
![image](https://user-images.githubusercontent.com/22501920/202913336-256b591b-f372-44fe-995c-5e08ec08a925.png)
## Features
### Fast TCP Latency Tracking
[@thebracket](https://github.com/thebracket/) has created [cpumap-pping](https://github.com/thebracket/cpumap-pping) which merges the functionality of the [xdp-cpumap-tc](https://github.com/xdp-project/xdp-cpumap-tc) and [ePPing](https://github.com/xdp-project/bpf-examples/tree/master/pping) projects, while keeping CPU use within ~1% of xdp-cpumap-tc.
### Integrations
- Added Splynx integration
- UISP integration overhaul by [@thebracket](https://github.com/thebracket/)
- [LMS integation](https://github.com/interduo/LMSLibreQoS) for Polish ISPs by [@interduo](https://github.com/interduo)
### Partial Queue Reload
In v1.2 and prior, the the entire queue structure had to be reloaded to make any changes. This led to a few milliseconds of packet loss for some clients each time that reload happened. The scheduled.py was set to reload all queues each morning at 4AM to avoid any potential disruptions that could theoretically cause.
Starting with v1.3 - LibreQoS tracks the state of the queues, and can do incremental changes without a full reload of all queues. Every 30 minutes - scheduler.py runs the CRM import, and runs a partial reload affecting just the queues that have changed. It still runs a full reload at 4AM.
### v1.3 Improvements to help scale
#### HTB major:minor handle
HTB uses a hex handle for classes. It is two 16-bit hex values joined by a colon - major:minor (<u16>:<u16>). In LibreQoS, each CPU core uses a different major handle.
In v1.2 and prior, the minor handle was unique across all CPUs, meaning only 30k subscribers could be added total.
Starting with LibreQoS v1.3 - minor handles are counted independently by CPU core. With this change, the maximum possible subscriber qdiscs/classes goes from a hard limit of 30k to instead be 30k x CPU core count. So for a higher end system with a 64 core processor such as the AMD EPYC™ 7713P, that would mean ~1.9 million possible subscriber classes. Of course CPU use will be the bottleneck well before class handles are in that scenario. But at least we have that arbitrary 30k limit out of the way.
#### "Circuit ID" Unique Identifier
In order to improve queue reload time in v1.3, it was necessary to use a unique identifier for each circuit. We went with Circuit ID. It can be a number or string, it just needs to be unique between circuits, and the same for multiple devices in the same circuit. This allows us to avoid costly lookups when sorting through the queue structure.
If you have your own script creating ShapedDevices.csv - you could use your CRM's unique identifier for customer services / circuits to serve as this Circuit ID. The UISP and Splynx integrations already do this automatically.
## Compatability Notes
The most major changes are the renaming of the fqorCake variable to "sqm",
and the addition of the Circuit identifier field.
Also after upgrading to LibreQos v1.3, a reboot is required to clear out the
old ebpf code.
old/v1.3/ShapedDevices.example.csv
-14
View File
@@ -1,14 +0,0 @@
#LibreQoS - autogenerated file - START
Circuit ID,Circuit Name,Device ID,Device Name,Parent Node,MAC,IPv4,IPv6,Download Min Mbps,Upload Min Mbps,Download Max Mbps,Upload Max Mbps,Comment
1,"968 Circle St., Gurnee, IL 60031",1,Device 1,AP_A,,"100.64.0.1, 100.64.0.14",,25,5,155,20,
2,"31 Marconi Street, Lake In The Hills, IL 60156",2,Device 2,AP_A,,100.64.0.2,,25,5,105,18,
3,"255 NW. Newport Ave., Jamestown, NY 14701",3,Device 3,AP_9,,100.64.0.3,,25,5,105,18,
4,"8493 Campfire Street, Peabody, MA 01960",4,Device 4,AP_9,,100.64.0.4,,25,5,105,18,
2794,"6 Littleton Drive, Ringgold, GA 30736",5,Device 5,AP_11,,100.64.0.5,,25,5,105,18,
2794,"6 Littleton Drive, Ringgold, GA 30736",6,Device 6,AP_11,,100.64.0.6,,25,5,105,18,
5,"93 Oklahoma Ave., Parsippany, NJ 07054",7,Device 7,AP_1,,100.64.0.7,,25,5,155,20,
6,"74 Bishop Ave., Bakersfield, CA 93306",8,Device 8,AP_1,,100.64.0.8,,25,5,105,18,
7,"9598 Peg Shop Drive, Lutherville Timonium, MD 21093",9,Device 9,AP_7,,100.64.0.9,,25,5,105,18,
8,"115 Gartner Rd., Gettysburg, PA 17325",10,Device 10,AP_7,,100.64.0.10,,25,5,105,18,
9,"525 Birchpond St., Romulus, MI 48174",11,Device 11,Site_1,,100.64.0.11,,25,5,105,18,
#LibreQoS - autogenerated file - EOF
1 #LibreQoS - autogenerated file - START
2 Circuit ID Circuit Name Device ID Device Name Parent Node MAC IPv4 IPv6 Download Min Mbps Upload Min Mbps Download Max Mbps Upload Max Mbps Comment
3 1 968 Circle St., Gurnee, IL 60031 1 Device 1 AP_A 100.64.0.1, 100.64.0.14 25 5 155 20
4 2 31 Marconi Street, Lake In The Hills, IL 60156 2 Device 2 AP_A 100.64.0.2 25 5 105 18
5 3 255 NW. Newport Ave., Jamestown, NY 14701 3 Device 3 AP_9 100.64.0.3 25 5 105 18
6 4 8493 Campfire Street, Peabody, MA 01960 4 Device 4 AP_9 100.64.0.4 25 5 105 18
7 2794 6 Littleton Drive, Ringgold, GA 30736 5 Device 5 AP_11 100.64.0.5 25 5 105 18
8 2794 6 Littleton Drive, Ringgold, GA 30736 6 Device 6 AP_11 100.64.0.6 25 5 105 18
9 5 93 Oklahoma Ave., Parsippany, NJ 07054 7 Device 7 AP_1 100.64.0.7 25 5 155 20
10 6 74 Bishop Ave., Bakersfield, CA 93306 8 Device 8 AP_1 100.64.0.8 25 5 105 18
11 7 9598 Peg Shop Drive, Lutherville Timonium, MD 21093 9 Device 9 AP_7 100.64.0.9 25 5 105 18
12 8 115 Gartner Rd., Gettysburg, PA 17325 10 Device 10 AP_7 100.64.0.10 25 5 105 18
13 9 525 Birchpond St., Romulus, MI 48174 11 Device 11 Site_1 100.64.0.11 25 5 105 18
14 #LibreQoS - autogenerated file - EOF
old/v1.3/cpumap-pping
-1
Submodule old/v1.3/cpumap-pping deleted from 0d4df7f918.
old/v1.3/graphInfluxDB.py
-558
View File
@@ -1,558 +0,0 @@
import subprocess
import json
import subprocess
from datetime import datetime
from pathlib import Path
import statistics
import time
import psutil
from influxdb_client import InfluxDBClient, Point
from influxdb_client.client.write_api import SYNCHRONOUS
from ispConfig import interfaceA, interfaceB, influxDBEnabled, influxDBBucket, influxDBOrg, influxDBtoken, influxDBurl, sqm
def getInterfaceStats(interface):
command = 'tc -j -s qdisc show dev ' + interface
jsonAr = json.loads(subprocess.run(command.split(' '), stdout=subprocess.PIPE).stdout.decode('utf-8'))
jsonDict = {}
for element in filter(lambda e: 'parent' in e, jsonAr):
flowID = ':'.join(map(lambda p: f'0x{p}', element['parent'].split(':')[0:2]))
jsonDict[flowID] = element
del jsonAr
return jsonDict
def chunk_list(l, n):
for i in range(0, len(l), n):
yield l[i:i + n]
def getCircuitBandwidthStats(subscriberCircuits, tinsStats):
interfaces = [interfaceA, interfaceB]
ifaceStats = list(map(getInterfaceStats, interfaces))
for circuit in subscriberCircuits:
if 'stats' not in circuit:
circuit['stats'] = {}
if 'currentQuery' in circuit['stats']:
circuit['stats']['priorQuery'] = circuit['stats']['currentQuery']
circuit['stats']['currentQuery'] = {}
circuit['stats']['sinceLastQuery'] = {}
else:
#circuit['stats']['priorQuery'] = {}
#circuit['stats']['priorQuery']['time'] = datetime.now().isoformat()
circuit['stats']['currentQuery'] = {}
circuit['stats']['sinceLastQuery'] = {}
#for entry in tinsStats:
if 'currentQuery' in tinsStats:
tinsStats['priorQuery'] = tinsStats['currentQuery']
tinsStats['currentQuery'] = {}
tinsStats['sinceLastQuery'] = {}
else:
tinsStats['currentQuery'] = {}
tinsStats['sinceLastQuery'] = {}
tinsStats['currentQuery'] = { 'Bulk': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'BestEffort': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'Video': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'Voice': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
}
tinsStats['sinceLastQuery'] = { 'Bulk': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'BestEffort': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'Video': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
'Voice': {'Download': {'sent_packets': 0.0, 'drops': 0.0}, 'Upload': {'sent_packets': 0.0, 'drops': 0.0}},
}
for circuit in subscriberCircuits:
for (interface, stats, dirSuffix) in zip(interfaces, ifaceStats, ['Download', 'Upload']):
element = stats[circuit['classid']] if circuit['classid'] in stats else False
if element:
bytesSent = float(element['bytes'])
drops = float(element['drops'])
packets = float(element['packets'])
if (element['drops'] > 0) and (element['packets'] > 0):
overloadFactor = float(round(element['drops']/element['packets'],3))
else:
overloadFactor = 0.0
if 'cake diffserv4' in sqm:
tinCounter = 1
for tin in element['tins']:
sent_packets = float(tin['sent_packets'])
ack_drops = float(tin['ack_drops'])
ecn_mark = float(tin['ecn_mark'])
tinDrops = float(tin['drops'])
trueDrops = ecn_mark + tinDrops - ack_drops
if tinCounter == 1:
tinsStats['currentQuery']['Bulk'][dirSuffix]['sent_packets'] += sent_packets
tinsStats['currentQuery']['Bulk'][dirSuffix]['drops'] += trueDrops
elif tinCounter == 2:
tinsStats['currentQuery']['BestEffort'][dirSuffix]['sent_packets'] += sent_packets
tinsStats['currentQuery']['BestEffort'][dirSuffix]['drops'] += trueDrops
elif tinCounter == 3:
tinsStats['currentQuery']['Video'][dirSuffix]['sent_packets'] += sent_packets
tinsStats['currentQuery']['Video'][dirSuffix]['drops'] += trueDrops
elif tinCounter == 4:
tinsStats['currentQuery']['Voice'][dirSuffix]['sent_packets'] += sent_packets
tinsStats['currentQuery']['Voice'][dirSuffix]['drops'] += trueDrops
tinCounter += 1
circuit['stats']['currentQuery']['bytesSent' + dirSuffix] = bytesSent
circuit['stats']['currentQuery']['packetDrops' + dirSuffix] = drops
circuit['stats']['currentQuery']['packetsSent' + dirSuffix] = packets
circuit['stats']['currentQuery']['overloadFactor' + dirSuffix] = overloadFactor
#if 'cake diffserv4' in sqm:
# circuit['stats']['currentQuery']['tins'] = theseTins
circuit['stats']['currentQuery']['time'] = datetime.now().isoformat()
allPacketsDownload = 0.0
allPacketsUpload = 0.0
for circuit in subscriberCircuits:
circuit['stats']['sinceLastQuery']['bitsDownload'] = circuit['stats']['sinceLastQuery']['bitsUpload'] = 0.0
circuit['stats']['sinceLastQuery']['bytesSentDownload'] = circuit['stats']['sinceLastQuery']['bytesSentUpload'] = 0.0
circuit['stats']['sinceLastQuery']['packetDropsDownload'] = circuit['stats']['sinceLastQuery']['packetDropsUpload'] = 0.0
circuit['stats']['sinceLastQuery']['packetsSentDownload'] = circuit['stats']['sinceLastQuery']['packetsSentUpload'] = 0.0
try:
circuit['stats']['sinceLastQuery']['bytesSentDownload'] = circuit['stats']['currentQuery']['bytesSentDownload'] - circuit['stats']['priorQuery']['bytesSentDownload']
circuit['stats']['sinceLastQuery']['bytesSentUpload'] = circuit['stats']['currentQuery']['bytesSentUpload'] - circuit['stats']['priorQuery']['bytesSentUpload']
except:
circuit['stats']['sinceLastQuery']['bytesSentDownload'] = 0.0
circuit['stats']['sinceLastQuery']['bytesSentUpload'] = 0.0
try:
circuit['stats']['sinceLastQuery']['packetDropsDownload'] = circuit['stats']['currentQuery']['packetDropsDownload'] - circuit['stats']['priorQuery']['packetDropsDownload']
circuit['stats']['sinceLastQuery']['packetDropsUpload'] = circuit['stats']['currentQuery']['packetDropsUpload'] - circuit['stats']['priorQuery']['packetDropsUpload']
except:
circuit['stats']['sinceLastQuery']['packetDropsDownload'] = 0.0
circuit['stats']['sinceLastQuery']['packetDropsUpload'] = 0.0
try:
circuit['stats']['sinceLastQuery']['packetsSentDownload'] = circuit['stats']['currentQuery']['packetsSentDownload'] - circuit['stats']['priorQuery']['packetsSentDownload']
circuit['stats']['sinceLastQuery']['packetsSentUpload'] = circuit['stats']['currentQuery']['packetsSentUpload'] - circuit['stats']['priorQuery']['packetsSentUpload']
except:
circuit['stats']['sinceLastQuery']['packetsSentDownload'] = 0.0
circuit['stats']['sinceLastQuery']['packetsSentUpload'] = 0.0
allPacketsDownload += circuit['stats']['sinceLastQuery']['packetsSentDownload']
allPacketsUpload += circuit['stats']['sinceLastQuery']['packetsSentUpload']
if 'priorQuery' in circuit['stats']:
if 'time' in circuit['stats']['priorQuery']:
currentQueryTime = datetime.fromisoformat(circuit['stats']['currentQuery']['time'])
priorQueryTime = datetime.fromisoformat(circuit['stats']['priorQuery']['time'])
deltaSeconds = (currentQueryTime - priorQueryTime).total_seconds()
circuit['stats']['sinceLastQuery']['bitsDownload'] = round(
((circuit['stats']['sinceLastQuery']['bytesSentDownload'] * 8) / deltaSeconds)) if deltaSeconds > 0 else 0
circuit['stats']['sinceLastQuery']['bitsUpload'] = round(
((circuit['stats']['sinceLastQuery']['bytesSentUpload'] * 8) / deltaSeconds)) if deltaSeconds > 0 else 0
else:
circuit['stats']['sinceLastQuery']['bitsDownload'] = (circuit['stats']['sinceLastQuery']['bytesSentDownload'] * 8)
circuit['stats']['sinceLastQuery']['bitsUpload'] = (circuit['stats']['sinceLastQuery']['bytesSentUpload'] * 8)
tinsStats['sinceLastQuery']['Bulk']['Download']['dropPercentage'] = tinsStats['sinceLastQuery']['Bulk']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['BestEffort']['Download']['dropPercentage'] = tinsStats['sinceLastQuery']['BestEffort']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Video']['Download']['dropPercentage'] = tinsStats['sinceLastQuery']['Video']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Voice']['Download']['dropPercentage'] = tinsStats['sinceLastQuery']['Voice']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Bulk']['Download']['percentage'] = tinsStats['sinceLastQuery']['Bulk']['Upload']['percentage'] = 0.0
tinsStats['sinceLastQuery']['BestEffort']['Download']['percentage'] = tinsStats['sinceLastQuery']['BestEffort']['Upload']['percentage'] = 0.0
tinsStats['sinceLastQuery']['Video']['Download']['percentage'] = tinsStats['sinceLastQuery']['Video']['Upload']['percentage'] = 0.0
tinsStats['sinceLastQuery']['Voice']['Download']['percentage'] = tinsStats['sinceLastQuery']['Voice']['Upload']['percentage'] = 0.0
try:
tinsStats['sinceLastQuery']['Bulk']['Download']['sent_packets'] = tinsStats['currentQuery']['Bulk']['Download']['sent_packets'] - tinsStats['priorQuery']['Bulk']['Download']['sent_packets']
tinsStats['sinceLastQuery']['BestEffort']['Download']['sent_packets'] = tinsStats['currentQuery']['BestEffort']['Download']['sent_packets'] - tinsStats['priorQuery']['BestEffort']['Download']['sent_packets']
tinsStats['sinceLastQuery']['Video']['Download']['sent_packets'] = tinsStats['currentQuery']['Video']['Download']['sent_packets'] - tinsStats['priorQuery']['Video']['Download']['sent_packets']
tinsStats['sinceLastQuery']['Voice']['Download']['sent_packets'] = tinsStats['currentQuery']['Voice']['Download']['sent_packets'] - tinsStats['priorQuery']['Voice']['Download']['sent_packets']
tinsStats['sinceLastQuery']['Bulk']['Upload']['sent_packets'] = tinsStats['currentQuery']['Bulk']['Upload']['sent_packets'] - tinsStats['priorQuery']['Bulk']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['BestEffort']['Upload']['sent_packets'] = tinsStats['currentQuery']['BestEffort']['Upload']['sent_packets'] - tinsStats['priorQuery']['BestEffort']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['Video']['Upload']['sent_packets'] = tinsStats['currentQuery']['Video']['Upload']['sent_packets'] - tinsStats['priorQuery']['Video']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['Voice']['Upload']['sent_packets'] = tinsStats['currentQuery']['Voice']['Upload']['sent_packets'] - tinsStats['priorQuery']['Voice']['Upload']['sent_packets']
except:
tinsStats['sinceLastQuery']['Bulk']['Download']['sent_packets'] = tinsStats['sinceLastQuery']['BestEffort']['Download']['sent_packets'] = 0.0
tinsStats['sinceLastQuery']['Video']['Download']['sent_packets'] = tinsStats['sinceLastQuery']['Voice']['Download']['sent_packets'] = 0.0
tinsStats['sinceLastQuery']['Bulk']['Upload']['sent_packets'] = tinsStats['sinceLastQuery']['BestEffort']['Upload']['sent_packets'] = 0.0
tinsStats['sinceLastQuery']['Video']['Upload']['sent_packets'] = tinsStats['sinceLastQuery']['Voice']['Upload']['sent_packets'] = 0.0
try:
tinsStats['sinceLastQuery']['Bulk']['Download']['drops'] = tinsStats['currentQuery']['Bulk']['Download']['drops'] - tinsStats['priorQuery']['Bulk']['Download']['drops']
tinsStats['sinceLastQuery']['BestEffort']['Download']['drops'] = tinsStats['currentQuery']['BestEffort']['Download']['drops'] - tinsStats['priorQuery']['BestEffort']['Download']['drops']
tinsStats['sinceLastQuery']['Video']['Download']['drops'] = tinsStats['currentQuery']['Video']['Download']['drops'] - tinsStats['priorQuery']['Video']['Download']['drops']
tinsStats['sinceLastQuery']['Voice']['Download']['drops'] = tinsStats['currentQuery']['Voice']['Download']['drops'] - tinsStats['priorQuery']['Voice']['Download']['drops']
tinsStats['sinceLastQuery']['Bulk']['Upload']['drops'] = tinsStats['currentQuery']['Bulk']['Upload']['drops'] - tinsStats['priorQuery']['Bulk']['Upload']['drops']
tinsStats['sinceLastQuery']['BestEffort']['Upload']['drops'] = tinsStats['currentQuery']['BestEffort']['Upload']['drops'] - tinsStats['priorQuery']['BestEffort']['Upload']['drops']
tinsStats['sinceLastQuery']['Video']['Upload']['drops'] = tinsStats['currentQuery']['Video']['Upload']['drops'] - tinsStats['priorQuery']['Video']['Upload']['drops']
tinsStats['sinceLastQuery']['Voice']['Upload']['drops'] = tinsStats['currentQuery']['Voice']['Upload']['drops'] - tinsStats['priorQuery']['Voice']['Upload']['drops']
except:
tinsStats['sinceLastQuery']['Bulk']['Download']['drops'] = tinsStats['sinceLastQuery']['BestEffort']['Download']['drops'] = 0.0
tinsStats['sinceLastQuery']['Video']['Download']['drops'] = tinsStats['sinceLastQuery']['Voice']['Download']['drops'] = 0.0
tinsStats['sinceLastQuery']['Bulk']['Upload']['drops'] = tinsStats['sinceLastQuery']['BestEffort']['Upload']['drops'] = 0.0
tinsStats['sinceLastQuery']['Video']['Upload']['drops'] = tinsStats['sinceLastQuery']['Voice']['Upload']['drops'] = 0.0
try:
dlPerc = tinsStats['sinceLastQuery']['Bulk']['Download']['drops'] / tinsStats['sinceLastQuery']['Bulk']['Download']['sent_packets']
ulPerc = tinsStats['sinceLastQuery']['Bulk']['Upload']['drops'] / tinsStats['sinceLastQuery']['Bulk']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['Bulk']['Download']['dropPercentage'] = max(round(dlPerc * 100.0, 3),0.0)
tinsStats['sinceLastQuery']['Bulk']['Upload']['dropPercentage'] = max(round(ulPerc * 100.0, 3),0.0)
dlPerc = tinsStats['sinceLastQuery']['BestEffort']['Download']['drops'] / tinsStats['sinceLastQuery']['BestEffort']['Download']['sent_packets']
ulPerc = tinsStats['sinceLastQuery']['BestEffort']['Upload']['drops'] / tinsStats['sinceLastQuery']['BestEffort']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['BestEffort']['Download']['dropPercentage'] = max(round(dlPerc * 100.0, 3),0.0)
tinsStats['sinceLastQuery']['BestEffort']['Upload']['dropPercentage'] = max(round(ulPerc * 100.0, 3),0.0)
dlPerc = tinsStats['sinceLastQuery']['Video']['Download']['drops'] / tinsStats['sinceLastQuery']['Video']['Download']['sent_packets']
ulPerc = tinsStats['sinceLastQuery']['Video']['Upload']['drops'] / tinsStats['sinceLastQuery']['Video']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['Video']['Download']['dropPercentage'] = max(round(dlPerc * 100.0, 3),0.0)
tinsStats['sinceLastQuery']['Video']['Upload']['dropPercentage'] = max(round(ulPerc * 100.0, 3),0.0)
dlPerc = tinsStats['sinceLastQuery']['Voice']['Download']['drops'] / tinsStats['sinceLastQuery']['Voice']['Download']['sent_packets']
ulPerc = tinsStats['sinceLastQuery']['Voice']['Upload']['drops'] / tinsStats['sinceLastQuery']['Voice']['Upload']['sent_packets']
tinsStats['sinceLastQuery']['Voice']['Download']['dropPercentage'] = max(round(dlPerc * 100.0, 3),0.0)
tinsStats['sinceLastQuery']['Voice']['Upload']['dropPercentage'] = max(round(ulPerc * 100.0, 3),0.0)
except:
tinsStats['sinceLastQuery']['Bulk']['Download']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Bulk']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['BestEffort']['Download']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['BestEffort']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Video']['Download']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Video']['Upload']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Voice']['Download']['dropPercentage'] = 0.0
tinsStats['sinceLastQuery']['Voice']['Upload']['dropPercentage'] = 0.0
try:
tinsStats['sinceLastQuery']['Bulk']['Download']['percentage'] = min(round((tinsStats['sinceLastQuery']['Bulk']['Download']['sent_packets']/allPacketsUpload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['Bulk']['Upload']['percentage'] = min(round((tinsStats['sinceLastQuery']['Bulk']['Upload']['sent_packets']/allPacketsUpload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['BestEffort']['Download']['percentage'] = min(round((tinsStats['sinceLastQuery']['BestEffort']['Download']['sent_packets']/allPacketsDownload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['BestEffort']['Upload']['percentage'] = min(round((tinsStats['sinceLastQuery']['BestEffort']['Upload']['sent_packets']/allPacketsUpload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['Video']['Download']['percentage'] = min(round((tinsStats['sinceLastQuery']['Video']['Download']['sent_packets']/allPacketsDownload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['Video']['Upload']['percentage'] = min(round((tinsStats['sinceLastQuery']['Video']['Upload']['sent_packets']/allPacketsUpload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['Voice']['Download']['percentage'] = min(round((tinsStats['sinceLastQuery']['Voice']['Download']['sent_packets']/allPacketsDownload)*100.0, 3),100.0)
tinsStats['sinceLastQuery']['Voice']['Upload']['percentage'] = min(round((tinsStats['sinceLastQuery']['Voice']['Upload']['sent_packets']/allPacketsUpload)*100.0, 3),100.0)
except:
# To avoid graphing 0.0 for all categories, which would show unusual graph results upon each queue reload, we just set these to None if the above calculations fail.
tinsStats['sinceLastQuery']['Bulk']['Download']['percentage'] = tinsStats['sinceLastQuery']['Bulk']['Upload']['percentage'] = None
tinsStats['sinceLastQuery']['BestEffort']['Download']['percentage'] = tinsStats['sinceLastQuery']['BestEffort']['Upload']['percentage'] = None
tinsStats['sinceLastQuery']['Video']['Download']['percentage'] = tinsStats['sinceLastQuery']['Video']['Upload']['percentage'] = None
tinsStats['sinceLastQuery']['Voice']['Download']['percentage'] = tinsStats['sinceLastQuery']['Voice']['Upload']['percentage'] = None
return subscriberCircuits, tinsStats
def getParentNodeBandwidthStats(parentNodes, subscriberCircuits):
for parentNode in parentNodes:
thisNodeDropsDownload = 0
thisNodeDropsUpload = 0
thisNodeDropsTotal = 0
thisNodeBitsDownload = 0
thisNodeBitsUpload = 0
packetsSentDownloadAggregate = 0.0
packetsSentUploadAggregate = 0.0
packetsSentTotalAggregate = 0.0
circuitsMatched = 0
thisParentNodeStats = {'sinceLastQuery': {}}
for circuit in subscriberCircuits:
if circuit['ParentNode'] == parentNode['parentNodeName']:
thisNodeBitsDownload += circuit['stats']['sinceLastQuery']['bitsDownload']
thisNodeBitsUpload += circuit['stats']['sinceLastQuery']['bitsUpload']
#thisNodeDropsDownload += circuit['packetDropsDownloadSinceLastQuery']
#thisNodeDropsUpload += circuit['packetDropsUploadSinceLastQuery']
thisNodeDropsTotal += (circuit['stats']['sinceLastQuery']['packetDropsDownload'] + circuit['stats']['sinceLastQuery']['packetDropsUpload'])
packetsSentDownloadAggregate += circuit['stats']['sinceLastQuery']['packetsSentDownload']
packetsSentUploadAggregate += circuit['stats']['sinceLastQuery']['packetsSentUpload']
packetsSentTotalAggregate += (circuit['stats']['sinceLastQuery']['packetsSentDownload'] + circuit['stats']['sinceLastQuery']['packetsSentUpload'])
circuitsMatched += 1
if (packetsSentDownloadAggregate > 0) and (packetsSentUploadAggregate > 0):
#overloadFactorDownloadSinceLastQuery = float(round((thisNodeDropsDownload/packetsSentDownloadAggregate)*100.0, 3))
#overloadFactorUploadSinceLastQuery = float(round((thisNodeDropsUpload/packetsSentUploadAggregate)*100.0, 3))
overloadFactorTotalSinceLastQuery = float(round((thisNodeDropsTotal/packetsSentTotalAggregate)*100.0, 1))
else:
#overloadFactorDownloadSinceLastQuery = 0.0
#overloadFactorUploadSinceLastQuery = 0.0
overloadFactorTotalSinceLastQuery = 0.0
thisParentNodeStats['sinceLastQuery']['bitsDownload'] = thisNodeBitsDownload
thisParentNodeStats['sinceLastQuery']['bitsUpload'] = thisNodeBitsUpload
thisParentNodeStats['sinceLastQuery']['packetDropsTotal'] = thisNodeDropsTotal
thisParentNodeStats['sinceLastQuery']['overloadFactorTotal'] = overloadFactorTotalSinceLastQuery
parentNode['stats'] = thisParentNodeStats
return parentNodes
def getParentNodeLatencyStats(parentNodes, subscriberCircuits):
for parentNode in parentNodes:
if 'stats' not in parentNode:
parentNode['stats'] = {}
parentNode['stats']['sinceLastQuery'] = {}
for parentNode in parentNodes:
thisParentNodeStats = {'sinceLastQuery': {}}
circuitsMatchedLatencies = []
for circuit in subscriberCircuits:
if circuit['ParentNode'] == parentNode['parentNodeName']:
if circuit['stats']['sinceLastQuery']['tcpLatency'] != None:
circuitsMatchedLatencies.append(circuit['stats']['sinceLastQuery']['tcpLatency'])
if len(circuitsMatchedLatencies) > 0:
thisParentNodeStats['sinceLastQuery']['tcpLatency'] = statistics.median(circuitsMatchedLatencies)
else:
thisParentNodeStats['sinceLastQuery']['tcpLatency'] = None
parentNode['stats'] = thisParentNodeStats
return parentNodes
def getCircuitLatencyStats(subscriberCircuits):
command = './cpumap-pping/src/xdp_pping'
listOfEntries = json.loads(subprocess.run(command.split(' '), stdout=subprocess.PIPE).stdout.decode('utf-8'))
tcpLatencyForClassID = {}
for entry in listOfEntries:
if 'tc' in entry:
handle = hex(int(entry['tc'].split(':')[0])) + ':' + hex(int(entry['tc'].split(':')[1]))
# To avoid outliers messing up avg for each circuit - cap at ceiling of 200ms
ceiling = 200.0
tcpLatencyForClassID[handle] = min(entry['avg'], ceiling)
for circuit in subscriberCircuits:
if 'stats' not in circuit:
circuit['stats'] = {}
circuit['stats']['sinceLastQuery'] = {}
for circuit in subscriberCircuits:
classID = circuit['classid']
if classID in tcpLatencyForClassID:
circuit['stats']['sinceLastQuery']['tcpLatency'] = tcpLatencyForClassID[classID]
else:
circuit['stats']['sinceLastQuery']['tcpLatency'] = None
return subscriberCircuits
def getParentNodeDict(data, depth, parentNodeNameDict):
if parentNodeNameDict == None:
parentNodeNameDict = {}
for elem in data:
if 'children' in data[elem]:
for child in data[elem]['children']:
parentNodeNameDict[child] = elem
tempDict = getParentNodeDict(data[elem]['children'], depth + 1, parentNodeNameDict)
parentNodeNameDict = dict(parentNodeNameDict, **tempDict)
return parentNodeNameDict
def parentNodeNameDictPull():
# Load network heirarchy
with open('network.json', 'r') as j:
network = json.loads(j.read())
parentNodeNameDict = getParentNodeDict(network, 0, None)
return parentNodeNameDict
def refreshBandwidthGraphs():
startTime = datetime.now()
with open('statsByParentNode.json', 'r') as j:
parentNodes = json.loads(j.read())
with open('statsByCircuit.json', 'r') as j:
subscriberCircuits = json.loads(j.read())
fileLoc = Path("tinsStats.json")
if fileLoc.is_file():
with open(fileLoc, 'r') as j:
tinsStats = json.loads(j.read())
else:
tinsStats = {}
fileLoc = Path("longTermStats.json")
if fileLoc.is_file():
with open(fileLoc, 'r') as j:
longTermStats = json.loads(j.read())
droppedPacketsAllTime = longTermStats['droppedPacketsTotal']
else:
longTermStats = {}
longTermStats['droppedPacketsTotal'] = 0.0
droppedPacketsAllTime = 0.0
parentNodeNameDict = parentNodeNameDictPull()
print("Retrieving circuit statistics")
subscriberCircuits, tinsStats = getCircuitBandwidthStats(subscriberCircuits, tinsStats)
print("Computing parent node statistics")
parentNodes = getParentNodeBandwidthStats(parentNodes, subscriberCircuits)
print("Writing data to InfluxDB")
client = InfluxDBClient(
url=influxDBurl,
token=influxDBtoken,
org=influxDBOrg
)
# Record current timestamp, use for all points added
timestamp = time.time_ns()
write_api = client.write_api(write_options=SYNCHRONOUS)
chunkedsubscriberCircuits = list(chunk_list(subscriberCircuits, 200))
queriesToSendCount = 0
for chunk in chunkedsubscriberCircuits:
queriesToSend = []
for circuit in chunk:
bitsDownload = float(circuit['stats']['sinceLastQuery']['bitsDownload'])
bitsUpload = float(circuit['stats']['sinceLastQuery']['bitsUpload'])
if (bitsDownload > 0) and (bitsUpload > 0):
percentUtilizationDownload = round((bitsDownload / round(circuit['maxDownload'] * 1000000))*100.0, 1)
percentUtilizationUpload = round((bitsUpload / round(circuit['maxUpload'] * 1000000))*100.0, 1)
p = Point('Bandwidth').tag("Circuit", circuit['circuitName']).tag("ParentNode", circuit['ParentNode']).tag("Type", "Circuit").field("Download", bitsDownload).field("Upload", bitsUpload).time(timestamp)
queriesToSend.append(p)
p = Point('Utilization').tag("Circuit", circuit['circuitName']).tag("ParentNode", circuit['ParentNode']).tag("Type", "Circuit").field("Download", percentUtilizationDownload).field("Upload", percentUtilizationUpload).time(timestamp)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
# print("Added " + str(len(queriesToSend)) + " points to InfluxDB.")
queriesToSendCount += len(queriesToSend)
queriesToSend = []
for parentNode in parentNodes:
bitsDownload = float(parentNode['stats']['sinceLastQuery']['bitsDownload'])
bitsUpload = float(parentNode['stats']['sinceLastQuery']['bitsUpload'])
dropsTotal = float(parentNode['stats']['sinceLastQuery']['packetDropsTotal'])
overloadFactor = float(parentNode['stats']['sinceLastQuery']['overloadFactorTotal'])
droppedPacketsAllTime += dropsTotal
if (bitsDownload > 0) and (bitsUpload > 0):
percentUtilizationDownload = round((bitsDownload / round(parentNode['maxDownload'] * 1000000))*100.0, 1)
percentUtilizationUpload = round((bitsUpload / round(parentNode['maxUpload'] * 1000000))*100.0, 1)
p = Point('Bandwidth').tag("Device", parentNode['parentNodeName']).tag("ParentNode", parentNode['parentNodeName']).tag("Type", "Parent Node").field("Download", bitsDownload).field("Upload", bitsUpload).time(timestamp)
queriesToSend.append(p)
p = Point('Utilization').tag("Device", parentNode['parentNodeName']).tag("ParentNode", parentNode['parentNodeName']).tag("Type", "Parent Node").field("Download", percentUtilizationDownload).field("Upload", percentUtilizationUpload).time(timestamp)
queriesToSend.append(p)
p = Point('Overload').tag("Device", parentNode['parentNodeName']).tag("ParentNode", parentNode['parentNodeName']).tag("Type", "Parent Node").field("Overload", overloadFactor).time(timestamp)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
# print("Added " + str(len(queriesToSend)) + " points to InfluxDB.")
queriesToSendCount += len(queriesToSend)
if 'cake diffserv4' in sqm:
queriesToSend = []
listOfTins = ['Bulk', 'BestEffort', 'Video', 'Voice']
for tin in listOfTins:
p = Point('Tin Drop Percentage').tag("Type", "Tin").tag("Tin", tin).field("Download", tinsStats['sinceLastQuery'][tin]['Download']['dropPercentage']).field("Upload", tinsStats['sinceLastQuery'][tin]['Upload']['dropPercentage']).time(timestamp)
queriesToSend.append(p)
# Check to ensure tin percentage has value (!= None) before graphing. During partial or full reload these will have a value of None.
if (tinsStats['sinceLastQuery'][tin]['Download']['percentage'] != None) and (tinsStats['sinceLastQuery'][tin]['Upload']['percentage'] != None):
p = Point('Tins Assigned').tag("Type", "Tin").tag("Tin", tin).field("Download", tinsStats['sinceLastQuery'][tin]['Download']['percentage']).field("Upload", tinsStats['sinceLastQuery'][tin]['Upload']['percentage']).time(timestamp)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
# print("Added " + str(len(queriesToSend)) + " points to InfluxDB.")
queriesToSendCount += len(queriesToSend)
# Graph CPU use
cpuVals = psutil.cpu_percent(percpu=True)
queriesToSend = []
for index, item in enumerate(cpuVals):
p = Point('CPU').field('CPU_' + str(index), item)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
queriesToSendCount += len(queriesToSend)
print("Added " + str(queriesToSendCount) + " points to InfluxDB.")
client.close()
with open('statsByParentNode.json', 'w') as f:
f.write(json.dumps(parentNodes, indent=4))
with open('statsByCircuit.json', 'w') as f:
f.write(json.dumps(subscriberCircuits, indent=4))
longTermStats['droppedPacketsTotal'] = droppedPacketsAllTime
with open('longTermStats.json', 'w') as f:
f.write(json.dumps(longTermStats, indent=4))
with open('tinsStats.json', 'w') as f:
f.write(json.dumps(tinsStats, indent=4))
endTime = datetime.now()
durationSeconds = round((endTime - startTime).total_seconds(), 2)
print("Graphs updated within " + str(durationSeconds) + " seconds.")
def refreshLatencyGraphs():
startTime = datetime.now()
with open('statsByParentNode.json', 'r') as j:
parentNodes = json.loads(j.read())
with open('statsByCircuit.json', 'r') as j:
subscriberCircuits = json.loads(j.read())
parentNodeNameDict = parentNodeNameDictPull()
print("Retrieving circuit statistics")
subscriberCircuits = getCircuitLatencyStats(subscriberCircuits)
print("Computing parent node statistics")
parentNodes = getParentNodeLatencyStats(parentNodes, subscriberCircuits)
print("Writing data to InfluxDB")
client = InfluxDBClient(
url=influxDBurl,
token=influxDBtoken,
org=influxDBOrg
)
# Record current timestamp, use for all points added
timestamp = time.time_ns()
write_api = client.write_api(write_options=SYNCHRONOUS)
chunkedsubscriberCircuits = list(chunk_list(subscriberCircuits, 200))
queriesToSendCount = 0
for chunk in chunkedsubscriberCircuits:
queriesToSend = []
for circuit in chunk:
if circuit['stats']['sinceLastQuery']['tcpLatency'] != None:
tcpLatency = float(circuit['stats']['sinceLastQuery']['tcpLatency'])
p = Point('TCP Latency').tag("Circuit", circuit['circuitName']).tag("ParentNode", circuit['ParentNode']).tag("Type", "Circuit").field("TCP Latency", tcpLatency).time(timestamp)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
queriesToSendCount += len(queriesToSend)
queriesToSend = []
for parentNode in parentNodes:
if parentNode['stats']['sinceLastQuery']['tcpLatency'] != None:
tcpLatency = float(parentNode['stats']['sinceLastQuery']['tcpLatency'])
p = Point('TCP Latency').tag("Device", parentNode['parentNodeName']).tag("ParentNode", parentNode['parentNodeName']).tag("Type", "Parent Node").field("TCP Latency", tcpLatency).time(timestamp)
queriesToSend.append(p)
write_api.write(bucket=influxDBBucket, record=queriesToSend)
queriesToSendCount += len(queriesToSend)
listOfAllLatencies = []
for circuit in subscriberCircuits:
if circuit['stats']['sinceLastQuery']['tcpLatency'] != None:
listOfAllLatencies.append(circuit['stats']['sinceLastQuery']['tcpLatency'])
if len(listOfAllLatencies) > 0:
currentNetworkLatency = statistics.median(listOfAllLatencies)
p = Point('TCP Latency').tag("Type", "Network").field("TCP Latency", currentNetworkLatency).time(timestamp)
write_api.write(bucket=influxDBBucket, record=p)
queriesToSendCount += 1
print("Added " + str(queriesToSendCount) + " points to InfluxDB.")
client.close()
with open('statsByParentNode.json', 'w') as f:
f.write(json.dumps(parentNodes, indent=4))
with open('statsByCircuit.json', 'w') as f:
f.write(json.dumps(subscriberCircuits, indent=4))
endTime = datetime.now()
durationSeconds = round((endTime - startTime).total_seconds(), 2)
print("Graphs updated within " + str(durationSeconds) + " seconds.")
if __name__ == '__main__':
refreshBandwidthGraphs()
refreshLatencyGraphs()
old/v1.3/influxDBdashboardTemplate.json
-2390
View File
File diff suppressed because it is too large. Load diff
old/v1.3/influxDBdashboardTemplate_v2.5.1.json
-1
View File
@@ -1 +0,0 @@
[{"apiVersion":"influxdata.com/v2alpha1","kind":"Dashboard","metadata":{"name":"unbridled-goodall-cd9001"},"spec":{"charts":[{"axes":[{"base":"10","name":"x","scale":"linear"},{"base":"10","label":"bits per second","name":"y","scale":"linear","suffix":"bps"}],"colorizeRows":true,"colors":[{"id":"d3804aa0-9930-45e5-879e-8a79b306d2bc","name":"Nineteen Eighty Four","type":"scale","hex":"#31C0F6"},{"id":"ff778217-93b4-41c1-baf9-5eac71c6969a","name":"Nineteen Eighty Four","type":"scale","hex":"#A500A5"},{"id":"770d44dd-3a18-4132-af57-86f3a5fb4738","name":"Nineteen Eighty Four","type":"scale","hex":"#FF7E27"}],"geom":"monotoneX","height":4,"hoverDimension":"auto","kind":"Xy","legendColorizeRows":true,"legendOpacity":1,"legendOrientationThreshold":100000000,"name":"Download Bandwidth | Nodes","opacity":1,"orientationThreshold":100000000,"position":"stacked","queries":[{"query":"from(bucket: \"libreqos\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"_measurement\"] == \"Bandwidth\")\n |> filter(fn: (r) => r[\"Type\"] == \"Parent Node\")\n |> filter(fn: (r) => r[\"_field\"] == \"Download\")\n |> aggregateWindow(every: v.windowPeriod, fn: last, createEmpty: false)\n |> yield(name: \"last\")"}],"staticLegend":{"colorizeRows":true,"opacity":1,"orientationThreshold":100000000,"widthRatio":1},"width":5,"widthRatio":1,"xCol":"_time","yCol":"_value"},{"axes":[{"label":"RTT (ms)","name":"x"}],"binCount":140,"colors":[{"id":"fd2dc3e9-d00b-44bd-97c0-b39748787656","name":"Nineteen Eighty Four","type":"scale","hex":"#31C0F6"},{"id":"9a32b733-aac8-4ee7-8f9e-41c0600b4270","name":"Nineteen Eighty Four","type":"scale","hex":"#A500A5"},{"id":"3247c366-9139-4bef-aadc-78e5df8ef217","name":"Nineteen Eighty Four","type":"scale","hex":"#FF7E27"}],"fillColumns":["ParentNode"],"height":4,"kind":"Histogram","legendColorizeRows":true,"legendOpacity":1,"legendOrientationThreshold":100000000,"name":"TCP Latency | Circuits","position":"stacked","queries":[{"query":"from(bucket: \"libreqos\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"_measurement\"] == \"TCP Latency\")\n |> filter(fn: (r) => r[\"Type\"] == \"Circuit\")\n |> filter(fn: (r) => r[\"_field\"] == \"TCP Latency\")\n |> aggregateWindow(every: v.windowPeriod, fn: last, createEmpty: false)\n |> yield(name: \"last\")"}],"staticLegend":{},"width":5,"xCol":"_value","yPos":4},{"axes":[{"base":"10","name":"x","scale":"linear"},{"base":"10","name":"y","scale":"linear"}],"colorizeRows":true,"colors":[{"id":"9a3213f8-38be-4075-a164-0af0b13fde1c","name":"Nineteen Eighty Four","type":"scale","hex":"#31C0F6"},{"id":"a236706f-192f-4538-b08f-b1a5944a67e9","name":"Nineteen Eighty Four","type":"scale","hex":"#A500A5"},{"id":"53c9ec3a-c023-4326-b624-ebd7d9890929","name":"Nineteen Eighty Four","type":"scale","hex":"#FF7E27"}],"geom":"monotoneX","height":4,"hoverDimension":"auto","kind":"Xy","legendColorizeRows":true,"legendOpacity":1,"legendOrientationThreshold":100000000,"name":"TCP Latency | Nodes","opacity":1,"orientationThreshold":100000000,"position":"overlaid","queries":[{"query":"from(bucket: \"libreqos\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"_measurement\"] == \"TCP Latency\")\n |> filter(fn: (r) => r[\"Type\"] == \"Parent Node\")\n |> filter(fn: (r) => r[\"_field\"] == \"TCP Latency\")\n |> aggregateWindow(every: v.windowPeriod, fn: last, createEmpty: false)\n |> yield(name: \"last\")"}],"staticLegend":{"colorizeRows":true,"opacity":1,"orientationThreshold":100000000,"widthRatio":1},"width":5,"widthRatio":1,"xCol":"_time","yCol":"_value","yPos":8},{"axes":[{"base":"10","name":"x","scale":"linear"},{"base":"10","label":"bits per second","name":"y","scale":"linear","suffix":"bps"}],"colorizeRows":true,"colors":[{"id":"d3804aa0-9930-45e5-879e-8a79b306d2bc","name":"Nineteen Eighty Four","type":"scale","hex":"#31C0F6"},{"id":"ff778217-93b4-41c1-baf9-5eac71c6969a","name":"Nineteen Eighty Four","type":"scale","hex":"#A500A5"},{"id":"770d44dd-3a18-4132-af57-86f3a5fb4738","name":"Nineteen Eighty Four","type":"scale","hex":"#FF7E27"}],"geom":"monotoneX","height":4,"hoverDimension":"auto","kind":"Xy","legendColorizeRows":true,"legendOpacity":1,"legendOrientationThreshold":100000000,"name":"Download Bandwidth | Circuits","opacity":1,"orientationThreshold":100000000,"position":"overlaid","queries":[{"query":"from(bucket: \"libreqos\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r[\"_measurement\"] == \"Bandwidth\")\n |> filter(fn: (r) => r[\"Type\"] == \"Circuit\")\n |> filter(fn: (r) => r[\"_field\"] == \"Download\")\n |> aggregateWindow(every: v.windowPeriod, fn: last, createEmpty: false)\n |> yield(name: \"last\")"}],"staticLegend":{"colorizeRows":true,"opacity":1,"orientationThreshold":100000000,"widthRatio":1},"width":5,"widthRatio":1,"xCol":"_time","yCol":"_value","yPos":12},{"axes":[{"base":"10","name":"x","Line truncated
old/v1.3/integrationCommon.py
-421
View File
@@ -1,421 +0,0 @@
# Provides common functionality shared between
# integrations.
from typing import List, Any
from ispConfig import allowedSubnets, ignoreSubnets, generatedPNUploadMbps, generatedPNDownloadMbps
import ipaddress
import enum
def isInAllowedSubnets(inputIP):
# Check whether an IP address occurs inside the allowedSubnets list
isAllowed = False
if '/' in inputIP:
inputIP = inputIP.split('/')[0]
for subnet in allowedSubnets:
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
isAllowed = True
return isAllowed
def isInIgnoredSubnets(inputIP):
# Check whether an IP address occurs within the ignoreSubnets list
isIgnored = False
if '/' in inputIP:
inputIP = inputIP.split('/')[0]
for subnet in ignoreSubnets:
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
isIgnored = True
return isIgnored
def isIpv4Permitted(inputIP):
# Checks whether an IP address is in Allowed Subnets.
# If it is, check that it isn't in Ignored Subnets.
# If it is allowed and not ignored, returns true.
# Otherwise, returns false.
return isInIgnoredSubnets(inputIP) == False and isInAllowedSubnets(inputIP)
def fixSubnet(inputIP):
# If an IP address has a CIDR other than /32 (e.g. 192.168.1.1/24),
# but doesn't appear as a network address (e.g. 192.168.1.0/24)
# then it probably isn't actually serving that whole subnet.
# This allows you to specify e.g. 192.168.1.0/24 is "the client
# on port 3" in the device, without falling afoul of UISP's inclusion
# of subnet masks in device IPs.
[rawIp, cidr] = inputIP.split('/')
if cidr != "32":
try:
subnet = ipaddress.ip_network(inputIP)
except:
# Not a network address
return rawIp + "/32"
return inputIP
class NodeType(enum.IntEnum):
# Enumeration to define what type of node
# a NetworkNode is.
root = 1
site = 2
ap = 3
client = 4
clientWithChildren = 5
device = 6
class NetworkNode:
# Defines a node on a LibreQoS network graph.
# Nodes default to being disconnected, and
# will be mapped to the root of the overall
# graph.
id: str
displayName: str
parentIndex: int
parentId: str
type: NodeType
downloadMbps: int
uploadMbps: int
ipv4: List
ipv6: List
address: str
mac: str
def __init__(self, id: str, displayName: str = "", parentId: str = "", type: NodeType = NodeType.site, download: int = generatedPNDownloadMbps, upload: int = generatedPNUploadMbps, ipv4: List = [], ipv6: List = [], address: str = "", mac: str = "") -> None:
self.id = id
self.parentIndex = 0
self.type = type
self.parentId = parentId
if displayName == "":
self.displayName = id
else:
self.displayName = displayName
self.downloadMbps = download
self.uploadMbps = upload
self.ipv4 = ipv4
self.ipv6 = ipv6
self.address = address
self.mac = mac
class NetworkGraph:
# Defines a network as a graph topology
# allowing any integration to build the
# graph via a common API, emitting
# ShapedDevices and network.json files
# via a common interface.
nodes: List
ipv4ToIPv6: Any
excludeSites: List # Copied to allow easy in-test patching
exceptionCPEs: Any
def __init__(self) -> None:
from ispConfig import findIPv6usingMikrotik, excludeSites, exceptionCPEs
self.nodes = [
NetworkNode("FakeRoot", type=NodeType.root,
parentId="", displayName="Shaper Root")
]
self.excludeSites = excludeSites
self.exceptionCPEs = exceptionCPEs
if findIPv6usingMikrotik:
from mikrotikFindIPv6 import pullMikrotikIPv6
self.ipv4ToIPv6 = pullMikrotikIPv6()
else:
self.ipv4ToIPv6 = {}
def addRawNode(self, node: NetworkNode) -> None:
# Adds a NetworkNode to the graph, unchanged.
# If a site is excluded (via excludedSites in ispConfig)
# it won't be added
if not node.displayName in self.excludeSites:
if node.displayName in self.exceptionCPEs.keys():
node.parentId = self.exceptionCPEs[node.displayName]
self.nodes.append(node)
def replaceRootNode(self, node: NetworkNode) -> None:
# Replaces the automatically generated root node
# with a new node. Useful when you have a top-level
# node specified (e.g. "uispSite" in the UISP
# integration)
self.nodes[0] = node
def addNodeAsChild(self, parent: str, node: NetworkNode) -> None:
# Searches the existing graph for a named parent,
# adjusts the new node's parentIndex to match the new
# node. The parented node is then inserted.
#
# Exceptions are NOT applied, since we're explicitly
# specifying the parent - we're assuming you really
# meant it.
if node.displayName in self.excludeSites: return
parentIdx = 0
for (i, node) in enumerate(self.nodes):
if node.id == parent:
parentIdx = i
node.parentIndex = parentIdx
self.nodes.append(node)
def __reparentById(self) -> None:
# Scans the entire node tree, searching for parents
# by name. Entries are re-mapped to match the named
# parents. You can use this to build a tree from a
# blob of raw data.
for child in self.nodes:
if child.parentId != "":
for (i, node) in enumerate(self.nodes):
if node.id == child.parentId:
child.parentIndex = i
def findNodeIndexById(self, id: str) -> int:
# Finds a single node by identity(id)
# Return -1 if not found
for (i, node) in enumerate(self.nodes):
if node.id == id:
return i
return -1
def findNodeIndexByName(self, name: str) -> int:
# Finds a single node by identity(name)
# Return -1 if not found
for (i, node) in enumerate(self.nodes):
if node.displayName == name:
return i
return -1
def findChildIndices(self, parentIndex: int) -> List:
# Returns the indices of all nodes with a
# parentIndex equal to the specified parameter
result = []
for (i, node) in enumerate(self.nodes):
if node.parentIndex == parentIndex:
result.append(i)
return result
def __promoteClientsWithChildren(self) -> None:
# Searches for client sites that have children,
# and changes their node type to clientWithChildren
for (i, node) in enumerate(self.nodes):
if node.type == NodeType.client:
for child in self.findChildIndices(i):
if self.nodes[child].type != NodeType.device:
node.type = NodeType.clientWithChildren
def __clientsWithChildrenToSites(self) -> None:
toAdd = []
for (i, node) in enumerate(self.nodes):
if node.type == NodeType.clientWithChildren:
siteNode = NetworkNode(
id=node.id + "_gen",
displayName="(Generated Site) " + node.displayName,
type=NodeType.site
)
siteNode.parentIndex = node.parentIndex
node.parentId = siteNode.id
if node.type == NodeType.clientWithChildren:
node.type = NodeType.client
for child in self.findChildIndices(i):
if self.nodes[child].type == NodeType.client or self.nodes[child].type == NodeType.clientWithChildren or self.nodes[child].type == NodeType.site:
self.nodes[child].parentId = siteNode.id
toAdd.append(siteNode)
for n in toAdd:
self.addRawNode(n)
self.__reparentById()
def __findUnconnectedNodes(self) -> List:
# Performs a tree-traversal and finds any nodes that
# aren't connected to the root. This is a "sanity check",
# and also an easy way to handle "flat" topologies and
# ensure that the unconnected nodes are re-connected to
# the root.
visited = []
next = [0]
while len(next) > 0:
nextTraversal = next.pop()
visited.append(nextTraversal)
for idx in self.findChildIndices(nextTraversal):
if idx not in visited:
next.append(idx)
result = []
for i, n in enumerate(self.nodes):
if i not in visited:
result.append(i)
return result
def __reconnectUnconnected(self):
# Finds any unconnected nodes and reconnects
# them to the root
for idx in self.__findUnconnectedNodes():
if self.nodes[idx].type == NodeType.site:
self.nodes[idx].parentIndex = 0
for idx in self.__findUnconnectedNodes():
if self.nodes[idx].type == NodeType.clientWithChildren:
self.nodes[idx].parentIndex = 0
for idx in self.__findUnconnectedNodes():
if self.nodes[idx].type == NodeType.client:
self.nodes[idx].parentIndex = 0
def prepareTree(self) -> None:
# Helper function that calls all the cleanup and mapping
# functions in the right order. Unless you are doing
# something special, you can use this instead of
# calling the functions individually
self.__reparentById()
self.__promoteClientsWithChildren()
self.__clientsWithChildrenToSites()
self.__reconnectUnconnected()
def doesNetworkJsonExist(self):
# Returns true if "network.json" exists, false otherwise
import os
return os.path.isfile("network.json")
def __isSite(self, index) -> bool:
return self.nodes[index].type == NodeType.ap or self.nodes[index].type == NodeType.site or self.nodes[index].type == NodeType.clientWithChildren
def createNetworkJson(self):
import json
topLevelNode = {}
self.__visited = [] # Protection against loops - never visit twice
for child in self.findChildIndices(0):
if child > 0 and self.__isSite(child):
topLevelNode[self.nodes[child].displayName] = self.__buildNetworkObject(
child)
del self.__visited
with open('network.json', 'w') as f:
json.dump(topLevelNode, f, indent=4)
def __buildNetworkObject(self, idx):
# Private: used to recurse down the network tree while building
# network.json
self.__visited.append(idx)
node = {
"downloadBandwidthMbps": self.nodes[idx].downloadMbps,
"uploadBandwidthMbps": self.nodes[idx].uploadMbps,
}
children = {}
hasChildren = False
for child in self.findChildIndices(idx):
if child > 0 and self.__isSite(child) and child not in self.__visited:
children[self.nodes[child].displayName] = self.__buildNetworkObject(
child)
hasChildren = True
if hasChildren:
node["children"] = children
return node
def __addIpv6FromMap(self, ipv4, ipv6) -> None:
# Scans each address in ipv4. If its present in the
# IPv4 to Ipv6 map (currently pulled from Mikrotik devices
# if findIPv6usingMikrotik is enabled), then matching
# IPv6 networks are appended to the ipv6 list.
# This is explicitly non-destructive of the existing IPv6
# list, in case you already have some.
for ipCidr in ipv4:
if '/' in ipCidr: ip = ipCidr.split('/')[0]
else: ip = ipCidr
if ip in self.ipv4ToIPv6.keys():
ipv6.append(self.ipv4ToIPv6[ip])
def createShapedDevices(self):
import csv
from ispConfig import bandwidthOverheadFactor
# Builds ShapedDevices.csv from the network tree.
circuits = []
for (i, node) in enumerate(self.nodes):
if node.type == NodeType.client:
parent = self.nodes[node.parentIndex].displayName
if parent == "Shaper Root": parent = ""
circuit = {
"id": node.id,
"name": node.address,
"parent": parent,
"download": node.downloadMbps,
"upload": node.uploadMbps,
"devices": []
}
for child in self.findChildIndices(i):
if self.nodes[child].type == NodeType.device and (len(self.nodes[child].ipv4)+len(self.nodes[child].ipv6)>0):
ipv4 = self.nodes[child].ipv4
ipv6 = self.nodes[child].ipv6
self.__addIpv6FromMap(ipv4, ipv6)
device = {
"id": self.nodes[child].id,
"name": self.nodes[child].displayName,
"mac": self.nodes[child].mac,
"ipv4": ipv4,
"ipv6": ipv6,
}
circuit["devices"].append(device)
if len(circuit["devices"]) > 0:
circuits.append(circuit)
with open('ShapedDevices.csv', 'w', newline='') as csvfile:
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
wr.writerow(['Circuit ID', 'Circuit Name', 'Device ID', 'Device Name', 'Parent Node', 'MAC',
'IPv4', 'IPv6', 'Download Min', 'Upload Min', 'Download Max', 'Upload Max', 'Comment'])
for circuit in circuits:
for device in circuit["devices"]:
#Remove brackets and quotes of list so LibreQoS.py can parse it
device["ipv4"] = str(device["ipv4"]).replace('[','').replace(']','').replace("'",'')
device["ipv6"] = str(device["ipv6"]).replace('[','').replace(']','').replace("'",'')
row = [
circuit["id"],
circuit["name"],
device["id"],
device["name"],
circuit["parent"],
device["mac"],
device["ipv4"],
device["ipv6"],
int(circuit["download"] * 0.98),
int(circuit["upload"] * 0.98),
int(circuit["download"] * bandwidthOverheadFactor),
int(circuit["upload"] * bandwidthOverheadFactor),
""
]
wr.writerow(row)
def plotNetworkGraph(self, showClients=False):
# Requires `pip install graphviz` to function.
# You also need to install graphviz on your PC.
# In Ubuntu, apt install graphviz will do it.
# Plots the network graph to a PDF file, allowing
# visual verification that the graph makes sense.
# Could potentially be useful in a future
# web interface.
import importlib.util
if (spec := importlib.util.find_spec('graphviz')) is None:
return
import graphviz
dot = graphviz.Digraph(
'network', comment="Network Graph", engine="fdp")
for (i, node) in enumerate(self.nodes):
if ((node.type != NodeType.client and node.type != NodeType.device) or showClients):
color = "white"
match node.type:
case NodeType.root: color = "green"
case NodeType.site: color = "red"
case NodeType.ap: color = "blue"
case NodeType.clientWithChildren: color = "magenta"
case NodeType.device: color = "white"
case default: color = "grey"
dot.node("N" + str(i), node.displayName, color=color)
children = self.findChildIndices(i)
for child in children:
if child != i:
if (self.nodes[child].type != NodeType.client and self.nodes[child].type != NodeType.device) or showClients:
dot.edge("N" + str(i), "N" + str(child))
dot.render("network.pdf")
old/v1.3/integrationSplynx.py
-126
View File
@@ -1,126 +0,0 @@
import requests
from ispConfig import excludeSites, findIPv6usingMikrotik, bandwidthOverheadFactor, exceptionCPEs, splynx_api_key, splynx_api_secret, splynx_api_url
from integrationCommon import isIpv4Permitted
import base64
from requests.auth import HTTPBasicAuth
if findIPv6usingMikrotik == True:
from mikrotikFindIPv6 import pullMikrotikIPv6
from integrationCommon import NetworkGraph, NetworkNode, NodeType
def buildHeaders():
credentials = splynx_api_key + ':' + splynx_api_secret
credentials = base64.b64encode(credentials.encode()).decode()
return {'Authorization' : "Basic %s" % credentials}
def spylnxRequest(target, headers):
# Sends a REST GET request to Spylnx and returns the
# result in JSON
url = splynx_api_url + "/api/2.0/" + target
r = requests.get(url, headers=headers, timeout=10)
return r.json()
def getTariffs(headers):
data = spylnxRequest("admin/tariffs/internet", headers)
tariff = []
downloadForTariffID = {}
uploadForTariffID = {}
for tariff in data:
tariffID = tariff['id']
speed_download = round((int(tariff['speed_download']) / 1000))
speed_upload = round((int(tariff['speed_upload']) / 1000))
downloadForTariffID[tariffID] = speed_download
uploadForTariffID[tariffID] = speed_upload
return (tariff, downloadForTariffID, uploadForTariffID)
def getCustomers(headers):
data = spylnxRequest("admin/customers/customer", headers)
#addressForCustomerID = {}
#customerIDs = []
#for customer in data:
# customerIDs.append(customer['id'])
# addressForCustomerID[customer['id']] = customer['street_1']
return data
def getRouters(headers):
data = spylnxRequest("admin/networking/routers", headers)
ipForRouter = {}
for router in data:
routerID = router['id']
ipForRouter[routerID] = router['ip']
return ipForRouter
def combineAddress(json):
# Combines address fields into a single string
# The API docs seem to indicate that there isn't a "state" field?
if json["street_1"]=="" and json["city"]=="" and json["zip_code"]=="":
return json["id"] + "/" + json["name"]
else:
return json["street_1"] + " " + json["city"] + " " + json["zip_code"]
def createShaper():
net = NetworkGraph()
print("Fetching data from Spylnx")
headers = buildHeaders()
tariff, downloadForTariffID, uploadForTariffID = getTariffs(headers)
customers = getCustomers(headers)
ipForRouter = getRouters(headers)
# It's not very clear how a service is meant to handle multiple
# devices on a shared tariff. Creating each service as a combined
# entity including the customer, to be on the safe side.
for customerJson in customers:
if customerJson['status'] == 'active':
services = spylnxRequest("admin/customers/customer/" + customerJson["id"] + "/internet-services", headers)
for serviceJson in services:
combinedId = "c_" + str(customerJson["id"]) + "_s_" + str(serviceJson["id"])
tariff_id = serviceJson['tariff_id']
customer = NetworkNode(
type=NodeType.client,
id=combinedId,
displayName=customerJson["name"],
address=combineAddress(customerJson),
download=downloadForTariffID[tariff_id],
upload=uploadForTariffID[tariff_id],
)
net.addRawNode(customer)
ipv4 = ''
ipv6 = ''
routerID = serviceJson['router_id']
# If not "Taking IPv4" (Router will assign IP), then use router's set IP
if serviceJson['taking_ipv4'] == 0:
ipv4 = ipForRouter[routerID]
elif serviceJson['taking_ipv4'] == 1:
ipv4 = serviceJson['ipv4']
# If not "Taking IPv6" (Router will assign IP), then use router's set IP
if serviceJson['taking_ipv6'] == 0:
ipv6 = ''
elif serviceJson['taking_ipv6'] == 1:
ipv6 = serviceJson['ipv6']
device = NetworkNode(
id=combinedId+"_d" + str(serviceJson["id"]),
displayName=serviceJson["description"],
type=NodeType.device,
parentId=combinedId,
mac=serviceJson["mac"],
ipv4=[ipv4],
ipv6=[ipv6]
)
net.addRawNode(device)
net.prepareTree()
net.plotNetworkGraph(False)
if net.doesNetworkJsonExist():
print("network.json already exists. Leaving in-place.")
else:
net.createNetworkJson()
net.createShapedDevices()
def importFromSplynx():
#createNetworkJSON()
createShaper()
if __name__ == '__main__':
importFromSplynx()
old/v1.3/integrationUISP.py
-220
View File
@@ -1,220 +0,0 @@
import requests
import os
import csv
from ispConfig import uispSite, uispStrategy
from integrationCommon import isIpv4Permitted, fixSubnet
def uispRequest(target):
# Sends an HTTP request to UISP and returns the
# result in JSON. You only need to specify the
# tail end of the URL, e.g. "sites"
from ispConfig import UISPbaseURL, uispAuthToken
url = UISPbaseURL + "/nms/api/v2.1/" + target
headers = {'accept': 'application/json', 'x-auth-token': uispAuthToken}
r = requests.get(url, headers=headers, timeout=10)
return r.json()
def buildFlatGraph():
# Builds a high-performance (but lacking in site or AP bandwidth control)
# network.
from integrationCommon import NetworkGraph, NetworkNode, NodeType
from ispConfig import generatedPNUploadMbps, generatedPNDownloadMbps
# Load network sites
print("Loading Data from UISP")
sites = uispRequest("sites")
devices = uispRequest("devices?withInterfaces=true&authorized=true")
# Build a basic network adding every client to the tree
print("Building Flat Topology")
net = NetworkGraph()
for site in sites:
type = site['identification']['type']
if type == "endpoint":
id = site['identification']['id']
address = site['description']['address']
name = site['identification']['name']
type = site['identification']['type']
download = generatedPNDownloadMbps
upload = generatedPNUploadMbps
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
download = int(round(site['qos']['downloadSpeed']/1000000))
upload = int(round(site['qos']['uploadSpeed']/1000000))
node = NetworkNode(id=id, displayName=name, type=NodeType.client, download=download, upload=upload, address=address)
net.addRawNode(node)
for device in devices:
if device['identification']['site'] is not None and device['identification']['site']['id'] == id:
# The device is at this site, so add it
ipv4 = []
ipv6 = []
for interface in device["interfaces"]:
for ip in interface["addresses"]:
ip = ip["cidr"]
if isIpv4Permitted(ip):
ip = fixSubnet(ip)
if ip not in ipv4:
ipv4.append(ip)
# TODO: Figure out Mikrotik IPv6?
mac = device['identification']['mac']
net.addRawNode(NetworkNode(id=device['identification']['id'], displayName=device['identification']
['name'], parentId=id, type=NodeType.device, ipv4=ipv4, ipv6=ipv6, mac=mac))
# Finish up
net.prepareTree()
net.plotNetworkGraph(False)
if net.doesNetworkJsonExist():
print("network.json already exists. Leaving in-place.")
else:
net.createNetworkJson()
net.createShapedDevices()
def buildFullGraph():
# Attempts to build a full network graph, incorporating as much of the UISP
# hierarchy as possible.
from integrationCommon import NetworkGraph, NetworkNode, NodeType
from ispConfig import generatedPNUploadMbps, generatedPNDownloadMbps
# Load network sites
print("Loading Data from UISP")
sites = uispRequest("sites")
devices = uispRequest("devices?withInterfaces=true&authorized=true")
dataLinks = uispRequest("data-links?siteLinksOnly=true")
# Do we already have a integrationUISPbandwidths.csv file?
siteBandwidth = {}
if os.path.isfile("integrationUISPbandwidths.csv"):
with open('integrationUISPbandwidths.csv') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
next(csv_reader)
for row in csv_reader:
name, download, upload = row
download = int(download)
upload = int(upload)
siteBandwidth[name] = {"download": download, "upload": upload}
# Find AP capacities from UISP
for device in devices:
if device['identification']['role'] == "ap":
name = device['identification']['name']
if not name in siteBandwidth and device['overview']['downlinkCapacity'] and device['overview']['uplinkCapacity']:
download = int(device['overview']
['downlinkCapacity'] / 1000000)
upload = int(device['overview']['uplinkCapacity'] / 1000000)
siteBandwidth[device['identification']['name']] = {
"download": download, "upload": upload}
print("Building Topology")
net = NetworkGraph()
# Add all sites and client sites
for site in sites:
id = site['identification']['id']
name = site['identification']['name']
type = site['identification']['type']
download = generatedPNDownloadMbps
upload = generatedPNUploadMbps
address = ""
if site['identification']['parent'] is None:
parent = ""
else:
parent = site['identification']['parent']['id']
match type:
case "site":
nodeType = NodeType.site
if name in siteBandwidth:
# Use the CSV bandwidth values
download = siteBandwidth[name]["download"]
upload = siteBandwidth[name]["upload"]
else:
# Add them just in case
siteBandwidth[name] = {
"download": download, "upload": upload}
case default:
nodeType = NodeType.client
address = site['description']['address']
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
download = int(round(site['qos']['downloadSpeed']/1000000))
upload = int(round(site['qos']['uploadSpeed']/1000000))
node = NetworkNode(id=id, displayName=name, type=nodeType,
parentId=parent, download=download, upload=upload, address=address)
# If this is the uispSite node, it becomes the root. Otherwise, add it to the
# node soup.
if name == uispSite:
net.replaceRootNode(node)
else:
net.addRawNode(node)
for device in devices:
if device['identification']['site'] is not None and device['identification']['site']['id'] == id:
# The device is at this site, so add it
ipv4 = []
ipv6 = []
for interface in device["interfaces"]:
for ip in interface["addresses"]:
ip = ip["cidr"]
if isIpv4Permitted(ip):
ip = fixSubnet(ip)
if ip not in ipv4:
ipv4.append(ip)
# TODO: Figure out Mikrotik IPv6?
mac = device['identification']['mac']
net.addRawNode(NetworkNode(id=device['identification']['id'], displayName=device['identification']
['name'], parentId=id, type=NodeType.device, ipv4=ipv4, ipv6=ipv6, mac=mac))
# Now iterate access points, and look for connections to sites
for node in net.nodes:
if node.type == NodeType.device:
for dl in dataLinks:
if dl['from']['device'] is not None and dl['from']['device']['identification']['id'] == node.id:
if dl['to']['site'] is not None and dl['from']['site']['identification']['id'] != dl['to']['site']['identification']['id']:
target = net.findNodeIndexById(
dl['to']['site']['identification']['id'])
if target > -1:
# We found the site
if net.nodes[target].type == NodeType.client or net.nodes[target].type == NodeType.clientWithChildren:
net.nodes[target].parentId = node.id
node.type = NodeType.ap
if node.displayName in siteBandwidth:
# Use the bandwidth numbers from the CSV file
node.uploadMbps = siteBandwidth[node.displayName]["upload"]
node.downloadMbps = siteBandwidth[node.displayName]["download"]
else:
# Add some defaults in case they want to change them
siteBandwidth[node.displayName] = {
"download": generatedPNDownloadMbps, "upload": generatedPNUploadMbps}
net.prepareTree()
net.plotNetworkGraph(False)
if net.doesNetworkJsonExist():
print("network.json already exists. Leaving in-place.")
else:
net.createNetworkJson()
net.createShapedDevices()
# Save integrationUISPbandwidths.csv
# (the newLine fixes generating extra blank lines)
with open('integrationUISPbandwidths.csv', 'w', newline='') as csvfile:
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
wr.writerow(['ParentNode', 'Download Mbps', 'Upload Mbps'])
for device in siteBandwidth:
entry = (
device, siteBandwidth[device]["download"], siteBandwidth[device]["upload"])
wr.writerow(entry)
def importFromUISP():
match uispStrategy:
case "full": buildFullGraph()
case default: buildFlatGraph()
if __name__ == '__main__':
importFromUISP()
old/v1.3/ispConfig.example.py
-101
View File
@@ -1,101 +0,0 @@
# 'fq_codel' or 'cake diffserv4'
# 'cake diffserv4' is recommended
# sqm = 'fq_codel'
sqm = 'cake diffserv4'
# Used to passively monitor the network for before / after comparisons. Leave as False to
# ensure actual shaping. After changing this value, run "sudo systemctl restart LibreQoS.service"
monitorOnlyMode = False
# How many Mbps are available to the edge of this network
upstreamBandwidthCapacityDownloadMbps = 1000
upstreamBandwidthCapacityUploadMbps = 1000
# Devices in ShapedDevices.csv without a defined ParentNode will be placed under a generated
# parent node, evenly spread out across CPU cores. Here, define the bandwidth limit for each
# of those generated parent nodes.
generatedPNDownloadMbps = 1000
generatedPNUploadMbps = 1000
# Interface connected to core router
interfaceA = 'eth1'
# Interface connected to edge router
interfaceB = 'eth2'
## WORK IN PROGRESS. Note that interfaceA determines the "stick" interface
## I could only get scanning to work if I issued ethtool -K enp1s0f1 rxvlan off
OnAStick = False
# VLAN facing the core router
StickVlanA = 0
# VLAN facing the edge router
StickVlanB = 0
# Allow shell commands. False causes commands print to console only without being executed.
# MUST BE ENABLED FOR PROGRAM TO FUNCTION
enableActualShellCommands = True
# Add 'sudo' before execution of any shell commands. May be required depending on distribution and environment.
runShellCommandsAsSudo = False
# Allows overriding queues / CPU cores used. When set to 0, the max possible queues / CPU cores are utilized. Please leave as 0.
queuesAvailableOverride = 0
# Some networks are flat - where there are no Parent Nodes defined in ShapedDevices.csv
# For such flat networks, just define network.json as {} and enable this setting
# By default, it balances the subscribers across CPU cores, factoring in their max bandwidth rates
# Past 25,000 subsribers this algorithm becomes inefficient and is not advised
useBinPackingToBalanceCPU = True
# Bandwidth & Latency Graphing
influxDBEnabled = True
influxDBurl = "http://localhost:8086"
influxDBBucket = "libreqos"
influxDBOrg = "Your ISP Name Here"
influxDBtoken = ""
# NMS/CRM Integration
# If a device shows a WAN IP within these subnets, assume they are behind NAT / un-shapable, and ignore them
ignoreSubnets = ['192.168.0.0/16']
allowedSubnets = ['100.64.0.0/10']
# Splynx Integration
automaticImportSplynx = False
splynx_api_key = ''
splynx_api_secret = ''
# Everything before /api/2.0/ on your Splynx instance
splynx_api_url = 'https://YOUR_URL.splynx.app'
# UISP integration
automaticImportUISP = False
uispAuthToken = ''
# Everything before /nms/ on your UISP instance
UISPbaseURL = 'https://examplesite.com'
# UISP Site - enter the name of the root site in your network tree
# to act as the starting point for the tree mapping
uispSite = ''
# Strategy:
# * "flat" - create all client sites directly off the top of the tree,
# provides maximum performance - at the expense of not offering AP,
# or site options.
# * "full" - build a complete network map
uispStrategy = "full"
# List any sites that should not be included, with each site name surrounded by '' and seperated by commas
excludeSites = []
# If you use IPv6, this can be used to find associated IPv6 prefixes for your clients' IPv4 addresses, and match them to those devices
findIPv6usingMikrotik = False
# If you want to provide a safe cushion for speed test results to prevent customer complains, you can set this to 1.15 (15% above plan rate).
# If not, you can leave as 1.0
bandwidthOverheadFactor = 1.0
# For edge cases, set the respective ParentNode for these CPEs
exceptionCPEs = {}
# 'CPE-SomeLocation1': 'AP-SomeLocation1',
# 'CPE-SomeLocation2': 'AP-SomeLocation2',
#}
# API Auth
apiUsername = "testUser"
apiPassword = "changeme8343486806"
apiHostIP = "127.0.0.1"
apiHostPost = 5000
old/v1.3/mikrotikDHCPRouterList.csv
-2
View File
@@ -1,2 +0,0 @@
Router Name / ID,IP,API Username,API Password, API Port
main,100.64.0.1,admin,password,8728
1 Router Name / ID IP API Username API Password API Port
2 main 100.64.0.1 admin password 8728
old/v1.3/mikrotikFindIPv6.py
-52
View File
@@ -1,52 +0,0 @@
#!/usr/bin/python3
import routeros_api
import csv
def pullMikrotikIPv6():
ipv4ToIPv6 = {}
routerList = []
with open('mikrotikDHCPRouterList.csv') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
next(csv_reader)
for row in csv_reader:
RouterName, IP, Username, Password, apiPort = row
routerList.append((RouterName, IP, Username, Password, apiPort))
for router in routerList:
RouterName, IP, inputUsername, inputPassword = router
connection = routeros_api.RouterOsApiPool(IP, username=inputUsername, password=inputPassword, port=apiPort, use_ssl=False, ssl_verify=False, ssl_verify_hostname=False, plaintext_login=True)
api = connection.get_api()
macToIPv4 = {}
macToIPv6 = {}
clientAddressToIPv6 = {}
list_dhcp = api.get_resource('/ip/dhcp-server/lease')
entries = list_dhcp.get()
for entry in entries:
try:
macToIPv4[entry['mac-address']] = entry['address']
except:
pass
list_dhcp = api.get_resource('/ipv6/dhcp-server/binding')
entries = list_dhcp.get()
for entry in entries:
try:
clientAddressToIPv6[entry['client-address']] = entry['address']
except:
pass
list_dhcp = api.get_resource('/ipv6/neighbor')
entries = list_dhcp.get()
for entry in entries:
try:
realIPv6 = clientAddressToIPv6[entry['address']]
macToIPv6[entry['mac-address']] = realIPv6
except:
pass
for mac, ipv6 in macToIPv6.items():
try:
ipv4 = macToIPv4[mac]
ipv4ToIPv6[ipv4] = ipv6
except:
print('Failed to find associated IPv4 for ' + ipv6)
return ipv4ToIPv6
if __name__ == '__main__':
print(pullMikrotikIPv6())
old/v1.3/network.example.json
-75
View File
@@ -1,75 +0,0 @@
{
"Site_1":
{
"downloadBandwidthMbps":1000,
"uploadBandwidthMbps":1000,
"children":
{
"AP_A":
{
"downloadBandwidthMbps":500,
"uploadBandwidthMbps":500
},
"Site_3":
{
"downloadBandwidthMbps":500,
"uploadBandwidthMbps":500,
"children":
{
"PoP_5":
{
"downloadBandwidthMbps":200,
"uploadBandwidthMbps":200,
"children":
{
"AP_9":
{
"downloadBandwidthMbps":120,
"uploadBandwidthMbps":120
},
"PoP_6":
{
"downloadBandwidthMbps":60,
"uploadBandwidthMbps":60,
"children":
{
"AP_11":
{
"downloadBandwidthMbps":30,
"uploadBandwidthMbps":30
}
}
}
}
}
}
}
}
},
"Site_2":
{
"downloadBandwidthMbps":500,
"uploadBandwidthMbps":500,
"children":
{
"PoP_1":
{
"downloadBandwidthMbps":200,
"uploadBandwidthMbps":200,
"children":
{
"AP_7":
{
"downloadBandwidthMbps":100,
"uploadBandwidthMbps":100
}
}
},
"AP_1":
{
"downloadBandwidthMbps":150,
"uploadBandwidthMbps":150
}
}
}
}
old/v1.3/scheduler.py
-62
View File
@@ -1,62 +0,0 @@
import time
import datetime
from LibreQoS import refreshShapers, refreshShapersUpdateOnly
from graphInfluxDB import refreshBandwidthGraphs, refreshLatencyGraphs
from ispConfig import influxDBEnabled, automaticImportUISP, automaticImportSplynx
if automaticImportUISP:
from integrationUISP import importFromUISP
if automaticImportSplynx:
from integrationSplynx import importFromSplynx
def importFromCRM():
if automaticImportUISP:
try:
importFromUISP()
except:
print("Failed to import from UISP")
elif automaticImportSplynx:
try:
importFromSplynx()
except:
print("Failed to import from Splynx")
def importAndShapeFullReload():
importFromCRM()
refreshShapers()
def importAndShapePartialReload():
importFromCRM()
refreshShapersUpdateOnly()
def graph():
time.sleep(10)
try:
refreshBandwidthGraphs()
except:
print("Failed to run refreshBandwidthGraphs()")
time.sleep(10)
try:
refreshBandwidthGraphs()
except:
print("Failed to run refreshBandwidthGraphs()")
time.sleep(10)
try:
refreshBandwidthGraphs()
except:
print("Failed to run refreshBandwidthGraphs()")
time.sleep(1)
try:
refreshLatencyGraphs()
except:
print("Failed to run refreshLatencyGraphs()")
if __name__ == '__main__':
importAndShapeFullReload()
while True:
finish_time = datetime.datetime.now() + datetime.timedelta(minutes=30)
while datetime.datetime.now() < finish_time:
if influxDBEnabled:
graph()
else:
time.sleep(1)
importAndShapePartialReload()
old/v1.3/testGraph.py
-333
View File
@@ -1,333 +0,0 @@
import unittest
class TestGraph(unittest.TestCase):
def test_empty_graph(self):
"""
Test instantiation of the graph type
"""
from integrationCommon import NetworkGraph
graph = NetworkGraph()
self.assertEqual(len(graph.nodes), 1) # There is an automatic root entry
self.assertEqual(graph.nodes[0].id, "FakeRoot")
def test_empty_node(self):
"""
Test instantiation of the GraphNode type
"""
from integrationCommon import NetworkNode, NodeType
node = NetworkNode("test")
self.assertEqual(node.type.value, NodeType.site.value)
self.assertEqual(node.id, "test")
self.assertEqual(node.parentIndex, 0)
def test_node_types(self):
"""
Test that the NodeType enum is working
"""
from integrationCommon import NetworkNode, NodeType
node = NetworkNode("Test", type = NodeType.root)
self.assertEqual(node.type.value, NodeType.root.value)
node = NetworkNode("Test", type = NodeType.site)
self.assertEqual(node.type.value, NodeType.site.value)
node = NetworkNode("Test", type = NodeType.ap)
self.assertEqual(node.type.value, NodeType.ap.value)
node = NetworkNode("Test", type = NodeType.client)
self.assertEqual(node.type.value, NodeType.client.value)
def test_add_raw_node(self):
"""
Adds a single node to a graph to ensure add works
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site"))
self.assertEqual(len(graph.nodes), 2)
self.assertEqual(graph.nodes[1].type.value, NodeType.site.value)
self.assertEqual(graph.nodes[1].parentIndex, 0)
self.assertEqual(graph.nodes[1].id, "Site")
def test_replace_root(self):
"""
Test replacing the default root node with a specified node
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
node = NetworkNode("Test", type = NodeType.site)
graph.replaceRootNode(node)
self.assertEqual(graph.nodes[0].id, "Test")
def add_child_by_named_parent(self):
"""
Tests inserting a node with a named parent
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site"))
graph.addNodeAsChild("site", NetworkNode("Client", type = NodeType.client))
self.assertEqual(len(graph.nodes), 3)
self.assertEqual(graph.nodes[2].parentIndex, 1)
self.assertEqual(graph.nodes[0].parentIndex, 0)
def test_reparent_by_name(self):
"""
Tests that re-parenting a tree by name is functional
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Missing Site", type=NodeType.client))
graph._NetworkGraph__reparentById()
self.assertEqual(len(graph.nodes), 7) # Includes 1 for the fake root
self.assertEqual(graph.nodes[1].parentIndex, 0) # Site 1 is off root
self.assertEqual(graph.nodes[2].parentIndex, 0) # Site 2 is off root
self.assertEqual(graph.nodes[3].parentIndex, 1) # Client 1 found Site 1
self.assertEqual(graph.nodes[4].parentIndex, 1) # Client 2 found Site 1
self.assertEqual(graph.nodes[5].parentIndex, 2) # Client 3 found Site 2
self.assertEqual(graph.nodes[6].parentIndex, 0) # Client 4 didn't find "Missing Site" and goes to root
def test_find_by_id(self):
"""
Tests that finding a node by name succeeds or fails
as expected.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
self.assertEqual(graph.findNodeIndexById("Site 1"), -1) # Test failure
graph.addRawNode(NetworkNode("Site 1"))
self.assertEqual(graph.findNodeIndexById("Site 1"), 1) # Test success
def test_find_by_name(self):
"""
Tests that finding a node by name succeeds or fails
as expected.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
self.assertEqual(graph.findNodeIndexByName("Site 1"), -1) # Test failure
graph.addRawNode(NetworkNode("Site 1", "Site X"))
self.assertEqual(graph.findNodeIndexByName("Site X"), 1) # Test success
def test_find_children(self):
"""
Tests that finding children in the tree works,
both for full and empty cases.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Missing Site", type=NodeType.client))
graph._NetworkGraph__reparentById()
self.assertEqual(graph.findChildIndices(1), [3, 4])
self.assertEqual(graph.findChildIndices(2), [5])
self.assertEqual(graph.findChildIndices(3), [])
def test_clients_with_children(self):
"""
Tests handling cases where a client site
itself has children. This is only useful for
relays where a site hasn't been created in the
middle - but it allows us to graph the more
pathological designs people come up with.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
graph._NetworkGraph__reparentById()
graph._NetworkGraph__promoteClientsWithChildren()
self.assertEqual(graph.nodes[5].type, NodeType.clientWithChildren)
self.assertEqual(graph.nodes[6].type, NodeType.client) # Test that a client is still a client
def test_client_with_children_promotion(self):
"""
Test locating a client site with children, and then promoting it to
create a generated site
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
graph._NetworkGraph__reparentById()
graph._NetworkGraph__promoteClientsWithChildren()
graph._NetworkGraph__clientsWithChildrenToSites()
self.assertEqual(graph.nodes[5].type, NodeType.client)
self.assertEqual(graph.nodes[6].type, NodeType.client) # Test that a client is still a client
self.assertEqual(graph.nodes[7].type, NodeType.site)
self.assertEqual(graph.nodes[7].id, "Client 3_gen")
def test_find_unconnected(self):
"""
Tests traversing a tree and finding nodes that
have no connection to the rest of the tree.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
graph._NetworkGraph__reparentById()
graph._NetworkGraph__promoteClientsWithChildren()
graph.nodes[6].parentIndex = 6 # Create a circle
unconnected = graph._NetworkGraph__findUnconnectedNodes()
self.assertEqual(len(unconnected), 1)
self.assertEqual(unconnected[0], 6)
self.assertEqual(graph.nodes[unconnected[0]].id, "Client 4")
def test_reconnect_unconnected(self):
"""
Tests traversing a tree and finding nodes that
have no connection to the rest of the tree.
Reconnects them and ensures that the orphan is now
parented.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
graph = NetworkGraph()
graph.addRawNode(NetworkNode("Site 1"))
graph.addRawNode(NetworkNode("Site 2"))
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
graph._NetworkGraph__reparentById()
graph._NetworkGraph__promoteClientsWithChildren()
graph.nodes[6].parentIndex = 6 # Create a circle
graph._NetworkGraph__reconnectUnconnected()
unconnected = graph._NetworkGraph__findUnconnectedNodes()
self.assertEqual(len(unconnected), 0)
self.assertEqual(graph.nodes[6].parentIndex, 0)
def test_network_json_exists(self):
from integrationCommon import NetworkGraph
import os
if os.path.exists("network.json"):
os.remove("network.json")
graph = NetworkGraph()
self.assertEqual(graph.doesNetworkJsonExist(), False)
with open('network.json', 'w') as f:
f.write('Dummy')
self.assertEqual(graph.doesNetworkJsonExist(), True)
os.remove("network.json")
def test_network_json_example(self):
"""
Rebuilds the network in network.example.json
and makes sure that it matches.
Should serve as an example for how an integration
can build a functional tree.
"""
from integrationCommon import NetworkGraph, NetworkNode, NodeType
import json
net = NetworkGraph()
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, 500, 500))
net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType.site, 500, 500))
net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType.site, 200, 200))
net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, 120, 120))
net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType.site, 60, 60))
net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, 30, 30))
net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType.site, 200, 200))
net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, 100, 100))
net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, 150, 150))
net.prepareTree()
net.createNetworkJson()
with open('network.json') as file:
newFile = json.load(file)
with open('src/network.example.json') as file:
exampleFile = json.load(file)
self.assertEqual(newFile, exampleFile)
def test_ipv4_to_ipv6_map(self):
"""
Tests the underlying functionality of finding an IPv6 address from an IPv4 mapping
"""
from integrationCommon import NetworkGraph
net = NetworkGraph()
ipv4 = [ "100.64.1.1" ]
ipv6 = []
# Test that it doesn't cause issues without any mappings
net._NetworkGraph__addIpv6FromMap(ipv4, ipv6)
self.assertEqual(len(ipv4), 1)
self.assertEqual(len(ipv6), 0)
# Test a mapping
net.ipv4ToIPv6 = {
"100.64.1.1":"dead::beef/64"
}
net._NetworkGraph__addIpv6FromMap(ipv4, ipv6)
self.assertEqual(len(ipv4), 1)
self.assertEqual(len(ipv6), 1)
self.assertEqual(ipv6[0], "dead::beef/64")
def test_site_exclusion(self):
from integrationCommon import NetworkGraph, NetworkNode, NodeType
net = NetworkGraph()
net.excludeSites = ['Site_2']
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
self.assertEqual(len(net.nodes), 2)
def test_site_exception(self):
from integrationCommon import NetworkGraph, NetworkNode, NodeType
net = NetworkGraph()
net.exceptionCPEs = {
"Site_2": "Site_1"
}
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
self.assertEqual(net.nodes[2].parentId, "Site_1")
net.prepareTree()
self.assertEqual(net.nodes[2].parentIndex, 1)
def test_graph_render_to_pdf(self):
"""
Requires that graphviz be installed with
pip install graphviz
And also the associated graphviz package for
your platform.
See: https://www.graphviz.org/download/
Test that it creates a graphic
"""
import importlib.util
if (spec := importlib.util.find_spec('graphviz')) is None:
return
from integrationCommon import NetworkGraph, NetworkNode, NodeType
net = NetworkGraph()
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, 500, 500))
net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType.site, 500, 500))
net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType.site, 200, 200))
net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, 120, 120))
net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType.site, 60, 60))
net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, 30, 30))
net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType.site, 200, 200))
net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, 100, 100))
net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, 150, 150))
net.prepareTree()
net.plotNetworkGraph(False)
from os.path import exists
self.assertEqual(exists("network.pdf.pdf"), True)
if __name__ == '__main__':
unittest.main()
old/v1.3/testIP.py
-54
View File
@@ -1,54 +0,0 @@
import unittest
import sys
class TestIP(unittest.TestCase):
def test_ignore(self):
"""
Test that we are correctly ignoring an IP address
"""
sys.path.append('testdata/')
from integrationCommon import isInIgnoredSubnets
self.assertEqual(isInIgnoredSubnets("192.168.1.1"),True)
def test_not_ignore(self):
"""
Test that we are not ignoring an IP address
"""
sys.path.append('testdata/')
from integrationCommon import isInIgnoredSubnets
self.assertEqual(isInIgnoredSubnets("10.0.0.1"),False)
def test_allowed(self):
"""
Test that we are correctly permitting an IP address
"""
sys.path.append('testdata/')
from integrationCommon import isInAllowedSubnets
self.assertEqual(isInAllowedSubnets("100.64.1.1"),True)
def test_not_allowed(self):
"""
Test that we are correctly not permitting an IP address
"""
sys.path.append('testdata/')
from integrationCommon import isInAllowedSubnets
self.assertEqual(isInAllowedSubnets("101.64.1.1"),False)
def test_is_permitted(self):
"""
Test the combined isIpv4Permitted function for true
"""
sys.path.append('testdata/')
from integrationCommon import isIpv4Permitted
self.assertEqual(isIpv4Permitted("100.64.1.1"),True)
def test_is_not_permitted(self):
"""
Test the combined isIpv4Permitted function for false
"""
sys.path.append('testdata/')
from integrationCommon import isIpv4Permitted
self.assertEqual(isIpv4Permitted("101.64.1.1"),False)
if __name__ == '__main__':
unittest.main()
old/v1.3/testdata/ispConfig.py
Vendored
-2
View File
@@ -1,2 +0,0 @@
ignoreSubnets = ['192.168.0.0/16']
allowedSubnets = ['100.64.0.0/10']
old/v1.3/testdata/sample_layout.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 62 KiB