Merge pull request #403 from janrovner/main

throughput_reader sizeof fix, buld_rust.sh fix

.readthedocs.yaml

@@ -0,0 +1,64 @@
+# Read the Docs configuration file for Sphinx projects
+
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+
+# Required
+
+version: 2
+
+
+# Set the OS, Python version and other tools you might need
+
+build:
+
+  os: ubuntu-22.04
+
+  tools:
+
+    python: "3.11"
+
+    # You can also specify other tool versions:
+
+    # nodejs: "20"
+
+    # rust: "1.70"
+
+    # golang: "1.20"
+
+
+# Build documentation in the "docs/" directory with Sphinx
+
+sphinx:
+
+  configuration: conf.py
+
+  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+
+  # builder: "dirhtml"
+
+  # Fail on all warnings to avoid broken references
+
+  # fail_on_warning: true
+
+
+# Optionally build your docs in additional formats such as PDF and ePub
+
+# formats:
+
+#    - pdf
+
+#    - epub
+
+
+# Optional but recommended, declare the Python requirements required
+
+# to build your documentation
+
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+
+python:
+
+   install:
+
+   - requirements: requirements.txt

README.md

@@ -15,7 +15,7 @@ Learn more about [Equinix Metal here](https://deploy.equinix.com/metal/).
 
 Please support the continued development of LibreQoS by sponsoring us via [GitHub Sponsors](https://github.com/sponsors/LibreQoE) or [Patreon](https://patreon.com/libreqos).
 
-## Documentation / Get Started
+## Documentation / Get Started / System Requirements
 
 [ReadTheDocs](https://libreqos.readthedocs.io/en/latest/)
 
@@ -24,94 +24,3 @@ Please support the continued development of LibreQoS by sponsoring us via [GitHu
 Our Matrix chat channel is available at [https://matrix.to/#/#libreqos:matrix.org](https://matrix.to/#/#libreqos:matrix.org).
 
 <img alt="LibreQoS" src="https://user-images.githubusercontent.com/22501920/223866474-603e1112-e2e6-4c67-93e4-44c17b1b7c43.png"></a>
-
-## Features
-
-### Flexible Hierarchical Shaping / Back-Haul Congestion Mitigation
-
-<img src="https://raw.githubusercontent.com/LibreQoE/LibreQoS/main/docs/nestedHTB2.png" width="350"></img>
-
-Starting in version v1.1+, operators can map their network hierarchy in LibreQoS. This enables both simple network hierarchies (Site>AP>Client) as well as much more complex ones (Site>Site>Micro-PoP>AP>Site>AP>Client). This can be used to ensure that a given site’s peak bandwidth will not exceed the capacity of its back-haul links (back-haul congestion control). Operators can support more users on the same network equipment with LibreQoS than with competing QoE solutions which only shape by AP and Client.
-
-### CAKE
-
-CAKE is the product of nearly a decade of development efforts to improve on fq\_codel. With the diffserv\_4 parameter enabled – CAKE groups traffic in to Bulk, Best Effort, Video, and Voice. This means that without having to fine-tune traffic priorities as you would with DPI products – CAKE automatically ensures your clients’ OS update downloads will not disrupt their zoom calls. It allows for multiple video conferences to operate on the same connection which might otherwise “fight” for upload bandwidth causing call disruptions. With work-from-home, remote learning, and tele-medicine becoming increasingly common – minimizing video call disruptions can save jobs, keep students engaged, and help ensure equitable access to medical care.
-
-### XDP
-
-Fast, multi-CPU queueing leveraging xdp-cpumap-tc and cpumap-pping. Currently tested in the real world past 11 Gbps (so far) with just 30% CPU use on a 16 core Intel Xeon Gold 6254. It's likely capable of 30Gbps or more.
-
-### Graphing
-
-You can graph bandwidth and TCP RTT by client and node (Site, AP, etc), using InfluxDB.
-
-### CRM Integrations
-
-## Server Recommendations
-It is most cost-effective to buy a used server with specifications matching your unique requirements, as laid out in the System Requirements section below.
-For those who do not have the time to do that, here are some off-the-shelf options to consider:
-* 1 Gbps | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
-* 10 Gbps | [Supermicro SuperServer 510T-ML (Choose E-2388G)](https://www.thinkmate.com/system/superserver-510t-ml)
-* 20 Gbps | [Dell R450 Config](https://www.dell.com/en-us/shop/servers-storage-and-networking/poweredge-r450-rack-server/spd/poweredge-r450/pe_r450_15127_vi_vp?configurationid=a7663c54-6e4a-4c96-9a21-bc5a69d637ba)
-
-The [AsRock 1U4LW-B6502L2T](https://www.thinkmate.com/system/asrock-1u4lw-b6502l2t/635744) can be a great lower-cost option as well.
-
-## System Requirements
-### VM or physical server
-* For VMs, NIC passthrough is required for optimal throughput and latency (XDP vs generic XDP). Using Virtio / bridging is much slower than NIC passthrough. Virtio / bridging should not be used for large amounts of traffic.
-
-### CPU
-* 2 or more CPU cores
-* A CPU with solid [single-thread performance](https://www.cpubenchmark.net/singleThread.html#server-thread) within your budget. Queuing is very CPU-intensive, and requires high single-thread performance.
-
-Single-thread CPU performance will determine the max throughput of a single HTB (cpu core), and in turn, what max speed plan you can offer customers.
-
-| Customer Max Plan   | Passmark Single-Thread   |
-| --------------------| ------------------------ |
-| 100 Mbps            | 1000                     |
-| 250 Mbps            | 1250                     |
-| 500 Mbps            | 1500                     |
-| 1 Gbps              | 1750                     |
-| 2 Gbps              | 2000                     |
-| 3 Gbps              | 2500                     |
-| 4 Gbps              | 3000                     |
-
-Below is a table of approximate aggregate throughput capacity, assuming a a CPU with a [single thread](https://www.cpubenchmark.net/singleThread.html#server-thread) performance of 2700 or greater:
-
-| Aggregate Throughput    | CPU Cores     |
-| ------------------------| ------------- |
-| 500 Mbps                | 2             |
-| 1 Gbps                  | 4             |
-| 5 Gbps                  | 6             |
-| 10 Gbps                 | 8             |
-| 20 Gbps                 | 16            |
-| 50 Gbps*                | 32            |
-
-(* Estimated)
-
-So for example, an ISP delivering 1Gbps service plans with 10Gbps aggregate throughput would choose a CPU with a 2500+ single-thread score and 8 cores, such as the Intel Xeon E-2388G @ 3.20GHz.
-
-### Memory
-* Minimum RAM = 2 + (0.002 x Subscriber Count) GB
-* Recommended RAM:
-
-| Subscribers   | RAM           |
-| ------------- | ------------- |
-| 100           | 4 GB          |
-| 1,000         | 8 GB          |
-| 5,000         | 16 GB         |
-| 10,000*       | 18 GB         |
-| 50,000*       | 24 GB         |
-
-(* Estimated)
-
-### Network Interface Requirements
-* One management network interface completely separate from the traffic shaping interfaces. Usually this would be the Ethernet interface built in to the motherboard.
-* Dedicated Network Interface Card for Shaping Interfaces
-  * NIC must have 2 or more interfaces for traffic shaping.
-  * NIC must have multiple TX/RX transmit queues. [Here's how to check from the command line](https://serverfault.com/questions/772380/how-to-tell-if-nic-has-multiqueue-enabled).
-  * Known supported cards:
-    * [NVIDIA Mellanox MCX512A-ACAT](https://www.fs.com/products/119649.html)
-    * NVIDIA Mellanox MCX416A-CCAT
-    * [Intel X710](https://www.fs.com/products/75600.html) * Note - possible i40e driver issue with XDP Redirect for high throughput 10G+
-    * Intel X520

conf.py

@@ -7,7 +7,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
 project = "LibreQoE"
-copyright = "2023, LibreQoE"
+copyright = "2023, LibreQoE, LLC"
 author = "Zach Biles"
 
 # -- General configuration ---------------------------------------------------

docs/SystemRequirements/Compute.md

@@ -11,10 +11,11 @@ Single-thread CPU performance will determine the max throughput of a single HTB
 | Customer Max Plan   | Passmark Single-Thread   |
 | --------------------| ------------------------ |
 | 100 Mbps            | 1000                     |
-| 250 Mbps            | 1500                     |
-| 500 Mbps            | 2000                     |
-| 1 Gbps              | 2500                     |
+| 250 Mbps            | 1250                     |
+| 500 Mbps            | 1500                     |
+| 1 Gbps              | 2000                     |
 | 2 Gbps              | 3000                     |
+| 4 Gbps              | 4000                     |
 
 Below is a table of approximate aggregate throughput capacity, assuming a a CPU with a [single thread](https://www.cpubenchmark.net/singleThread.html#server-thread) performance of 2700 or greater:
 
@@ -48,8 +49,12 @@ So for example, an ISP delivering 1Gbps service plans with 10Gbps aggregate thro
 ### Server Recommendations
 It is most cost-effective to buy a used server with specifications matching your unique requirements, as laid out in the System Requirements section below.
 For those who do not have the time to do that, here are some off-the-shelf options to consider:
-* 1 Gbps | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
-* 10 Gbps | [Supermicro SuperServer 510T-ML (Choose E-2388G)](https://www.thinkmate.com/system/superserver-510t-ml)
-* 20 Gbps | [Dell R450 Config](https://www.dell.com/en-us/shop/servers-storage-and-networking/poweredge-r450-rack-server/spd/poweredge-r450/pe_r450_15127_vi_vp?configurationid=a7663c54-6e4a-4c96-9a21-bc5a69d637ba)
 
-The [AsRock 1U4LW-B6502L2T](https://www.thinkmate.com/system/asrock-1u4lw-b6502l2t/635744) can be a great lower-cost option as well.
+|   Aggregate   | 100Mbps Plans |  1Gbps Plans  |  4Gbps Plans  |
+| ------------- | ------------- | ------------- | ------------- |
+| 1 Gbps Total  |       A       |               |               |
+| 10 Gbps Total |               |       B       |       C       |
+
+* A | [Supermicro SuperServer E100-9W-L](https://www.thinkmate.com/system/superserver-e100-9w-l)
+* B | [Supermicro SuperServer 510T-ML](https://www.thinkmate.com/system/superserver-510t-ml) (Select E-2388G)
+* C | [Supermicro AS-1015A-MT](https://store.supermicro.com/us_en/as-1015a-mt.html) (Ryzen 9 7700X, 2x16GB DDR5 4800MHz ECC, 1xSupermicro 10-Gigabit XL710+ X557)

docs/SystemRequirements/Networking.md

@@ -6,5 +6,5 @@
   * Known supported cards:
     * [NVIDIA Mellanox MCX512A-ACAT](https://www.fs.com/products/119649.html)
     * NVIDIA Mellanox MCX416A-CCAT
-    * [Intel X710](https://www.fs.com/products/75600.html)
+    * [Intel X710](https://www.fs.com/products/75600.html) * Note - possible i40e driver issue with XDP Redirect for high throughput 10G+
     * Intel X520

docs/TechnicalDocs/integrations.md

@@ -20,6 +20,8 @@ You have the option to run integrationUISP.py automatically on boot and every 30
 
 First, set the relevant parameters for Splynx (splynx_api_key, splynx_api_secret, etc.) in ispConfig.py.
 
+The Splynx Integration uses Basic authentication. For using this type of authentication, please make sure you enable [Unsecure access](https://splynx.docs.apiary.io/#introduction/authentication) in your Splynx API key settings. Also the Splynx API key should be granted access to the necessary permissions.
+
 To test the Splynx Integration, use
 
 ```shell

docs/TechnicalDocs/troubleshooting.md

@@ -22,3 +22,15 @@ This tends to show up when the MQ qdisc cannot be added correctly to the NIC int
 
 The scheduler (scheduler.py) runs the InfluxDB integration within a try/except statement. If it fails to update InfluxDB, it will report "Failed to update bandwidth graphs".
 To find the exact cause of the failure, please run ```python3 graphInfluxDB.py``` which will provde more specific errors.
+
+### All customer IPs are listed under Unknown IPs, rather than Shaped Devices in GUI
+```
+cd /opt/libreqos/src
+sudo systemctl stop lqos_scheduler
+sudo python3 LibreQoS.py
+```
+
+The console output from running LibreQoS.py directly provides more specific errors regarding issues with ShapedDevices.csv and network.json
+Once you have identified the error and fixed ShapedDevices.csv and/or Network.json, please then run
+
+```sudo systemctl start lqos_scheduler```

src/build_rust.sh

@@ -60,6 +60,10 @@ for prog in $PROGS
 do
     pushd $prog > /dev/null
     cargo build $BUILD_FLAGS
+    if [ $? -ne 0 ]; then
+      echo "Cargo build failed. Exiting with code 1."
+    exit 1
+    fi    
     popd > /dev/null
 done
 

src/rust/lqos_sys/src/bpf/lqos_kern.c

@@ -359,7 +359,7 @@ int throughput_reader(struct bpf_iter__bpf_map_elem *ctx)
 
     bpf_seq_write(seq, ip, sizeof(struct in6_addr));
     for (__u32 i=0; i<NUM_CPUS; i++) {
-        struct host_counter * content = counter+(i*48);
+        struct host_counter * content = counter+(i*sizeof(struct host_counter));
         bpf_seq_write(seq, content, sizeof(struct host_counter));
     }