This commit is contained in:
rchac committed 2026-03-11 18:22:13 +00:00
1 parent 1f7a4fb932
commit 3a7ae03d44
16 files changed
+203 -58

No files matched your search

+46 -1
View File
@@ -145,7 +145,52 @@ Notas:
#### Consideraciones de CPU
<img width="3276" height="1944" alt="cpu" src="https://github.com/user-attachments/assets/e4eeed5e-eeeb-4251-9258-d301c3814237" />
La planificación de CPU debe seguir el **árbol físico de shaping** (después de promoción), no el árbol lógico crudo de `network.json`.
```{mermaid}
flowchart LR
A[Topología lógica en network.json<br/>puede incluir nodos virtuales] --> B[Promoción durante el armado<br/>remover virtuales y promover hijos]
B --> C[Árbol físico HTB<br/>solo nodos reales]
C --> D[Asignación CPU/binpacking<br/>distribuir nodos físicos de primer nivel]
```
```{mermaid}
flowchart LR
subgraph Arbol Logico
L1[Region]
L2[Town virtual]
L3[AP_A]
L4[AP_B]
L1 --> L2
L2 --> L3
L2 --> L4
end
subgraph Arbol HTB Fisico
P1[Region]
P2[AP_A]
P3[AP_B]
P1 --> P2
P1 --> P3
end
```
```{mermaid}
flowchart TD
WAN[Objetivo WAN modelado 20 Gbps ejemplo] --> C1[CPU 1 presupuesto seguro ~5 Gbps]
WAN --> C2[CPU 2 presupuesto seguro ~5 Gbps]
WAN --> C3[CPU 3 presupuesto seguro ~5 Gbps]
WAN --> C4[CPU 4 presupuesto seguro ~5 Gbps]
C1 --> N1[Nodos fisicos de primer nivel asignados aqui]
C2 --> N2[Nodos fisicos de primer nivel asignados aqui]
C3 --> N3[Nodos fisicos de primer nivel asignados aqui]
C4 --> N4[Nodos fisicos de primer nivel asignados aqui]
```
Notas:
- Los nodos virtuales son solo lógicos y no crean clases HTB.
- La asignación de CPU/binpacking trabaja sobre el árbol físico post-promoción.
- Si la promoción produce colisiones de nombres entre hermanos, el build falla.
- Los valores por núcleo de arriba son ejemplos de planificación, no límites codificados.
#### Ayudante para convertir CSV a JSON
+8
View File
@@ -40,6 +40,14 @@ Cuando hay integraciones habilitadas:
- `network.json` también puede sobrescribirse según configuración (por ejemplo `always_overwrite_network_json`).
- Las ediciones manuales pueden sobrescribirse en el siguiente ciclo de refresco.
## Manejo común de velocidades de cliente
Para las integraciones incluidas que importan velocidades brutas de plan de suscriptor, LibreQoS aplica la misma regla compartida antes de escribir `ShapedDevices.csv`:
- velocidad máxima efectiva del cliente = `max(plan_rate * bandwidth_overhead_factor, plan_rate * client_bandwidth_multiplier)`
Las integraciones que ya consumen velocidades efectivas de shaping conservan esos valores tal como llegan, sin volver a aplicar el multiplicador.
## Páginas relacionadas
- [Quickstart](quickstart-es.md)
+1
View File
@@ -56,6 +56,7 @@ Los nodos virtuales ayudan para organización y agregación visual.
- Úselos para mejorar operabilidad y reportes.
- No los use para ocultar una topología física confusa.
- Valide colisiones de nombres tras promoción virtual.
- Para el flujo lógico-a-físico y asignación CPU/binpacking, consulte [Referencia avanzada de configuración](configuration-advanced-es.md).
## Guardrails de colas/clasificadores
+1
View File
@@ -175,6 +175,7 @@ Si sigue en blanco con tráfico normal, recolecte logs y abra issue.
Si `LibreQoS.py` falla con `Virtual node promotion collision: 'AP_A' already exists at this level.`, hay un nodo con `"virtual": true` cuyos hijos colisionan por nombre al promoverse.
Renombre nodos en conflicto o reestructure jerarquía para evitar colisiones.
Para un visual del flujo lógico-a-físico y la asignación de CPU, consulte [Referencia avanzada de configuración](configuration-advanced-es.md).
### Se alcanzó el límite de circuitos mapeados
+46 -1
View File
@@ -189,7 +189,52 @@ Notes:
#### CPU Considerations
<img width="3276" height="1944" alt="cpu" src="https://github.com/user-attachments/assets/e4eeed5e-eeeb-4251-9258-d301c3814237" />
CPU planning should follow the **physical shaping tree** (post-promotion), not the raw logical tree from `network.json`.
```{mermaid}
flowchart LR
A[Logical topology in network.json<br/>may include virtual nodes] --> B[Promotion for shaping build<br/>remove virtual nodes and promote children]
B --> C[Physical HTB shaping tree<br/>real nodes only]
C --> D[CPU/binpacking placement<br/>distribute physical top-level shaped nodes]
```
```{mermaid}
flowchart LR
subgraph Logical Tree
L1[Region]
L2[Town virtual]
L3[AP_A]
L4[AP_B]
L1 --> L2
L2 --> L3
L2 --> L4
end
subgraph Physical HTB Tree
P1[Region]
P2[AP_A]
P3[AP_B]
P1 --> P2
P1 --> P3
end
```
```{mermaid}
flowchart TD
WAN[Shaped WAN target 20 Gbps example] --> C1[CPU 1 safe budget ~5 Gbps]
WAN --> C2[CPU 2 safe budget ~5 Gbps]
WAN --> C3[CPU 3 safe budget ~5 Gbps]
WAN --> C4[CPU 4 safe budget ~5 Gbps]
C1 --> N1[Physical top-level nodes assigned here]
C2 --> N2[Physical top-level nodes assigned here]
C3 --> N3[Physical top-level nodes assigned here]
C4 --> N4[Physical top-level nodes assigned here]
```
Notes:
- Virtual nodes are logical-only and do not create HTB classes.
- CPU placement/binpacking acts on the physical post-promotion tree.
- If promotion creates sibling name collisions, shaping build fails.
- The per-core bandwidth numbers above are planning examples, not hard coded limits.
##### CSV to JSON conversion helper
+8
View File
@@ -51,6 +51,14 @@ When integrations are enabled:
- `network.json` may also be overwritten depending on integration settings (for example `always_overwrite_network_json`).
- Manual edits may be overwritten on the next refresh cycle.
## Common Client Rate Handling
For built-in integrations that import raw subscriber plan speeds, LibreQoS applies the same shared client-rate rule before writing `ShapedDevices.csv`:
- effective client max rate = `max(plan_rate * bandwidth_overhead_factor, plan_rate * client_bandwidth_multiplier)`
Integrations that already ingest effective shaped rates keep those values as-is rather than applying the multiplier a second time.
## Related Pages
- [Quickstart](quickstart.md)
+1
View File
@@ -56,6 +56,7 @@ Virtual nodes are useful for logical organization and aggregation views.
- Use virtual nodes to improve operability and reporting structure.
- Do not use virtual nodes as a substitute for physical topology clarity.
- Validate for name collisions after promotion behavior.
- For logical-to-physical promotion and CPU/binpacking flow, see [Advanced Configuration Reference](configuration-advanced.md).
## Queue/Classifier Guardrails
+1
View File
@@ -199,6 +199,7 @@ If still blank under normal traffic, collect recent logs and open an issue.
If LibreQoS.py fails with an error like `Virtual node promotion collision: 'AP_A' already exists at this level.`, you have a `"virtual": true` node whose children get promoted into a parent level where a node with the same name already exists.
Rename one of the colliding nodes (names must be unique among siblings after virtual-node promotion), or restructure the hierarchy so promoted children won’t collide.
For a visual of the logical-to-physical promotion flow and CPU placement, see [Advanced Configuration Reference](configuration-advanced.md).
### Mapped circuit limit reached
+13
View File
@@ -57,6 +57,19 @@ def fixSubnet(inputIP):
return rawIp + "/32"
return inputIP
def apply_client_bandwidth_multiplier(plan_rate_mbps):
# Convert a raw client/service plan rate into the effective shaped rate.
# The higher of bandwidth_overhead_factor and client_bandwidth_multiplier wins.
plan_rate_mbps = float(plan_rate_mbps or 0.0)
if plan_rate_mbps <= 0:
return 0.0
overhead = bandwidth_overhead_factor()
minimum = client_bandwidth_multiplier()
adjusted = plan_rate_mbps * overhead
floor_value = plan_rate_mbps * minimum
return max(adjusted, floor_value)
class NodeType(enum.IntEnum):
# Enumeration to define what type of node
# a NetworkNode is.
+11 -25
View File
@@ -7,8 +7,6 @@ from pythonCheck import checkPythonVersion
checkPythonVersion()
from liblqos_python import (
bandwidth_overhead_factor,
client_bandwidth_multiplier,
exclude_sites,
integration_common_use_mikrotik_ipv6,
netzur_api_key,
@@ -25,7 +23,7 @@ import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from integrationCommon import NetworkGraph, NetworkNode, NodeType
from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier
logging.basicConfig(level=logging.INFO)
LOG = logging.getLogger("netzur_integration")
@@ -64,18 +62,6 @@ def fetch_netzur_data() -> Tuple[List[dict], List[dict]]:
def _build_exclusion_set() -> set:
return {entry.lower().strip() for entry in exclude_sites() if entry}
def _apply_rate(plan_rate: float) -> float:
plan_rate = float(plan_rate or 0.0)
if plan_rate <= 0:
return 0.0
overhead = bandwidth_overhead_factor()
minimum = client_bandwidth_multiplier()
adjusted = plan_rate * overhead
floor_value = plan_rate * minimum
return max(adjusted, floor_value)
def createShaper() -> NetworkGraph:
LOG.info("[Netzur] Starting sync")
if integration_common_use_mikrotik_ipv6():
@@ -119,17 +105,17 @@ def createShaper() -> NetworkGraph:
circuit_id = f"netzur_{subscriber_id}"
net.addRawNode(
NetworkNode(
id=circuit_id,
displayName=customer_name,
type=NodeType.client,
parentId=parent_id,
address=address,
customerName=customer_name,
download=_apply_rate(subscriber.get("download")),
upload=_apply_rate(subscriber.get("upload")),
NetworkNode(
id=circuit_id,
displayName=customer_name,
type=NodeType.client,
parentId=parent_id,
address=address,
customerName=customer_name,
download=apply_client_bandwidth_multiplier(subscriber.get("download")),
upload=apply_client_bandwidth_multiplier(subscriber.get("upload")),
)
)
)
ipv4 = [subscriber["ip"]] if subscriber.get("ip") else []
ipv6 = [subscriber["ipv6"]] if subscriber.get("ipv6") else []
+3 -3
View File
@@ -8,7 +8,7 @@ import base64
from requests.auth import HTTPBasicAuth
if find_ipv6_using_mikrotik() == True:
from mikrotikFindIPv6 import pullMikrotikIPv6
from integrationCommon import NetworkGraph, NetworkNode, NodeType
from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier
from urllib3.exceptions import InsecureRequestWarning
def getCustomerInfo():
@@ -33,8 +33,8 @@ def getListServices():
for service in r.json():
if service['rate_down'] and service['rate_up']:
servicesDict[service['id']] = {}
servicesDict[service['id']]['downloadMbps'] = int(round(int(service['rate_down']) / 1000))
servicesDict[service['id']]['uploadMbps'] = int(round(int(service['rate_up']) / 1000))
servicesDict[service['id']]['downloadMbps'] = apply_client_bandwidth_multiplier(int(service['rate_down']) / 1000)
servicesDict[service['id']]['uploadMbps'] = apply_client_bandwidth_multiplier(int(service['rate_up']) / 1000)
return servicesDict
def createShaper():
+3 -3
View File
@@ -5,7 +5,7 @@ import subprocess
from liblqos_python import sonar_api_key, sonar_api_url, snmp_community, sonar_airmax_ap_model_ids, \
sonar_ltu_ap_model_ids, sonar_active_status_ids
all_models = sonar_airmax_ap_model_ids() + sonar_ltu_ap_model_ids()
from integrationCommon import NetworkGraph, NetworkNode, NodeType
from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier
from multiprocessing.pool import ThreadPool
### Requirements
@@ -235,8 +235,8 @@ def getAccounts(sonar_active_status_ids):
for item in account['addresses']['entities'][0]['inventory_items']['entities']:
devices.append({'id': item['id'], 'name': item['inventory_model']['name'], 'ips': findIPs(item), 'mac': item['inventory_model_field_data']['entities'][0]['value']})
if account['account_services']['entities'] and devices: # Make sure there is a data plan and devices on the account.
download = float(account['account_services']['entities'][0]['service']['data_service_detail']['download_speed_kilobits_per_second'])/1000
upload = float(account['account_services']['entities'][0]['service']['data_service_detail']['upload_speed_kilobits_per_second'])/1000
download = apply_client_bandwidth_multiplier(float(account['account_services']['entities'][0]['service']['data_service_detail']['download_speed_kilobits_per_second'])/1000)
upload = apply_client_bandwidth_multiplier(float(account['account_services']['entities'][0]['service']['data_service_detail']['upload_speed_kilobits_per_second'])/1000)
if download < 2:
download = 2
if upload < 2:
+4 -4
View File
@@ -6,10 +6,10 @@ import requests
import warnings
import os
import csv
from liblqos_python import exclude_sites, find_ipv6_using_mikrotik, bandwidth_overhead_factor, splynx_api_key, \
from liblqos_python import exclude_sites, find_ipv6_using_mikrotik, splynx_api_key, \
splynx_api_secret, splynx_api_url, splynx_strategy, overwrite_network_json_always
from integrationCommon import isIpv4Permitted
from integrationCommon import apply_client_bandwidth_multiplier, isIpv4Permitted
import base64
from requests.auth import HTTPBasicAuth
if find_ipv6_using_mikrotik() == True:
@@ -425,8 +425,8 @@ def getTariffs(headers):
speed_download = burstable_down
if burstable_up > speed_upload:
speed_upload = burstable_up
downloadForTariffID[tariffID] = speed_download
uploadForTariffID[tariffID] = speed_upload
downloadForTariffID[tariffID] = apply_client_bandwidth_multiplier(speed_download)
uploadForTariffID[tariffID] = apply_client_bandwidth_multiplier(speed_upload)
except:
print("Error, bad data returned from Splynx:")
print(data)
+5 -5
View File
@@ -4,7 +4,7 @@ import requests
import os
import csv
from datetime import datetime, timedelta
from integrationCommon import isIpv4Permitted, fixSubnet
from integrationCommon import apply_client_bandwidth_multiplier, isIpv4Permitted, fixSubnet
from liblqos_python import uisp_site, uisp_strategy, overwrite_network_json_always, uisp_suspended_strategy, \
airmax_capacity, ltu_capacity, use_ptmp_as_parent, uisp_base_url, uisp_auth_token, \
generated_pn_download_mbps, generated_pn_upload_mbps
@@ -43,8 +43,8 @@ def buildFlatGraph():
download = generated_pn_download_mbps()
upload = generated_pn_upload_mbps()
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
download = int(round(site['qos']['downloadSpeed']/1000000))
upload = int(round(site['qos']['uploadSpeed']/1000000))
download = apply_client_bandwidth_multiplier(site['qos']['downloadSpeed']/1000000)
upload = apply_client_bandwidth_multiplier(site['qos']['uploadSpeed']/1000000)
if site['identification'] is not None and site['identification']['suspended'] is not None and site['identification']['suspended'] == True:
if uisp_suspended_strategy() == "ignore":
print("WARNING: Site " + name + " is suspended")
@@ -465,8 +465,8 @@ def buildFullGraph():
except:
customerName = ""
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
download = int(round(site['qos']['downloadSpeed']/1000000))
upload = int(round(site['qos']['uploadSpeed']/1000000))
download = apply_client_bandwidth_multiplier(site['qos']['downloadSpeed']/1000000)
upload = apply_client_bandwidth_multiplier(site['qos']['uploadSpeed']/1000000)
if site['identification'] is not None and site['identification']['suspended'] is not None and site['identification']['suspended'] == True:
if uisp_suspended_strategy() == "ignore":
print("WARNING: Site " + name + " is suspended")
+9 -16
View File
@@ -13,8 +13,6 @@ from collections import Counter
from liblqos_python import (
get_libreqos_directory,
overwrite_network_json_always,
bandwidth_overhead_factor,
client_bandwidth_multiplier,
visp_client_id,
visp_client_secret,
visp_username,
@@ -24,24 +22,19 @@ from liblqos_python import (
visp_timeout_secs,
)
from integrationCommon import NetworkGraph, NetworkNode, NodeType, isIpv4Permitted
from integrationCommon import (
NetworkGraph,
NetworkNode,
NodeType,
apply_client_bandwidth_multiplier,
isIpv4Permitted,
)
TOKEN_URL = "https://data.visp.net/token"
GRAPHQL_URL = "https://integrations.visp.net/graphql"
def _apply_rate(plan_rate_mbps: float) -> float:
plan_rate_mbps = float(plan_rate_mbps or 0.0)
if plan_rate_mbps <= 0:
return 0.0
overhead = bandwidth_overhead_factor()
minimum = client_bandwidth_multiplier()
adjusted = plan_rate_mbps * overhead
floor_value = plan_rate_mbps * minimum
return max(adjusted, floor_value)
def _unit_to_mbps(value: Any, unit: Any) -> Optional[float]:
if value is None:
return None
@@ -316,8 +309,8 @@ def _add_circuit_and_device(
parentId="",
address=customer_name,
customerName=customer_name,
download=_apply_rate(down_mbps),
upload=_apply_rate(up_mbps),
download=apply_client_bandwidth_multiplier(down_mbps),
upload=apply_client_bandwidth_multiplier(up_mbps),
)
)
net.addRawNode(
+43
View File
@@ -278,6 +278,49 @@ class TestGraph(unittest.TestCase):
self.assertEqual(len(ipv6), 1)
self.assertEqual(ipv6[0], "dead::beef/64")
def test_apply_client_bandwidth_multiplier_uses_higher_factor(self):
import integrationCommon
old_overhead = integrationCommon.bandwidth_overhead_factor
old_multiplier = integrationCommon.client_bandwidth_multiplier
try:
integrationCommon.bandwidth_overhead_factor = lambda: 1.1
integrationCommon.client_bandwidth_multiplier = lambda: 1.25
self.assertEqual(integrationCommon.apply_client_bandwidth_multiplier(100), 125.0)
integrationCommon.bandwidth_overhead_factor = lambda: 1.4
integrationCommon.client_bandwidth_multiplier = lambda: 1.25
self.assertEqual(integrationCommon.apply_client_bandwidth_multiplier(100), 140.0)
finally:
integrationCommon.bandwidth_overhead_factor = old_overhead
integrationCommon.client_bandwidth_multiplier = old_multiplier
def test_create_shaped_devices_preserves_effective_client_rate(self):
import csv
import os
import tempfile
from integrationCommon import NetworkGraph, NetworkNode, NodeType
net = NetworkGraph()
net.addRawNode(NetworkNode("client_1", "Client 1", "", NodeType.client, 150.0, 75.0))
net.addRawNode(NetworkNode("device_1", "Device 1", "client_1", NodeType.device, ipv4=["100.64.1.10"], mac="AA:BB:CC:DD:EE:FF"))
net.prepareTree()
old_cwd = os.getcwd()
with tempfile.TemporaryDirectory() as tmpdir:
try:
os.chdir(tmpdir)
net.createShapedDevices()
with open("ShapedDevices.csv", newline="") as csvfile:
rows = list(csv.reader(csvfile))
finally:
os.chdir(old_cwd)
self.assertEqual(rows[1][8], "1")
self.assertEqual(rows[1][9], "1")
self.assertEqual(rows[1][10], "150.0")
self.assertEqual(rows[1][11], "75.0")
def test_site_exclusion(self):
from integrationCommon import NetworkGraph, NetworkNode, NodeType
net = NetworkGraph()