diff --git a/docs-es/v2.0/configuration-advanced-es.md b/docs-es/v2.0/configuration-advanced-es.md index 15f05e1f..cb182325 100644 --- a/docs-es/v2.0/configuration-advanced-es.md +++ b/docs-es/v2.0/configuration-advanced-es.md @@ -145,7 +145,52 @@ Notas: #### Consideraciones de CPU -cpu +La planificación de CPU debe seguir el **árbol físico de shaping** (después de promoción), no el árbol lógico crudo de `network.json`. + +```{mermaid} +flowchart LR + A[Topología lógica en network.json
puede incluir nodos virtuales] --> B[Promoción durante el armado
remover virtuales y promover hijos] + B --> C[Árbol físico HTB
solo nodos reales] + C --> D[Asignación CPU/binpacking
distribuir nodos físicos de primer nivel] +``` + +```{mermaid} +flowchart LR + subgraph Arbol Logico + L1[Region] + L2[Town virtual] + L3[AP_A] + L4[AP_B] + L1 --> L2 + L2 --> L3 + L2 --> L4 + end + subgraph Arbol HTB Fisico + P1[Region] + P2[AP_A] + P3[AP_B] + P1 --> P2 + P1 --> P3 + end +``` + +```{mermaid} +flowchart TD + WAN[Objetivo WAN modelado 20 Gbps ejemplo] --> C1[CPU 1 presupuesto seguro ~5 Gbps] + WAN --> C2[CPU 2 presupuesto seguro ~5 Gbps] + WAN --> C3[CPU 3 presupuesto seguro ~5 Gbps] + WAN --> C4[CPU 4 presupuesto seguro ~5 Gbps] + C1 --> N1[Nodos fisicos de primer nivel asignados aqui] + C2 --> N2[Nodos fisicos de primer nivel asignados aqui] + C3 --> N3[Nodos fisicos de primer nivel asignados aqui] + C4 --> N4[Nodos fisicos de primer nivel asignados aqui] +``` + +Notas: +- Los nodos virtuales son solo lógicos y no crean clases HTB. +- La asignación de CPU/binpacking trabaja sobre el árbol físico post-promoción. +- Si la promoción produce colisiones de nombres entre hermanos, el build falla. +- Los valores por núcleo de arriba son ejemplos de planificación, no límites codificados. #### Ayudante para convertir CSV a JSON diff --git a/docs-es/v2.0/integrations-es.md b/docs-es/v2.0/integrations-es.md index 9e860b19..1e16ff9d 100644 --- a/docs-es/v2.0/integrations-es.md +++ b/docs-es/v2.0/integrations-es.md @@ -40,6 +40,14 @@ Cuando hay integraciones habilitadas: - `network.json` también puede sobrescribirse según configuración (por ejemplo `always_overwrite_network_json`). - Las ediciones manuales pueden sobrescribirse en el siguiente ciclo de refresco. +## Manejo común de velocidades de cliente + +Para las integraciones incluidas que importan velocidades brutas de plan de suscriptor, LibreQoS aplica la misma regla compartida antes de escribir `ShapedDevices.csv`: + +- velocidad máxima efectiva del cliente = `max(plan_rate * bandwidth_overhead_factor, plan_rate * client_bandwidth_multiplier)` + +Las integraciones que ya consumen velocidades efectivas de shaping conservan esos valores tal como llegan, sin volver a aplicar el multiplicador. + ## Páginas relacionadas - [Quickstart](quickstart-es.md) diff --git a/docs-es/v2.0/scale-topology-es.md b/docs-es/v2.0/scale-topology-es.md index 72bcf5b4..377b4f38 100644 --- a/docs-es/v2.0/scale-topology-es.md +++ b/docs-es/v2.0/scale-topology-es.md @@ -56,6 +56,7 @@ Los nodos virtuales ayudan para organización y agregación visual. - Úselos para mejorar operabilidad y reportes. - No los use para ocultar una topología física confusa. - Valide colisiones de nombres tras promoción virtual. +- Para el flujo lógico-a-físico y asignación CPU/binpacking, consulte [Referencia avanzada de configuración](configuration-advanced-es.md). ## Guardrails de colas/clasificadores diff --git a/docs-es/v2.0/troubleshooting-es.md b/docs-es/v2.0/troubleshooting-es.md index afd4125d..a5e8fee6 100644 --- a/docs-es/v2.0/troubleshooting-es.md +++ b/docs-es/v2.0/troubleshooting-es.md @@ -175,6 +175,7 @@ Si sigue en blanco con tráfico normal, recolecte logs y abra issue. Si `LibreQoS.py` falla con `Virtual node promotion collision: 'AP_A' already exists at this level.`, hay un nodo con `"virtual": true` cuyos hijos colisionan por nombre al promoverse. Renombre nodos en conflicto o reestructure jerarquía para evitar colisiones. +Para un visual del flujo lógico-a-físico y la asignación de CPU, consulte [Referencia avanzada de configuración](configuration-advanced-es.md). ### Se alcanzó el límite de circuitos mapeados diff --git a/docs/v2.0/configuration-advanced.md b/docs/v2.0/configuration-advanced.md index f7c2984e..173fccb3 100644 --- a/docs/v2.0/configuration-advanced.md +++ b/docs/v2.0/configuration-advanced.md @@ -189,7 +189,52 @@ Notes: #### CPU Considerations -cpu +CPU planning should follow the **physical shaping tree** (post-promotion), not the raw logical tree from `network.json`. + +```{mermaid} +flowchart LR + A[Logical topology in network.json
may include virtual nodes] --> B[Promotion for shaping build
remove virtual nodes and promote children] + B --> C[Physical HTB shaping tree
real nodes only] + C --> D[CPU/binpacking placement
distribute physical top-level shaped nodes] +``` + +```{mermaid} +flowchart LR + subgraph Logical Tree + L1[Region] + L2[Town virtual] + L3[AP_A] + L4[AP_B] + L1 --> L2 + L2 --> L3 + L2 --> L4 + end + subgraph Physical HTB Tree + P1[Region] + P2[AP_A] + P3[AP_B] + P1 --> P2 + P1 --> P3 + end +``` + +```{mermaid} +flowchart TD + WAN[Shaped WAN target 20 Gbps example] --> C1[CPU 1 safe budget ~5 Gbps] + WAN --> C2[CPU 2 safe budget ~5 Gbps] + WAN --> C3[CPU 3 safe budget ~5 Gbps] + WAN --> C4[CPU 4 safe budget ~5 Gbps] + C1 --> N1[Physical top-level nodes assigned here] + C2 --> N2[Physical top-level nodes assigned here] + C3 --> N3[Physical top-level nodes assigned here] + C4 --> N4[Physical top-level nodes assigned here] +``` + +Notes: +- Virtual nodes are logical-only and do not create HTB classes. +- CPU placement/binpacking acts on the physical post-promotion tree. +- If promotion creates sibling name collisions, shaping build fails. +- The per-core bandwidth numbers above are planning examples, not hard coded limits. ##### CSV to JSON conversion helper diff --git a/docs/v2.0/integrations.md b/docs/v2.0/integrations.md index 8768977d..3042ce11 100644 --- a/docs/v2.0/integrations.md +++ b/docs/v2.0/integrations.md @@ -51,6 +51,14 @@ When integrations are enabled: - `network.json` may also be overwritten depending on integration settings (for example `always_overwrite_network_json`). - Manual edits may be overwritten on the next refresh cycle. +## Common Client Rate Handling + +For built-in integrations that import raw subscriber plan speeds, LibreQoS applies the same shared client-rate rule before writing `ShapedDevices.csv`: + +- effective client max rate = `max(plan_rate * bandwidth_overhead_factor, plan_rate * client_bandwidth_multiplier)` + +Integrations that already ingest effective shaped rates keep those values as-is rather than applying the multiplier a second time. + ## Related Pages - [Quickstart](quickstart.md) diff --git a/docs/v2.0/scale-topology.md b/docs/v2.0/scale-topology.md index 704f3321..c5da558d 100644 --- a/docs/v2.0/scale-topology.md +++ b/docs/v2.0/scale-topology.md @@ -56,6 +56,7 @@ Virtual nodes are useful for logical organization and aggregation views. - Use virtual nodes to improve operability and reporting structure. - Do not use virtual nodes as a substitute for physical topology clarity. - Validate for name collisions after promotion behavior. +- For logical-to-physical promotion and CPU/binpacking flow, see [Advanced Configuration Reference](configuration-advanced.md). ## Queue/Classifier Guardrails diff --git a/docs/v2.0/troubleshooting.md b/docs/v2.0/troubleshooting.md index 19a4ec10..853d63b6 100644 --- a/docs/v2.0/troubleshooting.md +++ b/docs/v2.0/troubleshooting.md @@ -199,6 +199,7 @@ If still blank under normal traffic, collect recent logs and open an issue. If LibreQoS.py fails with an error like `Virtual node promotion collision: 'AP_A' already exists at this level.`, you have a `"virtual": true` node whose children get promoted into a parent level where a node with the same name already exists. Rename one of the colliding nodes (names must be unique among siblings after virtual-node promotion), or restructure the hierarchy so promoted children won’t collide. +For a visual of the logical-to-physical promotion flow and CPU placement, see [Advanced Configuration Reference](configuration-advanced.md). ### Mapped circuit limit reached diff --git a/src/integrationCommon.py b/src/integrationCommon.py index 56e5fcb2..f5d9c9a1 100644 --- a/src/integrationCommon.py +++ b/src/integrationCommon.py @@ -57,6 +57,19 @@ def fixSubnet(inputIP): return rawIp + "/32" return inputIP + +def apply_client_bandwidth_multiplier(plan_rate_mbps): + # Convert a raw client/service plan rate into the effective shaped rate. + # The higher of bandwidth_overhead_factor and client_bandwidth_multiplier wins. + plan_rate_mbps = float(plan_rate_mbps or 0.0) + if plan_rate_mbps <= 0: + return 0.0 + overhead = bandwidth_overhead_factor() + minimum = client_bandwidth_multiplier() + adjusted = plan_rate_mbps * overhead + floor_value = plan_rate_mbps * minimum + return max(adjusted, floor_value) + class NodeType(enum.IntEnum): # Enumeration to define what type of node # a NetworkNode is. diff --git a/src/integrationNetzur.py b/src/integrationNetzur.py index e6afc4e6..172ee0b5 100644 --- a/src/integrationNetzur.py +++ b/src/integrationNetzur.py @@ -7,8 +7,6 @@ from pythonCheck import checkPythonVersion checkPythonVersion() from liblqos_python import ( - bandwidth_overhead_factor, - client_bandwidth_multiplier, exclude_sites, integration_common_use_mikrotik_ipv6, netzur_api_key, @@ -25,7 +23,7 @@ import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry -from integrationCommon import NetworkGraph, NetworkNode, NodeType +from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier logging.basicConfig(level=logging.INFO) LOG = logging.getLogger("netzur_integration") @@ -64,18 +62,6 @@ def fetch_netzur_data() -> Tuple[List[dict], List[dict]]: def _build_exclusion_set() -> set: return {entry.lower().strip() for entry in exclude_sites() if entry} - -def _apply_rate(plan_rate: float) -> float: - plan_rate = float(plan_rate or 0.0) - if plan_rate <= 0: - return 0.0 - overhead = bandwidth_overhead_factor() - minimum = client_bandwidth_multiplier() - adjusted = plan_rate * overhead - floor_value = plan_rate * minimum - return max(adjusted, floor_value) - - def createShaper() -> NetworkGraph: LOG.info("[Netzur] Starting sync") if integration_common_use_mikrotik_ipv6(): @@ -119,17 +105,17 @@ def createShaper() -> NetworkGraph: circuit_id = f"netzur_{subscriber_id}" net.addRawNode( - NetworkNode( - id=circuit_id, - displayName=customer_name, - type=NodeType.client, - parentId=parent_id, - address=address, - customerName=customer_name, - download=_apply_rate(subscriber.get("download")), - upload=_apply_rate(subscriber.get("upload")), + NetworkNode( + id=circuit_id, + displayName=customer_name, + type=NodeType.client, + parentId=parent_id, + address=address, + customerName=customer_name, + download=apply_client_bandwidth_multiplier(subscriber.get("download")), + upload=apply_client_bandwidth_multiplier(subscriber.get("upload")), + ) ) - ) ipv4 = [subscriber["ip"]] if subscriber.get("ip") else [] ipv6 = [subscriber["ipv6"]] if subscriber.get("ipv6") else [] diff --git a/src/integrationPowercode.py b/src/integrationPowercode.py index 756fcba2..ef31935a 100644 --- a/src/integrationPowercode.py +++ b/src/integrationPowercode.py @@ -8,7 +8,7 @@ import base64 from requests.auth import HTTPBasicAuth if find_ipv6_using_mikrotik() == True: from mikrotikFindIPv6 import pullMikrotikIPv6 -from integrationCommon import NetworkGraph, NetworkNode, NodeType +from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier from urllib3.exceptions import InsecureRequestWarning def getCustomerInfo(): @@ -33,8 +33,8 @@ def getListServices(): for service in r.json(): if service['rate_down'] and service['rate_up']: servicesDict[service['id']] = {} - servicesDict[service['id']]['downloadMbps'] = int(round(int(service['rate_down']) / 1000)) - servicesDict[service['id']]['uploadMbps'] = int(round(int(service['rate_up']) / 1000)) + servicesDict[service['id']]['downloadMbps'] = apply_client_bandwidth_multiplier(int(service['rate_down']) / 1000) + servicesDict[service['id']]['uploadMbps'] = apply_client_bandwidth_multiplier(int(service['rate_up']) / 1000) return servicesDict def createShaper(): diff --git a/src/integrationSonar.py b/src/integrationSonar.py index cf125127..b989ffb5 100644 --- a/src/integrationSonar.py +++ b/src/integrationSonar.py @@ -5,7 +5,7 @@ import subprocess from liblqos_python import sonar_api_key, sonar_api_url, snmp_community, sonar_airmax_ap_model_ids, \ sonar_ltu_ap_model_ids, sonar_active_status_ids all_models = sonar_airmax_ap_model_ids() + sonar_ltu_ap_model_ids() -from integrationCommon import NetworkGraph, NetworkNode, NodeType +from integrationCommon import NetworkGraph, NetworkNode, NodeType, apply_client_bandwidth_multiplier from multiprocessing.pool import ThreadPool ### Requirements @@ -235,8 +235,8 @@ def getAccounts(sonar_active_status_ids): for item in account['addresses']['entities'][0]['inventory_items']['entities']: devices.append({'id': item['id'], 'name': item['inventory_model']['name'], 'ips': findIPs(item), 'mac': item['inventory_model_field_data']['entities'][0]['value']}) if account['account_services']['entities'] and devices: # Make sure there is a data plan and devices on the account. - download = float(account['account_services']['entities'][0]['service']['data_service_detail']['download_speed_kilobits_per_second'])/1000 - upload = float(account['account_services']['entities'][0]['service']['data_service_detail']['upload_speed_kilobits_per_second'])/1000 + download = apply_client_bandwidth_multiplier(float(account['account_services']['entities'][0]['service']['data_service_detail']['download_speed_kilobits_per_second'])/1000) + upload = apply_client_bandwidth_multiplier(float(account['account_services']['entities'][0]['service']['data_service_detail']['upload_speed_kilobits_per_second'])/1000) if download < 2: download = 2 if upload < 2: diff --git a/src/integrationSplynx.py b/src/integrationSplynx.py index 1c3bbab5..28e6a077 100644 --- a/src/integrationSplynx.py +++ b/src/integrationSplynx.py @@ -6,10 +6,10 @@ import requests import warnings import os import csv -from liblqos_python import exclude_sites, find_ipv6_using_mikrotik, bandwidth_overhead_factor, splynx_api_key, \ +from liblqos_python import exclude_sites, find_ipv6_using_mikrotik, splynx_api_key, \ splynx_api_secret, splynx_api_url, splynx_strategy, overwrite_network_json_always -from integrationCommon import isIpv4Permitted +from integrationCommon import apply_client_bandwidth_multiplier, isIpv4Permitted import base64 from requests.auth import HTTPBasicAuth if find_ipv6_using_mikrotik() == True: @@ -425,8 +425,8 @@ def getTariffs(headers): speed_download = burstable_down if burstable_up > speed_upload: speed_upload = burstable_up - downloadForTariffID[tariffID] = speed_download - uploadForTariffID[tariffID] = speed_upload + downloadForTariffID[tariffID] = apply_client_bandwidth_multiplier(speed_download) + uploadForTariffID[tariffID] = apply_client_bandwidth_multiplier(speed_upload) except: print("Error, bad data returned from Splynx:") print(data) diff --git a/src/integrationUISP.py b/src/integrationUISP.py index 0a88a1c7..e9fd6299 100644 --- a/src/integrationUISP.py +++ b/src/integrationUISP.py @@ -4,7 +4,7 @@ import requests import os import csv from datetime import datetime, timedelta -from integrationCommon import isIpv4Permitted, fixSubnet +from integrationCommon import apply_client_bandwidth_multiplier, isIpv4Permitted, fixSubnet from liblqos_python import uisp_site, uisp_strategy, overwrite_network_json_always, uisp_suspended_strategy, \ airmax_capacity, ltu_capacity, use_ptmp_as_parent, uisp_base_url, uisp_auth_token, \ generated_pn_download_mbps, generated_pn_upload_mbps @@ -43,8 +43,8 @@ def buildFlatGraph(): download = generated_pn_download_mbps() upload = generated_pn_upload_mbps() if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']): - download = int(round(site['qos']['downloadSpeed']/1000000)) - upload = int(round(site['qos']['uploadSpeed']/1000000)) + download = apply_client_bandwidth_multiplier(site['qos']['downloadSpeed']/1000000) + upload = apply_client_bandwidth_multiplier(site['qos']['uploadSpeed']/1000000) if site['identification'] is not None and site['identification']['suspended'] is not None and site['identification']['suspended'] == True: if uisp_suspended_strategy() == "ignore": print("WARNING: Site " + name + " is suspended") @@ -465,8 +465,8 @@ def buildFullGraph(): except: customerName = "" if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']): - download = int(round(site['qos']['downloadSpeed']/1000000)) - upload = int(round(site['qos']['uploadSpeed']/1000000)) + download = apply_client_bandwidth_multiplier(site['qos']['downloadSpeed']/1000000) + upload = apply_client_bandwidth_multiplier(site['qos']['uploadSpeed']/1000000) if site['identification'] is not None and site['identification']['suspended'] is not None and site['identification']['suspended'] == True: if uisp_suspended_strategy() == "ignore": print("WARNING: Site " + name + " is suspended") diff --git a/src/integrationVISP.py b/src/integrationVISP.py index f6acb02b..972334a3 100644 --- a/src/integrationVISP.py +++ b/src/integrationVISP.py @@ -13,8 +13,6 @@ from collections import Counter from liblqos_python import ( get_libreqos_directory, overwrite_network_json_always, - bandwidth_overhead_factor, - client_bandwidth_multiplier, visp_client_id, visp_client_secret, visp_username, @@ -24,24 +22,19 @@ from liblqos_python import ( visp_timeout_secs, ) -from integrationCommon import NetworkGraph, NetworkNode, NodeType, isIpv4Permitted +from integrationCommon import ( + NetworkGraph, + NetworkNode, + NodeType, + apply_client_bandwidth_multiplier, + isIpv4Permitted, +) TOKEN_URL = "https://data.visp.net/token" GRAPHQL_URL = "https://integrations.visp.net/graphql" -def _apply_rate(plan_rate_mbps: float) -> float: - plan_rate_mbps = float(plan_rate_mbps or 0.0) - if plan_rate_mbps <= 0: - return 0.0 - overhead = bandwidth_overhead_factor() - minimum = client_bandwidth_multiplier() - adjusted = plan_rate_mbps * overhead - floor_value = plan_rate_mbps * minimum - return max(adjusted, floor_value) - - def _unit_to_mbps(value: Any, unit: Any) -> Optional[float]: if value is None: return None @@ -316,8 +309,8 @@ def _add_circuit_and_device( parentId="", address=customer_name, customerName=customer_name, - download=_apply_rate(down_mbps), - upload=_apply_rate(up_mbps), + download=apply_client_bandwidth_multiplier(down_mbps), + upload=apply_client_bandwidth_multiplier(up_mbps), ) ) net.addRawNode( diff --git a/src/testGraph.py b/src/testGraph.py index 0ce07dc8..e2900eb1 100644 --- a/src/testGraph.py +++ b/src/testGraph.py @@ -278,6 +278,49 @@ class TestGraph(unittest.TestCase): self.assertEqual(len(ipv6), 1) self.assertEqual(ipv6[0], "dead::beef/64") + def test_apply_client_bandwidth_multiplier_uses_higher_factor(self): + import integrationCommon + + old_overhead = integrationCommon.bandwidth_overhead_factor + old_multiplier = integrationCommon.client_bandwidth_multiplier + try: + integrationCommon.bandwidth_overhead_factor = lambda: 1.1 + integrationCommon.client_bandwidth_multiplier = lambda: 1.25 + self.assertEqual(integrationCommon.apply_client_bandwidth_multiplier(100), 125.0) + + integrationCommon.bandwidth_overhead_factor = lambda: 1.4 + integrationCommon.client_bandwidth_multiplier = lambda: 1.25 + self.assertEqual(integrationCommon.apply_client_bandwidth_multiplier(100), 140.0) + finally: + integrationCommon.bandwidth_overhead_factor = old_overhead + integrationCommon.client_bandwidth_multiplier = old_multiplier + + def test_create_shaped_devices_preserves_effective_client_rate(self): + import csv + import os + import tempfile + from integrationCommon import NetworkGraph, NetworkNode, NodeType + + net = NetworkGraph() + net.addRawNode(NetworkNode("client_1", "Client 1", "", NodeType.client, 150.0, 75.0)) + net.addRawNode(NetworkNode("device_1", "Device 1", "client_1", NodeType.device, ipv4=["100.64.1.10"], mac="AA:BB:CC:DD:EE:FF")) + net.prepareTree() + + old_cwd = os.getcwd() + with tempfile.TemporaryDirectory() as tmpdir: + try: + os.chdir(tmpdir) + net.createShapedDevices() + with open("ShapedDevices.csv", newline="") as csvfile: + rows = list(csv.reader(csvfile)) + finally: + os.chdir(old_cwd) + + self.assertEqual(rows[1][8], "1") + self.assertEqual(rows[1][9], "1") + self.assertEqual(rows[1][10], "150.0") + self.assertEqual(rows[1][11], "75.0") + def test_site_exclusion(self): from integrationCommon import NetworkGraph, NetworkNode, NodeType net = NetworkGraph()