mirror of
https://github.com/LibreQoE/LibreQoS.git
synced 2025-02-25 18:55:32 -06:00
Merge pull request #145 from thebracket/integration-common-graph
Common graph integrations
This commit is contained in:
Robert Chacón
GitHub
commit
7359727729
149
v1.3/INTEGRATION_HOWTO.md
Normal file
149
v1.3/INTEGRATION_HOWTO.md
Normal file
@ -0,0 +1,149 @@
|
||||
# LibreQoS Integrations
|
||||
|
||||
If you need to create an integration for your network, we've tried to give you the tools you need. We currently ship integrations for UISP and Spylnx. We'd love to include more.
|
||||
|
||||
### Overall Concept
|
||||
|
||||
LibreQoS enforces customer bandwidth limits, and applies CAKE-based optimizations at several levels:
|
||||
|
||||
* Per-user Cake flows are created. These require the maximum bandwidth permitted per customer.
|
||||
* Customers can have more than one device that share a pool of bandwidth. Customers are grouped into "circuits"
|
||||
* *Optional* Access points can have a speed limit/queue, applied to all customers associated with the access point.
|
||||
* *Optional* Sites can contain access points, and apply a speed limit/queue to all access points (and associated circuits).
|
||||
* *Optional* Sites can be nested beneath other sites and access point, providing for a queue hierarchy that represents physical limitations of backhaul connections.
|
||||
|
||||
Additionally, you might grow to have more than one shaper - and need to express your network topology from the perspective of different parts of your network. (For example, if *Site A* and *Site B* both have Internet connections - you want to generate an efficient topology for both sites. It's helpful if you can derive this from the same overall topology)
|
||||
|
||||
LibreQoS's network modeling accomplishes this by modeling your network as a *graph*: a series of interconnected nodes, each featuring a "parent". Any "node" (entry) in the graph can be turned into a "root" node, allowing you to generate the `network.json` and `ShapedDevices.csv` files required to manage your customers from the perspective of that root node.
|
||||
|
||||
### Flat Shaping
|
||||
|
||||
The simplest form of integration produces a "flat" network. This is the highest performance model in terms of raw throughput, but lacks the ability to provide shaping at the access point or site level: every customer site is parented directly off the root.
|
||||
|
||||
> For an integration, it's recommended that you fetch the customer/device data from your management system rather than type them all in Python.
|
||||
|
||||
A flat integration is relatively simple. Start by importing the common API:
|
||||
|
||||
```python
|
||||
from integrationCommon import isIpv4Permitted, fixSubnet, NetworkGraph, NetworkNode, NodeType
|
||||
```
|
||||
|
||||
Then create an empty network graph (it will grow to represent your network):
|
||||
|
||||
```python
|
||||
net = NetworkGraph()
|
||||
```
|
||||
|
||||
Once you have your `NetworkGraph` object, you start adding customers and devices. Customers may have any number of devices. You can add a single customer with one device as follows:
|
||||
|
||||
```python
|
||||
# Add the customer
|
||||
customer = NetworkNode(
|
||||
id="Unique Customer ID",
|
||||
displayName="The Doe Family",
|
||||
type=NodeType.client,
|
||||
download=100, # Download is in Mbit/second
|
||||
upload=20, # Upload is in Mbit/second
|
||||
address="1 My Road, My City, My State")
|
||||
net.addRawNode(customer) # Insert the customer ID
|
||||
|
||||
# Give them a device
|
||||
device = NetworkNode(
|
||||
id="Unique Device ID",
|
||||
displayName="Doe Family CPE",
|
||||
parentId="Unique Customer ID", # must match the customer's ID
|
||||
type=NodeType.device,
|
||||
ipv4=["100.64.1.5/32"], # As many as you need, express networks as the network ID - e.g. 192.168.100.0/24
|
||||
ipv6=["feed:beef::12/64"], # Same again. May be [] for none.
|
||||
mac="00:00:5e:00:53:af"
|
||||
)
|
||||
net.addRawNode(device)
|
||||
```
|
||||
|
||||
If the customer has multiple devices, you can add as many as you want - with `ParentId` continuing to match the parent customer's `id`.
|
||||
|
||||
Once you have entered all of your customers, you can finish the integration:
|
||||
|
||||
```python
|
||||
net.prepareTree() # This is required, and builds parent-child relationships.
|
||||
net.createNetworkJson() # Create `network.json`
|
||||
net.createShapedDevices() # Create the `ShapedDevices.csv` file.
|
||||
```
|
||||
|
||||
### Detailed Hierarchies
|
||||
|
||||
Creating a full hierarchy (with as many levels as you want) uses a similar strategy to flat networks---we recommend that you start by reading the "flat shaping" section above.
|
||||
|
||||
Start by importing the common API:
|
||||
|
||||
```python
|
||||
from integrationCommon import isIpv4Permitted, fixSubnet, NetworkGraph, NetworkNode, NodeType
|
||||
```
|
||||
|
||||
Then create an empty network graph (it will grow to represent your network):
|
||||
|
||||
```python
|
||||
net = NetworkGraph()
|
||||
```
|
||||
|
||||
Now you can start to insert sites and access points. Sites and access points are inserted like customer or device nodes: they have a unique ID, and a `ParentId`. Customers can then use a `ParentId` of the site or access point beneath which they should be located.
|
||||
|
||||
For example, let's create `Site_1` and `Site_2` - at the top of the tree:
|
||||
|
||||
```python
|
||||
net.addRawNode(NetworkNode(id="Site_1", displayName="Site_1", parentId="", type=NodeType.site, download=1000, upload=1000))
|
||||
net.addRawNode(NetworkNode(id="Site_2", displayName="Site_2", parentId="", type=NodeType.site, download=500, upload=500))
|
||||
```
|
||||
|
||||
Let's attach some access points and point-of-presence sites:
|
||||
|
||||
```python
|
||||
net.addRawNode(NetworkNode(id="AP_A", displayName="AP_A", parentId="Site_1", type=NodeType.ap, download=500, upload=500))
|
||||
net.addRawNode(NetworkNode(id="Site_3", displayName="Site_3", parentId="Site_1", type=NodeType.site, download=500, upload=500))
|
||||
net.addRawNode(NetworkNode(id="PoP_5", displayName="PoP_5", parentId="Site_3", type=NodeType.site, download=200, upload=200))
|
||||
net.addRawNode(NetworkNode(id="AP_9", displayName="AP_9", parentId="PoP_5", type=NodeType.ap, download=120, upload=120))
|
||||
net.addRawNode(NetworkNode(id="PoP_6", displayName="PoP_6", parentId="PoP_5", type=NodeType.site, download=60, upload=60))
|
||||
net.addRawNode(NetworkNode(id="AP_11", displayName="AP_11", parentId="PoP_6", type=NodeType.ap, download=30, upload=30))
|
||||
net.addRawNode(NetworkNode(id="PoP_1", displayName="PoP_1", parentId="Site_2", type=NodeType.site, download=200, upload=200))
|
||||
net.addRawNode(NetworkNode(id="AP_7", displayName="AP_7", parentId="PoP_1", type=NodeType.ap, download=100, upload=100))
|
||||
net.addRawNode(NetworkNode(id="AP_1", displayName="AP_1", parentId="Site_2", type=NodeType.ap, download=150, upload=150))
|
||||
```
|
||||
|
||||
When you attach a customer, you can specify a tree entry (e.g. `PoP_5`) as a parent:
|
||||
|
||||
```python
|
||||
# Add the customer
|
||||
customer = NetworkNode(
|
||||
id="Unique Customer ID",
|
||||
displayName="The Doe Family",
|
||||
parentId="PoP_5",
|
||||
type=NodeType.client,
|
||||
download=100, # Download is in Mbit/second
|
||||
upload=20, # Upload is in Mbit/second
|
||||
address="1 My Road, My City, My State")
|
||||
net.addRawNode(customer) # Insert the customer ID
|
||||
|
||||
# Give them a device
|
||||
device = NetworkNode(
|
||||
id="Unique Device ID",
|
||||
displayName="Doe Family CPE",
|
||||
parentId="Unique Customer ID", # must match the customer's ID
|
||||
type=NodeType.device,
|
||||
ipv4=["100.64.1.5/32"], # As many as you need, express networks as the network ID - e.g. 192.168.100.0/24
|
||||
ipv6=["feed:beef::12/64"], # Same again. May be [] for none.
|
||||
mac="00:00:5e:00:53:af"
|
||||
)
|
||||
net.addRawNode(device)
|
||||
```
|
||||
|
||||
Once you have entered all of your network topology and customers, you can finish the integration:
|
||||
|
||||
```python
|
||||
net.prepareTree() # This is required, and builds parent-child relationships.
|
||||
net.createNetworkJson() # Create `network.json`
|
||||
net.createShapedDevices() # Create the `ShapedDevices.csv` file.
|
||||
```
|
||||
|
||||
You can also add a call to `net.plotNetworkGraph(False)` (use `True` to also include every customer; this can make for a HUGE file) to create a PDF file (currently named `network.pdf.pdf`) displaying your topology. The example shown here looks like this:
|
||||
|
||||
|
||||
@ -1,32 +1,419 @@
|
||||
# Provides common functionality shared between
|
||||
# integrations.
|
||||
|
||||
from ispConfig import allowedSubnets, ignoreSubnets
|
||||
import ipaddress;
|
||||
from typing import List, Any
|
||||
from ispConfig import allowedSubnets, ignoreSubnets, generatedPNUploadMbps, generatedPNDownloadMbps
|
||||
import ipaddress
|
||||
import enum
|
||||
|
||||
|
||||
def isInAllowedSubnets(inputIP):
|
||||
# Check whether an IP address occurs inside the allowedSubnets list
|
||||
isAllowed = False
|
||||
if '/' in inputIP:
|
||||
inputIP = inputIP.split('/')[0]
|
||||
for subnet in allowedSubnets:
|
||||
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
|
||||
isAllowed = True
|
||||
return isAllowed
|
||||
isAllowed = False
|
||||
if '/' in inputIP:
|
||||
inputIP = inputIP.split('/')[0]
|
||||
for subnet in allowedSubnets:
|
||||
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
|
||||
isAllowed = True
|
||||
return isAllowed
|
||||
|
||||
|
||||
def isInIgnoredSubnets(inputIP):
|
||||
# Check whether an IP address occurs within the ignoreSubnets list
|
||||
isIgnored = False
|
||||
if '/' in inputIP:
|
||||
inputIP = inputIP.split('/')[0]
|
||||
for subnet in ignoreSubnets:
|
||||
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
|
||||
isIgnored = True
|
||||
return isIgnored
|
||||
isIgnored = False
|
||||
if '/' in inputIP:
|
||||
inputIP = inputIP.split('/')[0]
|
||||
for subnet in ignoreSubnets:
|
||||
if (ipaddress.ip_address(inputIP) in ipaddress.ip_network(subnet)):
|
||||
isIgnored = True
|
||||
return isIgnored
|
||||
|
||||
|
||||
def isIpv4Permitted(inputIP):
|
||||
# Checks whether an IP address is in Allowed Subnets.
|
||||
# If it is, check that it isn't in Ignored Subnets.
|
||||
# If it is allowed and not ignored, returns true.
|
||||
# Otherwise, returns false.
|
||||
return isInIgnoredSubnets(inputIP)==False and isInAllowedSubnets(inputIP)
|
||||
return isInIgnoredSubnets(inputIP) == False and isInAllowedSubnets(inputIP)
|
||||
|
||||
|
||||
def fixSubnet(inputIP):
|
||||
# If an IP address has a CIDR other than /32 (e.g. 192.168.1.1/24),
|
||||
# but doesn't appear as a network address (e.g. 192.168.1.0/24)
|
||||
# then it probably isn't actually serving that whole subnet.
|
||||
# This allows you to specify e.g. 192.168.1.0/24 is "the client
|
||||
# on port 3" in the device, without falling afoul of UISP's inclusion
|
||||
# of subnet masks in device IPs.
|
||||
[rawIp, cidr] = inputIP.split('/')
|
||||
if cidr != "32":
|
||||
try:
|
||||
subnet = ipaddress.ip_network(inputIP)
|
||||
except:
|
||||
# Not a network address
|
||||
return rawIp + "/32"
|
||||
return inputIP
|
||||
|
||||
class NodeType(enum.IntEnum):
|
||||
# Enumeration to define what type of node
|
||||
# a NetworkNode is.
|
||||
root = 1
|
||||
site = 2
|
||||
ap = 3
|
||||
client = 4
|
||||
clientWithChildren = 5
|
||||
device = 6
|
||||
|
||||
|
||||
class NetworkNode:
|
||||
# Defines a node on a LibreQoS network graph.
|
||||
# Nodes default to being disconnected, and
|
||||
# will be mapped to the root of the overall
|
||||
# graph.
|
||||
|
||||
id: str
|
||||
displayName: str
|
||||
parentIndex: int
|
||||
parentId: str
|
||||
type: NodeType
|
||||
downloadMbps: int
|
||||
uploadMbps: int
|
||||
ipv4: List
|
||||
ipv6: List
|
||||
address: str
|
||||
mac: str
|
||||
|
||||
def __init__(self, id: str, displayName: str = "", parentId: str = "", type: NodeType = NodeType.site, download: int = generatedPNDownloadMbps, upload: int = generatedPNUploadMbps, ipv4: List = [], ipv6: List = [], address: str = "", mac: str = "") -> None:
|
||||
self.id = id
|
||||
self.parentIndex = 0
|
||||
self.type = type
|
||||
self.parentId = parentId
|
||||
if displayName == "":
|
||||
self.displayName = id
|
||||
else:
|
||||
self.displayName = displayName
|
||||
self.downloadMbps = download
|
||||
self.uploadMbps = upload
|
||||
self.ipv4 = ipv4
|
||||
self.ipv6 = ipv6
|
||||
self.address = address
|
||||
self.mac = mac
|
||||
|
||||
|
||||
class NetworkGraph:
|
||||
# Defines a network as a graph topology
|
||||
# allowing any integration to build the
|
||||
# graph via a common API, emitting
|
||||
# ShapedDevices and network.json files
|
||||
# via a common interface.
|
||||
|
||||
nodes: List
|
||||
ipv4ToIPv6: Any
|
||||
excludeSites: List # Copied to allow easy in-test patching
|
||||
exceptionCPEs: Any
|
||||
|
||||
def __init__(self) -> None:
|
||||
from ispConfig import findIPv6usingMikrotik, excludeSites, exceptionCPEs
|
||||
self.nodes = [
|
||||
NetworkNode("FakeRoot", type=NodeType.root,
|
||||
parentId="", displayName="Shaper Root")
|
||||
]
|
||||
self.excludeSites = excludeSites
|
||||
self.exceptionCPEs = exceptionCPEs
|
||||
if findIPv6usingMikrotik:
|
||||
from mikrotikFindIPv6 import pullMikrotikIPv6
|
||||
self.ipv4ToIPv6 = pullMikrotikIPv6()
|
||||
else:
|
||||
self.ipv4ToIPv6 = {}
|
||||
|
||||
def addRawNode(self, node: NetworkNode) -> None:
|
||||
# Adds a NetworkNode to the graph, unchanged.
|
||||
# If a site is excluded (via excludedSites in ispConfig)
|
||||
# it won't be added
|
||||
if not node.displayName in self.excludeSites:
|
||||
if node.displayName in self.exceptionCPEs.keys():
|
||||
node.parentId = self.exceptionCPEs[node.displayName]
|
||||
self.nodes.append(node)
|
||||
|
||||
def replaceRootNote(self, node: NetworkNode) -> None:
|
||||
# Replaces the automatically generated root node
|
||||
# with a new node. Useful when you have a top-level
|
||||
# node specified (e.g. "uispSite" in the UISP
|
||||
# integration)
|
||||
self.nodes[0] = node
|
||||
|
||||
def addNodeAsChild(self, parent: str, node: NetworkNode) -> None:
|
||||
# Searches the existing graph for a named parent,
|
||||
# adjusts the new node's parentIndex to match the new
|
||||
# node. The parented node is then inserted.
|
||||
#
|
||||
# Exceptions are NOT applied, since we're explicitly
|
||||
# specifying the parent - we're assuming you really
|
||||
# meant it.
|
||||
if node.displayName in self.excludeSites: return
|
||||
parentIdx = 0
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.id == parent:
|
||||
parentIdx = i
|
||||
node.parentIndex = parentIdx
|
||||
self.nodes.append(node)
|
||||
|
||||
def __reparentById(self) -> None:
|
||||
# Scans the entire node tree, searching for parents
|
||||
# by name. Entries are re-mapped to match the named
|
||||
# parents. You can use this to build a tree from a
|
||||
# blob of raw data.
|
||||
for child in self.nodes:
|
||||
if child.parentId != "":
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.id == child.parentId:
|
||||
child.parentIndex = i
|
||||
|
||||
def findNodeIndexById(self, id: str) -> int:
|
||||
# Finds a single node by identity(id)
|
||||
# Return -1 if not found
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.id == id:
|
||||
return i
|
||||
return -1
|
||||
|
||||
def findNodeIndexByName(self, name: str) -> int:
|
||||
# Finds a single node by identity(name)
|
||||
# Return -1 if not found
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.displayName == name:
|
||||
return i
|
||||
return -1
|
||||
|
||||
def findChildIndices(self, parentIndex: int) -> List:
|
||||
# Returns the indices of all nodes with a
|
||||
# parentIndex equal to the specified parameter
|
||||
result = []
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.parentIndex == parentIndex:
|
||||
result.append(i)
|
||||
return result
|
||||
|
||||
def __promoteClientsWithChildren(self) -> None:
|
||||
# Searches for client sites that have children,
|
||||
# and changes their node type to clientWithChildren
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.type == NodeType.client:
|
||||
for child in self.findChildIndices(i):
|
||||
if self.nodes[child].type != NodeType.device:
|
||||
node.type = NodeType.clientWithChildren
|
||||
|
||||
def __clientsWithChildrenToSites(self) -> None:
|
||||
toAdd = []
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.type == NodeType.clientWithChildren:
|
||||
siteNode = NetworkNode(
|
||||
id=node.id + "_gen",
|
||||
displayName="(Generated Site) " + node.displayName,
|
||||
type=NodeType.site
|
||||
)
|
||||
siteNode.parentIndex = node.parentIndex
|
||||
node.parentId = siteNode.id
|
||||
if node.type == NodeType.clientWithChildren:
|
||||
node.type = NodeType.client
|
||||
for child in self.findChildIndices(i):
|
||||
if self.nodes[child].type == NodeType.client or self.nodes[child].type == NodeType.clientWithChildren or self.nodes[child].type == NodeType.site:
|
||||
self.nodes[child].parentId = siteNode.id
|
||||
toAdd.append(siteNode)
|
||||
|
||||
for n in toAdd:
|
||||
self.addRawNode(n)
|
||||
|
||||
self.__reparentById()
|
||||
|
||||
def __findUnconnectedNodes(self) -> List:
|
||||
# Performs a tree-traversal and finds any nodes that
|
||||
# aren't connected to the root. This is a "sanity check",
|
||||
# and also an easy way to handle "flat" topologies and
|
||||
# ensure that the unconnected nodes are re-connected to
|
||||
# the root.
|
||||
visited = []
|
||||
next = [0]
|
||||
|
||||
while len(next) > 0:
|
||||
nextTraversal = next.pop()
|
||||
visited.append(nextTraversal)
|
||||
for idx in self.findChildIndices(nextTraversal):
|
||||
if idx not in visited:
|
||||
next.append(idx)
|
||||
|
||||
result = []
|
||||
for i, n in enumerate(self.nodes):
|
||||
if i not in visited:
|
||||
result.append(i)
|
||||
return result
|
||||
|
||||
def __reconnectUnconnected(self):
|
||||
# Finds any unconnected nodes and reconnects
|
||||
# them to the root
|
||||
for idx in self.__findUnconnectedNodes():
|
||||
if self.nodes[idx].type == NodeType.site:
|
||||
self.nodes[idx].parentIndex = 0
|
||||
for idx in self.__findUnconnectedNodes():
|
||||
if self.nodes[idx].type == NodeType.clientWithChildren:
|
||||
self.nodes[idx].parentIndex = 0
|
||||
for idx in self.__findUnconnectedNodes():
|
||||
if self.nodes[idx].type == NodeType.client:
|
||||
self.nodes[idx].parentIndex = 0
|
||||
|
||||
def prepareTree(self) -> None:
|
||||
# Helper function that calls all the cleanup and mapping
|
||||
# functions in the right order. Unless you are doing
|
||||
# something special, you can use this instead of
|
||||
# calling the functions individually
|
||||
self.__reparentById()
|
||||
self.__promoteClientsWithChildren()
|
||||
self.__clientsWithChildrenToSites()
|
||||
self.__reconnectUnconnected()
|
||||
|
||||
def doesNetworkJsonExist(self):
|
||||
# Returns true if "network.json" exists, false otherwise
|
||||
import os
|
||||
return os.path.isfile("network.json")
|
||||
|
||||
def __isSite(self, index) -> bool:
|
||||
return self.nodes[index].type == NodeType.ap or self.nodes[index].type == NodeType.site or self.nodes[index].type == NodeType.clientWithChildren
|
||||
|
||||
def createNetworkJson(self):
|
||||
import json
|
||||
topLevelNode = {}
|
||||
self.__visited = [] # Protection against loops - never visit twice
|
||||
|
||||
for child in self.findChildIndices(0):
|
||||
if child > 0 and self.__isSite(child):
|
||||
topLevelNode[self.nodes[child].displayName] = self.__buildNetworkObject(
|
||||
child)
|
||||
|
||||
del self.__visited
|
||||
|
||||
with open('network.json', 'w') as f:
|
||||
json.dump(topLevelNode, f, indent=4)
|
||||
|
||||
def __buildNetworkObject(self, idx):
|
||||
# Private: used to recurse down the network tree while building
|
||||
# network.json
|
||||
self.__visited.append(idx)
|
||||
node = {
|
||||
"downloadBandwidthMbps": self.nodes[idx].downloadMbps,
|
||||
"uploadBandwidthMbps": self.nodes[idx].uploadMbps,
|
||||
}
|
||||
children = {}
|
||||
hasChildren = False
|
||||
for child in self.findChildIndices(idx):
|
||||
if child > 0 and self.__isSite(child) and child not in self.__visited:
|
||||
children[self.nodes[child].displayName] = self.__buildNetworkObject(
|
||||
child)
|
||||
hasChildren = True
|
||||
if hasChildren:
|
||||
node["children"] = children
|
||||
return node
|
||||
|
||||
def __addIpv6FromMap(self, ipv4, ipv6) -> None:
|
||||
# Scans each address in ipv4. If its present in the
|
||||
# IPv4 to Ipv6 map (currently pulled from Mikrotik devices
|
||||
# if findIPv6usingMikrotik is enabled), then matching
|
||||
# IPv6 networks are appended to the ipv6 list.
|
||||
# This is explicitly non-destructive of the existing IPv6
|
||||
# list, in case you already have some.
|
||||
for ipCidr in ipv4:
|
||||
if '/' in ipCidr: ip = ipCidr.split('/')[0]
|
||||
else: ip = ipCidr
|
||||
if ip in self.ipv4ToIPv6.keys():
|
||||
ipv6.append(self.ipv4ToIPv6[ip])
|
||||
|
||||
def createShapedDevices(self):
|
||||
import csv
|
||||
from ispConfig import bandwidthOverheadFactor
|
||||
# Builds ShapedDevices.csv from the network tree.
|
||||
circuits = []
|
||||
nextId = 0
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if node.type == NodeType.client:
|
||||
parent = self.nodes[node.parentIndex].displayName
|
||||
if parent == "Shaper Root": parent = ""
|
||||
circuit = {
|
||||
"id": nextId,
|
||||
"name": node.address,
|
||||
"parent": parent,
|
||||
"download": node.downloadMbps,
|
||||
"upload": node.uploadMbps,
|
||||
"devices": []
|
||||
}
|
||||
for child in self.findChildIndices(i):
|
||||
if self.nodes[child].type == NodeType.device and (len(self.nodes[child].ipv4)+len(self.nodes[child].ipv6)>0):
|
||||
ipv4 = self.nodes[child].ipv4
|
||||
ipv6 = self.nodes[child].ipv6
|
||||
self.__addIpv6FromMap(ipv4, ipv6)
|
||||
device = {
|
||||
"id": self.nodes[child].id,
|
||||
"name": self.nodes[child].displayName,
|
||||
"mac": self.nodes[child].mac,
|
||||
"ipv4": ipv4,
|
||||
"ipv6": ipv6,
|
||||
}
|
||||
circuit["devices"].append(device)
|
||||
if len(circuit["devices"]) > 0:
|
||||
circuits.append(circuit)
|
||||
nextId += 1
|
||||
|
||||
with open('ShapedDevices.csv', 'w', newline='') as csvfile:
|
||||
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
|
||||
wr.writerow(['Circuit ID', 'Circuit Name', 'Device ID', 'Device Name', 'Parent Node', 'MAC',
|
||||
'IPv4', 'IPv6', 'Download Min', 'Upload Min', 'Download Max', 'Upload Max', 'Comment'])
|
||||
for circuit in circuits:
|
||||
for device in circuit["devices"]:
|
||||
row = [
|
||||
circuit["id"],
|
||||
circuit["name"],
|
||||
device["id"],
|
||||
device["name"],
|
||||
circuit["parent"],
|
||||
device["mac"],
|
||||
device["ipv4"],
|
||||
device["ipv6"],
|
||||
int(circuit["download"] * 0.98),
|
||||
int(circuit["upload"] * 0.98),
|
||||
int(circuit["download"] * bandwidthOverheadFactor),
|
||||
int(circuit["upload"] * bandwidthOverheadFactor),
|
||||
""
|
||||
]
|
||||
wr.writerow(row)
|
||||
|
||||
def plotNetworkGraph(self, showClients=False):
|
||||
# Requires `pip install graphviz` to function.
|
||||
# You also need to install graphviz on your PC.
|
||||
# In Ubuntu, apt install graphviz will do it.
|
||||
# Plots the network graph to a PDF file, allowing
|
||||
# visual verification that the graph makes sense.
|
||||
# Could potentially be useful in a future
|
||||
# web interface.
|
||||
import importlib.util
|
||||
if (spec := importlib.util.find_spec('graphviz')) is None:
|
||||
return
|
||||
|
||||
import graphviz
|
||||
dot = graphviz.Digraph(
|
||||
'network', comment="Network Graph", engine="fdp")
|
||||
|
||||
for (i, node) in enumerate(self.nodes):
|
||||
if ((node.type != NodeType.client and node.type != NodeType.device) or showClients):
|
||||
color = "white"
|
||||
match node.type:
|
||||
case NodeType.root: color = "green"
|
||||
case NodeType.site: color = "red"
|
||||
case NodeType.ap: color = "blue"
|
||||
case NodeType.clientWithChildren: color = "magenta"
|
||||
case NodeType.device: color = "white"
|
||||
case default: color = "grey"
|
||||
dot.node("N" + str(i), node.displayName, color=color)
|
||||
children = self.findChildIndices(i)
|
||||
for child in children:
|
||||
if child != i:
|
||||
if (self.nodes[child].type != NodeType.client and self.nodes[child].type != NodeType.device) or showClients:
|
||||
dot.edge("N" + str(i), "N" + str(child))
|
||||
|
||||
dot.render("network.pdf")
|
||||
|
||||
@ -1,40 +1,26 @@
|
||||
import requests
|
||||
import os
|
||||
import csv
|
||||
import ipaddress
|
||||
from ispConfig import excludeSites, findIPv6usingMikrotik, bandwidthOverheadFactor, exceptionCPEs, splynx_api_key, splynx_api_secret, splynx_api_url
|
||||
from integrationCommon import isIpv4Permitted
|
||||
import shutil
|
||||
import json
|
||||
import time
|
||||
import base64
|
||||
from requests.auth import HTTPBasicAuth
|
||||
if findIPv6usingMikrotik == True:
|
||||
from mikrotikFindIPv6 import pullMikrotikIPv6
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
|
||||
|
||||
def createShaper():
|
||||
print("Creating ShapedDevices.csv")
|
||||
|
||||
#nonce = round((time.time() * 1000) * 100)
|
||||
#def generate_signature():
|
||||
# key = str(nonce)+splynx_api_key
|
||||
# key_bytes= bytes(key , 'latin-1') # Commonly 'latin-1' or 'ascii'
|
||||
# data_bytes = bytes(splynx_api_secret, 'latin-1') # Assumes `data` is also an ascii string.
|
||||
# return hmac.new(key_bytes, data_bytes , hashlib.sha256).hexdigest()
|
||||
#signature = generate_signature().upper()
|
||||
# Authorization: Splynx-EA (key=<key>&nonce=<nonce>&signature=<signature>)
|
||||
#splynxAuth = 'Splynx-EA (key=' + splynx_api_key + '&nonce=' + str(nonce) + '&signature=' + signature + ')'
|
||||
#headers = {'Authorization' : splynxAuth}
|
||||
def buildHeaders():
|
||||
credentials = splynx_api_key + ':' + splynx_api_secret
|
||||
credentials = base64.b64encode(credentials.encode()).decode()
|
||||
splynxAuth = 'Basic ' + credentials + ''
|
||||
headers = {'Authorization' : "Basic %s" % credentials}
|
||||
|
||||
# Tariffs
|
||||
url = splynx_api_url + "/api/2.0/" + "admin/tariffs/internet"
|
||||
return {'Authorization' : "Basic %s" % credentials}
|
||||
|
||||
def spylnxRequest(target, headers):
|
||||
# Sends a REST GET request to Spylnx and returns the
|
||||
# result in JSON
|
||||
url = splynx_api_url + "/api/2.0/" + target
|
||||
r = requests.get(url, headers=headers)
|
||||
data = r.json()
|
||||
return r.json()
|
||||
|
||||
def getTariffs(headers):
|
||||
data = spylnxRequest("admin/tariffs/internet", headers)
|
||||
tariff = []
|
||||
downloadForTariffID = {}
|
||||
uploadForTariffID = {}
|
||||
@ -44,73 +30,92 @@ def createShaper():
|
||||
speed_upload = round((int(tariff['speed_upload']) / 1000))
|
||||
downloadForTariffID[tariffID] = speed_download
|
||||
uploadForTariffID[tariffID] = speed_upload
|
||||
|
||||
# Customers
|
||||
addressForCustomerID = {}
|
||||
url = splynx_api_url + "/api/2.0/" + "admin/customers/customer"
|
||||
r = requests.get(url, headers=headers)
|
||||
data = r.json()
|
||||
customerIDs = []
|
||||
for customer in data:
|
||||
customerIDs.append(customer['id'])
|
||||
addressForCustomerID[customer['id']] = customer['street_1']
|
||||
|
||||
# Routers
|
||||
return (tariff, downloadForTariffID, uploadForTariffID)
|
||||
|
||||
def getCustomers(headers):
|
||||
data = spylnxRequest("admin/customers/customer", headers)
|
||||
#addressForCustomerID = {}
|
||||
#customerIDs = []
|
||||
#for customer in data:
|
||||
# customerIDs.append(customer['id'])
|
||||
# addressForCustomerID[customer['id']] = customer['street_1']
|
||||
return data
|
||||
|
||||
def getRouters(headers):
|
||||
data = spylnxRequest("admin/networking/routers", headers)
|
||||
ipForRouter = {}
|
||||
url = splynx_api_url + "/api/2.0/" + "admin/networking/routers"
|
||||
r = requests.get(url, headers=headers)
|
||||
data = r.json()
|
||||
for router in data:
|
||||
routerID = router['id']
|
||||
ipForRouter[routerID] = router['ip']
|
||||
|
||||
# Customer services
|
||||
circuits = []
|
||||
for customerID in customerIDs:
|
||||
url = splynx_api_url + "/api/2.0/" + "admin/customers/customer/" + customerID + "/internet-services"
|
||||
r = requests.get(url, headers=headers)
|
||||
data = r.json()
|
||||
for service in data:
|
||||
return ipForRouter
|
||||
|
||||
def combineAddress(json):
|
||||
# Combines address fields into a single string
|
||||
# The API docs seem to indicate that there isn't a "state" field?
|
||||
if json["street_1"]=="" and json["city"]=="" and json["zip_code"]=="":
|
||||
return json["id"] + "/" + json["name"]
|
||||
else:
|
||||
return json["street_1"] + " " + json["city"] + " " + json["zip_code"]
|
||||
|
||||
def createShaper():
|
||||
net = NetworkGraph()
|
||||
|
||||
print("Fetching data from Spylnx")
|
||||
headers = buildHeaders()
|
||||
tariff, downloadForTariffID, uploadForTariffID = getTariffs(headers)
|
||||
customers = getCustomers(headers)
|
||||
ipForRouter = getRouters(headers)
|
||||
|
||||
# It's not very clear how a service is meant to handle multiple
|
||||
# devices on a shared tariff. Creating each service as a combined
|
||||
# entity including the customer, to be on the safe side.
|
||||
for customerJson in customers:
|
||||
services = spylnxRequest("admin/customers/customer/" + customerJson["id"] + "/internet-services", headers)
|
||||
for serviceJson in services:
|
||||
combinedId = "c_" + str(customerJson["id"]) + "_s_" + str(serviceJson["id"])
|
||||
tariff_id = serviceJson['tariff_id']
|
||||
customer = NetworkNode(
|
||||
type=NodeType.client,
|
||||
id=combinedId,
|
||||
displayName=customerJson["name"],
|
||||
address=combineAddress(customerJson),
|
||||
download=downloadForTariffID[tariff_id],
|
||||
upload=uploadForTariffID[tariff_id],
|
||||
)
|
||||
net.addRawNode(customer)
|
||||
|
||||
ipv4 = ''
|
||||
ipv6 = ''
|
||||
routerID = service['router_id']
|
||||
routerID = serviceJson['router_id']
|
||||
# If not "Taking IPv4" (Router will assign IP), then use router's set IP
|
||||
if service['taking_ipv4'] == 0:
|
||||
if serviceJson['taking_ipv4'] == 0:
|
||||
ipv4 = ipForRouter[routerID]
|
||||
elif service['taking_ipv4'] == 1:
|
||||
ipv4 = service['ipv4']
|
||||
elif serviceJson['taking_ipv4'] == 1:
|
||||
ipv4 = serviceJson['ipv4']
|
||||
# If not "Taking IPv6" (Router will assign IP), then use router's set IP
|
||||
if service['taking_ipv6'] == 0:
|
||||
if serviceJson['taking_ipv6'] == 0:
|
||||
ipv6 = ''
|
||||
elif service['taking_ipv6'] == 1:
|
||||
ipv6 = service['ipv6']
|
||||
serviceID = service['id']
|
||||
tariff_id = service['tariff_id']
|
||||
mac = service['mac']
|
||||
dlMbps = downloadForTariffID[tariff_id]
|
||||
ulMbps = uploadForTariffID[tariff_id]
|
||||
address = addressForCustomerID[customerID]
|
||||
circuit = {
|
||||
'circuitID': serviceID,
|
||||
'circuitName': address,
|
||||
'deviceID': routerID,
|
||||
'deviceName': '',
|
||||
'parentNode': '',
|
||||
"mac": mac,
|
||||
'ipv4': ipv4,
|
||||
'ipv6': ipv6,
|
||||
'minDownload': round(dlMbps*.98),
|
||||
'minUpload': round(ulMbps*.98),
|
||||
'maxDownload': dlMbps,
|
||||
'maxUpload': ulMbps
|
||||
}
|
||||
circuits.append(circuit)
|
||||
|
||||
with open('ShapedDevices.csv', 'w') as csvfile:
|
||||
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
|
||||
wr.writerow(['Circuit ID', 'Circuit Name', 'Device ID', 'Device Name', 'Parent Node', 'MAC', 'IPv4', 'IPv6', 'Download Min', 'Upload Min', 'Download Max', 'Upload Max', 'Comment'])
|
||||
for circuit in circuits:
|
||||
wr.writerow((circuit['circuitID'], circuit['circuitName'], circuit['deviceID'], circuit['deviceName'], circuit['parentNode'], circuit['mac'], circuit['ipv4'], circuit['ipv6'], circuit['minDownload'], circuit['minUpload'], circuit['maxDownload'], circuit['maxUpload'], ''))
|
||||
elif serviceJson['taking_ipv6'] == 1:
|
||||
ipv6 = serviceJson['ipv6']
|
||||
|
||||
device = NetworkNode(
|
||||
id=combinedId+"_d" + str(serviceJson["id"]),
|
||||
displayName=serviceJson["description"],
|
||||
type=NodeType.device,
|
||||
parentId=combinedId,
|
||||
mac=serviceJson["mac"],
|
||||
ipv4=[ipv4],
|
||||
ipv6=[ipv6]
|
||||
)
|
||||
net.addRawNode(device)
|
||||
|
||||
net.prepareTree()
|
||||
net.plotNetworkGraph(False)
|
||||
if net.doesNetworkJsonExist():
|
||||
print("network.json already exists. Leaving in-place.")
|
||||
else:
|
||||
net.createNetworkJson()
|
||||
net.createShapedDevices()
|
||||
|
||||
def importFromSplynx():
|
||||
#createNetworkJSON()
|
||||
|
||||
@ -1,267 +1,216 @@
|
||||
import requests
|
||||
import os
|
||||
import csv
|
||||
import ipaddress
|
||||
from ispConfig import UISPbaseURL, uispAuthToken, excludeSites, findIPv6usingMikrotik, bandwidthOverheadFactor, exceptionCPEs
|
||||
from integrationCommon import isIpv4Permitted
|
||||
import shutil
|
||||
import json
|
||||
if findIPv6usingMikrotik == True:
|
||||
from mikrotikFindIPv6 import pullMikrotikIPv6
|
||||
from ispConfig import uispSite, uispStrategy
|
||||
from integrationCommon import isIpv4Permitted, fixSubnet
|
||||
|
||||
knownRouterModels = ['ACB-AC', 'ACB-ISP']
|
||||
knownAPmodels = ['LTU-Rocket', 'RP-5AC', 'RP-5AC-Gen2', 'LAP-GPS', 'Wave-AP']
|
||||
|
||||
def createTree(sites,accessPoints,bandwidthDL,bandwidthUL,siteParentDict,siteIDtoName,sitesWithParents,currentNode):
|
||||
currentNodeName = list(currentNode.items())[0][0]
|
||||
childrenList = []
|
||||
for site in sites:
|
||||
try:
|
||||
thisOnesParent = siteIDtoName[site['identification']['parent']['id']]
|
||||
if thisOnesParent == currentNodeName:
|
||||
childrenList.append(site['id'])
|
||||
except:
|
||||
thisOnesParent = None
|
||||
aps = []
|
||||
for ap in accessPoints:
|
||||
if ap['device']['site'] is None:
|
||||
print("Unable to read site information for: " + ap['device']['name'])
|
||||
else:
|
||||
thisOnesParent = ap['device']['site']['name']
|
||||
if thisOnesParent == currentNodeName:
|
||||
if ap['device']['model'] in knownAPmodels:
|
||||
aps.append(ap['device']['name'])
|
||||
apDict = {}
|
||||
for ap in aps:
|
||||
maxDL = min(bandwidthDL[ap],bandwidthDL[currentNodeName])
|
||||
maxUL = min(bandwidthUL[ap],bandwidthUL[currentNodeName])
|
||||
apStruct = {
|
||||
ap :
|
||||
{
|
||||
"downloadBandwidthMbps": maxDL,
|
||||
"uploadBandwidthMbps": maxUL,
|
||||
}
|
||||
}
|
||||
apDictNew = apDict | apStruct
|
||||
apDict = apDictNew
|
||||
if bool(apDict):
|
||||
currentNode[currentNodeName]['children'] = apDict
|
||||
counter = 0
|
||||
tempChildren = {}
|
||||
for child in childrenList:
|
||||
name = siteIDtoName[child]
|
||||
maxDL = min(bandwidthDL[name],bandwidthDL[currentNodeName])
|
||||
maxUL = min(bandwidthUL[name],bandwidthUL[currentNodeName])
|
||||
childStruct = {
|
||||
name :
|
||||
{
|
||||
"downloadBandwidthMbps": maxDL,
|
||||
"uploadBandwidthMbps": maxUL,
|
||||
}
|
||||
}
|
||||
childStruct = createTree(sites,accessPoints,bandwidthDL,bandwidthUL,siteParentDict,siteIDtoName,sitesWithParents,childStruct)
|
||||
tempChildren = tempChildren | childStruct
|
||||
counter += 1
|
||||
if tempChildren != {}:
|
||||
if 'children' in currentNode[currentNodeName]:
|
||||
currentNode[currentNodeName]['children'] = currentNode[currentNodeName]['children'] | tempChildren
|
||||
else:
|
||||
currentNode[currentNodeName]['children'] = tempChildren
|
||||
return currentNode
|
||||
def uispRequest(target):
|
||||
# Sends an HTTP request to UISP and returns the
|
||||
# result in JSON. You only need to specify the
|
||||
# tail end of the URL, e.g. "sites"
|
||||
from ispConfig import UISPbaseURL, uispAuthToken
|
||||
url = UISPbaseURL + "/nms/api/v2.1/" + target
|
||||
headers = {'accept': 'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
return r.json()
|
||||
|
||||
def createNetworkJSON():
|
||||
if os.path.isfile("network.json"):
|
||||
print("network.json already exists. Leaving in place.")
|
||||
else:
|
||||
print("Generating network.json")
|
||||
bandwidthDL = {}
|
||||
bandwidthUL = {}
|
||||
url = UISPbaseURL + "/nms/api/v2.1/sites?type=site"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
sites = r.json()
|
||||
url = UISPbaseURL + "/nms/api/v2.1/devices/aps/profiles"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
apProfiles = r.json()
|
||||
listOfTopLevelParentNodes = []
|
||||
if os.path.isfile("integrationUISPbandwidths.csv"):
|
||||
with open('integrationUISPbandwidths.csv') as csv_file:
|
||||
csv_reader = csv.reader(csv_file, delimiter=',')
|
||||
next(csv_reader)
|
||||
for row in csv_reader:
|
||||
name, download, upload = row
|
||||
download = int(download)
|
||||
upload = int(upload)
|
||||
listOfTopLevelParentNodes.append(name)
|
||||
bandwidthDL[name] = download
|
||||
bandwidthUL[name] = upload
|
||||
for ap in apProfiles:
|
||||
name = ap['device']['name']
|
||||
model = ap['device']['model']
|
||||
apID = ap['device']['id']
|
||||
if model in knownAPmodels:
|
||||
url = UISPbaseURL + "/nms/api/v2.1/devices/airmaxes/" + apID + '?withStations=false'
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
thisAPairmax = r.json()
|
||||
# See if there is a reported capacity of the AP we can use. If not, use a default (1000 Mbit)
|
||||
defaultCap = 1000
|
||||
if ('downlinkCapacity' in thisAPairmax['overview']) and ('uplinkCapacity' in thisAPairmax['overview']):
|
||||
if thisAPairmax['overview']['downlinkCapacity'] != None:
|
||||
downloadCap = int(round(thisAPairmax['overview']['downlinkCapacity']/1000000))
|
||||
uploadCap = int(round(thisAPairmax['overview']['uplinkCapacity']/1000000))
|
||||
else:
|
||||
downloadCap = defaultCap
|
||||
uploadCap = defaultCap
|
||||
else:
|
||||
downloadCap = defaultCap
|
||||
uploadCap = defaultCap
|
||||
# If operator already included bandwidth definitions for this ParentNode, do not overwrite what they set
|
||||
if name not in listOfTopLevelParentNodes:
|
||||
print("Found " + name)
|
||||
listOfTopLevelParentNodes.append(name)
|
||||
bandwidthDL[name] = downloadCap
|
||||
bandwidthUL[name] = uploadCap
|
||||
for site in sites:
|
||||
name = site['identification']['name']
|
||||
if name not in excludeSites:
|
||||
# If operator already included bandwidth definitions for this ParentNode, do not overwrite what they set
|
||||
if name not in listOfTopLevelParentNodes:
|
||||
print("Found " + name)
|
||||
listOfTopLevelParentNodes.append(name)
|
||||
bandwidthDL[name] = 1000
|
||||
bandwidthUL[name] = 1000
|
||||
with open('integrationUISPbandwidths.csv', 'w') as csvfile:
|
||||
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
|
||||
wr.writerow(['ParentNode', 'Download Mbps', 'Upload Mbps'])
|
||||
for device in listOfTopLevelParentNodes:
|
||||
entry = (device, bandwidthDL[device], bandwidthUL[device])
|
||||
wr.writerow(entry)
|
||||
url = UISPbaseURL + "/nms/api/v2.1/devices?role=ap"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
accessPoints = r.json()
|
||||
siteIDtoName = {}
|
||||
siteParentDict = {}
|
||||
sitesWithParents = []
|
||||
topLevelSites = []
|
||||
for site in sites:
|
||||
siteIDtoName[site['id']] = site['identification']['name']
|
||||
try:
|
||||
siteParentDict[site['id']] = site['identification']['parent']['id']
|
||||
sitesWithParents.append(site['id'])
|
||||
except:
|
||||
siteParentDict[site['id']] = None
|
||||
if site['identification']['name'] not in excludeSites:
|
||||
topLevelSites.append(site['id'])
|
||||
tLname = siteIDtoName[topLevelSites.pop()]
|
||||
topLevelNode = {
|
||||
tLname :
|
||||
{
|
||||
"downloadBandwidthMbps": bandwidthDL[tLname],
|
||||
"uploadBandwidthMbps": bandwidthUL[tLname],
|
||||
}
|
||||
}
|
||||
tree = createTree(sites,apProfiles, bandwidthDL, bandwidthUL, siteParentDict,siteIDtoName,sitesWithParents,topLevelNode)
|
||||
with open('network.json', 'w') as f:
|
||||
json.dump(tree, f, indent=4)
|
||||
def buildFlatGraph():
|
||||
# Builds a high-performance (but lacking in site or AP bandwidth control)
|
||||
# network.
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
from ispConfig import generatedPNUploadMbps, generatedPNDownloadMbps
|
||||
|
||||
# Load network sites
|
||||
print("Loading Data from UISP")
|
||||
sites = uispRequest("sites")
|
||||
devices = uispRequest("devices?withInterfaces=true&authorized=true")
|
||||
|
||||
# Build a basic network adding every client to the tree
|
||||
print("Building Flat Topology")
|
||||
net = NetworkGraph()
|
||||
|
||||
for site in sites:
|
||||
type = site['identification']['type']
|
||||
if type == "endpoint":
|
||||
id = site['identification']['id']
|
||||
address = site['description']['address']
|
||||
name = site['identification']['name']
|
||||
type = site['identification']['type']
|
||||
download = generatedPNDownloadMbps
|
||||
upload = generatedPNUploadMbps
|
||||
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
|
||||
download = int(round(site['qos']['downloadSpeed']/1000000))
|
||||
upload = int(round(site['qos']['uploadSpeed']/1000000))
|
||||
|
||||
node = NetworkNode(id=id, displayName=name, type=NodeType.client, download=download, upload=upload, address=address)
|
||||
net.addRawNode(node)
|
||||
for device in devices:
|
||||
if device['identification']['site'] is not None and device['identification']['site']['id'] == id:
|
||||
# The device is at this site, so add it
|
||||
ipv4 = []
|
||||
ipv6 = []
|
||||
|
||||
for interface in device["interfaces"]:
|
||||
for ip in interface["addresses"]:
|
||||
ip = ip["cidr"]
|
||||
if isIpv4Permitted(ip):
|
||||
ipv4.append(fixSubnet(ip))
|
||||
|
||||
# TODO: Figure out Mikrotik IPv6?
|
||||
mac = device['identification']['mac']
|
||||
|
||||
net.addRawNode(NetworkNode(id=device['identification']['id'], displayName=device['identification']
|
||||
['name'], parentId=id, type=NodeType.device, ipv4=ipv4, ipv6=ipv6, mac=mac))
|
||||
|
||||
# Finish up
|
||||
net.prepareTree()
|
||||
net.plotNetworkGraph(False)
|
||||
if net.doesNetworkJsonExist():
|
||||
print("network.json already exists. Leaving in-place.")
|
||||
else:
|
||||
net.createNetworkJson()
|
||||
net.createShapedDevices()
|
||||
|
||||
def buildFullGraph():
|
||||
# Attempts to build a full network graph, incorporating as much of the UISP
|
||||
# hierarchy as possible.
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
from ispConfig import generatedPNUploadMbps, generatedPNDownloadMbps
|
||||
|
||||
# Load network sites
|
||||
print("Loading Data from UISP")
|
||||
sites = uispRequest("sites")
|
||||
devices = uispRequest("devices?withInterfaces=true&authorized=true")
|
||||
dataLinks = uispRequest("data-links?siteLinksOnly=true")
|
||||
|
||||
# Do we already have a integrationUISPbandwidths.csv file?
|
||||
siteBandwidth = {}
|
||||
if os.path.isfile("integrationUISPbandwidths.csv"):
|
||||
with open('integrationUISPbandwidths.csv') as csv_file:
|
||||
csv_reader = csv.reader(csv_file, delimiter=',')
|
||||
next(csv_reader)
|
||||
for row in csv_reader:
|
||||
name, download, upload = row
|
||||
download = int(download)
|
||||
upload = int(upload)
|
||||
siteBandwidth[name] = {"download": download, "upload": upload}
|
||||
|
||||
# Find AP capacities from UISP
|
||||
for device in devices:
|
||||
if device['identification']['role'] == "ap":
|
||||
name = device['identification']['name']
|
||||
if not name in siteBandwidth and device['overview']['downlinkCapacity'] and device['overview']['uplinkCapacity']:
|
||||
download = int(device['overview']
|
||||
['downlinkCapacity'] / 1000000)
|
||||
upload = int(device['overview']['uplinkCapacity'] / 1000000)
|
||||
siteBandwidth[device['identification']['name']] = {
|
||||
"download": download, "upload": upload}
|
||||
|
||||
print("Building Topology")
|
||||
net = NetworkGraph()
|
||||
# Add all sites and client sites
|
||||
for site in sites:
|
||||
id = site['identification']['id']
|
||||
name = site['identification']['name']
|
||||
type = site['identification']['type']
|
||||
download = generatedPNDownloadMbps
|
||||
upload = generatedPNUploadMbps
|
||||
address = ""
|
||||
if site['identification']['parent'] is None:
|
||||
parent = ""
|
||||
else:
|
||||
parent = site['identification']['parent']['id']
|
||||
match type:
|
||||
case "site":
|
||||
nodeType = NodeType.site
|
||||
if name in siteBandwidth:
|
||||
# Use the CSV bandwidth values
|
||||
download = siteBandwidth[name]["download"]
|
||||
upload = siteBandwidth[name]["upload"]
|
||||
else:
|
||||
# Add them just in case
|
||||
siteBandwidth[name] = {
|
||||
"download": download, "upload": upload}
|
||||
case default:
|
||||
nodeType = NodeType.client
|
||||
address = site['description']['address']
|
||||
if (site['qos']['downloadSpeed']) and (site['qos']['uploadSpeed']):
|
||||
download = int(round(site['qos']['downloadSpeed']/1000000))
|
||||
upload = int(round(site['qos']['uploadSpeed']/1000000))
|
||||
|
||||
node = NetworkNode(id=id, displayName=name, type=nodeType,
|
||||
parentId=parent, download=download, upload=upload, address=address)
|
||||
# If this is the uispSite node, it becomes the root. Otherwise, add it to the
|
||||
# node soup.
|
||||
if name == uispSite:
|
||||
net.replaceRootNote(node)
|
||||
else:
|
||||
net.addRawNode(node)
|
||||
|
||||
for device in devices:
|
||||
if device['identification']['site'] is not None and device['identification']['site']['id'] == id:
|
||||
# The device is at this site, so add it
|
||||
ipv4 = []
|
||||
ipv6 = []
|
||||
|
||||
for interface in device["interfaces"]:
|
||||
for ip in interface["addresses"]:
|
||||
ip = ip["cidr"]
|
||||
if isIpv4Permitted(ip):
|
||||
ipv4.append(fixSubnet(ip))
|
||||
|
||||
# TODO: Figure out Mikrotik IPv6?
|
||||
mac = device['identification']['mac']
|
||||
|
||||
net.addRawNode(NetworkNode(id=device['identification']['id'], displayName=device['identification']
|
||||
['name'], parentId=id, type=NodeType.device, ipv4=ipv4, ipv6=ipv6, mac=mac))
|
||||
|
||||
# Now iterate access points, and look for connections to sites
|
||||
for node in net.nodes:
|
||||
if node.type == NodeType.device:
|
||||
for dl in dataLinks:
|
||||
if dl['from']['device'] is not None and dl['from']['device']['identification']['id'] == node.id:
|
||||
if dl['to']['site'] is not None and dl['from']['site']['identification']['id'] != dl['to']['site']['identification']['id']:
|
||||
target = net.findNodeIndexById(
|
||||
dl['to']['site']['identification']['id'])
|
||||
if target > -1:
|
||||
# We found the site
|
||||
if net.nodes[target].type == NodeType.client or net.nodes[target].type == NodeType.clientWithChildren:
|
||||
net.nodes[target].parentId = node.id
|
||||
node.type = NodeType.ap
|
||||
if node.displayName in siteBandwidth:
|
||||
# Use the bandwidth numbers from the CSV file
|
||||
node.uploadMbps = siteBandwidth[node.displayName]["upload"]
|
||||
node.downloadMbps = siteBandwidth[node.displayName]["download"]
|
||||
else:
|
||||
# Add some defaults in case they want to change them
|
||||
siteBandwidth[node.displayName] = {
|
||||
"download": generatedPNDownloadMbps, "upload": generatedPNUploadMbps}
|
||||
|
||||
net.prepareTree()
|
||||
net.plotNetworkGraph(False)
|
||||
if net.doesNetworkJsonExist():
|
||||
print("network.json already exists. Leaving in-place.")
|
||||
else:
|
||||
net.createNetworkJson()
|
||||
net.createShapedDevices()
|
||||
|
||||
# Save integrationUISPbandwidths.csv
|
||||
# (the newLine fixes generating extra blank lines)
|
||||
with open('integrationUISPbandwidths.csv', 'w', newline='') as csvfile:
|
||||
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
|
||||
wr.writerow(['ParentNode', 'Download Mbps', 'Upload Mbps'])
|
||||
for device in siteBandwidth:
|
||||
entry = (
|
||||
device, siteBandwidth[device]["download"], siteBandwidth[device]["upload"])
|
||||
wr.writerow(entry)
|
||||
|
||||
def createShaper():
|
||||
print("Creating ShapedDevices.csv")
|
||||
devicesToImport = []
|
||||
url = UISPbaseURL + "/nms/api/v2.1/sites?type=site"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
sites = r.json()
|
||||
siteIDtoName = {}
|
||||
for site in sites:
|
||||
siteIDtoName[site['id']] = site['identification']['name']
|
||||
url = UISPbaseURL + "/nms/api/v2.1/sites?type=client&ucrm=true&ucrmDetails=true"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
clientSites = r.json()
|
||||
url = UISPbaseURL + "/nms/api/v2.1/devices"
|
||||
headers = {'accept':'application/json', 'x-auth-token': uispAuthToken}
|
||||
r = requests.get(url, headers=headers)
|
||||
allDevices = r.json()
|
||||
ipv4ToIPv6 = {}
|
||||
if findIPv6usingMikrotik:
|
||||
ipv4ToIPv6 = pullMikrotikIPv6()
|
||||
for uispClientSite in clientSites:
|
||||
#if (uispClientSite['identification']['status'] == 'active') and (uispClientSite['identification']['suspended'] == False):
|
||||
if (uispClientSite['identification']['suspended'] == False):
|
||||
foundCPEforThisClientSite = False
|
||||
if (uispClientSite['qos']['downloadSpeed']) and (uispClientSite['qos']['uploadSpeed']):
|
||||
downloadSpeedMbps = int(round(uispClientSite['qos']['downloadSpeed']/1000000))
|
||||
uploadSpeedMbps = int(round(uispClientSite['qos']['uploadSpeed']/1000000))
|
||||
address = uispClientSite['description']['address']
|
||||
uispClientSiteID = uispClientSite['id']
|
||||
|
||||
UCRMclientID = uispClientSite['ucrm']['client']['id']
|
||||
siteName = uispClientSite['identification']['name']
|
||||
AP = 'none'
|
||||
thisSiteDevices = []
|
||||
#Look for station devices, use those to find AP name
|
||||
for device in allDevices:
|
||||
if device['identification']['site'] != None:
|
||||
if device['identification']['site']['id'] == uispClientSite['id']:
|
||||
deviceName = device['identification']['name']
|
||||
deviceRole = device['identification']['role']
|
||||
deviceModel = device['identification']['model']
|
||||
deviceModelName = device['identification']['modelName']
|
||||
if (deviceRole == 'station'):
|
||||
if device['attributes']['apDevice']:
|
||||
AP = device['attributes']['apDevice']['name']
|
||||
#Look for router devices, use those as shaped CPE
|
||||
for device in allDevices:
|
||||
if device['identification']['site'] != None:
|
||||
if device['identification']['site']['id'] == uispClientSite['id']:
|
||||
deviceModel = device['identification']['model']
|
||||
deviceName = device['identification']['name']
|
||||
deviceRole = device['identification']['role']
|
||||
if device['identification']['mac']:
|
||||
deviceMAC = device['identification']['mac'].upper()
|
||||
else:
|
||||
deviceMAC = ''
|
||||
if (deviceRole == 'router') or (deviceModel in knownRouterModels):
|
||||
ipv4 = device['ipAddress']
|
||||
if '/' in ipv4:
|
||||
ipv4 = ipv4.split("/")[0]
|
||||
ipv6 = ''
|
||||
if ipv4 in ipv4ToIPv6.keys():
|
||||
ipv6 = ipv4ToIPv6[ipv4]
|
||||
if isIpv4Permitted(ipv4):
|
||||
deviceModel = device['identification']['model']
|
||||
deviceModelName = device['identification']['modelName']
|
||||
maxSpeedDown = round(bandwidthOverheadFactor*downloadSpeedMbps)
|
||||
maxSpeedUp = round(bandwidthOverheadFactor*uploadSpeedMbps)
|
||||
minSpeedDown = min(round(maxSpeedDown*.98),maxSpeedDown)
|
||||
minSpeedUp = min(round(maxSpeedUp*.98),maxSpeedUp)
|
||||
#Customers directly connected to Sites
|
||||
if deviceName in exceptionCPEs.keys():
|
||||
AP = exceptionCPEs[deviceName]
|
||||
if AP == 'none':
|
||||
try:
|
||||
AP = siteIDtoName[uispClientSite['identification']['parent']['id']]
|
||||
except:
|
||||
AP = 'none'
|
||||
devicesToImport.append((uispClientSiteID, address, '', deviceName, AP, deviceMAC, ipv4, ipv6, str(minSpeedDown), str(minSpeedUp), str(maxSpeedDown),str(maxSpeedUp),''))
|
||||
foundCPEforThisClientSite = True
|
||||
else:
|
||||
print("Failed to import devices from " + uispClientSite['description']['address'] + ". Missing QoS.")
|
||||
if foundCPEforThisClientSite != True:
|
||||
print("Failed to import devices for " + uispClientSite['description']['address'])
|
||||
|
||||
with open('ShapedDevices.csv', 'w') as csvfile:
|
||||
wr = csv.writer(csvfile, quoting=csv.QUOTE_ALL)
|
||||
wr.writerow(['Circuit ID', 'Circuit Name', 'Device ID', 'Device Name', 'Parent Node', 'MAC', 'IPv4', 'IPv6', 'Download Min', 'Upload Min', 'Download Max', 'Upload Max', 'Comment'])
|
||||
for device in devicesToImport:
|
||||
wr.writerow(device)
|
||||
|
||||
def importFromUISP():
|
||||
createNetworkJSON()
|
||||
createShaper()
|
||||
match uispStrategy:
|
||||
case "full": buildFullGraph()
|
||||
case default: buildFlatGraph()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
importFromUISP()
|
||||
importFromUISP()
|
||||
|
||||
@ -65,6 +65,12 @@ automaticImportUISP = False
|
||||
uispAuthToken = ''
|
||||
# Everything before /nms/ on your UISP instance
|
||||
UISPbaseURL = 'https://examplesite.com'
|
||||
# Strategy:
|
||||
# * "flat" - create all client sites directly off the top of the tree,
|
||||
# provides maximum performance - at the expense of not offering AP,
|
||||
# or site options.
|
||||
# * "full" - build a complete network map
|
||||
uispStrategy = "full"
|
||||
# List any sites that should not be included, with each site name surrounded by '' and seperated by commas
|
||||
excludeSites = []
|
||||
# If you use IPv6, this can be used to find associated IPv6 prefixes for your clients' IPv4 addresses, and match them to those devices
|
||||
|
||||
333
v1.3/testGraph.py
Normal file
333
v1.3/testGraph.py
Normal file
@ -0,0 +1,333 @@
|
||||
import unittest
|
||||
|
||||
class TestGraph(unittest.TestCase):
|
||||
def test_empty_graph(self):
|
||||
"""
|
||||
Test instantiation of the graph type
|
||||
"""
|
||||
from integrationCommon import NetworkGraph
|
||||
graph = NetworkGraph()
|
||||
self.assertEqual(len(graph.nodes), 1) # There is an automatic root entry
|
||||
self.assertEqual(graph.nodes[0].id, "FakeRoot")
|
||||
|
||||
def test_empty_node(self):
|
||||
"""
|
||||
Test instantiation of the GraphNode type
|
||||
"""
|
||||
from integrationCommon import NetworkNode, NodeType
|
||||
node = NetworkNode("test")
|
||||
self.assertEqual(node.type.value, NodeType.site.value)
|
||||
self.assertEqual(node.id, "test")
|
||||
self.assertEqual(node.parentIndex, 0)
|
||||
|
||||
def test_node_types(self):
|
||||
"""
|
||||
Test that the NodeType enum is working
|
||||
"""
|
||||
from integrationCommon import NetworkNode, NodeType
|
||||
node = NetworkNode("Test", type = NodeType.root)
|
||||
self.assertEqual(node.type.value, NodeType.root.value)
|
||||
node = NetworkNode("Test", type = NodeType.site)
|
||||
self.assertEqual(node.type.value, NodeType.site.value)
|
||||
node = NetworkNode("Test", type = NodeType.ap)
|
||||
self.assertEqual(node.type.value, NodeType.ap.value)
|
||||
node = NetworkNode("Test", type = NodeType.client)
|
||||
self.assertEqual(node.type.value, NodeType.client.value)
|
||||
|
||||
def test_add_raw_node(self):
|
||||
"""
|
||||
Adds a single node to a graph to ensure add works
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site"))
|
||||
self.assertEqual(len(graph.nodes), 2)
|
||||
self.assertEqual(graph.nodes[1].type.value, NodeType.site.value)
|
||||
self.assertEqual(graph.nodes[1].parentIndex, 0)
|
||||
self.assertEqual(graph.nodes[1].id, "Site")
|
||||
|
||||
def test_replace_root(self):
|
||||
"""
|
||||
Test replacing the default root node with a specified node
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
node = NetworkNode("Test", type = NodeType.site)
|
||||
graph.replaceRootNote(node)
|
||||
self.assertEqual(graph.nodes[0].id, "Test")
|
||||
|
||||
def add_child_by_named_parent(self):
|
||||
"""
|
||||
Tests inserting a node with a named parent
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site"))
|
||||
graph.addNodeAsChild("site", NetworkNode("Client", type = NodeType.client))
|
||||
self.assertEqual(len(graph.nodes), 3)
|
||||
self.assertEqual(graph.nodes[2].parentIndex, 1)
|
||||
self.assertEqual(graph.nodes[0].parentIndex, 0)
|
||||
|
||||
def test_reparent_by_name(self):
|
||||
"""
|
||||
Tests that re-parenting a tree by name is functional
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Missing Site", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
self.assertEqual(len(graph.nodes), 7) # Includes 1 for the fake root
|
||||
self.assertEqual(graph.nodes[1].parentIndex, 0) # Site 1 is off root
|
||||
self.assertEqual(graph.nodes[2].parentIndex, 0) # Site 2 is off root
|
||||
self.assertEqual(graph.nodes[3].parentIndex, 1) # Client 1 found Site 1
|
||||
self.assertEqual(graph.nodes[4].parentIndex, 1) # Client 2 found Site 1
|
||||
self.assertEqual(graph.nodes[5].parentIndex, 2) # Client 3 found Site 2
|
||||
self.assertEqual(graph.nodes[6].parentIndex, 0) # Client 4 didn't find "Missing Site" and goes to root
|
||||
|
||||
def test_find_by_id(self):
|
||||
"""
|
||||
Tests that finding a node by name succeeds or fails
|
||||
as expected.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
self.assertEqual(graph.findNodeIndexById("Site 1"), -1) # Test failure
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
self.assertEqual(graph.findNodeIndexById("Site 1"), 1) # Test success
|
||||
|
||||
def test_find_by_name(self):
|
||||
"""
|
||||
Tests that finding a node by name succeeds or fails
|
||||
as expected.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
self.assertEqual(graph.findNodeIndexByName("Site 1"), -1) # Test failure
|
||||
graph.addRawNode(NetworkNode("Site 1", "Site X"))
|
||||
self.assertEqual(graph.findNodeIndexByName("Site X"), 1) # Test success
|
||||
|
||||
def test_find_children(self):
|
||||
"""
|
||||
Tests that finding children in the tree works,
|
||||
both for full and empty cases.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Missing Site", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
self.assertEqual(graph.findChildIndices(1), [3, 4])
|
||||
self.assertEqual(graph.findChildIndices(2), [5])
|
||||
self.assertEqual(graph.findChildIndices(3), [])
|
||||
|
||||
def test_clients_with_children(self):
|
||||
"""
|
||||
Tests handling cases where a client site
|
||||
itself has children. This is only useful for
|
||||
relays where a site hasn't been created in the
|
||||
middle - but it allows us to graph the more
|
||||
pathological designs people come up with.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
graph._NetworkGraph__promoteClientsWithChildren()
|
||||
self.assertEqual(graph.nodes[5].type, NodeType.clientWithChildren)
|
||||
self.assertEqual(graph.nodes[6].type, NodeType.client) # Test that a client is still a client
|
||||
|
||||
def test_client_with_children_promotion(self):
|
||||
"""
|
||||
Test locating a client site with children, and then promoting it to
|
||||
create a generated site
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
graph._NetworkGraph__promoteClientsWithChildren()
|
||||
graph._NetworkGraph__clientsWithChildrenToSites()
|
||||
self.assertEqual(graph.nodes[5].type, NodeType.client)
|
||||
self.assertEqual(graph.nodes[6].type, NodeType.client) # Test that a client is still a client
|
||||
self.assertEqual(graph.nodes[7].type, NodeType.site)
|
||||
self.assertEqual(graph.nodes[7].id, "Client 3_gen")
|
||||
|
||||
def test_find_unconnected(self):
|
||||
"""
|
||||
Tests traversing a tree and finding nodes that
|
||||
have no connection to the rest of the tree.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
graph._NetworkGraph__promoteClientsWithChildren()
|
||||
graph.nodes[6].parentIndex = 6 # Create a circle
|
||||
unconnected = graph._NetworkGraph__findUnconnectedNodes()
|
||||
self.assertEqual(len(unconnected), 1)
|
||||
self.assertEqual(unconnected[0], 6)
|
||||
self.assertEqual(graph.nodes[unconnected[0]].id, "Client 4")
|
||||
|
||||
def test_reconnect_unconnected(self):
|
||||
"""
|
||||
Tests traversing a tree and finding nodes that
|
||||
have no connection to the rest of the tree.
|
||||
Reconnects them and ensures that the orphan is now
|
||||
parented.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
graph = NetworkGraph()
|
||||
graph.addRawNode(NetworkNode("Site 1"))
|
||||
graph.addRawNode(NetworkNode("Site 2"))
|
||||
graph.addRawNode(NetworkNode("Client 1", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 2", parentId="Site 1", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 3", parentId="Site 2", type=NodeType.client))
|
||||
graph.addRawNode(NetworkNode("Client 4", parentId="Client 3", type=NodeType.client))
|
||||
graph._NetworkGraph__reparentById()
|
||||
graph._NetworkGraph__promoteClientsWithChildren()
|
||||
graph.nodes[6].parentIndex = 6 # Create a circle
|
||||
graph._NetworkGraph__reconnectUnconnected()
|
||||
unconnected = graph._NetworkGraph__findUnconnectedNodes()
|
||||
self.assertEqual(len(unconnected), 0)
|
||||
self.assertEqual(graph.nodes[6].parentIndex, 0)
|
||||
|
||||
def test_network_json_exists(self):
|
||||
from integrationCommon import NetworkGraph
|
||||
import os
|
||||
if os.path.exists("network.json"):
|
||||
os.remove("network.json")
|
||||
graph = NetworkGraph()
|
||||
self.assertEqual(graph.doesNetworkJsonExist(), False)
|
||||
with open('network.json', 'w') as f:
|
||||
f.write('Dummy')
|
||||
self.assertEqual(graph.doesNetworkJsonExist(), True)
|
||||
os.remove("network.json")
|
||||
|
||||
def test_network_json_example(self):
|
||||
"""
|
||||
Rebuilds the network in network.example.json
|
||||
and makes sure that it matches.
|
||||
Should serve as an example for how an integration
|
||||
can build a functional tree.
|
||||
"""
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
import json
|
||||
net = NetworkGraph()
|
||||
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
|
||||
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
|
||||
net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, 500, 500))
|
||||
net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType.site, 500, 500))
|
||||
net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType.site, 200, 200))
|
||||
net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, 120, 120))
|
||||
net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType.site, 60, 60))
|
||||
net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, 30, 30))
|
||||
net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType.site, 200, 200))
|
||||
net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, 100, 100))
|
||||
net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, 150, 150))
|
||||
net.prepareTree()
|
||||
net.createNetworkJson()
|
||||
with open('network.json') as file:
|
||||
newFile = json.load(file)
|
||||
with open('v1.3/network.example.json') as file:
|
||||
exampleFile = json.load(file)
|
||||
self.assertEqual(newFile, exampleFile)
|
||||
|
||||
def test_ipv4_to_ipv6_map(self):
|
||||
"""
|
||||
Tests the underlying functionality of finding an IPv6 address from an IPv4 mapping
|
||||
"""
|
||||
from integrationCommon import NetworkGraph
|
||||
net = NetworkGraph()
|
||||
ipv4 = [ "100.64.1.1" ]
|
||||
ipv6 = []
|
||||
# Test that it doesn't cause issues without any mappings
|
||||
net._NetworkGraph__addIpv6FromMap(ipv4, ipv6)
|
||||
self.assertEqual(len(ipv4), 1)
|
||||
self.assertEqual(len(ipv6), 0)
|
||||
|
||||
# Test a mapping
|
||||
net.ipv4ToIPv6 = {
|
||||
"100.64.1.1":"dead::beef/64"
|
||||
}
|
||||
net._NetworkGraph__addIpv6FromMap(ipv4, ipv6)
|
||||
self.assertEqual(len(ipv4), 1)
|
||||
self.assertEqual(len(ipv6), 1)
|
||||
self.assertEqual(ipv6[0], "dead::beef/64")
|
||||
|
||||
def test_site_exclusion(self):
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
net = NetworkGraph()
|
||||
net.excludeSites = ['Site_2']
|
||||
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
|
||||
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
|
||||
self.assertEqual(len(net.nodes), 2)
|
||||
|
||||
def test_site_exception(self):
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
net = NetworkGraph()
|
||||
net.exceptionCPEs = {
|
||||
"Site_2": "Site_1"
|
||||
}
|
||||
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
|
||||
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
|
||||
self.assertEqual(net.nodes[2].parentId, "Site_1")
|
||||
net.prepareTree()
|
||||
self.assertEqual(net.nodes[2].parentIndex, 1)
|
||||
|
||||
def test_graph_render_to_pdf(self):
|
||||
"""
|
||||
Requires that graphviz be installed with
|
||||
pip install graphviz
|
||||
And also the associated graphviz package for
|
||||
your platform.
|
||||
See: https://www.graphviz.org/download/
|
||||
Test that it creates a graphic
|
||||
"""
|
||||
import importlib.util
|
||||
if (spec := importlib.util.find_spec('graphviz')) is None:
|
||||
return
|
||||
|
||||
from integrationCommon import NetworkGraph, NetworkNode, NodeType
|
||||
net = NetworkGraph()
|
||||
net.addRawNode(NetworkNode("Site_1", "Site_1", "", NodeType.site, 1000, 1000))
|
||||
net.addRawNode(NetworkNode("Site_2", "Site_2", "", NodeType.site, 500, 500))
|
||||
net.addRawNode(NetworkNode("AP_A", "AP_A", "Site_1", NodeType.ap, 500, 500))
|
||||
net.addRawNode(NetworkNode("Site_3", "Site_3", "Site_1", NodeType.site, 500, 500))
|
||||
net.addRawNode(NetworkNode("PoP_5", "PoP_5", "Site_3", NodeType.site, 200, 200))
|
||||
net.addRawNode(NetworkNode("AP_9", "AP_9", "PoP_5", NodeType.ap, 120, 120))
|
||||
net.addRawNode(NetworkNode("PoP_6", "PoP_6", "PoP_5", NodeType.site, 60, 60))
|
||||
net.addRawNode(NetworkNode("AP_11", "AP_11", "PoP_6", NodeType.ap, 30, 30))
|
||||
net.addRawNode(NetworkNode("PoP_1", "PoP_1", "Site_2", NodeType.site, 200, 200))
|
||||
net.addRawNode(NetworkNode("AP_7", "AP_7", "PoP_1", NodeType.ap, 100, 100))
|
||||
net.addRawNode(NetworkNode("AP_1", "AP_1", "Site_2", NodeType.ap, 150, 150))
|
||||
net.prepareTree()
|
||||
net.plotNetworkGraph(False)
|
||||
from os.path import exists
|
||||
self.assertEqual(exists("network.pdf.pdf"), True)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
BIN
v1.3/testdata/sample_layout.png
vendored
Normal file
BIN
v1.3/testdata/sample_layout.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Loading…
Reference in New Issue
Block a user