Commit Graph

1367 Commits

Author SHA1 Message Date
Herbert Wolverson
6575517040 Squish JSON into MsgPack for most common queries, resulting in a big reduction in bandwidth usage. 2023-03-08 21:08:38 +00:00
Herbert Wolverson
b1091a94aa Add the beginnings of internal stats keeping for lqosd and a not-advertised /api/stats page to view. 2023-03-08 19:25:30 +00:00
Herbert Wolverson
bbbe1e5b83 The watched queues structure no longer RwLocks - it's an interior mutable DashMap now. 2023-03-08 18:17:17 +00:00
Herbert Wolverson
c5dca04ade Update tree tracker RTT store to use a DashSet for lock-free access. 2023-03-08 18:01:10 +00:00
Herbert Wolverson
10cf95e5c9 Success: the ThroughputTracker system is lock-free, other than interior locking on the DashMap. 2023-03-08 17:48:36 +00:00
Herbert Wolverson
395b8e0857 Throughput tracker unlocking part 2, cycle is an atomic. 2023-03-08 17:29:15 +00:00
Herbert Wolverson
ae1d5efb4a Throughput tracker refactor part 1: raw_data is now a DashMap (no locking required). 2023-03-08 17:19:35 +00:00
Herbert Wolverson
497abd15f8 The last of the periodic lqos_node_manager bus requests is gone. Unknown devices and host counts are calculated when requested, leading to zero bus usage when you aren't viewing the GUI. 2023-03-08 17:06:25 +00:00
Herbert Wolverson
b47032d006 Overall RTT histogram stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:45:25 +00:00
Herbert Wolverson
67c8f9dec0 Worst 10 stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:38:19 +00:00
Herbert Wolverson
9f8bf22ff9 Top 10 stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:32:08 +00:00
Herbert Wolverson
1053693785 Remove a mut I forgot about 2023-03-08 16:16:44 +00:00
Herbert Wolverson
b6e2091152 Further reduce write locking of NETWORK_JSON by utilizing interior mutability on the RTT tracker. 2023-03-08 16:16:21 +00:00
Herbert Wolverson
1376d7e812 Elide some write locks on NETWORK_JSON altogether by making
the throughput counts atomic with interior mutability.
2023-03-08 15:58:52 +00:00
Herbert Wolverson
ee47549728 Tighten up write locks for NETWORK_JSON 2023-03-08 15:31:29 +00:00
Herbert Wolverson
1196f7a7fb Remove unused code. 2023-03-07 21:44:31 +00:00
Herbert Wolverson
67cc8d8e99 Large batch of improvements:
* The JavaScript RingBuffer structure updated correctly.
* Replaced the funnel graph with text - easier to read.
* Discovered that the current "parking_lot" could become unstable
  under very heavy load, and only with "fat" LTO. Since it's
  no longer recommended (recent change), removed it.
* Replaced the "lazy_static" macro suite with the newly recommended
  "once_cell" system. Less code.
* Full source format.
* Update some dependency versions.
2023-03-07 21:37:23 +00:00
Herbert Wolverson
9fa1318350 Lots of polish. Funnel chart on the front page, rename menu item to tree, move login to the right section, breadcrumbs in the tree. 2023-03-07 15:41:15 +00:00
Herbert Wolverson
fc5b9ad3d4 Stop storing a node-manager wide ringbuffer of data utilization, instead request it when it's wanted. Use the same MultiRingBuffer class, with added render method, to handle front-page utilization. 2023-03-06 22:28:36 +00:00
Herbert Wolverson
39f605e195 Add a MultiRingBuffer type, and use it to cleanly emit stacked charts for the tree system. 2023-03-06 21:44:04 +00:00
Herbert Wolverson
664d29e783 Create RttHistogram class, and use it for both histogram displays. (DRY). 2023-03-06 21:12:07 +00:00
Herbert Wolverson
a2b4c85886 Colorize based on percentage of limit used 2023-03-06 19:41:47 +00:00
Herbert Wolverson
e536f47395 Funnel now displays network sites and clients, supports redaction, and graphing. 2023-03-06 19:23:34 +00:00
Herbert Wolverson
0042bb5f1b Don't include auto-generated 'children' nodes in the display tree. 2023-03-06 16:12:20 +00:00
Herbert Wolverson
78a45515e6 Update to latest crate dependencies. 2023-03-06 16:04:43 +00:00
Herbert Wolverson
9392b43e3c Proper fix for submitting IP/CPU/Queue mapping batches.
* Bring the per-client buffer size back down to a reasonable 2k.
* Divide submission batches into groups and submit those. It's
  still MASSIVELY faster, but it can't fall victim to guessing
  the number of batches incorrectly.
2023-03-06 15:58:57 +00:00
Herbert Wolverson
bfe9601be1 Merge branch 'main' of https://github.com/LibreQoE/LibreQoS 2023-03-05 14:29:21 +00:00
Dave Taht
875ab4b8dc Support 80k Queues in a batch
a 2048000 size read buffer will support 80k queues being installed
in a batch.

H/T Lake Linx and Herbert for this improvement. Perhaps it too
can be improved to have no limit, or the right limit, in the future.
2023-03-04 19:57:05 -08:00
Herbert Wolverson
f64862a8ff Network funnel - work in progress
Step 1 of the network funnel

* network.json reader now tags throughput entries with their tree
  location and parents to whom data should be applied.
* Data flows "up the tree", giving totals all the way up.
* Simple network map page for displaying the data while it's worked
  on.
2023-03-04 16:58:17 +00:00
Robert Chacón
2843061bbd
Merge pull request #278 from LibreQoE/toggleDisplayNameOrAddress
Integrations Improvement - Toggle using display name or address for Circuit Name
2023-03-03 20:40:28 -07:00
Robert Chacón
2e50a2a108
Update integrationSplynx.py 2023-03-03 20:38:32 -07:00
Robert Chacón
fae48c85a5
Update ispConfig.example.py 2023-03-03 20:35:15 -07:00
Robert Chacón
0f9a2b0e21
Update integrationSplynx.py 2023-03-03 20:34:18 -07:00
Robert Chacón
b100870e3c
Update integrationCommon.py 2023-03-03 20:28:22 -07:00
Robert Chacón
6c84100832
Update integrationUISP.py 2023-03-03 20:27:36 -07:00
Robert Chacón
aa3184c64b
Update integrationSplynx.py 2023-03-03 19:50:47 -07:00
Robert Chacón
38fba4a0a4
Add warnings, try to troubleshoot issue 273 2023-03-03 18:30:17 -07:00
Herbert Wolverson
7e5b432253 First refactor towards the "funnel" - shaped devices in lqosd
ShapedDevices.csv is now monitored in lqosd. This brings some
advantages:

* The Tracked Devices list now knows the circuit id association
  for every tracked IP.
* The associations auto-update after a ShapedDevices reload.
* The webserver is no longer doing Trie lookups to figure
  out what name to display.

Moving forwards, this will allow for stats gathering to group
IPs by circuit, and allow calculation of the "funnel".
2023-03-03 21:42:22 +00:00
Herbert Wolverson
42c2c63f55 In lqosd, replace lazy_static with once_cell, per Rust best practices
These days the Rust team are recommending "once_cell", which has
simpler syntax and does the same thing.
2023-03-03 20:26:14 +00:00
Herbert Wolverson
99dc9fc066 Add batching to LibreQoS.py python XDP mapping system
* Add a new Python-exported class ('BatchedCommands') to the
  Python-Rust library.
* Replace direct calls to xdp_iphash_cpu_cmdline with batched
  commands.
* Execute the single batch and obtain counts from the batch.
2023-03-03 16:52:34 +00:00
Herbert Wolverson
24722aa608 Force types to be integers in Python
Replace:
min(data[node]['downloadBandwidthMbps'],parentMaxDL)

With:
min(int(data[node]['downloadBandwidthMbps']),int(parentMaxDL))

Python thought my current configuration contained a string for one
of the numbers. It was a string representing an int. Force strong
typing (a non-numeric will fail, but it would fail anyway).
2023-03-03 14:25:36 +00:00
Herbert Wolverson
433e7ac877 Consolidate hex string to u32 conversion
Rather than having lots of "cpu.replace("0x", "").str_radix(...)
calls around, move to a single, unit-tested function in lqos_utils
and use it repeatedly.
2023-03-03 14:17:31 +00:00
Robert Chacón
308453fecf
Update integrationSplynx.py 2023-03-02 21:21:40 -07:00
Robert Chacón
55ca0adcf0
Update integrationSplynx.py 2023-03-02 10:24:58 -07:00
Robert Chacón
765056f8be
Update integrationSplynx.py 2023-03-02 10:22:40 -07:00
Herbert Wolverson
dca743850a Fix redacting of individual IPs in web console. 2023-03-02 14:41:37 +00:00
Robert Chacón
cae04e1930
Update integrationSplynx.py 2023-03-01 19:34:03 -07:00
Robert Chacón
b33951a8fb
Update integrationSplynx.py 2023-03-01 17:54:51 -07:00
Robert Chacón
eb02c95688
Fix https://github.com/LibreQoE/LibreQoS/issues/273#issuecomment-1451049170 2023-03-01 17:51:59 -07:00
Herbert Wolverson
44ddde6cc5 After coffee, I realized the previous commit had a bug - this should take care of removing comments altogether by mistake. 2023-03-01 15:08:22 +00:00