IntenseWebs/LibreQoS
3
0
Fork 0
mirror of https://github.com/LibreQoE/LibreQoS.git synced 2025-02-25 18:55:32 -06:00
Code Issues Packages Projects Releases Wiki Activity
1,352 Commits 71 Branches 23 Tags 134 MiB
7b571213d9
Commit Graph

483 Commits

Author SHA1 Message Date
Herbert Wolverson
5ff671567a Some relatively simple performance enhancements that reduce the burden of Heimdall when he's not looking. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
810823c8bb Fix verifier dislike bpf_probe 2023-03-22 14:39:32 +00:00
Herbert Wolverson
951ee4cbe5 Make it easier to change capture size at compile time, set to 128 bytes. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
17100415dd Add libpcap format support and ability to download a snapshot of the last 10 seconds in pcap format for Wireshark (or other tool) analysis. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
8288bb3f9b Extend the capture to include TCP flags, window and ACK data. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
d542bf1660 Abuse error bars to show packet durations on dump view. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
521b383ef7 Oops - packet buttons didn't render. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
eba74042dd Fix warning 2023-03-22 14:39:32 +00:00
Herbert Wolverson
d58c1cb459 Slightly more efficient circuit_queues page, and fix the issue with tin data not appearing. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
70b4230339 Switch to a lock free method of storing packets. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
f6d4542598 Add a packet dump page, and the beginnings of a packet browser at NS resolution. 2023-03-22 14:39:32 +00:00
Herbert Wolverson
450466976a Fix warnings 2023-03-22 14:39:32 +00:00
Herbert Wolverson
7180eb48dc WIP: You can now view the last 10 seconds of flow data in the API. 2023-03-22 14:39:31 +00:00
Herbert Wolverson
8f3a8a78eb CPU melting work in progress. 2023-03-22 14:33:49 +00:00
Herbert Wolverson
c70d01b3f6 VERY work in progress - do not try to use this. The good: creates a ringbuffer and successfully sends events to userspace. The bad: it's currently watching all packets, eats a lot of CPU and has a horrible lack of abstractions. 2023-03-22 14:33:48 +00:00
Herbert Wolverson
e97fb4ac11 Super ugly start to getting perf events working for Heimdall. 2023-03-22 14:32:31 +00:00
Herbert Wolverson
73ee423592 Re-hide the scary error messages that aren't actually errors. 2023-03-22 14:32:31 +00:00
Herbert Wolverson
9ba5bb2628 Add bounds check 2023-03-22 14:32:31 +00:00
Herbert Wolverson
7d055d9796 Cleanup ICMP tracking, add documentation to source code. 
This should make the Heimdall branch usable.
 2023-03-22 14:32:31 +00:00
Herbert Wolverson
d3feb64911 Improve ECN display 2023-03-22 14:32:31 +00:00
Herbert Wolverson
07354b868d Change rendering of marks and drops on circuits. 2023-03-22 14:32:31 +00:00
Herbert Wolverson
4372b501c9 Continue cleaning up flow display system. 2023-03-22 14:32:31 +00:00
Herbert Wolverson
64641c6849 Port map_txq_config_base_setup from C to Rust 
The previous C function was pretty simple, it'd been left as C
for fast porting. It's now a safe (other than the syscall,
which is properly wrapped) Rust function.
 2023-03-22 14:32:31 +00:00
Herbert Wolverson
93e8afae71 Replace unsafe bpf_num_possible_cpus() with safe Rust function. 
Port the C from `bpf_num_possible_cpus()` over to safe Rust,
with proper error handling and unit tests to ensure that it
is giving the correct answers. Change usages to the new safe
function.
 2023-03-22 14:32:31 +00:00
Herbert Wolverson
130c888e22 Heimdall now only watches those who request to be watched by default. 2023-03-22 14:32:30 +00:00
Herbert Wolverson
e1346f9d47 Rename the branch to Heimdall 2023-03-22 14:31:32 +00:00
Herbert Wolverson
fad3e7858a Minimally working functionality 2023-03-22 14:30:48 +00:00
Herbert Wolverson
aecc5f0d95 First play with the idea of storing stats about each TCP/UDP/ICMP flow as it traverses the network. 2023-03-22 14:30:14 +00:00
Herbert Wolverson
9639ceeb6b Merge branch 'anonymouse' 2023-03-21 17:48:53 +00:00
Herbert Wolverson
f6318d8652 Change 'map ip' to add or insert, rather than just insert. This allows for speedy changes without a remove, making it an atomic operation. So you can add a new queue, change the IP/net, and wait a few seconds before deleting the old one to avoid packet loss during reloads. 2023-03-21 17:46:51 +00:00
Herbert Wolverson
769b9e73bd Include high traffic watermark in stats 2023-03-21 17:22:48 +00:00
Herbert Wolverson
98364a8526 More granular version numbering for DPKG builds 2023-03-21 16:33:58 +00:00
Herbert Wolverson
f42a1a5908 build_rust.sh checks for dependencies 
On start, the "build_rust.sh" script checks for the availability
of bpftool, pkg-config and clang/LLVM. If these aren't found, it
fires up an "apt-get install" for the appropriate packages.
 2023-03-21 13:17:46 +00:00
Herbert Wolverson
e415d699db Make the SQL and unique id less dumb! Use the machine-id file that is automatically generated and hash it for node id. 2023-03-20 20:25:32 +00:00
Herbert Wolverson
84e4306b22 Don't change lqos.conf, generate a node ID hash based on ip link output. 2023-03-20 19:47:30 +00:00
Herbert Wolverson
26efcdb208 Proof of concept with a tiny webserver 2023-03-20 19:17:21 +00:00
Herbert Wolverson
b19b8b356b Truly minimal stats server, but it works. 2023-03-20 19:17:21 +00:00
Herbert Wolverson
ef5ffa4e64 Start to add sqlite db - we'll probably use postgres, but I wanted something for testing. 2023-03-20 19:17:21 +00:00
Herbert Wolverson
c1c5f3bba4 Initial stats collection protocol. 2023-03-20 19:17:20 +00:00
Herbert Wolverson
6d1519b9aa Setup asks about anonymous access. 2023-03-20 19:16:07 +00:00
Herbert Wolverson
adc3bb8eeb dpkg setup adds an auto-generated node-id 2023-03-20 19:16:07 +00:00
Herbert Wolverson
8146472479 More data in anonymous stats. 2023-03-20 19:16:07 +00:00
Herbert Wolverson
08e7788efb Basic data structures and collection for anonymous usage collection. 2023-03-20 19:16:07 +00:00
Herbert Wolverson
5805d9860e Add anonymous usage options to lqos.conf and the example file. 2023-03-20 19:16:07 +00:00
Herbert Wolverson
beead5a303 Backport the TC_HANDLE parser to main. 
Replace C code with native Rust, including unit tests to find the
edge-cases.
 2023-03-20 17:40:03 +00:00
Herbert Wolverson
e7e7a089ee Round memory numbers. 2023-03-20 16:52:17 +00:00
Herbert Wolverson
f0aaa05e38 Tin graphs have a height of 200 instead of 150. 
This removes the wrapping error on the legend.
 2023-03-20 16:50:00 +00:00
Herbert Wolverson
62ea9c7038 Backport markers mode for drops graphs 
Tins display markers rather than lines for drops graphs.
 2023-03-20 16:43:09 +00:00
Herbert Wolverson
6668214d09 So 0 number of seconds is actually zero. 2023-03-20 16:33:40 +00:00
Herbert Wolverson
b56e44b1d2 At the expense of a nice margin/gutter. Sigh. 2023-03-20 16:12:56 +00:00
Herbert Wolverson
efdf522d1a italic to make it more visible. 2023-03-20 16:10:06 +00:00
Herbert Wolverson
d14526e0a5 ISSUE #183 - Remove the redundant "dashboard" link from all pages, 
since the logo already links to the dashboard.
 2023-03-20 16:07:08 +00:00
Herbert "TheBracket
1ada4e5a5c
Build deb as part of Continuous Integration (#281) 
Build the .deb release as part of the Github Actions run. The .deb file is available for download as an artifact of that build for 90 days. Please squash this commit.
 2023-03-20 08:32:04 -07:00
Robert Chacón
44518e76a5
Fix race condition where partial reload runs at same time 2023-03-19 20:50:22 -06:00
Robert Chacón
b9adea70f4
Multiple fixes to partial reload 2023-03-15 20:25:56 -06:00
Herbert "TheBracket
4f9bc7e867
Merge pull request #279 from LibreQoE/the-funnel 
Merge the funnel branch
 2023-03-09 09:12:36 -06:00
Dave Taht
38a6ccbbba Merge branch 'main' of github.com:LibreQoE/LibreQoS 2023-03-09 06:52:50 -08:00
Dave Taht
c2098880c3 InterfaceA vs InterfaceB 
So far new users have got these two parameters inverted more
than 50% of the time, with roughly the same probability of
toast landing buttered side down.

Rewriting the python to read the toml, or accessing rust via
the bus, would make this mistake impossible.

I would really like to retire ispConfig.py entirely.
 2023-03-09 06:51:07 -08:00
Herbert Wolverson
23736eaab7 Correct logic for detecting if the bandwidth test is already running. compare_exchange returns the CURRENT value, not the NEW value. 2023-03-09 14:25:43 +00:00
Herbert Wolverson
798e65b107 Properly redact funnel entry names on the circuit_queue page. 2023-03-08 22:23:18 +00:00
Herbert Wolverson
6575517040 Squish JSON into MsgPack for most common queries, resulting in a big reduction in bandwidth usage. 2023-03-08 21:08:38 +00:00
Herbert Wolverson
b1091a94aa Add the beginnings of internal stats keeping for lqosd and a not-advertised /api/stats page to view. 2023-03-08 19:25:30 +00:00
Herbert Wolverson
bbbe1e5b83 The watched queues structure no longer RwLocks - it's an interior mutable DashMap now. 2023-03-08 18:17:17 +00:00
Herbert Wolverson
c5dca04ade Update tree tracker RTT store to use a DashSet for lock-free access. 2023-03-08 18:01:10 +00:00
Herbert Wolverson
10cf95e5c9 Success: the ThroughputTracker system is lock-free, other than interior locking on the DashMap. 2023-03-08 17:48:36 +00:00
Herbert Wolverson
395b8e0857 Throughput tracker unlocking part 2, cycle is an atomic. 2023-03-08 17:29:15 +00:00
Herbert Wolverson
ae1d5efb4a Throughput tracker refactor part 1: raw_data is now a DashMap (no locking required). 2023-03-08 17:19:35 +00:00
Herbert Wolverson
497abd15f8 The last of the periodic lqos_node_manager bus requests is gone. Unknown devices and host counts are calculated when requested, leading to zero bus usage when you aren't viewing the GUI. 2023-03-08 17:06:25 +00:00
Herbert Wolverson
b47032d006 Overall RTT histogram stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:45:25 +00:00
Herbert Wolverson
67c8f9dec0 Worst 10 stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:38:19 +00:00
Herbert Wolverson
9f8bf22ff9 Top 10 stats are no longer stored globally in lqos_node_manager. Instead, they are only calculated when requested. Slightly higher utilization with lots of people using the node manager, and no bus traffic during the 99% of the time when its idle. 2023-03-08 16:32:08 +00:00
Herbert Wolverson
1053693785 Remove a mut I forgot about 2023-03-08 16:16:44 +00:00
Herbert Wolverson
b6e2091152 Further reduce write locking of NETWORK_JSON by utilizing interior mutability on the RTT tracker. 2023-03-08 16:16:21 +00:00
Herbert Wolverson
1376d7e812 Elide some write locks on NETWORK_JSON altogether by making 
the throughput counts atomic with interior mutability.
 2023-03-08 15:58:52 +00:00
Herbert Wolverson
ee47549728 Tighten up write locks for NETWORK_JSON 2023-03-08 15:31:29 +00:00
Herbert Wolverson
1196f7a7fb Remove unused code. 2023-03-07 21:44:31 +00:00
Herbert Wolverson
67cc8d8e99 Large batch of improvements: 
* The JavaScript RingBuffer structure updated correctly.
* Replaced the funnel graph with text - easier to read.
* Discovered that the current "parking_lot" could become unstable
  under very heavy load, and only with "fat" LTO. Since it's
  no longer recommended (recent change), removed it.
* Replaced the "lazy_static" macro suite with the newly recommended
  "once_cell" system. Less code.
* Full source format.
* Update some dependency versions.
 2023-03-07 21:37:23 +00:00
Herbert Wolverson
9fa1318350 Lots of polish. Funnel chart on the front page, rename menu item to tree, move login to the right section, breadcrumbs in the tree. 2023-03-07 15:41:15 +00:00
Herbert Wolverson
fc5b9ad3d4 Stop storing a node-manager wide ringbuffer of data utilization, instead request it when it's wanted. Use the same MultiRingBuffer class, with added render method, to handle front-page utilization. 2023-03-06 22:28:36 +00:00
Herbert Wolverson
39f605e195 Add a MultiRingBuffer type, and use it to cleanly emit stacked charts for the tree system. 2023-03-06 21:44:04 +00:00
Herbert Wolverson
664d29e783 Create RttHistogram class, and use it for both histogram displays. (DRY). 2023-03-06 21:12:07 +00:00
Herbert Wolverson
a2b4c85886 Colorize based on percentage of limit used 2023-03-06 19:41:47 +00:00
Herbert Wolverson
e536f47395 Funnel now displays network sites and clients, supports redaction, and graphing. 2023-03-06 19:23:34 +00:00
Herbert Wolverson
0042bb5f1b Don't include auto-generated 'children' nodes in the display tree. 2023-03-06 16:12:20 +00:00
Herbert Wolverson
78a45515e6 Update to latest crate dependencies. 2023-03-06 16:04:43 +00:00
Herbert Wolverson
9392b43e3c Proper fix for submitting IP/CPU/Queue mapping batches. 
* Bring the per-client buffer size back down to a reasonable 2k.
* Divide submission batches into groups and submit those. It's
  still MASSIVELY faster, but it can't fall victim to guessing
  the number of batches incorrectly.
 2023-03-06 15:58:57 +00:00
Herbert Wolverson
bfe9601be1 Merge branch 'main' of https://github.com/LibreQoE/LibreQoS 2023-03-05 14:29:21 +00:00
Dave Taht
875ab4b8dc Support 80k Queues in a batch 
a 2048000 size read buffer will support 80k queues being installed
in a batch.

H/T Lake Linx and Herbert for this improvement. Perhaps it too
can be improved to have no limit, or the right limit, in the future.
 2023-03-04 19:57:05 -08:00
Herbert Wolverson
f64862a8ff Network funnel - work in progress 
Step 1 of the network funnel

* network.json reader now tags throughput entries with their tree
  location and parents to whom data should be applied.
* Data flows "up the tree", giving totals all the way up.
* Simple network map page for displaying the data while it's worked
  on.
 2023-03-04 16:58:17 +00:00
Robert Chacón
2e50a2a108
Update integrationSplynx.py 2023-03-03 20:38:32 -07:00
Robert Chacón
fae48c85a5
Update ispConfig.example.py 2023-03-03 20:35:15 -07:00
Robert Chacón
0f9a2b0e21
Update integrationSplynx.py 2023-03-03 20:34:18 -07:00
Robert Chacón
b100870e3c
Update integrationCommon.py 2023-03-03 20:28:22 -07:00
Robert Chacón
6c84100832
Update integrationUISP.py 2023-03-03 20:27:36 -07:00
Robert Chacón
aa3184c64b
Update integrationSplynx.py 2023-03-03 19:50:47 -07:00
Robert Chacón
38fba4a0a4
Add warnings, try to troubleshoot issue 273 2023-03-03 18:30:17 -07:00
Herbert Wolverson
7e5b432253 First refactor towards the "funnel" - shaped devices in lqosd 
ShapedDevices.csv is now monitored in lqosd. This brings some
advantages:

* The Tracked Devices list now knows the circuit id association
  for every tracked IP.
* The associations auto-update after a ShapedDevices reload.
* The webserver is no longer doing Trie lookups to figure
  out what name to display.

Moving forwards, this will allow for stats gathering to group
IPs by circuit, and allow calculation of the "funnel".
 2023-03-03 21:42:22 +00:00
Herbert Wolverson
42c2c63f55 In lqosd, replace lazy_static with once_cell, per Rust best practices 
These days the Rust team are recommending "once_cell", which has
simpler syntax and does the same thing.
 2023-03-03 20:26:14 +00:00
Herbert Wolverson
99dc9fc066 Add batching to LibreQoS.py python XDP mapping system 
* Add a new Python-exported class ('BatchedCommands') to the
  Python-Rust library.
* Replace direct calls to xdp_iphash_cpu_cmdline with batched
  commands.
* Execute the single batch and obtain counts from the batch.
 2023-03-03 16:52:34 +00:00
Herbert Wolverson
24722aa608 Force types to be integers in Python 
Replace:
min(data[node]['downloadBandwidthMbps'],parentMaxDL)

With:
min(int(data[node]['downloadBandwidthMbps']),int(parentMaxDL))

Python thought my current configuration contained a string for one
of the numbers. It was a string representing an int. Force strong
typing (a non-numeric will fail, but it would fail anyway).
 2023-03-03 14:25:36 +00:00