Herbert Wolverson
a61070529b
Promote and rename on the network tree
2024-05-02 08:49:18 -05:00
Herbert Wolverson
386003214b
Include IP conflict detection.
2024-05-01 15:41:56 -05:00
Herbert Wolverson
d21f4fa8e0
Shaped Devices save button and validation, row removal and addition. Actual saving not implemented yet.
2024-05-01 14:53:40 -05:00
Herbert Wolverson
e2e22ea7ae
Round-trip load/edit/save cycle is basically working for lqos.conf files. More testing needed, passed the easy cases I tried.
2024-05-01 12:36:47 -05:00
Herbert Wolverson
001f01da98
Setting the in-memory config from changes after validation is apparently working.
2024-05-01 11:00:21 -05:00
Herbert Wolverson
861c6e2a8d
Initial validation workflow for saving configuration.
2024-05-01 10:09:11 -05:00
Herbert Wolverson
d4f479dab8
More config work in progress.
2024-04-30 09:32:46 -05:00
Herbert Wolverson
1ad80a278c
Basic network json viewer, shaped devices viewer. All read-only still, but progress.
2024-04-26 12:56:06 -05:00
Herbert Wolverson
3e9fb1e518
Add NIC drop-down lists to config editor
2024-04-25 12:53:16 -05:00
Herbert Wolverson
7ad6001646
Config system can now display the whole lqos.conf file.
2024-04-25 12:31:17 -05:00
Herbert Wolverson
f3c6b344b7
WIP - Current status on config page
2024-04-25 09:02:04 -05:00
Herbert Wolverson
062e1e7eb8
Minimal read-only view of most of the configuration - a good start.
2024-04-24 15:43:43 -05:00
Herbert Wolverson
9ba992db5f
Update package versions in Rust, to remove some more theoretical CVEs.
2024-04-09 14:44:39 -05:00
Herbert Wolverson
a4f08f2bd5
Add a flow map icon to the main page in lqos_node_manager.
2024-03-21 08:39:44 -05:00
Herbert Wolverson
82d213dd89
Update to latest Rust dependencies, to fix a CVE that didn't appear to actually apply to us but showed up in the audit log.
2024-03-20 11:38:57 -05:00
Herbert Wolverson
d72dd29996
Add the ability to ignore subnets from TCP RTT tracking.
...
If you add:
[netflow]
do_not_track_subnets = [ "192.168.66.0/24" ]
to your lqos.conf file, the listed subnets will never have RTT data
collected. This is useful if you wish to ignore the time spent
updating a cloud management system for CPEs, for example.
2024-03-20 09:31:57 -05:00
Herbert Wolverson
c6830abae6
First try at a bubble map.
2024-03-20 08:38:25 -05:00
Herbert Wolverson
280447f79d
Display worst TCP retransmits as a button in the UI.
2024-03-19 15:39:58 -05:00
Herbert Wolverson
a4c8093401
Circuit data now includes TCP retransmits. Made the tables a little less unpleasant to look at.
2024-03-19 14:13:04 -05:00
Herbert Wolverson
b01ab42a3b
Nice highlighting for top 10 buttons.
2024-03-19 13:29:04 -05:00
Herbert Wolverson
e43bd00717
Fix typo
2024-03-19 12:47:56 -05:00
Herbert Wolverson
7e6146b79a
Add the protocols top 10 summary
2024-03-19 12:43:49 -05:00
Herbert Wolverson
c6feebe229
Add plumbing for the top 10 displays. Display ethertypes from expired flows.
2024-03-19 12:29:52 -05:00
Herbert Wolverson
b69748dcf6
REmember to scale the unit on display
2024-03-19 09:01:12 -05:00
Herbert Wolverson
a2afa7fa9d
This should fix the country RTTs.
2024-03-19 08:48:05 -05:00
Herbert Wolverson
e45bedc7ce
One more try before I go and eat something...
2024-03-18 16:13:06 -05:00
Herbert Wolverson
ebc3ab73da
Make the country RTTs actually add up
2024-03-18 16:05:37 -05:00
Herbert Wolverson
019d301913
Make the country RTTs actually add up
2024-03-18 16:04:43 -05:00
Herbert Wolverson
bad4429729
Last commit, but now with sensible logic.
2024-03-18 15:57:07 -05:00
Herbert Wolverson
98155a04c3
Only update flow RTT if there is data for it.
2024-03-18 15:49:31 -05:00
Herbert Wolverson
7b6da8e32e
Limit RTT tracking to flows that have achieved 1 Mbps or more.
2024-03-18 10:19:50 -05:00
Herbert Wolverson
dec1440b31
Flow data: the RTT tracker now keeps track of if new data has arrived for a flow, and only reports an RTT if it has. This prevents stuck flows with no new values coming in from biasing our results.
2024-03-16 09:16:05 -05:00
Herbert Wolverson
c9600f831d
RTT gathering improvement: RTTs are grouped per circuit, and then added as a single value to help preserve per-circuit sample stability.
2024-03-16 09:08:31 -05:00
Herbert Wolverson
b6eb08751c
Fix order of magnitude on the display histogram.
2024-03-15 13:27:12 -05:00
Herbert Wolverson
d0e575f317
Add compatibility with the 5.x kernel version eBPF verifier.
2024-03-15 13:24:58 -05:00
Herbert Wolverson
6ce7e7a3f9
Strongly type RTT data in the throughput tracker to reduce confusion.
2024-03-15 13:03:49 -05:00
Herbert Wolverson
b8937bf9c2
Basic web UI update to show numbers in the right units
2024-03-15 12:55:23 -05:00
Herbert Wolverson
9d0e536089
Use a dual-structure to store RTT in both directions.
2024-03-15 12:29:47 -05:00
Herbert Wolverson
9938a94caf
Comment out console log that was annoying me
2024-03-15 12:19:59 -05:00
Herbert Wolverson
beda42194a
Include RTT events per second tracking in the /api/stats call
2024-03-15 12:18:19 -05:00
Herbert Wolverson
f44af37670
WIP - Compiles RTT data into the tracker, strong type RTTs to clarify the unit confusion. Web side is not done yet.
2024-03-15 12:15:11 -05:00
Herbert Wolverson
56b170f7e4
Truly minimal RTT to userspace eBPF ringbuffer implementation. This
...
breaks a lot of things - no RTTs are currently recorded or acted
upon. The goal is to measure the scale of the load hit for receiving
these events.
2024-03-15 09:45:49 -05:00
Herbert Wolverson
3ad023f69c
Fix circuit flows display from last patch.
2024-03-15 09:02:07 -05:00
Herbert Wolverson
318e20dad7
These numbers line up on the iZones tracker, with wireshark dumps for comparison. This is a WIP - don't apply this. Unbreaking some of my previous code.
2024-03-15 08:32:06 -05:00
Herbert Wolverson
56dd4b6750
It's not quite done yet, but I spent the last 2 hours matching observed RTT, checking with Wireshark, and looking to see if the numbers line up. Implements a small buffer for RTT values per flow. Shrinks some stack entries. Will require a map rebuild.
2024-03-14 14:07:06 -05:00
Herbert Wolverson
69508d2753
Remove two variables and use pointers instead to save stack space.
2024-03-14 12:40:33 -05:00
Herbert Wolverson
a937820a87
Move rate estimate from 64-bits to 32-bits, saving some bytes.
2024-03-14 11:03:43 -05:00
Herbert Wolverson
8ec361b095
Fix a reporting message that was meant to be compile-conditional.
2024-03-14 10:42:22 -05:00
Herbert Wolverson
fbb3960b50
Warning fix
2024-03-14 08:52:36 -05:00
Herbert Wolverson
eab9df4874
Missing file from previous commit
2024-03-13 14:56:59 -05:00
Herbert Wolverson
fb91e8313a
Very silly commit - adds a /showoff page to the node manager firing particles from all endpoints at my ISP in Missouri. Will turn into something useful in the future.
2024-03-13 14:45:53 -05:00
Herbert Wolverson
a63ff0a6f1
Rename all flowbee "retries" to "tcp_retransmits" in code, and
...
"retransmits" in visible HTML.
2024-03-13 08:20:36 -05:00
Herbert Wolverson
fae96280b9
Actually limit the top 10 endpoints to being 10
2024-03-12 16:41:57 -05:00
Herbert Wolverson
c16f06b0ab
Sort countries by download bytes not upload
2024-03-12 15:47:29 -05:00
Herbert Wolverson
66a19c04a4
First example of an actual summary report - we can group flow endpoints in a 5-minute report.
2024-03-12 15:19:07 -05:00
Herbert Wolverson
1fb5838ebe
Remove a test call that was spamming the logs.
2024-03-12 14:46:47 -05:00
Herbert Wolverson
e46aafe5ae
Silly performance: making 12k mutex locks when I need one was really dumb.
2024-03-12 14:20:35 -05:00
Herbert Wolverson
5a3f90412d
Working geocode system, albeit not as useful as I hoped.
2024-03-12 14:02:36 -05:00
Herbert Wolverson
55f24cf71b
Revert previous bad idea
2024-03-12 11:38:19 -05:00
Herbert Wolverson
07239b3d24
That should avoid some flow duplication that was hitting too hard.
2024-03-12 11:30:13 -05:00
Herbert Wolverson
3ca7ca8a0d
Lower threshold for capturing RTT by rate
2024-03-12 10:52:41 -05:00
Herbert Wolverson
e20d6d39b0
Another try at a flow system that cleans up.
2024-03-12 10:29:08 -05:00
Herbert Wolverson
3d9b52e627
More cleanup and logic improvements based on observation. In particular, flow cleanup is more accurate now, and we're a bit more aggressive in what RTT data we accept.
2024-03-12 08:57:29 -05:00
Herbert Wolverson
213a27498e
Improve the flow cleanup logic.
2024-03-12 08:21:33 -05:00
Herbert Wolverson
79247e07f0
Fix a really silly mistake. I did NOT mean to always store the worst RTT we've ever seen.
2024-03-11 15:26:23 -05:00
Herbert Wolverson
d93726e538
Only report RTT for flows exceeding 4kbps - to eliminate noise from basically idle connections.
2024-03-11 15:03:47 -05:00
Herbert Wolverson
71fd1d558f
Probable fix for building this on earlier kernel versions.
2024-03-11 14:04:41 -05:00
Herbert Wolverson
445cdcda81
Remove most of the Heimdall mode 1 path, cleaning up the execution path now that we have global flow tracking.
2024-03-11 13:27:42 -05:00
Herbert Wolverson
91a48bc275
Use the new flows system rather than Heimdall to display the circuits flows tab. Not complete yet - the backend is all there still. Also needs some tweaking on the hyperfocus for packet capture mode.
2024-03-11 12:24:18 -05:00
Herbert Wolverson
eb281b3edd
Revert from a dashmap to a regular mutex for the flow container. Performance improved, and flow removal is less troublesome.
2024-03-11 11:20:27 -05:00
Herbert Wolverson
82ecd5eb17
Eliminate all time fetching calls except one, and store the result in the dissector. Minor speed improvement.
2024-03-10 21:11:55 -05:00
Herbert Wolverson
4a8be30c09
First update of the 'recently finished flows' mechanism for tracking the last 5 minutes of data.
2024-03-09 10:29:49 -06:00
Herbert Wolverson
79fa1d42a8
Revert per-ms timings for now.
2024-03-08 14:53:49 -06:00
Herbert Wolverson
1fb151aa1b
Minimal protocol analysis beginnings.
2024-03-08 14:52:02 -06:00
Herbert Wolverson
5202f447dc
Begin adding a little UI
2024-03-08 14:03:29 -06:00
Herbert Wolverson
6b384b2a5f
Include ASN assessment in flows data.
2024-03-08 13:15:49 -06:00
Herbert Wolverson
b1cd8e5ed4
Setup initial stage of ASN analysis - downloading and loading the table, periodic upgrades.
2024-03-08 12:11:37 -06:00
Herbert Wolverson
9c7a9849ad
Replace mutex locked vector with a dashmap for flow data, to ease sharing and update rather than replace cycle.
2024-03-08 09:32:15 -06:00
Herbert Wolverson
b7d43567ff
Reworked the NetFlow code to batch packets into single submissions of up to 30 packets at a time.
2024-03-07 12:51:08 -06:00
Herbert Wolverson
04b0cd4246
Fix warning
2024-03-07 11:38:12 -06:00
Herbert Wolverson
33c1efdd2c
IPv6 encoding matches the same pattern, needs testing.
2024-03-07 11:27:33 -06:00
Herbert Wolverson
34a2ec7b88
Refactor netflow v9 into readable code, and the IPv4 version is at least somewhat working now.
2024-03-07 11:19:02 -06:00
Herbert Wolverson
10c56f9353
This time with a theoretically valid header...
2024-03-05 14:54:57 -06:00
Herbert Wolverson
b649f7004e
Netflow v9. Probably doesn't work yet, committing before I change PC.
2024-03-05 14:05:18 -06:00
Herbert Wolverson
a5bef2851a
Refactor to clean code up.
2024-03-05 11:02:47 -06:00
Herbert Wolverson
f0ddbe62f8
Netflow V5 is largely working. Still a few kinks to work out, but the exporter sends them out - and they are received correctly by my test ehnt setup.
2024-03-05 09:31:49 -06:00
Herbert Wolverson
43befe6c9c
Enable binpacking for on-a-stick mode.
2024-03-05 09:08:41 -06:00
Herbert Wolverson
bc5a6d68a1
Fix on-a-stick support.
2024-03-05 08:45:36 -06:00
Herbert Wolverson
28e861aeee
First attempt at adding NetFlow v5 support.
2024-03-05 08:44:57 -06:00
Herbert Wolverson
13f2fabe4c
Merge branch 'develop' into per_flow
2024-03-02 09:16:08 -06:00
Herbert Wolverson
d741a24ed2
Merge branch 'circuit_weights' into develop
2024-03-01 14:28:41 -06:00
Herbert Wolverson
b9d2be426c
Add a safety feature that if a site isn't found, auto-allocate its bin.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
cdadbb2c1a
Undoes last commit. This is stable.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
a1314cccbf
Add support for virtual nodes by accident, finishing up stage 2 that allows us to skip layers in the tree.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
1ac7a44355
Stage 1 of the revamp. Instead of modifying traverseNetwork directly, builds a layer per CPU in the network and then uses the previous code.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
7836dbc3bb
Add a safety feature that if a site isn't found, auto-allocate its bin.
2024-03-01 14:17:59 -06:00
Herbert Wolverson
e9530dff54
Undoes last commit. This is stable.
2024-03-01 13:59:34 -06:00
Herbert Wolverson
728b0a9153
Add support for virtual nodes by accident, finishing up stage 2 that allows us to skip layers in the tree.
2024-03-01 13:50:39 -06:00
Herbert Wolverson
008236796c
Stage 1 of the revamp. Instead of modifying traverseNetwork directly, builds a layer per CPU in the network and then uses the previous code.
2024-03-01 13:36:44 -06:00
Herbert Wolverson
fcf07db112
On flat networks, read calculated LTS (or not) weights and apply them to the binpacking circuit weights if found. This makes binpacking more dynamic for flat networks, taking into account usual usage patterns.
2024-03-01 12:12:32 -06:00
Herbert Wolverson