Herbert Wolverson
e20d6d39b0
Another try at a flow system that cleans up.
2024-03-12 10:29:08 -05:00
Herbert Wolverson
3d9b52e627
More cleanup and logic improvements based on observation. In particular, flow cleanup is more accurate now, and we're a bit more aggressive in what RTT data we accept.
2024-03-12 08:57:29 -05:00
Herbert Wolverson
213a27498e
Improve the flow cleanup logic.
2024-03-12 08:21:33 -05:00
Herbert Wolverson
79247e07f0
Fix a really silly mistake. I did NOT mean to always store the worst RTT we've ever seen.
2024-03-11 15:26:23 -05:00
Herbert Wolverson
d93726e538
Only report RTT for flows exceeding 4kbps - to eliminate noise from basically idle connections.
2024-03-11 15:03:47 -05:00
Herbert Wolverson
71fd1d558f
Probable fix for building this on earlier kernel versions.
2024-03-11 14:04:41 -05:00
Herbert Wolverson
445cdcda81
Remove most of the Heimdall mode 1 path, cleaning up the execution path now that we have global flow tracking.
2024-03-11 13:27:42 -05:00
Herbert Wolverson
91a48bc275
Use the new flows system rather than Heimdall to display the circuits flows tab. Not complete yet - the backend is all there still. Also needs some tweaking on the hyperfocus for packet capture mode.
2024-03-11 12:24:18 -05:00
Herbert Wolverson
eb281b3edd
Revert from a dashmap to a regular mutex for the flow container. Performance improved, and flow removal is less troublesome.
2024-03-11 11:20:27 -05:00
Herbert Wolverson
82ecd5eb17
Eliminate all time fetching calls except one, and store the result in the dissector. Minor speed improvement.
2024-03-10 21:11:55 -05:00
Herbert Wolverson
4a8be30c09
First update of the 'recently finished flows' mechanism for tracking the last 5 minutes of data.
2024-03-09 10:29:49 -06:00
Herbert Wolverson
79fa1d42a8
Revert per-ms timings for now.
2024-03-08 14:53:49 -06:00
Herbert Wolverson
1fb151aa1b
Minimal protocol analysis beginnings.
2024-03-08 14:52:02 -06:00
Herbert Wolverson
5202f447dc
Begin adding a little UI
2024-03-08 14:03:29 -06:00
Herbert Wolverson
6b384b2a5f
Include ASN assessment in flows data.
2024-03-08 13:15:49 -06:00
Herbert Wolverson
b1cd8e5ed4
Setup initial stage of ASN analysis - downloading and loading the table, periodic upgrades.
2024-03-08 12:11:37 -06:00
Herbert Wolverson
9c7a9849ad
Replace mutex locked vector with a dashmap for flow data, to ease sharing and update rather than replace cycle.
2024-03-08 09:32:15 -06:00
Herbert Wolverson
b7d43567ff
Reworked the NetFlow code to batch packets into single submissions of up to 30 packets at a time.
2024-03-07 12:51:08 -06:00
Herbert Wolverson
04b0cd4246
Fix warning
2024-03-07 11:38:12 -06:00
Herbert Wolverson
33c1efdd2c
IPv6 encoding matches the same pattern, needs testing.
2024-03-07 11:27:33 -06:00
Herbert Wolverson
34a2ec7b88
Refactor netflow v9 into readable code, and the IPv4 version is at least somewhat working now.
2024-03-07 11:19:02 -06:00
Herbert Wolverson
10c56f9353
This time with a theoretically valid header...
2024-03-05 14:54:57 -06:00
Herbert Wolverson
b649f7004e
Netflow v9. Probably doesn't work yet, committing before I change PC.
2024-03-05 14:05:18 -06:00
Herbert Wolverson
a5bef2851a
Refactor to clean code up.
2024-03-05 11:02:47 -06:00
Herbert Wolverson
f0ddbe62f8
Netflow V5 is largely working. Still a few kinks to work out, but the exporter sends them out - and they are received correctly by my test ehnt setup.
2024-03-05 09:31:49 -06:00
Herbert Wolverson
28e861aeee
First attempt at adding NetFlow v5 support.
2024-03-05 08:44:57 -06:00
Herbert Wolverson
13f2fabe4c
Merge branch 'develop' into per_flow
2024-03-02 09:16:08 -06:00
Herbert Wolverson
b9d2be426c
Add a safety feature that if a site isn't found, auto-allocate its bin.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
cdadbb2c1a
Undoes last commit. This is stable.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
a1314cccbf
Add support for virtual nodes by accident, finishing up stage 2 that allows us to skip layers in the tree.
2024-03-01 14:28:36 -06:00
Herbert Wolverson
1ac7a44355
Stage 1 of the revamp. Instead of modifying traverseNetwork directly, builds a layer per CPU in the network and then uses the previous code.
2024-03-01 14:28:36 -06:00
Robert Chacón
57c99cfa04
Merge pull request #462 from LibreQoE/main
...
Update Develop
2024-03-01 11:22:33 -07:00
Herbert Wolverson
fcf07db112
On flat networks, read calculated LTS (or not) weights and apply them to the binpacking circuit weights if found. This makes binpacking more dynamic for flat networks, taking into account usual usage patterns.
2024-03-01 12:12:32 -06:00
Herbert Wolverson
64fc075066
First try at binpacking the top-level shaper nodes.
2024-03-01 11:17:48 -06:00
Herbert Wolverson
0efa269816
Weight calculation for tree nodes.
2024-03-01 10:24:24 -06:00
Herbert Wolverson
ff0b397395
Dedupe the RTT estimation code.
2024-02-29 12:26:31 -06:00
Herbert Wolverson
0173798981
WIP - Fix my botch at deduping the flow key code.
2024-02-29 12:16:45 -06:00
Herbert Wolverson
28ebecc44d
Forgot to comment out a debug line.
2024-02-29 11:53:10 -06:00
Herbert Wolverson
2be8ce4aa9
Add wraparound logic to handle ACK sequence wraparound.
2024-02-29 11:35:25 -06:00
Herbert Wolverson
370b65fa67
Compromise on 10ms sampling time for rate estimation. Giving pretty accurate results on my tests, and not hitting the CPU too hard.
2024-02-29 11:19:35 -06:00
Herbert Wolverson
23f2cfb9d8
Reduce the flow rate estimation frequency from 1 second to 1 ms.
...
This reduces round-error issues and is generally more accurate -
at the expense of slightly higher CPU usage and a "more bouncy"
rate - that is, for a widely variable flow such as Netflix it
jumps around a lot more. I remain unconvinced that 1ms is the
right number - will continue to test.
Thanks again to Simon Sundberg for this suggestion.
2024-02-29 11:05:37 -06:00
Herbert Wolverson
7d6cbd417a
* Remove most code duplication by tracking rate_index and its reciprocal.
...
* Significantly nicer comments.
* I used a ternary operator. Please shoot me.
Thanks to Simon Sundberg for this recommendation.
2024-02-29 10:17:28 -06:00
Herbert Wolverson
8f343b7c3b
More flexible API for 'top N' calculations based on flow buffer.
2024-02-29 09:10:31 -06:00
Herbert Wolverson
0659cda225
Managed to reduce the amount of space required in the XDP metadata buffer to 32 bits.
2024-02-29 08:45:41 -06:00
Herbert Wolverson
9de53853e7
Improve comments on metadata
2024-02-29 08:28:33 -06:00
Herbert Wolverson
f84798885b
Add a stub for listing top flows by current bitrate.
2024-02-28 15:03:03 -06:00
Herbert Wolverson
7516715874
Add a real-time flow counter to the display and API.
2024-02-28 14:44:20 -06:00
Herbert Wolverson
393c3adc2a
* Restore the tc_dissector, I didn't mean to remove that.
...
* Add SKB metadata support for pass-through and eliding the
second LPM check if we can. Checks at run-time to see if
it is possible.
2024-02-28 12:52:04 -06:00
Herbert Wolverson
95b7c9ac5c
No more second stage dissector.
2024-02-28 10:32:02 -06:00
Herbert Wolverson
22d56a71bf
Correct the ordering of src/dst ports in the flow map.
2024-02-28 09:56:59 -06:00
Herbert Wolverson
Robert Chacón