IntenseWebs/LibreQoS
3
0
Fork 0
mirror of https://github.com/LibreQoE/LibreQoS.git synced 2026-09-03 20:52:53 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,029 Commits 269 Branches 4 Tags
fbb3960b5009ca9c4473fbbdeca3ca2793a8b07a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Herbert Wolverson fbb3960b50 Warning fix 2024-03-14 08:52:36 -05:00
Herbert Wolverson eab9df4874 Missing file from previous commit 2024-03-13 14:56:59 -05:00
Herbert Wolverson fb91e8313a Very silly commit - adds a /showoff page to the node manager firing particles from all endpoints at my ISP in Missouri. Will turn into something useful in the future. 2024-03-13 14:45:53 -05:00
Herbert Wolverson a63ff0a6f1 Rename all flowbee "retries" to "tcp_retransmits" in code, and 
"retransmits" in visible HTML.
 2024-03-13 08:20:36 -05:00
Herbert Wolverson fae96280b9 Actually limit the top 10 endpoints to being 10 2024-03-12 16:41:57 -05:00
Herbert Wolverson c16f06b0ab Sort countries by download bytes not upload 2024-03-12 15:47:29 -05:00
Herbert Wolverson 66a19c04a4 First example of an actual summary report - we can group flow endpoints in a 5-minute report. 2024-03-12 15:19:07 -05:00
Herbert Wolverson 1fb5838ebe Remove a test call that was spamming the logs. 2024-03-12 14:46:47 -05:00
Herbert Wolverson e46aafe5ae Silly performance: making 12k mutex locks when I need one was really dumb. 2024-03-12 14:20:35 -05:00
Herbert Wolverson 5a3f90412d Working geocode system, albeit not as useful as I hoped. 2024-03-12 14:02:36 -05:00
Herbert Wolverson 55f24cf71b Revert previous bad idea 2024-03-12 11:38:19 -05:00
Herbert Wolverson 07239b3d24 That should avoid some flow duplication that was hitting too hard. 2024-03-12 11:30:13 -05:00
Herbert Wolverson 3ca7ca8a0d Lower threshold for capturing RTT by rate 2024-03-12 10:52:41 -05:00
Herbert Wolverson e20d6d39b0 Another try at a flow system that cleans up. 2024-03-12 10:29:08 -05:00
Herbert Wolverson 3d9b52e627 More cleanup and logic improvements based on observation. In particular, flow cleanup is more accurate now, and we're a bit more aggressive in what RTT data we accept. 2024-03-12 08:57:29 -05:00
Herbert Wolverson 213a27498e Improve the flow cleanup logic. 2024-03-12 08:21:33 -05:00
Herbert Wolverson 79247e07f0 Fix a really silly mistake. I did NOT mean to always store the worst RTT we've ever seen. 2024-03-11 15:26:23 -05:00
Herbert Wolverson d93726e538 Only report RTT for flows exceeding 4kbps - to eliminate noise from basically idle connections. 2024-03-11 15:03:47 -05:00
Herbert Wolverson 71fd1d558f Probable fix for building this on earlier kernel versions. 2024-03-11 14:04:41 -05:00
Herbert Wolverson 445cdcda81 Remove most of the Heimdall mode 1 path, cleaning up the execution path now that we have global flow tracking. 2024-03-11 13:27:42 -05:00
Herbert Wolverson 91a48bc275 Use the new flows system rather than Heimdall to display the circuits flows tab. Not complete yet - the backend is all there still. Also needs some tweaking on the hyperfocus for packet capture mode. 2024-03-11 12:24:18 -05:00
Herbert Wolverson eb281b3edd Revert from a dashmap to a regular mutex for the flow container. Performance improved, and flow removal is less troublesome. 2024-03-11 11:20:27 -05:00
Herbert Wolverson 82ecd5eb17 Eliminate all time fetching calls except one, and store the result in the dissector. Minor speed improvement. 2024-03-10 21:11:55 -05:00
Herbert Wolverson 4a8be30c09 First update of the 'recently finished flows' mechanism for tracking the last 5 minutes of data. 2024-03-09 10:29:49 -06:00
Herbert Wolverson 79fa1d42a8 Revert per-ms timings for now. 2024-03-08 14:53:49 -06:00
Herbert Wolverson 1fb151aa1b Minimal protocol analysis beginnings. 2024-03-08 14:52:02 -06:00
Herbert Wolverson 5202f447dc Begin adding a little UI 2024-03-08 14:03:29 -06:00
Herbert Wolverson 6b384b2a5f Include ASN assessment in flows data. 2024-03-08 13:15:49 -06:00
Herbert Wolverson b1cd8e5ed4 Setup initial stage of ASN analysis - downloading and loading the table, periodic upgrades. 2024-03-08 12:11:37 -06:00
Herbert Wolverson 9c7a9849ad Replace mutex locked vector with a dashmap for flow data, to ease sharing and update rather than replace cycle. 2024-03-08 09:32:15 -06:00
Herbert Wolverson b7d43567ff Reworked the NetFlow code to batch packets into single submissions of up to 30 packets at a time. 2024-03-07 12:51:08 -06:00
Herbert Wolverson 04b0cd4246 Fix warning 2024-03-07 11:38:12 -06:00
Herbert Wolverson 33c1efdd2c IPv6 encoding matches the same pattern, needs testing. 2024-03-07 11:27:33 -06:00
Herbert Wolverson 34a2ec7b88 Refactor netflow v9 into readable code, and the IPv4 version is at least somewhat working now. 2024-03-07 11:19:02 -06:00
Herbert Wolverson 10c56f9353 This time with a theoretically valid header... 2024-03-05 14:54:57 -06:00
Herbert Wolverson b649f7004e Netflow v9. Probably doesn't work yet, committing before I change PC. 2024-03-05 14:05:18 -06:00
Herbert Wolverson a5bef2851a Refactor to clean code up. 2024-03-05 11:02:47 -06:00
Herbert Wolverson f0ddbe62f8 Netflow V5 is largely working. Still a few kinks to work out, but the exporter sends them out - and they are received correctly by my test ehnt setup. 2024-03-05 09:31:49 -06:00
Herbert Wolverson 28e861aeee First attempt at adding NetFlow v5 support. 2024-03-05 08:44:57 -06:00
Herbert Wolverson 13f2fabe4c Merge branch 'develop' into per_flow 2024-03-02 09:16:08 -06:00
Herbert Wolverson d741a24ed2 Merge branch 'circuit_weights' into develop 2024-03-01 14:28:41 -06:00
Herbert Wolverson b9d2be426c Add a safety feature that if a site isn't found, auto-allocate its bin. 2024-03-01 14:28:36 -06:00
Herbert Wolverson cdadbb2c1a Undoes last commit. This is stable. 2024-03-01 14:28:36 -06:00
Herbert Wolverson a1314cccbf Add support for virtual nodes by accident, finishing up stage 2 that allows us to skip layers in the tree. 2024-03-01 14:28:36 -06:00
Herbert Wolverson 1ac7a44355 Stage 1 of the revamp. Instead of modifying traverseNetwork directly, builds a layer per CPU in the network and then uses the previous code. 2024-03-01 14:28:36 -06:00
Herbert Wolverson 7836dbc3bb Add a safety feature that if a site isn't found, auto-allocate its bin. 2024-03-01 14:17:59 -06:00
Herbert Wolverson e9530dff54 Undoes last commit. This is stable. 2024-03-01 13:59:34 -06:00
Herbert Wolverson 728b0a9153 Add support for virtual nodes by accident, finishing up stage 2 that allows us to skip layers in the tree. 2024-03-01 13:50:39 -06:00
Herbert Wolverson 008236796c Stage 1 of the revamp. Instead of modifying traverseNetwork directly, builds a layer per CPU in the network and then uses the previous code. 2024-03-01 13:36:44 -06:00
Herbert Wolverson fcf07db112 On flat networks, read calculated LTS (or not) weights and apply them to the binpacking circuit weights if found. This makes binpacking more dynamic for flat networks, taking into account usual usage patterns. 2024-03-01 12:12:32 -06:00
Herbert Wolverson 64fc075066 First try at binpacking the top-level shaper nodes. 2024-03-01 11:17:48 -06:00
Herbert Wolverson 0efa269816 Weight calculation for tree nodes. 2024-03-01 10:24:24 -06:00
Herbert Wolverson ff0b397395 Dedupe the RTT estimation code. 2024-02-29 12:26:31 -06:00
Herbert Wolverson 0173798981 WIP - Fix my botch at deduping the flow key code. 2024-02-29 12:16:45 -06:00
Herbert Wolverson 28ebecc44d Forgot to comment out a debug line. 2024-02-29 11:53:10 -06:00
Herbert Wolverson 2be8ce4aa9 Add wraparound logic to handle ACK sequence wraparound. 2024-02-29 11:35:25 -06:00
Herbert Wolverson 370b65fa67 Compromise on 10ms sampling time for rate estimation. Giving pretty accurate results on my tests, and not hitting the CPU too hard. 2024-02-29 11:19:35 -06:00
Herbert Wolverson 23f2cfb9d8 Reduce the flow rate estimation frequency from 1 second to 1 ms. 
This reduces round-error issues and is generally more accurate -
at the expense of slightly higher CPU usage and a "more bouncy"
rate - that is, for a widely variable flow such as Netflix it
jumps around a lot more. I remain unconvinced that 1ms is the
right number - will continue to test.

Thanks again to Simon Sundberg for this suggestion.
 2024-02-29 11:05:37 -06:00
Herbert Wolverson 7d6cbd417a * Remove most code duplication by tracking rate_index and its reciprocal. 
* Significantly nicer comments.
* I used a ternary operator. Please shoot me.

Thanks to Simon Sundberg for this recommendation.
 2024-02-29 10:17:28 -06:00
Herbert Wolverson 8f343b7c3b More flexible API for 'top N' calculations based on flow buffer. 2024-02-29 09:10:31 -06:00
Herbert Wolverson 0659cda225 Managed to reduce the amount of space required in the XDP metadata buffer to 32 bits. 2024-02-29 08:45:41 -06:00
Herbert Wolverson 9de53853e7 Improve comments on metadata 2024-02-29 08:28:33 -06:00
Herbert Wolverson f84798885b Add a stub for listing top flows by current bitrate. 2024-02-28 15:03:03 -06:00
Herbert Wolverson 7516715874 Add a real-time flow counter to the display and API. 2024-02-28 14:44:20 -06:00
Herbert Wolverson 393c3adc2a * Restore the tc_dissector, I didn't mean to remove that. 
* Add SKB metadata support for pass-through and eliding the
  second LPM check if we can. Checks at run-time to see if
  it is possible.
 2024-02-28 12:52:04 -06:00
Herbert Wolverson 95b7c9ac5c No more second stage dissector. 2024-02-28 10:32:02 -06:00
Herbert Wolverson 22d56a71bf Correct the ordering of src/dst ports in the flow map. 2024-02-28 09:56:59 -06:00
Herbert Wolverson ff4c070b7d Improve the JSON transfer data by listing protocol explicitly for flowbee. 2024-02-28 09:51:43 -06:00
Herbert Wolverson 29b0e07867 Massive improvement in RTT tracking accuracy. 2024-02-28 09:29:20 -06:00
Herbert Wolverson e98a1864ad Add a new api call - api/flows/dump_all - that lists all recent flows that have been collected. Intended for debugging. 2024-02-27 16:22:22 -06:00
Herbert Wolverson df2b9dfe32 Integrate the flow data into the moving average system for RTTs, giving a better spread of results. 2024-02-27 14:54:29 -06:00
Herbert Wolverson f33d22faa0 In progress. The RTT data from the flows system is integrated into the high-level per-IP RTT tracker. 2024-02-27 14:06:13 -06:00
Herbert Wolverson b7c02d251d Warning fix 2024-02-27 11:56:36 -06:00
Herbert Wolverson 8fa53782c6 Work in progress. Add endian conversions. Add Rust interface for iterating the flows list. Add a temporary interface that dumps flow data to the console to prove that it works. 2024-02-27 11:53:18 -06:00
Herbert Wolverson bfc9b8227c Skeleton for including UDP and ICMP flow data. 2024-02-27 10:38:17 -06:00
Herbert Wolverson c7df905e24 WIP - cleanup done, TCP is mostly working on the server-side. 2024-02-27 10:30:48 -06:00
Herbert Wolverson 0fd6b29e6c Considerably cleaned up flow tracking system. 2024-02-26 20:36:37 -06:00
Herbert Wolverson 23487f3328 The flows now include their very own rate estimator, and only count RTT when rate exceeds a threshold - 5mbps right now, which is probably not a good choice. 2024-02-14 14:56:46 -06:00
Herbert Wolverson 1ca595b16c Hacking away - remove the fin/reset flags because they shouldn't be needed. 2024-02-14 14:40:58 -06:00
Herbert Wolverson 295caaad91 Very early days - kinda/sorta working per-flow work. 2024-02-14 13:51:55 -06:00
Herbert Wolverson 088eed555a Correct URL in device_weights.rs 2024-02-08 13:21:56 -06:00
Herbert Wolverson 05d2a398a3 Add a device_weights() call to the Python API 
Adds the ability to call device_weights() from liblqos_python.
This returns a set of objects containing circuit_id and weight.

1. The weights are loaded from ShapedDevices.csv using HALF the
   maximum download mbps.
2. If LTS is enabled, a REST call is made to an LTS API located at
   https:/stats.libreqos.io/api/device_weights and the results
   merged in (overwiting existing entries).
3. The merged weights are returned as a Python array of classes.

Example Python code:

```python
from liblqos_python import get_weights;
weights = get_weights();
for w in weights:
    print(w.circuit_id + " : " + str(w.weight));
```
 2024-02-08 12:01:12 -06:00
Herbert Wolverson d412851560 WIP 2024-02-08 08:32:58 -06:00
Herbert "TheBracket 206ba2641f Merge pull request #459 from LibreQoE/unifig 
Unified Configuration System
 2024-02-07 12:55:32 -06:00
Herbert Wolverson 7180dd7950 BUGFIX: We weren't correctly honouring requests to use a Linux bridge. This patch fixes it. 2024-02-05 11:33:32 -06:00
Herbert Wolverson f7c9e82173 Just starting - display some packet info 2024-02-05 11:28:31 -06:00
Herbert Wolverson 3ab165a591 Re-add InfluxDB Support to the unified configuration. 2024-02-04 21:29:02 -06:00
Herbert Wolverson 1ee3543eb1 Add documentation updates for the unified config 2024-02-04 20:55:31 -06:00
Herbert Wolverson 699e265850 Add some small documentation for adding config items. 2024-01-27 10:12:57 -06:00
Herbert Wolverson e0e81ed715 Finish porting the Sonar integration over 2024-01-27 10:06:46 -06:00
Herbert Wolverson 8a812fe190 Update the parts of the Sonar configuration that are ported. Still have to figure out some types. 2024-01-27 09:46:46 -06:00
Herbert Wolverson 9c2747de37 Update the scheduler to use the latest config system 2024-01-27 09:42:11 -06:00
Herbert Wolverson ed5cbbc29c Add the beginnings of the Sonar integration into the config system. Add defaults for every Python field to reduce the chances of getting a partial config. 2024-01-27 09:30:54 -06:00
Herbert Wolverson a24c51065f Fix on-a-stick typo messing up vlans 2024-01-26 16:36:36 -06:00
Herbert Wolverson d851161670 General warning and documentation pass. 2024-01-26 16:33:35 -06:00
Herbert Wolverson 50bc071db1 Fix issue with queueRefreshInterval being optional but not treated as such, and breaking things. 2024-01-26 16:33:26 -06:00
Herbert Wolverson 84f7562d54 Add line to example file to complete on-a-stick outline. 2024-01-26 16:08:29 -06:00
Herbert Wolverson b1b01b89a6 Update cargo packages to latest releases, update rocket and sysinfo to newer syntaxes. 2024-01-26 12:50:30 -06:00
Herbert Wolverson 8cd8b7d404 Update example.toml to include the powercode required fields 2024-01-26 11:49:24 -06:00
Herbert Wolverson 3769bc3d84 Merge branch 'main' into unifig 2024-01-26 11:42:19 -06:00