LibreQoS/docs/v2.0/htb_fq_codel_cake.md

HTB + fq_codel + CAKE: Detailed Queueing Behavior in LibreQoS

This page is the canonical queueing deep-dive companion to LibreQoS Backend Architecture.

It explains:

1. Why LibreQoS layers HTB with leaf qdiscs (fq_codel or CAKE)
2. How fq_codel works in practice
3. How CAKE works in practice
4. When to choose fq_codel vs CAKE
5. Operator troubleshooting and observability patterns

1) Why These Three Components Exist Together

In production LibreQoS:

1. HTB provides hierarchical rate policy (rate, ceil, borrow, hierarchy)
2. fq_codel or CAKE provides per-flow queue service and AQM behavior inside each shaped envelope

This split is intentional:

• Policy problem: "How much can this class send?" -> HTB
• Queueing problem: "Which packet should send next while controlling latency?" -> fq_codel/CAKE

2) Runtime Placement in LibreQoS

Conceptually, packets pass through:

1. mq root
2. HTB hierarchy
3. leaf qdisc per shaped class (CAKE by default, fq_codel optional)

Operationally, this is commonly:

mq root -> per-CPU HTB parents -> per-circuit HTB classes -> leaf qdisc (cake diffserv4 or fq_codel)

Each HTB class has a child qdisc attachment point. If no explicit leaf qdisc is attached, behavior falls back to kernel default queueing for that class.

Practical LibreQoS behavior model:

1. Default behavior uses HTB + cake diffserv4 for shaped circuits.
2. TreeGuard can dynamically switch circuit directions between cake diffserv4 and fq_codel based on low-load/RTT guardrails.
3. Current builds keep TreeGuard enabled by default for circuit SQM management, while runtime link virtualization is disabled by default in favor of static queue policy from lqos_topology.

See TreeGuard for configuration and rollout details.

3) HTB Summary for AQM Users

3.1 Core HTB mechanics

1. Tokens are consumed by packet bytes and refill over time.
2. rate defines guaranteed service.
3. ceil defines borrowable maximum when parent capacity exists.
4. Children borrow only from ancestor spare capacity.
5. Sibling contention is affected by prio, scheduler behavior, and class proportions.

3.2 Key HTB controls

• rate, ceil
• prio
• burst, cburst
• quantum, r2q
• default class minor ID (Linux HTB concept; see LibreQoS behavior note below)

3.3 Why this matters for fq_codel and CAKE

fq_codel and CAKE do not replace HTB's hierarchy and class-rate policy. They shape queue service inside the class envelope HTB allows.

3.4 Undefined traffic behavior in LibreQoS

LibreQoS behavior is explicit here:

1. traffic not mapped to a shaped circuit is passed through
2. LibreQoS does not direct undefined traffic into HTB default classes
3. HTB default class behavior still exists in Linux tc, but it is not how LibreQoS handles undefined traffic

Operationally, this means troubleshooting undefined traffic starts with classification/mapping validation, not with HTB default-class tuning.

3.5 Compact HTB skeleton (reference pattern)

Illustrative Linux HTB+leaf pattern:

tc qdisc add dev <ifname> root handle 1: htb default 30
tc class add dev <ifname> parent 1: classid 1:1 htb rate 1gbit ceil 1gbit
tc class add dev <ifname> parent 1:1 classid 1:10 htb rate 700mbit ceil 1gbit prio 1
tc class add dev <ifname> parent 1:1 classid 1:20 htb rate 300mbit ceil 1gbit prio 2
tc class add dev <ifname> parent 1:1 classid 1:30 htb rate 10mbit ceil 1gbit prio 7
tc qdisc add dev <ifname> parent 1:10 cake diffserv4
tc qdisc add dev <ifname> parent 1:20 fq_codel
tc qdisc add dev <ifname> parent 1:30 cake diffserv4
tc filter add dev <ifname> protocol ip parent 1:0 prio 1 u32 match ip src <A>/32 flowid 1:10
tc filter add dev <ifname> protocol ip parent 1:0 prio 2 u32 match ip src <B>/32 flowid 1:20

In LibreQoS, queue/class commands are generated automatically and undefined traffic is passed through rather than sent to an HTB default class.

4) fq_codel Deep Dive

4.1 What fq_codel is

fq_codel combines:

1. stochastic flow queueing (hashed queues)
2. DRR-style fair scheduling across queues
3. CoDel AQM per queue

Core references:

• tc-fq_codel(8)
• RFC 8290 (Flow Queue CoDel)

4.2 Scheduler behavior and sparse-flow benefit

FQ-CoDel maintains "new" and "old" active queue lists. Newly active queues are prioritized over persistent backlogged queues, which naturally benefits sparse/interactive traffic.

It also uses byte-credit (quantum) scheduling so fairness is byte-oriented, not packet-count oriented.

4.3 Flow hashing model

By default, packets are classified using a 5-tuple hash into a configurable number of buckets (flows). Hash collisions are possible and are a known tradeoff of stochastic queueing.

4.4 fq_codel parameters that operators actually tune

tc-fq_codel(8) parameters worth knowing:

1. limit PACKETS: hard queue packet cap (default 10240)
2. memory_limit BYTES: memory cap (default 32MB), lower of limit and memory cap is enforced
3. flows NUMBER: hash buckets (default 1024, set at creation time)
4. target TIME: acceptable minimum persistent delay (default 5ms)
5. interval TIME: CoDel control window, generally order of worst RTT through bottleneck (default 100ms)
6. quantum BYTES: DRR deficit quantum (default 1514)
7. ecn/noecn: ECN on/off (ecn is default in fq_codel)
8. ce_threshold TIME: shallow ECN marking threshold for DCTCP-style use
9. ce_threshold_selector VALUE/MASK: apply CE threshold only to selected traffic
10. drop_batch: max drop batch when limits exceed (default 64)

4.5 fq_codel observability (tc -s qdisc show)

Common counters/fields to inspect:

1. dropped, overlimits, requeues
2. drop_overlimit
3. new_flow_count
4. ecn_mark
5. new_flows_len, old_flows_len
6. backlog

Interpretation pattern:

1. Verify queue pressure exists (backlog, requeues, overlimits)
2. Check whether AQM signaling is engaged (ecn_mark, dropped)
3. Correlate new_flows_len/old_flows_len with traffic mix (sparse vs bulk)

5) CAKE Deep Dive

5.1 CAKE architecture

CAKE integrates multiple layers in one qdisc:

1. deficit-mode shaper
2. priority queue (tins)
3. flow isolation (DRR++)
4. AQM (COBALT, combining CoDel + BLUE)
5. packet management and overhead compensation

Core references:

• tc-cake(8)
• Bufferbloat CAKE and CakeTechnical pages
• Piece of CAKE paper (cake.pdf)

5.2 Shaped vs unshaped operation

When bandwidth is set, CAKE's shaper and its derived tuning drive tin thresholds and timing behavior.

Without shaping (unlimited), CAKE still provides queue service and AQM logic, but tin and service behavior are no longer operating against a fixed shaped bottleneck target.

5.3 Flow isolation modes

CAKE supports multiple fairness modes:

1. flowblind (no flow isolation)
2. flows (5-tuple flow fairness)
3. srchost, dsthost, hosts
4. dual-srchost, dual-dsthost
5. triple-isolate (default in tc-cake(8))

Operational note:

• triple-isolate is a safe general default when you need both per-flow and host fairness controls.

5.4 NAT awareness

nat/nonat controls whether CAKE performs NAT lookup before applying flow isolation.

Why it matters:

• Without nat, fairness sees post-NAT addresses only.
• With nat, fairness can better represent internal hosts behind NAT (when NAT is on the same box/path).

5.5 DiffServ modes and tins

Main priority presets:

1. besteffort (single tin, no priority queue)
2. diffserv3
3. diffserv4
4. diffserv8
5. precedence (legacy, discouraged in modern deployments)

tc-cake(8) currently documents diffserv3 as default, while LibreQoS typically uses cake diffserv4 as operator-facing default policy.

5.6 LibreQoS diffserv4 DSCP mapping

LibreQoS commonly runs CAKE with diffserv4. Practical class mapping:

1. Latency Sensitive: CS7, CS6, EF, VA, CS5, CS4
2. Streaming Media: AF4x, AF3x, CS3, AF2x, TOS4, CS2, TOS1
3. Best Effort: CS0, AF1x, TOS2, and unrecognized codepoints
4. Background Traffic: CS1

Known codepoints in common operator use:

1. CS1 (Least Effort)
2. CS0 (Best Effort)
3. TOS1 (Max Reliability / LLT "Lo")
4. TOS2 (Max Throughput)
5. TOS4 (Min Delay)
6. TOS5 (LLT "La")
7. AF1x
8. AF2x
9. AF3x
10. AF4x
11. CS2
12. CS3
13. CS4
14. CS5
15. CS6
16. CS7
17. VA
18. EF

RFC 4594-style traffic-class framing (high-level):

1. Network Control: CS6, CS7
2. Telephony: EF, VA
3. Signaling: CS5
4. Multimedia Conferencing: AF4x
5. Realtime Interactive: CS4
6. Multimedia Streaming: AF3x
7. Broadcast Video: CS3
8. Low Latency Data: AF2x, TOS4
9. Ops/Admin/Management: CS2, TOS1
10. Standard Service: CS0 and unrecognized codepoints
11. High Throughput Data: AF1x, TOS2
12. Low Priority Data: CS1

fq_codel note:

1. fq_codel has no CAKE tin model and no CAKE-style DSCP class scheduler.
2. DSCP marking can still be used by external classification/policy, but not via CAKE diffserv4 tin behavior.
3. In LibreQoS, DSCP priority behavior described above applies when CAKE with diffserv4 is selected.

5.7 Overhead and framing compensation

CAKE can account for link-layer overhead/framing using:

1. overhead N
2. mpu N
3. atm, ptm, noatm
4. shortcut keywords (ethernet, docsis, etc.)
5. raw and conservative

Operator rule:

• If overhead/framing is wrong, shaping accuracy is wrong. Validate with realistic traffic tests.

5.8 GSO handling (split-gso)

By default, CAKE splits GSO superpackets to reduce latency impact on competing flows, especially at lower rates.

At very high link rates (e.g. >10 Gbps), no-split-gso can improve peak throughput, but may trade away latency smoothness.

5.9 ACK filtering

CAKE supports:

1. ack-filter
2. ack-filter-aggressive
3. no-ack-filter (default)

Best use case is asymmetric links where upstream ACK load limits downstream goodput. Apply conservatively and validate with application traffic, not only synthetic tests.

5.10 Ingress mode and autorate

ingress mode changes accounting and tuning for downlink shaping realities (including counting dropped packets as already-transited data).

autorate-ingress can estimate capacity from arriving traffic and is primarily useful on highly variable links (for example some cellular paths). It cannot estimate bottlenecks downstream of where CAKE is attached.

5.11 DOCSIS modem headroom

When shaping downstream traffic behind a DOCSIS modem, set the shaper below the modem's actual provisioned downstream rate—usually 85–95% after confirming the real rate with traffic tests. Otherwise, traffic can queue in the modem's downstream buffer (notably on modems with large buffers such as some Puma 6 devices), where LibreQoS and CAKE/fq_codel cannot manage it; latency then rises even though the LibreQoS shaper appears correctly configured. This is the same downstream bottleneck that autorate-ingress cannot observe. Keep the queue under CAKE or fq_codel by leaving enough headroom, and adjust the percentage for the modem and service tier rather than treating one value as universal.

5.12 CAKE observability (tc -s qdisc show)

Useful fields commonly present:

1. top-level: dropped, overlimits, backlog, memory used, capacity estimate
2. per tin: thresh, target, interval
3. delay telemetry: pk_delay, av_delay, sp_delay
4. hashing: way_inds, way_miss, way_cols
5. signaling: drops, marks
6. ACK filtering: ack_drop
7. queue activity: sp_flows, bk_flows, un_flows, max_len, quantum

Interpretation pattern:

1. check if tins and thresholds match intended policy
2. inspect delay EWMAs by tin
3. correlate drops/marks with user-visible latency and throughput
4. monitor hash-collision indicators (way_cols) under peak concurrency

6) LibreQoS Policy Framing: CAKE vs fq_codel

For LibreQoS operators, start with platform behavior first:

1. Default operation is cake diffserv4 in HTB class leaves.
2. TreeGuard (upcoming feature) can move selected circuit directions to fq_codel during sustained low-load conditions and back to cake as utilization/guardrail pressure rises.
3. Manual per-circuit sqm overrides still provide explicit operator control.

Use this matrix for tradeoff context:

Dimension	fq_codel	CAKE
Configuration complexity	Lower	Higher (more integrated features)
Resource footprint at scale	Often lower	Often higher
Integrated shaping features	No (needs parent shaper like HTB)	Yes (deficit-mode shaper built in)
DiffServ/tin behavior	Basic/indirect	Strong native tin model
Host isolation modes	Not CAKE-style host modes	Rich host/flow isolation modes
Overhead compensation	Limited	Rich built-in overhead/framing controls
Asymmetric-link ACK optimizations	None	ACK filtering modes available
Best fit	Large queue count with tight resources	Mixed traffic where policy richness and smoothness matter

7) LibreQoS Operator Notes

From maintainer testing and deployment feedback:

1. fq_codel has no intrinsic rate limiting; it relies on HTB for rate policy.
2. fq_codel and CAKE both keep per-flow state tables, so RAM/hash behavior matters at high queue counts.
3. CAKE and HTB are viable down to very low and asymmetric rates in LibreQoS deployments.
4. A top-level "sandwich" limiter pattern using HTB+fq_codel is a practical deployment option in some environments.
5. Some dashboard traffic views can reflect pre-drop context; interpret counters with drop/mark semantics together.
6. "Only hard saturation benefits from AQM" is too narrow; AQM and fair queueing can improve latency when managed queues are under burst/contended pressure, even before interface-wide utilization is fully pegged.
7. Older shared/default-bucket discussion patterns reinforce that queue dynamics, not only "link fully pegged" moments, drive AQM value; in current LibreQoS, undefined traffic is pass-through, so apply this principle to managed HTB leaf queues.

8) Practical Observability Workflow

Start with:

tc -s qdisc show dev <ifname>
tc -s class show dev <ifname>

Then:

1. confirm HTB classes exist where expected
2. confirm leaf qdisc type (cake vs fq_codel) per class
3. inspect class and qdisc counters together
4. verify directionality (ingress/egress) matches the problem being diagnosed
5. correlate with user-visible latency/throughput, not counters alone

9) Common Misunderstandings

1. "fq_codel or CAKE replaces HTB"
   • False for LibreQoS hierarchy operation; HTB remains the policy envelope.
2. "Undefined traffic goes to an HTB default queue in LibreQoS"
   • False; LibreQoS passes undefined traffic through.
3. "Only hard saturation events benefit from AQM"
   • False. See Why AQM still helps even below line-rate saturation.
4. "Leaf qdisc tuning can fix broken hierarchy/class mapping"
   • False; mapping/hierarchy errors must be corrected first.
5. "fq_codel can rate-limit on its own"
   • False; use HTB (or another shaper) for explicit rate policy.

10) HTB HOWTO Context (Historical, Still Useful)

Classic HTB HOWTO material remains useful for operator mental models when translated to modern LibreQoS:

1. classify traffic
2. schedule queue service
3. shape at the bottleneck (or immediately upstream)
4. define explicit class intent with rate/ceil

Modern translation notes:

1. confirm behavior with tc -s counters, not assumptions about package defaults
2. keep classifier order intentional (specific rules before broader matches)
3. include explicit catch-all handling in manual tc deployments
4. in LibreQoS specifically, undefined traffic is pass-through unless mapped into shaped hierarchy

11) References

• LibreQoS Backend Architecture
• tc-htb man page (man7)
• tc-fq_codel man page (man7)
• tc-cake man page (man7)
• FlowQueue-Codel RFC 8290
• IANA DSCP Registry
• CAKE wiki (Bufferbloat)
• CAKE technical notes (Bufferbloat)
• FQ_Codel wiki (Bufferbloat)
• CAKE vs FQ_Codel (Bufferbloat)
• CoDel/fq_codel project wiki index (Bufferbloat)
• Toke Høiland-Jorgensen, Dave Taht, Jonathan Morton. Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways, IEEE LANMAN, 2018.