Introduction

Congestion control algorithms (CCAs) are the unsung heroes of the internet, silently managing how fast data travels across networks. Among them, CUBIC stands out as the default CCA in Linux, standardized in RFC 9438, governing the majority of TCP and QUIC connections on the public internet. At Cloudflare, our open-source QUIC implementation, quiche, relies on CUBIC as its default congestion controller. This places CUBIC's code in the critical path for a substantial portion of the traffic we handle. In this article, we unravel a peculiar bug where CUBIC's congestion window (cwnd) becomes permanently stuck at its minimum, unable to recover from a congestion collapse event. The story begins with a Linux kernel change intended to align CUBIC with the app-limited exclusion rules of RFC 9438 §4.2-12—a fix for TCP that, when ported to quiche, exposed unexpected behaviours. The outcome? An elegant, near-one-line fix that broke the cycle.

CUBIC's Logic in a Nutshell

Before diving into the bug, a quick refresher on CCA fundamentals helps set the stage. The central knob a CCA turns is the congestion window (cwnd)—a sender-side cap on how many bytes can be in flight (sent but not acknowledged) at any moment. A larger cwnd lets the sender push more data per round trip; a smaller one throttles it. Every loss-based CCA, CUBIC included, is a policy for growing cwnd when the network appears healthy and shrinking it when it doesn't.

In essence, CCAs aim to maximize data transfer by inferring available bandwidth. Loss-based algorithms like CUBIC operate on a fundamental premise: if there's no packet loss, increase the sending rate (boost bandwidth utilization); if loss occurs, assume network capacity is exceeded and back off (reduce utilization). This logic rests on several assumptions that have been revisited over the years, but we'll save that discussion for another time.

The Symptom: A Test That Failed 61% of the Time

Our investigation began with reports of unexpected failures in our ingress proxy integration test pipeline. This erratic behaviour surfaced in tests where CUBIC was evaluated under heavy loss early in the connection. Recovery after a congestion collapse is an uncommon regime, but exactly the regime a congestion controller exists to handle. Most tests exercise steady-state and growth phases; far fewer probe what happens at minimum cwnd, after the connection has been beaten down. Bugs in this corner of the state space are invisible in throughput data.

The failing test simulated a scenario where the connection experienced severe packet loss early on, forcing CUBIC to reduce its cwnd to the minimum. Instead of recovering and gradually increasing the window as the network cleared, the cwnd remained pinned at its floor. The connection never escaped this collapsed state, leading to persistent underutilization and test failures—61% of the time, to be precise.

Root Cause: The App-Limited Exclusion in a New Context

The problem traced back to a Linux kernel change that added an app-limited exclusion check to CUBIC. In TCP, this change prevented the congestion window from being artificially limited when the application itself was not sending enough data to fill the window. The logic was sound for TCP, but when the same logic was ported to quiche—our QUIC implementation—it interacted poorly with QUIC's different pacing and loss detection mechanisms.

Specifically, the app-limited exclusion caused CUBIC to misinterpret a normal app-limited state as a congestion event, preventing it from increasing cwnd even when the network had spare capacity. Once cwnd hit its minimum value, the exclusion logic kept it there indefinitely, because every subsequent opportunity to increase cwnd was blocked by the app-limited condition. This created a deadlock: the connection could never recover.

The Fix: A One-Line Correction

The solution turned out to be elegantly simple. By adjusting the condition under which CUBIC considered the connection to be app-limited, we broke the cycle. The change effectively said: unless the sender is truly idle (i.e., has no data to send), do not apply the app-limited exclusion that was preventing cwnd growth. This near-one-line fix restored CUBIC's ability to recover from congestion collapse, passing all tests reliably.

This bug highlights how a well-intentioned optimization in one protocol (TCP) can introduce unexpected behaviours when ported to another (QUIC). It also underscores the importance of testing edge cases—like congestion recovery at minimum cwnd—which are often overlooked.

For more details, see our original blog post.

Lessons Learned

This episode offers several takeaways for network engineers and protocol implementors:

• Cross-protocol porting requires caution: Assumptions valid in TCP may not hold in QUIC due to differences in loss detection, pacing, and app-limited states.
• Test edge cases thoroughly: Congestion collapse recovery at minimum cwnd is rare but critical—bugs there can cripple real connections.
• Simplicity wins: The fix was a single line, demonstrating that sometimes the most elegant solutions are the smallest.

With this fix deployed, quiche's CUBIC implementation now correctly handles early heavy loss, ensuring robust recovery for millions of QUIC connections.