Path Quality Part 3: Is BBR the Future of Congestion Avoidance?

In our previous research titled "Path Quality Part 1: The Surprising Impact of 1% Packet Loss," we established that a 1% packet loss can cause more than a 70% drop in throughput in environments using CUBIC, the default congestion avoidance algorithm in all major operating systems. This drop occurs regardless of the network's topology, be it symmetric or asymmetric. These findings naturally indicate the importance of addressing even minor levels of packet loss and underscores the significance of network monitoring.

Recognizing that packet loss can have compounding adverse effects from an early stage, we wanted to examine the impact on throughput in environments using other congestion avoidance algorithms, namely Bottleneck Bandwidth and Round-Trip Time (BBR).

Enter BBR

CUBIC was designed to overcome the limitations of previous TCP congestion control algorithms in handling high-speed, long-distance networks by implementing a cubic polynomial function. The use of a cubic function, allows CUBIC to handle congestion with significantly greater complexity compared to the linear functions employed by algorithms, such as Reno. However, CUBIC must rely entirely on its packet loss and congestion window to govern the forwarding rate.

Loss-based congestion control has its limitations. Loss happens before congestion for network gear with shallow buffers that are frequently exceeded by brief bursts of traffic. Quality of Service (QoS) also creates challenges for loss-based TCPs, such as Reno and CUBIC. In paths with active queue management (AQM), there can be active loss with low latency. In these types of scenarios, the response of loss-based congestion controls can often result in a dramatic and often overly aggressive drop in the forwarding rate. Additionally, loss-based control can also pose problems for larger buffer network equipment, resulting in constant deep queues.

Google developed BBR to overcome the limitations of loss-based algorithms. BBR is designed to optimize network utilization and throughput by continuously probing for available bandwidth. The algorithm has the logic to adjust the forwarding rate continuously according to available bandwidth. This combination of continuous probing and dynamic adjustment of the forwarding rate gives BBR its unique capabilities.

How It Works

At a high level, the congestion avoidance model is based on a particular algorithm that uses various input parameters, including throughput, delay, ECN, and loss. It maintains a state machine that adapts quickly and closely monitors and probes Round-trip Time (RTT) as a signal of queue utilization at the bottleneck. Unlike Reno and CUBIC, it doesn't rely on packet loss as the only signal to back off. Instead, it uses RTT spikes to indicate queuing. The premise here is that path queue health can be continuously gauged by BBR’s recurring measurement of RTT, enabling the TCP to adjust to changes in end-to-end performance dynamically. Higher latencies correspond to lengthier queue depths. This approach is designed to provide high-throughput utilization with low queue pressure. You can find details of the algorithm's inner workings in this research paper.

At a high level, BBR does the following:

• Bandwidth estimation estimates available bandwidth by measuring packet delivery rates. The algorithm utilizes a concept called pacing to ensure a steady flow of packets without causing undue congestion. 

• Round-trip time (RTT) estimation maintains an estimate of the connection's minimum RTT and base RTT (RTT when there is no congestion). RTT variations are used to adjust the pacing rate, ensuring smooth transmission and reduced latency.

• Bottleneck detection identifies the bottleneck link in the network path through various techniques like probing for increased delivery rates and utilizing RTT feedback.

• Congestion window management adjusts the forwarding rate by maintaining pacing gain and probing.

• Low latency operation tries to keep the queue size low, which helps in reducing latency.

The Key Differences Between BBR and CUBIC

One of the congestion avoidance algorithm’s responsibilities is Congestion Window Adjustment. CUBIC adjusts the congestion window based on a cubic function, reacting strongly to packet loss (causing forwarding rate drops by close to 30%). In contrast, BBR dynamically adjusts the forwarding rate based on bandwidth and RTT estimations, avoiding unnecessary loss. 

Regarding Bandwidth Estimation, CUBIC relies on packet loss as an indicator of congestion, while BBR actively probes for available bandwidth and adjusts the forwarding rate, minimizing latency. 

Latency-wise, CUBIC prioritizes throughput over latency, potentially leading to increased latency under heavy congestion. Meanwhile BBR maintains low latency by continuously monitoring network conditions and adjusting congestion control parameters accordingly.

What Did We Test?

As a reminder, we tested throughput using iperf3 in the topology with five Linux hosts configured for forwarding, as part of which routing was configured statically. We configured routing to replicate a symmetric networking topology, which is common in many enterprise environments (for its ease of troubleshooting and security), and an asymmetric network topology (which resembles traffic on the Internet). We introduced the loss using the Linux tool "tc" (more specifically, "netem," a tc module that has capabilities to introduce loss and latency, among other conditions). 

Establishing the Baseline Using BBR

As Table 1 below shows, in the symmetric network topology where the client is configured with BBR, we achieved an average of 868.50 Mbps. In the CUBIC experiment, we got 804.6 Mbps for symmetric network topology, a 7.9% difference compared to BBR. 

Regarding the asymmetric network topology using BBR, we achieved 827.2 Mbps (Table 2) compared to 864.1 Mbps in the same topology using CUBIC, a 4.4% difference on average.

Table 1: Establishing baseline in a symmetric topology using BBR

Table 2: Establishing baseline in an asymmetric topology using BBR

Introducing the Loss

Upon introducing the 1% packet loss, we achieved 794.06 Mbps in the symmetric network topology, as shown in Table 3, and 763.42 Mbps in the asymmetric network topology (Table 4), a 3.85% decrease compared to symmetric network topology. 

Table 3: Achieved throughput in a symmetric topology with 1% packet loss

Table 4: Achieved throughput in an asymmetric topology with 1% packet loss

Throughput in the symmetric networking topology decreased by 8.57% compared to baseline, while in the asymmetric network topology, the decrease was even smaller at 7.7%. 

Overall results

Table 5: Overall results in a symmetric network topology

Table 6: Overall results in an asymmetric network topology

Comparison With CUBIC at 1% Packet Loss

In our first blog post on this topic, we published the results that we achieved using the default congestion avoidance algorithm, CUBIC. For the symmetric network topology, we achieved 235.51 Mbps, a 70.7% decrease compared to the baseline. However, in this round of testing, BBR achieved 794.06 Mbps, an 8.57% decrease compared to the baseline. That represents a 70.3% difference in achieved throughput between CUBIC and BBR!

In terms of the asymmetric network topology, we achieved a throughput of 222.49 Mbps using CUBIC, which resulted in a 74.2% decrease when compared to the baseline. On the other hand, we achieved a throughput of 763.4 Mbps using BBR, which resulted in a 7.7% decrease when compared to the baseline. This finding illustrates a significant difference of 70.8% between the achieved throughput using CUBIC and BBR. Our results were consistent with those we obtained for the symmetric network topology.

What Do We Expect, and Where Are We Now? 

Many see TCP as effective, fair, and efficient. Its efficiency is expected to make use of all available bandwidth, while at the same time taking into account other TCP flows that require fairness. The development of congestion avoidance algorithms from Tahoe to Reno to the current default CUBIC has been a journey of balancing fairness and efficiency. 

The mechanisms of these algorithms for dealing with packet loss are aggressive, with Reno backing off by 50% and CUBIC by 30%. However, these algorithms are slow when utilizing total bandwidth after a random loss event, which is inefficient for modern, high-capacity networks. 

The rapid improvements in ethernet over the last decade, with hundreds of gigabits per second of capacity per single interface, necessitate more to be done regarding congestion avoidance. 

However, BBRv1 has faced several initial challenges, with the primary concern of flow fairness, such as crowding out Reno and CUBIC flows, not reacting to ECN, being too loss-agnostic, and having low throughput for Wi-Fi paths. 

BBRv2 was released several years ago, addressing each of these issues and improving its overall performance.

Adoption, Adoption, Adoption

Enabling BBR is a straightforward process, requiring only a few sysctl commands on popular Linux distributions. The fact that it only needs to be enabled on the sender side further simplifies its adoption. BBR has been part of the Linux kernel for the good part of the past decade, and it has a reputable company behind it. The fact that major companies have successfully adopted it is a testament to its effectiveness and reliability. Enabling it on Windows is a matter of executing a few commands, as well. 

BBR demonstrates significant benefits in maximizing throughput in the face of packet loss. BBR’s goal is not necessarily to provide more bandwidth. While loss-based algorithms like Cubic, as well as delay-based TCPs like Vegas, can also achieve high throughput across loss-free paths, BBR is a hybrid algorithm designed to maintain bandwidth efficiency and provide better controls over buffer-induced latency. And importantly for our topic of packet loss, its use of both loss and delay enables it to maximize throughput across lossy as well as non-lossy links. 

Unsurprisingly, Google publicly announced that it is using BBR for its search engine, YouTube, and Google Cloud. Reports also suggest that other companies like Spotify, Dropbox, and Netflix are using it as well. If BBR is suitable for these large enterprises, is it good enough for you?

To learn more about ThousandEyes Internet and Cloud Intelligence Research, visit our research hub.