Packet loss and other networking-related issues have plagued networks for decades. Often, network connections fail or slow down, impacting user experiences. To resolve these problems, network operators are often faced with the daunting task of troubleshooting using traditional monitoring tools that lack comprehensive visibility. Moreover, troubleshooting is often performed from a single, usually affected, vantage point, further limiting visibility.
When diagnosing an issue, network operations (NetOps) teams must consider multiple possible contributing reasons for potential failures, such as blackholing, dropping traffic due to overutilization, dirty fiber, and more. Usually, these problems are related to issues on the data plane. However, occasionally, they arise due to control plane behavior. More specifically, routing decisions, or the lack thereof, can significantly contribute to blackholing.
Today, we are announcing a new tool for network operators: the BGP Stuck Route Observatory, which helps identify one of the common reasons for blackholing—BGP zombies. BGP zombies, or stuck routes, are routes that persist in routers despite their withdrawal by the origin Autonomous System (AS).
Just like our popular Internet Outages Map, the BGP Stuck Route Observatory is free and available to all.
Stuck Routes? BGP Zombies?
In Border Gateway Protocol (BGP), an AS, a network under single administrative control, announces a prefix (a block of IP addresses) that it owns to its neighboring ASes. These ASes subsequently propagate the announcement to their neighbors, continuing this process iteratively until the prefix becomes reachable from all ASes across the Internet. Once the originating AS no longer wishes for the prefix to be reachable via this specific path or due to reasons such as peering going down, congestion, or maintenance, it withdraws the prefix from its neighbors. Similar to the announcement, the withdrawal is propagated to all ASes, and the prefix is removed from the global routing table.
BGP zombies (stuck routes) occur during the withdrawal process when a router in an AS fails to propagate the withdrawal message. Its neighbor ASes are not notified and will continue to consider the route valid. All subsequent neighbors will do the same. These stuck routes falsely indicate that a prefix is still reachable even though the path does not exist for the associated route.
Why Do Stuck BGP Routes Happen? What’s the Impact?
Stuck BGP routes can occur due to software bugs, hardware and configuration issues, or BGP protocol flaws that prevent routers from properly withdrawing or updating routes in their BGP routing tables.
Stuck BGP routes can lead to suboptimal routing decisions, network instability, routing loops, and disruptions in traffic flow. These issues cause operational problems, including performance deterioration and outages.
Why Are Stuck BGP Routes So Hard To Catch?
Automated detection of BGP zombies in operational networks is inherently difficult due to the lack of ground-truth data. Without knowing the intentions of each network operator, it is challenging to infer with confidence why a withdrawal message was sent to an Autonomous System. Beyond that, detecting stuck routes requires visibility from multiple vantage points, strategically deployed across the globe, spanning Tier 1, Tier 2, and other networks such as Internet exchanges.
How Does the BGP Stuck Route Observatory Work?
To address these challenges, the ThousandEyes BGP Stuck Route Observatory uses beacon prefixes to detect stuck routes. Beacon prefixes are prefixes that are periodically announced and withdrawn at specific times. Knowing exactly when a prefix should be removed from the routing table globally increases confidence in identifying a route as stuck.
The Observatory uses a new beacon methodology that incorporates significant improvements to beacon advertisements, giving us the ability to track how long it takes to withdraw a prefix, whether the number of stuck routes is increasing or decreasing, and how long operators take to identify. (To learn more about this methodology, check out the BGP zombies blog series.)
This approach allows the BGP Stuck Route Observatory to provide a streamlined way to determine whether your AS is potentially impacted by certain ASes on the path (such as an upstream provider) or whether your AS might have an error (in other words, the issue might be happening in your network). By processing data ingested from hundreds of BGP monitors strategically deployed across the globe, the BGP Stuck Route Observatory detects stuck routes based on our beacon prefix advertisements.
When you enter an Autonomous System Number (ASN) in the BGP Stuck Route Observatory’s lookup field, you will be presented with results that indicate whether the ASN is not affected, potentially affected (by another ASN), or potentially the source of the issue (i.e., it is contributing to the problem). The results will also show the different AS paths where we observed the potential issue.
If your ASN is indicated as potentially affected or as possibly having an error, you can reach out to the ThousandEyes team to gain more insights about the scope of the problem detected by the BGP Stuck Route Observatory, as well as guidance on identifying where the issues are happening.
Why Does Catching Stuck BGP Routes Matter?
Zombie or stuck routes remain prevalent on the Internet, despite improvements to the BGP protocol (RFC 9687) and hardware. As mentioned, these stuck BGP routes can decrease performance or cause outages, ultimately leading to subpar digital experiences for your users. To guard against these impacts, systematic detection and root cause analysis is vital, helping you remove stuck routes faster and pinpoint or discover flaws that cause them.
Additionally, monitoring your participation in stuck route outbreaks and acting on them is an important part of being a responsible member of the global Internet community. The Internet is a connected graph; one AS’s flaw can cause problems for any other AS’s operations on the Internet.
When diagnosing any network problem, identifying that an issue is happening is only a piece of the puzzle. You also need to quickly determine where the issue is occurring. With the ThousandEyes BGP Stuck Route Observatory, NetOps teams have a powerful tool that helps answer the “if” and provides insights on the possible “where.”