At Cisco ThousandEyes, we are dedicated to uncovering and addressing critical network issues that impact the global Internet. One such issue is the phenomenon of "BGP zombies," which can lead to infinite routing loops, suboptimal data paths, and network instability. In this post, we explain what BGP zombies are, how they occur, and the risks they pose to Internet routing. Using the ThousandEyes platform, we identify and analyze three real-world cases of BGP zombies that could potentially impact hundreds of Autonomous Systems (ASes). Our aim is to raise awareness about this recurring problem and foster discussions around potential solutions to mitigate its impact.
Background: BGP Zombies
Border Gateway Protocol (BGP) is the de facto standard protocol used to manage a network's reachability on the Internet. It is used to exchange routing information between different Autonomous Systems (ASes), ensuring data packets are routed across multiple networks efficiently. BGP enables routers to determine the best paths for data transmission based on various factors, such as network policies and path attributes.
One well-known but not widely discussed BGP issue involves stuck or “Zombie” BGP routes, which falsely indicate that a prefix is still visible even though the origin AS has withdrawn the associated route. Stuck BGP routes can occur due to misconfigurations, software bugs, network issues, or even inherent flaws in the BGP protocol that prevent routers from withdrawing or updating routes properly in their BGP routing tables. Stuck BGP routes can potentially lead to suboptimal routing decisions, network instability, and disruptions in traffic flow within a network, causing operational issues such as performance degradation and outages.
In BGP, an Autonomous System (AS) announces a prefix that it owns to its neighboring ASes. These ASes propagate the announcement to their own neighbors. Consequently, announcements of the prefix propagate to the rest of the Internet so that other ASes can reach that prefix. This reachability information is encoded in BGP Update messages. The two most important types of these messages are Announcements and Withdrawals:
- Announcement: This message contains information on how to reach the associated prefix(es). Among other attributes, it includes:
-
- AS Path: The sequence of ASes that propagated this message. Traffic directed toward the announced prefix will traverse these ASes to reach its destination.
- Next-hop: The IP address to which packets are initially sent in order to reach the announced prefix
-
- Withdrawal: A message sent by a router to its BGP neighbor to indicate that it no longer has a route to the withdrawn prefix(es).
The BGP AS graph is composed of numerous ASes; each AS is connected to a specific set of neighboring ASes and applies different policies during its BGP sessions based on business relationships. Thus, when a new announcement is made, an AS will receive multiple BGP messages regarding the same prefix but with different attribute values, which are processed by the BGP selection algorithm to determine the optimal route. One of the criteria used by this algorithm is the AS path length, which gives priority to routes with shorter AS paths. All of the different valid routes will be recorded in the Routing Information Base (RIB) of the router, and only the optimal one will be installed in the routing table to handle traffic directed toward the prefix.
We define stuck routes, or BGP zombies, as prefixes that remain in the RIBs of some routers even though the originating AS has withdrawn the prefix. For example, imagine the scenario depicted in Figure 1: AS1 advertises the prefix 2001:db8::/48, even though it owns its covering prefix 2001:db8::/32. At some point, AS1 sells the /32 prefix to AS2, and naturally, AS1 stops advertising the /48, and AS2 starts announcing the /32. If the /48 route of AS1 does not get completely withdrawn and instead remains in a dominant AS3 (e.g., Tier 1, IXP), at least some of the Internet traffic towards the /48 route will be forwarded based on the Zombie route towards AS1, causing a partial outage affecting AS2.
For example, if a user within ASY starts sending traffic towards the IP address 2001:db8::1, due to the Longest Prefix Match (LPM) strategy implemented by routers, the traffic will be forwarded along the path ASY [...] AS3 [...] AS1. However, the AS following AS3 (left branch) has only the route towards the prefix 2001:db8::/32, and therefore, when the traffic reaches it, it will forward it back to AS3, creating a loop. Eventually, the routers will drop the packets when they decrement the IPv4 Time to Live field to zero.
BGP Beacons
RIPE routing beacons are IPv4/IPv6 prefixes that are announced every four hours by specific RIPE Routing Information Service (RIS) collectors and are withdrawn two hours after each announcement. The Cisco ThousandEyes platform tracks the reachability of configured prefix targets over time. Using Cisco ThousandEyes, we illustrate the reachability of RIPE beacon prefixes. Figure 2 depicts this for the prefix 2001:7fb:fe01::/48 on 2024-07-01. Although the prefix is withdrawn during the pinpointed interval, its reachability does not drop to 0%; instead, it drops to 5.7% as three monitors do not send any withdrawals for the prefix.
Three Instances of BGP Zombies
At Cisco ThousandEyes, our research has identified a strong likelihood that a BGP beacon withdrawal from an origin AS can trigger a BGP Zombie outbreak. This means that BGP zombies appear regularly. Using custom IPv6 beacons, we examine three interesting instances of BGP zombies:
- A Tier 2 ISP in France: Our beacon prefix 2a0d:3dc1:2327::/48 remained in the RIBs of eight RIPE RIS peer routers from six peer ASes, which shared the common subpath “30781 5511 25091 8298 210312”. This indicates that AS30781 (Free Pro SAS, FR), a Tier 2 ISP in France with approximately 200 ASes in its customer cone, could be the root cause of the Zombie outbreak. We visualize the Zombie AS paths and highlight the possible root cause AS (b) in Figure 3.
The non-zero reachability is visible on the ThousandEyes platform, which reported 0.6% reachability for the withdrawn prefix 2a0d:3dc1:2327::/48.
- A Tier 2 ISP in Germany: Our beacon prefix 2a0d:3dc1:1737::/48 remained in the RIBs of seven peer routers from seven peer ASes, which shared the common subpath “24961 210312”. This indicates that AS24961 (WIIT AG, DE), a Tier 2 ISP in Germany with approximately 200 ASes in its customer cone, could be the root cause of the Zombie outbreak. We visualize the Zombie AS paths and highlight the possible root cause AS (a) in Figure 5.
The non-zero reachability is visible on the ThousandEyes platform, which reported 4.7% reachability for the withdrawn prefix 2a0d:3dc1:1737::/48.
- Another Tier 2 ISP in Germany: Our beacon prefix 2a0d:3dc1:2233::/48 remained in the RIBs of 24 peer routers and 21 peer ASes, which shared the common subpath “33891 25091 8298 210312”. This indicates that AS33891 (Core-Backbone GmbH, DE), a Tier-2 in Germany with approximately 2100 ASes in its customer cone, could be the root cause of the Zombie outbreak.
This research highlights the importance of robust monitoring and analysis solutions in addressing complex issues like BGP zombies. Cisco ThousandEyes empowers organizations with unparalleled visibility into routing anomalies, enabling them to detect and mitigate the operational risks associated with stuck routes and Zombie paths.
