VIDEO
The McLaren Formula 1 Team keeps their network race-ready with ThousandEyes.

Outage Analyses

When Routes Go Wrong: Solving BGP AS-Path Prepending Errors

By Doug Royer
| October 21, 2014 | 4 min read

Summary


One scenario we captured and demonstrated during Networking Field Day 8 involved a BGP change at Country Financial. Follow along with this interactive share link if you would like to see the data for yourself, or watch the recorded presentation from NFD8 below.

Around 17:25 UTC our cloud agents started alerting to connection issues with Country Financial’s website. We saw a dip in availability and alerting on this test for over an hour (Figure 1).

Figure 1: Availability to Country Financial below 50% for over an hour.
Figure 1: Availability to Country Financial below 50% for over an hour.

In order to get some context into Country Financial’s network, let’s take a look at a time before the event. When we look at the Path Visualization at 17:15 UTC on September 4th we see one upstream internet service provider, Access2Go (Figure 2).

Figure 2: All traffic to Country Financial’s web server, green node on the far right, travels through the Access2Go network, shown as a blue node.
Figure 2: All traffic to Country Financial’s web server, green node on the far right, travels through the Access2Go network, shown as a blue node.

The BGP Route Visualization confirms this (Figure 3). Autonomous System (AS) 40948 belongs to Access2Go and is shown in grey, indicating it is the provider. The origin network, AS 10511, is registered to CC Services Inc, another name for Country Financial.

Figure 3: Country Financial’s AS 10511 is show in green with ISP Access2Go’s AS 40948 in grey.
Figure 3: Country Financial’s AS 10511 is show in green with ISP Access2Go’s AS 40948 in grey.

At 18:30 UTC, after the event, we looked again at the Path Visualization. It appears that during the last hour the provider changed from Access2Go to Qwest (Figure 4).

Figure 4: Country Financial is now served by Qwest.
Figure 4: Country Financial is now served by Qwest.

To confirm this we can look at the BGP Route Visualization view. However instead of seeing AS 209 (Qwest) in grey as the only provider, we also see AS 15011 (Jaguar Communications). Even more interestingly, none of the monitors are peering with AS 15011 (Figure 5).

Figure 5: Country Financial’s AS 10511 is connected to both AS 209 (Qwest), the correct upstream ISP, and AS 15011 (Jaguar Communications), a BGP prepending misconfiguration.
Figure 5: Country Financial’s AS 10511 is connected to both AS 209 (Qwest), the correct upstream ISP, and AS 15011 (Jaguar Communications), a BGP prepending misconfiguration.

Qwest (AS 209) is the provider, however during the provider change someone trying to increase the cost of the path using AS-path prepending mistyped 15011 instead of 10511. I had a quick look at the raw data and confirmed this is what happened.

208.74.228.0/24 | 3561 209 10511 10511 15011 10511
Figure 6: Raw BGP data showing the AS path from AS 3561 (Saavis) on the left to AS 10511 (Country Financial) on the right, with two correctly prepended paths and one incorrect path to AS 15011 (Jaguar Communications).

What is BGP prepending? BGP prefers routes with the shortest AS path. To manipulate the inbound route taken, administrators can manipulate the AS path by prepending, or adding AS numbers to a route advertisement so that a route looks less favorable. A typical use case for prepending would be on multi-homed networks, where administrators will use prepending to disfavor a route that has higher costs or is the backup link.

Most of the time routing changes take place and they go fine - however sometimes just one keystroke can introduce latency, errors, or even unintentional peering sessions.

related blogs

Blog Thumbnail: Starlink Outage Analysis: July 24, 2025
Outage Analyses
Starlink Outage Analysis: July 24, 2025
On July 24, Starlink experienced a 2.5-hour outage that impacted users globally. This analysis explores our observations, which indicate a centralized control plane failure of Starlink’s Low Earth Orbit (LEO) service. Our findings also provide important lessons for enterprises…
By Internet Research Team | July 24, 2025 | 10 min read
Blog Thumbnail: Cloudflare Outage Analysis: July 14, 2025
Outage Analyses
Cloudflare Outage Analysis: July 14, 2025
On July 14, Cloudflare's public DNS service was disrupted for around one hour when configuration errors caused route withdrawals. Explore what happened and unpack important lessons for enterprise network operations and infrastructure planning teams.
By Internet Research Team | July 14, 2025 | 19 min read
Blog Thumbnail: Google Cloud Outage Analysis: June 12, 2025
Outage Analyses
Google Cloud Outage Analysis: June 12, 2025
On June 12, some applications that use Google Cloud services experienced performance and availability issues. See how the outage unfolded in this analysis.
By Internet Research Team | June 12, 2025 | 21 min read