One scenario we captured and demonstrated during Networking Field Day 8 involved a BGP change at Country Financial. Follow along with this interactive share link if you would like to see the data for yourself, or watch the recorded presentation from NFD8 below.
Around 17:25 UTC our cloud agents started alerting to connection issues with Country Financial’s website. We saw a dip in availability and alerting on this test for over an hour (Figure 1).
In order to get some context into Country Financial’s network, let’s take a look at a time before the event. When we look at the Path Visualization at 17:15 UTC on September 4th we see one upstream internet service provider, Access2Go (Figure 2).
The BGP Route Visualization confirms this (Figure 3). Autonomous System (AS) 40948 belongs to Access2Go and is shown in grey, indicating it is the provider. The origin network, AS 10511, is registered to CC Services Inc, another name for Country Financial.
At 18:30 UTC, after the event, we looked again at the Path Visualization. It appears that during the last hour the provider changed from Access2Go to Qwest (Figure 4).
To confirm this we can look at the BGP Route Visualization view. However instead of seeing AS 209 (Qwest) in grey as the only provider, we also see AS 15011 (Jaguar Communications). Even more interestingly, none of the monitors are peering with AS 15011 (Figure 5).
Qwest (AS 209) is the provider, however during the provider change someone trying to increase the cost of the path using AS-path prepending mistyped 15011 instead of 10511. I had a quick look at the raw data and confirmed this is what happened.
208.74.228.0/24 | 3561 209 10511 10511 15011 10511
What is BGP prepending? BGP prefers routes with the shortest AS path. To manipulate the inbound route taken, administrators can manipulate the AS path by prepending, or adding AS numbers to a route advertisement so that a route looks less favorable. A typical use case for prepending would be on multi-homed networks, where administrators will use prepending to disfavor a route that has higher costs or is the backup link.
Most of the time routing changes take place and they go fine - however sometimes just one keystroke can introduce latency, errors, or even unintentional peering sessions.