Understanding the Meta, Comcast, and LinkedIn Outages

Outage Analyses

Rostelecom Route Leak Targets E-Commerce Services

By Nick Kephart
| | 15 min read


Last week Rostelecom, one of Russia’s largest, partially state-owned Internet service providers, leaked dozens of routes pertaining to IP addresses that belong to major financial services firms. This route leak redirected Internet traffic bound for these financial firms to Rostelcom, as some routers around the Internet decided that the new, leaked routes were preferable to the legitimate ones.

After multiple inquiries from our own customers, some of which were impacted in this event, we’ve decided to add some additional insights. Most importantly, the affected prefixes (ranges of IP addresses) not only belonged to financial services firms, they also included prefixes that host critical e-commerce and payments services. And not only were routes leaked, we can also confirm that traffic passed through the Rostelecom network en route to its intended destinations.

While it is hard to confirm that the leaking of routes to the Internet was intentional (e.g., to inspect traffic that should be going to financial services firms’ networks), the targeted selection of e-commerce services would not appear to be a coincidence.

Impact to E-Commerce Services

What is so intriguing about this route leak is the set of impacted services that use the prefixes, or IP address ranges, that were leaked. We performed reverse DNS lookups on the impacted prefixes to find which domains and services were affected.

Impacted services included:

  • Mastercard SecureCode, Smart Data and MasterPass
  • Verified by Visa and Visa-owned CardinalCommerce
  • Symantec WebSecurity and Geotrust
  • RSA’s email servers
  • Online banking sites for French banks BNP Paribas and CIT, and Polish Bank Zachodni owned by Santander
  • Trading and custody sites for HSBC and MUFG’s MoneyPort
  • Payment processing for HSBC’s EPS and PPS services, UBS’s CardCenter, Redsys and Atos Worldline

Several of these services form the core of online payment infrastructure. SecureCode and Verified by Visa, for example, authenticate cardholders at the time of an e-commerce purchase to reduce fraud. Symantec’s GeoTrust is a major digital certificate provider for TLS/SSL encryption that, among many other things, secures payments traffic.

Regardless of whether this route leak was intentional or not, the selection of prefixes seems to be very intentional and would suggest that Rostelecom was attempting to steer e-commerce payment traffic on at least their own network.

The Rostelecom Route Leak

On the evening of Wednesday, April 26th from 22:36-22:43 UTC (3:36-3:43pm Pacific), Rostelecom originated 137 prefixes to the Internet via peers such as Telecom Italia, Telstra, Hurricane Electric, KDDI, Cogent, Telia and Tata. Of these 137 prefixes, approximately 100 belong to organizations in Russia and, as such, likely as part of normal network operations. However, 36 prefixes belonged to companies such as Symantec, Visa, Mastercard, BNP Paribas and EMC. These numbers agree with BGPmon’s reporting on the event.

Over the course of 7 minutes, traffic destined for e-commerce and payment processing services run by these financial services firms, as well as web security and Internet security services, was redirected to the Rostelecom network. The route leak impacted large portions of Internet traffic, though not all, as not all networks accepted or preferred Rostelecom’s routes over the legitimate ones.

What are Route Leaks?

Internet traffic is delivered to its intended destination, represented by an IP address, after moving through a series of networks called Autonomous Systems. These Autonomous Systems (ASes) are the networks of companies such as Comcast, Verizon, GE or Coca-Cola. Each AS has ranges of IP addresses called prefixes that are registered to that organization. To help routers throughout the Internet determine which AS traffic should flow through, the Border Gateway Protocol (BGP) is used to communicate new or changed routes for each prefix.

Route leaks occur when an AS advertises illegitimate routes, whether accidentally or intentionally. Routes can be illegitimate when:

  • The hijacker AS claims to be the origin of a prefix when it is not (as in this case with Rostelecom)
  • The hijacker AS advertises a new, more specific prefix that routers will prefer
  • The hijacker AS advertises a shorter AS Path than actually exists, making its routes appear preferential

In each case, Internet traffic can be steered toward the offending AS, allowing it to inspect traffic or deny service.

Anatomy of a Route Leak

Let’s take a look at what the data actually shows about this route leak. ThousandEyes collects routing information from more than a hundred vantage points (we call them Route Monitors) around the Internet.

In Figure 1, we can see routes from a subset of these vantage points. The BGP Route Visualization shows Rostelecom (AS 12389) advertising and then withdrawing routes. Peers such as Cogent (AS 174), Hurricane Electric (AS 6939) and Tata (AS 6453) accepted these routes and propagated them across the Internet. Route Monitors in orange and red were impacted, while those in green continued sending traffic to the legitimate origin network.

Figure 1
Figure 1: Rostelecom (blue dotted circle) advertised and then withdrew routes (red dotted lines). Route Monitors (diamonds) in orange and red
were impacted, while those in green continued sending traffic to the legitimate origin network (green circle).

Correlating Traffic with Routes

ThousandEyes data includes not just routing information but also the traffic path taken by IP packets across the Internet in a view called Path Visualization. You can see in Figure 2 that traffic entered the Rostelecom network and then returned back out to the destination network, in this case via Cogent in Stockholm. But Rostelecom took the traffic for a ride, through 60+ interfaces in what was either an unintentional routing loop or a very intentional series of devices to inspect the traffic. You can see these as the white, unresponsive, interfaces in the visualization.

Figure 2
Figure 2: Traffic from Toronto destined for an impacted e-commerce service enters the Rostelecom network, travels through
60+ unresponsive (white) interfaces and exits back to Cogent’s network and then the destination network.

Alerting and Mitigating Route Leaks

Route leaks are especially concerning because a network operator cannot completely control the propagation of routes across the Internet. Route propagation is trust-based, dependent on the preferences and policies that each network places on routes, which they accept from peers and advertise themselves.

In the short term, you can monitor for route leaks with services that you operate or ones which you rely on for critical operations (e.g., a payment gateway). There are three types of route hijacks and leaks you can detect in ThousandEyes:

  • Another network originates your prefix (the Rostelecom leak is a good example). Set an alert for Origin AS not in [Your ASNs].
  • Another network originates a more specific prefix in your address space. Set an alert for Covered Prefix [Exists] or Covered Prefix not in [Your Prefixes].
  • Another network inserts their ASN into the AS path. Set an alert for Next Hop ASN not in {Your Upstream ASNs]. Note, this will not include hijacks that are more than one hop away.

See the in-depth article on alerting for route leaks and hijacks.

Figure 3
Figure 3: Alerting options to detect route leaks of your prefixes.

Aside from alerting, you may be able to take actions such as contacting upstream ISPs, announcing covered prefixes, announcing more preferred routes, changing the IP addresses you have in use or publishing ROAs. Read our best practices on combating route leaks and hijacks for more details, as well as longer-term, communal mitigation strategies such as route filtering, RPKI, RPSL, BGPSEC and TCP MD5.

Don’t miss our upcoming webinar on Detecting BGP Hijacks and Leaks, where we’ll talk through how you can diagnose and prevent these routing phenomena.

Subscribe to the ThousandEyes Blog

Stay connected with blog updates and outage reports delivered while they're still fresh.

Upgrade your browser to view our website properly.

Please download the latest version of Chrome, Firefox or Microsoft Edge.

More detail