Craigslist reported a DNS outage on Sunday, November 23rd starting at 5pm PST when, according to the craigslist CEO, “DNS records maintained at one of our domain registrars were compromised, diverting users to various non-craigslist sites.” By 7:18am Monday morning, craigslist confirmed that the DNS records had been corrected by the registrar, Network Solutions. During the course of the hijacking, we found that upwards of 26% of DNS queries directed users to rogue name servers. These name servers then directed traffic to an infamous prank site, digitalgangster.com, causing a denial of service attack, as reported by Ars Technica. And even after Monday morning, hijack records remained in the caches of DNS resolvers around the globe.
So what is a DNS hijack? What were the effects? And how long did it take to resolve? We monitored the craigslist.org hijack; the data shows just how widespread and long-lived DNS hijacks can be.
What Is DNS Hijacking and Cache Poisoning
First, a little background on DNS vulnerabilities and attacks. DNS records, and web traffic destined for those domains, are typically compromised in one of two ways, hijacking or cache poisoning. In both cases, redirected web traffic can be used to execute denial of service attacks, install malware and phish for passwords.
Hijacking involves compromising the DNS server or registrar itself, typically through a phishing attack or a compromised password. The hijacker gains administrative access to the DNS account in order to changes the records directly. Typically a hijacker will change the name server (NS) record to point future DNS queries to a name server under their control. A hijacker may also directly change address (A or AAAA) records themselves. Examples of this type of attack are hijacks of some New York Times and Twitter domains in 2013.
Cache poisoning occurs on DNS resolvers distributed throughout networks that comprise the Internet. When end users make a DNS query, resolvers in the DNS hierarchy serve a response if they have a valid record in their cache; therefore, most queries only resolve to the authoritative DNS server at periodic intervals proportional to the time-to-live (TTL) of the record. In the case of cache poisoning, an attacker inserts a forged DNS record into a DNS resolver, using a variety of tactics that typically involve racing to provide a valid response or brute forcing less secure DNS configurations. The poisoned records, again typically NS records, direct future DNS queries to the attacker’s name server, which then serves up an authoritative record of the attacker’s choice.
Hijacking at craigslist
In the craigslist DNS interruption last Sunday, it was a hijacking that triggered the misdirected traffic. The hijacked NS records directed users to four of the attacker’s name servers hosted at Network Solutions’ WorldNic and Hostinger Group’s 000webhost:
Below you can see the proportion of DNS queries that resolved to different name servers across more than 1300 global DNS resolvers, the source used for all of the following data. As of Sunday at 9:20pm Pacific, 4 hours into the attack, 26% of DNS resolvers were directing queries to the rogue name servers (Figure 1).
From a country perspective, Asia was hardest hit with more than 30% of DNS resolvers serving hijacked records in Japan, South Korea, Australia and China (Figure 2).
Within and across countries there is a high degree of variance by network. Some major networks had over 50% of resolvers serving hijacked records, including China Unicom, Telstra of Australia and Qwest of the U.S (Figure 3).
Clearing the Cache
In a DNS hijacking, a critical part of response and recovery is that network operators flush their caches as soon as the proper NS records are restored. Over time the NS records in caches should expire, varying on the TTL of records in each resolver, but a manual flush expedites the process. Therefore, we’d hope to see that hijacked records would be flushed within the first day after the hijacking. By Monday at 9:20pm Pacific, 3.5% of resolvers were returning non-craigslist name servers, an 87% reduction over 24 hours. By Sunday November 30th, a week after the hijack, 0.3% of DNS resolvers still served hijacked records (Figure 4).
Across countries, we see that despite a high prevalence of hijacked records on resolvers in Japan and Australia, these caches were cleared within 48 hours. Other countries, including Germany and Norway still do not have all resolvers returning the correct NS records (Figure 5).
And from a network view, most major networks had flushed their caches within 24 hours, though Qwest took nearly 5 days and TeliaSonera still has resolvers serving hijacked records after a week (Figure 6).
We can look into a specific resolver on the TeliaSonera network and find that, indeed, the DNS trace resolves to a 000webhost name server rather than a craigslist name server (Figure 7).
Even a week after a highly publicized (in the network operator world) DNS hijack, some major networks still have not flushed their resolvers’ caches. This highlights the importance of being vigilant before, during and after a hijack.
Detecting and Resolving a DNS Hijack
You’ll want to be on the look out for a DNS hijack in two cases. First, in a hijack of your own name space, you’ll need to both restore the proper records and encourage network operators to flush their caches (just as craigslist.org did). Second, in the case of someone else’s name space being hijacked you’ll want to flush the relevant records (just as major ISPs did). For more examples of DDoS Attacks, BGP Hijacks and Cache Poisoning watch the Monitoring for Network Security webinar.