In our previous post, Monitoring Application Delivery in China, we discussed a myriad of issues that can affect the performance of applications delivered to users in China. Of these issues, difficulties with DNS in China is a topic that deserves its own post — when the packets of a critical service like DNS routinely go missing, the impacts on the delivery of important applications can be enormous.
The common issues of high packet loss and latency in China especially affect DNS because it generally employs the unreliable protocol UDP. When a UDP message is sent, there is no way to make sure that it successfully reaches its destination. Since there is no concept of acknowledgement, retransmission or timeout, very often a UDP packet sent out onto the Chinese Internet is simply lost.
In this post, we’ll discuss data collected by our 14 Cloud Agents located in China. We’ll first explore DNS issues likely related to congestion and resulting high packet loss, and then we’ll dive into an example where censorship is obviously taking place. To brush up on the specific DNS tampering tactics employed by the system of censorship in China, read our post Deconstructing the Great Firewall of China, which summarizes the range of techniques that make up the Great Firewall.
Frequent Packet Loss
We’ll first look at an example where low DNS performance results more likely from congested networks that frequently see high levels of packet loss and latency, rather than from direct censorship. Our example comes from a DNS server test for GitHub’s DNS records, which are served out of Route 53, AWS’s DNS service. As we saw with HTTP traffic crossing the Great Firewall, DNS traffic traveling across China’s borders to AWS DNS servers must overcome great obstacles, including significant levels of delay (around 200 ms) within the China Telecom network, likely as a result of congestion, stringent filtering, or both.
within the China Telecom network (red links).
As a result of high packet loss and high latency conditions, many of the China agents could not reach the Route 53 DNS name servers at all, after two tries with 3 second timeouts. The China agents that were able to reach AWS’s name servers received the correct mappings, though not without resolution times much higher than even Hong Kong, which is geographically close but located outside the Great Firewall.
reach GitHub’s name servers received the correct mappings, though not without high resolution times.
In contrast, looking at a DNS server test for the AWS S3 China service, we see that performance is far better if DNS records are hosted within China. In this case, the DNS records for S3 China are hosted within an ISP in Beijing. Packet loss and latency are greatly improved relative to traffic traveling across the Great Firewall, but issues still frequently arise due to the turbulent nature of networks and censorship in China.
Packet loss and latency are greatly improved relative to traffic traveling across the Great Firewall.
Outright Censorship
Next, we’ll look at data that clearly shows the effect of censorship on DNS performance. This example comes from a DNS Server test that sends test traffic from all of our China agents to all name servers serving up the A record for nytimes.com. The New York Times has long been blocked by the Great Firewall, and we’ll see what the repercussions of that block look like for DNS in this example. To follow along with the analysis, explore the data in this share link.
The nytimes.com A record is hosted in 6 name servers, 4 of which are external name servers in Dyn, and 2 of which are internal name servers maintained by the New York Times. We’ll first look at the Path Visualization showing each hop that test traces take from a selection of US and China agents to the Dyn name server ns1.p24.dynect.net. We see that traces from both US and China agents are able to reach the anycast Dyn name server, and traces originating in China successfully cross the Great Firewall into nodes in NTT in Frankfurt and Tokyo.
successfully cross the Great Firewall into nodes in NTT in Frankfurt and Tokyo.
Censoring NS Records
However, the Path Visualization looks drastically different when we look at those same agents targeting the New York Times’ internal name server in Newark, dns-plx.ewr1.nytimes.com. The US agents are able to look up the correct IP address for the name server (170.149.161.134) and reach the name server in Newark with no issue.
In contrast, the China agents fail miserably: on the far right, we see a variety of IP addresses that have been returned for the DNS lookup for dns-plx.ewr1.nytimes.com. With a quick whois lookup, we see that these IP addresses don’t belong to the New York Times at all, but rather to a set of services like Facebook and Dropbox that are known to be blocked in China. Our China agents received bogus records likely as a result of DNS cache poisoning or hijacking from within the Great Firewall, a set of techniques long studied by academic researchers.
Once the agents received the spoofed DNS reply, they then sent traces off to these blocked IP addresses. The traces were then of course terminated within Chinese ISPs that practice IP blocking, which entails blackholing traffic destined for blacklisted IP addresses like the ones we see below. This manifests itself in the Path Visualization as terminating traces in the networks of a number of major Chinese ISPs, including China Telecom and China Unicom.
As a result, traces destined for these IP addresses are blackholed by major Chinese ISPs.
The story is the same for the other name servers hosting the nytimes.com A record: traces to the Dyn name servers go unhindered, while traces to the New York Times internal name servers are consistently blocked. This data is strong evidence of censorship — the New York Times name servers are blocked likely because they only host their own DNS records, which are deemed to be sensitive, while the Dyn name servers host a variety of services’ records (both sensitive and otherwise) and should not be blocked on the name server level.
Censoring Address Records
However, blocking name servers’ hostnames is not the only DNS trick at play here. The Great Firewall also employs cache poisoning and DNS hijacking to cover all its bases. Even though it can’t entirely block Dyn’s name servers, it can stop sensitive records from being served up by Dyn’s name servers in other ways.
We can see this when we look at the Server Metrics view that shows the mappings for the nytimes.com A record that each agent received from the first New York Times internal name server we looked at previously. While the US agents all receive the same set of four IP addresses beginning with 151.101, the China agents each get a mapping to a single IP address, each of which belongs to a known blocked service like Facebook, similar to the mappings we saw earlier for the New York Times’ name server’s hostname.
The DNS records for nytimes.com that the China agents received are obviously spoofed, but can we tell more about how the tampering was done? Resolution time gives us some clues — notice that the resolution times for the China agents range from 7-27 ms, while resolution times for the US agents range from 15-73 ms. Keep in mind the target name server is located in Newark. It’s impossible that the China agents are able to achieve resolution times lower than even US agents that are geographically much closer to the destination.
These impossibly low resolution times indicate that the spoofed mapping was served from somewhere within the Great Firewall, likely via local cache poisoning, which is more lightweight and generally much quicker than keyword-based DNS hijacking.
indicate that the spoofed records were served up from within the Great Firewall, likely via local cache poisoning.
It should now be clear that DNS can fail in a number of ways within China’s borders; packets can be easily lost on frequently congested networks, or intentionally dropped as a result of censorship techniques. These conditions can change at the drop of a hat — with the political climate, new and evolving censorship policies and even with diurnal patterns and the time of day.
It’s important to remember that application delivery completely relies on the availability of accurate DNS records. To keep abreast of any changes that affect important records, use DNS Server and Trace tests to continuously monitor the state of your DNS records, including their availability, accuracy and resolution time.