As businesses expand their operations to rapidly developing new markets like China, we are excited to announce that we too are expanding our horizons at ThousandEyes. We have increased our set of monitoring points to include nine new Cloud Agent locations in China: Foshan, Jinan, Nanjing, Shenyang, Shijiazhuang, Wenzhou, Wuxi, Xi'an and Zhuhai. This brings the total number of agents located in China up to 14.
This post is the first in a series of blogs that will discuss how you can best take advantage of the new Cloud Agents to understand network performance in China. In this post, we’ll talk about how you can baseline performance impacts in China and set expectations for application delivery, along with examples from our own testing.
A Different Internet
As we’ve discussed in previous posts on the Great Firewall of China and Internet censorship, the Chinese Internet looks very different from what the Internet looks like in the US. In China, Internet service is primarily dominated by two ISPs (China Telecom and China Unicom) that are underdeveloped and often congested. On top of that, network access from China to the rest of the world is restricted through a small number of choke points. In addition, the Great Firewall heavily filters traffic both at the borders and within China, using a wide range of methods including IP blocking, DNS tampering and hijacking and deep packet inspection.
Needless to say, high latency and packet loss are issues much more common in China than in the US or Europe. When monitoring network performance to, from or within China, you’ll need to work with different baselines in mind and understand the common issues that may arise, so you can distinguish typical underperformance from atypical outages.
Performance Impacts
Now we’ll discuss the performance impacts you can expect when monitoring in China. In general, you should expect significantly higher levels of packet loss and latency if either the source or destination of traffic is located in China and packets must pass through China’s border networks where censorship and filtering are most severe. However, the exact levels of packet loss and latency depend on a number of factors, including how “sensitive” the accessed content is to the Chinese government, the Great Firewall’s current level of stringency, current levels of congestion and how well the content has been adapted to be delivered to or from China.
To show how you can start to get an understanding of and baseline performance in China, we set up tests from a selection of China, US and Hong Kong Cloud Agents that targeted five websites each in the US, China and Hong Kong. We collected two weeks’ worth of data and present the findings below — feel free to follow along at this share link to see the data in more detail. While five sites is not enough to be representative of performance across the Great Firewall, it does provide a starting point for setting expectations on performance impacts.
Performance to US Sites
To compare performance to US sites with performance to China sites, we set up tests to the US domains of AWS S3, Starbucks, KFC, Evernote and Johnson & Johnson, as well as to their Chinese counterparts. We selected all 14 China agents, 10 US agents, and the Hong Kong agent to test from.
Looking at mean packet loss and latency to the group of US sites over two weeks, the results show unsurprisingly significant differences. The China agents see much higher overall packet loss (6.9%) and latency (218 ms) than the US agents (0.04% and 17 ms, respectively), due to factors of censorship, congestion and sheer distance. Packet loss and latency from the Hong Kong agents (0.2% and 81 ms), while higher than the US numbers, are several times lower than the measurements from the China agents, showing that geographic distance is not the only factor impacting performance from China.
Hong Kong show that geographic distance is not the only factor in impacted performance from China.
But there is a more nuanced picture in the timelines of packet loss and latency over time. While loss and latency from Hong Kong and the US are fairly stable over time, the same metrics are highly unstable from China. Interestingly, both packet loss and latency follow diurnal patterns, with latency ranging from 150 ms at the troughs and up to 300 ms at the peaks. Lows in latency generally occur around 4-6am China Standard Time, while the high plateaus occur from 1pm-1am (usually peaking around 7-9pm).
These daily fluctuations are likely due to traffic patterns and the strained networks and filtering structures within Chinese networks. During time periods with high levels of traffic, ISPs are likely also congested and take longer to filter traffic according to the government’s censorship standards. This explanation is in line with researchers’ findings that scanning activity for Tor servers follows diurnal patterns.
Performance to China Sites
We’ll now look at performance in the other direction, to the Chinese domains of the companies mentioned above: AWS S3, Starbucks, KFC, Evernote and Johnson & Johnson.
Now the picture is flipped: China agents see the lowest levels of packet loss and latency on average, the Hong Kong agent observes packet loss and latency roughly 1-2 times higher, and the US agents see the most impacted network performance. It’s clear that it’s no small feat for a packet to get across the Great Firewall in a timely fashion.
and latency roughly 1-2 times higher, and the US agents see the most impacted network performance.
Looking at the timelines of packet loss and latency to China sites, it’s interesting to note that packet loss and latency of traffic going into China do not have such clear diurnal patterns as traffic going out of China. This may suggest that the Great Firewall is more stringent, or that infrastructure is more congested, for outbound traffic coming out of China.
Performance to Hong Kong Sites
To isolate the performance impact of the Great Firewall, we set up tests to five different sites hosted in Hong Kong and only tested from three agents: Hong Kong, Zhuhai and Foshan. Zhuhai and Foshan are located within China and are geographically very close to Hong Kong, so any differences in latency due to geographic distance should be negligible.
As a result, any actual differences in performance between the Hong Kong and China agents can be attributed to the performance cost associated with crossing the Great Firewall. Below, we see that traffic from China to Hong Kong sees packet loss roughly five times higher and latency twice as high as traffic from Hong Kong.
Looking at packet loss and latency over time, we see diurnal patterns similar to what we saw when monitoring US sites, strengthening our hypothesis that the Great Firewall filters more heavily, or that infrastructure is more congested, for outbound traffic coming out of China.
When monitoring in China, particularly for outbound traffic, it’s important to set expectations that packet loss and latency can see large fluctuations over the day, often in line with diurnal patterns.
Use the Path Visualization to spot issues traffic is experiencing in crossing the Great Firewall. In our test to www.cuhk.edu.hk, the Zhuhai and Foshan agents frequently saw loss and latency issues in the border infrastructure between China and Hong Kong, while the Hong Kong agent saw no issues and took much fewer hops to the destination. The below Path Visualization shows high latency in the link from China Unicom to PCCW in Hong Kong, as well as high loss in PCCW close to the border, common issues given that much of China’s censorship and filtering occurs around the border.
in PCCW close to the border, common issues given that much of China’s censorship and filtering occurs around the border.
If anything, these results show the importance of hosting domains within China for application delivery to users located in China. Keep in mind that hosting sites in Hong Kong is no substitute for being within the borders of the Great Firewall — while performance to Hong Kong is still better than all the way back to the US, it still incurs performance impacts from having to cross China’s Great Firewall.
With 14 monitoring points located all over China, you’re now well equipped to begin setting up tests to baseline performance from China and understand the experience of end users located in China. Read the next post in this series, Monitoring Application Delivery in China, for tips to monitor common issues unique to operating in China, including highly unreliable DNS, blocked page components and monitoring and alerting on the performance of ISPs and third party providers.