New Innovations Unveiled at Cisco Live 2024

Internet and Cloud Research

What 160M Public Cloud Performance Measurements Mean

By Archana Kesavan
| | 9 min read


Public cloud providers are everywhere, literally. In fact, it’s not an exaggeration to say that in echoes of the “Great Game” of olde, the big three cloud providers—Amazon Web Services (AWS), Microsoft Azure and Google Cloud (GCP) are in a great race for cloud dominance. Analyst research firms predict that the global public cloud market will continue to rise briskly at a CAGR of 22% and will be predominantly influenced by the top three players. So much so that at the beginning of 2018, Forrester predicted the Big 3 would capture at least 76% of the cloud platform revenue in 2018, and rise to 80% by 2020. That’s big.

My Kingdom for Some Cloud Performance Data!

Unfortunately, what’s not so big is the amount of metric performance data that IT architects and leaders have had at their disposal, with which to make decisions. To be clear, there’s lots of data out there comparing public cloud providers. In fact, you if you google “AWS vs. Azure vs. GCP,” you’ll get a lot of hits, but mostly you’ll find comparisons of market share, service catalogs, pricing and data center presence and the like. When it comes to real performance data, it’s pretty slim pickings. And it’s not just that you can’t find many studies of public cloud performance, but when you do, the data itself is lacking. Most such studies we looked at didn’t offer much breadth. For example, they looked at single providers without cross-vendor comparisons, focused on performance to the “front door” of the cloud—user locations to data center regions, while ignoring the back-end performance within cloud provider backbones. Further, most cloud performance studies lacked depth. They used fairly simplistic measurements like ping tests that didn’t show deeper metrics like packet loss or jitter. The studies and analyses we observed were mostly moment-in-time snapshots rather than more extended duration tests that could surface trends and averages. Finally, performance isn’t just about averages, but most existing studies didn’t look at standard deviations to measure performance predictability.

Shallow data
Figure 1: Enterprise IT leaders couldn’t rely on the previously available performance data to make cloud decisions due to lack of breadth and depth.

As a result of this large performance data gap, IT leaders and architects have had to rely on instincts, educated guesses and vendor claims to formulate their cloud strategies and connectivity architectures. But that’s no way to run a business, especially today.

The Public Cloud Performance Benchmark Report is Born

In 2017, we conducted some tests within AWS measuring things like Inter-AZ performance. In 2018, with the help of an intrepid product marketing intern (see Kieran’s blog post on automating tests with the ThousandEyes API), we decided to expand the scope and conduct the industry’s most comprehensive set of public cloud network performance tests. We utilized the ThousandEyes Network Intelligence platform that offers active monitoring tests from a global fleet of software monitoring agents deployed around the Internet and across the Big 3 cloud providers. To ensure proper depth of measurements, we ran bi-directional network performance tests between monitoring agents, measuring latency, loss and jitter plus recording precise network paths with both end-to-end and hop-by-hop metrics as well as per-node Internet and cloud backbone topology data.

To make this study useful, we knew we also needed breadth, so we tested across four major vectors. First, we tested between 27 global user locations and 55 regions across AWS, Azure and GCP to understand “front-door” behavior.

We also knew that deploying applications in the cloud means redundancy and load balancing, so second, we tested Inter-AZ performance. Many organizations need to regionalize their cloud deployments to serve geographically diverse user bases and often use tiered architectures where storage or database components are centralized while compute is distributed. So to look at this angle, we (third) tested Inter-Region performance within each cloud provider’s network--15 in AWS, 25 in Azure and 15 in GCP. Finally, it has become clear to use that multi-cloud is fast becoming a reality, so we tested performance between all regions of all three providers.

Measurements for benchmarking cloud performance
Figure 2: Four major vectors of cloud performance measurement create significant breadth to the report’s data.

We took all these measurements every ten minutes over a period of thirty days across multiple metrics mentioned above. The result of the breadth, depth, frequency and duration was a total of 160 million unique measurements. Even 160M is somewhat conservative because we’re counting all the individual data points within network path data as a single measurement.

Details of extensive data set used for benchmarking cloud performance
Figure 3: The Public Cloud Performance Benchmark produced an extensive dataset.

What Does 160M Cloud Performance Measurements Mean?

Having lots of data is nice of course, but what does it mean? First of all, it means that IT leaders and architects can emerge from the darkness of assumptions to light of data-driven decisions. The report offers a wealth of tabular data, network architecture insights, and even baselines Inter-Region measurements against comparable Internet performance metrics.

The report also uncovers some interesting findings, addressing topics such as:

  • Does Inter-AZ performance meet the expectations set by the Big 3?
  • Do the Big 3 all run their network architectures in the same fashion?
  • What regional pairs can serve load balancing or data replication?
  • Where, if any exist, are there user to region performance anomalies?
  • Is multi-cloud a safe alternative in the face of any regional performance anomalies?

In subsequent blog posts, we’ll be exploring a number of these topics in specific, so if you want to keep up to date, make sure to subscribe to our blog.

Ultimately, cloud architectures are complex and every organization has to make its own choices about cloud services, pricing and performance. The Public Cloud Performance Benchmark Report is a resource for you to use in your decision making. Get your copy of the report today.

Subscribe to the ThousandEyes Blog

Stay connected with blog updates and outage reports delivered while they're still fresh.

Upgrade your browser to view our website properly.

Please download the latest version of Chrome, Firefox or Microsoft Edge.

More detail