Making Our Website Faster: A Comparison of Cloud Content Delivery Networks (CDNs) with Cloud Storage, Cloud Hosting and Old-Fashioned Hosting

We have been using Amazon CloudFront CDN to deliver our software downloads for almost a year with excellent results. To find the best CDN provider for our upcoming website we set up accounts on selected CDN sites and selected hosting platforms. Then we monitored the performance from six locations around the globe. The contestants are:

Vendor	Bandwidth Cost (US$ per TB)
Amazon CloudFront CDN	$170-$200 plus some S3 cost
Amazon S3	$170
Rackspace Cloud Files CDN	$220
Google AppEngine	$120
SimpleCDN	$39
Softlayer CloudLayer CDN	$200
Joyent Cloud	(included in flat rate)
As a reference we added our own web server www.paessler.com (hosted in the Rackspace datacenter in Dallas TX)	several hundred $

For this test the PRTG Network Monitor monitoring probes of our cloud monitoring system www.cloudclimate.com requested a 64 kbyte file every 30 seconds between Aug 8th and today. We used probes located at GoGrid.com (San Francisco CA), Softlayer (Dallas TX), Newservers.com (Miami FL), Amazon EC2 (US and EU data center) and Flexiscale (UK). Here are the results:

Average Request Times

Let's remember that our goal is to provide fast downloads to people everywhere. So we would like to see short bars which should have the same short height for all locations. Note: We haven been talking to people at Amazon about the elevated Cloudfront values for EC2 US (blue bar) and they were able to fix the routing so that the latest values are around the 60ms area. The same process is currently ongoing for Newservers (purple bar). So actually we can consider all CloudFront values to be excellent.

Request Time Observations

CloudFront CDN and Cloud Files CDN lead the pack, closely followed by CloudLayer CDN. These three are also the most expensive solutions, but they are still much cheaper than our Rackspace bandwidth and much faster. From a performance perspective SimpleCDN is not better than "normal" hosting, but it is much cheaper, so it may be a good and cheap solution for very bandwidth hungry downloads (e.g. ISO images, virtual machines) that don't need the latest bit of performance. Google's AppEngine was the surprise for me in this test. It is not a CDN and it is slower than the CDNs, but request times are quite evenly distributed across the US and EU. As we all know Google stores all data in a distributed database and this shows. I'd rate them the price/performance-ratio leader. Storage system S3, cloud hoster Joyent and our own web servers in Dallas all show request times that corelate to the geographical distance between server and probes.

Ratio "Average Request Times" vs. "95th Percentile"

The ratio between the average and the 95th percentile gives us an estimate how consistent the request times were. A higher ratio tells us that request times were more volatile. Of course we would like to have constant times (so many small bars on the graph are better).

Observations

From this perspective the hosting servers and S3 are better than the CDNs. But even when CloudFront needs twice as long as the average (+100%) it is still twice as fast as an average S3 request. Cloudfront is not only faster than Cloud Files, it also delivers more consistently, too.

Conclusion

Here are the average request times for all contestants from all six locations: We will stay with Amazon CloudFront. It delivers our files in only 25% of the time that our own webserver in Dallas would need (or you could say: our webpages will load 4 times faster). CloudFront delivers files a little more consistent than Cloud Files, making Cloud Files a close runner-up. Softlayer's CDN also performs quite well and we rate it third in our test. An impressive fourth place goes to AppEngine. SimpleCDN may be cheap, but it is not fast enough for us. In our search for a fast way to quickly deliver files to our website visitors regardless of where they are all other contestants are no options.

Notes:

What about Asia and the rest of the world? We also did some monitoring from Panama and Singapore and results showed quite similar patterns. But our probes there are not reliable enough to be included in the official test.