Measuring and Evaluating Large-Scale CDNs

Problem: How can we compare content distribution networks for performance and availability?

Is it important? Most major websites are probably interested in the results. It's interesting to ask what they use to make the decision on hosts however; this study is done purely from the outside, but CDNs must have internal performance metrics they present to potential customers.

Nugget: Use DNS from a large list of open recursive DNS servers to identify the servers in each CDN and measure latencies to them from a large number of geographical locations. The particular technique used comes from a system called King. To measure the latency from an open recursive DNS server D to a target server T, one first uses a domain one owns to install a name server entry for a subdomain with T's IP address listed as the name server. One then tries to make a query for this subdomain through D, thus causing D to send a message to T and receive a reply. There were also other neat pieces of methodology in the paper, such as using a list of logs from MSN Live to find a large number of open recursive DNS servers.

Results: The paper compares Akamai with Limelight and finds a number of interesting differences between them, largely resulting from their different architectures. Akamai has many small clusters of servers in ISPs, while Limelight has a few larger clusters each of which are placed near a number of ISPs at the same time. We see the following high-level results:

• Akamai has more servers in total by nearly (~25K versus ~5K).
• Akamai's services are more geographically distributed (e.g. Limelight has none in Oceania and only 3 in Asia).
  
• Individual Akamai servers have lower availability (although it's not clear whether availability for the service as a whole is affected), presumably because there are more of them so they are harder to manage.
• Akamai has lower 95th percentile latency across the whole suite of tests (from each of the locations with open recursive DNS servers). However, interestingly, Limelight does very well within particular regions, beating Akamai in North America and Europe. The difference seems to be mainly due to Limelight not having focused so much on Asia and Oceania yet, so perhaps its approach will work fine as it evolves.
  

Finally, the main thing missing in the paper is a study of throughput. This was not possible to do through the open DNS server technique, but it would've been valuable to try a few measurements from a smaller number of sites (e.g. PlanetLab).