Saturday, February 7, 2009

eBay Scaling Odyssey and All I Need is Scale

These are two very interesting slide sets on web application design principles and cloud computing at eBay. Building a web application that serves 2 billion page views per day and manages transactions with real money ($60 billion per year) is definitely nontrivial, especially if you have to balance reliability and performance with "feature velocity" and adaptability (the ability to constantly upgrade the product). Is it a real problem? Heck yeah. I think that supporting rapid evolution is one of the most underestimated aspects of web application development in the research community, and one of the main challenges in practice. I've seen this during my internship at Facebook too. It's nice that the presentations point out this problem right away. Evolvability is also one of the things that it's hardest to plan for when building a web application, especially given the need to constantly push out features to compete, and it gets harder and harder as your application grows. Having a set of guiding principles to follow from day 1 is very helpful.

The five principles the slides point out are familiar to anyone who's read about GFS, Dynamo, MapReduce, etc. It's nice to see consensus! The most surprising one to me was that eBay uses machine learning for automation, although this seemed to be more for the search side than the operations side.

The design patterns provide some good ideas as well. While I didn't fully understand the point of the scalable multicast and event queue patterns from the slides, I really liked the idea of a global message bus with logging to deal with failures, and of rollback. This reminds me of some of the elements of the Facebook architecture - for example, a common RPC format, Thrift, for service interactions, along with an RPC logging layer (Scribe). However, the Facebook architecture is somewhat different because you don't have a very clear concept of independent services yet, whereas eBay seems to be a more traditional SOA.

Their view on clouds is pretty promising - although they identify some "faux PaaS", they are actually considering using external cloud providers. This provides a strong case for cloud computing. One difference between eBay and other web applications though is that much of the data on eBay is public, so for example putting a search service in the cloud does not introduce a risk of losing consumer data. If a company like Facebook wanted to put services in the cloud on the other hand, they'd have to think very carefully because most of the site is private information.

Finally, the presentations also do a good job talking about tradeoffs. They mention CAP directly, and they say that they go for the A and P as much as possible. They also talk about the friction between performance, reliability etc and extensibility, though they don't try to say that this is an unavoidable tradeoff.

3 comments:

Andy Konwinski said...

"However, the Facebook architecture is somewhat different because you don't have a very clear concept of independent services yet, whereas eBay seems to be a more traditional SOA."

Do you think that the more Facebook matures, the more it will move towards SOA? It seems like SOA would better facilitate code reuse within a large company (e.g. Amazon, Y!, Facebook, etc.) Why isn't Facebook doing this?

Matei Zaharia said...

I think the difficulty has been adding new features at a rapid rate as opposed to spending a lot of time to refine an existing product. A site like ebay doesn't change its basic service very often, but Facebook has had a lot of big changes, such as the introduction of news feed, the introduction of Facebook apps, the new profile, etc. The company was probably just not at the scale of something like ebay where they could set aside a team of people and tell them "okay guys, you design version 2 of the site from scratch". SOA is undoubtedly the way to go though, and a lot of features (e.g. chat, photos) are now separate "services" with their dedicated pools of machines and such.

Ion said...

Maybe one should come up with an architecture that allows rapid deployment of new features... Such an architecture should probably provide a pretty high level (declarative?) API to enable fast prototyping and development of new features. An important question here is what are the services exposed by such API. Maybe something similar to Google Apps?