Hive: Data Warehousing and Analytics on Hadoop

Problem: Mining event data generated by a large web application.

Nugget: Store data in a large distributed filesystem (HDFS) and query it with MapReduce (Hadoop). Provide an interface on top of this to let users express queries in SQL, create user-defined functions in Python, etc. Most importantly, manage metadata and schemas -- this is the primary difference between Hive and something like Pig.

Why is it different from previous work? This is a really interesting question given that the large-scale analytic DB space is pretty hot right now. The main reason given in the presentation is scale - Facebook expects the data warehouse to grow very large. They also mention that commercial products didn't have the appropriate availability, presumably due to not coping well with scale. However, perhaps one reason not covered is cost. From public pricing numbers, it seems that data warehouse DB products seem to cost from $100,000 per TB of user data for smaller databases down to $15,000 per TB of user data for very large ones (Teradata example). These prices may be discounted when you make an actual sale, but still, Hive is storing 200 TB of compressed data (and they seem to get 5x compression from the import numbers, so it might be 1 PB uncompressed), so this could cost tens of millions of dollars. At that price, building a cluster of 300 Hadoop machines and hiring ~10 programmers to build a data warehouse suddenly seems kind of attractive, even if the raw performance of Hadoop is not as good as that of an optimized DB engine. It will be interesting to see how this field plays out, as newer analytics DB startups (Greenplum, Aster Data, etc) are moving towards more shared-nothing, commodity-hardware systems.

Any hard tradeoffs? Certainly Hive had to sacrifice ACID and some amount of per-node performance in using Hadoop. On the performance side, it will be interesting to see how optimizations in Hive and Hadoop help (both systems have a lot of opportunity for optimization).

Will it be influential in 10 years? It's certainly influential today at Facebook. As more companies are starting to use Hadoop for data analysis (mostly of logs), they might use similar solutions (or even Hive itself, because Hive is open-source).