Apache Hadoop and Hive课件.pptVIP

This is the architecture of our backend data warehouing system. This system provides important information on the usage of our website, including but not limited to the number page views of each page, the number of active users in each country, etc. We generate 3TB of compressed log data every day. All these data are stored and processed by the hadoop cluster which consists of over 600 machines. The summary of the log data is then copied to Oracle and MySQL databases, to make sure it is easy for people to access. Apache Hadoop and Hive Dhruba Borthakur Apache Hadoop Developer Facebook Data Infrastructure dhruba@, dhruba@ Condor Week, April 22, 2009 精品文档 Outline Architecture of Hadoop Distributed File System Hadoop usage at Facebook Ideas for Hadoop related research 精品文档 Who Am I? Hadoop Developer Core contributor since Hadoop’s infancy Project Lead for Hadoop Distributed File System Facebook (Hadoop, Hive, Scribe) Yahoo! (Hadoop in Yahoo Search) Veritas (San Point Direct, Veritas File System) IBM Transarc (Andrew File System) UW Computer Science Alumni (Condor Project) 精品文档 Hadoop, Why? Need to process Multi Petabyte Datasets Expensive to build reliability in each application. Nodes fail every day – Failure is expected, rather than exceptional. – The number of nodes in a cluster is not constant. Need common infrastructure – Efficient, reliable, Open Source Apache License The above goals are same as Condor, but Workloads are IO bound and not CPU bound 精品文档 Hive, Why? Need a Multi Petabyte Warehouse Files are insufficient data abstractions Need tables, schemas, partitions, indices SQL is highly popular Need for an open data format – RDBMS have a closed data format – flexible schema Hive is a Hadoop subproject! 精品文档 Hadoop Hive History Dec 2004 – Google GFS paper published July 2005 – Nutch uses MapReduce Feb 2006 – Becomes Lucene subproject Apr 2007 – Yahoo! on 1000-node cluster Jan 2008 – An Apache Top Level Project Jul 2008 – A 4000 node test cluster Sep