Introduction to Hive Cloudera Engineering Blog（蜂巢Cloudera工程概论的博客）.pdfVIP

Introduction to Hive © 2009 Cloudera, Inc. Outline • Motivation • Overview • Data Model • Working with Hive • Wrap up Conclusions © 2009 Cloudera, Inc. Background • Started at Facebook • Data was collected by nightly cron jobs into Oracle DB • “ETL” via hand-coded python • Grew from 10s of GBs (2006) to 1 TB/day new data (2007), now 10x that. © 2009 Cloudera, Inc. Hadoop as Enterprise Data Warehouse • Scribe and MySQL data loaded into Hadoop HDFS • Hadoop MapReduce jobs to process data • Missing components: – Command-line interface for “end users” – Ad-hoc query support • … without writing full MapReduce jobs – Schema information © 2009 Cloudera, Inc. Hive Applications • Log processing • Text mining • Document indexing • Customer-facing business intelligence (e.g., Google Analytics) • Predictive modeling, hypothesis testing © 2009 Cloudera, Inc. Hive Components • Shell: allows interactive queries like MySQL shell connected to database – Also supports web and JDBC clients • Driver: session handles, fetch, execute • Compiler: parse, plan, optimize • Execution engine: DAG of stages (M/R, HDFS, or metadata) • Metastore: schema, location in HDFS, SerDe © 2009 Cloudera, Inc. Data Model • Tables – Typed columns (int, float, string, date, boolean) – Also, list: map (for JSON-like data) • Partitions – e.g., to range-partition table