20100521肖韬组会报告MapReduce编程实例浅析.pptVIP

* * * * * * * * * * * * job.setReducerClass(IntSumReducer.class); Set the?Reducer?for the job. job.setOutputKeyClass(Text.class); Set the key class for the job output data. job.setOutputValueClass(IntWritable.class); Set the value class for job outputs. FileInputFormat.addInputPath(job, new Path(otherArgs[0])); 原型：static?void addInputPath(JobConf?conf,?Path?path); Add a?Path?to the list of inputs for the map-reduce job. FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 原型：static?void setOutputPath (JobConf conf,?Path?outputDir) ?Set the?Path?of the output directory for the map-reduce job. System.exit(job.waitForCompletion(true) ? 0 : 1); Submit the job to the cluster and wait for it to finish. 对TokenizerMapper的代码分析 1. TokenizerMapper类的定义 class TokenizerMapper extends MapperObject, Text, Text, IntWritable ｛ … ｝ 所有的mapper类都必须实现接口Mapper，该接口有4个类型参数，分别是： Object：Input Key Type Text: Input Value Type Text: Output Key Type IntWritable: Output Value Type 因为所有的mapper类都必须实现接口Mapper，即必须实 现Mapper中的map函数 public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } Writable的主要特点是它使得Hadoop框架知道 对一个Writable类型的对象怎样进行serialize以 及deserialize. WritableComparable在Writable的基础上增加 了compareT接口，使得Hadoop框架知道怎样对WritableComparable类型的对象进行排序。 参考书目 Hadoop: The Definitive Guide. Pro Hadoop. Hadoop Streaming Guide. The Quick Python Book (2nd Edition) * * * * * * * * * * * * * * * * * * * * * * * * * MapReduce程序实例浅析 肖韬 南京大学计算机科学与技术系 2010.05.21 MapReduce任务执行过程回顾 下载JDK及Hadoop软件 JDK最好使用1.6以上的版本，安装好了之后需要进行环境变量的设置 在/etc/profile文件中增加3行： export JAVA_HOME=/home/xt/soft/jdk1.6.0_18 export HADOOP_HOME_0_20_2=/home/hadoop_home/ hadoop-0.20.2/bin export PATH=$JAVA_HOME/bin:$HADOOP_HOME_0_20_2:$PATH 然后， source /etc/profile将该文件重新加载一遍 编写MapReduce程序