MapReduce编程环境设置和注意事项2010说课材料.doc

Good is good, but better carries it.精益求精，善益求善。MapReduce编程环境设置和注意事项2010-一 MapReduce编程环境设置和注意事项远程登陆服务器二 请同学们使用服务器分配的原始密码通过SSH工具登陆到实验集群bl6000的登陆节点：202.38.192.99，然后使用命令passwd对初始密码进行修改。PS：Linux下密码不可见。接着在自己的账号下使用命令mkdir分别建立1个目录：MapReduce用来存放不同实验数据和代码。设置编译环境使用命令vi/.bash_profile设置环境变量，添加路径如下文，然后使用./.bash_profile（注意前有个点+空格）使设置生效，整个操作过程如下:ming.taobl6000$cat.bash_profile#.bash_profile#Getthealiasesandfunctionsif-f/.bashrc;then./.bashrcfi#UserspecificenvironmentandstartupprogramsPATH=/opt/gcc-4.2.4/bin:/opt/mpich-1.2.7pl/bin:/opt/intel/Compiler/11.0/069/bin/ia32:$PATH:$HOME/bin:/usr/sbinLD_LIBRARY_PATH=/opt/gcc-4.2.4/lib64:/usr/lib:opt/mpich-1.2.7pl/lib/:/opt/intel/Compiler/11.0/069/lib/ia32:$LD_LIBRARY_PATHHADOOP_CLASSPATH=/opt/hadoop/hadoop-0.20.1/hadoop-0.20.1-core.jar:/opt/hadoop/hadoop/hadoop-0.20.1/lib/:/opt/hadoop/hadoop-0.20.1/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-6.1.14.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-util-6.1.14.jarexportPATHexportLD_LIBRARY_PATHexportHADOOP_CLASSPATHunsetUSERNAMEming.taobl6000$ming.taobl6000$./.bash_profile然后使用echo$HADOOP_CLASSPATH命令检测环境变量是否设置成功,正常的显示如下：ming.taobl6000$echo$HADOOP_CLASSPATH三 /opt/hadoop/hadoop-0.20.1/hadoop-0.20.1-core.jar:/opt/hadoop/hadoop/hadoop-0.20.1/lib/:/opt/hadoop/hadoop-0.20.1/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-6.1.14.jar:/opt/hadoop/hadoop-0.20.1/lib/jetty-util-6.1.14.jar跳转计算节点本次实验除了bl6000的登陆头节点（202.38.192.99）还有4个计算节点，分别是：c0-0,c0-1,c0-2,c0-3使用命令ssh可以从当前节点转到计算节点,例如sshc0-1则可从当前节点转到计算节四 点c0-1,其他如此类推。编译运行MapReduce程序示例如何编译运行MapReduce程序以附录的程序WordCount.java为例:ming.taobl6000MapReduce$lsWordCount.javaming.taobl6000MapReduce$javac-classpath$HADOOP_CLASSPATHWordCount.javaming.taobl6000MapReduce$lsWordCount.classWordCount$IntSumReducer.classWordCount.javaWordCount$TokenizerMapper.classming.taobl6000MapReduce$jarcvfwd.jar*.classming.taobl6000MapReduce$viinputdataming.taobl6000MapReduce$catinputdataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-putinputdataInput-dataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-lsFound8items-rw-r-r-3ming.taosupergroup462009-12-2818:39/user/ming.tao/Input-dataming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-cat/user/ming.tao/Input-dataThisisaMapReduceExp.HadoopMapRedueHDFSming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopjarwd.jarWordCountInput-dataOutput-dir09/12/2818:46:12INFOinput.FileInputFormat:Totalinputpathstoprocess:109/12/2818:46:13INFOmapred.JobClient:Runningjob:job_200912242113_000509/12/2818:46:14INFOmapred.JobClient:map0%reduce0%ming.taobl6000MapReduce$ming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-lsFound10items-rw-r-r-3ming.taosupergroup462009-12-2818:39/user/ming.tao/Input-dirdrwxrwxrwx-ming.taosupergroup02009-12-2818:46/user/ming.tao/Output-dirming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-ls/user/ming.tao/Output-dirFound2itemsdrwxrwxrwx-ming.taosupergroup02009-12-2818:46/user/ming.tao/Output-dir/_logs-rw-r-r-3ming.taosupergroup622009-12-2818:46/user/ming.tao/Output-dir/part-r-00000ming.taobl6000MapReduce$/opt/hadoop/hadoop-0.20.1/bin/hadoopdfs-cat/user/ming.tao/Output-dir/part-r-00000Exp.1HDFS1Hadoop1MapReduce2This1a1is1ming.taobl6000MapReduce$五.编译运行MapReduce程序5.1编译：javac-classpath$HADOOP_CLASSPATyourprog.java我们采用的Java版本是：javaversion"1.6.0_14"5.2打包：jarcvf*.jar*.class5.3运行：/opt/hadoop/hadoop-0.20.1/bin/hadoopjar*.jaryourprogInput-dataOutput-dir六. 注意：如果你需要用bin/hadoopjar命令，需要在源程序中加入：Job.setJarByClass(yourprog.class);可参见demoprogram下的WordCount.java文件(job.setJarByClass(WordCount.class);)一些常见问题6.1 如何将本地文件放入HDFS文件系统中对策：将本地的localfile文件拷贝到HDFS文件系统的HDFSfile运用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsputlocalfileHDFSfile或/opt/hadoop/hadoop-0.20.1/bin/hadoopdfscopyFromLocallocalfileHDFSfile6.2如何将本地文件放入HDFS文件系统中对策：将HDFS文件系统的HDFSfile拷贝到本地的localfile文件运用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsgetHDFSfilelocalfile或/opt/hadoop/hadoop-0.20.1/bin/hadoopdfscopyToLocalHDFSfilelocalfile6.3如何将在HDFS文件系统中创建目录对策：将HDFS文件系统创建mydir目录运用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsmkdirmydir6.4如何查看HDFS中的文件及文件内容对策：运用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfsls/user/you_account/运用/opt/hadoop/hadoop-0.20.1/bin/hadoopdfscat/user/you_account/*file*6.5如何获取Hadoop的API对策：Hadoop的API实验附录的HadoopAPI目录6.6其它问题可查阅l Hadoop的Map/ReduceTutorial：http:/hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.htmll SortBenchmark：http:/sortbenchmark.org-