hadoop完全分布式搭建文档.pdf

《hadoop完全分布式搭建文档.pdf》由会员分享，可在线阅读，更多相关《hadoop完全分布式搭建文档.pdf（14页珍藏版）》请在淘文阁 - 分享文档赚钱的网站上搜索。

1、hadoop完全分布式部署（linux）超详细版概述：利多台Linux主机来进部署Hadoop，对Linux机器集群进规划，使得Hadoop各个模块分别部署在不同的多台机器上。新建个虚拟机hadoop1为主站，分别克隆两个名为hadoop2和hadoop3的从站。HostnameIPhadoop1192.168.100.10hadoop2192.168.100.11hadoop3192.168.100.12服务器功能规划：分别在3台主机上安装不同的功能hadoop1hadoop2hadoop3NameNodeResourceManage DataNodeDataNodeDataNodeNode

2、ManagerNodeManagerNodeManagerHistoryServer SecondaryNameNode 安装步骤：1、安装虚拟机，设置络 2、在主站上安装好java和hadoop，3、关闭防墙 systemctl stop firewalld 4、配置core-site.xml、hdfs-site.xml、slaves、yarn-site.xml、mapred-site.xml 5、分别在三台主机上设置SSH密码登录 6、分发主站的hadoop和java到两个从站上 7：启动集群，查看三台主机的进程。验证：在hadoop1上运namenode，在hadoop2上运yarn，在

3、hadoop3上运SecondaryNameNode。通过测试个例，分别在三个主机上查看完全分布式安装1、设置虚拟机络 新增NAT模式，修改IP为192.168.100.0修改NAT设置 2、安装Linux系统 hadoop1，安装完成后右击克隆两个作为从站的主机 分别修改主站和两个从站的IP和hostname,具体参考概述的表格。即主站为hadoop1，两个从站分别为hadoop2及hadoop3sudo vim/etc/hostname配置hosts：sudo vim/etc/hosts三台机器hosts都配置为：设置完成后可以ping下来测试三个主机之间是否可以连接。192.168.10

4、0.10 hadoop1192.168.100.11 hadoop2192.168.100.12 hadoop3 3、在主站安装Java 和 Hadoop（主站配置好后，通过分发复制到其他从站）安装步骤可以参考我的其他篇博客，配置如下。JAVA_HOME=/usr/local/jdk1.8.0_171CLASSPATH=.:$JAVA_HOME/lib.tools.jarHADOOP_HOME=/usr/local/hadoop-2.9.2PATH=$JAVA_HOME/bin:$PATH/bin:$HADOOP_HOME/binexport JAVA_HOME CLASSPATH PATH

5、HADOOP_HOME 验证验证是否成功：roothadoop1 hadoop-2.9.2#java-versionroothadoop1 hadoop-2.9.2#java-versionJava(TM)SE Runtime Environment(build 1.8.0_171-b11)Java HotSpot(TM)64-Bit Server VM(build 25.171-b11,mixed mode)roothadoop1 hadoop-2.9.2#hadoop versionHadoop 2.9.2Subversion https:/git-wip-us.apache.org/re

6、pos/asf/hadoop.git-r 826afbeae31ca687bc2f8471dc841b66ed2c6704Compiled by ajisaka on 2018-11-13T12:42ZCompiled with protoc 2.5.0From source with checksum 3a9939967262218aa556c684d107985This command was run using/usr/local/hadoop-2.9.2/share/hadoop/common/hadoop-common-2.9.2.jar 4、配置Hadoop JDK路径修改hado

7、op-env.sh、mapred-env.sh、yarn-env.sh件中的JDK路径：5、配置core-site.xml vim core-site.xml fs.defaultFS hdfs:/hadoop1:8020 hadoop.tmp.dir /usr/local/hadoop-tmp dfs.namenode.name.dir file:/$hadoop.tmp.dir/dfs/name dfs.datanode.data.dir file:/$hadoop.tmp.dir/dfs/data fs.defaultFS为NameNode的地址。hadoop.tmp.dir为hadoo

8、p临时录的地址，默认情况下，NameNode和DataNode的数据件都会存在这个录下的对应录下。应该保证此录是存在的，如果不存在，先创建。6、配置hdfs-site.xml vim hdfs-site.xml dfs.namenode.secondary.http-address hadoop3:50090 dfs.namenode.secondary.http-address是指定secondaryNameNode的http访问地址和端号，因为在规划中，我们将hadoop3规划为SecondaryNameNode服务器。7、配置slavesvim etc/hadoop/slaveshado

9、op1hadoop2hadoop38、配置yarn-site.xml vim yarn-site.xml yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.hostname hadoop2 yarn.log-aggregation-enable true yarn.log-aggregation.retain-seconds 106800 根据规划yarn.resourcemanager.hostname这个指定resourcemanager服务器指向hadoop2。yarn.log-aggregation

10、-enable是配置是否启志聚集功能。yarn.log-aggregation.retain-seconds是配置聚集的志在HDFS上最多保存多长时间。9、配置mapred-site.xml从mapred-site.xml.template复制个mapred-site.xml件。mapreduce.framework.name yarn mapreduce.jobhistory.address hadoop1:10020 mapreduce.jobhistory.webapp.address hadoop1:19888 10、设置SSH密码登录在hadoop1上成公钥，不断回车即可。rooth

11、adoop1 hadoop-2.9.2#ssh-keygen-t rsaGenerating public/private rsa key pair.Enter file in which to save the key(/root/.ssh/id_rsa):Created directory/root/.ssh.Enter passphrase(empty for no passphrase):Enter same passphrase again:Your identification has been saved in/root/.ssh/id_rsa.Your public key h

12、as been saved in/root/.ssh/id_rsa.pub.The key fingerprint is:SHA256:j3UJ6KS7FajeLVCd3/KS62GjA9ANPDWZDRftgcn6RSo roothadoop1The keys randomart image is:+-RSA 2048-+|.+*o=|+o+=+|.=oo.+.|.o=E.o.|oo S+.oo|.=+.|.o=+|.+.ooo.|.o.oooo|+-SHA256-+在hadoop1分发公钥，分别分发给三台主机。（如下，操作中需要回复yes确认以及登录密码）roothadoop1 hadoo

13、p-2.9.2#ssh-copy-id hadoop1The authenticity of host hadoop1(192.168.100.10)cant be established.ECDSA key fingerprint is SHA256:D/yWgW1PoFwxoBex4pbN4LqUAUfheZ2CRJ8ejEeregk.ECDSA key fingerprint is MD5:79:25:23:98:0c:c2:6d:a1:6d:56:1b:5f:ce:32:62:4e.Are you sure you want to continue connecting(yes/no)

14、?yes/usr/bin/ssh-copy-id:INFO:attempting to log in with the new key(s),to filter out any that are already installed/usr/bin/ssh-copy-id:INFO:1 key(s)remain to be installed-if you are prompted now it is to install the new keysroothadoop1s password:Number of key(s)added:1Number of key(s)added:1and che

15、ck to make sure that only the key(s)you wanted were added.roothadoop1 hadoop-2.9.2#ssh-copy-id hadoop2The authenticity of host hadoop2(192.168.100.11)cant be established.ECDSA key fingerprint is SHA256:D/yWgW1PoFwxoBex4pbN4LqUAUfheZ2CRJ8ejEeregk.ECDSA key fingerprint is MD5:79:25:23:98:0c:c2:6d:a1:6

16、d:56:1b:5f:ce:32:62:4e.Are you sure you want to continue connecting(yes/no)?yes/usr/bin/ssh-copy-id:INFO:attempting to log in with the new key(s),to filter out any that are already installed/usr/bin/ssh-copy-id:INFO:1 key(s)remain to be installed-if you are prompted now it is to install the new keys

17、roothadoop2s password:Number of key(s)added:1Number of key(s)added:1and check to make sure that only the key(s)you wanted were added.roothadoop1 hadoop-2.9.2#ssh-copy-id hadoop3The authenticity of host hadoop3(192.168.100.12)cant be established.ECDSA key fingerprint is SHA256:D/yWgW1PoFwxoBex4pbN4Lq

18、UAUfheZ2CRJ8ejEeregk.ECDSA key fingerprint is MD5:79:25:23:98:0c:c2:6d:a1:6d:56:1b:5f:ce:32:62:4e.Are you sure you want to continue connecting(yes/no)?yes/usr/bin/ssh-copy-id:INFO:attempting to log in with the new key(s),to filter out any that are already installed/usr/bin/ssh-copy-id:INFO:1 key(s)r

19、emain to be installed-if you are prompted now it is to install the new keysroothadoop3s password:Number of key(s)added:1Number of key(s)added:1and check to make sure that only the key(s)you wanted were added.同样的在hadoop2、hadoop3上成公钥和私钥后，将公钥分发到三台机器上。11、分发Hadoop件通过Scp分发hadoop安装包到两个从站。scp-r/usr/local/ha

20、doop-2.9.2 hadoop2:/usr/local/scp-r/usr/local/hadoop-2.9.2 hadoop3:/usr/local/12、关闭防墙：CentOS系统默认开启了防墙，在开启 Hadoop 集群之前，需要关闭集群中每个节点的防墙。有防墙会导致ping 得通但 telnet 端不通，从导致 DataNode 启动了，但 Live datanodes 为 0 的情况。systemctl stop firewalld.service#关闭firewall systemctl disable firewalld.service#禁firewall开机启动 启动集群1

21、、在主站格式化NameNode$HADOOP_HOME/bin/hdfs namenode format注意：如果需要重新格式化NameNode,需要先将原来NameNode和DataNode下的件全部删除，不然会报错，NameNode和DataNode所在录是在core-site.xml中hadoop.tmp.dir、dfs.namenode.name.dir、dfs.datanode.data.dir属性配置的。2、在hadoop1上启动HDFS$HADOOP_HOME/sbin/start-dfs.shroothadoop1 hadoop-2.9.2#$HADOOP_HOME/sbin/

22、start-dfs.shStarting namenodes on hadoop1hadoop1:starting namenode,logging to/usr/local/hadoop-2.9.2/logs/hadoop-root-namenode-hadoop1.outhadoop1:starting datanode,logging to/usr/local/hadoop-2.9.2/logs/hadoop-root-datanode-hadoop1.outhadoop2:starting datanode,logging to/usr/local/hadoop-2.9.2/logs/

23、hadoop-root-datanode-hadoop2.outhadoop3:starting datanode,logging to/usr/local/hadoop-2.9.2/logs/hadoop-root-datanode-hadooop3.outStarting secondary namenodes hadoop3hadoop3:starting secondarynamenode,logging to/usr/local/hadoop-2.9.2/logs/hadoop-root-secondarynamenode-hadooop3.out查看进程启动情况：roothadoo

24、p1 hadoop-2.9.2#jps7030 DataNode6888 NameNode7276 Jpsroothadoop2 hadoop-2.9.2#jps3385 DataNode3434 Jpsroothadooop3 hadoop-2.9.2#jps3668 DataNode3764 SecondaryNameNode3807 Jps3、在hadoop2 启动YARN$HADOOP_HOME/sbin/start-yarn.shroothadoop2 hadoop-2.9.2#$HADOOP_HOME/sbin/start-yarn.shstarting yarn daemonss

25、tarting resourcemanager,logging to/usr/local/hadoop-2.9.2/logs/yarn-root-resourcemanager-hadoop2.outhadoop3:starting nodemanager,logging to/usr/local/hadoop-2.9.2/logs/yarn-root-nodemanager-hadooop3.outhadoop1:starting nodemanager,logging to/usr/local/hadoop-2.9.2/logs/yarn-root-nodemanager-hadoop1.

26、outhadoop2:starting nodemanager,logging to/usr/local/hadoop-2.9.2/logs/yarn-root-nodemanager-hadoop2.out 查看进程启动情况：roothadoop1 hadoop-2.9.2#jps7347 NodeManager7030 DataNode6888 NameNode7469 Jpsroothadoop2 hadoop-2.9.2#jps3652 NodeManager3974 Jps3385 DataNode3551 ResourceManagerroothadooop3 hadoop-2.9

27、.2#jps3668 DataNode3764 SecondaryNameNode4021 Jps3896 NodeManager4、在hadoop1上启动志服务器$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver 查看进程：roothadooop1 hadoop-2.9.2#jps4115 Jps3668 DataNode3764 SecondaryNameNode3896 NodeManager4074 JobHistoryServer5、查看HDFS Web页根据规划，我们部署在主站，所以访问路径为 /hadoop1

28、:500706、查看YARN Web 页根据规划，我们部署在从站 hadoop2，所以访问路径为 /hadoop2:80887、查看志历史服务 Web页测试 接下来运带的 wordcounter 来验证1、创建测试的Input件$HADOOP_HOME/bin/hdfs dfs-mkdir-p/wordcountdemo/input2、上传个测试件到input件夹$HADOOP_HOME/bin/hdfs dfs-put$HADOOP_HOME/etc/hadoop/core-site.xml/wordcountdemo/input 3、运WordCount MapReduce Job$HADOOP_HOME/bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount/wordcountdemo/input/wordcountdemo/out4、查看结果