2022年hadoop平台搭建多节点详细教程一次就能搭建成功 .pdf

Hadoop 平台搭建说明1. Hadoop节点规划本次安装规划使用三个节点，每个节点都使用centos 系统。三个节点的hostname 分别规划为： centoshadoop1 、centoshadoop2 、centoshadoop3 （此处为本教程参数，可根据实际环境情况修改）三个节点的ip 地址分别规划为：192.168.65.57、192.168.65.58 、192.168.65.59（此处为本教程参数，根据实际环境情况修改）2. 平台搭建使用的软件下载如下软件操作系统安装包：Centos6.3_x64 Jdk安装包： jdk-6u37-linux-x64.bin Hadoop 安装包： hadoop-1.1.2.tar.gz 3. 安装 centos操作系统安装三个节点的操作系统，安装过程省略。4. 配置 centoshadoop1节点4.1. 修改节点 hostname rootlocalhost # vi /etc/sysconfig/network HOSTNAME=centoshadoop1 rootlocalhost # vi /etc/hosts 192.168.65.57 centoshadoop1 192.168.65.58centoshadoop2 192.168.65.59centoshadoop3 rootlocalhost #reboot 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 1 页，共 8 页 - - - - - - - - - 4.2. 关闭 iptables防火墙rootcentoshadoop1#service iptables stop 注意每次操作系统重启后都要操作4.3. 建立无 ssh 密码登陆生成签名文件rootcentoshadoop1#cd /root rootcentoshadoop1#ssh-keygen -t dsa -P -f /.ssh/id_dsa rootcentoshadoop1#cat /.ssh/id_dsa.pub /.ssh/authorized_keys rootcentoshadoop1# 测试本地SSH无密码登录rootcentoshadoop1#sshcentoshadoop1 4.4. 安装 jdk 上传 jdk-6u37-linux-x64.bin 到/root 目录下rootcentoshadoop1#chmod 777 jdk-6u37-linux-x64.bin rootcentoshadoop1#./jdk-6u37-linux-x64.bin rootcentoshadoop1#ll 查看生成jdk-6u37-linux-x64 目录4.5. 安装 hadoop软件上传 hadoop-1.1.2.tar.gz 到/root 目录下rootcentoshadoop1#tar -zvxf hadoop-1.1.2.tar.gz rootcentoshadoop1#ll 查看生成hadoop-1.1.2 目录rootcentoshadoop1#vi /conf/core-site.xml fs.default.name hdfs:/192.168.65.57:9000 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 2 页，共 8 页 - - - - - - - - - hadoop.tmp.dir /root/hadoop-1.1.2/tmp rootcentoshadoop1#vi hdfs-site.xml dfs.replication 1 rootcentoshadoop1#vi mapred-site.xml mapred.job.tracker 192.168.65.57:9001 4.6. 配置 hadoop数据节点rootcentoshadoop1#vi /root/hadoop-1.1.2/conf/masters 192.168.65.57 #secondaryNameNode rootcentoshadoop1#vi /root/hadoop-1.1.2/conf/slaves 192.168.65.58 #datanode 192.168.65.59 #datanode 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 3 页，共 8 页 - - - - - - - - - 4.7. 添加环境变量rootcentoshadoop1#cd /root rootcentoshadoop1#vi /etc/profile export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_HOME_WARN_SUPPRESS=1 export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH rootcentoshadoop1#vi /.bashrc export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_DEV_HOME=/root/hadoop-1.1.2 export HADOOP_COMMON_HOME=/root/hadoop-1.1.2 export HADOOP_HDFS_HOME=/root/hadoop-1.1.2 export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf rootcentoshadoop1#vi /root/hadoop-1.1.2/conf/hadoop-env.sh export JAVA_HOME=/root/jdk1.6.0_37 export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib 5. 配置 centoshadoop2节点5.1. 修改节点 hostname rootlocalhost # vi /etc/sysconfig/network HOSTNAME=centoshadoop2 rootlocalhost # vi /etc/hosts 192.168.65.57 centoshadoop1 192.168.65.58 centoshadoop2 192.168.65.59 centoshadoop3 rootlocalhost #reboot 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 4 页，共 8 页 - - - - - - - - - 5.2. 关闭 iptables防火墙rootcentoshadoop1#service iptables stop 注意每次操作系统重启后都要操作5.3. 建立无 ssh 密码登陆将 centoshadoop1 节点上 /root/.ssh目录下的authorized_keys、known_hosts 、id_rsa.pub、id_rsa 四个文件拷贝到本机的/root/.ssh 目录下5.4. 安装 jdk 上传 jdk-6u37-linux-x64.bin 到/root 目录下rootcentoshadoop1#chmod 777 jdk-6u37-linux-x64.bin rootcentoshadoop1#./jdk-6u37-linux-x64.bin rootcentoshadoop1#ll 查看生成jdk-6u37-linux-x64 目录5.5. 安装 hadoop软件上传 hadoop-1.1.2.tar.gz 到/root 目录下rootcentoshadoop1#tar -zvxf hadoop-1.1.2.tar.gz rootcentoshadoop1#ll 查看生成hadoop-1.1.2 目录rootcentoshadoop1#vi /conf/core-site.xml fs.default.name hdfs:/192.168.65.57:9000 hadoop.tmp.dir /root/hadoop-1.1.2/tmp 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 5 页，共 8 页 - - - - - - - - - rootcentoshadoop1#vi hdfs-site.xml dfs.replication 1 rootcentoshadoop1#vi mapred-site.xml mapred.job.tracker 192.168.65.57:9001 5.6. 添加环境变量rootcentoshadoop1#cd /root rootcentoshadoop1#vi /etc/profile export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_HOME_WARN_SUPPRESS=1 export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$PATH rootcentoshadoop1#vi /.bashrc export JAVA_HOME=/root/jdk1.6.0_37 export JRE_HOME=/root/jdk1.6.0_37/jre 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 6 页，共 8 页 - - - - - - - - - export HADOOP_HOME=/root/hadoop-1.1.2 export HADOOP_DEV_HOME=/root/hadoop-1.1.2 export HADOOP_COMMON_HOME=/root/hadoop-1.1.2 export HADOOP_HDFS_HOME=/root/hadoop-1.1.2 export HADOOP_CONF_DIR=/root/hadoop-1.1.2/conf rootcentoshadoop1#vi /root/hadoop-1.1.2/conf/hadoop-env.sh export JAVA_HOME=/root/jdk1.6.0_37 export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib 6. 配置 centoshadoop3节点6.1. 修改节点 hostname rootlocalhost # vi /etc/sysconfig/network HOSTNAME=centoshadoop3 rootlocalhost # vi /etc/hosts 192.168.65.57 centoshadoop1 192.168.65.58 centoshadoop2 192.168.65.59 centoshadoop3 rootlocalhost #reboot 关闭 iptables 防火墙、建立无ssh密码登陆、安装jdk、安装 hadoop 软件、添加环境变量等操作跟“配置centoshadoop2 节点”完全相同7. 启动运行 hadoop平台7.1. 首先执行格式化rootcentoshadoop1#hadoop namenode format 注意： core-site.xml、hdfs-site.xml 、mapred-site.xml 等配置文件变化后都需要执行格式化7.2. 启动 hadoop rootcentoshadoop1#start-all.sh 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 7 页，共 8 页 - - - - - - - - - 7.3. 停止 hadoop rootcentoshadoop1#stop-all.sh 7.4. 查看 hadoop运行状态查看 hadoop 启动的进程rootcentoshadoop1#jps 查看 hadoop 集群状态rootcentoshadoop1#hadoop dfsadmin -report 查看 hadoop 中的目录情况rootcentoshadoop1#hadoop fs ls / 名师资料总结 - - -精品资料欢迎下载 - - - - - - - - - - - - - - - - - - 名师精心整理 - - - - - - - 第 8 页，共 8 页 - - - - - - - - -