hadoop2.6完全分布式安装.docx
系统准备Hadoop完全分布式安装，服务器最好都是基数，我用了三台虚拟机。hadoop2.6.0 完全分布式 masterhadoop2.6.0 完全分布式 salvesOlhadoop2.6.0 完全分布式 salves02系统环境设置修改虚拟机的主机名称代码如下：1、修改第一台虚拟机 vim /etc/sysconfig/networkNETWORKING=yesHOSTNAME=masterroot<atocalhost # vim /etc/sysconfig/network rootlocalhost # cat /etc/sysconfig/network NETWORKING=yesHOSTNAME=masterootlocalhost # 2、修改第一台虚拟机vim /etc/sysconfig/networkNETWORKING=yesHOSTNAME=salves01rootlocalhost # vim /etc/sysconfig/network rootlocalhost # cat /etc/sysconfig/network NETWORKING=yesH0STNAME=salves01rootlocalhost*# |3、修改第一台虚拟机vim /etc/sysconfig/networkNETWORKING=yesHOSTNAME=salves02rootlocalhost vim /etc/sysconfig/network rootlocalhost -# cat /etc/sysconfig/network NETWORKING二yesH0STNAME=salves02rootlocalhost # |配置IP地址代码如下：1、修改第一台服务器vim /etc/sysconfig/network-scripts/ifcfg-ethODEVICE=ethO#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec3ONBOOT=yes虚拟机启动时就启动网络NM_CONTROLLED=yesBOOTPROTO=staticIPADDR=192.168.83.100root(aiocalhost -# vim /etc/sysconfig/network-scripts/ifcfg-ethO rootlocalhost cat /etc/sysconfig/network-scripts/ifcfg-ethO DEVICE=eth0#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec30NB00T=yesNM_CONTROLLED=yesB00TPR0T0=staticrootcaiocalhost |2、修改第二台服务器vim /etc/sysconfig/network-scripts/ifcfg-ethODEVICE=ethO#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec3ONBOOT=yes 虚拟机启动时就启动网络NM_CONTROLLED=yesBOOTPROTO=staticroot(atocalhost vim /etc/sysconfig/network-scripts/ifcfg-ethO root(aiocalhost j# cat /etc/sysconfig/network-scripts/ifcfg-ethO DEVICE=ethQ#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec3ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=staticroot(aiocalhost T# |3、修改第三台服务器vim /etc/sysconfig/network-scripts/ifcfg-ethODEVICE=ethO#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec3ONBOOT=yes虚拟机启动时就启动网络NM_CONTROLLED=yesBOOTPROTO=staticIPADDR=192.168.83.102rootlocalhost -# vim /etc/sysconfig/network-scripts/ifcfg-ethO root(aiocalhost -# cat /etc/sysconfig/network-scripts/ifcfg-ethO DEVICE=ethQ#HWADDR=00:0C:29:8C:FB:39TYPE=EthernetUUID=92d31d5c-369a-4e3d-8fbc-140ef4ff3ec30NB00T=yesNM_CONTROLLED=yesB00TPR0T0=staticrootlocalhost -#service network restart 使配置的 ip 起作用修改主机名和IP的映射关系以及其他虚拟机的关系(hosts)代码如下:vim /etc/hosts127.0.0.1 localhost localhost.localdomain Iocalhost4 Iocalhost4.localdomain4:1localhost localhost.localdomain Iocalhost6 Iocalhost6.localdomain6mastersalvesOlsalves02rootlocalhost -# vim /etc/hostsrootQlocalhost j# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 Iocalhost4.localdomain4:1localhost localhost.localdomain Localhost6 localhost6.Iocaldomain6192.168. 83.100 master192.168.83.101 salvesOl192.168.83.102 salves02rootlocalhost -# |三台虚拟机都需要配置，三台虚拟机的hosts 一样关闭防火墙代码如下:service iptables status service iptables stop chkconfig iptables -list chkconfig iptables off查看防火墙的状态关闭防火墙查看防火墙开机启动状态 关闭防火墙开机启动rootlocalhost -# service iptables status Table: filterChain INPUTChain INPUT(policy ACCEPT)num12345target ACCEPT ACCEPT ACCEPT ACCEPT REJECTprot all icmp all tcp alloptsourcestate RELATED ESTABLISHEDstate NEW tcp dpt:22 reject-with icmp-host-prohibitedChain FORWARD num target 1 REJECT(policyprotallACCEPT) opt sourcereject-with icmp-host-prohibitedChain OUTPUT num target(policyprotACCEPT) opt sourcedestinationrootlocal.host *# service iptables stop iptables: Setting chains to policy ACCEPT: filter iptables: Flushing firewall rules:iptables: Unloading modules:OKOKOKrootlocal.host iptablesrootlocalhost rootlocalhost一# chkconfig 0:off l:off # chkconfig7# Iiptables2: oniptables-list 3:on 4:on 5:onoff6:off三台虚拟机都需要关闭防火墙重启系统代码如下：reboot 重启系统安装jdk准备jdk在网上下载64位的jdk,下载好了之后上传到虚拟机中在Ubuntu下切换到root用户代码如下：进入：cd /usr/local/仓犍：mkdirjava解压 jdk(jdk-7u55-linux-x64.tar.gz)代码如下:tar -zxvf jdk-7u55-1 inux-x64.tar.gz -C /usr/local/java/配置坏境变量代码如下：vim /etc/profile(在后面添加)export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexport PATH=$JAVA HOME/bin:$JAVA JRE/bin:$PATH重启/etc/profile代码如下：source /etc/profile查看是否配置成功（jdk的版本）代码如下：java -versionrootmaster # java -versionjava version "1.7.055"Java(TM) SE Runtime Environment (build 1.7.0 55-bl3)Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03r mixed mode)一# |统默认自己配置的jdk代码如下：sudo update-alternatives -config java配置ssh免密码登录代码如下：ssh-keygen -t rsa仓ll建 keyGenerating public/private rsa key pair.Enter file in which to save the key (/home/salve/.ssh/idrsa)：Created directory 1/home/salve/.ssh,.Enter passphrase (empty for no passphrase):Enter same passphrase again：Your identification has been saved in /home/salve/.ssh/idrsa.Your public key has been saved in /home/salve/.ssh/idrsa.pub. The key fingerprint is:19:5b:If:9b:38:c4:c9:c1:9e:f6:51:b4:74:84:cf:c6 salvemasterThe key's randomart image is:+- RSA 2048-十I.0001I0.0 .00 III|*+0.+ E|S.O. + .cat /.ssh/id_rsa.pub » /.ssh/authorized_keys 把 key 写在 authorized_keys 中II /.ssh/authorized_keys 查看是否有这个文件chmod 600 authorized_keys给这个文件赋予权限ssh localhost 执行切换，看是否是免密码登录把三台服务器中的id_rsa.pub (key),分别加入三台服务器中的authorized_keys文件中,打通三台服务器免密码登录。rootmaster # ssh master Last login: Tue May 24 23:48:40 rootmaster # ssh salvesOlLast root Last rootlogin: Tue May 24 ©salvesOl # ssh login: Tue May 24 ©salves02 # sshssh: could not resolve rootsalves02 # ssh Last login: Tue May 24 rootsalvesOl # ssh23:44:47 salves02 23:44:41 savlesOl hostname salvesOl 23:49:00 master23:48:49Last login: Tue May 24 rootmaster # ssh salves02Last rootlogin: Tue May 24 23:49:07 salves02 # ssh masterLast login: Tue May 24 23:49:36 rootmaster #201620162016fromfromfromsavlesOl:2016201620162016fromfromfromfromsalves02masterName or servi ce not knownmastermastersalvesOlsalvesOl安装 准备hadoop压缩文件这个压缩文件在apache官网中下载的都是32位的，如果系统是64位的，那么自己需要 编译为64位的压缩包，不过不编译也没事，影响不大。把下载的文件上传到虚拟机中解压 代码如下：在master服务器中解压tar -zxvf /home/salve/package/hadoop-2.6.0.tar.gz -C /home/hadoop/cd /home/hadoop/hadoop-2.6.0 进入hadoop解压目录下创立以下目录 mkdir tmpmkdir-p dfs/namemkdir-p dfs/datarootmaster hadoop-2.6.0# mkdir tmprootmaster hadoop-2.6.0# mkdir -p dfs/namerootmaster hadoop-2.6.0# mkdir -p dfs/datarootmaster hadoop-2.6.0# IsLICENSE.tXt NOTICE.tXt README.tXt配置hadoop,修改配置文件（所有节点上的配置都一样）.配置 hadoop-env.sh代码如下：vim hadoop-env.sh# The java implementation to use.ftexport JAVA_HOME=$JAVA_HOMEexport # The java implementation to use.# export JAVA_HOME=5JAVA-HOME export JAVA_HOME=/usr/local/java/jdkl.7.O_551 .配置 yarn-env.sh代码如下：vim yarn-env.shsome Java parameters# some Java parameters# export JAVA_HOME=»/ho<ne/y/libexec/idkl. 6. 0/ export JAVA_HOME=/usr/Iocal/java/jdkl.7.O_55if f,“"" ; then#echo "run java in $java_home"JAVA_HOME=fi3 .配置从节点slaves代码如下：vim slaveslocalhostsalvesOlsalvesO2rootmaster hadoop# vim siaves local hostsalvesOlsalves02.配置 core-site.xml代码如下：vim core-site.xml<configuration><property><name>fs.defaultFS</name><value>hdfs:/master:8020</value></property><property><name>io.file.buffer.size</name>
