hadoop的HA机制+zookeeper

Posted 我想静静

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了hadoop的HA机制+zookeeper相关的知识,希望对你有一定的参考价值。

 关于hadoop的HA配置以及wordcount测试

一,简单环境配置

1,查看centos版本位数:

$>getconf LONG_BIT,

2,桌面模式和文本模式之间进行切换:

1),在终端命令行进行设置时只能暂时改变模式,

     $>init 3    表示切换到文本模式

     $>init 5    表示切换到桌面模式

2),永久改变模式需要修改配置文件,进入到etc目录下

      $>sudo nano inittab   修改该文件最后一行

     若需要文本模式则改为 id:3:initdefault

     若需要桌面模式则改为:id:5:initdefault

注意:改为桌面模式时最好加大内存,改为4G左右

3,配置静态ip,修改主机名,配置hosts文件

    1),进入桌面模式下进行配置,

2),查看配置的静态ip

     cd /etc/sysconfig/networking/devices/ifcfg-eth2

3),修改主机名

          nano /etc/sysconfig/network  本机的主机名

4,域名映射的配置:

      nano /etc/hosts  集群中的主机域名映射表

5,检验配置是否生效

ping hostname观察网络配置是否生效

6,关闭防火墙

          1)查看防火墙状态

                   service iptables status

          2)关闭防火墙

             service iptables stop

          3)查看防火墙开机启动状态

                   chkconfig iptables --list

          4)关闭防火墙开机启动

                   chkconfig iptables off

7,配置ssh免登陆

           1)生成ssh免登陆密钥

              进入到我的home目录

                 cd ~/.ssh

ssh-keygen -t rsa (四个回车)

           2)执行完这个命令后,会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)

              将公钥拷贝到要免登陆的机器上

                 ssh-copy-id localhost

        

 

二,配置HA

1,让两个NN节点在某个时间只能有一个节点正常响应客户端请求,响应请求的必须为ACTIVE状态的那一台

2,standby状态的节点(第二个namenode)必须能够无缝的切换为ACTIVE状态,意味着两个NN必须时时刻刻保持一致

3,管理edits,写了一个分布式应用qjournal:它依赖于zookeeper实现

4,如何避免状态切换时发生brain split现象

   fencing: ssh发送kill指令

   执行自定义shell脚本

  

5,部署方式:

3台机器:

第一台                 第二台                              第三台

nn1                    nn2

zkfc1                  zkfc2                                   dn1                                                                   

zk1,                   zk2,                                    zk3

jn1,                   jn2,                                     jn3

RM                     RM                                    NM

 

7台机器:

1、        namenode                    zkfc

2,         namenode                    zkfc

3,         resourcemanager

4,         resourcemanager

5,         zookeeper  journalnode  datanode  nodemanager

6,         zookeeper  journalnode  datanode  nodemanager

7,         zookeeper  journalnode  datanode  nodemanager

hadoop2.0已经发布了稳定版本了,增加了很多特性,比如HDFS HA、YARN等。最新的hadoop-2.4.1又增加了YARN HA

前期准备就不详细说了,我下面的HA配置选择的是3个节点

1.修改Linux主机名

2.修改IP

3.修改主机名和IP的映射关系

4.关闭防火墙

5.ssh免登陆

6.安装JDK,配置环境变量等

集群规划:

         主机名              IP                       安装的软件                                          运行的进程

         s104               192.168.43.104  jdk、hadoop,zookeeper1            NN1,zkfc1,RM1,QPM,JN1 

         s106               192.168.43.106     jdk、hadoop,zookeeper2      NN2,zkfc2,RM2,QPM,JN2 

         s108               192.168.43.108     jdk、hadoop、zookeeper3        DN,NM,QPM,JN3     

说明:

1.在hadoop2.0中通常由两个NameNode组成,一个处于active状态,另一个处于standby状态。Active NameNode对外提供服务,而Standby NameNode则不对外提供服务,仅同步active namenode的状态,以便能够在它失败时快速进行切换。

         hadoop2.0官方提供了两种HDFS HA的解决方案,一种是NFS,另一种是QJM。这里我们使用简单的QJM。在该方案中,主备NameNode之间通过一组JournalNode同步元数据信息,一条数据只要成功写入多数JournalNode即认为写入成功。通常配置奇数个JournalNode

         这里还配置了一个zookeeper集群,用于ZKFC(DFSZKFailoverController)故障转移,当Active NameNode挂掉了,会自动切换Standby NameNode为standby状态

2.hadoop-2.2.0中依然存在一个问题,就是ResourceManager只有一个,存在单点故障,hadoop-2.4.1解决了这个问题,有两个ResourceManager,一个是Active,一个是Standby,状态由zookeeper进行协调

安装步骤:

1.安装配置zooekeeper集群(在s104上)

1).上传zk安装包到~/目录下

2).解压 tar -zxvf zookeeper-3.4.5.tar.gz -C app

3),进入zookeeper-3.4.5,然后进入到文件夹conf进行配置(先在一台节点上配置)

3.1添加一个zoo.cfg配置文件

                  $>zookeeper-3.4.5/conf

                  $>mv zoo_sample.cfg zoo.cfg

3.2修改配置文件(zoo.cfg)

                        dataDir=/home/hadoop/app/zookeeper-3.4.5/data

                         server.1=itcast05:2888:3888

                         server.2=itcast06:2888:3888

                         server.3=itcast07:2888:3888

3.3在(dataDir=/home/hadoop/app/zookeeper-3.4.5/data)创建一个myid文件,里面内容是server.N中的N(server.2里面内容为2)

                       echo "1" > myid

3.4将配置好的zk拷贝到其他节点

                       scp -r /home/hadoop/app/zookeeper-3.4.5/ s106:/home/hadoop/app

                       scp -r /home/hadoop/app/zookeeper-3.4.5/ s108:/home/hadoop/app

3.5注意:在其他节点上一定要修改myid的内容

                      在s106应该讲myid的内容改为2 (echo "2" > myid)

                      在s108应该讲myid的内容改为3 (echo "3" > myid)

4).启动集群(后面配置完成进行集群测试时再关闭,然后按步骤启动)

                分别启动zk,进入到zookeeper的bin目录下

       ./zkServer.sh start

           默认端口号查看:netstat -nltp | grep 2181

           查看运行状态: ./zkServer.sh status

5),设置成功后,进入命令行客户端,连接到集群访问数据,做测试

           cd zookeeper-3.4.5/bin          ./zkCli.sh

6),zookeeper管理客户存放的数据采用的是类似于文件树的结构,每一个节点叫做node。

           注意:zookeeper集群能够开启,开启的节点数必须超过配置的一半    

2.安装配置hadoop集群(在s104上操作)

2.1解压

                            tar -zxvf hadoop-2.4.1.tar.gz -C app

2.2配置HDFS(hadoop2.0所有的配置文件都在$HADOOP_HOME/etc/hadoop目录下)

                     #将hadoop添加到环境变量中

                            vim /etc/profile

                            export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

                            export HADOOP_HOME=/home/hadoop/app/hadoop-2.4.1

                            export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

                     #hadoop2.0的配置文件全部在$HADOOP_HOME/etc/hadoop下

                            cd /home/hadoop/app/hadoop-2.4.1/etc/hadoop

2.2.1修改hadoop-env.sh

                                     export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

2.2.2修改core-site.xml

<configuration>

                             <!-- 指定hdfs的nameservice为ns1 -->

                                               <property>

                                                        <name>fs.defaultFS</name>

                                                        <value>hdfs://ns1/</value>

                                               </property>

                                               <!-- 指定hadoop临时目录 -->

                                               <property>

                                                        <name>hadoop.tmp.dir</name>

                                                        <value>/home/hadoop/app/hadoop-2.4.1/tmp</value>

                                               </property>

                                               <!-- 指定zookeeper地址 -->

                                               <property>

                                                        <name>ha.zookeeper.quorum</name>

                                                        <value>s104:2181,s106:2181,s108:2181</value>

                                               </property>

                                     </configuration>

2.2.3修改hdfs-site.xml

                                     <configuration>

                                     <!--指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 -->

                                               <property>

                                                        <name>dfs.nameservices</name>

                                                        <value>ns1</value>

                                               </property>

                                     <!-- ns1下面有两个NameNode,分别是nn1,nn2 -->

                                               <property>

                                                        <name>dfs.ha.namenodes.ns1</name>

                                                        <value>nn1,nn2</value>

                                               </property>

                                               <!-- nn1的RPC通信地址 -->

                                               <property>

                                                        <name>dfs.namenode.rpc-address.ns1.nn1</name>

                                                        <value>s104:9000</value>

                                               </property>

                                               <!-- nn1的http通信地址 -->

                                               <property>

                                                        <name>dfs.namenode.http-address.ns1.nn1</name>

                                                        <value>s104:50070</value>

                                               </property>

                                              <!-- nn2的RPC通信地址 -->

                                               <property>

                                                        <name>dfs.namenode.rpc-address.ns1.nn2</name>

                                                        <value>s106:9000</value>

                                               </property>

                                               <!-- nn2的http通信地址 -->

                                               <property>

                                                        <name>dfs.namenode.http-address.ns1.nn2</name>

                                                        <value>s106:50070</value>

                                               </property>

                                               <!-- 指定NameNode的元数据在JournalNode上的存放位置 -->

                                               <property>

                                                        <name>dfs.namenode.shared.edits.dir</name>

                                                        <value>qjournal://s104:8485;s106:8485;s108:8485/ns1</value>

                                               </property>

                                               <!-- 指定JournalNode在本地磁盘存放数据的位置 -->

                                               <property>

                                                        <name>dfs.journalnode.edits.dir</name>

                                                        <value>/home/hadoop/app/hadoop-2.4.1/journaldata</value>

                                               </property>

                                               <!-- 开启NameNode失败自动切换 -->

                                               <property>

                                                        <name>dfs.ha.automatic-failover.enabled</name>

                                                        <value>true</value>

                                               </property>

                                               <!-- 配置失败自动切换实现方式 -->

                                               <property>

<name>dfs.client.failover.proxy.provider.ns1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行-->

                                               <property>

                                                        <name>dfs.ha.fencing.methods</name>

                                                        <value>

                                                                 sshfence

                                                                 shell(/bin/true)

                                                        </value>

                                               </property>

         <!-- 使用sshfence隔离机制时需要ssh免登陆 -->

                                               <property>

                                                        <name>dfs.ha.fencing.ssh.private-key-files</name>

                                                        <value>/home/hadoop/.ssh/id_rsa</value>

                                               </property>

                                               <!-- 配置sshfence隔离机制超时时间 -->

                                               <property>

                                                        <name>dfs.ha.fencing.ssh.connect-timeout</name>

                                                        <value>30000</value>

                                               </property>

                                     </configuration>

                           

2.2.4修改mapred-site.xml

                                     <configuration>

                                               <!-- 指定mr框架为yarn方式 -->

                                               <property>

                                                        <name>mapreduce.framework.name</name>

                                                        <value>yarn</value>

                                               </property>

                                     </configuration>     

                           

2.2.5修改yarn-site.xml

                                     <configuration>

                                                        <!-- 开启RM高可用 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.ha.enabled</name>

                                                           <value>true</value>

                                                        </property>

                                                        <!-- 指定RM的cluster id -->

                                                        <property>

                                                           <name>yarn.resourcemanager.cluster-id</name>

                                                           <value>yrc</value>

                                                        </property>

                                                        <!-- 指定RM的名字 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.ha.rm-ids</name>

                                                           <value>rm1,rm2</value>

                                                        </property>

                                                        <!-- 分别指定RM的地址 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.hostname.rm1</name>

                                                           <value>s104</value>

                                                        </property>

                                                        <property>

                                                           <name>yarn.resourcemanager.hostname.rm2</name>

                                                           <value>s106</value>

                                                        </property>

                                                        <!-- 指定zk集群地址 -->

                                                        <property>

                                                           <name>yarn.resourcemanager.zk-address</name>

                                                           <value>s104:2181,s106:2181,s108:2181</value>

                                                        </property>

                                                        <property>

                                                           <name>yarn.nodemanager.aux-services</name>

                                                           <value>mapreduce_shuffle</value>

                                                        </property>

                                     </configuration>     

                                    

2.2.6修改slaves(slaves是指定子节点的位置,因为要在s104上启动HDFS、在s104又启动yarn,所以s104上的slaves文件既指定的是datanode的位置,又指定的是nodemanager的位置)

                                     s108

2.2.7配置免密码登陆

                   #首先要配置s104到s106,s108的免密码登陆

                   #在s104上生产一对钥匙

                      ssh-keygen -t rsa

                   #将公钥拷贝到其他节点,包括自己

                      ssh-coyp-id s104

                     ssh-coyp-id s106

                      ssh-coyp-id s108

                   #配置s106到s104,s108的免密码登陆(两个namenode之间要配置ssh免密码登陆,别忘了配置s106到s104的免登陆)

                   #在s106上生产一对钥匙

                      ssh-keygen -t rsa

                   #将公钥拷贝到其他节点

                      ssh-coyp-id s104

                      ssh-coyp-id s106

                      ssh-coyp-id s108                                    

2.4将配置好的hadoop拷贝到其他节点

                            scp -r /home/hadoop/app/hadoop-2.4.1/ hadoop@s106:/home/hadoop/app/

                            scp -r /home/hadoop/app/hadoop-2.4.1/ hadoop@s108:/home/hadoop/app/

###注意:严格按照下面的步骤

2.5启动zookeeper集群(分别在s104,s105,s106上启动zk)

                            cd /weekend/zookeeper-3.4.5/bin/

                            ./zkServer.sh start

                            #查看状态:一个leader,两个follower

                            ./zkServer.sh status        

2.6启动journalnode(分别在在s104,s105,s106上执行)

                            cd /home/hadoop/app/hadoop-2.4.1

                            sbin/hadoop-daemon.sh start journalnode

                            #运行jps命令检验,s104,s105,s106上多了JournalNode进程

2.7格式化HDFS

                            #在s104上执行命令:

                            hdfs namenode -format

                          #格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件,这里我配置的是/home/hadoop/app/hadoop-2.4.1/tmp,将其拷贝到s106 的/home/hadoop/app/hadoop-2.4.1/下。

                           scp -r tmp/ s106:/home/hadoop/app/hadoop-2.4.1/

                            ##也可以这样,建议hdfs namenode –bootstrapStandby

2.8格式化ZKFC(在s104上执行

以上是关于hadoop的HA机制+zookeeper的主要内容,如果未能解决你的问题,请参考以下文章

Hadoop HA 机制学习

好程序员大数据学习路线分享Hadoop阶段的高可用配置

hadoop的HA机制+zookeeper

HA机制下的Hadoop配置

Hadoop2 HA介绍

Hadoop (HDFS) HA架构通信机制