大数据集群HA的搭建

Posted 2022-03-30 yjmxx1314

一:手动HA切换搭建

实现步骤:

    1.修改hdfs-site.xml

<configuration>

    

          <property>

                    <name>dfs.nameservices</name>

                    <value>mycluster</value>

         </property>

          

 

          <property>

                    <name>dfs.ha.namenodes.mycluster</name>

                    <value>nn1,nn2</value>

         </property>

        

 

         <property>

           <name>dfs.namenode.rpc-address.mycluster.nn1</name>

           <value>hadoop01:8020</value>

         </property>

         <property>

           <name>dfs.namenode.rpc-address.mycluster.nn2</name>

           <value>hadoop02:8020</value>

         </property>

        

 

         <property>

           <name>dfs.namenode.http-address.mycluster.nn1</name>

           <value>hadoop01:50070</value>

         </property>

         <property>

           <name>dfs.namenode.http-address.mycluster.nn2</name>

           <value>hadoop02:50070</value>

         </property>

        

         <property>

           <name>dfs.namenode.shared.edits.dir</name>

           <value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/mycluster</value>

        </property>

        

 

         <property>

           <name>dfs.client.failover.proxy.provider.mycluster</name>

           <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

         </property>

        

 

         <property>

          <name>dfs.ha.fencing.methods</name>

          <value>sshfence</value>

        </property>

 

        <property>

          <name>dfs.ha.fencing.ssh.private-key-files</name>

          <value>/home/hadoop/.ssh/id_rsa</value>

        </property>

        

        <property>

          <name>dfs.ha.fencing.ssh.connect-timeout</name>

          <value>30000</value>

        </property>

 

          <property>

            <name>dfs.permissions.enabled</name>

            <value>false</value>

        </property>

        

</configuration>

 

 

2.修改core-site.xml配置文件

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://mycluster</value>

    </property>

 

3.配置yarn-site.xml配置文件

<configuration>

   <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

   </property>

   <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>hadoop02</value>

   </property>

  

   <property>

        <name>yarn.log-aggregation-enable</name>

        <value>true</value>

   </property>

  

   <property>

        <name>yarn.log-aggregation.retain-seconds</name>

        <value>604800</value>

   </property>

  

</configuration>

 

4.配置mapred-site.xml

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

        

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>hadoop02:10020</value>

    </property>

        

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>hadoop02:19888</value>

    </property>

</configuration>

 

5.完成配置之后,就可以把配置文件分发到其他的节点上,分发之前必须保证/etc/hosts文件和主机名称做好映射

[[email protected] hadoop-2.7.2]# scp -r etc/hadoop/ hadoop02:/opt/app/hadoop-2.7.2/etc/

[[email protected] hadoop-2.7.2]# scp -r etc/hadoop/ hadoop03:/opt/app/hadoop-2.7.2/etc/

 

注解: 1):把hadoop的目录的用户权限改为hadoop.如果是在root启动直接启动就可以了

        2):启动之前,需要把残留进程以及相关临时文件清除掉 /tmp  hadoop目录中/data/tmp 全部清空

        3):启动之前,先把hadoop的权限修改为hadoop用户

 

*6*.启动的顺序

    1):先启动zookeeper

[[email protected] zookeeper-3.4.10]$ bin/zkServer.sh start

    2):再启动jouranlnode

[[email protected] hadoop-2.7.2]$ sbin/hadoop-daemon.sh start journalnode

 

注解: 所有节点都要进行启动

 

    3):格式化namenode

[[email protected] hadoop-2.7.2]$ bin/hadoop namenode -format

   

     4):先启动格式化好的namenode

[[email protected] hadoop-2.7.2]$ sbin/hadoop-daemon.sh start namenode

    5):要在第二台节点上,同步namenode1的元数据

[[email protected] hadoop-2.7.2]$ bin/hdfs namenode -bootstrapStandby

    6);手动切换 active

[[email protected] hadoop-2.7.2]$ bin/hdfs haadmin -transitionToActive nn1

 

二:自动HA的搭建

    

    1.配置hdfs-site.xml配置文件

        <property>

            <name>dfs.ha.automatic-failover.enabled</name>

            <value>true</value>

         </property>

说明: 开启自动故障切换

 

    2.配置core-site.xml配置文件

 <property>

   <name>ha.zookeeper.quorum</name>

   <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>

 </property>

说明:把以上的配置分发到其他的节点上

 

    3.初始化HA状态到zookeeper中

[[email protected] hadoop-2.7.2]$ bin/hdfs zkfc  -formatZK

 

    4.启动hdfs

 

sbin/start-dfs.sh