大数据学习5（基于yarn的高可用集群搭建）

Posted 2020-12-19 bug开发工程师

	NN-1	NN-2	DN	ZK	ZKFC	JNN	RS	NM(和DN在一起)
node0001	*				*	*
node0002		*	*	*	*	*		*
node0003			*	*		*	*	*
node0004			*	*			*	*

 

 

 

 

 

 

1，修改配置文件

cd /opt/sxt/hadoop-2.6.5/etc/hadoop

mv mapred-site.xml.template mapred-site.xml

vi mapred-site.xml    (插入)

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>

vi yarn-site.xml

<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
<property>
   <name>yarn.resourcemanager.ha.enabled</name>
   <value>true</value>
 </property>
 <property>
   <name>yarn.resourcemanager.cluster-id</name>
   <value>cluster1</value>
 </property>
 <property>
   <name>yarn.resourcemanager.ha.rm-ids</name>
   <value>rm1,rm2</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm1</name>
   <value>node0003</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname.rm2</name>
   <value>node0004</value>
 </property>
 <property>
   <name>yarn.resourcemanager.zk-address</name>
   <value>node0002:2181,node0003:2181,node0004:2181</value>
 </property>

 

2,分发

scp mapred-site.xml  yarn-site.xml node0002:`pwd`

scp mapred-site.xml  yarn-site.xml node0003:`pwd`

scp mapred-site.xml  yarn-site.xml node0004:`pwd`

3,两个ResourceManager相互免密钥

node0003:cd .ssh

ssh-keygen -t dsa -P ‘‘ -f ./id_dsa

cat id_dsa.pub >> authorized_keys

scp id_dsa.pub node0004:`pwd`/node0003.pub

node0004:cd .ssh

cat node0003.pub >> authorized_keys

ssh-keygen -t dsa -P ‘‘ -f ./id_dsa

cat id_dsa.pub >> authorized_keys

scp id_dsa.pub node0003:`pwd`/node0004.pub

node0003:

cat node0004.pub >> authorized_keys

相互访问没问题ok

4，启动

zkServer.sh start（启动zookeeper,002,003,004）

start-dfs.sh  (启动hdfs，mapreduce基于hdfs,0001)

start-yarn.sh   (启动yarn,0001)

jps 查看 0002，0003，0004 都有NodeManager进程

yarn-daemon.sh start resourcemanager    （启动resourcemanager ，0003，0004）

jps 查看 0003，0004 都有ResourceManager 进程

5，浏览器访问

node0003:8088

6,停止

node0003,node0004: yarn-daemon.sh stop resourcemanager （停止resourcemanager）

node0001: stop-yarn.sh (停止nodemanager)

node0001: stop-dfs.sh

node0002,node0003,node0004: zkServer.sh stop