spark（HA）集群安装

Posted 2022-09-24 捡黄金的少年

事先搭建好zookeeper和hadoop集群，（启动zk和hadoop）

1、下载安装包

• https://archive.apache.org/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz

• spark-2.3.3-bin-hadoop2.7.tgz

2、解压文件

cd /kkb/soft
tar -zxvf  spark-2.3.3-bin-hadoop2.7.tgz  -C /kkb/install 

3、修改配置文件spark-env.sh  

• cd /kkb/install/spark-2.3.3-bin-hadoop2.7/conf/

• cp spark-env.sh.template spark-env.sh

• vim spark-env.sh

#配置java的环境变量
export JAVA_HOME=/opt/install/jdk1.8.0_141

#配置history日志信息
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=10 -Dspark.history.fs.logDirectory=hdfs://node01:8020/spark_log"

#配置zk相关信息
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER  -Dspark.deploy.zookeeper.url=node01:2181,node02:2181,node03:2181  -Dspark.deploy.zookeeper.dir=/spark"

4、修改slaves配置文件  

• cp slaves.template slaves

• vim slaves

node01
node02
node03 

 5、修改spark-defaults.conf配置选项

cd /kkb/install/spark-2.3.3-bin-hadoop2.7/conf/
cp spark-defaults.conf.template spark-defaults.conf

vim spark-defaults.conf

spark.eventLog.enabled true
spark.eventLog.compress true
spark.eventLog.dir  hdfs://node01:8020/spark_log

如果spark 运行过程中， 报lzo错误，将一下两项添加进来

spark.driver.extraClassPath   /kkb/install/hadoop-2.6.0-cdh5.14.2/share/hadoop/comm
on/hadoop-lzo-0.4.20.jar
spark.executor.extraClassPath /kkb/install/hadoop-2.6.0-cdh5.14.2/share/hadoop/comm
on/hadoop-lzo-0.4.20.jar

6、分发安装目录到其他机器

• node01执行以下命令分发安装包

 cd /kkb/install/
scp -r spark-2.3.3-bin-hadoop2.7/ node02:$PWD
scp -r spark-2.3.3-bin-hadoop2.7/ node03:$PWD

7、hdfs创建文件

hdfs  dfs -mkdir -p /spark_log

8、启动

node01上面

cd /kkb/install/spark-2.3.3-bin-hadoop2.7/sbin
 

./start-all.sh

node02上面

cd /kkb/install/spark-2.3.3-bin-hadoop2.7/sbin

 ./start-master.sh

访问master主节点web管理界面 :http://node01:8080/

  访问备份master节点 :http://node02:8080/

在node01上面启动 historyserver

cd /kkb/install/spark-2.3.3-bin-hadoop2.7/sbin

./start-history-server.sh

 访问historyserver历史任务访问界面 http://node01:4000/

 参考博客一

参考博客二

参考博客三