从0开始搭建基于Zookeeper的Spark集群

Posted 2020-06-15

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了从0开始搭建基于Zookeeper的Spark集群相关的知识，希望对你有一定的参考价值。

完全从0搭建Spark集群

备注：这个步骤，只适合用root来搭建，正式环境下应该要有权限类的东西后面另外再进行实验写教程

1、安装各个软件，设置环境变量（每种软件需自己单独下载）

export JAVA_HOME=/usr/java/jdk1.8.0_71

export JAVA_BIN=/usr/java/jdk1.8.0_71/bin

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME JAVA_BIN PATH CLASSPATH

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"

export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

export SCALA_HOME=/usr/local/scala-2.10.4

export PATH=${SCALA_HOME}/bin:$PATH

export SPARK_HOME=/usr/local/spark/spark-1.6.0-bin-hadoop2.6

export PATH=${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH

export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.6

2、ssh设置

ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa //生成key到~/.ssh/id_dsa中

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys //追加到key中

3、主机名和域名的设置

vi /etc/hostname 改成Master或者Worker1、2、3、4

vim /etc/hosts 改域名，各个系统ip对应的域名

4、Hadoop的配置

1）cd $HADOOP_HOME/etc/hadoop/ 下改动core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://Master:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/tmp</value>

</property>

<name>hadoop.native.lib</name>

<description>Should native hadoop libraries,if present,be used</description>

</property>

</configuration>

2）还是cd $HADOOP_HOME/etc/hadoop/ 下改动hdfs-site.xml

<name>dfs.replication</name>

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>Master:50090</value>

<description>The secondary namenode http server address and port</description>

</property>

<name>dfs.namenode.name.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/dfs/name</value>

</property>

<name>dfs.datanode.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/dfs/data</value>

</property>

<name>dfs.namenode.checkpoint.dir</name>

<value>file:///usr/local/hadoop/hadoop-2.6.0/dfs/namesecondary</value>

<description>Determines where on the local filesystem the DFSsecondary name node should store th temporary images to merge.If this is acomma-delimited list of directories then the image is replicated in all of the irectories foe redundancy.</description>

</property>

</configuration>

3）还是cd $HADOOP_HOME/etc/hadoop/ 下改动 mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

4）还是cd $HADOOP_HOME/etc/hadoop/ 下改动 yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>Master</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

5）还是cd $HADOOP_HOME/etc/hadoop/ 下改动 hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_71 对应的jdk目录

如果你想把你的Master也作为一个节点，也可以Master加入，但是机器不够多Driver如果是Master，Driver还有其它程序比如web查询等在跑，不建议把Master作为节点来跑。

=======一台机器配置到上面这个程度，然后开始复制机器，再继续下面的操作=============

6）先再看第3步里面改域名的，到各个机器上把主机名称和域名改好了

7）还是cd $HADOOP_HOME/etc/hadoop/ 下改动 slaves

看你有多少slaves，分别把那些机器的域名加入进来，比如

Worker1

Worker2

Worker3

然后分别复制给几台机器：

scp slaves [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop/slaves

scp slaves [email protected]/usr/local/hadoop-2.6.0/etc/hadoop/slaves

8）还是cd $HADOOP_HOME/etc/hadoop/ 下改动 Master，内容就是Master

在Master不做集群的情况下，需要把Master分别拷贝到各个机器上，其实应该拷贝，这样如果不启动集群，也能跑

如果Master是集群，ZooKeeper，在ZooKeeper进行配置

scp Master [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop/Master

scp Master [email protected]/usr/local/hadoop-2.6.0/etc/hadoop/Master

9）在Master上格式化系统

mkdir /usr/local/hadoop/hadoop-2.6.0/tmp 如果原来存在，就删除

hdfs namenode -format

10)启动dfs

cd $HADOOP_HOME/sbin

./start-dfs.sh

然后

http://Master:50070/dfshealth.html 可以看dfs文件状态

看不到，比如Configured Capacity只有0B，尝试每台机器防火墙关闭：

systemctl stop firewalld.service

systemctl disable firewalld.service

但是这个只适合于开发机，实际生产环境需要仔细看什么端口来确定。

***************如果只做Spark，到这里就足够了，做Hadoop另外说****************

5、Spark的配置

1）spark-env.sh

cd $SPARK_HOME/conf

cp出来spark-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_71

export SCALA_HOME=/usr/local/scala-2.10.4

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop

#export SPARK_CLASSPATH=$SPARK_CLASSPATH:$SPARK_HOME/lib/ojdbc-14.jar:$SPARK_HOME/lib/jieyi-tools-1.2.0.7.RELEASE.jar

#export SPARK_MASTER_IP=Master

export SPARK_WORKER_MEMORY=2g

export SPARK_EXCUTOR_MEMORY=2g

export SPARK_DRIVER_MEMORY=2g

export SPARK_WORKER_CORES=8

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=Master:2181,Worker1:2181,Worker2:2181 -Dspark.deploy.zookeeper.dir=/spark"

参数意思讲解：

export JAVA_HOME=/usr/java/jdk1.8.0_71

export SCALA_HOME=/usr/local/scala-2.10.4

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop //运行在yarn模式下必须配置

export SPARK_MASTER_IP=Master //Saprk运行的主ip

export SPARK_WORKER_MEMORY=2g //具体机器

export SPARK_EXCUTOR_MEMORY=2g //具体计算

export SPARK_DRIVER_MEMORY=2g

export SPARK_WORKER_CORES=8 //线程池并发数

其中export SPARK_MASTER_IP=Master是作为单机的时候，export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=Master:2181,Worker1:2181,Worker2:2181 -Dspark.deploy.zookeeper.dir=/spark"是作为集群的时候的配置

改完之后同步：

scp spark-env.sh [email protected]:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/spark-env.sh

2）slaves

cd $SPARK_HOME/conf

cp出来slaves

内容为为：

Worker1

Worker2

Worker3

改完之后同步：

scp slaves [email protected]:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/slaves

3）spark-defaults.conf

cd $SPARK_HOME/conf

cp spark-defaults.conf出来

spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.eventLog.enabled true

spark.eventLog.dir hdfs://Master:9000/historyserverforSpark1

spark.yarn.historyServer.address Master:18080

spark.history.fs.logDirectory hdfs://Master:9000/historyserverforSpark1

改完之后同步：

scp spark-defaults.conf [email protected]:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/spark-defaults.conf

或者上面三步一起：

cd $SPARK_HOME

scp -r ./spark-1.6.0-bin-hadoop2.6/ [email protected]:/usr/local/spark

4）创建历史目录（第一次安装必须做）

hadoop dfs -rmr /historyserverforSpark

hadoop dfs -mkdir /historyserverforSpark

然后这里就有了：

5）启动spark

cd $SPARK_HOME/sbin

./start-all.sh

看web控制台

master:8080/

6）启动历史信息的服务

cd $SPARK_HOME/sbin

./start-history-server.sh

7）实验下Pi的算法：

./spark-submit --class org.apache.spark.examples.SparkPi --master spark://Master:7077 ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100

开始神奇的Spark之旅吧！

***************如果Spark单机，到这里就足够了，开始补充Zookeeper做集群的东西****************

6、ZooKeeper安装集群的东西

1）先在第一台机器上解压zookeeper，目录按照开头的环境变量解压就可以

进入到zookeeper下，创建data和logs两个目录

[email protected]:/usr/local/zookeeper-3.4.6# mkdir data

[email protected]:/usr/local/zookeeper-3.4.6# mkdir logs

2）从zoo_sample.cfg中cp出zoo.cfg并设置

[email protected]:/usr/local/zookeeper-3.4.6/conf# cp zoo_sample.cfg zoo.cfg

[email protected]:/usr/local/zookeeper-3.4.6/conf# vi zoo.cfg

修改(做3台机器的集群)

dataDir=/usr/local/zookeeper-3.4.6/data

dataLogDir=/usr/local/zookeeper-3.4.6/logs

server.0=Master:2888:3888

server.1=Worker1:2888:3888

server.2=Worker2:2888:3888

3）在data下面为机器编号

[email protected]:/usr/local/zookeeper-3.4.6/conf# cd ../data/

为机器编号

[email protected]:/usr/local/zookeeper-3.4.6/data# echo 0>myid

[email protected]:/usr/local/zookeeper-3.4.6/data# echo 0>>myid

[email protected]:/usr/local/zookeeper-3.4.6/data# ls

myid

[email protected]:/usr/local/zookeeper-3.4.6/data# cat myid

[email protected]:/usr/local/zookeeper-3.4.6/data# vi myid 在这里在里面写一个0

[email protected]:/usr/local/zookeeper-3.4.6/data# cat myid

到这个时候一台机器已经配置好了

4）拷贝给其它两台机器同时更改myid

[email protected]:/usr/local# scp -r ./zookeeper-3.4.6 [email protected]:/usr/local

然后分别进去Worker1和Worker2更改myid为1和2

到这个时候3台机器的Zookeeper已经配置好了

5)下一步就是让Spark支持zookeeper下HA

到spark-env.sh中配置

[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# vi spark-env.sh

//整个集群的状态的维护和恢复都是通过zookeeper的，状态信息都是（下面的这段就是上面被注释的东西，要切单机和集群就靠这个了）

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=Master:2181,Worker1:2181,Worker2:2181 -Dspark.deploy.zookeeper.dir=/spark"

已经配置集群了，所以还要注释

#export SPARK_MASTER_IP=Master

[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# scp spark-env.sh [email protected]:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/spark-env.sh

spark-env.sh 100% 500 0.5KB/s 00:00

[email protected]:/usr/local/spark-1.6.0-bin-hadoop2.6/conf# scp spark-env.sh [email protected]:/usr/local/spark /spark-1.6.0-bin-hadoop2.6/conf/spark-env.sh

spark-env.sh 100% 500 0.5KB/s 00:00

到这个时候3台机器的Spark也已经配置好了，下面就是启动了

6）整体启动步骤

启动Hadoop hdfs

cd $HADOOP_HOME/sbin

./start-dfs.sh

三台Zookeeper的机器分别启动Zookeeper：

cd $ZOOKEEPER_HOME/bin

./zkServer.sh start

启动Spark

在Master启动：

cd $SPARK_HOME/sbin