hadoop+spark详细的部署过程
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了hadoop+spark详细的部署过程相关的知识，希望对你有一定的参考价值。
准备软件包
1、hadoop-2.7.2.tar.gz
http://mirror.bit.edu.cn/apache/hadoop/common/
2、scala-2.10.4.tgz
http://www.scala-lang.org/download/2.10.4.html
3、spark-2.0.0-bin-hadoop2.7.tar
http://spark.apache.org/downloads.html
一、环境准备
3台Centos7的虚拟机：
172.16.92.115 spark01 #namenode
172.16.92.117 spark02 #datanode
172.16.92.80 spark03 #datanode
1.1、防火墙、selinux关闭
1.2、配置hosts设置ssh免密码登录，使三台机能够互访
[[email protected] ~]# vi /etc/hosts 172.16.92.115 spark01 #namenode 172.16.92.117 spark02 #datanode 172.16.92.80 spark03 #datanode [[email protected] ~]# ssh-keygen [[email protected] ~]# ssh-copy-id spark01 [[email protected] ~]# ssh-copy-id spark02 [[email protected] ~]# ssh-copy-id spark03 [[email protected] ~]# scp /etc/hosts spark02:/etc/ [[email protected] ~]# scp /etc/hosts spark03:/etc/ [[email protected] ~]# ssh-keygen [[email protected] ~]# ssh-copy-id spark01 [[email protected] ~]# ssh-copy-id spark02 [[email protected] ~]# ssh-copy-id spark03 [[email protected] ~]# ssh-keygen [[email protected] ~]# ssh-copy-id spark01 [[email protected] ~]# ssh-copy-id spark02 [[email protected] ~]# ssh-copy-id spark03
1.3、安装JDK1.8
这里已安装
[[email protected] ~]# yum list|grep jdk Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast jdk1.8.0_91.x86_64 2000:1.8.0_91-fcs installed
1.4、配置环境变量
[[email protected] ~]# export JAVA_HOME=/usr/java/jdk1.8.0_91 [[email protected] ~]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar: $JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib:$CLASSPATH [r[email protected] ~]# export PATH=$JAVA_HOME/bin:$PATH
二、开始安装hadoop
2.1、解压并创建hadoop文件目录
[[email protected] ~]#tar -xzvf hadoop-2.7.2.tar.gz -C /opt [[email protected] ~]#cd /opt/hadoop-2.7.2 [[email protected] hadoop-2.7.2]#mkdir -p hdfs/user [[email protected] hadoop-2.7.2]#mkdir -p hdfs/data [[email protected] hadoop-2.7.2]#mkdir tmp/
2.2、修改配置文件
主要是修改以下文件
etc/hadoop/slaves
etc/hadoop/core-site.xml
etc/hadoop/hdfs-site.xml
etc/hadoop/mapred-site.xml
etc/hadoop/yarn-site.xml
2.2.1、修改slaves文件（添加数据节点）
[[email protected] hadoop-2.7.2]#vi etc/hadoop/slaves spark02 spark03
2.2.2、修改core-site.xml文件（增加hadoop核心配置，hdfs文件端口是9000）
[[email protected] hadoop-2.7.2]#vi etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://spark01:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> #默认64MB，这里改为128MB </property> <property> <name>hadoop.tmp.dir</name> <value>file:/opt/hadoop-2.7.2/tmp</value> <description>Abasefor other temporary directories.</description> </property> <property> <name>hadoop.proxyuser.spark.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.spark.groups</name> <value>*</value> </property> </configuration>
2.2.3、
修改hdfs-site.xml 文件（增加hdfs配置信息、namenode、datanode端口和目录位置）
[[email protected] hadoop-2.7.2]#vi etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>spark01:9001</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/opt/hadoop-2.7.2/hdfs/user</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/opt/hadoop-2.7.2/hdfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> #复本个数==datanode个数 </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>
2.2.4、修改mapred-site.xml 文件（增加mapreduce配置、使用yarn框架、jobhistory使用地址以及web地址）
[[email protected] hadoop-2.7.2]#vi etc/hadoop/mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>spark01:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>spark01:19888</value> </property> </configuration>
[[email protected] hadoop-2.7.2]# vi etc/hadoop/yarn-site.xml <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux- services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>spark01:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>spark01:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>spark01:8035</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>spark01:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>spark01:8088</value> </property> </configuration>
2.2.5、修改hadoop_env.sh配置文件的JAVA_HOME
[[email protected] hadoop-2.7.2]# vi etc/hadoop/hadoop-env.sh # The java implementation to use. export JAVA_HOME=/usr/java/jdk1.8.0_91
2.3、将配置好的hadoop文件copy到其他的所有的slave机器
[[email protected] ~]# scp -r /opt/hadoop-2.7.2/ spark02:/opt/ [[email protected] ~]# scp -r /opt/hadoop-2.7.2/ spark03:/opt/
2.4、配置hadoop环境变量
[[email protected] ~]# export HADOOP_HOME=/opt/hadoop-2.7.2 [[email protected] ~]# export PATH=$HADOOP_HOME/bin:$PATH [[email protected] ~]# export HADOOP_HOME=/opt/hadoop-2.7.2 [[email protected] ~]# export PATH=$HADOOP_HOME/bin:$PATH [[email protected] ~]# export HADOOP_HOME=/opt/hadoop-2.7.2 [[email protected] ~]# export PATH=$HADOOP_HOME/bin:$PATH
注意：这里配置的是本地的环境变量，在hadoop中不一定会生效。
hadoop的环境变量在：etc/hadoop/hadoop-env.sh
spark的环境变量也一样，当使用spark-submit提交任务到集群，
如果要调用库等，需要在spark中配置环境变量，就是添加环境变量到spark-env.sh文件。
2.5、格式化namenode节点
[[email protected] hadoop-2.7.2]# ./bin/hdfs namenode -format 17/07/18 09:52:36 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = spark01/172.16.92.115 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.7.2 。。。。。。 /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at spark01/172.16.92.115 ************************************************************/
2.6、启动hadoop文件系统
[[email protected] hadoop-2.7.2]# ./sbin/start-dfs.sh Starting namenodes on [spark01] spark01: Warning: Permanently added the ECDSA host key for IP address ‘172.16.92.115‘ to the list of known hosts. spark01: starting namenode, logging to /opt/hadoop-2.7.2/logs/hadoop-root-namenode-spark01.out spark03: starting datanode, logging to /opt/hadoop-2.7.2/logs/hadoop-root-datanode-spark03.out spark02: starting datanode, logging to /opt/hadoop-2.7.2/logs/hadoop-root-datanode-spark02.out Starting secondary namenodes [spark01] spark01: starting secondarynamenode, logging to /opt/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-spark01.out
2.7、查看进程jps
[[email protected] hadoop-2.7.2]# jps 25954 Jps 25749 SecondaryNameNode 25533 NameNode
2.8、其他命令
2.8.1、关闭文件系统
[[email protected] ~]#./sbin/stop-dfs.sh
2.8.2、开启或者关闭hadoop所有服务
[[email protected] ~]#./sbin/start-all.sh [[email protected] ~]#./sbin/stop-all.sh
用浏览器输入以下地址查看hadoop集群
http://spark01:50070/
三、安装Spark2.0.0
已经在3个节点中安装hadoop集群
3.1.1、安装scala
[[email protected] ~]# tar -zxvf scala-2.10.4.tgz -C /usr/local
3.1.2、配置scala环境变量（spark是用scala开发的）
[[email protected] ~]# vi /etc/profile export SCALA_HOME=/usr/local/scala-2.10.4 export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${SCALA_HOME}/bin:$PATH [[email protected] ~]# source /etc/profile
3.1.3、测试scala运行环境
[[email protected] ~]# scala Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91). Type in expressions to have them evaluated. Type :help for more information. scala> 9*3 res0: Int = 27 scala> exit
3.2、安装spark
3.2.1、解压spark包
[[email protected] ~]# tar -zxvf spark-2.0.0-bin-hadoop2.7.tar -C /usr/local/
3.2.2、配置本地环境变量
[[email protected] ~]# vi /etc/profile export SPARK_HOME=/usr/local/spark-2.0.0-bin-hadoop2.7/ export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH [[email protected] ~]# source /etc/profile
3.2.3、配置Spark环境变量
[[email protected] ~]# cd $SPARK_HOME/conf [[email protected] conf]# cp spark-env.sh.template spark-env.sh [[email protected] conf]# vi spark-env.sh export JAVA_HOME=/usr/java/jdk1.8.0_91 export SCALA_HOME=/usr/local/scala-2.10.4 export SPARK_MASTER_IP=spark01 #Spark集群的主节点的主机名 export SPARK_WORKER_CORES=1 #Spark集群工作节点的cpu核数 export SPARK_WORKER_MEMORY=512M #Spark集群工作节点的可用内存 export HADOOP_CONF_DIR=/opt/hadoop-2.7.2/etc/hadoop
3.2.4、配置工作节点
[[email protected] conf]# cp slaves.template slaves [[email protected] conf]# vi slaves spark01 spark02 spark03
3.2.5、将配置好的hadoop文件copy到其他的所有的slave机器
[[email protected] ~]# scp -r /usr/local/spark-2.0.0-bin-hadoop2.7/ spark02:/usr/local/ [[email protected] ~]# scp -r /usr/local/spark-2.0.0-bin-hadoop2.7/ spark03:/usr/local/
3.2.6、启动Spark
[[email protected] ~]# cd $SPARK_HOME/sbin [[email protected] sbin]# ./start-all.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.0.0-bin-hadoop2.7//logs/spark-root-org.apache.spark.deploy.master.Master-1-spark01.out localhost: Warning: Permanently added ‘localhost‘ (ECDSA) to the list of known hosts. spark02: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-spark02.out spark01: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-spark01.out localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-spark01.out spark03: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-spark03.out
3.2.7、jps查看进程，多了一个Master和Worker进程
[[email protected] sbin]# jps 25749 SecondaryNameNode 13413 Worker 13719 Jps 13322 Master 13434 Worker 25533 NameNode
查看spark集群
http://spark01:8080/
本文出自 “运维笔记” 博客，请务必保留此出处http://quliren.blog.51cto.com/9849266/1948609
以上是关于hadoop+spark详细的部署过程的主要内容，如果未能解决你的问题，请参考以下文章