Spark集群搭建与测试(DT大数据梦工厂)

Posted 2020-06-13

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Spark集群搭建与测试(DT大数据梦工厂)相关的知识，希望对你有一定的参考价值。

Spark流行的两种文件存储方式：1、Hadoop的HDFS；2、H3云存储

tux yarn +HDFS是未来3、5年的趋势

看你用的是bash，可能ubuntu里的bash不会自动source /etc/profile，所以你将那条export命令放在~/.bashrc里试试

计算的集群和数据存储的集群不在同一个集群上的话，性能不高不可接受，tux yarn解决了这个问题，它用JAVA写的

ubuntu

设置root登录见http://jingyan.baidu.com/article/148a1921a06bcb4d71c3b1af.html

1、ssh的安装：

sudo apt-get install ssh

sudo apt-get install rsync 以后集群用到

2、安装JAVA

sudo apt-get install rpm 将rpm的安装的东西先安装好

3、ssh生成key

ssh-keygen -t dsa -P ‘‘ -f ~/.ssh/id_dsa //生成key到~/.ssh/id_dsa中

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys //追加到key中

然后可以

[email protected]:~# ssh localhost

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-30-generic x86_64)

* Documentation: https://help.ubuntu.com/

429 packages can be updated.

220 updates are security updates.

Last login: Mon Feb 1 23:52:43 2016 from localhost

4、vim /etc/hostname

//改本机的名字

vim /etc/hosts

//改域名

5、安装Hadoop

cd hadoop/etc/hadoop下面的各个文件

文件：core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://Master:9000</value> //默认文件系统的方式

</property>

<name>hadoop.tmp.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/tmp</value> //放临时数据，尤其eclipse开发时候

</property>

<name>hadoop.native.lib</name> //初学的时候暂时不用关心

<description>Should native hadoop libraries,if present,be used</description>

</property>

</configuration>

文件：hdfs-site.xml

<name>dfs.replication</name>

<value>2</value> //存副本的数量

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>Master:50090</value> //Master、Slave结构，做镜像用的

<description>The secondary namenode http server address and port</description>

</property>

<name>dfs.namenode.name.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/dfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/usr/local/hadoop/hadoop-2.6.0/dfs/data</value>

</property>

<name>dfs.namenode.checkpoint.dir</name>

<value>file:///usr/local/hadoop/hadoop-2.6.0/dfs/namesecondary</value>

<description>Determines where on the local filesystem the DFSsecondary name node should store th temporary images to merge.If this is acomma-delimited list of directories then the image is replicated in all of the irectories foe redundancy.</description>

</property>

</configuration>

计算模型：

文件：mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>// 【指定运行mapreduce的环境是yarn，与hadoop1截然不同的地方】

</property>

</configuration>

文件：yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>Master</value> //自定ResourceManager的地址，还是单点，这是隐患

</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

</configuration>

文件：hadoop-env.sh

# The java implementation to use.

export JAVA_HOME=/usr/java/jdk1.8.0_71

6、配置Hadoop环境变量

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"

export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

. /etc/profile

7、看slaves

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# vi slaves

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# pwd

/usr/local/hadoop-2.6.0/etc/hadoop

老师的电脑：

7、格式化系统

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# scp masters [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop

masters 100% 7 0.0KB/s 00:00

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# scp masters [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop

masters 100% 7 0.0KB/s 00:00

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# scp slaves [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop

slaves 100% 23 0.0KB/s 00:00

[email protected]:/usr/local/hadoop-2.6.0/etc/hadoop# scp slaves [email protected]:/usr/local/hadoop-2.6.0/etc/hadoop

slaves 100% 23 0.0KB/s 00:00

hdfs namenode -format

8、启动HDFS系统

[email protected]:/usr/local/hadoop-2.6.0/sbin# pwd

/usr/local/hadoop-2.6.0/sbin

[email protected]:/usr/local/hadoop-2.6.0/sbin# ./start-dfs.sh

如果启动出错，改动hosts之后怎么样copy

[email protected]:~# jps //查看进程的

8290 NameNode

8723 Jps

8403 DataNode

8590 SecondaryNameNode

http://Master:50070/dfshealth.html 可以看dfs文件状态

9、启动资源管理框架

cd /usr/local/hadoop-2.6.0/sbin

./start-yarn.sh

[email protected]:/usr/local/hadoop-2.6.0/sbin# jps

8960 NodeManager

8290 NameNode

8403 DataNode

9256 Jps

8843 ResourceManager

8590 SecondaryNameNode

http://master:8088/cluster 可以看

//////////////Spark角度讲，Hadoop安装到这个程度差不多了///////////////

10、安装spark-1.6.0-bin-hadoop2.6.tgz

tar -zxvf spark-1.6.0-bin-hadoop2.6.tgz

解压到/usr/local

cp出来spark-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_71

export SCALA_HOME=/usr/local/scala-2.10.4

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop //运行在yarn模式下必须配置

export SPARK_MASTER_IP=Master //Saprk运行的主ip

export SPARK_WORKER_MEMORY=2g //具体机器

export SPARK_EXCUTOR_MEMORY=2g //具体计算

export SPARK_DRIVER_MEMORY=2g

export SPARK_WORKER_CORES=8 //线程池并发数

cp出来slaves

配置系统环境变量

export SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6

export PATH=${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH

scp ~/.bashrc [email protected]:~/.bashrc

改完后再同步到其它机器，用上面的方式

cp spark-defaults.conf出来

# Example:

# spark.master spark://master:7077

# spark.eventLog.enabled true //程序运行的信息

# spark.eventLog.dir hdfs://namenode:8021/directory

# spark.serializer org.apache.spark.serializer.KryoSerializer

# spark.driver.memory 5g

# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

spark.eventLog.enabled true

spark.eventLog.dir hdfs://Master:9000/historyserverforSpark

spark.yarn.historyServer.address Master:18080

spark.history.fs.logDorectory hdfs://Master:9000/historyserverforSpark

#spark.default.paralielism 100

然后是拷贝到各其它机器，所有机器的路径都一样的

scp -r ./spark-1.6.0-bin-hadoop2.6/ [email protected]:/usr/local/spark

到spark的sbin

第一次跑的时候

hadoop dfs -rmr /historyserverforSpark

hadoop dfs -mkdir /historyserverforSpark

然后这里就有了：

启动Spark

./start-all.sh

看web控制台

http://master:8080/

再启动

./start-history-server.sh

cd ../bin

http://spark.apache.org/docs/1.6.0/submitting-applications.html

官网例子

# Run on a Spark standalone cluster in client deploy mode ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --executor-memory 20G \ --total-executor-cores 100 \ /path/to/examples.jar \ 1000 //并行运行线程数量

改进运行

./spark-submit --class org.apache.spark.examples.SparkPi --master spark://Master:7077 ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100

此时在管理控制台也可以看到

老师机器，机器太好，一会就跑完：

我的机器，机器太烂：

貌似不是集群跑不起来

晚上加了集群：

改成本地试试

# Run application locally on 8 cores ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master local[8] \ /path/to/examples.jar \

100

./spark-submit --class org.apache.spark.examples.SparkPi --master local[8] ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100

真跑出来了，看来要好好搞下集群那个了

程序启动时候分配资源叫粗粒度Coarse Grained

程序计算时分配资源，用完就回收叫细粒度

spark-shell演示程序

./spark-shell --master spark://Master:7077

./spark-shell --master local[8]

sc.textFile("/library/wordcount/input/Data").flatMap(_.split(" ")).map(word=>(word,1)).reduceByKey(_+_).map(pair=>(pair._2,pair._1)).sortByKey(false).map(pair=>(pair._1,pair._2)).saveAsTextFile("/library/wordcount/output/dt_spark_clicked1")

//把key、value互换位置然后升序排列，然后再换回来，就可以根据value进行排序了

sc.textFile("/library/wordcount/input/Data").flatMap(_.split(" ")).map(word=>(word,1)).reduceByKey(_+_).map(pair=>(pair._2,pair._1)).sortByKey(false,1).map(pair=>(pair._1,pair._2)).saveAsTextFile("/library/wordcount/output/dt_spark_clicked2") //变成1个文件

最后，hadoop自带的wordcount

hadoop jar hadoop-examples-1.2.1.jar wordcount input putput

作业：

自己构件Spark集群或者本地模式，并且截图放在模式

王家林老师名片：

中国Spark第一人

新浪微博：http://weibo.com/ilovepains

微信公众号：DT_Spark

博客：http://blog.sina.com.cn/ilovepains

手机：18610086859

QQ：1740415547

邮箱：[email protected]