Spark1.4.1 编译与安装

Posted 2020-09-30 mthoutai

1、下载

下载地址：

http://spark.apache.org/downloads.html

选择下载源代码

2、源代码编译

1）解压

tar -zxvf spark-1.4.1.tgz

2、编译

进入根文件夹下，採用make-distribution.sh进行编译。

cd spark-1.4.1

sudo ./make-distribution.sh --tgz --skip-java-test -Pyarn -Phadoop-2.2-Dhadoop.version=2.2.0 -Phive -Phive-thriftserver -DskipTests clean package

假设中间有报错，请又一次跑，多试几次，一般都能成功。

编译成功后，其安装文件在根文件夹下：

spark-1.4.1-bin-2.2.0.tgz

3、安装

省略，和之前版本号一样。就不写了。

4、报错问题

集群启动时问题：

1）问题1 : worek节点不能启动

localhost:starting org.apache.spark.deploy.worker.Worker, logging to/home/lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-is xxxx.out

localhost:failed to launch org.apache.spark.deploy.worker.Worker:

localhost:      at org.apache.spark.launcher.SparkClassCommandBuilder.buildCommand(SparkClassCommandBuilder.java:98)

localhost:      atorg.apache.spark.launcher.Main.main(Main.java:74)

localhost:full log in/home/lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-is xxxx.out

localhost:Connection to localhost closed.

原因是系统自带java问题

rpm -qa | grep java

gcc-java-4.4.7-4.el6.x86_64

java_cup-0.10k-5.el6.x86_64

java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

卸载

rpm -e --nodeps java_cup-0.10k-5.el6.x86_64

rpm -e --nodepsjava-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

2）问题2 :JAVA_HOME is not set

localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/lib/spark-1.4.1/sbin/../logs/spark-org.apache.spark.deploy.worker.Worker-1-is xxxx.out
localhost: failed to launch org.apache.spark.deploy.worker.Worker:
localhost:   JAVA_HOME is not set
localhost: full log in /lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-isxxxx.out
localhost: Connection to localhost closed.

找到出错的shell文件。增加export JAVA_HOME=... 就可以

spark-env.sh。增加export JAVA_HOME=... 就可以

启动成功后的界面：

转载请注明出处：

http://blog.csdn.net/sunbow0