Linux安装Spark

Posted 2020-11-23 始于2019

1、下载

http://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz

2、解压

tar -xvf spark-2.4.1-bin-hadoop2.7.tgz

3、配置环境变量SPARK_HOME

• vim /etc/profile 或者 vim ~/.bashrc

#Spark Environment
export SPARK_HOME=$USER_HOME/software/spark-2.4.1-bin-hadoop2.7

• source /etc/profile 或者 source ~/.bashrc

4、修改配置文件

• slave配置文件复制

cp $SPARK_HOME/conf/slaves.template $SPARK_HOME/conf/slaves

• vim $SPARK_HOME/conf/slaves

修改主机名，暂时默认为localhost，不修改

• spark-env配置文件修改

cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh

• vim $SPARK_HOME/conf/spark-env.sh

export USER_HOME=/home/hadoop
export JAVA_HOME=$USER_HOME/software/jdk1.8.0_201 
export SCALA_HOME=$USER_HOME/software/scala-2.12.8 
export SPARK_WORKER_MEMORY=1G 
export HADOOP_HOME=$USER_HOME/software/hadoop-2.7.3 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export SPARK_MASTER_IP=hadoop

5、Spark启停

$SPARK_HOME/sbin/start-all.sh
$SPARK_HOME/sbin/stop-all.sh

启动后可以通过访问如下网址查看SPARK的控制台

http://hadoop:8080/

6、进入spark shell控制台

$SPARK_HOME/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_   version 2.4.1
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_201)

启动成功后，可访问http://hadoop:4040/jobs/

7、Spark测试

• Hadoop命令

hadoop fs -mkdir -p /usr/hadoop 创建/usr/hadoop文件夹
hadoop fs -ls -R /          列举出根目录/下面的文件和文件夹
hadoop fs -mkdir -p /usr/data/input 创建/usr/hadoop/input文件夹
hadoop fs -put /home/hadoop/aaa.txt /usr/data/input 上传本地文件/home/hadoop/aaa.txt到/usr/data/input目录
hadoop fs -ls -R /

• Spark Shell测试

var test_file = sc.textFile("/usr/data/input/aaa.txt")
test_file.count()
scala> var test_file = sc.textFile("/usr/data/input/aaa.txt")
test_file: org.apache.spark.rdd.RDD[String] = /usr/data/input/aaa.txt MapPartitionsRDD[1] at textFile at <console>:24

scala> test_file.count()
[Stage 0:> (0 + 1) / 1    res0: Long = 5

8、Web界面查看

http://hadoop:8100/ 默认端口为50070
http://localhost:8100   默认端口为50070
http://localhost:8088/cluster
http://hadoop:8080/
http://hadoop:4040/jobs/

9、命令测试

$HADOOP_HOME/bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /bbb.txt /out