hive on spark配置

Posted 2020-10-16 ChavinKing

1、安装java、maven、scala、hadoop、mysql、hive

略

2、编译spark

./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-2.6,parquet-provided"

3、安装spark

tar -zxvf spark-1.6.0-bin-hadoop2-without-hive.tgz -C /opt/cdh5/

4、配置spark

：spark-env.sh

export JAVA_HOME=/opt/service/jdk1.8.0_151
export SCALA_HOME=/opt/service/scala-2.10.5
export HADOOP_HOME=/opt/cdh5/hadoop-2.6.0-cdh5.10.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HIVE_CONF_DIR=/opt/cdh5/hive-2.1.0/conf
export SPARK_WORKER_CORES=4
export SPARK_WORKER_INSTANCES=4
export SPARK_WORKER_MEMORY=1g
export SPARK_DRIVER_MEMORY=1g
export SPARK_MASTER_IP=chavin.king
export SPARK_LIBRARY_PATH=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib
export SPARK_MASTER_WEBUI_PORT=8080
export SPARK_WORKER_WEBUI_PORT=8081
export SPARK_WORKER_DIR=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/work
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_PORT=7078
export SPARK_LOG_DIR=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/log

：spark-default.xml

#spark.master                     yarn
spark.master                     spark://chavin.king:7077
spark.home                       /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://chavin.king:8020/spark-log
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.executor.memory            1g
spark.driver.memory              1g
spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

：slaves

chavin.king

5、配置yarn

：yarn-site.xml

<property>
   <name>yarn.resourcemanager.scheduler.class</name>
   <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

6、配置hive

<property>
   <name>hive.execution.engine</name>
   <value>spark</value>
</property>

<property>
   <name>hive.enable.spark.execution.engine</name>
   <value>true</value>
</property>

<property>
   <name>spark.home</name>
   <value>/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive</value>
</property>
<property>
   <name>spark.master</name>
   <value>spark://chavin.king:7077</value>
</property>
<property>
   <name>spark.enentLog.enabled</name>
   <value>true</value>
</property>
<property>
   <name>spark.enentLog.dir</name>
   <value>hdfs://chavin.king:8020/spark-log</value>
</property>
<property>
   <name>spark.serializer</name>
   <value>org.apache.spark.serializer.KryoSerializer</value>
</property>
<property>
   <name>spark.executor.memeory</name>
   <value>1g</value>
</property>
<property>
   <name>spark.driver.memeory</name>
   <value>1g</value>
</property>
<property>
   <name>spark.executor.extraJavaOptions</name>
   <value>-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</value>
</property>

7、为hive添加spark jar包：

cp /opt/software/spark-1.6.0/core/target/spark-core_2.10-1.6.0.jar /opt/cdh5/hive-2.1.0/lib/
ln -s /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar /opt/cdh5/hive-2.1.0/lib/

bin/hdfs dfs -put /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar hdfs://chavin.king:8020/spark-assembly-1.6.0-hadoop2.6.0.jar

在hive-site.xml中添加：

<property>
   <name>spark.yarn.jar</name>
   <value>hdfs://chavin.king:8020/spark-assembly-1.6.0-hadoop2.6.0.jar</value>
</property>

8、验证hive on spark是否成功配置

$ bin/hive
which: no hbase in (/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/bin:/opt/service/maven-3.3.3/bin:/opt/service/scala-2.10.5/bin:/opt/service/jdk1.8.0_151/bin:/opt/service/jdk1.8.0_151/jre/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cdh5/hive-2.1.0/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in file:/opt/cdh5/hive-2.1.0/conf/hive-log4j2.properties Async: true
hive (default)> show tables ;
OK
tab_name
t1
Time taken: 0.966 seconds, Fetched: 1 row(s)
hive (default)> select count(*) from t1;
Query ID = hadoop_20171204024017_cda99c42-21eb-480f-9d2a-e0dbb18a9b63
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
   set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
   set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
   set mapreduce.job.reduces=<number>
Starting Spark Job = e8b4ccc6-2dfa-43b9-99cc-7a066e2c0a0f

Query Hive on Spark job[0] stages:
0
1

Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
2017-12-04 02:40:32,861    Stage-0_0: 0/1    Stage-1_0: 0/1   
... ...
2017-12-04 02:44:11,388    Stage-0_0: 1/1 Finished    Stage-1_0: 0(+1)/1   
2017-12-04 02:44:50,826    Stage-0_0: 1/1 Finished    Stage-1_0: 1/1 Finished   
Status: Finished successfully in 268.11 seconds
OK
c0
3
Time taken: 338.493 seconds, Fetched: 1 row(s)
hive (default)> exit;