Hive 使用Tez引擎的配置

Posted 2021-03-02 yangxusun9

前提

Hive  需要是 2.0以上版本

Tez配置

下载好tar包后，将tar包上传到HDFS集群路径下

hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez

将Linux本机上的tar包解压

Hive配置

在hive/conf下新建tez-site.xml文件，并添加以下参数

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!--指明hdfs集群上的tez的tar包，使Hadoop可以自动分布式缓存该jar包-->
    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>
</property>
<!--tez是否可用Hadoop的jar包 -->

    <property>
         <name>tez.use.cluster.hadoop-libs</name>
         <value>true</value>
</property>
<!--以何种形式形成日志文件 -->

    <property>
         <name>tez.history.logging.service.class</name>        
         <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
    </property>
</configuration>

 

在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置

# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=/opt/module/hadoop-2.7.2

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/module/hive/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
    export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done

export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS

在hive-site.xml文件中添加如下配置，更改hive计算引擎

<property>
    <name>hive.execution.engine</name>
    <value>tez</value>
</property>

yarn配置

如果运行时报错

Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

修改yarn-site.xml，关掉虚拟内存检查

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>