hadoop+hive+spark搭建

Posted 寂夜云

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了hadoop+hive+spark搭建相关的知识,希望对你有一定的参考价值。

上传hive软件包到任意节点

一、安装hive软件 

  解压缩hive软件包到/usr/local/hadoop/目录下

     

  重命名hive文件夹

     

 

  在/etc/profile文件中添加环境变量

      export HIVE_HOME=/usr/local/hadoop/hive

      export PATH=$HIVE_HOME/bin:$PATH

  运行命令source /etc/profile

 

   使用mysql作为数据库时需要安装mysql

   在mysql中创建hive用户,数据库等

     create user \'hive\' identified by \'hive\';
     create database hive;
     grant all on hive.* to hive@\'%\'  identified by \'hive\';
     grant all on hive.* to hive@\'localhost\'  identified by \'hive\';
     flush privileges;

   退出mysql

   拷贝mysql-connector-java.jar到hive目录下lib/中

二、修改配置文件

    修改hive目录中conf/hive-default.xml.template文件为conf/hive-site.xml

    在conf目录中修改配置文件hive-site.xml

      (使用默认数据库)

     

      <property>
        <name>hive.exec.local.scratchdir</name>
        <value>/home/hive/iotmp</value>
        <description>Local scratch space for Hive jobs</description>
      </property>
      <property>
        <name>hive.downloaded.resources.dir</name>
        <value>/home/hive/iotmp</value>
        <description>Temporary local directory for added resources in the remote file system.</description>
      </property>
      <property>
        <name>hive.querylog.location</name>
        <value>/home/hive/iotmp</value>
        <description>Location of Hive run time structured log file</description>
      </property>

      <property>
        <name>hive.exec.local.scratchdir</name>
        <value>/home/hive/iotmp</value>
        <description>Local scratch space for Hive jobs</description>
      </property>

     (mysql数据库)
     
      <property> 
         <name>javax.jdo.option.ConnectionURL </name> 
         <value>jdbc:mysql://localhost:3306/hive </value> 
      </property> 
 
      <property> 
         <name>javax.jdo.option.ConnectionDriverName </name> 
         <value>com.mysql.jdbc.Driver </value> 
      </property>
 
      <property> 
         <name>javax.jdo.option.ConnectionPassword </name> 
         <value>hive </value> 
      </property>

      <property>
        <name>hive.exec.local.scratchdir</name>
        <value>/home/hive/iotmp</value>
        <description>Local scratch space for Hive jobs</description>
      </property>
      

      <property>
        <name>hive.downloaded.resources.dir</name>
        <value>/home/hive/iotmp</value>
        <description>Temporary local directory for added resources in the remote file system.</description>
      </property>
      <property>
        <name>hive.querylog.location</name>
        <value>/home/hive/iotmp</value>
        <description>Location of Hive run time structured log file</description>
      </property>

      <property>
        <name>hive.exec.local.scratchdir</name>
        <value>/home/hive/iotmp</value>
        <description>Local scratch space for Hive jobs</description>
      </property>

 
 

三、运行hive

  输入命令格式化数据库

    默认数据库  schematool -initSchema -dbType derby 

    mysql数据库 schematool -initSchema -dbType mysql 

 

  启动hive

      输入命令hive

      

 

hive安装完毕

  

 

以上是关于hadoop+hive+spark搭建的主要内容,如果未能解决你的问题,请参考以下文章

hadoop+hive+spark搭建

hadoop集群搭建(Hadoop 3.1.3 /Hive 3.1.2/Spark 3.0.0)

hadoop + spark+ hive 集群搭建(apache版本)

Mac下hadoop,hive, hbase,spark单机环境搭建

Hadoop+Hive+Mysql环境搭建

Mac上搭建Hadoop+Hive+Spark开发环境