在apache hadoop2.6 上部署hive 并将hive数据源存储于Mysql

Posted 2020-06-18 zhangl

集成hive 的前提是apache hadoop 集群能够正常启动。

hadoop 版本 apach2.6.0  hive 版本：1.2.1

1、安装mysql 并赋予权限：

 1.1：创建hive 用户和密码：

Create user  ‘hive’ identified by ‘123456’

1.2：创建数据库：create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

1.3：授权允许远程登陆：grant all privileges on *.* to ‘user‘@‘hostname‘ identified by ‘passwd‘ with grant option;

flush privileges;

1.4 将mysql 驱动复制到 hive/lib 目录下

　　mysql-connector-java-5.1.38-bin.jar

 

2、hive 配置文件在conf 目录下：hive-env.sh

cp hive-env.sh.template hive-env.sh

最后添加如下配置：

export JAVA_HOME=/usr/java/jdk1.7.0_79
export HIVE_HOME=/home/hadoop/hive-1.2.1-bin
export HADOOP_HOME=/home/hadoop/hadoop-2.6.0

3、hive-site.xml

3.1hdfs 上的文件需要事先创建，如果出现权限问题修改对应的权限即可：hadoop dfs -chmod -R 777 hdfs://node17:9000/hive/scratchdir

3.2需要将下述jar文件复制到hive/lib目录下
home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar
file:///home/hadoop/hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/zookeeper-3.4.6.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar

 

<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://node17:9000/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://node17:9000/hive/scratchdir</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive-1.2.1-bin/logs</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://node15:3306/hive?createDatabaseIfNotExist=true</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>

<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>

<property>
<name>hive.aux.jars.path</name>
<value>file:///home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar ,file:///home/hadoop/hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/zookeeper-3.4.6.jar,file:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar</value>
</property>

<property>
<name>hive.metastore.uris</name>
<value>thrift://node17:9083</value>       
</property>

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/hadoop/hive-1.2.1-bin/tmp</value>
  </property>

</configuration>

出现问题：

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

 

解决办法：

启动metastore service服务即可。

./hive --service metastore

 

解决办法：

删除 yarn 下的 jline-0.9.94.jar jar 包即可。

./hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar