HBASE_HIVE整合
Posted 李杰然
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了HBASE_HIVE整合相关的知识,希望对你有一定的参考价值。
Hive与HBase的整合功能的实现是利用两者本身对外的API接口互相进行通信,相互通信主要是依靠hive_hbase-handler.jar工具类
首先保证版本一致
cd /home/hadoop/hive-1.1.0-cdh5.5.2/lib
查看版本是否一致hbase-server-1.0.0-cdh5.5.2.jar zookeeper-3.4.5-cdh5.5.2.jar hive-hbase-handler-1.1.0-cdh5.5.2.jar
如果不一致删除原有jar cp 原hbase和zookeeper下的jar包到hive的lib中
2.修改 hive/conf下hive-site.xml文件
[[email protected] hive-0.7.1-cdh3u5]$ mkdir tmp
[[email protected] hive-0.7.1-cdh3u5]$ mkdir logs
[[email protected] conf]$ vi hive-site.xml
以下内容 添加到文件底部 </configuration> 之上
<!--
<property>
<name>hive.exec.scratchdir</name>
<value>/home/hadoop/hive-1.1.0-cdh5.5.2/tmp/</value>
</property>
-->
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive-1.1.0-cdh5.5.2/logs/</value>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>
file:///home/hadoop/hive-1.1.0-cdh5.5.2/lib/hive-hbase-handler-1.1.0-cdh5.5.2.jar,
file:///home/hadoop/hive-1.1.0-cdh5.5.2/lib/hbase-server-1.0.0-cdh5.5.2.jar,
file:///home/hadoop/hive-1.1.0-cdh5.5.2/lib/zookeeper-3.4.5-cdh5.5.2.jar
</value>
</property>
3.拷贝hbase-server-1.0.0-cdh5.5.2.jar 到所有hadoop节点 lib下(包括master节点)
[[email protected] hbase-0.90.6-cdh3u5]$ cp hbase-server-1.0.0-cdh5.5.2.jar /home/hadoop/hadoop-2.6.0-cdh5.5.2/lib
[[email protected] hbase-0.90.6-cdh3u5]$ scp hbase-server-1.0.0-cdh5.5.2.jar [email protected]:/home/hadoop/hadoop-2.6.0-cdh5.5.2/lib
[[email protected] hbase-0.90.6-cdh3u5]$ scp [email protected]:/home/hadoop/hadoop-2.6.0-cdh5.5.2/lib
4.拷贝hbase/conf下的hbase-site.xml文件到所有hadoop节点(包括master)的hadoop/conf下
[[email protected] conf]$ cp hbase-site.xml /home/hadoop/hadoop-2.6.0-cdh5.5.2/conf/
[[email protected] conf]$ scp hbase-site.xml [email protected]:/home/hadoop/hadoop-2.6.0-cdh5.5.2/conf/
[[email protected] conf]$ scp hbase-site.xml [email protected]:/home/hadoop/hadoop-2.6.0-cdh5.5.2/conf/
5.启动hive
[[email protected] hive-0.7.1-cdh3u5]$ bin/hive -hiveconf hbase.zookeeper.quorum=h201,h202,h203
----------------------------------------------------------------
例子
1创建 hbase识别的表
hive>
CREATE TABLE hbase_table_1(key int, value string)
STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler‘
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "sq");
***(hbase.table.name 定义在hbase的table名称
hbase.columns.mapping 定义在hbase的列族)****
2.创建 hive表
hive> create table ha1(id int,name string)
row format delimited
fields terminated by ‘\t‘
stored as textfile;
[[email protected] ~]$ vi ha1.txt
11 zs
22 ls
33 ww
hive> load data local inpath ‘/home/hadoop/ha1.txt‘ into table ha1;
hive> insert into table hbase_table_1 select * from ha1;
~~hive> select * from hbase_table_1;
3.[[email protected] hbase-0.90.6-cdh3u5]$ bin/hbase shell
hbase(main):002:0> scan ‘sq‘
(能看到结果 说明hive把数据存到hbase中)
第二种方法直接修改 hive/conf下hive-site.xml文件
<!--
<property>
<name>hive.exec.scratchdir</name>
<value>/home/hadoop/hive-1.1.0-cdh5.5.2/tmp/</value>
</property>
-->
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive-1.1.0-cdh5.5.2/logs/</value>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>
file:///home/hadoop/hive-1.1.0-cdh5.5.2/lib/hive-hbase-handler-1.1.0-cdh5.5.2.jar,
file:///home/hadoop/hive-1.1.0-cdh5.5.2/lib/guava-14.0.1.jar,
file:///home/hadoop/hbase-1.0.0-cdh5.5.2/lib/hbase-common-1.0.0-cdh5.5.2.jar,
file:///home/hadoop/hbase-1.0.0-cdh5.5.2/lib/hbase-client-1.0.0-cdh5.5.2.jar,
file:///home/hadoop/hbase-1.0.0-cdh5.5.2/lib/hbase-server-1.0.0-cdh5.5.2.jar,
file:///home/hadoop/hbase-1.0.0-cdh5.5.2/lib/netty-all-4.0.23.Final.jar,
file:///home/hadoop/hbase-1.0.0-cdh5.5.2/lib/hbase-hadoop2-compat-1.0.0-cdh5.5.2.jar,
file:///home/hadoop/zookeeper-3.4.5-cdh5.5.2/zookeeper-3.4.5-cdh5.5.2.jar
</value>
</property>
然后重启hive即可。
以上是关于HBASE_HIVE整合的主要内容,如果未能解决你的问题,请参考以下文章
SSM框架整合—详细整合教程(Spring+SpringMVC+MyBatis)
SpringBoot系列八:SpringBoot整合消息服务(SpringBoot 整合 ActiveMQSpringBoot 整合 RabbitMQSpringBoot 整合 Kafka)
SpringBoot——整合SSM(主要整合MyBatis)