hive的安装和使用

Posted 2022-09-19 linyouyi

基础依赖环境：

1, jdk 已装
2, hadoop 2.x 已装
3, hive 2.3.6
4, mysql 
5, mysql-connector-jar

1.下载

[linyouyi@hadoop01 software]$ wget https://mirrors.aliyun.com/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz--2019-09-01 00:08:04--  https://mirrors.aliyun.com/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz
Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 119.147.158.241, 119.147.158.240, 183.2.199.237, ...
Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|119.147.158.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 232225538 (221M) [application/gzip]
Saving to: ‘apache-hive-2.3.6-bin.tar.gz’

100%[=======================================================================>] 232,225,538 5.65MB/s   in 38s    

2019-09-01 00:08:43 (5.80 MB/s) - ‘apache-hive-2.3.6-bin.tar.gz’ saved [232225538/232225538]
[linyouyi@hadoop01 software]$ ll
total 1028752
-rw-rw-r-- 1 linyouyi linyouyi 232225538 Aug 23 02:53 apache-hive-2.3.6-bin.tar.gz
-rw-rw-r-- 1 linyouyi linyouyi 312465430 Apr 30 06:17 apache-storm-2.0.0.tar.gz
-rw-r--r-- 1 linyouyi linyouyi 218720521 Aug  3 17:56 hadoop-2.7.7.tar.gz
-rw-rw-r-- 1 linyouyi linyouyi 132569269 Mar 18 14:28 hbase-2.0.5-bin.tar.gz
-rw-rw-r-- 1 linyouyi linyouyi  63999924 Mar 23 08:57 kafka_2.11-2.2.0.tgz
-rw-r--r-- 1 linyouyi linyouyi  54701720 Aug  3 17:47 server-jre-8u144-linux-x64.tar.gz
-rw-r--r-- 1 linyouyi linyouyi  37676320 Aug  8 09:36 zookeeper-3.4.14.tar.gz

2.解压拷贝

[linyouyi@hadoop01 software]$ tar -zxvf apache-hive-2.3.6-bin.tar.gz -C /hadoop/module/
[linyouyi@hadoop01 software]$ ll /hadoop/module/
total 28
drwxrwxr-x 10 linyouyi linyouyi 4096 Sep  1 00:10 apache-hive-2.3.6-bin
drwxrwxr-x 18 linyouyi linyouyi 4096 Aug 12 21:24 apache-storm-2.0.0
drwxr-xr-x 12 linyouyi linyouyi 4096 Aug  9 22:51 hadoop-2.7.7
drwxrwxr-x  7 linyouyi linyouyi 4096 Aug 11 12:10 hbase-2.0.5
drwxr-xr-x  7 linyouyi linyouyi 4096 Jul 22  2017 jdk1.8.0_144
drwxr-xr-x  7 linyouyi linyouyi 4096 Aug 17 15:19 kafka_2.11-2.2.0
drwxr-xr-x 15 linyouyi linyouyi 4096 Aug  8 11:03 zookeeper-3.4.14

安装mysql用于存放元数据,把mysql_connector拷贝到lib目录,修改配置文件

[linyouyi@hadoop01 apache-hive-2.3.6-bin]# unzip mysql_connector_java8.0.13.zip 
[linyouyi@hadoop01 apache-hive-2.3.6-bin]# cp mysql-connector-java-8.0.13/mysql-connector-java-8.0.13.jar lib
[linyouyi@hadoop01 apache-hive-2.3.6-bin]# ls lib/mysql-*
lib/mysql-connector-java-8.0.13.jar  lib/mysql-metadata-storage-0.9.2.jar

//修改hive-site.xml
[linyouyi@hadoop01 apache-hive-2.3.6-bin]$ cp conf/hive-default.xml.template conf/hive-site.xml
[linyouyi@hadoop01 apache-hive-2.3.6-bin]$ vim conf/hive-site.xml 
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <!--<value>jdbc:derby:;databaseName=metastore_db;create=true</value>-->
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseInfoNotExist=true</value>

<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <!--<value>org.apache.derby.jdbc.EmbeddedDriver</value>-->
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>linyouyi</value>
    <description>Username to use against metastore database</description>
  </property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>linyouyi</value>
    <description>password to use against metastore database</description>
  </property>

//修改tmp dir
修改将含有"system:java.io.tmpdir"的配置项的值修改为如下地址
/tmp/hive

创建数据库,创建用户赋予权限

MariaDB [(none)]> create database hive;
MariaDB [(none)]> grant all privileges on *.* to linyouyi@‘localhost‘ identified by ‘linyouyi‘;
MariaDB [(none)]> flush privileges;

初始化hive(初始化metadata)

[linyouyi@hadoop01 apache-hive-2.3.6-bin]$ bin/schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/module/apache-hive-2.3.6-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/module/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:     jdbc:mysql://localhost:3306/hive?createDatabaseInfoNotExist=true
Metastore Connection Driver :     com.mysql.jdbc.Driver
Metastore connection User:     linyouyi
Loading class `com.mysql.jdbc.Driver‘. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver‘. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

3.启动hive

[linyouyi@hadoop01 apache-hive-2.3.6-bin]$ bin/hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/module/apache-hive-2.3.6-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/module/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/hadoop/module/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
Loading class `com.mysql.jdbc.Driver‘. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver‘. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
OK
default
Time taken: 3.063 seconds, Fetched: 1 row(s)

如果出现 which: no hbase in 是因为没有配置hbase环境变量

[root@hadoop01 apache-hive-2.3.6-bin]# vim /etc/profile
#HBASE_HOME
export HBASE_HOME=/hadoop/module/hbase-2.0.5
export PATH=$PATH:$HBASE_HOME/bin
[root@hadoop01 apache-hive-2.3.6-bin]# vim /etc/profile
[root@hadoop01 apache-hive-2.3.6-bin]# exit
//退出用户重新登陆,让环境变量生效
[linyouyi@hadoop01 apache-hive-2.3.6-bin]$ exit


//再次登陆执行,就没有刚才的错误提示了

 

4.使用

hive> create database lin;
OK
Time taken: 0.088 seconds
hive> show databases;
OK
default
lin
Time taken: 0.008 seconds, Fetched: 2 row(s)
hive> use lin;
OK
Time taken: 0.017 seconds
hive> create table t_lin(id int,name string,salary string);
OK
Time taken: 0.3 seconds
hive> show tables;
OK
t_lin
Time taken: 0.017 seconds, Fetched: 1 row(s)

 插入数据一般都很慢

hive> insert into table t_lin values(1,"youyi","10K");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = linyouyi_20190901170958_4cd00921-8614-4a8d-8d12-41843d92e21d
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there‘s no reduce operator
Starting Job = job_1565368243366_0001, Tracking URL = http://hadoop02:8088/proxy/application_1565368243366_0001/
Kill Command = /hadoop/module/hadoop-2.7.7/bin/hadoop job  -kill job_1565368243366_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-09-01 17:10:06,685 Stage-1 map = 0%,  reduce = 0%
2019-09-01 17:10:11,867 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.19 sec
MapReduce Total cumulative CPU time: 2 seconds 190 msec
Ended Job = job_1565368243366_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://mycluster/user/hive/warehouse/lin.db/t_lin/.hive-staging_hive_2019-09-01_17-09-58_657_115403678722774435-1/-ext-10000
Loading data to table lin.t_lin
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 2.19 sec   HDFS Read: 4317 HDFS Write: 77 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 190 msec
OK
Time taken: 14.633 seconds


hive> select * from t_lin;
OK
1    youyi    10K
Time taken: 0.103 seconds, Fetched: 1 row(s)