Hive基础练习一

Posted 2020-11-22 youngchaolin

下面是hive基本练习，持续补充中。

简述Hive工作原理

hive是基于hadoop，可以管理hdfs上的数据的工具，它本质上是执行MapReduce程序，只是使用了类sql语句更加方便开发，hive驱动器会将类sql语句转换成MapReduce的task来执行，因此执行速度会比较慢。

hive的核心是驱动器，它可以连接sql和hdfs，将sql转换成了MapReduce任务，驱动器主要包括：

（1）解析器：解析sql语句，划分为不同的stage

（2）编译器：将不同阶段的stage编译，变成一个个的MR任务

（3）优化器：对逻辑执行进行优化

（4）执行器：将sql里的逻辑任务转换为hdfs的物理任务，hive的执行器就是MapReduce

hive 内部表和外部表区别

内部表：主要用在数据仓库层（DW层），为个人独自占有，如果删除表格，对应的原始数据也将删除，但是对其他人没有影响。

外部表：主要用在源数据层（ODS层），删除表格不会删除对应的数据。

在建表时，如果是内部表，不需要使用external关键字，外部表需要使用external关键字。

创建表格导入数据练习1

战狼2,吴京:吴刚:卢婧姗,2017-08-16
大话西游,周星驰:吴孟达,1995-09-01
哪吒,吕艳婷:瀚墨,2019-07-26
使徒行者2,张家辉:古天乐:吴镇宇,2019-08-07
鼠胆英雄,岳云鹏:佟丽娅:田雨:袁弘,2019-08-02

创建表格，导入数据。

# 建表
0: jdbc:hive2://node01:10000> create table movie_info(moviename string,actors array<string>,showtime string) row format delimited by ',' collection items teminated by ':';
Error: Error while compiling statement: FAILED: ParseException line 1:100 cannot recognize input near 'by' '','' 'collection' in serde properties specification (state=42000,code=40000)
0: jdbc:hive2://node01:10000> create table movie_info(moviename string,actors array<string>,showtime string) row format delimited fields terminated by ',' collection items teminated by ':';
Error: Error while compiling statement: FAILED: ParseException line 1:142 mismatched input 'teminated' expecting TERMINATED near 'items' in table row format's column separator (state=42000,code=40000)
0: jdbc:hive2://node01:10000> create table movie_info(moviename string,actors array<string>,showtime string) row format delimited fields terminated by ',' collection items terminated by ':';
INFO  : Compiling command(queryId=hadoop_20191115220202_01acb251-d8e2-46c5-bf20-5b354d6d2923): create table movie_info(moviename string,actors array<string>,showtime string) row format delimited fields terminated by ',' collection items terminated by ':'
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115220202_01acb251-d8e2-46c5-bf20-5b354d6d2923); Time taken: 0.035 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115220202_01acb251-d8e2-46c5-bf20-5b354d6d2923): create table movie_info(moviename string,actors array<string>,showtime string) row format delimited fields terminated by ',' collection items terminated by ':'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hadoop_20191115220202_01acb251-d8e2-46c5-bf20-5b354d6d2923); Time taken: 0.286 seconds
INFO  : OK
No rows affected (0.359 seconds)
# 加载数据
0: jdbc:hive2://node01:10000> load data local inpath '/kkb/install/hivedatas/move_info.txt' into table movie_info;
INFO  : Compiling command(queryId=hadoop_20191115220505_1c2ea52f-37cf-43d2-85b0-e2b54049aa33): load data local inpath '/kkb/install/hivedatas/move_info.txt' into table movie_info
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115220505_1c2ea52f-37cf-43d2-85b0-e2b54049aa33); Time taken: 0.061 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115220505_1c2ea52f-37cf-43d2-85b0-e2b54049aa33): load data local inpath '/kkb/install/hivedatas/move_info.txt' into table movie_info
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table db_hive.movie_info from file:/kkb/install/hivedatas/move_info.txt
INFO  : Starting task [Stage-1:STATS] in serial mode
INFO  : Table db_hive.movie_info stats: [numFiles=1, totalSize=235]
INFO  : Completed executing command(queryId=hadoop_20191115220505_1c2ea52f-37cf-43d2-85b0-e2b54049aa33); Time taken: 0.645 seconds
INFO  : OK
No rows affected (0.726 seconds)
# 查询加载后结果
0: jdbc:hive2://node01:10000> select * from movie_info;
INFO  : Compiling command(queryId=hadoop_20191115220505_0e4cbd35-3f1c-44d8-8a03-2b7dcd7ab03e): select * from movie_info
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:movie_info.moviename, type:string, comment:null), FieldSchema(name:movie_info.actors, type:array<string>, comment:null), FieldSchema(name:movie_info.showtime, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115220505_0e4cbd35-3f1c-44d8-8a03-2b7dcd7ab03e); Time taken: 0.09 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115220505_0e4cbd35-3f1c-44d8-8a03-2b7dcd7ab03e): select * from movie_info
INFO  : Completed executing command(queryId=hadoop_20191115220505_0e4cbd35-3f1c-44d8-8a03-2b7dcd7ab03e); Time taken: 0.0 seconds
INFO  : OK
+-----------------------+--------------------------+----------------------+--+
| movie_info.moviename  |    movie_info.actors     | movie_info.showtime  |
+-----------------------+--------------------------+----------------------+--+
| 战狼2                   | ["吴京","吴刚","卢婧姗"]        | 2017-08-16           |
| 大话西游                  | ["周星驰","吴孟达"]            | 1995-09-01           |
| 哪吒                    | ["吕艳婷","瀚墨"]             | 2019-07-26           |
| 使徒行者2                 | ["张家辉","古天乐","吴镇宇"]      | 2019-08-07           |
| 鼠胆英雄                  | ["岳云鹏","佟丽娅","田雨","袁弘"]  | 2019-08-02           |
+-----------------------+--------------------------+----------------------+--+
5 rows selected (0.177 seconds)

3.1 查询出每个电影的第二个主演

0: jdbc:hive2://node01:10000> select moviename,actors[1] from movie_info;
INFO  : Compiling command(queryId=hadoop_20191115220909_702e25ac-bbf9-4fb3-a39e-44156379fcf3): select moviename,actors[1] from movie_info
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:moviename, type:string, comment:null), FieldSchema(name:_c1, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115220909_702e25ac-bbf9-4fb3-a39e-44156379fcf3); Time taken: 0.134 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115220909_702e25ac-bbf9-4fb3-a39e-44156379fcf3): select moviename,actors[1] from movie_info
INFO  : Completed executing command(queryId=hadoop_20191115220909_702e25ac-bbf9-4fb3-a39e-44156379fcf3); Time taken: 0.0 seconds
INFO  : OK
+------------+------+--+
| moviename  | _c1  |
+------------+------+--+
| 战狼2        | 吴刚   |
| 大话西游       | 吴孟达  |
| 哪吒         | 瀚墨   |
| 使徒行者2      | 古天乐  |
| 鼠胆英雄       | 佟丽娅  |
+------------+------+--+

3.2 查询每部电影有几名主演

0: jdbc:hive2://node01:10000> select moviename,size(actors) as actorcount from movie_info;
INFO  : Compiling command(queryId=hadoop_20191115221010_1b46f1ce-12bf-406d-9cfc-395df7f26816): select moviename,size(actors) as actorcount from movie_info
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:moviename, type:string, comment:null), FieldSchema(name:actorcount, type:int, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115221010_1b46f1ce-12bf-406d-9cfc-395df7f26816); Time taken: 0.075 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115221010_1b46f1ce-12bf-406d-9cfc-395df7f26816): select moviename,size(actors) as actorcount from movie_info
INFO  : Completed executing command(queryId=hadoop_20191115221010_1b46f1ce-12bf-406d-9cfc-395df7f26816); Time taken: 0.001 seconds
INFO  : OK
+------------+-------------+--+
| moviename  | actorcount  |
+------------+-------------+--+
| 战狼2        | 3           |
| 大话西游       | 2           |
| 哪吒         | 2           |
| 使徒行者2      | 3           |
| 鼠胆英雄       | 4           |
+------------+-------------+--+

3.3主演里面包含古天乐的电影

# 需要使用lateral view
0: jdbc:hive2://node01:10000> Select t.moviename,t.actor from
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > select moviename,actor from movie_info lateral view explode(actors)temp as actor
. . . . . . . . . . . . . . > ) t
. . . . . . . . . . . . . . > Where t.actor='古天乐';
INFO  : Compiling command(queryId=hadoop_20191115222525_532eeeb8-06db-4828-9e6d-73594e31877e): Select t.moviename,t.actor from
(
select moviename,actor from movie_info lateral view explode(actors)temp as actor
) t
Where t.actor='古天乐'
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t.moviename, type:string, comment:null), FieldSchema(name:t.actor, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115222525_532eeeb8-06db-4828-9e6d-73594e31877e); Time taken: 0.225 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115222525_532eeeb8-06db-4828-9e6d-73594e31877e): Select t.moviename,t.actor from
(
select moviename,actor from movie_info lateral view explode(actors)temp as actor
) t
Where t.actor='古天乐'
INFO  : Completed executing command(queryId=hadoop_20191115222525_532eeeb8-06db-4828-9e6d-73594e31877e); Time taken: 0.0 seconds
INFO  : OK
+--------------+----------+--+
| t.moviename  | t.actor  |
+--------------+----------+--+
| 使徒行者2        | 古天乐      |
+--------------+----------+--+

创建表格导入数据练习2

1,张三,18:male:北京
2,李四,29:female:上海
3,杨朝来,22:male:深圳?
4,蒋平,34:male:成都
5,唐灿华,25:female:哈尔滨
6,马达,17:male:北京
7,赵小雪,23:female:杭州
8,薛文泉,26:male:上海
9,丁建,29:male:北京

创建表格，加载数据。

# 创建表格
0: jdbc:hive2://node01:10000> create table dept(id int,name string,info struct<age:int,gender:string,city:string>) row format delimited fields terminated by ',' collection items terminated by ':';
INFO  : Compiling command(queryId=hadoop_20191115223434_1d5c9e49-c4e7-4930-938d-4a5823486099): create table dept(id int,name string,info struct<age:int,gender:string,city:string>) row format delimited fields terminated by ',' collection items terminated by ':'
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115223434_1d5c9e49-c4e7-4930-938d-4a5823486099); Time taken: 0.008 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115223434_1d5c9e49-c4e7-4930-938d-4a5823486099): create table dept(id int,name string,info struct<age:int,gender:string,city:string>) row format delimited fields terminated by ',' collection items terminated by ':'
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hadoop_20191115223434_1d5c9e49-c4e7-4930-938d-4a5823486099); Time taken: 0.086 seconds
INFO  : OK
No rows affected (0.131 seconds)
0: jdbc:hive2://node01:10000> desc dept;
INFO  : Compiling command(queryId=hadoop_20191115223434_270cf471-d325-40ed-af27-4afff5a6aabf): desc dept
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115223434_270cf471-d325-40ed-af27-4afff5a6aabf); Time taken: 0.1 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115223434_270cf471-d325-40ed-af27-4afff5a6aabf): desc dept
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hadoop_20191115223434_270cf471-d325-40ed-af27-4afff5a6aabf); Time taken: 0.042 seconds
INFO  : OK
+-----------+--------------------------------------------+----------+--+
| col_name  |                 data_type                  | comment  |
+-----------+--------------------------------------------+----------+--+
| id        | int                                        |          |
| name      | string                                     |          |
| info      | struct<age:int,gender:string,city:string>  |          |
+-----------+--------------------------------------------+----------+--+
3 rows selected (0.169 seconds)
# 加载数据
0: jdbc:hive2://node01:10000> load data local inpath '/kkb/install/hivedatas/dept.txt' overwrite into table dept;
INFO  : Compiling command(queryId=hadoop_20191115223535_d81bba57-47cc-473b-be36-fe32053922b8): load data local inpath '/kkb/install/hivedatas/dept.txt' overwrite into table dept
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115223535_d81bba57-47cc-473b-be36-fe32053922b8); Time taken: 0.026 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115223535_d81bba57-47cc-473b-be36-fe32053922b8): load data local inpath '/kkb/install/hivedatas/dept.txt' overwrite into table dept
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table db_hive.dept from file:/kkb/install/hivedatas/dept.txt
INFO  : Starting task [Stage-1:STATS] in serial mode
INFO  : Table db_hive.dept stats: [numFiles=1, totalSize=240]
INFO  : Completed executing command(queryId=hadoop_20191115223535_d81bba57-47cc-473b-be36-fe32053922b8); Time taken: 0.303 seconds
INFO  : OK
# 查询数据
0: jdbc:hive2://node01:10000> select * from dept;
INFO  : Compiling command(queryId=hadoop_20191115223737_b10807f4-8f05-4b3b-aa33-4e0bc8c408f6): select * from dept
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:dept.id, type:int, comment:null), FieldSchema(name:dept.name, type:string, comment:null), FieldSchema(name:dept.info, type:struct<age:int,gender:string,city:string>, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115223737_b10807f4-8f05-4b3b-aa33-4e0bc8c408f6); Time taken: 0.054 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115223737_b10807f4-8f05-4b3b-aa33-4e0bc8c408f6): select * from dept
INFO  : Completed executing command(queryId=hadoop_20191115223737_b10807f4-8f05-4b3b-aa33-4e0bc8c408f6); Time taken: 0.0 seconds
INFO  : OK
+----------+------------+--------------------------------------------+--+
| dept.id  | dept.name  |                 dept.info                  |
+----------+------------+--------------------------------------------+--+
| 1        | 张三         | {"age":18,"gender":"male","city":"北京"}     |
| 2        | 李四         | {"age":29,"gender":"female","city":"上海"}   |
| 3        | 杨朝来        | {"age":22,"gender":"male","city":"深圳?"}    |
| 4        | 蒋平         | {"age":34,"gender":"male","city":"成都"}     |
| 5        | 唐灿华        | {"age":25,"gender":"female","city":"哈尔滨"}  |
| 6        | 马达         | {"age":17,"gender":"male","city":"北京"}     |
| 7        | 赵小雪        | {"age":23,"gender":"female","city":"杭州"}   |
| 8        | 薛文泉        | {"age":26,"gender":"male","city":"上海"}     |
| 9        | 丁建         | {"age":29,"gender":"male","city":"北京"}     |
+----------+------------+--------------------------------------------+--+

4.1 查询出每个人的id,名字,居住地址

0: jdbc:hive2://node01:10000> select id,name,info.city from dept;
INFO  : Compiling command(queryId=hadoop_20191115224040_796610ca-d24a-48ac-a8aa-d1eaee1ddce9): select id,name,info.city from dept
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:id, type:int, comment:null), FieldSchema(name:name, type:string, comment:null), FieldSchema(name:city, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hadoop_20191115224040_796610ca-d24a-48ac-a8aa-d1eaee1ddce9); Time taken: 0.081 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20191115224040_796610ca-d24a-48ac-a8aa-d1eaee1ddce9): select id,name,info.city from dept
INFO  : Completed executing command(queryId=hadoop_20191115224040_796610ca-d24a-48ac-a8aa-d1eaee1ddce9); Time taken: 0.001 seconds
INFO  : OK
+-----+-------+-------+--+
| id  | name  | city  |
+-----+-------+-------+--+
| 1   | 张三    | 北京    |
| 2   | 李四    | 上海    |
| 3   | 杨朝来   | 深圳?   |
| 4   | 蒋平    | 成都    |
| 5   | 唐灿华   | 哈尔滨   |
| 6   | 马达    | 北京    |
| 7   | 赵小雪   | 杭州    |
| 8   | 薛文泉   | 上海    |
| 9   | 丁建    | 北京    |
+-----+-------+-------+--+

以上为hive练习部分，记录一下。