hadoop的Hive操作hdfs文件第一天

Posted 2020-08-04

hadoop的Hive操作hdfs文件第一天

hive的具体练习以下4个目标
1. 第一普通的hdfs文件能导入到hive中以供我们查询。

create table dept(deptID int,deptName string,address string);

load data local inpath ‘/home/dyq/Documents/dept‘ overwrite into table dept;

select * from dept;

hive> select * from dept;

OK

NULL NULL NULL

NULL NULL NULL

NULL NULL NULL

NULL NULL NULL

Time taken: 0.316 seconds, Fetched: 4 row(s)

从上面看导入成功但是查询的都是null那是因为没有加分隔.test表默认的有terminated by ‘\t‘ 
lines terminated by ‘\n‘  分隔符所以尽管有报错数据也是插入的。
正确的创建语法为

create table dept1(deptID int,deptName string,address string) row format delimited 

fields terminated by ‘,‘ lines terminated by ‘\n‘ stored as textfile

导入数据的时候语句不变

load data local inpath ‘/home/dyq/Documents/dept‘ overwrite into table dept1;

hive> select * from dept1;

OK

10 ACCOUNTING NEW YORK

20 RESEARCH DALLAS

30 SALES CHICAGO

40 OPERATIONS BOSTON

Time taken: 0.153 seconds, Fetched: 4 row(s)

看看成功了

二、查询结果导出来。

从hive中把表中的数据导出来保存成文本类型。
先检索索要的结果

’

hive> select deptID,deptName from dept1;

OK

10 ACCOUNTING NEW YORK

20 RESEARCH DALLAS

30 SALES CHICAGO

40 OPERATIONS BOSTON

Time taken: 4.735 seconds, Fetched: 4 row(s)

将查询结果输出至本地目录

insert overwrite local directory ‘/home/dyq/Documents/outputdept‘ select a.deptID,a.deptName from dept1 a; 

hive> insert overwrite local directory ‘/home/dyq/Documents/outputdept‘ select a.deptID,a.deptName from dept1 a;  

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.

Query ID = dyq_20160828100454_07848914-19c7-47a7-9dd6-7fe50a1f9a82

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there‘s no reduce operator

Starting Job = job_1472347049416_0003, Tracking URL = http://ubuntu:8088/proxy/application_1472347049416_0003/

Kill Command = /opt/hadoop-2.6.2/bin/hadoop job  -kill job_1472347049416_0003

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0

2016-08-28 10:05:14,004 Stage-1 map = 0%,  reduce = 0%

2016-08-28 10:05:26,132 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.23 sec

MapReduce Total cumulative CPU time: 2 seconds 230 msec

Ended Job = job_1472347049416_0003

Moving data to local directory /home/dyq/Documents/outputdept

MapReduce Jobs Launched: 

Stage-Stage-1: Map: 1   Cumulative CPU: 2.23 sec   HDFS Read: 3564 HDFS Write: 49 SUCCESS

Total MapReduce CPU Time Spent: 2 seconds 230 msec

OK

Time taken: 34.259 seconds

三、查看到处结果

在/home/dyq/Documents下出现了outdept目录里面有000000_0文件打开

10ACCOUNTING

20RESEARCH

30SALES

40OPERATIONS

呵呵成功了额