Sqoop_mysql,hive,hdfs导入导出操作

Posted 2020-08-19

一、准备数据

# 在我的mysql下创建数据库和表,并插入几条数据
  mysql> create database if not exists student default character set utf8 collate utf8_general_ci;
  mysql> use student;
  mysql> create table if not exists stu_info( id int(10) primary key not null auto_increment, name varchar(20) not null) default character set utf8 collate utf8_general_ci;
  mysql> insert into stu_info(name) values("李建");
  mysql> insert into stu_info(name) values("张明");
  mysql> insert into stu_info(name) values("赵兴");
  mysql> insert into stu_info(name) values("陈琦");
  mysql> insert into stu_info(name) values("刘铭");
  mysql> select id,name from stu_info;
  +----+--------+
  | id | name   |
  +----+--------+
  |  1 | 李建   |
  |  2 | 张明   |
  |  3 | 赵兴   |
  |  4 | 陈琦   |
  |  5 | 刘铭   |
  +----+--------+
  5 rows in set (0.00 sec)

二、使用sqoop将mysql中的这张表导入到hdfs上

bin/sqoop import --connect jdbc:mysql://10.0.0.108:3306/student --username root --password root --table stu_info --target-dir /student --num-mappers 1 --fields-terminated-by ‘\t‘

三、使用sqoop将mysql中的这张表导入到hive

方式一、
1. 在hive中创建数据库和表
    create database if not exists student;
    create table if not exists stu_info(id int,name string) row format delimited fields terminated by ‘\t‘;
2. bin/sqoop import     --connect jdbc:mysql://hadoop09-linux-01.ibeifeng.com:3306/student     --username root --password root     --table stu_info     --delete-target-dir     --target-dir /user/hive/warehouse/student.db/stu_info     --hive-import     --hive-database student     --hive-table stu_info     --hive-overwrite     --num-mappers 1     --fields-terminated-by ‘\t‘
方式二、
1. 使用sqoop create-hive-table,但必须创建出自定义数据库,否则目标路径将是元数据库
2. bin/sqoop create-hive-table 、
    --connect jdbc:mysql://10.0.0.108:3306/student 、
    --username root --password root     --table stu_info     --hive-table student.stu_info
3. bin/sqoop import --connect jdbc:mysql://10.0.0.108:3306/student     --username root --password root     --table stu_info     --hive-import     --hive-database student     --hive-table stu_info     --hive-overwrite      --num-mappers 1     --fields-terminated-by ‘\t‘     --delete-target-dir     --target-dir /user/hive/warehouse/student.db/stu_info
4. 在hive中查询会发现数据全部为NULL
    但是从hdfs上查看却是正常的,确定hive无法解析数据,定位在分隔符问题
    使用--fields-terminated-by ‘\001‘ 即可  # \001就是ctrl+A,hive默认分隔符,mysql默认分隔符为","

五、从hdfs或hive导出数据到mysql表

1. 在mysql上准备好数据库和表
2. 数据库我就直接使用student数据库
    create table if not exists stu_info_export like stu_info;
3. 根据hdfs/hive表数据分隔符为主
    bin/sqoop export     --connect jdbc:mysql://10.0.0.108/student     --username root --password root     --table stu_info_export     --export-dir /user/hive/warehouse/student.db/stu_info     --num-mappers 1     --input-fields-terminated-by ‘\001‘