DataX mysql与hive间传输数据
Posted 羲凡丞相
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了DataX mysql与hive间传输数据相关的知识，希望对你有一定的参考价值。
@羲凡——只为了更好的活着
DataX mysql与hive间传输数据
前期准备
a.下载并配置DataX，见官网
b.有hive环境和mysql库
0.注意事项
a.mysql数据导入到hive 中 connection 中的 jdbcUrl必须是个list，也就是要加中括号
b.hive数据导出到mysql 中 connection 中的 jdbcUrl必须是string，也就是别加中括号
c.hive数据导出到mysql 中,hdfsreader数据类型转化,hive中的TINYINT,SMALLINT,INT,BIGINT都对应LONG
d.hive数据导出到mysql 中,对于hdfsreader指定Column信息，type必须填写，index/value必须选择其一,默认情况下，可以全部按照String类型读取数据 “column”: ["*"]
1.mysql数据导入到hive(新建文件script/mysql2hive.json)
"job":
"setting":
"speed":
"channel": 2
,
"errorLimit":
"record": 0,
"percentage": 0.02
,
"content": [
"reader":
"name": "mysqlreader",
"parameter":
"username": "root",
"password": "123456",
"column": ["id","name","age","city"],
"splitPk": "id",
"connection": [
"table": ["t_user_info"],
"jdbcUrl": ["jdbc:mysql://10.218.223.96:3306/test"]
]
,
"writer":
"name": "hdfswriter",
"parameter":
"defaultFS": "hdfs://10.218.223.97:8020",
"fileType": "text",
"path": "/user/hive/warehouse/test.db/h_user_info",
"fileName": "mysql2hive",
"column": [
"name": "id","type": "TINYINT",
"name": "name","type": "STRING",
"name": "age","type": "INT",
"name": "city","type": "STRING"
],
"writeMode": "nonConflict",
"fieldDelimiter": "\\t",
"compress":"gzip"
]
cd $DATAX_HOME
python bin/datax.py script/mysql2hive.json
2.hive数据导出到mysql
"job":
"setting":
"speed":
"channel": 2
,
"errorLimit":
"record": 0,
"percentage": 0.02
,
"content": [
"reader":
"name": "hdfsreader",
"parameter":
"defaultFS": "hdfs://10.218.223.97:8020",
"fileType": "text",
"encoding": "UTF-8",
"path": "/user/hive/warehouse/test.db/h_user_info/*",
"column": [
"index": 0,"type": "LONG",
"index": 1,"type": "STRING",
"index": 2,"type": "LONG",
"value": "南京","type": "STRING"
],
"fieldDelimiter": "\\t",
"compress":"gzip"
,
"writer":
"name": "mysqlwriter",
"parameter":
"writeMode": "insert",
"username": "root",
"password": "123456",
"column": ["id","name","age","city"],
"session": ["set session sql_mode='ANSI'"],
"preSql": ["delete from t_user_info"],
"connection": [
"jdbcUrl": "jdbc:mysql://10.218.223.96:3306/test?useUnicode=true&characterEncoding=gbk",
"table": ["t_user_info"]
]
]
====================================================================
@羲凡——只为了更好的活着
若对博客中有任何问题，欢迎留言交流
以上是关于DataX mysql与hive间传输数据的主要内容，如果未能解决你的问题，请参考以下文章