DataX mysql与hive间传输数据

Posted 2021-09-17 羲凡丞相

@羲凡——只为了更好的活着

DataX mysql与hive间传输数据

官网

前期准备
a.下载并配置DataX，见官网
b.有hive环境和mysql库

0.注意事项

a.mysql数据导入到hive 中 connection 中的 jdbcUrl必须是个list，也就是要加中括号
b.hive数据导出到mysql 中 connection 中的 jdbcUrl必须是string，也就是别加中括号
c.hive数据导出到mysql 中,hdfsreader数据类型转化,hive中的TINYINT,SMALLINT,INT,BIGINT都对应LONG
d.hive数据导出到mysql 中,对于hdfsreader指定Column信息，type必须填写，index/value必须选择其一,默认情况下，可以全部按照String类型读取数据 “column”: ["*"]

1.mysql数据导入到hive(新建文件script/mysql2hive.json)

{
    "job": {
        "setting": {
            "speed": {
                "channel": 2
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "123456",
                        "column": ["id","name","age","city"],
                        "splitPk": "id",
                        "connection": [
                            {
                                "table": ["t_user_info"],
                                "jdbcUrl": ["jdbc:mysql://10.218.223.96:3306/test"]
                            }
                        ]
                    }
                },
               "writer": {
                    "name": "hdfswriter",
                    "parameter": {
                        "defaultFS": "hdfs://10.218.223.97:8020",
                        "fileType": "text",
                        "path": "/user/hive/warehouse/test.db/h_user_info",
                        "fileName": "mysql2hive",
                        "column": [
                            {"name": "id","type": "TINYINT"},
                            {"name": "name","type": "STRING"},
                            {"name": "age","type": "INT"},
                            {"name": "city","type": "STRING"}
                        ],
                        "writeMode": "nonConflict",
                        "fieldDelimiter": "\\t",
						"compress":"gzip"
                    }
                }
            }
        ]
    }
}

cd $DATAX_HOME
python bin/datax.py script/mysql2hive.json

2.hive数据导出到mysql

{
    "job": {
        "setting": {
            "speed": {
                "channel": 2
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.02
            }
        },
        "content": [
            {
                "reader": {
                    "name": "hdfsreader",
                    "parameter": {
                        "defaultFS": "hdfs://10.218.223.97:8020",
                        "fileType": "text",
                        "encoding": "UTF-8",
                        "path": "/user/hive/warehouse/test.db/h_user_info/*",
                        "column": [
                            {"index": 0,"type": "LONG"},
                            {"index": 1,"type": "STRING"},
                            {"index": 2,"type": "LONG"},
                            {"value": "南京","type": "STRING"}
                        ],
                        "fieldDelimiter": "\\t",
                        "compress":"gzip"
                    }
                },
               "writer": {
                    "name": "mysqlwriter",
                    "parameter": {
                        "writeMode": "insert",
                        "username": "root",
                        "password": "123456",
                        "column": ["id","name","age","city"],
                        "session": ["set session sql_mode='ANSI'"],
                        "preSql": ["delete from t_user_info"],
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://10.218.223.96:3306/test?useUnicode=true&characterEncoding=gbk",
                                "table": ["t_user_info"]
                            }
                        ]
                    }
                }
            }
        ]
    }
}

====================================================================

@羲凡——只为了更好的活着

若对博客中有任何问题，欢迎留言交流