sqoop提交参数说明

Posted 2022-12-09 上官沐雪

sqoop提交参数解析

/opt/cloudera/parcels/CDH/lib/sqoop/bin/sqoop import \\
--connect jdbc:mysql://ip:port/T_Cloud_Promote?tinyInt1isBit=false \\
--username rt_center \\
--password pwd\\
--query "select KeywordID,KeyType,Root,KeywordName,UID,Status,AddDate,UpdateDate,MigrateOID,PromotionStatus,WordType,IsEmphasis,KeywordLength,IsViolate,ViolateWord from T_Cloud_Promote.T_Cloud_User_KeywordLibrary where KeywordID < $[a+10000000] AND KeywordID >= $a AND AddDate != '0000-00-00 00:00:00' AND \\$CONDITIONS" \\
--fields-terminated-by '\\001' \\
--target-dir /tmp/t_cloud_order \\
--delete-target-dir \\
--hive-import \\
--hive-database T_Cloud_Promote \\
--hive-table ods_t_cloud_user_keywordlibraryb_tmp_01 \\
--null-string '\\\\N' \\
--null-non-string '\\\\N' \\
--hive-drop-import-delims \\
--num-mappers 5 \\
--split-by KeywordID \\
--hive-overwrite

注释

1. –connect
   mysql连接

2. –username
   用户名

3. –password
   密码

4. –query
   条件查询语句

5. –fields-terminated-by
   分隔符

6. –target-dir
   临时存放位置

7. –delete-target-dir
   程序结束删除文件夹

8. –hive-import
   导入到hive

9. –hive-database T_Cloud_Promote
   hive的数据库

10. –hive-table
    hive表

11. –null-string ‘\\N’
    –null-non-string ‘\\N’
    转换为hive空

12. –hive-drop-import-delims
    删除特殊分隔符如：\\n\\r \\0x01

13. –num-mappers 5
    指定map的个数

14. –split-by KeywordID
    指定按哪个字段切分

15. –hive-overwrite
    覆盖hive表，若不写则追加

16. -Dorg.apache.sqoop.splitter.allow_text_splitter=true
    允许切割字符串