Sqoop导入具有带有where子句和并行处理的SQL查询
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Sqoop导入具有带有where子句和并行处理的SQL查询相关的知识，希望对你有一定的参考价值。
我在mysql中有一个如下表：
订单详细信息 ：
+---------+------------+-------------------+--------------+
| orderid | order_date | order_customer_id | order_status |
+---------+------------+-------------------+--------------+
| A001 | 10/30/2018 | C003 | Completed |
| A002 | 10/30/2018 | C005 | Completed |
| A451 | 11/02/2018 | C376 | Pending |
| P9209 | 10/30/2018 | C234 | Completed |
| P92099 | 10/30/2018 | C244 | Pending |
| P9210 | 10/30/2018 | C035 | Completed |
| P92398 | 10/30/2018 | C346 | Pending |
| P9302 | 10/30/2018 | C034 | Completed |
+---------+------------+-------------------+--------------+
和描述如下：
mysql> desc Order_Details_Sankha;
+-------------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+---------+-------+
| orderid | varchar(20) | NO | PRI | | |
| order_date | varchar(20) | YES | | NULL | |
| order_customer_id | varchar(20) | YES | | NULL | |
| order_status | varchar(20) | YES | | NULL | |
+-------------------+-------------+------+-----+---------+-------+
我正在使用以下sqoop导入并行处理：
sqoop import
--connect jdbc:mysql://ip-10-0-1-10.ec2.internal/06july2018_new
--username labuser
--password abc123
--driver com.mysql.jdbc.Driver
--query "select * from Order_Details where order_date = '10/30/2018' AND $CONDITIONS"
--target-dir /user/sankha087_gmail_com/outputs/EMP_Sankha_1112201888
--split-by ","
--m 3
我收到以下错误消息
18/12/15 17:15:14 WARN security.UserGroupInformation: PriviledgedActionException as:sankha087_gmail_com (auth:SIMPLE) cause:java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You hav
e an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), MAX(,) FROM (select * from Order_Details_Sankha where order_date = '10/30/201' a
t line 1
18/12/15 17:15:14 ERROR tool.ImportTool: Import failed: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to
your MySQL server version for the right syntax to use near '), MAX(,) FROM (select * from Order_Details_Sankha where order_date = '10/30/201' at line 1
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:207)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:305)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
请告知我的import语句需要更改的内容。
答案
垂直分割不会发生Sqoop并行执行，它发生在水平分割中。
--split-by应该是列名。列应该是均匀分布的列。
https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html#id1765770
阅读：7.2.4。控制并行性
以上是关于Sqoop导入具有带有where子句和并行处理的SQL查询的主要内容，如果未能解决你的问题，请参考以下文章
Sqoop 导入映射器失败，但 sqoop 作业显示正在运行
带有 lambda 表达式的 LINQ where 子句具有 OR 子句和返回不完整结果的空值
当少数表没有主键时，sqoop import-all 具有并行性