Sqoop 导入失败，“on 子句”中有未知列

Posted 2023-04-18

【中文标题】Sqoop 导入失败，“on 子句”中有未知列

【英文标题】：Sqoop import fails with Unknown column in 'on clause'

【发布时间】：2017-01-29 13:32:26

【问题描述】：

该查询与 mysql 完美配合，但与 Sqoop 一起使用时将数据从 MySql 导入 Hive 时失败。

select cmpdl.*,ct.value from (SELECT cmp.domain_id,
       cmp.model,
       cmp.name,
       cmp.status,
       br.name,
       cat.category,
       dl.start_date,
       dl.end_date,
       cmp.created_at,
       cmp.updated_at
FROM domain cmp
JOIN brands br ON cmp.brand_id=br.brands_id
JOIN delxxx dl ON cmp.del_id=dl.del_id
JOIN
  (SELECT c.domain_id,
          group_concat(b.category_name) AS category
   FROM domain c
   JOIN categories b ON find_in_set(b.category_id,xxxxx)
   GROUP BY c.domain_id) cat ON cmp.domain_id=cat.domain_id
WHERE cmp.domain_type='XXXXXX'
  AND cmp.category='XXXXXXX'
  AND cmp.model != 'XXX') cmpdl join context_targeting ct on cmpdl.domain_id=ct.domain_id and ct.context_targeting_group='xxxxxxx' AND $CONDITIONS;

错误日志

Error: java.io.IOException: SQLException in nextKeyValue
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'cmp.domain_id' in 'on clause'
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:389)
    at com.mysql.jdbc.Util.getInstance(Util.java:372)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2535)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1911)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2034)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:111)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
    ... 12 more

我无法弄清楚别名是如何超出范围并且找不到列的确切问题。

【问题讨论】：

domain 表有 domain_id 列吗？这里使用ON cmp.domain_id=cat.domain_id.

是的，当我在 mysql shell 中运行时，查询工作正常。

分享你的 sqoop 导入命令。

【参考方案1】：

我的 sqoop split-by 属性存在问题

其中我提到 cmp.domain_id 代替 cmpdl.domain_id 是因为 Sqoop Map-Reduce 作业失败