sqoop 导入未识别显示 ClassNotFountException 错误的 mysql 表

Posted 2023-04-18

【中文标题】sqoop 导入未识别显示 ClassNotFountException 错误的 mysql 表

【英文标题】：sqoop import is not identifying mysql tables showing ClassNotFountException Error

【发布时间】：2017-10-31 05:25:57

【问题描述】：

我有一个 mysql 表“客户”并尝试使用 sqoop 导入将数据从 mysql 导入到 hdfs 位置。以下是我在我的机器上安装的版本： sqoop 版本：1.4.6 蜂巢版本：2.3.0 Hadoop版本：2.8.1

Sqoop 导入命令：

sqoop import --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234 --table customers -m 1 --target-dir /user/hduser/sqoop_import/customers1/

并显示在 ClassNotFountException 下方：

Tue Oct 31 09:57:21 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.17/10/31 09:57:21 INFO db.DBInputFormat: Using read commited transaction isolation             17/10/31 09:57:21 INFO mapred.MapTask: Processing split: 1=1 AND 1=1 
17/10/31 09:57:21 INFO mapred.LocalJobRunner: map task executor complete. 
17/10/31 09:57:22 WARN mapred.LocalJobRunner: job_local1437452057_0001java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found 
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class customers not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2216) 
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:237)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:263)     at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:515)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) 
Caused by: java.lang.ClassNotFoundException: Class customers not found     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
... 12 more
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 running in uber mode : false  17/10/31 09:57:22 INFO mapreduce.Job:  map 0% reduce 0%
17/10/31 09:57:22 INFO mapreduce.Job: Job job_local1437452057_0001 failed with state FAILED due to: NA  17/10/31 09:57:22 INFO mapreduce.Job: Counters: 0
17/10/31 09:57:22 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead  17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 4.105 seconds (0 bytes/sec)
17/10/31 09:57:22 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead  17/10/31 09:57:22 INFO mapreduce.ImportJobBase: Retrieved 0 records.
17/10/31 09:57:22 ERROR tool.ImportTool: Error during import: Import job failed!

但是当我尝试使用 sqoop 命令列出表时，它工作正常并显示客户表。 Sqoop 命令：

sqoop list-tables --connect jdbc:mysql://localhost/localdb --username root --password mnbv@1234;

输出正确显示如下：

17/10/31 10:07:09 INFO manager.MySQLManager: Preparing to use a MySQL  streaming resultset.          
Tue Oct 31 10:07:09 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+,  5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not  using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true  and provide truststore for server certificate verification.                                                                                                customers

可能是什么问题，为什么从表中导入的 sqoop 无法识别来自 mysql 的表。请帮助我。

提前致谢。

【参考方案1】：

使用 --bindir 选项并指向您当前的工作目录。在 --bindir 选项下给出 $SQOOP_HOME/lib 的路径

抱歉回复晚了。根据最新评论更新答案。请检查以下内容：

sqoop 进程运行时需要检查 jar 文件 cmd：Writing jar file:/tmp/sqoop.../*.jar 使用导入命令：sqoop import -fs local -jt local -libjars /tmp/..../*.jar --connect "jdbc:mysql://example.com/sqoop" --username <>--password <>--table customers 验证生成的用于 sqoop 导入的 Java 文件。它应该有 与您正常导入的表同名。 sqoop codegen --connect <""> --username <> --password <> --table customers

希望这会有所帮助！

【讨论】：

17/11/01 09:32:24 INFO mapreduce.Job：作业 job_local349901698_0001 在 uber 模式下运行：false 17/11/01 09:32:24 INFO mapreduce.Job：地图 0% 减少0% 17/11/01 09:32:24 INFO mapreduce.Job：作业 job_local349901698_0001 失败，状态为 FAILED 由于：NA 17/11/01 09:32:24 INFO mapreduce.Job：计数器：8 Map-Reduce 框架映射输入记录=2 映射输出记录=2 输入拆分字节=87 溢出记录=0 失败随机播放=0 合并映射输出=0 文件输入格式计数器字节读取=0 文件输出格式计数器写入字节=164

17/11/01 09:32:24 警告 mapreduce.Counters：Group FileSystemCounters 已弃用。改用 org.apache.hadoop.mapreduce.FileSystemCounter 17/11/01 09:32:24 INFO mapreduce.ImportJobBase：在 3.8506 秒内传输 0 个字节（0 个字节/秒） 17/11/01 09:32:24 INFO mapreduce .ImportJobBase：检索到 2 条记录。 17/11/01 09:32:24 ERROR tool.ImportTool：导入期间出错：导入作业失败！找不到问题请帮忙。