使用 sqoop 从 sql server 导入表时出错

Posted 2023-04-18

【中文标题】使用 sqoop 从 sql server 导入表时出错

【英文标题】：Error while importing tables form sqlserver with sqoop

【发布时间】：2016-04-08 14:29:25

【问题描述】：

我在将数据库表表单SQL-server 导入到hadoop HDFS 时遇到此错误，但是 list-databases 可以成功运行并且Hadoop 作业可以正确执行。这些是我得到的一些错误：

  sqoop import --connect 'jdbc:sqlserver://192.168.1.15:1433;username=root;password=root;database=project ' --query 'SELECT * FROM Car WHERE $CONDITIONS' --target-dir /data/Database --split-by 'car_id' --as-textfile

16/04/08 16:50:42 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /app/hadoop/tmp/mapred/local/1460123441737/parquet-avro-1.4.1.jar <- //parquet-avro-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Localized file:/usr/lib/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/parquet-avro-1.4.1.jar as file:/app/hadoop/tmp/mapred/local/1460123441737/parquet-avro-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1460123441738/paranamer-2.3.jar <- //paranamer-2.3.jar
16/04/08 16:50:42 WARN fs.FileUtil: Command 'ln -s /app/hadoop/tmp/mapred/local/1460123441738/paranamer-2.3.jar /paranamer-2.3.jar' failed 1 with: ln: failed to create symbolic link ‘/paranamer-2.3.jar’: Permission denied

16/04/08 16:50:42 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /app/hadoop/tmp/mapred/local/1460123441738/paranamer-2.3.jar <- //paranamer-2.3.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Localized file:/usr/lib/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/paranamer-2.3.jar as file:/app/hadoop/tmp/mapred/local/1460123441738/paranamer-2.3.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1460123441739/parquet-jackson-1.4.1.jar <- //parquet-jackson-1.4.1.jar
16/04/08 16:50:42 WARN fs.FileUtil: Command 'ln -s /app/hadoop/tmp/mapred/local/1460123441739/parquet-jackson-1.4.1.jar /parquet-jackson-1.4.1.jar' failed 1 with: ln: failed to create symbolic link ‘/parquet-jackson-1.4.1.jar’: Permission denied

16/04/08 16:50:42 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /app/hadoop/tmp/mapred/local/1460123441739/parquet-jackson-1.4.1.jar <- //parquet-jackson-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Localized file:/usr/lib/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/parquet-jackson-1.4.1.jar as file:/app/hadoop/tmp/mapred/local/1460123441739/parquet-jackson-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1460123441740/commons-codec-1.4.jar <- //commons-codec-1.4.jar
16/04/08 16:50:42 WARN fs.FileUtil: Command 'ln -s /app/hadoop/tmp/mapred/local/1460123441740/commons-codec-1.4.jar /commons-codec-1.4.jar' failed 1 with: ln: failed to create symbolic link ‘/commons-codec-1.4.jar’: Permission denied
16/04/08 16:50:42 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /app/hadoop/tmp/mapred/local/1460123441741/parquet-generator-1.4.1.jar <- //parquet-generator-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: Localized file:/usr/lib/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/parquet-generator-1.4.1.jar as file:/app/hadoop/tmp/mapred/local/1460123441741/parquet-generator-1.4.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1460123441710/commons-logging-1.1.1.jar
16/04/08 16:50:42 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1460123441711/jackson-annotations-2.3.0.jar

16/04/08 16:50:42 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/04/08 16:50:42 INFO mapreduce.Job: Running job: job_local194572897_0001
16/04/08 16:50:42 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/04/08 16:50:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/04/08 16:50:42 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/04/08 16:50:42 INFO mapred.LocalJobRunner: Waiting for map tasks
16/04/08 16:50:42 INFO mapred.LocalJobRunner: Starting task: attempt_local194572897_0001_m_000000_0
16/04/08 16:50:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
16/04/08 16:50:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/04/08 16:50:42 INFO db.DBInputFormat: Using read commited transaction isolation
16/04/08 16:50:42 INFO mapred.MapTask: Processing split: car_id >= 1 AND car_id <= 4
16/04/08 16:50:42 INFO mapred.LocalJobRunner: map task executor complete.
16/04/08 16:50:42 WARN mapred.LocalJobRunner: job_local194572897_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
    at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
    at org.apache.sqoop.mapreduce.db.SQLServerDBInputFormat.createDBRecordReader(SQLServerDBInputFormat.java:56)
    at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:263)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:515)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:758)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class QueryResult not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
    ... 12 more

结果是：

16/04/08 16:50:43 INFO mapreduce.Job: Job job_local194572897_0001 running in uber mode : false
16/04/08 16:50:43 INFO mapreduce.Job:  map 0% reduce 0%
16/04/08 16:50:43 INFO mapreduce.Job: Job job_local194572897_0001 failed with state FAILED due to: NA
16/04/08 16:50:43 INFO mapreduce.Job: Counters: 0
16/04/08 16:50:43 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
16/04/08 16:50:43 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 2.6174 seconds (0 bytes/sec)
16/04/08 16:50:43 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/04/08 16:50:43 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/04/08 16:50:43 ERROR tool.ImportTool: Error during import: Import job failed!

【问题讨论】：

这里没有太多细节，但在输出“权限被拒绝”的消息中肯定有一个共同的主题。我会从那里开始。

检查您在 sql server 中使用的登录名。即用户名和密码，是否有所需的权限，请检查..

我使用的用户名具有登录和使用 DML 命令所需的权限

【参考方案1】：

转到安全登录并检查您的登录用途 Sql Server 身份验证并取消选中这两个复选框 的启用密码策略并强制密码过期

还要确保在您的服务器属性中启用了 sql server auth 并选择了登录审核“无”单选按钮

【讨论】：

谢谢我的朋友，但上面提到的所有属性都正确完成