添加外部Jar时Hadoop NoClassDefFoundError

Posted 2023-04-14

【中文标题】添加外部Jar时Hadoop NoClassDefFoundError

【英文标题】：Hadoop NoClassDefFoundError when adding external Jar

【发布时间】：2016-01-31 02:30:05

【问题描述】：

我正在尝试使用外部 Jar 在 Hadoop 上运行 MapReduce 作业。我已经使用copyFromLocal 到/user/hduser/lib/ 将Jar 添加到HDFS。

在我的主要方法中，我将 Jar 添加到 DistributedCache。但是，当我运行 MapReduce 程序时，我在 Mapper 类中收到 NoClassDefFoundError。我已经尝试了许多在 SO 上发布的解决方案，这些解决方案来自其他有类似错误的人，但我还没有解决这个问题。任何指导表示赞赏。

来自主方法：

Configuration conf = new Configuration();
String jarToAdd1 = "/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar";
String jarToAdd2 = "/user/hduser/lib/joda-time-2.9.1.jar";
addJarToDistributedCache(jarToAdd1, conf);
addJarToDistributedCache(jarToAdd2, conf);
.........

添加到分布式缓存：

private static void addJarToDistributedCache(String jarToAdd, Configuration conf) throws IOException 

        Path hdfsJar = new Path(jarToAdd);
        DistributedCache.addFileToClassPath(hdfsJar,conf);
    

发生错误的映射器：

public static class Map1 extends Mapper<LongWritable, Text, IntWritable, UserData> 

Map<IntWritable, UserData> userLog = new HashMap<IntWritable, UserData>();

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException 

    String line = value.toString();                             
    StringTokenizer tokenizer = new StringTokenizer(line);      
    DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyyddmm HH:mm:ss");    // *********ERROR HAPPENS HERE **********

堆栈跟踪：

16/01/30 20:45:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/30 20:45:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/01/30 20:45:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/01/30 20:45:16 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
16/01/30 20:45:16 INFO input.FileInputFormat: Total input paths to process : 1
16/01/30 20:45:16 INFO mapreduce.JobSubmitter: number of splits:3
16/01/30 20:45:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1787244132_0001
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1-no-tzdb.jar as file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar <- /usr/local/hadoop/sbin/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: Localized hdfs://localhost:54310/user/hduser/lib/joda-time-2.9.1.jar as file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717451/joda-time-2.9.1-no-tzdb.jar
16/01/30 20:45:17 INFO mapred.LocalDistributedCacheManager: file:/app/hadoop/tmp/mapred/local/1454204717452/joda-time-2.9.1.jar
16/01/30 20:45:17 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/01/30 20:45:17 INFO mapreduce.Job: Running job: job_local1787244132_0001
16/01/30 20:45:17 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/01/30 20:45:18 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Waiting for map tasks
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000000_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/sentimentFeedback7.csv:0+143748596
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000001_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:0+134217728
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1787244132_0001_m_000002_0
16/01/30 20:45:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/01/30 20:45:18 INFO mapred.MapTask: Processing split: hdfs://localhost:54310/user/hduser/input/allergyConsumption7.csv:134217728+105486421
16/01/30 20:45:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/01/30 20:45:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/01/30 20:45:18 INFO mapred.MapTask: soft limit at 83886080
16/01/30 20:45:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/01/30 20:45:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/01/30 20:45:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/01/30 20:45:18 INFO mapred.MapTask: Starting flush of map output
16/01/30 20:45:18 INFO mapred.LocalJobRunner: map task executor complete.
16/01/30 20:45:18 WARN mapred.LocalJobRunner: job_local1787244132_0001
java.lang.Exception: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:45)
    at org.peach.fooddiary.FoodDiaryMR$Map1.map(FoodDiaryMR.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java:55)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 12 more
16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 running in uber mode : false
16/01/30 20:45:18 INFO mapreduce.Job:  map 0% reduce 0%
16/01/30 20:45:18 INFO mapreduce.Job: Job job_local1787244132_0001 failed with state FAILED due to: NA
16/01/30 20:45:19 INFO mapreduce.Job: Counters: 0

【参考方案1】：

首先 DistributedCache.addFileToClassPath 现在已弃用，但您可以使用：

    Configuration conf = getConf();
    Job job = Job.getInstance(conf, "My Job Name");
    job.addFileToClassPath(new Path("path"));

您是否仍然遇到同样的问题，手动执行，我的意思是通过将您的 jar 文件放入 hadoop 的类路径，将您的代码部署到 hadoop 本身：

1) 使用以下命令将需要执行的类导出到 hadoop：

在项目资源管理器中，右键单击项目名称并选择“导出...” 在“Java”部分下，选择“Jar 文件”，然后选择“下一步” 选择要存储导出的“jar”的目录

2) 转到正在运行 hadoop 的服务器并运行以下命令以了解 hadoop 正在查看的目录是什么，其中一个示例是 "$HADOOP_HOME/share/hadoop/common/lib/*"

$ hadoop classpath 

3) 将步骤 1 中导出的 jar 文件复制到您在步骤 2 中获得的目录之一，然后再次开始测试。

提示最好直接在您的服务器中测试 map reduce 并使用以下命令在您的服务器中运行它：

$ hadoop jar JarFileName.jar PakageName.ClassName /path/to/input /path/to/output

完整示例及详细说明请查看this