configuration.class 定义在 hadoop 2.0 中移动了吗？

Posted 2023-02-16

【中文标题】configuration.class 定义在 hadoop 2.0 中移动了吗？

【英文标题】：Has configuration.class definition moved in hadoop 2.0?

【发布时间】：2016-01-23 08:58:50

【问题描述】：

我正在将我的一些 MapReduce 代码从 Hadoop 1.0 迁移到 Hadoop 2.0。我从简单的 WordCount 工作开始。我将以下 jars 添加到我的构建路径并删除了相应的 Hadoop 1.0 jars。

hadoop-mapreduce-client-app-2.7.1.jar hadoop-mapreduce-client-common-2.7.1.jar hadoop-mapreduce-client-core-2.7.1.jar hadoop-mapreduce-client-core-2.7.1.jar hadoop-mapreduce-client-hs-2.7.1.jar hadoop-mapreduce-client-hs-plugins-2.7.1.jar hadoop-mapreduce-client-jobclient-2.7.1-tests.jar hadoop-mapreduce-client-jobclient-2.7.1.jar hadoop-mapreduce-client-shuffle-2.7.1.jar

我保留了带有以下客户端定义的简单 WordCount 类。

JobConf conf = new JobConf(new Configuration(), WordCount.class);
conf.setJobName("WordCount");

conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(WordCountMap.class);
//conf.setCombinerClass(Reduce.class);
conf.setReducerClass(WordCountReduce.class);

conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf, new Path(params.getInput()));
FileOutputFormat.setOutputPath(conf, new Path(params.getOutput()));

JobClient.runJob(conf);

实例化 JobConf 时出现以下错误。

java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2957)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1210)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1690)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at bigdata.hadoop.WordCount.execute(WordCount.java:55)
at bigdata.hadoop.WordCount.execute(WordCount.java:1)
at bigdata.hadoop.BigDataJobDriver.executeJobDriver(BigDataJobDriver.java:15)
at bigdata.jobs.WordCountJob.executeJob(WordCountJob.java:50)
at bigdata.quartz.BigDataJob.execute(BigDataJob.java:30)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 14 more
2016-01-23 03:03:35,408 ERROR [QuartzScheduler_Worker-1] core.ErrorLogger (QuartzScheduler.java:schedulerError(2361)) - Job (Hadoop.Hadoop 7 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration]
at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2957)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1210)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1690)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at bigdata.hadoop.WordCount.execute(WordCount.java:55)
at bigdata.hadoop.WordCount.execute(WordCount.java:1)
at bigdata.hadoop.BigDataJobDriver.executeJobDriver(BigDataJobDriver.java:15)
at bigdata.jobs.WordCountJob.executeJob(WordCountJob.java:50)
at bigdata.quartz.BigDataJob.execute(BigDataJob.java:30)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 14 more

我执行了以下命令，但该命令找不到 Configuration.class。 hadoop 1.0 上的相同命令显示了 Configuration.class。

tar tvf hadoop-mapreduce-client-core-2.7.1.jar | grep Configuration.class

是否已将 Configuration.class 移至 Hadoop 2.7.1 的不同 jar 文件？

【参考方案1】：

通过制作配置对象进行检查并将其传递给 Job 对象。

喜欢： 配置 conf = new Configuration(); Job job = new Job(conf, "WordCount");

【参考方案2】：

尝试用这个替换

Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);

job.setMapperClass(WordCountMap.class);
job.setReducerClass(WordCountReduce.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.waitForCompletion(true);