踩坑：Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer

Posted 2022-05-02 T&King

今天在做MapReduce的时候，为了做个降序排列，所以定义了一个自定义序列，然后运行的时候就出现了这个问题，查了好多教程，最后发现错在了一个比较离谱的地方，记录一下，顺带写上其他几种可能。（1-3来源网络，4亲身经历）
先放个报错信息以供参考

2022-04-29 01:04:34,361 WARN  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(418)) - Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more
2022-04-29 01:04:34,364 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2022-04-29 01:04:34,370 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1220408697_0001
java.lang.Exception: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:424)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        ... 10 more
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more

情况1

包导入错了，这是网上最多情况的报错原因，比如Text应该要导入org.apache.hadoop.io.Text之类的，仔细检查一下，看看有没有。

情况2

自定义序列的时候没有没有完全继承Writable和Comparable, 这种情况的话，只要继承一下WritableComparable就ok了, WritableComparable已经继承了Writable和Comparable。比如我的自定义序列：

public class classes implements WritableComparable<classes>...

情况3

如果是自定义序列的话，再MapReduce操作中是要求其可迭代的，也就是说需要重写一个compareTo方法，比如我的

public int compareTo(classes o) 
        //降序
       return (int) (-this.avgsorce+o.getAvgsorce());
    

如果没写的话也可能出现报错

情况4

最后就是我的问题了，很简单，在自定义序列的时候，我只定义了一个有参的构造器，总所周知，在Java里如果已经定义了构造器，系统就不再有默认构造器了，这就导致我定义的这个自定义序列没有无参构造器，好巧不巧，MapReduce排序的时候就偏偏要用到这个无参构造器，得嘞，这可不就报错了，到此我定义了一个无参构造器，问题解决

public classes()

    

这坑有点深，找了我好久，再加上现在是北京时间1：21，半夜啦！！忍不住发一下吐槽一下（这句纯吐槽，不想看可以略过哈哈哈）

转载请注明出处，谢谢！

创作打卡挑战赛 赢取流量/现金/CSDN周边激励大奖