java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$

Posted 2020-11-25 一字千金

1.问题描述：在调试mapreduce辅助排序（二次排序）的过程中，运行程序总是报错

Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2021)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8860 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
8872 [Thread-6] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1887386767_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

 

2.原因分析：

（1）初步判断是是调用了空指针；

（2）当点击WritableComparator.java:157 会跳转到报错的地方是this.buffer.reset(b1, s1, l1);

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
    try {
        this.buffer.reset(b1, s1, l1);
        this.key1.readFields(this.buffer);
        this.buffer.reset(b2, s2, l2);
        this.key2.readFields(this.buffer);
        this.buffer.reset((byte[])null, 0, 0);
    } catch (IOException var8) {
        throw new RuntimeException(var8);
    }
（3）也就是说WritableComparator buffer没有初始化

public class WritableComparator implements RawComparator, Configurable {
    private static final ConcurrentHashMap<Class, WritableComparator> comparators = new ConcurrentHashMap();
    private Configuration conf;
    private final Class<? extends WritableComparable> keyClass;
    private final WritableComparable key1;
    private final WritableComparable key2;
    private final DataInputBuffer buffer;

（4）查找程序中使用到WritableComparator 相关的类的定义是分组类ReducerGroupingComparator  和第一字段的比较类FirstSecondComparator 没有默认的构造函数

public static class ReducerGroupingComparator  extends WritableComparator
{//一定要加默认构造函数，且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
    public ReducerGroupingComparator()
    {
        super(IntPair.class,true);
    }
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        IntPair pairA=(IntPair)a;
        IntPair pairB=(IntPair)b;
        //年份相同，返回值为0的分为一组
        return (int) (pairA.getFirst()-pairB.getFirst());
    }
}
//与FirstSecondComparator中定义的默认对比函数功能相同，这个用于显示设置对比类
public  static class FirstSecondComparator extends WritableComparator
{
    //

//一定要加默认构造函数，且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c

    public FirstSecondComparator()
    {
        super(IntPair.class,true);
    }
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        IntPair pairA=(IntPair)a;
        IntPair pairB=(IntPair)b;
        int cmp= (int) (pairA.getFirst()-pairA.getFirst());
        if (cmp!=0)
        {
            return cmp;
        }
        double A=pairA.getSecond();
        double B=pairB.getSecond();
        return -((A==B)? 0:(A>B? 1:-1));
    }
}
（5）为什么一定要加这个才不报java.lang.NullPointerException异常？点开super(IntPair.class,true);

会跳转到WritableComparator的构造函数

protected WritableComparator(Class<? extends WritableComparable> keyClass, boolean createInstances) {
    this(keyClass, (Configuration)null, createInstances);
}
在点击this，进一步跳转到另外一个重载构造函数，看到这里大家应该名明白了，上面报异常的地方，就是这些变量没有申请内存空间

protected WritableComparator(Class<? extends WritableComparable> keyClass, Configuration conf, boolean createInstances) {
    this.keyClass = keyClass;
    this.conf = conf != null ? conf : new Configuration();
    if (createInstances) {
        this.key1 = this.newKey();
        this.key2 = this.newKey();
        this.buffer = new DataInputBuffer();
    } else {
        this.key1 = this.key2 = null;
        this.buffer = null;
    }


}

自己开发了一个股票智能分析软件，功能很强大，需要的点击下面的链接获取：

https://www.cnblogs.com/bclshuai/p/11380657.html