大数据-Hadoop生态(19)-MapReduce框架原理-Combiner合并

Posted 2021-01-26 duoduotouhenying

1. Combiner概述

 

 2. 自定义Combiner实现步骤

1). 定义一个Combiner继承Reducer,重写reduce方法

public class WordcountCombiner extends Reducer<Text, IntWritable, Text,IntWritable>{

    @Override
    protected void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {

        // 1 汇总操作
        int count = 0;
        for(IntWritable v :values){
            count += v.get();
        }

        // 2 写出
        context.write(key, new IntWritable(count));
    }
}

2). 在Driver类中添加设置

job.setCombinerClass(WordcountCombiner.class);

 

效果