MapReduce单元测试

Posted 2020-09-29 代号菜鸟

MapReduce进行单元测试的步骤：

1. 在POM中添加MRUnit

<dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>1.1.0</version>
            <classifier>hadoop2</classifier>
            <scope>test</scope>
 </dependency>

2. Mapper测试用例

    @Test
    public void testCountMapper() throws IOException {
        LongWritable key = new LongWritable(0);
        Text value = new Text("hadoop yarn");
        new MapDriver<LongWritable,Text,Text,IntWritable>()
                .withMapper(new WordCountMapper())
                .withInput(key,value)
                .withOutput(new Text("hadoop"),new IntWritable(1))
                .withOutput(new Text("yarn"),new IntWritable(1))
                .runTest();
    }

 3. Reducer测试用例

    @Test
    public void testCountReducer() throws IOException {
        new ReduceDriver<Text,IntWritable,Text,IntWritable>()
                .withReducer(new WordCountReducer())
                .withInput(new Text("hadoop"), Arrays.asList(new IntWritable(1),new IntWritable(1)))
                .withOutput(new Text("hadoop"),new IntWritable(2))
                .runTest();
    }