MapReduce 编程 系列九 Reducer数目
Posted zhchoutai
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了MapReduce 编程 系列九 Reducer数目相关的知识，希望对你有一定的参考价值。
本篇介绍怎样控制reduce的数目。前面观察结果文件，都会发现通常是以part-r-00000 形式出现多个文件，事实上这个reducer的数目有关系。reducer数目多，结果文件数目就多。
在初始化job的时候。是能够设置reducer的数目的。example4在example的基础上做了改动。改动了pom.xml。使得结束一个參数作为reducer的数目。改动了LogJob.java的代码，作为设置reducer数目。
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.freebird</groupId>
<artifactId>mr1_example4</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>mr1_example4</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.3.2</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>hadoop</executable>
<arguments>
<argument>jar</argument>
<argument>target/mr1_example4-1.0-SNAPSHOT.jar</argument>
<argument>org.freebird.LogJob</argument>
<argument>/user/chenshu/share/logs</argument>
<argument>1</argument>
</arguments>
</configuration>
</plugin>
</plugins>
</build>
</project>
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.freebird.reducer.LogReducer;
import org.freebird.mapper.LogMapper;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.fs.FileSystem;
import java.io.IOException;
public class LogJob {
public static void main(String[] args) throws Exception {
String inputPath = args[0];
if (inputPath.endsWith("/")) {
inputPath = inputPath.substring(0, inputPath.length() -1);
}
System.out.println("args[0] indicates input folder path, the last / will be removed if it exists:" + inputPath);
String outputPath = inputPath + "/output";
System.out.println("output folder path is:" + outputPath);
int numReducer = Integer.parseInt(args[1]);
System.out.println("reducer number is: " + args[1]);
Configuration conf = new Configuration();
Job job = new Job(conf, "sum_did_from_log_file");
job.setJarByClass(LogJob.class);
job.setMapperClass(org.freebird.mapper.LogMapper.class);
job.setReducerClass(org.freebird.reducer.LogReducer.class);
job.setNumReduceTasks(numReducer);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
Path path1 = new Path(inputPath);
Path path2 = new Path(outputPath);
removeFolder(path2, conf);
MultipleOutputs.addNamedOutput(job, "result", TextOutputFormat.class, Text.class, IntWritable.class);
FileInputFormat.addInputPath(job, path1);
FileOutputFormat.setOutputPath(job, path2);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
private static void removeFolder(Path path, Configuration conf) throws IOException {
FileSystem fs = path.getFileSystem(conf);
if (fs.exists(path)) {
fs.delete(path);
}
}
}
执行结果，通过观察jobtracker。的确reducer数目为1了。
而且结果文件也变成了仅仅有一个：
[[email protected] example4]$ hdfs dfs -ls /user/chenshu/share/logs/output/ 14/10/03 14:18:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 4 items -rw-r--r-- 3 chenshu chenshu 0 2014-10-03 12:53 /user/chenshu/share/logs/output/_SUCCESS drwxr-xr-x - chenshu chenshu 0 2014-10-03 12:52 /user/chenshu/share/logs/output/_logs -rw-r--r-- 3 chenshu chenshu 0 2014-10-03 12:53 /user/chenshu/share/logs/output/part-r-00000 -rw-r--r-- 3 chenshu chenshu 4391668 2014-10-03 12:53 /user/chenshu/share/logs/output/result-r-00000
以上是关于MapReduce 编程 系列九 Reducer数目的主要内容，如果未能解决你的问题，请参考以下文章
Map Reduce编程中reducer中的shuffle和sorting阶段的目的是啥？
学习笔记Hadoop（十四）—— MapReduce开发入门—— MapReduce API介绍MapReduce实例