MapReduce 编程 系列九 Reducer数目

Posted zhchoutai

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了MapReduce 编程 系列九 Reducer数目相关的知识,希望对你有一定的参考价值。

本篇介绍怎样控制reduce的数目。前面观察结果文件,都会发现通常是以part-r-00000 形式出现多个文件,事实上这个reducer的数目有关系。reducer数目多,结果文件数目就多。

在初始化job的时候。是能够设置reducer的数目的。example4在example的基础上做了改动。改动了pom.xml。使得结束一个參数作为reducer的数目。改动了LogJob.java的代码,作为设置reducer数目。

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.freebird</groupId>
  <artifactId>mr1_example4</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>mr1_example4</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-core</artifactId>
      <version>1.2.1</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.3.2</version>
        <executions>
          <execution>
            <goals>
              <goal>exec</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <executable>hadoop</executable>
          <arguments>
            <argument>jar</argument>
            <argument>target/mr1_example4-1.0-SNAPSHOT.jar</argument>
            <argument>org.freebird.LogJob</argument>
            <argument>/user/chenshu/share/logs</argument>
            <argument>1</argument>
          </arguments>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>


LogJob.java代码:
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.freebird.reducer.LogReducer;
import org.freebird.mapper.LogMapper;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.fs.FileSystem;
import java.io.IOException;



public class LogJob {                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                  
    public static void main(String[] args) throws Exception {                                                                                                                                                                                                                     
        String inputPath = args[0];                                                                                                                                                                                                                                               
        if (inputPath.endsWith("/")) {                                                                                                                                                                                                                                            
            inputPath = inputPath.substring(0, inputPath.length() -1);                                                                                                                                                                                                            
        }                                                                                                                                                                                                                                                                         
        System.out.println("args[0] indicates input folder path, the last / will be removed if it exists:" + inputPath);                                                                                                                                                          
        String outputPath = inputPath + "/output";                                                                                                                                                                                                                                
        System.out.println("output folder path is:" + outputPath);                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                  
        int numReducer = Integer.parseInt(args[1]);                                                                                                                                                                                                                               
        System.out.println("reducer number is: " + args[1]);                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                  
        Configuration conf = new Configuration();                                                                                                                                                                                                                                 
        Job job = new Job(conf, "sum_did_from_log_file");                                                                                                                                                                                                                         
        job.setJarByClass(LogJob.class);                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                  
        job.setMapperClass(org.freebird.mapper.LogMapper.class);                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                  
        job.setReducerClass(org.freebird.reducer.LogReducer.class);                                                                                                                                                                                                               
        job.setNumReduceTasks(numReducer);                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                  
        job.setOutputKeyClass(Text.class);                                                                                                                                                                                                                                        
        job.setOutputValueClass(IntWritable.class);                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                  
        Path path1 = new Path(inputPath);                                                                                                                                                                                                                                         
        Path path2 = new Path(outputPath);                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                  
        removeFolder(path2, conf);                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                  
        MultipleOutputs.addNamedOutput(job, "result", TextOutputFormat.class, Text.class, IntWritable.class);                                                                                                                                                                     
                                                                                                                                                                                                                                                                                  
        FileInputFormat.addInputPath(job, path1);                                                                                                                                                                                                                                 
        FileOutputFormat.setOutputPath(job, path2);                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                  
        System.exit(job.waitForCompletion(true) ? 0 : 1);                                                                                                                                                                                                                         
    }                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                  
    private static void removeFolder(Path path, Configuration conf) throws IOException {                                                                                                                                                                                          
        FileSystem fs = path.getFileSystem(conf);                                                                                                                                                                                                                                 
        if (fs.exists(path)) {                                                                                                                                                                                                                                                    
            fs.delete(path);                                                                                                                                                                                                                                                      
        }                                                                                                                                                                                                                                                                         
    }                                                                                                                                                                                                                                                                             
}

执行结果,通过观察jobtracker。的确reducer数目为1了。

而且结果文件也变成了仅仅有一个:

[[email protected] example4]$ hdfs dfs -ls /user/chenshu/share/logs/output/
14/10/03 14:18:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
-rw-r--r--   3 chenshu chenshu          0 2014-10-03 12:53 /user/chenshu/share/logs/output/_SUCCESS
drwxr-xr-x   - chenshu chenshu          0 2014-10-03 12:52 /user/chenshu/share/logs/output/_logs
-rw-r--r--   3 chenshu chenshu          0 2014-10-03 12:53 /user/chenshu/share/logs/output/part-r-00000
-rw-r--r--   3 chenshu chenshu    4391668 2014-10-03 12:53 /user/chenshu/share/logs/output/result-r-00000



以上是关于MapReduce 编程 系列九 Reducer数目的主要内容,如果未能解决你的问题,请参考以下文章

Map Reduce编程中reducer中的shuffle和sorting阶段的目的是啥?

学习笔记Hadoop(十四)—— MapReduce开发入门—— MapReduce API介绍MapReduce实例

MapReduce 任务数

MapReduce编程模型

大数据之Hadoop(MapReduce):MapReduce编程规范

MAPREDUCE实践篇