Mapreducer任务启动

Posted 2020-12-02 zwcoding

一、Streaming

 

#!/bin/bash

hadoop fs -rm -r/data/apps/zhangwenchao/mapreduce/streaming/wordcount/output

 

hadoop jar /data/tools/hadoop/hadoop-2.6.2/share/hadoop/tools/lib/hadoop-streaming-2.6.2.jar
       -input /data/apps/zhangwenchao/mapreduce/streaming/wordcount/input 
       -output /data/apps/zhangwenchao/mapreduce/streaming/wordcount/output 
       -mapper "sh -x mapper.sh" 
       -reducer "sh -x reducer.sh" 
       -file mapper.sh 
       -file reducer.sh 
       -jobconf mapred.job.name=wordcount 
       -jobconf mapred.job.tasks=5 
       -jobconf mapred.reduce.tasks=3

 

注意：输出目录不能存在，故启动任务前，删除。mapper.sh，reducer.sh文件与start.sh文件同属于当前目录下

 

二、Python

 

#!/bin/bash

hadoop fs -rm -r /data/apps/zhangwenchao/mapreduce/python/wordcount/output

hadoop jar /data/tools/hadoop/hadoop-2.6.2/share/hadoop/tools/lib/hadoop-streaming-2.6.2.jar         -input /data/apps/zhangwenchao/mapreduce/python/wordcount/input         -output /data/apps/zhangwenchao/mapreduce/python/wordcount/output         -mapper "mapper.py"         -reducer "reducer.py"         -file mapper.py         -file reducer.py         -jobconf mapred.job.name=wordcount         -jobconf mapred.job.tasks=5         -jobconf mapred.reduce.tasks=3

 

注意：输出目录不能存在，故启动任务前，删除。mapper.sh，reducer.sh文件与start.sh文件同属于当前目录下

 

三、Java

 

hadoop jar x.jar 是运行包的一种方式，完成的方式为

 

hadoop jar  x.jar  ×××.MainClassName inputPath outputPath

 

同时解释一下：

 

x.jar为包的名称，包括路径，直接写包名称，则为默认路径

 

×××.MainClassName为运行的类名称

 

inputPath为输入路径

 

outputPath为输出路径。

 

这里需要注意的是：

 

输入路径和输出路径，一定要对应。

 

 

 

我们也可以在java代码里指定输入输出路径，这里启动mapreduce任务就可以不用指定路径了

 

如hadoop jar wordcount.jar Main

 

Main函数如下：

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class Main {
public static void main(String[] args) throws Exception {

String input = "/test/zhang*/mapreduce/java/wordcount/input";
String output = "/test/zhang*/mapreduce/java/wordcount/output";
Configuration conf = new Configuration();

Job job = new Job(conf);
job.setJobName("test");
job.setJarByClass(Main.class);

FileInputFormat.addInputPath(job, new Path(input));
FileOutputFormat.setOutputPath(job, new Path(output));

job.setMapperClass(MyMap.class);
job.setReducerClass(MyReduce.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setCombinerClass(MyReduce.class);

job.setNumReduceTasks(3);

job.waitForCompletion(true);
}
}