MapReduce--2--MapReduce全局计数器

Posted 2023-01-17 中琦2513

MapReduce的全局计数器

1.1、介绍

计数器是用来记录job的执行进度和状态的。它的作用可以理解为日志。我们可以在程序的某个位置插入计数器，记录数据或者进度的变化情况。

 

MapReduce 计数器（Counter）为我们提供一个窗口，用于观察 MapReduce Job 运行期的各种细节数据。对MapReduce性能调优很有帮助，MapReduce性能优化的评估大部分都是基于这些 Counter 的数值表现出来的。

 

MapReduce 自带了许多默认Counter，现在我们来分析这些默认Counter 的含义，方便大家观察 Job 结果，如输入的字节数、输出的字节数、Map端输入/输出的字节数和条数、Reduce端的输入/输出的字节数和条数等

1.2、需求

在实际生产代码中，常常需要将数据处理过程中遇到的不合规数据行进行全局计数，类似这种需求可以借助MapReduce框架中提供的全局计数器来实现

1.3、场景介绍

现在我们通过全局计数器来实现，统计一个目录下的所有文件或者一个文件中的总行数和总单词数

1.4、代码实现

package com.ghgj.mazh.mapreduce.counter;

import java.io.IOException;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Counter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordAndLineCounter 

	/**
	 * mapreduce全局计数器,是用枚举类进行计数器的定义
	 * COUNT_LINES用来统计总行数， 
	 * COUNT_WORDS用来统计总单词数
	 */
	enum MyCouterWordCount 
		COUNT_WORDS, COUNT_LINES
	

	// 驱动程序
	public static void main(String[] args) throws Exception 

		// 指定hdfs相关的参数
		Configuration conf = new Configuration();
		Job job = Job.getInstance(conf);
		// 设置jar包所在路径
		job.setJarByClass(WordAndLineCounter.class);

		job.setMapperClass(WordAndLineCounterMapper.class);

		job.setMapOutputKeyClass(Text.class);
		job.setMapOutputValueClass(LongWritable.class);

		// 本地路径
		Path inputPath = new Path("D:\\\\bigdata\\\\wordcount\\\\input");
		FileInputFormat.setInputPaths(job, inputPath);

		Path outputPath = new Path("D:\\\\bigdata\\\\wordcount\\\\output");
		FileSystem fs = FileSystem.get(conf);
		if (fs.exists(outputPath)) 
			fs.delete(outputPath, true);
		
		FileOutputFormat.setOutputPath(job, outputPath);

		// 最后提交任务
		boolean waitForCompletion = job.waitForCompletion(true);
		System.exit(waitForCompletion ? 0 : 1);
	

	private static class WordAndLineCounterMapper extends Mapper<LongWritable, Text, Text, LongWritable> 
		@Override
		protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException 

			// 获取计数器，行数计数器
			Counter counter = context.getCounter(MyCouterWordCount.COUNT_LINES);
			// 计数器计数
			counter.increment(1L);

			String[] words = value.toString().split(" ");
			for (String word : words) 
				if(!StringUtils.isBlank(word))
					context.getCounter(MyCouterWordCount.COUNT_WORDS).increment(1L);
				
			
			
		
	

1.5、输入数据

hello huangbo
hello xuzheng
hello wangbaoqiang
one two three four five
one two three four
one two three
one two
hello hi

1.6、计算结果

 INFO [main] - Job job_local1030848363_0001 completed successfully
 INFO [main] - Counters: 35
	File System Counters
		FILE: Number of bytes read=614
		FILE: Number of bytes written=506970
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
	Map-Reduce Framework
		Map input records=8
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=104
		Combine input records=0
		Combine output records=0
		Reduce input groups=0
		Reduce shuffle bytes=6
		Reduce input records=0
		Reduce output records=0
		Spilled Records=0
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
		Total committed heap usage (bytes)=514850816
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	com.ghgj.mazh.mapreduce.counter.WordAndLineCounter$MyCouterWordCount
		COUNT_LINES=8
		COUNT_WORDS=22
	File Input Format Counters 
		Bytes Read=127
	File Output Format Counters 
		Bytes Written=8

这就是最后的计算结果：在上面结果中的倒数第5-7行

com.ghgj.mazh.mapreduce.counter.WordAndLineCounter$MyCouterWordCount
		COUNT_LINES=8
		COUNT_WORDS=22

各位看官，有没有看懂的，如果有疑问的地方，可以举手提问。