Hadoop——利用MapReduce求平均成绩

Posted 2022-12-08 mmい

环境配置

在linux中使用Eclipse编译运行MapReduce程序请参考这篇文章

数据集

输入数据集每一行是学生的名字和学生的成绩（一个学生可有多门学科成绩）；
在本地生成file1.txt file2.txt file3.txt三个文件并且上传到hdfs上的averageScore文件夹；

源代码

    public static class Map extends Mapper<LongWritable,Text,Text,IntWritable>
        public void map(LongWritable key,Text value,Context context)throws IOException,InterruptedException
            String line = value.toString(); //change text to String
            StringTokenizer tokenizerArticle = new StringTokenizer(line,"\\n"); //cut input  to lines
            while(tokenizerArticle.hasMoreElements())
                StringTokenizer tokenizerLine = new StringTokenizer(tokenizerArticle.nextToken());
                String strName = tokenizerLine.nextToken(); //name
                String strScore = tokenizerLine.nextToken(); //score
                Text name = new Text(strName);
                System.out.println(strScore);
                System.out.println(name);
                int scoreInt = Integer.parseInt(strScore);
                context.write(name, new IntWritable(scoreInt));
            
        
    
    public static class Reduce extends Reducer<Text,IntWritable,Text,IntWritable>
        public void reduce(Text key,Iterable<IntWritable> values,Context context)throws IOException,InterruptedException
            int sum = 0;
            int count = 0;
            Iterator<IntWritable> iterator = values.iterator();
            while(iterator.hasNext())
                sum += iterator.next().get();
                count++;
            
            int averageScore = (int)sum/count;
            System.out.println(key);
            System.out.println(sum);
            context.write(key, new IntWritable(averageScore));
        
       

结果

使用命令行模式编译运行hadoop程序