java程序中的编译错误以使用hadoop计算文件中的单词

Posted 2023-02-19

【中文标题】java程序中的编译错误以使用hadoop计算文件中的单词

【英文标题】：Compile error in java program to count words in file using hadoop

【发布时间】：2017-10-28 05:30:45

【问题描述】：

我的 Java 版本：

➜  test git:(dev) ✗ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

我正在尝试运行以下简单的 java 程序，该程序使用来自 here 的 hadoop map reduce 提供文件的字数。以下是整个java代码：

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;

public class WordCount extends Configured implements Tool 
   private final static LongWritable ONE = new LongWritable(1L);

// Mapper Class, Counts words in each line. For each line, break the line into words and emits them as (word, 1)

public static class MapClass extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> 
  private final static IntWritable one = new IntWritable(1);
 private Text word = new Text();

public void map(LongWritable key, Text value,
   OutputCollector<text, intwritable> output,
   Reporter reporter) throws IOException 

  String line = value.toString();
  StringTokenizer itr = new StringTokenizer(line);
  while (itr.hasMoreTokens()) 
    word.set(itr.nextToken());
     output.collect(word, one);
   
 


// Reducer class that just emits the sum of the input values.

public static class Reduce extends MapReduceBase implements Reducer< Text, IntWritable, Text, IntWritable > 
public void reduce(Text key, Iterator values,
 OutputCollector<text, intwritable=""> output,
 Reporter reporter) throws IOException 
      int sum = 0;
      while (values.hasNext()) 
        sum += values.next().get();
      
      output.collect(key, new IntWritable(sum));
    
  

static int printUsage() 
System.out.println("wordcount [-m #mappers ] [-r #reducers] input_file output_file");
    ToolRunner.printGenericCommandUsage(System.out);
    return -1;
  

public int run(String[] args) throws Exception 

    JobConf conf = new JobConf(getConf(), WordCount.class);
    conf.setJobName("wordcount");

// the keys are words (strings)
   conf.setOutputKeyClass(Text.class);
// the values are counts (ints)
   conf.setOutputValueClass(IntWritable.class);

   conf.setMapperClass(MapClass.class);
// Here we set the combiner!!!!
   conf.setCombinerClass(Reduce.class);
   conf.setReducerClass(Reduce.class);

  List other_args = new ArrayList();
   for(int i=0; i < args.length; ++i) 
     try 
        if ("-m".equals(args[i])) 
conf.setNumMapTasks(Integer.parseInt(args[++i]));
         else if ("-r".equals(args[i])) 
conf.setNumReduceTasks(Integer.parseInt(args[++i]));
         else 
          other_args.add(args[i]);
        
       catch (NumberFormatException except) 
        System.out.println("ERROR: Integer expected instead of " + args[i]);
        return printUsage();
       catch (ArrayIndexOutOfBoundsException except) 
        System.out.println("ERROR: Required parameter missing from " +
            args[i-1]);
        return printUsage();
      
    
// Make sure there are exactly 2 parameters left.
   if (other_args.size() != 2) 
      System.out.println("ERROR: Wrong number of parameters: " +
          other_args.size() + " instead of 2.");
      return printUsage();
    
    FileInputFormat.setInputPaths(conf, other_args.get(0));
    FileOutputFormat.setOutputPath(conf, new Path(other_args.get(1)));

    JobClient.runJob(conf);
    return 0;
  

public static void main(String[] args) throws Exception 
    int res = ToolRunner.run(new Configuration(), new WordCount(), args);
    System.exit(res);
  

但在上述代码的下面部分：

// Reducer class that just emits the sum of the input values.

public static class Reduce extends MapReduceBase implements Reducer< Text, IntWritable, Text, IntWritable > 
public void reduce(Text key, Iterator values,
 OutputCollector<text, intwritable=""> output,
 Reporter reporter) throws IOException 
      int sum = 0;
      while (values.hasNext()) 
        sum += values.next().get();
      
      output.collect(key, new IntWritable(sum));
    
  

我收到以下编译错误：

➜ 测试 git:(dev) ✗ javac WordCount.java WordCount.java:43：错误：> 预期 OutputCollector 输出， ^

WordCount.java:43: 错误: ')' 预期 OutputCollector 输出， ^

WordCount.java:43: 错误：';'预期的 OutputCollector 输出， ^

WordCount.java:43：错误：预期 OutputCollector 输出， ^

WordCount.java:44: 错误：';'预期的 记者记者）抛出 IOException ^

WordCount.java:44：错误：预期 记者记者）抛出 IOException ^ WordCount.java:44：错误：类型的非法开始 记者记者）抛出 IOException ^

WordCount.java:44: 错误：';'预期的 记者记者）抛出 IOException ^

8 个错误

我确信这是一些非常基本的错误，但无法弄清楚。

【参考方案1】：

我不确定这是什么，但它并不完全是 Java。例如：

OutputCollector<text, intwritable="">

text 和 intwrtiable 不是类型。那应该是Text 和IntWritable。而默认分配对我来说完全是个谜。 Java 不支持默认泛型，即使支持，我也不确定在这种情况下空白字符串意味着什么。

【讨论】：

你在写，同样让我感到困惑。