在windows 使用eclipse 跨平台运行mapreduce 提示连接不上

Posted 2020-10-27 geek-sq

错误提示：call from 10.32.6.150：8020 failed on connection 

 如果你遇到这个问题了，在网络正常的情况下，更改你的代码，如下：

 

 1 System.setProperty("HADOOP_USER_NAME", "root");        
 2         Configuration conf = new Configuration();  
 3         conf.set("fs.defaultFS", "hdfs://10.32.6.150:9000");
 4         conf.set("mapreduce.jobtracker.address", "hdfs://10.32.6.150:9001");
 5         conf.set("mapreduce.framework.name", "yarn");
 6         conf.set("mapreduce.app-submission.cross-platform", "true");//为了解决跨平台的问题
 7         conf.set("fs.defaultFS", "hdfs://10.32.6.150:9000");
 8         conf.set("mapreduce.jobtracker.address", "hdfs://10.32.6.150:9001");
 9         conf.set("mapreduce.framework.name", "yarn");
10         conf.set("mapreduce.app-submission.cross-platform","true");
11         Job wordCountJob = Job.getInstance(conf,"mr");  
12         //重要：指定本job所在的jar包  
13         wordCountJob.setJar("E:\\Desktop\\WordCount1.jar");
14         wordCountJob.setJarByClass(WordCount.class);  
15           
16         //设置wordCountJob所用的mapper逻辑类为哪个类  
17         wordCountJob.setMapperClass(WordCountMapper.class);  
18         //设置wordCountJob所用的reducer逻辑类为哪个类  
19         wordCountJob.setReducerClass(WordCountReducer.class);  
20           
21         //设置map阶段输出的kv数据类型  
22         wordCountJob.setMapOutputKeyClass(Text.class);  
23         wordCountJob.setMapOutputValueClass(IntWritable.class);  
24           
25         //设置最终输出的kv数据类型  
26         wordCountJob.setOutputKeyClass(Text.class);  
27         wordCountJob.setOutputValueClass(IntWritable.class);  
28           
29         //设置要处理的文本数据所存放的路径  
30         FileInputFormat.setInputPaths(wordCountJob, "hdfs://10.32.6.150:9000/1.txt");  
31         FileOutputFormat.setOutputPath(wordCountJob, new Path("hdfs://10.32.6.150/outputsq111/"));  
32           
33         //提交job给hadoop集群  
34         wordCountJob.waitForCompletion(true);  

 1 System.setProperty("HADOOP_USER_NAME", "root");
 2         Configuration conf = new Configuration();
 3         conf.set("fs.defaultFS", "hdfs://10.32.6.150:9000");
 4         conf.set("mapreduce.jobtracker.address", "hdfs://10.32.6.150:9001");
 5         conf.set("mapreduce.framework.name", "yarn");
 6         conf.set("mapreduce.app-submission.cross-platform","true");
 7         //创建工作对象
 8         Job job = Job.getInstance(conf, "wc");
 9         //添加架包路径
10         job.setJar("E:\\Desktop\\WordCount.jar");
11         job.setJarByClass(WordCountOther.class);
12         
13         //指定map类和reduce类
14         job.setMapperClass(WordCountMapper.class);
15         job.setReducerClass(WordCountReducer.class);
16         
17         //指定map的输出类型
18         job.setMapOutputKeyClass(Text.class);
19         job.setMapOutputValueClass(IntWritable.class);
20         
21         //指定最终的输出类型
22         job.setOutputKeyClass(Text.class);
23         job.setOutputValueClass(IntWritable.class);
24         
25         //指定job的输入输出目录
26         FileInputFormat.setInputPaths(job, new Path("hdfs://10.32.6.150:9000/1.txt"));
27         FileOutputFormat.setOutputPath(job, new Path("hdfs://10.32.6.150:9000/output321/"));
28         
29         boolean result = job.waitForCompletion(true);
30         System.exit(result?0:1);

上边的是报错的代码，下边是可以运行的代码，两者看不出有什么区别，我反复以为网络的问题（因为是内网，觉得会有防火墙之类的），后来执行下同学写的，运行成功

解决方案：这个问题找不到原因，不是网络问题，应该也不是eclipse的问题，遇到了就重新写份job代码：）