Eclipse调用hadoop2运行MR程序

Posted fansy1990

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Eclipse调用hadoop2运行MR程序相关的知识,希望对你有一定的参考价值。

hadoop:hadoop2.2 ,windows myeclipse环境;

Eclipse调用hadoop运行MR程序其实就是普通的java程序可以提交MR任务到集群执行而已。在Hadoop1中,只需指定jt(jobtracker)和fs(namenode)即可,一般如下:

Configuration conf = new Configuration();
conf.set("mapred.job.tracker", "192.168.128.138:9001");
conf.set("fs.default.name","192.168.128.138:9000");
上面的代码在hadoop1中运行是ok的,完全可以使用java提交任务到集群运行。但是,hadoop2却是没有了jt,新增了yarn。这个要如何使用呢?最简单的想法,同样指定其配置,试试。

Configuration conf = new YarnConfiguration();
			  conf.set("fs.defaultFS", "hdfs://node31:9000");
			  conf.set("mapreduce.framework.name", "yarn");
			  conf.set("yarn.resourcemanager.address", "node31:8032");
恩,这样配置后,可以运行,首先是下面的错误:

2014-04-03 21:20:21,568 ERROR [main] util.Shell (Shell.java:getWinUtilsPath(303)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\\bin\\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
	at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:345)
	at org.fansy.hadoop.mr.WordCount.getConf(WordCount.java:104)
	at org.fansy.hadoop.mr.WordCount.runJob(WordCount.java:84)
	at org.fansy.hadoop.mr.WordCount.main(WordCount.java:47)

这个错误不用管,这个好像是windows调用的时候就会出的错误。

然后是什么权限问题之类的,这个时候就需要去调整下权限,至少我目前是这样做的。调整的权限主要有/tmp 以及运行wordcount的输入、输出目录。命令如下: $HADOOP_HOME/bin/hadoop fs -chmod -R 777 /tmp 。

然后直到你出现了下面的错误,那么,好了,可以说你已经成功了一半了。

2014-04-03 20:32:36,596 ERROR [main] security.UserGroupInformation (UserGroupInformation.java:doAs(1494)) - PriviledgedActionException as:Administrator (auth:SIMPLE) cause:java.io.IOException: Failed to run job : Application application_1396459813671_0001 failed 2 times due to AM Container for appattempt_1396459813671_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
	at org.apache.hadoop.util.Shell.run(Shell.java:379)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)


.Failing this attempt.. Failing the application.
用上面出现的错误去google,可以得到这个网页:https://issues.apache.org/jira/browse/MAPREDUCE-5655 。 恩,对的。这个网页就是我们的solution。

我们分为1、2、3步骤吧。

1. 修改MRapps.java 、YARNRunner.java的源码,然后打包替换原来的jar包中的相应class文件,这两个jar我已经打包,可以在这里下载http://download.csdn.net/detail/fansy1990/7143547 。然后替换集群中相应的jar吧,同时需要注意替换Myeclipse中导入的包。额,说起Myeclipse中的jar包,这里还是先上幅jar包的图吧:



2. 修改mapred-default.xml ,添加:(这个只需在eclipse中导入的jar包修改即可,修改后的jar包不用上传到集群)

<property>
	<name>mapred.remote.os</name>
	<value>Linux</value>
	<description>
		Remote MapReduce framework's OS, can be either Linux or Windows
	</description>
</property>
(题外话,添加了这个属性后,按说我new一个Configuration后,我使用conf.get("mapred.remote.os")的时候应该是可以得到Linux的,但是我得到的却是null,这个就不清楚是怎么了。)

其文件在:


这时,你再运行程序,额好吧程序基本可以提交了,但是还是报错,查看log,可以看到下面的错误:

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
额,说了这么久,还是把我的wordcount程序贴出来吧:

package org.fansy.hadoop.mr;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.ClusterStatus;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class WordCount 
		private static Logger log = LoggerFactory.getLogger(WordCount.class);
	   public static class WCMapper extends  Mapper<LongWritable, Text, LongWritable, Text> 
		  
	      public void map(LongWritable key, Text value, Context cxt) throws IOException,InterruptedException 
	      // String[] values= value.toString().split("[,| ]");
	       cxt.write(key, value);
	      
	   
		
	    public static class WCReducer extends  Reducer<LongWritable, Text, LongWritable,Text> 
		    public void reduce(LongWritable key, Iterable<Text> values, Context cxt) throws IOException,InterruptedException 
		        StringBuffer buff = new StringBuffer();
			    for (Text v:values) 
			    	buff.append(v.toString()+"\\t");
		       
			    cxt.write(key, new Text(buff.toString()));
		    
	    

		public static void main(String[] args) throws Exception 
	//	  checkFS();
		  String input ="hdfs://node31:9000/input/test.dat";
		  String output="hdfs://node31:9000/output/wc003";
			runJob(input,output);
		//	runJob(args[0],args[1]);
		//  upload();
		
		
		/**
		 * test operate the hdfs
		 * @throws IOException 
		 */
		public static void checkFS() throws IOException
			Configuration conf=getConf(); 
			Path f= new Path("/user");
			FileSystem fs = FileSystem.get(f.toUri(),conf);
			
			RemoteIterator<LocatedFileStatus> paths=fs.listFiles(f, true);
			while(paths.hasNext())
				System.out.println(paths.next());
			
			
		
		
		public static void upload() throws IOException
			Configuration conf = getConf();
			Path f= new Path("d:\\\\wordcount.jar");
			FileSystem fs = FileSystem.get(f.toUri(),conf);
			fs.copyFromLocalFile(true, f, new Path("/input/wordcount.jar"));
			System.out.println("done ...");
		
		
		/**
		 *  test the job submit
		 * @throws IOException
		 * @throws InterruptedException 
		 * @throws ClassNotFoundException 
		 */
		public static void runJob(String input,String output) throws IOException, ClassNotFoundException, InterruptedException
			
			  Configuration conf=getConf(); 
			  Job job = new Job(conf,"word count");
		//	  job.setJar("hdfs://node31:9000/input/wordcount.jar");
		      job.setJobName("wordcount");
		      job.setJarByClass(WordCount.class);
		    //  job.setOutputFormatClass(SequenceFileOutputFormat.class);
		      job.setOutputKeyClass(LongWritable.class);
		      job.setOutputValueClass(Text.class);
	
		      job.setMapperClass(WCMapper.class);
			  job.setCombinerClass(WCReducer.class);
		      job.setReducerClass(WCReducer.class);
	
		      FileInputFormat.addInputPath(job, new Path(input));
			//  SequenceFileOutputFormat.setOutputPath(job, new Path(args[1]));
		      FileOutputFormat.setOutputPath(job, new Path(output));
			  System.exit(job.waitForCompletion(true)?0:1);
		
		
		private static Configuration getConf() throws IOException
			Configuration conf = new YarnConfiguration();
			  conf.set("fs.defaultFS", "hdfs://node31:9000");
			  conf.set("mapreduce.framework.name", "yarn");
			  conf.set("yarn.resourcemanager.address", "node31:8032");
		//	  conf.set("mapred.remote.os", "Linux");
			  System.out.println(conf.get("mapred.remote.os"));
		//	  JobClient client = new JobClient(conf);
		//	  ClusterStatus cluster = client.getClusterStatus();
			  return conf;
		

3. 如何修复上面的报错?按照那个链接的solution,需要修改mapred-default.xml 和yarn-default.xml ,其中mapred-default.xml刚才已经修改过了,这次再次修改,添加:

<property>
	<name>mapreduce.application.classpath</name>
	<value>
		$HADOOP_CONF_DIR,
		$HADOOP_COMMON_HOME/share/hadoop/common/*,
		$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
		$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
		$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
		$HADOOP_YARN_HOME/share/hadoop/yarn/*,
		$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
	</value>
</property>
对于yarn-default.xml也是同样的修改,其在hadoop-yarn-common-2.2.0.jar包中,修改内容如下:

<property>
	<name>mapreduce.application.classpath</name>
	<value>
		$HADOOP_CONF_DIR,
		$HADOOP_COMMON_HOME/share/hadoop/common/*,
		$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
		$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
		$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
		$HADOOP_YARN_HOME/share/hadoop/yarn/*,
		$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
	</value>
  </property>

同样的,上面两个jar包只用替换myeclipse中的jar包即可,不需要替换集群中的。

4. 经过上面的替换,然后再次运行,出现下面的错误:

Caused by: java.lang.ClassNotFoundException: Class org.fansy.hadoop.mr.WordCount$WCMapper not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
	... 8 more
额,好吧,我应该不用多少了,这样的错误,应该已经说明我们的myeclipse可以提交任务到hadoop2了,并且可以运行了。好吧最后一步,上传我们打包的wordcount程序的jar文件到$HADOOP_HOME/share/hadoop/mapreduce/lib下面,然后再次运行。(这里上传后不用重启集群)呵呵,最后得到下面的结果:

2014-04-03 21:17:34,289 ERROR [main] util.Shell (Shell.java:getWinUtilsPath(303)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\\bin\\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
	at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:345)
	at org.fansy.hadoop.mr.WordCount.getConf(WordCount.java:104)
	at org.fansy.hadoop.mr.WordCount.runJob(WordCount.java:84)
	at org.fansy.hadoop.mr.WordCount.main(WordCount.java:47)
Linux
2014-04-03 21:18:19,853 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-03 21:18:20,499 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(56)) - Connecting to ResourceManager at node31/192.168.0.31:8032
2014-04-03 21:18:20,973 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(149)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2014-04-03 21:18:21,020 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(287)) - Total input paths to process : 1
2014-04-03 21:18:21,313 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(394)) - number of splits:1
2014-04-03 21:18:21,336 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - user.name is deprecated. Instead, use mapreduce.job.user.name
2014-04-03 21:18:21,337 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-04-03 21:18:21,337 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-04-03 21:18:21,338 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
2014-04-03 21:18:21,338 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
2014-04-03 21:18:21,339 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
2014-04-03 21:18:21,339 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.name is deprecated. Instead, use mapreduce.job.name
2014-04-03 21:18:21,339 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
2014-04-03 21:18:21,340 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-04-03 21:18:21,340 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-04-03 21:18:21,342 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-04-03 21:18:21,343 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-04-03 21:18:21,343 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-04-03 21:18:21,513 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(477)) - Submitting tokens for job: job_1396463733942_0003
2014-04-03 21:18:21,817 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(174)) - Submitted application application_1396463733942_0003 to ResourceManager at node31/192.168.0.31:8032
2014-04-03 21:18:21,859 INFO  [main] mapreduce.Job (Job.java:submit(1272)) - The url to track the job: http://node31:8088/proxy/application_1396463733942_0003/
2014-04-03 21:18:21,860 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1317)) - Running job: job_1396463733942_0003
2014-04-03 21:18:31,307 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1338)) - Job job_1396463733942_0003 running in uber mode : false
2014-04-03 21:18:31,311 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 0% reduce 0%
2014-04-03 21:19:02,346 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 0%
2014-04-03 21:19:11,416 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1345)) -  map 100% reduce 100%
2014-04-03 21:19:11,425 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1356)) - Job job_1396463733942_0003 completed successfully
2014-04-03 21:19:11,552 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1363)) - Counters: 43
	File System Counters
		FILE: Number of bytes read=11139
		FILE: Number of bytes written=182249
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=8646
		HDFS: Number of bytes written=10161
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=29330
		Total time spent by all reduces in occupied slots (ms)=5825
	Map-Reduce Framework
		Map input records=235
		Map output records=235
		Map output bytes=10428
		Map output materialized bytes=11139
		Input split bytes=98
		Combine input records=235
		Combine output records=235
		Reduce input groups=235
		Reduce shuffle bytes=11139
		Reduce input records=235
		Reduce output records=235
		Spilled Records=470
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=124
		CPU time spent (ms)=21920
		Physical memory (bytes) snapshot=299376640
		Virtual memory (bytes) snapshot=1671372800
		Total committed heap usage (bytes)=152834048
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=8548
	File Output Format Counters 
		Bytes Written=10161

上面你看到Linux,是因为我使用了conf.set("mapred.remote.os", "Linux"); 不过在实际运行的时候却不需要设置。

另外,如果是linux系统部署的tomcat调用hadoop2集群运行MR程序的话,应该不需要替换其jar吧的,这个还有待验证。

哈,总算搞定了。这个问题也算是困扰了我好久了,期间几次想要冲破,结果都是无果而归,甚是郁闷。额,其实这个也不算是原创了,哎,国外在02/Dec/13 18:35这个时间点就搞定了。不过,我搜了好久,都没有中文的相关介绍。(如果有的话,那就是我搜索能力的问题了,居然没有搜到,哎)。


分享,成长,快乐

转载请注明blog地址:http://blog.csdn.net/fansy1990



以上是关于Eclipse调用hadoop2运行MR程序的主要内容,如果未能解决你的问题,请参考以下文章

使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS

Windows下Eclipse提交MR程序到HadoopCluster

在eclipse上开发hadoop2.5.2程序的快捷方法

Hive性能调优

Eclipse环境搭建并且运行wordcount程序

hadoop2.5.2学习13-MR之新浪微博-DF的实现