Hadoop 程序在“java -jar”下运行良好，但在“hadoop jar”下运行良好

Posted 2023-04-18

【中文标题】Hadoop 程序在“java -jar”下运行良好，但在“hadoop jar”下运行良好

【英文标题】：Hadoop program runs well with "java -jar" but not with "hadoop jar"

【发布时间】：2014-09-10 15:32:27

【问题描述】：

我编写了一个连接到两个 HBase 数据库的 MapReduce 程序。我已经在 Eclipse 上编写了它，并使用“Runnable Jar”选项（以及所有库）导出了它。它与java -jar 命令一起运行良好。输出如下：

$ java -jar mr.jar
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

然后是预期的输出。但是，如果我使用hadoop jar 命令运行相同的程序，我不知道究竟会发生什么，但我猜不出什么。没有出现预期的输出。

$ hadoop jar mr.jar
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
14/09/10 17:22:07 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/09/10 17:22:07 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
14/09/10 17:22:07 INFO Configuration.deprecation: dfs.df.interval is deprecated. Instead, use fs.df.interval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.tracker.http.address is deprecated. Instead, use mapreduce.tasktracker.http.address
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.userlog.retain.hours is deprecated. Instead, use mapreduce.job.userlog.retain.hours
14/09/10 17:22:07 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.local.dir.minspacestart is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacestart
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.shuffle.read.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.read.timeout
14/09/10 17:22:07 INFO Configuration.deprecation: io.sort.spill.percent is deprecated. Instead, use mapreduce.map.sort.spill.percent
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.parallel.copies is deprecated. Instead, use mapreduce.reduce.shuffle.parallelcopies
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.local.dir.minspacekill is deprecated. Instead, use mapreduce.tasktracker.local.dir.minspacekill
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.profile is deprecated. Instead, use mapreduce.task.profile
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.heartbeats.in.second is deprecated. Instead, use mapreduce.jobtracker.heartbeats.in.second
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.healthChecker.interval is deprecated. Instead, use mapreduce.tasktracker.healthchecker.interval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir
14/09/10 17:22:07 INFO Configuration.deprecation: jobclient.completion.poll.interval is deprecated. Instead, use mapreduce.client.completion.pollinterval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.active
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.shuffle.merge.percent is deprecated. Instead, use mapreduce.reduce.shuffle.merge.percent
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.reduce.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.input.buffer.percent
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.cache.levels is deprecated. Instead, use mapreduce.jobtracker.taskcache.levels
14/09/10 17:22:07 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jobtracker.instrumentation is deprecated. Instead, use mapreduce.jobtracker.instrumentation
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.userlog.limit.kb is deprecated. Instead, use mapreduce.task.userlog.limit.kb
14/09/10 17:22:07 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use mapreduce.job.speculative.slownodethreshold
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.skip.map.max.skip.records is deprecated. Instead, use mapreduce.map.skip.maxrecords
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.lru.cache.size
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.hours
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.handler.count is deprecated. Instead, use mapreduce.jobtracker.handler.count
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
14/09/10 17:22:07 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.profile.maps is deprecated. Instead, use mapreduce.task.profile.maps
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.child.java.opts is deprecated. Instead, use mapreduce.reduce.java.opts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.dns.nameserver is deprecated. Instead, use mapreduce.tasktracker.dns.nameserver
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated. Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.expiry.interval is deprecated. Instead, use mapreduce.jobtracker.expire.trackers.interval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.max.tracker.failures is deprecated. Instead, use mapreduce.job.maxtaskfailures.per.tracker
14/09/10 17:22:07 INFO Configuration.deprecation: mapreduce.jobtracker.split.metainfo.maxsize is deprecated. Instead, use mapreduce.job.split.metainfo.maxsize
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use mapreduce.jobtracker.persist.jobstatus.dir
14/09/10 17:22:07 INFO Configuration.deprecation: job.end.retry.attempts is deprecated. Instead, use mapreduce.job.end-notification.retry.attempts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/09/10 17:22:07 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.tracker.task-controller is deprecated. Instead, use mapreduce.tasktracker.taskcontroller
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use mapreduce.jobtracker.maxtasks.perjob
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.max.attempts is deprecated. Instead, use mapreduce.reduce.maxattempts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.output.compression.codec is deprecated. Instead, use mapreduce.map.output.compress.codec
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use mapreduce.reduce.shuffle.input.buffer.percent
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.tracker.report.address is deprecated. Instead, use mapreduce.tasktracker.report.address
14/09/10 17:22:07 INFO Configuration.deprecation: keep.failed.task.files is deprecated. Instead, use mapreduce.task.files.preserve.failedtasks
14/09/10 17:22:07 INFO Configuration.deprecation: tasktracker.http.threads is deprecated. Instead, use mapreduce.tasktracker.http.threads
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use mapreduce.job.speculative.slowtaskthreshold
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.acls.enabled is deprecated. Instead, use mapreduce.cluster.acls.enabled
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.child.java.opts is deprecated. Instead, use mapreduce.map.java.opts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.max.tracker.blacklists is deprecated. Instead, use mapreduce.jobtracker.tasktracker.maxblacklists
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.indexcache.mb is deprecated. Instead, use mapreduce.tasktracker.indexcache.mb
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.skip.attempts.to.start.skipping is deprecated. Instead, use mapreduce.task.skip.start.attempts
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.reduce.tasks.maximum
14/09/10 17:22:07 INFO Configuration.deprecation: jobclient.output.filter is deprecated. Instead, use mapreduce.client.output.filter
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jobtracker.restart.recover is deprecated. Instead, use mapreduce.jobtracker.restart.recover
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.speculative.execution.speculativeCap is deprecated. Instead, use mapreduce.job.speculative.speculativecap
14/09/10 17:22:07 INFO Configuration.deprecation: jobclient.progress.monitor.poll.interval is deprecated. Instead, use mapreduce.client.progressmonitor.pollinterval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use mapreduce.jobtracker.retiredjobs.cache.size
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.dns.interface is deprecated. Instead, use mapreduce.tasktracker.dns.interface
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.task.profile.reduces is deprecated. Instead, use mapreduce.task.profile.reduces
14/09/10 17:22:07 INFO Configuration.deprecation: job.end.retry.interval is deprecated. Instead, use mapreduce.job.end-notification.retry.interval
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jobtracker.job.history.block.size is deprecated. Instead, use mapreduce.jobtracker.jobhistory.block.size
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.child.tmp is deprecated. Instead, use mapreduce.task.tmp.dir
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use mapreduce.tasktracker.map.tasks.maximum
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.jobtracker.taskScheduler is deprecated. Instead, use mapreduce.jobtracker.taskscheduler
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.skip.reduce.max.skip.groups is deprecated. Instead, use mapreduce.reduce.skip.maxgroups
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.instrumentation is deprecated. Instead, use mapreduce.tasktracker.instrumentation
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use mapreduce.job.jvm.numtasks
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.inmem.merge.threshold is deprecated. Instead, use mapreduce.reduce.merge.inmem.threshold
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb
14/09/10 17:22:07 INFO Configuration.deprecation: topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.reduce.slowstart.completed.maps is deprecated. Instead, use mapreduce.job.reduce.slowstart.completedmaps
14/09/10 17:22:07 INFO Configuration.deprecation: dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode
14/09/10 17:22:07 INFO Configuration.deprecation: topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.merge.recordsBeforeProgress is deprecated. Instead, use mapreduce.task.merge.progress.records
14/09/10 17:22:07 INFO Configuration.deprecation: mapred.shuffle.connect.timeout is deprecated. Instead, use mapreduce.reduce.shuffle.connect.timeout
14/09/10 17:22:08 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/09/10 17:22:08 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max
14/09/10 17:22:08 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
14/09/10 17:22:08 INFO Configuration.deprecation: fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.block.size is deprecated. Instead, use dfs.blocksize
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
14/09/10 17:22:08 INFO Configuration.deprecation: fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
14/09/10 17:22:08 INFO Configuration.deprecation: dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
14/09/10 17:22:08 INFO Configuration.deprecation: fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
14/09/10 17:22:08 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.1:8032
14/09/10 17:22:08 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/09/10 17:22:08 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max
14/09/10 17:22:08 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/09/10 17:22:08 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
14/09/10 17:22:08 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/09/10 17:22:08 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/09/10 17:22:08 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:host.name=xxx
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_55
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.class.path=[...]
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/hadoop/hadoop-2.2.0/lib/native
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.17.1.el6.x86_64
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:user.name=xxx
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/xxx
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/xxx/programas
14/09/10 17:22:08 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x18dddc42, quorum=localhost:2181, baseZNode=/hbase
14/09/10 17:22:08 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x18dddc42 connecting to ZooKeeper ensemble=localhost:2181
14/09/10 17:22:08 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
14/09/10 17:22:08 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
14/09/10 17:22:08 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14859c8510b004c, negotiated timeout = 90000
14/09/10 17:22:09 INFO mapreduce.TableOutputFormat: Created table instance for s
14/09/10 17:22:09 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/09/10 17:22:11 INFO util.RegionSizeCalculator: Calculating region sizes for table "e".
14/09/10 17:22:11 WARN mapreduce.TableInputFormatBase: Cannot resolve the host name for hadoop2/192.168.0.2 because of javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '2.0.168.192.in-addr.arpa'
14/09/10 17:22:12 INFO mapreduce.JobSubmitter: number of splits:1
14/09/10 17:22:12 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
14/09/10 17:22:12 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max
14/09/10 17:22:12 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
14/09/10 17:22:12 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
14/09/10 17:22:12 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/09/10 17:22:12 INFO Configuration.deprecation: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
14/09/10 17:22:12 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/09/10 17:22:12 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
14/09/10 17:22:12 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/09/10 17:22:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1405689757338_0555
14/09/10 18:26:53 INFO impl.YarnClientImpl: Submitted application application_1405689757338_0560 to ResourceManager at /192.168.0.1:8032
14/09/10 18:26:53 INFO mapreduce.Job: The url to track the job: http://xxx.xxx.xxx:8088/proxy/application_1405689757338_0560/
14/09/10 18:26:53 INFO mapreduce.Job: Running job: job_1405689757338_0560
14/09/10 18:27:18 INFO mapreduce.Job: Job job_1405689757338_0560 running in uber mode : false
14/09/10 18:27:18 INFO mapreduce.Job:  map 0% reduce 0%
14/09/10 18:37:46 INFO mapreduce.Job: Task Id : attempt_1405689757338_0560_m_000000_0, Status : FAILED
AttemptID:attempt_1405689757338_0560_m_000000_0 Timed out after 600 secs
14/09/10 18:48:17 INFO mapreduce.Job: Task Id : attempt_1405689757338_0560_m_000000_1, Status : FAILED
AttemptID:attempt_1405689757338_0560_m_000000_1 Timed out after 600 secs
14/09/10 18:58:47 INFO mapreduce.Job: Task Id : attempt_1405689757338_0560_m_000000_2, Status : FAILED
AttemptID:attempt_1405689757338_0560_m_000000_2 Timed out after 600 secs
14/09/10 19:09:18 INFO mapreduce.Job:  map 100% reduce 100%
14/09/10 19:09:19 INFO mapreduce.Job: Job job_1405689757338_0560 failed with state FAILED due to: Task failed task_1405689757338_0560_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/09/10 19:09:19 INFO mapreduce.Job: Counters: 9
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Rack-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=25139550
        Total time spent by all reduces in occupied slots (ms)=0
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
Exception in thread "main" java.io.IOException: JOB ERROR.
    at Principal.main(Principal.java:44)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

非常感谢。

【问题讨论】：

以下链接可能对***.com/questions/25180142/…有帮助

【参考方案1】：

与您的 hadoop 安装相比，您的输出看起来像您的配置文件较旧。如果你还没有自定义配置文件，你可能需要刷新一下；否则，您将必须检查所有警告并确保它们不重要。

错误输出的结尾表明您正在运行的程序具有某种类型的本机编译组件。如果您将其提交到分布式 map/reduce 集群，系统如何知道如何调用您的本地库？本机库是否已分发到所有节点，并且位于同一文件位置？您是否设置了您的 java 本机库路径，以便它在每个系统上为您的 map/reduce 任务的每次执行包含此位置？

【讨论】：

我在我的代码上试过这个：TableMapReduceUtil.addHBaseDependencyJars(config); 和这个：TableMapReduceUtil.addDependencyJars(job);。但没有什么新鲜事。我在集群中运行程序。它还没有基于它，所以管理员几天前安装了它（0.98）并运行了 hadoop 2.2.0。我还使用-libjars ./mr_lib/*.jar 运行 jar。不知道你设置java原生库路径是什么意思。谢谢。

【参考方案2】：

看起来，hadoop 默认 jar 不是从 hadoop 类路径中获取的，请确保在 jar 中添加的 jar 在 hadoop 类路径中引用。

【讨论】：

这意味着我使用的所有额外的 jars (hbase) 都必须列在 hadoop 类路径中？谢谢。