查看YARN任务日志的几种方式

Posted 2021-05-01 大数据技术与架构

回复”资源“获取更多资源

点击右侧关注，大数据开发领域最强公众号！

点击右侧关注，暴走大数据！

通过history server

通过history server，直接在web ui上查看（如果任务异常退出，可能会看不到）

 

通过yarn命令(用户要和提交任务的用户一致）

yarn application -list -appStates ALL（这个不显示时间信息）

[hdfs@node1 opt]$ yarn application -list -appStates ALL17/07/26 10:51:33 INFO impl.TimelineClientImpl: Timeline service address: http://node2.hde.h3c.com:8188/ws/v1/timeline/17/07/26 10:51:33 INFO client.RMProxy: Connecting to ResourceManager at node2.hde.h3c.com/192.38.0.232:8050Total number of applications (application-types: [] and states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED]):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URLapplication_1500702983136_0002 word count MAPREDUCE ambari-qa default FINISHED SUCCEEDED 100% http://node2.hde.h3c.com:19888/jobhistory/job/job_1500702983136_0002application_1500702983136_0001 DistributedShell YARN ambari-qa default FINISHED SUCCEEDED 100% N/A

yarn logs -applicationId application_1493700892407_0007

[ambari-qa@node1 opt]$ yarn logs -applicationId application_1500702983136_000217/07/26 10:56:23 INFO impl.TimelineClientImpl: Timeline service address: http://node2.hde.h3c.com:8188/ws/v1/timeline/17/07/26 10:56:23 INFO client.RMProxy: Connecting to ResourceManager at node2.hde.h3c.com/192.38.0.232:805017/07/26 10:56:25 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library17/07/26 10:56:25 INFO compress.CodecPool: Got brand-new decompressor [.deflate]  Container: container_1500702983136_0002_01_000002 on node1.hde.h3c.com_45454==============================================================================LogType:stderrLog Upload Time:Sat Jul 22 01:59:41 -0400 2017LogLength:0Log Contents:End of LogType:stderr LogType:stdoutLog Upload Time:Sat Jul 22 01:59:41 -0400 2017LogLength:0Log Contents:End of LogType:stdout LogType:syslogLog Upload Time:Sat Jul 22 01:59:41 -0400 2017LogLength:3674Log Contents:2017-07-22 01:59:16,446 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties

直接查看hdfs路径的log

查看yarn-site.xml，确定log配置目录

<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value></property>```2）查看日志文件信息（注意日期和时间）[hdfs@node1 root]$ hdfs dfs -ls /app-logs/hdfs/logsFound 1 itemsdrwxrwx--- - hdfs hadoop 0 2017-05-02 04:18 /app-logs/hdfs/logs/application_1493700892407_00073）查看日志详情（注意查看节点重启前的几个敏感app）yarn logs -applicationId application_1493700892407_0007（同2）

欢迎点赞+收藏+转发朋友圈素质三连

文章不错？点个【在看】吧！ 

以上是关于查看YARN任务日志的几种方式的主要内容，如果未能解决你的问题，请参考以下文章

Eclipse查看以往的错误日志的几种简单方式

Flink On Yarn如何查看任务日志

Spark On Yarn如何查看任务日志

spring定时任务的几种实现方式

Yarn开启ACL用户认证之后无法查看Yarn历史任务日志解决办法

Spring定时任务的几种实现