SparkSession使用[error]日志记录到控制台

Posted 2021-04-04

如果我运行以下Main.scala类：

object main extends App {

  import org.apache.spark.sql.SparkSession

  val spark = SparkSession
    .builder()
    .config("spark.master", "local")
    .config("spark.network.timeout", "10000s")
    .config("spark.executor.heartbeatInterval", "5000s")
    .getOrCreate()

  println("Hello World")

  //stop spark
  spark.stop()

}

*编辑：这是我的log4j.properties文件，位于main esources下：

log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.Threshold=INFO

log4j会将所有INFO日志记录到控制台，但INFO日志将被归类为[error]，如下所示：

[info] Running controller.main compile:last run
[error] Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
[error] 19/03/14 22:35:08 INFO SparkContext: Running Spark version 2.3.0
[error] 19/03/14 22:35:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[error] 19/03/14 22:35:09 INFO SparkContext: Submitted application: 289d7336-196c-4b3a-8bf9-6e247b7a6883
[error] 19/03/14 22:35:09 INFO SecurityManager: Changing view acls to: elicrane
[error] 19/03/14 22:35:09 INFO SecurityManager: Changing modify acls to: elicrane
[error] 19/03/14 22:35:09 INFO SecurityManager: Changing view acls groups to: 
[error] 19/03/14 22:35:09 INFO SecurityManager: Changing modify acls groups to: 
[error] 19/03/14 22:35:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(elicrane); groups with view permissions: Set(); users  with modify permissions: Set(elicrane); groups with modify permissions: Set()
[error] 19/03/14 22:35:09 INFO Utils: Successfully started service 'sparkDriver' on port 51903.
[error] 19/03/14 22:35:09 INFO SparkEnv: Registering MapOutputTracker
[error] 19/03/14 22:35:09 INFO SparkEnv: Registering BlockManagerMaster
[error] 19/03/14 22:35:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
[error] 19/03/14 22:35:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
[error] 19/03/14 22:35:09 INFO DiskBlockManager: Created local directory at /private/var/folders/tj/qnhq1s7x6573bbxlkps7p3wc0000gn/T/blockmgr-e2e86467-a8bc-4cec-8ee9-fa55ef8ce99b
[error] 19/03/14 22:35:09 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
[error] 19/03/14 22:35:09 INFO SparkEnv: Registering OutputCommitCoordinator
[error] 19/03/14 22:35:09 INFO Utils: Successfully started service 'SparkUI' on port 4040.
[error] 19/03/14 22:35:10 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.158:4040
[error] 19/03/14 22:35:10 INFO Executor: Starting executor ID driver on host localhost
[error] 19/03/14 22:35:10 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51904.
[error] 19/03/14 22:35:10 INFO NettyBlockTransferService: Server created on 192.168.1.158:51904
[error] 19/03/14 22:35:10 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
[error] 19/03/14 22:35:10 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.158, 51904, None)
[error] 19/03/14 22:35:10 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.158:51904 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.158, 51904, None)
[error] 19/03/14 22:35:10 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.158, 51904, None)
[error] 19/03/14 22:35:10 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.158, 51904, None)
[info] Hello World
[error] 19/03/14 22:35:10 INFO SparkUI: Stopped Spark web UI at http://192.168.1.158:4040
[error] 19/03/14 22:35:10 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
[error] 19/03/14 22:35:10 INFO MemoryStore: MemoryStore cleared
[error] 19/03/14 22:35:10 INFO BlockManager: BlockManager stopped
[error] 19/03/14 22:35:10 INFO BlockManagerMaster: BlockManagerMaster stopped
[error] 19/03/14 22:35:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
[error] 19/03/14 22:35:10 INFO SparkContext: Successfully stopped SparkContext
[error] 19/03/14 22:35:10 INFO ShutdownHookManager: Shutdown hook called
[error] 19/03/14 22:35:10 INFO ShutdownHookManager: Deleting directory /private/var/folders/tj/qnhq1s7x6573bbxlkps7p3wc0000gn/T/spark-bd7cacce-4431-442e-8942-62eec678717a
[success] Total time: 11 s, completed Mar 14, 2019 10:35:10 PM

这些错误似乎不会阻止应用程序运行，但它们很烦人。所以我有两个问题：

1）导致这些错误的原因是什么？

2）我应该压制它们，如果是这样的话怎么样？

答案

您能否按照下面的说明更新您的log4j属性。

# Root logger option
log4j.rootLogger=INFO, stdout

# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %-5p %c{3}:%L - %m%n //set according to your logs. 

如果它不起作用，请告诉我。谢谢！

希望能帮助到你！