Spark 运行时错误 - ClassDefNotFound: SparkConf

Posted 2023-03-24

【中文标题】Spark 运行时错误 - ClassDefNotFound: SparkConf

【英文标题】：Spark Runtime Error - ClassDefNotFound: SparkConf

【发布时间】：2015-09-19 05:31:55

【问题描述】：

在安装和构建 Apache Spark 之后（尽管有很多警告），我们的 Spark 应用程序的编译（使用“sbt 包”）成功完成。但是，当尝试使用 spark-submit 脚本运行我们的应用程序时，会出现运行时错误，指出未找到 SparkConf 类定义。 SparkConf.scala 文件存在于我们的系统中，但似乎没有正确构建。关于如何解决这个问题的任何想法？

user@compname:~/Documents/TestApp$ /opt/Spark/spark-1.4.0/bin/spark-submit --master local[4] --jars /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar target/scala-2.11/sparkesingest_2.11-1.0.0.jar ~/Desktop/CSV/data.csv es-index localhost
Warning: Local jar /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar does not exist, skipping.
log4j:WARN No appenders could be found for logger (App).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/01 13:56:58 INFO SparkContext: Running Spark version 1.4.0
15/07/01 13:56:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/01 13:56:59 WARN Utils: Your hostname, compname resolves to a loopback address: 127.0.1.1; using [IP ADDRESS] instead (on interface eth0)
15/07/01 13:56:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/07/01 13:56:59 INFO SecurityManager: Changing view acls to: user
15/07/01 13:56:59 INFO SecurityManager: Changing modify acls to: user
15/07/01 13:56:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user)
15/07/01 13:56:59 INFO Slf4jLogger: Slf4jLogger started
15/07/01 13:56:59 INFO Remoting: Starting remoting
15/07/01 13:56:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@[IP ADDRESS]]
15/07/01 13:56:59 INFO Utils: Successfully started service 'sparkDriver' on port 34276.
15/07/01 13:56:59 INFO SparkEnv: Registering MapOutputTracker
15/07/01 13:56:59 INFO SparkEnv: Registering BlockManagerMaster
15/07/01 13:56:59 INFO DiskBlockManager: Created local directory at /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45
15/07/01 13:56:59 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/07/01 13:56:59 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/httpd-f4c3c67a-d058-4aba-bd65-5352feb5f12e
15/07/01 13:56:59 INFO HttpServer: Starting HTTP Server
15/07/01 13:56:59 INFO Utils: Successfully started service 'HTTP file server' on port 33599.
15/07/01 13:56:59 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/01 13:56:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/07/01 13:56:59 INFO SparkUI: Started SparkUI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 ERROR SparkContext: Jar not found at file:/opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar
15/07/01 13:57:00 INFO SparkContext: Added JAR file:/home/user/Documents/TestApp/target/scala-2.11/sparkesingest_2.11-1.0.0.jar at http://[IP ADDRESS]:33599/jars/sparkesingest_2.11-1.0.0.jar with timestamp 1435784220028
15/07/01 13:57:00 INFO Executor: Starting executor ID driver on host localhost
15/07/01 13:57:00 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44746.
15/07/01 13:57:00 INFO NettyBlockTransferService: Server created on 44746
15/07/01 13:57:00 INFO BlockManagerMaster: Trying to register BlockManager
15/07/01 13:57:00 INFO BlockManagerMasterEndpoint: Registering block manager localhost:44746 with 265.4 MB RAM, BlockManagerId(driver, localhost, 44746)
15/07/01 13:57:00 INFO BlockManagerMaster: Registered BlockManager
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(143840) called with curMem=0, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 140.5 KB, free 265.3 MB)
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(12635) called with curMem=143840, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 12.3 KB, free 265.3 MB)
15/07/01 13:57:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44746 (size: 12.3 KB, free: 265.4 MB)
15/07/01 13:57:00 INFO SparkContext: Created broadcast 0 from textFile at Ingest.scala:159
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
    at org.elasticsearch.spark.rdd.CompatUtils.<clinit>(CompatUtils.java:20)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.elasticsearch.hadoop.util.ObjectUtils.loadClass(ObjectUtils.java:71)
    at org.elasticsearch.spark.package$.<init>(package.scala:14)
    at org.elasticsearch.spark.package$.<clinit>(package.scala)
    at build.Ingest$.main(Ingest.scala:176)
    at build.Ingest.main(Ingest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 17 more
15/07/01 13:57:00 INFO SparkContext: Invoking stop() from shutdown hook
15/07/01 13:57:00 INFO SparkUI: Stopped Spark web UI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 INFO DAGScheduler: Stopping DAGScheduler
15/07/01 13:57:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/07/01 13:57:00 INFO Utils: path = /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45, already present as root for deletion.
15/07/01 13:57:00 INFO MemoryStore: MemoryStore cleared
15/07/01 13:57:00 INFO BlockManager: BlockManager stopped
15/07/01 13:57:01 INFO BlockManagerMaster: BlockManagerMaster stopped
15/07/01 13:57:01 INFO SparkContext: Successfully stopped SparkContext
15/07/01 13:57:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/07/01 13:57:01 INFO Utils: Shutdown hook called
15/07/01 13:57:01 INFO Utils: Deleting directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd

这里是 build.sbt 文件：

scalaVersion := "2.11.6"

name := "SparkEsIngest"

version := "1.0.0"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.4.0" % "provided",
    "org.apache.spark" %% "spark-streaming" % "1.4.0" % "provided",
    "org.apache.spark" %% "spark-sql" % "1.4.0" % "provided",
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")
  )

if ( System.getenv("QUERY_ES_RESOURCE") != null) 
  println("[info] Using lib/es-hadoop-build-snapshot/ unmanagedBase dir")
  unmanagedBase <<= baseDirectory  base => base / "lib/es-hadoop-build-snapshot" 
 else 
  println("[info] Using lib/ unmanagedBase dir")
  unmanagedBase <<= baseDirectory  base => base / "lib" 


resolvers += "conjars.org" at "http://conjars.org/repo"

resolvers += "clojars" at "https://clojars.org/repo"

【问题讨论】：

您的 CLASSPATH 和 SPARK_HOME 环境变量设置为什么？此外，输出中还有进一步的错误和警告，指出 /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar 丢失。你检查过吗？我意识到不会出现 NoClassDefFoundError: org/apache/spark/SparkConf ，但是将那个 jar 放在正确的位置会消除潜在的原因，特别是考虑到错误似乎是在那个 jar 的一个类中生成的。

PATH变量设置为：/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games: /usr/local/games:/opt/Scala/scala-2.11.6/bin:/opt/Spark/spark-1.4.0 和 SPARK_HOME 设置为：/opt/Spark/spark-1.4.0 就我而言知道，elasticsearch-hadoop jar 不是必需的，因为它包含在依赖项中并安装在机器上。在没有 --jars 的情况下运行它会产生相同的错误。

【参考方案1】：

您提交的 JAR 中是否包含 spark JAR？似乎您告诉 sbt 您正在提供 jar，但如果您计算 jar 在机器中，我在您的 sbt 中看不到“unmanagedJars in Compile += file(...)”建议您不要这样做，因为它会导致此类问题。

尝试 unrar you're JAR 并查看 spark JAR 是否存在，如果没有，请使用 sbt-assembly 或您选择的其他工具。

【讨论】：

elasticsearch-hadoop jar 不存在，但包含在依赖项中。在没有 --jars 的情况下运行它会产生相同的错误。我曾尝试使用 sbt-assembly 工具，但这也会产生错误。

即使使用 sbt 组件，jar 也不在你的 jar 中？

【参考方案2】：

不要使用provided 范围，因为它使依赖项仅在编译时可用，而不是在运行代码时可用。更改后的代码看起来像..

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.4.0",
    "org.apache.spark" %% "spark-streaming" % "1.4.0",
    "org.apache.spark" %% "spark-sql" % "1.4.0",
    "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")
  )