Spark Shell 启动报NullPointerException（包含getZipEntry）

Posted 2022-11-30 fansy1990

文章目录

• Spark Shell 启动NullPointerException提示getZipEntry
• • 1. 软件版本：
  • 2. 问题描述：
  • 3. 问题解决思路
  • 4. 后续

Spark Shell 启动NullPointerException提示getZipEntry

1. 软件版本：

软件	版本
CDH	CDH-5.7.3-1.cdh5.7.3.p0.5
Spark	Standalone模式，1.6.0
JDK	1.8.0_131

2. 问题描述：

在使用Spark Shell的过程中，CDH其中的一个节点，忽然执行不了spark-shell命令，报类似如下的错误：

Exception in thread "main" java.lang.NullPointerException
	at java.util.zip.ZipFile.getZipEntry(ZipFile.java:566)
	at java.util.zip.ZipFile.access$900(ZipFile.java:60)
	at java.util.zip.ZipFile$ZipEntryIterator.next(ZipFile.java:524)
	at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:499)
	at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:480)
	at scala.reflect.io.FileZipArchive.iterator(ZipArchive.scala:135)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:91)
	at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:308)
	at scala.tools.nsc.util.DirectoryClassPath.x$16$lzycompute(ClassPath.scala:317)
	at scala.tools.nsc.util.DirectoryClassPath.x$16(ClassPath.scala:317)
	at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:317)
	at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:317)
	at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:297)
	at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:375)
	at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:375)
	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:375)
	at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:370)
	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:243)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:194)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
	at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:240)
	at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:59)
	at scala.tools.nsc.Global.rootMirror(Global.scala:57)
	at scala.tools.nsc.Global.rootMirror(Global.scala:37)
	at scala.reflect.internal.Definitions$DefinitionsClass.<init>(Definitions.scala:166)
	at scala.reflect.internal.Definitions$definitions$.<init>(Definitions.scala:20)
	at scala.reflect.internal.SymbolTable.definitions$lzycompute(SymbolTable.scala:13)
	at scala.reflect.internal.SymbolTable.definitions(SymbolTable.scala:13)
	at scala.tools.nsc.Global$Run.<init>(Global.scala:1290)
	at org.apache.spark.repl.SparkIMain.org$apache$spark$repl$SparkIMain$$_initialize(SparkIMain.scala:198)
	at org.apache.spark.repl.SparkIMain.initializeSynchronous(SparkIMain.scala:227)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
	at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
	at org.apache.spark.repl.Main$.main(Main.scala:31)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).

初步怀疑是classpath的问题，经查询：https://stackoverflow.com/questions/14993621/how-to-debug-nullpointerexception-occurred-in-library-code，确实是classpath的问题。

3. 问题解决思路

1. 初步怀疑可能是当前节点的spark-assemble.jar损坏，拷贝其他节点相应jar到此节点，无效；
2. 对比/opt/cloudera/parcels/CDH-5.7.3-1.cdh5.7.3.p0.5/lib/spark/conf 目录下面的classpath.txt （此文件是classpath的加载路径，会在进行初始化的时候读取），发现当前节点多了一些记录，故此判断是多的记录引起的，把多余的记录删掉，发现启动成功；

4. 后续

解决后，要还原此问题，发现：

1. 有多2条记录，其中一个是mysql jdbc jar；所以把mysql jdbc jar添加到classpath发现仍不会报错；
2. 由于另一个记录是一个自定义的jar包，所以是否是因为这个自定义的jar包不存在而导致的。经试验，在classpath.txt中随便加一个不存在的jar包记录，发现仍能启动；故此，添加的Jar包应该是存在的。
3. 既然Jar包存在，找到其Jar包，并查看其内容，出现如下所示的异常：

[root@server1 ~]# jar vtf demo.Driver.jar 
java.lang.NullPointerException
	at java.util.zip.ZipFile.getZipEntry(ZipFile.java:566)
	at java.util.zip.ZipFile.access$900(ZipFile.java:60)
	at java.util.zip.ZipFile$ZipEntryIterator.next(ZipFile.java:524)
	at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:499)
	at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:480)
	at sun.tools.jar.Main.list(Main.java:1118)
	at sun.tools.jar.Main.run(Main.java:293)
	at sun.tools.jar.Main.main(Main.java:1288)

所以确认是这个Jar包的问题，经确认，这个Jar包损坏。