Apache Livy 0.7.0 无法创建交互式会话

Posted 2023-03-23

【中文标题】Apache Livy 0.7.0 无法创建交互式会话

【英文标题】：Apache Livy 0.7.0 Failed to create Interactive session

【发布时间】：2021-09-02 14:44:16

【问题描述】：

使用 apache Livy 0.7.0 创建新会话时，出现以下错误。 我也在使用 zeppelin notebook（livy 解释器）来创建会话。

使用 Scala 版本 2.12.10、Java HotSpot(TM) 64 位服务器 VM、11.0.11 火花 3.0.2 齐柏林飞艇 0.9.0

知道为什么会出现错误吗？

我已经检查了类路径中有 livy-repl_2.11-0.7.1-incubating.jar 并且 JAR 已经有它无法找到的类。

$ jar tf livy-repl_2.11-0.7.1-incubating.jar | grep "ReplDriver"
org/apache/livy/repl/ReplDriver$$anonfun$addFile$1.class
org/apache/livy/repl/ReplDriver$$anonfun$1.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$1.class
org/apache/livy/repl/ReplDriver$$anonfun$addJarOrPyFile$1.class
org/apache/livy/repl/ReplDriver.class
org/apache/livy/repl/ReplDriver$$anonfun$handle$2.class
org/apache/livy/repl/ReplDriver$$anonfun$2.class

错误日志

    21/06/17 13:13:28 INFO LineBufferedStream: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.livy.repl.ReplDriver
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    21/06/17 13:13:28 INFO LineBufferedStream:      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)


21/06/17 13:13:28 WARN ContextLauncher: Child process exited with code 1.
21/06/17 13:13:28 ERROR RSCClient: Failed to connect to context.
java.io.IOException: Child process exited with code 1.
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:397)
        at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:448)
        at java.base/java.lang.Thread.run(Thread.java:834)
21/06/17 13:13:28 ERROR SparkProcApp: spark-submit exited with code 1
21/06/17 13:13:28 INFO RSCClient: Failing pending job f05c2bac-6e0a-4732-a419-0bde4dcfd06b due to shutdown.
21/06/17 13:13:28 INFO InteractiveSession: Stopping InteractiveSession 0...
21/06/17 13:13:28 INFO InteractiveSession: Failed to ping RSC driver for session 0. Killing application.
21/06/17 13:13:28 INFO InteractiveSession: Stopped InteractiveSession 0.
21/06/17 13:13:28 WARN InteractiveSession: Fail to get rsc uri
java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.livy.server.interactive.InteractiveSession$$anonfun$18.apply(InteractiveSession.scala:411) 

【参考方案1】：

Spark 3.0.x 带有 scala 2.12 版本。您需要使用 scal 2.12 使用 Spark 3.0.x 构建 livy 来解决此问题。然后你需要调整你的 livy.conf 这里是关于如何使用 maven 重建你的 livy 的文章 (How to rebuild apache Livy with scala 2.12)

祝你好运。我遇到了同样的问题，并能够通过上述步骤解决。