PySpark python错误：异常：Java网关进程在向驱动程序发送其端口号之前退出

Posted 2023-04-15

【中文标题】PySpark python错误：异常：Java网关进程在向驱动程序发送其端口号之前退出

【英文标题】：PySpark python error: Exception: Java gateway process exited before sending the driver its port number

【发布时间】：2017-11-08 19:11:40

【问题描述】：

我在 python 3 中使用 PySpark，我刚刚用命令下载了 pyspark

conda install pyspark

并成功下载，我将文件导入Jupyter Notebook，然后尝试运行以下代码

sc1 = sp.SparkContext.getOrCreate()

我收到以下异常

Exception                                 Traceback (most recent call last)
<ipython-input-9-a4daca8dcb9d> in <module>()
----> 1 sc1 = sp.SparkContext.getOrCreate()

~\Anaconda3\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
    332         with SparkContext._lock:
    333             if SparkContext._active_spark_context is None:
--> 334                 SparkContext(conf=conf or SparkConf())
    335             return SparkContext._active_spark_context
    336 

~\Anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    113         """
    114         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 115         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    116         try:
    117             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

~\Anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    281         with SparkContext._lock:
    282             if not SparkContext._gateway:
--> 283                 SparkContext._gateway = gateway or launch_gateway(conf)
    284                 SparkContext._jvm = SparkContext._gateway.jvm
    285 

~\Anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf)
     93                 callback_socket.close()
     94         if gateway_port is None:
---> 95             raise Exception("Java gateway process exited before sending the driver its port number")
     96 
     97         # In Windows, ensure the Java child processes do not linger after Python has exited.

Exception: Java gateway process exited before sending the driver its port number

我尝试了几个解决方案，但没有一个有效.. 有什么建议吗？？

【问题讨论】：

我得到了 windows 的解决方案，medium.com/@GalarnykMichael/… 按照此链接上的说明进行操作，现在一切正常

很遗憾 conda install pyspark 没有更简单的解决方案

【参考方案1】：

Solution为windows用户提供了解决方案，如何安装，一步一步的说明，我跟着它工作，试试这个。

【参考方案2】：

Windows 中的问题在于 JAVA_HOME 目录路径不能像“C:\Program Files”那样有任何空格。我在“C:\Java”目录中重新安装了Java，问题就消失了。​