Apache Spark：启动 PySpark 的问题

Posted 2023-04-18

【中文标题】Apache Spark：启动 PySpark 的问题

【英文标题】：Apache Spark: Problems to start PySpark

【发布时间】：2015-12-14 21:02:58

【问题描述】：

当我尝试启动 Pyspark 时，它显示：

Python 2.7.10 (default, Aug 22 2015, 20:33:39) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/12/14 15:32:56 INFO SparkContext: Running Spark version 1.5.2
15/12/14 15:32:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/14 15:32:58 INFO SparkContext: Successfully stopped SparkContext
15/12/14 15:32:58 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
Traceback (most recent call last):
File "/usr/local/Cellar/spark-1.5.2/python/pyspark/shell.py", line 43, in <module>
sc = SparkContext(pyFiles=add_files)
File "/usr/local/Cellar/spark-1.5.2/python/pyspark/context.py", line 113, in __init__
conf, jsc, profiler_cls)
File "/usr/local/Cellar/spark-1.5.2/python/pyspark/context.py", line 174, in _do_init
self._accumulatorServer = accumulators._start_update_server()
File "/usr/local/Cellar/spark-1.5.2/python/pyspark/accumulators.py", line 259, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 420, in __init__
self.server_bind()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 434, in server_bind
self.socket.bind(self.server_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known                                             15/12/14 15:32:58 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/12/14 15:32:58 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/12/14 15:32:58 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

“远程关闭”是什么意思？如何解决这个问题？谢谢！

【问题讨论】：

你目前的安装是什么？

【参考方案1】：

在我重置 JAVA_HOME 并运行 pyspark 后，问题就解决了。