Conda 搭建jupyter notebook + pyspark

Posted 2020-09-07 Jiang_Xi

 python3.6不支持pyspark, 好在用的是Anaconda这种神器，可以随意切换python版本。因为我的Spark是1.6的，所以python2.7应该是可以的。

首先

conda create -n py27 python=2.7 anaconda
source activate py27 

conda install python=2.7
首就将当前的python环境切换到了2.7， 然后修改

/usr/local/share/jupyter/kernels/pyspark/kernel.json

{
"display_name": "PySpark",
"language": "python",
"argv": [ "/home/.../anaconda3/envs/py27/bin/python", "-m", "ipykernel", "-f", "{connection_file}" ],
"env": {
"SPARK_HOME": "/.../spark/spark-1.6.0-bin-hadoop2.6/",
"PYSPARK_PYTHON": "/.../anaconda3/envs/py27/bin/python",
"PYSPARK_DRIVER_PYTHON": "ipython2",                       
"PYTHONPATH": "/.../spark/spark-1.6.0-bin-hadoop2.6/python/:/.../spark/spark-1.6.0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip",
"PYTHONSTARTUP": "/.../spark/spark-1.6.0-bin-hadoop2.6/python/pyspark/shell.py",
"PYSPARK_SUBMIT_ARGS": "--name pyspark --master yarn --deploy-mode client --conf spark.executor.instances=8 --conf spark.executor.memory=5g --conf spark.driver.memory=8g --conf spark.sql.caseSensitive=false --conf spark.yarn.queue=queue1 pyspark-shell"
}
}

注意该文件路径不要改，会有程序到这个路径直接读取。

为了确保不会跑到别的python上去，将PATH变量也修改一下：export PATH=/.../anaconda3/envs/py27/bin/python:$PATH