python pyspark入门篇
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了python pyspark入门篇相关的知识,希望对你有一定的参考价值。
一.环境介绍:
1.安装jdk 7以上
2.python 2.7.11
3.IDE pycharm
4.package: spark-1.6.0-bin-hadoop2.6.tar.gz
二.Setup
1.解压spark-1.6.0-bin-hadoop2.6.tar.gz 到目录D:\\spark-1.6.0-bin-hadoop2.6
2.配置环境变量Path,添加D:\\spark-1.6.0-bin-hadoop2.6\\bin,此后可以在cmd端输入pySpark,返回如下则安装完成:
3.将D:\\spark-1.6.0-bin-hadoop2.6\\python下的pySpark文件拷贝到C:\\Python27\\Lib\\site-packages
4.安装py4j , pip install py4j -i https://pypi.douban.com/simple
5.配置pychar环境变量:
三.Example
1.make a new python file: wordCount.py
#!/usr/bin/env python # -*- coding: utf-8 -*- import sys from pyspark import SparkContext from operator import add import re def main(): sc = SparkContext(appName= "wordsCount") lines = sc.textFile(‘words.txt‘) counts = lines.flatMap(lambda x: x.split(‘ ‘)) .map( lambda x : (x, 1)) .reduceByKey(add) output = counts.collect() print output for (word, count) in output: print "%s: %i" %(word, count) sc.stop() if __name__ =="__main__": main()
2.代码中的words.txt如下:
The dynamic lifestyle people lead nowadays causes many reactions in our bodies and the one that is the most frequent of all is the headache
3.配置pycharm环境变量:
3.1 工具栏 run --> Edit configuration-->点击箭头位置
3.2 然后点击 + ,输入key:SPARK_HOME, value: D:\\spark-1.6.0-bin-hadoop2.6
4.输出结果如下图:
四.深入练习:
1.文档:http://spark.apache.org/docs/latest/api/python/pyspark.html
2.在解压的Spark文档下,有example下有很多实例可以练习。D:\\spark-1.6.0-bin-hadoop2.6\\examples\\src\\main\\python
以上是关于python pyspark入门篇的主要内容,如果未能解决你的问题,请参考以下文章
七python-PySpark篇(黑马程序猿-python学习记录)