python pyspark-wordcount-numtasks.py

Posted 2021-05-09

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了python pyspark-wordcount-numtasks.py相关的知识，希望对你有一定的参考价值。

inputPath = "/public/randomtextwriter/part-m-0000*"
outputPath = "/user/dgadiraju/wordcount"

# Ideal number of tasks could be 4 while processing 1 file
sc.textFile(inputPath). \
  flatMap(lambda rec: rec.split(" ")). \
  map(lambda rec: (rec, 1)). \
  reduceByKey(lambda total, agg: total + agg, 10). \
  saveAsTextFile(outputPath)

以上是关于python pyspark-wordcount-numtasks.py的主要内容，如果未能解决你的问题，请参考以下文章

Python代写，Python作业代写，代写Python，代做Python