spark wordcount
Posted yaoyao66123
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了spark wordcount相关的知识，希望对你有一定的参考价值。
object WordCount {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("WordCount").setMaster("local[*]")
val sc = new SparkContext(conf)
val lines: RDD[String] = sc.textFile("/Users/yaoyao/hobby/AntiReptile/ip/wordcount.txt")
val words: RDD[String] = lines.flatMap(_.split(" "))
val wordOne: RDD[(String, Int)] = words.map((_,1))
val reduces: RDD[(String, Int)] = wordOne.reduceByKey(_+_)
println(reduces.collect().toBuffer)
sc.stop()
}
}
以上是关于spark wordcount的主要内容，如果未能解决你的问题，请参考以下文章