Spark基础知识汇总

Posted 2020-07-28 创业者-春跃-增长黑客

2，wordcount：

val wordcount = sc.textFile("/user/s-44/wordcount.txt").flatMap(_.split(\' \')).map((_, 1)).reduceByKey(_ + _).sortByKey().collect

val wordcount = sc.textFile("/user/s-44/wordcount.txt").flatMap(_.split(\' \')).map((_, 1)).reduceByKey(_ + _).sortByKey().saveAsTextFile("/user/s-44/result.txt")

下面这个是按value排序

val wordcount = sc.textFile("/user/s-44/wordcount.txt").flatMap(_.split(\' \')).map((_, 1)).reduceByKey(_ + _).map(_.swap).sortByKey().collect

val wordcount = sc.textFile("/user/s-44/wordcount.txt").flatMap(_.split(\' \')).map((_, 1)).reduceByKey(_ + _).map(_.swap).sortByKey().saveAsTextFile("/user/s-44/result.txt")

View Code

 

 

1，集合变成rdd

val rdd = sc.parallelize(List(1, 2, 3, 4, 5))

View Code