Spark-shell交互式编程

Posted daisy99lijing

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Spark-shell交互式编程相关的知识,希望对你有一定的参考价值。

1.该系总共有多少学生
val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val par = lines.map(row=>row.split(",")(0)) 
val distinct_par = par.distinct() //去重操作
distinct_par.count //取得总数

2.该系共开设来多少门课程

val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val par = lines.map(row=>row.split(",")(1)) 
val distinct_par = par.distinct() 
distinct_par.count

3.Tom 同学的总成绩平均分是多少

val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val pare = lines.filter(row=>row.split(",")(0)=="Tom")
pare.foreach(println)
Tom,DataBase,26
Tom,Algorithm,12
Tom,OperatingSystem,16
Tom,Python,40
Tom,Software,60
pare.map(row=>(row.split(",")(0),row.split(",")(2).toInt)).mapValues(x=>(x,1)).reduceByKey((x,y
) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect()
//res9: Array[(String, Int)] = Array((Tom,30))

4.求每名同学的选修的课程门数

val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val pare = lines.filter(row=>row.split(",")(1)=="DataBase")
pare.count
res1: Long = 126

5.该系 DataBase 课程共有多少人选修

val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val pare = lines.filter(row=>row.split(",")(1)=="DataBase")
pare.count
res1: Long = 126

6.各门课程的平均分是多少

val lines = sc.textFile("file:///usr/local/spark/sparksqldata/Data01.txt")
val pare = lines.map(row=>(row.split(",")(1),row.split(",")(2).toInt))
pare.mapValues(x=>(x,1)).reduceByKey((x,y) => (x._1+y._1,x._2 + y._2)).mapValues(x => (x._1 / x._2)).collect()
res0: Array[(String, Int)] = Array((Python,57), (OperatingSystem,54), (CLanguage,50), 
(Software,50), (Algorithm,48), (DataStructure,47), (DataBase,50), (ComputerNetwork,51))

 

以上是关于Spark-shell交互式编程的主要内容,如果未能解决你的问题,请参考以下文章

Spark学习散点总结

2020年寒假假期总结0206

如何使用 java 连接到 spark-shell 并执行 hive 查询

Spark交互式工具spark-shell

如何从 spark-shell/spark-submit 运行交互式 Spark 应用程序

Spark-Shell的启动与运行