Spark:scala集合转化为DS/DF

Posted 2020-10-03 yy

scala集合转化为DS/DF

 1 case class TestPerson(name: String, age: Long, salary: Double)
 2 val tom = TestPerson("Tom Hanks",37,35.5)
 3 val sam = TestPerson("Sam Smith",40,40.5)
 4 val PersonList = mutable.MutableList[TestPerson]()
 5 PersonList += tom
 6 PersonList += sam
 7 
 8 val personDS = PersonList.toDS()
 9 println(personDS.getClass)
10 personDS.show()
11 
12 val personDF = PersonList.toDF()
13 println(personDF.getClass)
14 personDF.show()
15 personDF.select("name", "age").show()

输出结果：

class org.apache.spark.sql.Dataset

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| 37|  35.5|
|Sam Smith| 40|  40.5|
+---------+---+------+

class org.apache.spark.sql.DataFrame

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| 37|  35.5|
|Sam Smith| 40|  40.5|
+---------+---+------+

+---------+---+
|     name|age|
+---------+---+
|Tom Hanks| 37|
|Sam Smith| 40|
+---------+---+

参考资料：

https://stackoverflow.com/questions/39397652/convert-scala-list-to-dataframe-or-dataset