Spark 1.6.0 DataFrame 自加入问题

Posted 2023-04-15

【中文标题】Spark 1.6.0 DataFrame 自加入问题

【英文标题】：Spark 1.6.0 DataFrame selfjoin issue

【发布时间】：2016-10-12 04:25:35

【问题描述】：

我正在尝试使用 DataFrame Scala API 执行 selfjoin。 这是我的代码sn-ps； 你能告诉我第一个解决方案有什么问题吗？

val df=sqlc.read.json("empMgr.json");

empMgr.json

"ID":101,"ename":"Peter","sal":24.24,"dept":"11","country":"US","doj":"1/12/2017 ","经理":201 "ID":201,"ename":"John","sal":1300,"dept":"232","country":"IN","doj":"4/22/2016","经理":111 "ID":301,"ename":"Sam","dept":"22","country":"KR","doj":"5/22/2015","mgr":201

// 1. following is not working
var df_right=df; 
df.join(df_right, df("mgr") === df_right("ID")).show()
df.join(df, df("mgr") === df("ID")).show()

/*
 * Output:
 * +---+-------+----+---+-----+---+---+---+-------+----+---+-----+---+---+
    | ID|country|dept|doj|ename|mgr|sal| ID|country|dept|doj|ename|mgr|sal|
    +---+-------+----+---+-----+---+---+---+-------+----+---+-----+---+---+
    +---+-------+----+---+-----+---+---+---+-------+----+---+-----+---+---+
 * */


//2. following works fine
df_right= sqlc.read.json("file:///opt/data/empMgr.json");  
df.join(df_right, df("mgr") === df_right("ID")).show()

/*
 *Output:
 * +---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+
    | ID|country|dept|      doj|ename|mgr|  sal| ID|country|dept|      doj|ename|mgr|   sal|
    +---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+
    |101|     US|  11|1/12/2017|Peter|201|24.24|201|     IN| 232|4/22/2016| John|111|1300.0|
    |301|     KR|  22|5/22/2015|  Sam|201| null|201|     IN| 232|4/22/2016| John|111|1300.0|
    +---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+ 
 * */


//3. following works fine
df.registerTempTable("empMgr")
sqlc.sql("select b.ename, a.ename as mgr,b.mgr from empMgr a join empMgr b on a.ID=b.mgr").show();

/*
 * output
 * +-----+----+---+
  |ename| mgr|mgr|
  +-----+----+---+
  |Peter|John|201|
  |  Sam|John|201|
  +-----+----+---+
 * */

【问题讨论】：

您的问题是什么？我是不是弄错了，或者第 1 点中有一条不应该存在的额外线？请澄清。

【参考方案1】：

在引用相似名称时使用 Dataframe 的 as() 方法消除歧义。

df.as("a").join(df.as("b"), $"a.mgr" === $"b.ID").show

+---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+
| ID|country|dept|      doj|ename|mgr|  sal| ID|country|dept|      doj|ename|mgr|   sal|
+---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+
|101|     US|  11|1/12/2017|Peter|201|24.24|201|     IN| 232|4/22/2016| John|111|1300.0|
|301|     KR|  22|5/22/2015|  Sam|201| null|201|     IN| 232|4/22/2016| John|111|1300.0|
+---+-------+----+---------+-----+---+-----+---+-------+----+---------+-----+---+------+

【讨论】：

我无法测试这个，因为我遇到了错误。 **'value $ 不是 StringContext 的成员'** 这是我的 maven 依赖项` org.apache.sparkspark-core_2.101.6.0org.apache.sparkspark-sql_2.101.6.0`

尝试导入隐式，看看是否能解决问题：import sqlContext.implicits._

谢谢，它成功了。投票赞成。 1. 你能告诉我解决方案#1 的问题吗？语法不正确吗？ 2.谁能给我提供Java修复？