如何在Spark sql中将列转换为Int

Posted 2021-04-19

我想在数据帧中取最后6个字符串的字符串

 val loc ="/data/published/omega/omega_logs/20171205_4801"
 val df =sqlContext.read.avro(loc)
 val df1 = df.withColumn("sub_str", substring(df("broadcast_end_date_time"),9,14)).select("broadcast_end_date_time","sub_str")

上面的代码有效。但我不想在我的子串功能中硬编码14。如何找到列的长度并将其转换为Int

 +-----------------------+-------+
 |broadcast_end_date_time|sub_str|
  +-----------------------+-------+
 |20171205124000         |124000 |
 |20171205254000         |254000 |
 |20171205143000         |143000 |
 |20171205111000         |111000 |
 |20171205124000         |124000 |

我正在尝试以下代码，我得到以下错误

 val df1 = df.withColumn("sub_str", substring(df("broadcast_end_date_time"),9,length(df("broadcast_end_date_time")))).select("broadcast_end_date_time","sub_str")

似乎length函数返回一个Column。我如何将其转换为Int

 scala> val df1 = df.withColumn("sub_str", substring(df("broadcast_end_date_time"),9,length(df("broadcast_end_date_time")))).select("broadcast_end_date_time","sub_str")
 <console>:52: error: type mismatch;
 found   : org.apache.spark.sql.Column
 required: Int
   val df1 = df.withColumn("sub_str", substring(df("broadcast_end_date_time"),9,length(df("broadcast_end_date_time")))).select("broadcast_end_date_time","sub_str")
                                                                                      ^

有人可以帮我解决这个问题吗？

答案

你是直接在你的数据集上写SQL查询我是spark sql开发人员我完成了你的要求，这可能对你有所帮助。

actualDataset.createOrReplaceTempView("Test");
actualDataset.show();
 Dataset<Row> resultDataSet = spark.sql("SELECT value, substring(value, 9,length(value)) substr, length(value) length from Test");
resultDataSet.show();

输出：

+--------------+
|         value|
+--------------+
|20171205124000|
|20171205254000|
|20171205143000|
+--------------+

+--------------+------+------+
|         value|substr|length|
+--------------+------+------+
|20171205124000|124000|    14|
|20171205254000|254000|    14|
|20171205143000|143000|    14|
+--------------+------+------+