pyspark：删除作为另一列值的子字符串，并从给定列的值中包含正则表达式字符

Posted 2023-04-15

【中文标题】pyspark：删除作为另一列值的子字符串，并从给定列的值中包含正则表达式字符

【英文标题】：pyspark: Remove substring that is the value of another column and includes regex characters from the value of a given column

【发布时间】：2020-11-26 12:54:25

【问题描述】：

假设我有一个类似的数据框

df = spark.createDataFrame(
  [
    ('Test1 This is a test Test2','This is a test'),
    ('That is','That')
  ],
  ['text','name'])


+--------------------------+--------------+
|text                      |name          |
+--------------------------+--------------+
|Test1 This is a test Test2|This is a test|
|That is                   |That          |
+--------------------------+--------------+

如果我申请 df.withColumn("new",F.expr("regexp_replace(text,name,'')")).show(truncate=False) 它可以正常工作并导致

+--------------------------+--------------+------------+
|text                      |name          |new         |
+--------------------------+--------------+------------+
|Test1 This is a test Test2|This is a test|Test1  Test2|
|That is                   |That          | is         |
+--------------------------+--------------+------------+

假设我有以下数据框

+-----------------------------+-----------------+
|text                         |name             |
+-----------------------------+-----------------+
|Test1 This is a test(+1 Test2|This is a test(+1|
|That is                      |That             |
+-----------------------------+-----------------+

如果我应用上面的命令，我会收到以下错误消息：

java.util.regex.PatternSyntaxException：悬空元字符 '+'

我该怎么做才能使这个异常不会以最“pyspark”的方式发生并保持文本中的值不变？

谢谢

【参考方案1】：

在 spark 中使用 replace 函数代替 regexp_replace。

replace(str, search[, replace]) - 替换所有出现的搜索 用替换。

Example:

df.show(10,False)
#+-----------------------------+-----------------+
#|text                         |name             |
#+-----------------------------+-----------------+
#|Test1 This is a test(+1 Test2|This is a test(+1|
#|That is                      |That             |
#+-----------------------------+-----------------+

df.withColumn("new",expr("replace(text,name,'')")).show(10,False)
#+-----------------------------+-----------------+------------+
#|text                         |name             |new         |
#+-----------------------------+-----------------+------------+
#|Test1 This is a test(+1 Test2|This is a test(+1|Test1  Test2|
#|That is                      |That             | is         |
#+-----------------------------+-----------------+------------+