TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class 'list'> colum

Posted

技术标签:

【中文标题】TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class \'list\'> column literals use \'lit\' \'array\' \'struct\' or \'create_map\'【英文标题】:TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class 'list'> column literals use 'lit' 'array' 'struct' or 'create_map'TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class 'list'> column literals use 'lit' 'array' 'struct' or 'create_map' 【发布时间】:2021-10-11 02:58:00 【问题描述】:

我在 PySpark UDF 中遇到问题,它抛出错误

PythonException: An exception was thrown from a UDF: 'TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class 'list'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.

我正在尝试根据优先级数组获取一个数字,并且它们的优先级随着它们从左到右而降低。

precedence = [1,2,11,12,13,20,20131,200,202,203,210,220,223,226,235,236,237,242,244,245,247,253,254,257,259,260,262,278,283,701,20107,20108,20109,20112,20115,20123,20135,20141,20144,20152,20162,20163,20167,20168,20169,20170,20171,20172,20173,20174,20175,14,211,213,258,270,273,274,275,277,280,281,287,288,20120,20122,20124,20125,20126,20130,20133,20136,20137,20138,20140,20142,20143,20154,20155,20156,20157]
reverse_order = precedence[::-1]

def get_p(row):
    if (row!=None) and (row!="null"):
        temp = row.split(",")
        test = []
        for i in temp:
            if (i.find('=')!=-1):
                i = i.split('=')[0]
            if int(i) in reverse_order:
                test.append(reverse_order.index(int(i)))
            else:
                test.append(-1)
        if max(test)!=-1:
            return reverse_order[max(test)]
        return -999
    else:
        return None

get_array = udf(get_p, IntegerType())

bronze_table = bronze_table.withColumn("precedence", get_array("event_list"))
bronze_table.select("event_list","precedence").show(100, False)

这里是示例记录,

+---------------------------------------------------------------------------------------+
|event_list                                                                             |
+---------------------------------------------------------------------------------------+
|276,100,101,202,176                                                                    |
|276,100,2,124,176                                                                      |
|246,100,101,257,115,116,121,123,124,125,135,138,145,146,153,167,168,170,171,173,189,191|
|246,100,101,278,123,124,135,170,189,191                                                |
|20131=16,20151,100,101,102,115,116,121,123,124,125,135,138,145,146,153,168,170,171     |
|null                                                                                   |
|20107=9,20151,100,101,102,123,124,135,170,189,191                                      |
|20108=3,20151,100,101,102,123,124,125,135,170,171,189,191                              |
|null                                                                                   |
+---------------------------------------------------------------------------------------+

我期待什么

+---------------------------------------------------------------------------------------+----------+
|event_list                                                                             |precedence|
+---------------------------------------------------------------------------------------+----------+
|276,100,101,202,176                                                                    |202       |
|276,100,2,124,176                                                                      |2         |
|246,100,101,257,115,116,121,123,124,125,135,138,145,146,153,167,168,170,171,173,189,191|257       |
|246,100,101,278,123,124,135,170,189,191                                                |278       |
|20131=16,20151,100,101,102,115,116,121,123,124,125,135,138,145,146,153,168,170,171     |20131     |
|null                                                                                   |null      |
|20107=9,20151,100,101,102,123,124,135,170,189,191                                      |20107     |
|20108=3,20151,100,101,102,123,124,125,135,170,171,189,191                              |20108     |
|null                                                                                   |null      |
+---------------------------------------------------------------------------------------+----------+

我的UDF 在 python 中按预期工作,但在 pyspark 中不工作。我请求有人帮我解决这个问题。

【问题讨论】:

【参考方案1】:

PySpark 数据帧中的null 在 Python 中是 None,所以这个条件 if row!="null": 是不正确的。请改用if row!= None:

但是,您的 get_p 函数对我来说运行不佳,例如:

get_p('276,100,101,202,176')
# output: 2
# expected: 202

get_p('20131=16,20151,100,101,102,115,116,121,123,124,125,135,138,145,146,153,168,170,171')
# output: Exception `invalid literal for int() with base 10: ''`
# expected: 20131

【讨论】:

感谢您对此进行调查,我确实更新了 get_p 函数,@pltc【参考方案2】:

谢谢大家,谁试图解决这个问题。我可以使用这个链接解决它https://***.com/a/63654269/6187792

链接中指定的 max 函数的问题。

这是更新后的代码。

import builtins as p
def get_p(row):
    if (row!=None) and (row!="null"):
        temp = row.split(",")
        test = []
        for i in temp:
            if (i.find('=')!=-1):
                i = i.split('=')[0]
            if int(i) in reverse_order:
                test.append(reverse_order.index(int(i)))
            else:
                test.append(-1)
        if p.max(test)!=-1:
            return reverse_order[p.max(test)]
        return None
    else:
        return None

【讨论】:

我认为max 函数没有任何问题。问题是您导入 PySpark 函数的方式不正确。所以,不要使用import builtins as p,然后使用p.max 等内置函数,您应该使用from pyspark.sql import functions as F,然后使用F.udfF.col

以上是关于TypeError: Invalid argument, not a string or column: [79, -1, -1] of type <class 'list'> colum的主要内容,如果未能解决你的问题,请参考以下文章

throw new ERR_INVALID_ARG_TYPE('chunk',['string','Buffer'],chunk);TypeError[ERR_INVALID_ARG_TYPE]:“c

TypeError [ERR_INVALID_ARG_TYPE] 尝试将错误写入 Nodejs 中的文件时

TypeError [ERR_INVALID_ARG_TYPE]:“listener”参数必须是函数类型。收到未定义

Firebase - TypeError [ERR_INVALID_ARG_TYPE]:“路径”参数必须是字符串类型

电子中的奇怪错误。未捕获的异常:TypeError [ERR_INVALID_ARG_TYPE]:“路径”

NodeJS - TypeError [ERR_INVALID_ARG_TYPE]:“路径”参数必须是字符串类型。收到未定义