pyarrow.lib.ArrowInvalid: ('Could not convert X with type Y: did not identify Python value type when

Posted 2023-03-11

【中文标题】pyarrow.lib.ArrowInvalid: (\'Could not convert X with type Y: did not identify Python value type when inferring an Arrow data type\')

【英文标题】：pyarrow.lib.ArrowInvalid: ('Could not convert X with type Y: did not recognize Python value type when inferring an Arrow data type')pyarrow.lib.ArrowInvalid: ('Could not convert X with type Y: did not identify Python value type when inferring an Arrow data type')

【发布时间】：2020-04-25 10:50:07

【问题描述】：

使用pyarrow将包含Player对象的pandas.DataFrame转换为pyarrow.Table，代码如下

import pandas as pd
import pyarrow as pa

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def __repr__(self):
        return f'<self.name (self.age)>'

data = [
    Player('Jack', 21, 'm'),
    Player('Ryan', 18, 'm'),
    Player('Jane', 35, 'f'),
]
df = pd.DataFrame(data, columns=['player'])
print(pa.Table.from_pandas(df))

我们得到错误：

pyarrow.lib.ArrowInvalid: ('Could not convert <Jack (21)> with type Player: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column 0 with type object')

使用时遇到同样的错误

df.to_parquet('players.pq')

pyarrow 是否可以回退到使用pickle 序列化这些 Python 对象？还是有更好的解决方案？ pyarrow.Table 最终将使用Parquet.write_table() 写入磁盘。

使用 Python 3.8.0、pandas 0.25.3、pyarrow 0.13.0。 pandas.DataFrame.to_parquet() 不支持多索引，因此首选使用pq.write_table(pa.Table.from_dataframe(pandas.DataFrame)) 的解决方案。

谢谢！

【问题讨论】：

你能用 Apache Arrow 打开一个 JIRA 问题吗？我们并没有真正与 *** 上的用户或开发人员互动。 github.com/apache/arrow/blob/master/CONTRIBUTING.md

你有想过这个吗？

【参考方案1】：

据我了解，“类型”存在问题，因为 repr 试试这个方法（它有效）：

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def other(self):
        return f'<self.name (self.age)>'

data = [
    Player('Jack', 21, 'm').other(),
    Player('Ryan', 18, 'm').other(),
    Player('Jane', 35, 'f').other(),
]
df = pd.DataFrame(data, columns=['player'])
print(df)
        player
0  <Jack (21)>
1  <Ryan (18)>
2  <Jane (35)>

print(pa.Table.from_pandas(df))

pyarrow.Table
player: string

【参考方案2】：

我的建议是将数据插入到已经序列化的 DataFrame 中。

最佳选择 - 使用数据类 (python >=3.7)

通过装饰器将 Player 类定义为数据类，并让序列化在本地为您完成（到 JSON）。

import pandas as pd
from dataclasses import dataclass

@dataclass
class PlayerV2:
    name:str
    age:int
    gender:str

    def __repr__(self):
        return f'<self.name (self.age)>'


dataV2 = [
    PlayerV2(name='Jack', age=21, gender='m'),
    PlayerV2(name='Ryan', age=18, gender='m'),
    PlayerV2(name='Jane', age=35, gender='f'),
]

# The serialization is done natively to JSON
df_v2 = pd.DataFrame(data, columns=['player'])
print(df_v2)

# Can still get the objects's attributes by deserializeing the record
json.loads(df_v2["player"][0])['name']

手动序列化对象（python

在 Player 类中定义一个序列化函数，并在创建 Dataframe 之前对每个实例进行序列化。

import pandas as pd
import json

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def __repr__(self):
        return f'<self.name (self.age)>'
    
    # The serialization function for JSON, if for some reason you really need pickle you can use it instead
    def toJSON(self):
        return json.dumps(self, default=lambda o: o.__dict__)

# Serialize the objects before inserting it into the DataFrame
data = [
    Player('Jack', 21, 'm').toJSON(),
    Player('Ryan', 18, 'm').toJSON(),
    Player('Jane', 35, 'f').toJSON(),
]
df = pd.DataFrame(data, columns=['player'])

# You can see all the data inserted as a serialized json into the column player
print(df)

# Can still get the objects's attributes by deserializeing the record
json.loads(df["player"][0])['name']