numpy vectorize np.prod 无法构造超过 32 个操作数的 ufunc
Posted
技术标签:
【中文标题】numpy vectorize np.prod 无法构造超过 32 个操作数的 ufunc【英文标题】:numpy vectorize np.prod Cannot construct a ufunc with more than 32 operands 【发布时间】:2022-01-08 13:56:22 【问题描述】:我知道这里有一个类似的问题:Python numpy.vectorize: ValueError: Cannot construct a ufunc with more than 32 operands
但我的情况不同。
我有一个 32 列的 df,你可以通过运行以下代码来获得它:
import numpy as np
import pandas as pd
from io import StringIO
dfs = """
M0 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27 M28 M29 M30 age
1 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 1 2 3 4 3.2
2 7 5 4 5 8 3 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 1 2 3 4 4.5
3 4 8 9 3 5 2 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 1 2 3 4 6.7
"""
df = pd.read_csv(StringIO(dfs.strip()), sep='\s+', )
df
基于业务逻辑我构建了一个向量化的函数,如果函数的参数总数小于32就可以了:
M=["M0","M1","M2","M3","M4","M5","M6","M7","M8","M9","M10","M11","M12","M13","M14","M15","M16","M17","M18","M19",
"M20","M21","M22","M23","M24","M25","M26","M27","M28","M29"]
def func2(df, M):
return [df[i].values for i in M]
def func(age,*Ms):
newcol=np.prod(Ms[0:age])
return newcol
vfunc = np.frompyfunc(func, len(M)+1, 1)
df['newcol']=vfunc(df['age'].values.astype(int), *func2(df,M))
为了便于理解,func2只是让代码更简洁,它为func生成所有参数,没有func2的代码如下:
def func(age,M0,M1,M2,...,M29):
newcol=np.prod(Ms[0:age])
return newcol
vfunc = np.frompyfunc(func, 31, 1)
df['newcol']=vfunc(df['age'].values.astype(int), df['M1'].values,...,df['M29'].values)
真正的问题是,一旦参数的数量等于或大于 32,如下所示:
M=["M0","M1","M2","M3","M4","M5","M6","M7","M8","M9","M10","M11","M12","M13","M14","M15","M16","M17","M18","M19",
"M20","M21","M22","M23","M24","M25","M26","M27","M28","M29","M30"] # M30 is the only difference from the above function
def func2(df, M):
return [df[i].values for i in M]
def func(age,*Ms):
newcol=np.prod(Ms[0:age])
return newcol
vfunc = np.frompyfunc(func, len(M)+1, 1)
df['newcol']=vfunc(df['age'].values.astype(int), *func2(df,M))
我收到错误:
ValueError Traceback (most recent call last)
<ipython-input-66-9a042ad44f9b> in <module>()
76 return newcol
77
---> 78 vfunc = np.frompyfunc(func, len(M)+1, 1)
79
80 df['newcol']=vfunc(df['age'].values.astype(int), *func2(df,M))
ValueError: Cannot construct a ufunc with more than 32 operands (requested number were: inputs = 32 and outputs = 1)
在我的真实业务逻辑中,我有超过 100 列需要使用 np.pro 来计算,所以这真的让我很困惑。有朋友可以帮忙吗?
【问题讨论】:
This 可以回答你的问题吗? 请完整回溯! 更新了我的问题并进行了完整的追溯。 @jezza_99 不,谢谢您的回复。 @QuangHoang,显然去年我发现np.vectorize
比pandas apply
或行迭代更快。但那是因为pandas 的所有索引包袱太慢了。我没有用它的raw
模式测试apply
,它绕过了很多。
【参考方案1】:
这是实现您的结果的一种方法。选中所有带有filter
的M列,使用where
将列位置高于年龄列的所有值替换为nan,然后沿列使用prod
。
df['newcol'] = (
# keep only Mx columns
df.filter(like='M')
# keep only the values when the position of the column
# is less than the age
.where(lambda x: (np.arange(x.shape[1])+1)<df['age'].to_numpy()[:, None])
# multiply all the non-nan values per row
.prod(axis=1)
)
print(df)
【讨论】:
@QuangHoang 更新了我的熊猫版本后,它现在可以工作了,谢谢! 如果不使用 MO ...M30,我需要使用 1-M0..1-M30 怎么办?我试过 1-(np.arange(x.shape[1])+1)和 (np.arange(1-x.shape[1])+1),但收到错误 @William 如果您的意思是列的名称以 1-M 开头?然后在过滤器中的参数like='1-M'
应该选择列
感谢您的回复,不是名称,而是值应该是 1-df['M0]...1-df['M30'],抱歉造成混淆
非常感谢以上是关于numpy vectorize np.prod 无法构造超过 32 个操作数的 ufunc的主要内容,如果未能解决你的问题,请参考以下文章