python 属性错误：无法腌制本地对象。使用多处理

Posted 2023-02-16

【中文标题】python 属性错误：无法腌制本地对象。使用多处理

【英文标题】：python attribute error : can't pickle local object. Using multiprocessing

【发布时间】：2021-11-10 09:59:15

【问题描述】：

我正在编写一个程序，旨在包装 C 代码以启动相位计采集。

这段代码应该同时在多个进程上运行，所以我使用multiprocessing。

我阅读了关于这个主题的几乎所有已回答的问题，但我认为我遗漏了一些东西。如果有人可以帮我解决这个问题，我会很高兴。

我有以下错误：

wdir='/home/castaing/Documents/LISA/lisa_zifo_monitoring/Python Drivers/Phasemeter/Py_Code') Traceback（最近一次调用最后一次）：

文件“/home/castaing/Documents/LISA/lisa_zifo_monitoring/Python Drivers/Phasemeter/Py_Code/DriverPhasemeter.py”，第 297 行，在 Process.map(start,Phasemeter.phase_list)

文件 “/home/castaing/anaconda3/envs/LISA/lib/python3.7/multiprocessing/pool.py”， 第 268 行，在地图中 return self._map_async(func, iterable, mapstar, chunksize).get()

文件 “/home/castaing/anaconda3/envs/LISA/lib/python3.7/multiprocessing/pool.py”， 第 657 行，在获取 提高self._value

文件 “/home/castaing/anaconda3/envs/LISA/lib/python3.7/multiprocessing/pool.py”， 第 431 行，在 _handle_tasks 中 放（任务）

文件 “/home/castaing/anaconda3/envs/LISA/lib/python3.7/multiprocessing/connection.py”， 第 206 行，在发送中 self._send_bytes(_ForkingPickler.dumps(obj))

文件 “/home/castaing/anaconda3/envs/LISA/lib/python3.7/multiprocessing/reduction.py”， 第 51 行，在转储中 cls(buf, 协议).dump(obj)

AttributeError: Can't pickle local object 'CDLL.init.._FuncPtr'

我尝试定义一个全局函数来调用相位计对象的方法，如下所示。一开始我在相位计对象中有一个多启动方法，代码是：

def multistart(self) : 
   with multiprocessing.Pool(len(Phasemeter.phase_list)) as Process :
        Process.map(lambda x :x.start,Phasemeter.phase_list)

这是代码（我只放了与我相关的部分）：

#%% Initialization

#%% Function definition

#Fix pickle problem ?
def start(Phasemeter_object):
    Phasemeter_object.start()
        
#%% Class definiton
class Phasemeter :
    # List of all phasemeters objects, accessed by calling Phasemeter.phase_list
    phase_list=[]
    
    #%% Initialization
    def __init__(self,*args,**kwargs) :
        #%% Robustness. Check type of passed arguments 

        #%% Path setting, parsing config file 
        
        #%% Option handling

        #%% Debug, used only if verbose argument is passed in start method 
        
        #%% Defining path to shared object file
        self.path=os.path.abspath(os.path.join(os.getcwd(),
                                               self.path_to_so_file))
        
        # LIBC is now an object and its method are C Code's functions
        self.LIBC = ctypes.CDLL(self.path)

        # Settings C library's signature datatype with ctypes data structre tool
        # INFO: To see all datas types that can be transmited to C Code
        # read ctypes documentation
        # First argument is int : argc
        # Second argument is string array : argv
        # Third is a string : path_to_log_file
        self.LIBC.lisaf_phasemeter_main.argtypes= [ctypes.c_int,
                                            ctypes.POINTER(ctypes.c_char_p),
                                            ctypes.c_char_p,]
        # Return type is int : Return code
        self.LIBC.lisaf_phasemeter_main.restypes = [ctypes.c_int,]
        
        # Add object to phase_list, list used in multistart method 
        Phasemeter.phase_list.append(self)

    #%% Start
    def start(self):
        #%% Marshalling data for C library
        # Create a string array with option list length size
        self.c_char_pointer_array = ctypes.c_char_p * len(self.options)
        # Encode option list
        self.encoded_options = [str.encode(str(i)) for i in self.options ]
        # Fill the string array with encoded strings
        # REMINDER: C code only understand encoded strings
        self.encoded_options = self.c_char_pointer_array (*self.encoded_options)


        #%% Calling C library wihth encoded options
        # If the logfile option is activated then the encoded 
        # string is transmited
        if self.encoded_path_to_log_file :
            self.status = self.LIBC.lisaf_phasemeter_main(
                len(self.encoded_options),
                self.encoded_options,
                self.encoded_path_to_log_file)
        # Otherwise None pointer is transmited
        else :
            self.status = self.LIBC.lisaf_phasemeter_main(len(self.encoded_options),
                                                      self.encoded_options,None)
#%% Multistart
if __name__ == '__main__' :
    # This function is used to start acquisition on multiple phasemeters 
    my_phase = Phasemeter(name="PH1")
    my_phase = Phasemeter(name="PH2")
    with multiprocessing.Pool(len(Phasemeter.phase_list)) as Process :
        Process.map(start,Phasemeter.phase_list)

【问题讨论】：

可能重复：***.com/a/8805244/15826727

这就是我的想法，但我尝试了建议的解决方案，但它不起作用

但是我可能遗漏了什么，欢迎任何帮助

【参考方案1】：

您通过摆脱 lambda 函数解决了一半的问题。但现在你仍然处于“泡菜”状态。在您的 Phasemeter.__init__ 方法中，您有：

        self.LIBC = ctypes.CDLL(self.path)

该代码在主进程中执行，而方法Phasemeter.start 在另一个进程中执行。因此，尝试使用pickle 将self.LIBC 从一个地址空间序列化/反序列化到另一个地址空间，但失败了。我建议将此属性的设置推迟到 start 方法之前，这样它就不必被腌制：

import multiprocessing

#%% Initialization

#%% Function definition

#Fix pickle problem ?
def start(Phasemeter_object):
    Phasemeter_object.start()
        
#%% Class definiton
class Phasemeter :
    # List of all phasemeters objects, accessed by calling Phasemeter.phase_list
    phase_list=[]
    
    #%% Initialization
    def __init__(self,*args,**kwargs) :
        #%% Robustness. Check type of passed arguments 

        #%% Path setting, parsing config file 
        
        #%% Option handling

        #%% Debug, used only if verbose argument is passed in start method 
        
        #%% Defining path to shared object file
        self.path=os.path.abspath(os.path.join(os.getcwd(),
                                               self.path_to_so_file))
                
        # Add object to phase_list, list used in multistart method 
        Phasemeter.phase_list.append(self)

    #%% Start
    def start(self):
        # LIBC is now an object and its method are C Code's functions
        self.LIBC = ctypes.CDLL(self.path)

        # Settings C library's signature datatype with ctypes data structre tool
        # INFO: To see all datas types that can be transmited to C Code
        # read ctypes documentation
        # First argument is int : argc
        # Second argument is string array : argv
        # Third is a string : path_to_log_file
        self.LIBC.lisaf_phasemeter_main.argtypes= [ctypes.c_int,
                                            ctypes.POINTER(ctypes.c_char_p),
                                            ctypes.c_char_p,]
        # Return type is int : Return code
        self.LIBC.lisaf_phasemeter_main.restypes = [ctypes.c_int,]

        #%% Marshalling data for C library
        # Create a string array with option list length size
        self.c_char_pointer_array = ctypes.c_char_p * len(self.options)
        # Encode option list
        self.encoded_options = [str.encode(str(i)) for i in self.options ]
        # Fill the string array with encoded strings
        # REMINDER: C code only understand encoded strings
        self.encoded_options = self.c_char_pointer_array (*self.encoded_options)


        #%% Calling C library wihth encoded options
        # If the logfile option is activated then the encoded 
        # string is transmited
        if self.encoded_path_to_log_file :
            self.status = self.LIBC.lisaf_phasemeter_main(
                len(self.encoded_options),
                self.encoded_options,
                self.encoded_path_to_log_file)
        # Otherwise None pointer is transmited
        else :
            self.status = self.LIBC.lisaf_phasemeter_main(len(self.encoded_options),
                                                      self.encoded_options,None)
#%% Multistart
if __name__ == '__main__' :
    # This function is used to start acquisition on multiple phasemeters 
    my_phase = Phasemeter(name="PH1")
    my_phase = Phasemeter(name="PH2")
    with multiprocessing.Pool(len(Phasemeter.phase_list)) as Process :
        Process.map(start,Phasemeter.phase_list)

作为奖励，您可以并行化更多代码。

【讨论】：

它就像一个魅力！万分感谢 ！您能否链接任何资源以帮助我理解为什么会这样？

并非如此。但应该清楚的是，传递给您的工作函数的所有参数（这里您使用map 指定参数）都需要从一个地址空间序列化/反序列化到另一个地址空间。您的Phasemeter.__init__ 方法正在由主进程运行，因此您传递给map 的对象具有self.LIBC = ctypes.CDLL(self.path) 属性，作为对象的一部分必须被腌制到新的地址空间但不能。解决方案是在Phasemeter.start中设置该属性，该属性在池进程中运行。

哦，好的，这是一个很好的答案，非常感谢。我希望我能给你一些 BAT 小费。祝你有美好的一天

“你在泡菜中”的评论让我很开心（也是一个很好的答案）