cProfiler 与多处理一起工作很奇怪

Posted 2023-02-16

【中文标题】cProfiler 与多处理一起工作很奇怪

【英文标题】：cProfiler working weirdly with multiprocessing

【发布时间】：2019-05-16 09:01:14

【问题描述】：

我收到此代码的错误：

from pathos.multiprocessing import ProcessingPool
def diePlz(im):
    print('Whoopdepoop!')   
def caller():
    im = 1
    pool = ProcessingPool()
    pool.map(diePlz,[im,im,im,im]) 

if __name__=='__main__':
    caller()    

当我使用 cProfiler 运行它时：(python3 -m cProfile testProfiler.py)

multiprocess.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/rohit/.local/lib/python3.6/site-packages/pathos/helpers/mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "testProfiler.py", line 3, in diePlz
    print('Whoopdepoop!')
NameError: name 'print' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/cProfile.py", line 160, in <module>
    main()
  File "/usr/lib/python3.6/cProfile.py", line 153, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib/python3.6/cProfile.py", line 20, in runctx
    filename, sort)
  File "/usr/lib/python3.6/profile.py", line 64, in runctx
    prof.runctx(statement, globals, locals)
  File "/usr/lib/python3.6/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "testProfiler.py", line 11, in <module>
    caller()    
  File "testProfiler.py", line 8, in caller
    pool.map(diePlz,[im,im,im,im]) 
  File "/home/rohit/.local/lib/python3.6/site-packages/pathos/multiprocessing.py", line 137, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
NameError: name 'print' is not defined

但是当我在没有cProfiler 的情况下运行它时：

$ python3 testProfiler.py 
Whoopdepoop!
Whoopdepoop!
Whoopdepoop!
Whoopdepoop!

我提供的代码是该问题的最小工作示例。我想调试一个更大的代码，但由于cProfiler 不断引发奇怪的错误，我无法这样做。

在这种情况下，重点是

NameError: name 'print' is not defined

这意味着python3 无法识别print 本身。在我的代码中，它无法识别range。

【参考方案1】：

所以，我意识到这是在原始帖子之后很长时间，但我遇到了完全相同的问题。

在我的情况下，我得到了与原始帖子完全相同的错误 - python 内置函数，如 print() 或 len() 导致如下错误：

NameError: name 'len' is not defined

我目前正在运行multiprocess 0.70.11.1 版和dill 0.3.3 版（使基于进程的并行性工作的悲惨组件）。

根据我在问题评论中发现的内容：https://github.com/uqfoundation/pathos/issues/129#issuecomment-536081859 软件包作者之一建议尝试：

import dill
dill.settings['recurse'] = True

至少在我的情况下，以上解决了错误！