IPython ipyparallel map_sync ImportError

Posted 2023-02-22

【中文标题】IPython ipyparallel map_sync ImportError

【英文标题】：

【发布时间】：2015-09-28 03:49:33

【问题描述】：

我是ipyparallel 的新手，我想使用这个包在我的机器学习应用程序中实现并行计算。

以下是对ipyparallel的测试，我在func.py文件中定义了一个名为add的函数，在test.py文件中定义了main函数。

func.py 的代码是：

#!/usr/bin/env python
# coding=utf-8

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers

test.py 的代码是：

#!/usr/bin/env python
# coding=utf-8

from func import add
from ipyparallel import Client

if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3]
    print rc[0].map_sync(add, [1, 2, 3, 4])

既然你知道map可以正常运行，但是运行map_sync时，命令行返回：

☁  test  python test.py 
[[2], [3], [4]]
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print rc[0].map_sync(add, [1, 2, 3, 4])
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 353, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 54, in sync_results
    ret = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 618, in map
    return pf.map(*sequences)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 268, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 75, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 251, in __call__
    return r.get()
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/asyncresult.py", line 104, in get
    raise self._exception
ipyparallel.error.CompositeError: one or more exceptions from call to method: add
[0:apply]: ImportError: No module named func

如果我在 test.py 文件中定义函数，map_sync 可以运行：

#!/usr/bin/env python
# coding=utf-8

#from func import add
from ipyparallel import Client

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers


if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3])

    print rc[0].map_sync(add, [1, 2, 3, 4])

结果是：

☁  test  python test.py
[[2], [3], [4]]
[[2], [3], [4], [5]]

我想知道map_sync如何使用其他文件中定义的函数？以及如何导入这些功能？由于from py_file import func 不适用于map_sync。

【参考方案1】：

需要的模块应该复制（或者，您可以推送或模块）到engine machines，并且应该在engine machines上安装3方包，否则会出现ImportError。

但是，在运行程序时，你应该运行：

$ ipcontroller --ip=client_ip --reuse

在client machine上，会在/home/user/.ipython/profile_default/security目录下生成2个文件

$ ls /home/user/.ipython/profile_default/security 
ipcontroller-client.json  ipcontroller-engine.json

所以，ipcontroller-client.json 和 ipcontroller-engine.json 都必须复制到 engine machines 并运行

$ ipengine --file=/path/to/ipcontroller-engine.json

在engine machines上，所以parallel computing环境已经搭建好了。

接下来，您可以定义您的 parallel computing 任务并运行您的程序。