Pybind11 多处理挂起
Posted
技术标签:
【中文标题】Pybind11 多处理挂起【英文标题】:Pybind11 multiprocessing hangs 【发布时间】:2020-06-05 06:54:09 【问题描述】:我正在编写一个使用 Pybind11 嵌入 Python 解释器的应用程序(Windows,64 位,Visual C++ 2017)。从 Python 中,我需要生成多个进程,但它似乎不起作用。我尝试以下代码作为测试:
import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
__name__ = "__mp_main__"
def print_cube(num):
"""
function to print cube of given num
"""
print("Cube: ".format(num * num * num))
def print_square(num):
"""
function to print square of given num
"""
print("Square: ".format(num * num))
print(__name__)
if __name__ == "__main__":
print(prefix, "checkpoint 1")
# creating processes
p1 = multiprocessing.Process(target=print_square, args=(10, ))
p1.daemon = True
p2 = multiprocessing.Process(target=print_cube, args=(10, ))
# starting process 1
p1.start()
print(prefix, "checkpoint 2")
# starting process 2
p2.start()
print(prefix, "checkpoint 3")
# wait until process 1 is finished
print(prefix, "checkpoint 4")
p1.join()
print(prefix, "checkpoint 5")
# wait until process 2 is finished
p2.join()
print(prefix, "checkpoint 6")
# both processes finished
print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)
在命令提示符下使用 Python 运行它,我得到:
This is the name of the script: mp.py
Number of arguments: 1
The arguments are: ['mp.py']
__main__
12872- checkpoint 1
12872- checkpoint 2
This is the name of the script: C:\tmp\mp.py
Number of arguments: 1
The arguments are: ['C:\\tmp\\mp.py']
__mp_main__
7744- checkpoint 7
Square: 100
12872- checkpoint 3
12872- checkpoint 4
12872- checkpoint 5
This is the name of the script: C:\tmp\mp.py
Number of arguments: 1
The arguments are: ['C:\\tmp\\mp.py']
__mp_main__
15020- checkpoint 7
Cube: 1000
12872- checkpoint 6
Done!
12872- checkpoint 7
这是正确的。如果我在 Pybind11 的 C++ 项目中尝试相同的操作,则输出为:
This is the name of the script: C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe
Number of arguments: 1
The arguments are: ['C:\\AGPX\\Documenti\\TestPyBind\\x64\\Debug\\TestPyBind.exe']
__main__
4440- checkpoint 1
This is the name of the script: C:\AGPX\Documenti\TestPyBind\x64\Debug\TestPyBind.exe
Number of arguments: 4
The arguments are: ['C:\\AGPX\\Documenti\\TestPyBind\\x64\\Debug\\TestPyBind.exe', '-c', 'from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=4440, pipe_handle=128)', '--multiprocessing-fork']
__mp_main__
10176- checkpoint 7
请注意,在这种情况下,变量 __name__
始终设置为“__main__
”,因此我必须手动将其(对于衍生的进程)更改为“__mp_main__
”(我可以检测到孩子进程感谢 sys.argv)。这是第一个奇怪的行为。
父进程的 pid 为 4440,我可以在进程资源管理器中看到该进程。
第一个子进程的 pid 为 10176,它到达末尾“检查点 7”,进程从进程资源管理器中消失。但是,主进程没有打印“检查点 2”,看起来它挂在“p1.start()”上,我不明白为什么。
完整的C++代码是:
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <pybind11/embed.h>
#include <iostream>
namespace py = pybind11;
using namespace py::literals;
int wmain(int argc, wchar_t **argv)
py::initialize_interpreter();
PySys_SetArgv(argc, argv);
std::string pyCode = std::string(R"(
import multiprocessing
import os
import sys
import time
print("This is the name of the script: ", sys.argv[0])
print("Number of arguments: ", len(sys.argv))
print("The arguments are: " , str(sys.argv))
prefix=str(os.getpid())+"-"
if len(sys.argv) > 1:
__name__ = "__mp_main__"
def print_cube(num):
"""
function to print cube of given num
"""
print("Cube: ".format(num * num * num))
def print_square(num):
"""
function to print square of given num
"""
print("Square: ".format(num * num))
print(__name__)
if __name__ == "__main__":
print(prefix, "checkpoint 1")
# creating processes
p1 = multiprocessing.Process(target=print_square, args=(10, ))
p1.daemon = True
p2 = multiprocessing.Process(target=print_cube, args=(10, ))
# starting process 1
p1.start()
print(prefix, "checkpoint 2")
# starting process 2
p2.start()
print(prefix, "checkpoint 3")
# wait until process 1 is finished
print(prefix, "checkpoint 4")
p1.join()
print(prefix, "checkpoint 5")
# wait until process 2 is finished
p2.join()
print(prefix, "checkpoint 6")
# both processes finished
print("Done!")
print(prefix, "checkpoint 7")
time.sleep(10)
)");
try
py::exec(pyCode);
catch (const std::exception &e)
std::cout << e.what();
py::finalize_interpreter();
谁能告诉我如何解决这个问题?
提前致谢(我为我的英语道歉)。
【问题讨论】:
【参考方案1】:好的,感谢这个链接:https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows,我解决了这个奇怪的问题(这似乎与 Windows 相关)。
这不是 Pybind11 问题,而是 Python C API 本身。
您可以通过将sys.executable
设置为python 解释器可执行文件(python.exe) 的路径并将python 代码写入文件并将路径设置为__file__
变量来解决此问题。也就是说,我必须添加:
import sys
sys.executable = "C:\\Users\\MyUserName\\Miniconda3\\python.exe"
__file__ = "C:\\tmp\\run.py"
而我需要将python代码写入__file__
指定的文件,即:
FILE *f = nullptr;
fopen_s(&f, "c:\\tmp\\run.py", "wt");
fprintf(f, "%s", pyCode.c_str());
fclose(f);
就在执行py::exec(pyCode)
之前。
另外代码:
if len(sys.argv) > 1:
__name__ = "__mp_main__"
不再需要。但是,请注意,以这种方式运行的进程不再嵌入,不幸的是,如果您想直接将 C++ 模块传递给它们,则无法这样做。
希望这可以帮助其他人。
【讨论】:
以上是关于Pybind11 多处理挂起的主要内容,如果未能解决你的问题,请参考以下文章