我可以在 python 程序上使用 cuda-memcheck
Posted
技术标签:
【中文标题】我可以在 python 程序上使用 cuda-memcheck【英文标题】：Can I use cuda-memcheck on a python program 【发布时间】：2019-07-07 21:01:05 【问题描述】：我有很多 cuda 内核要测试，所以我希望能够通过从 python 程序执行它们来测试它们（python 程序调用一个启动 cuda 内核的库），即类似于cuda-memcheck --leak-check full -- "python my_test_program.py"。与编写 C++ 程序来做同样的事情相比，这将节省时间。这是否与测试内核形成 *.cc 或 *.cu 程序的方式相同？
【问题讨论】：
我同时使用cuda-memcheck 和 nvprof 与 python + numba/cuda 和 python + pycuda。
【参考方案1】：
是的，您可以在 python 程序中使用 cuda-memcheck 和 nvprof 等工具。例如，考虑以下（故意）有缺陷的 Numba 示例：
$ cat numbatest.py
from __future__ import division
from numba import cuda
import numpy
import math
# CUDA kernel
@cuda.jit
def my_kernel(io_array):
pos = cuda.grid(1)
if pos > io_array.size:
io_array[pos] *= 2 # do the computation
# Host code
data = numpy.ones(62)
threadsperblock = 64
blockspergrid = math.ceil(data.shape[0] / threadsperblock)
my_kernel[blockspergrid, threadsperblock](data)
print(data)
这个内核在运行时会产生越界的内存访问。使用 cuda-memcheck 执行此操作表明情况确实如此：
$ cuda-memcheck python numbatest.py
========= CUDA-MEMCHECK
========= Invalid __global__ read of size 8
========= at 0x000000b8 in cudapy::__main__::my_kernel$241(Array<double, int=1, C, mutable, aligned>)
========= by thread (63,0,0) in block (0,0,0)
========= Address 0x7040601f8 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so (cuLaunchKernel + 0x2cd) [0x2486ed]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call_unix64 + 0x4c) [0x6adc]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call + 0x1f2) [0x6282]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (_ctypes_callproc + 0x2ce) [0x12e4e]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so [0x13885]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x19642e]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
========= Host Frame:python [0x196d8b]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
========= Host Frame:python [0x196124]
========= Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
========= Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
========= Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python [0x171127]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x19642e]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
========= Host Frame:python [0x196124]
========= Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
========= Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
========= Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python [0x171127]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x196124]
========= Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
========= Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
========= Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python [0x171127]
========= Host Frame:python (_PyObject_FastCallDict + 0x8b) [0x1135bb]
========= Host Frame:python [0x19cd6e]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
========= Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
========= Host Frame:python (PyEval_EvalCode + 0x1c) [0x19887c]
========= Host Frame:python [0x219074]
========= Host Frame:python (PyRun_FileExFlags + 0xa1) [0x219471]
========= Host Frame:python (PyRun_SimpleFileExFlags + 0x1c3) [0x219673]
========= Host Frame:python (Py_Main + 0x650) [0x21d1b0]
========= Host Frame:python (main + 0xee) [0xe4b4e]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21f45]
========= Host Frame:python [0x1c61a8]
=========
========= Program hit CUDA_ERROR_LAUNCH_FAILED (error 719) due to "unspecified launch failure" on CUDA API call to cuMemcpyDtoH_v2.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so (cuMemcpyDtoH_v2 + 0x173) [0x2436a3]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call_unix64 + 0x4c) [0x6adc]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/../../libffi.so.6 (ffi_call + 0x1f2) [0x6282]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (_ctypes_callproc + 0x2ce) [0x12e4e]
========= Host Frame:/opt/miniconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so [0x13885]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python (PyEval_EvalCodeEx + 0x966) [0x198116]
========= Host Frame:python [0x1988f4]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x196124]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
========= Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
========= Host Frame:python [0x198a06]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x1962c6]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
========= Host Frame:python [0x19642e]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
========= Host Frame:python [0x19642e]
========= Host Frame:python [0x196fc1]
========= Host Frame:python [0x19ccf5]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x10c8) [0x1c04d8]
========= Host Frame:python [0x196124]
========= Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
========= Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
========= Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python [0x171127]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x198a) [0x1c0d9a]
========= Host Frame:python [0x196124]
========= Host Frame:python (_PyFunction_FastCallDict + 0x1bc) [0x19749c]
========= Host Frame:python (_PyObject_FastCallDict + 0x26f) [0x11379f]
========= Host Frame:python (_PyObject_Call_Prepend + 0x63) [0x118303]
========= Host Frame:python (PyObject_Call + 0x3e) [0x1131de]
========= Host Frame:python [0x171127]
========= Host Frame:python (_PyObject_FastCallDict + 0x8b) [0x1135bb]
Traceback (most recent call last):
========= Host Frame:python [0x19cd6e]
========= Host Frame:python (_PyEval_EvalFrameDefault + 0x30a) [0x1bf71a]
File "numbatest.py", line 17, in <module>
========= Host Frame:python (PyEval_EvalCodeEx + 0x329) [0x197ad9]
========= Host Frame:python (PyEval_EvalCode + 0x1c) [0x19887c]
========= Host Frame:python [0x219074]
========= Host Frame:python (PyRun_FileExFlags + 0xa1) [0x219471]
========= Host Frame:python (PyRun_SimpleFileExFlags + 0x1c3) [0x219673]
========= Host Frame:python (Py_Main + 0x650) [0x21d1b0]
========= Host Frame:python (main + 0xee) [0xe4b4e]
=========
my_kernel[blockspergrid, threadsperblock](data)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 704, in __call__
cfg(*args)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 484, in __call__
sharedmem=self.sharedmem)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 586, in _kernel_call
wb()
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/compiler.py", line 601, in <lambda>
retr.append(lambda: devary.copy_to_host(val, stream=stream))
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
return fn(*args, **kws)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 241, in copy_to_host
_driver.device_to_host(hostary, self, self.alloc_size, stream=stream)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1606, in device_to_host
fn(host_pointer(dst), device_pointer(src), size, *varargs)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
self._check_error(fname, retcode)
File "/opt/miniconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [719] Call to cuMemcpyDtoH results in CUDA_ERROR_LAUNCH_FAILED
========= ERROR SUMMARY: 2 errors
这将适用于任何使用 CUDA 加速库（Tensorflow、PyTorch 等）或生成 CUDA 代码并运行它（Numba、PyCUDA）的 Python 进程。
【讨论】：
以上是关于我可以在 python 程序上使用 cuda-memcheck的主要内容，如果未能解决你的问题，请参考以下文章
我可以在 ASE 上移植我现有的 python 应用程序吗？
使用 cx_freeze 在 Mac、Linux 和 Windows 上分发 python：我可以从一个平台生成所有应用程序吗？
使用 Python 的 Tkinter，我如何保存数据以仅在该应用程序上使用？
如何在我的桌面（Windows 7）上使用我的python程序在没有notify2，notify-send，pqt5的桌面上显示简单的通知？