Theano：设备 gpu 初始化失败！原因=CNMEM_STATUS_OUT_OF_MEMORY

Posted 2023-03-16

【中文标题】Theano：设备 gpu 初始化失败！原因=CNMEM_STATUS_OUT_OF_MEMORY

【英文标题】：Theano: Initialisation of device gpu failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY

【发布时间】：2016-07-06 02:59:50

【问题描述】：

我正在运行Keras 的example kaggle_otto_nn.py，后端为theano。当我设置cnmem=1时，出现以下错误：

cliu@cliu-ubuntu:keras-examples$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,lib.cnmem=1 python kaggle_otto_nn.py
Using Theano backend.
ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1

/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
  "downsample module has been moved to the theano.tensor.signal.pool module.")
Traceback (most recent call last):
  File "kaggle_otto_nn.py", line 28, in <module>
    from keras.models import Sequential
  File "build/bdist.linux-x86_64/egg/keras/models.py", line 15, in <module>
  File "build/bdist.linux-x86_64/egg/keras/backend/__init__.py", line 46, in <module>
  File "build/bdist.linux-x86_64/egg/keras/backend/theano_backend.py", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/__init__.py", line 111, in <module>
    theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
  File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/sandbox/cuda/tests/test_driver.py", line 38, in test_nvidia_driver1
    if not numpy.allclose(f(), a.sum()):
  File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
RuntimeError: Cuda error: kernel_reduce_ccontig_node_97496c4d3cf9a06dc4082cc141f918d2_0: out of memory. (grid: 1 x 1; block: 256 x 1 x 1)

Apply node that caused the error: GpuCAReduceadd1(<CudaNdarrayType(float32, vector)>)
Toposort index: 0
Inputs types: [CudaNdarrayType(float32, vector)]
Inputs shapes: [(10000,)]
Inputs strides: [(1,)]
Inputs values: ['not shown']
Outputs clients: [[HostFromGpu(GpuCAReduceadd1.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

似乎我无法将cnmem 设置为一个非常大的值（大约 > 0.9），因为它可能会导致 GPU 的内存溢出。当我设置cnmem=0.9 时，它工作正常。根据this，它

表示内存池的起始大小（以 MB 或总 GPU 内存的百分比为单位）。

和

这可能会导致内存碎片。所以如果你在使用cnmem时出现内存错误，试着在开始时分配更多的内存或者禁用它。如果您尝试这样做，请在 :reftheano-dev 上报告您的结果。

但是如果我得到内存错误，我为什么要在开始时分配更多的内存？在我的情况下，在开始时分配更多的内存似乎会导致错误。

【参考方案1】：

这是基于this解决的问题。

如here所示，cnmem实际上只允许分配为float。

0：未启用。

0

> 1：使用此数字，以兆字节 (MB) 为单位。

因此，如果 cnmem=1.0 而不是 cnmem=1，它将起作用。