CUDA 驱动程序版本对于 CUDA 运行时版本、CUDA 版本 10.0 不足

Posted 2023-04-15

【中文标题】CUDA 驱动程序版本对于 CUDA 运行时版本、CUDA 版本 10.0 不足

【英文标题】：CUDA driver version is insufficient for CUDA runtime version , CUDA version 10.0

【发布时间】：2019-12-31 18:38:51

【问题描述】：

我在 GPU 上运行 keras 时收到此错误。

  File "modeltra.py", line 165, in <module>
    x_train = a_model.predict(y_training)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2697, in __call__
    if hasattr(get_session(), '_make_callable_from_options'):
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 186, in get_session
    _SESSION = tf.Session(config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1570, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 693, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

我的GPU信息如下

(gp_ker_pip) [hgam@xx]$ nvidia-smi
Tue Aug 27 17:29:34 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  On   | 00000000:03:00.0 Off |                    0 |
| N/A   34C    P0    33W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  On   | 00000000:82:00.0 Off |                    0 |
| N/A   32C    P0    31W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

(gp_ker_pip) [hgam@xx]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

首先使用安装 tensorflow pip install tensorflow-gpu

并安装了 keras conda install keras-gpu

我应该遵循什么来完成这项工作？还有如何查看 CUDA 运行时版本和驱动版本？

PS： 我还使用

降级了 cuda 版本

>    export PATH=$PATH:/usr/local/cuda-9.0/bin
>    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.0/lib64

我仍然得到错误。

【问题讨论】：

What should I follow to make this work? 为您的 GPU 安装最新的驱动程序

【参考方案1】：

按照@Robert 的建议安装 CUDA 10.1 后，此问题已解决。

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:03:00.0 Off |                    0 |
| N/A   39C    P0    64W /  70W |  14503MiB / 15079MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:82:00.0 Off |                    0 |
| N/A   35C    P0    25W /  70W |    111MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+