尝试在 AWS EMR 中获得 GPU 支持时出现错误“一个 NVIDIA 内核模块 'nvidia' 似乎已加载到您的内核中”

Posted 2023-04-15

【中文标题】尝试在 AWS EMR 中获得 GPU 支持时出现错误“一个 NVIDIA 内核模块 \'nvidia\' 似乎已加载到您的内核中”

【英文标题】：Error 'An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel' when trying to get GPU support in AWS EMR尝试在 AWS EMR 中获得 GPU 支持时出现错误“一个 NVIDIA 内核模块 'nvidia' 似乎已加载到您的内核中”

【发布时间】：2021-06-15 06:07:23

【问题描述】：

我在尝试将公开可用的 AWS DL AMI 应用到 EMR 集群（emr-6.2.0、spark 3.0.1）时遇到以下错误

来自应用阶段的 puppet.log 文件：

ERROR: An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel. This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading. Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver. If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occured that has corrupted an NVIDIA kernel module's usage count, for which the simplest remedy is to reboot your computer.

集群内的Linux版本是

Linux ip-10-14-1-68 4.14.26-46.32.amzn1.x86_64 #1 SMP Fri Mar 30 22:29:54 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

DLAMI = ami-058964fc61ad6c7c8 TensorFlow 版本 = 2.4.1

是否有针对此问题的修复或解决方法？也许有一些干净的方法可以删除 Nvidia 并允许重新安装它或杀死一些持有它的进程？或者别的什么。

【参考方案1】：

如果您尝试安装新版本的 nvidia 驱动程序，您需要确保没有其他进程使用旧版本的驱动程序。

您可以通过以下方式找到这些流程：

sudo lsof /dev/nvidia*

然后，通过对应的PID 和kill -9 PID 杀死进程。

【讨论】：

谢谢，@OmriKaduri。我们已经尝试过非常相似的东西。基本上，在引导操作中：ps -ef | grep [n]vidia | awk ' print $2 ' | xargs kill -9 但是，通过该命令：Shutting down nvidia processes...listing nvidia procs after the proc kill attempt:root 6208 2 0 15:02 ? 00:00:00 [irq/109-nvidia]root 6209 2 0 15:02 ? 00:00:00 [nvidia] 这些进程不容易被杀死。必须弄清楚如何彻底关闭它们。