Nvidia & Docker – Failed to initialize NVML

09 Feb 2023

Reading time ~2 minutes

This one is not going to be crazy technical but I haven’t written in a while and thought this might be helpful to many out there.

After containerizing my GPUs workloads in my home lab, I noticed that seemingly randomly my GPU-enabled containers would start throwing the following error (e.g. when running nvidia-smi):

Failed to initialize NVML: Unknown Error

As it turns out, dozens have been reporting the issue in the past couple of years – but never were there a solid solution. It was pointed out that it wasn’t “random” after all, and that running systemctl daemon-reload would instantly trigger the issue, and that the container(s) had to be restarted before the GPU(s) could be used within the containers once again. It all started with Docker 20.10, which then started using cgroup v2 (aka unified cgroup hierarchy) instead of cgroupfs (if enabled on the operating system), and as distros started enabling it by default (e.g. Ubuntu 22.04).

Under the hood, the nvidia-container-runtime runc prestart hook injects the GPU(s)’s userland drivers, char devices and associated cgroups through libnvidia-container.

nvidia-container-runtime's prestart hook

Unfortunately, the hook does all of this “behind the back” of runc which requires /dev/char symlinks, and a systemd reload reverts that work while re-evaluating all of the cgroups rules. Until Docker adds supports for the Container Device Interface (CDI) (now in CRI-O & containerd) and until the nvidia-container-runtime becomes CDI-aware, we need to work around the issue by setting up permanent links to the char devices.

On Kubernetes with the Nvidia GPU Operator

The Nvidia GPU Operator v22.9.2 released on Jan 30, 2023 now handles the symlinks in their validator pod. If not using the operator, you may refer to the workaround below.

On standalone Docker with the nvidia-ctk

The Nvidia Container Toolkit v1.12.0 released on Feb 6, 2023 provides a new utility to create the char symlinks automatically:

$ nvidia-ctk system create-dev-char-symlinks --create-all
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/195:254 => /dev/nvidia-modeset"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/195:255 => /dev/nvidiactl"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/510:0 => /dev/nvidia-uvm"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/510:1 => /dev/nvidia-uvm-tools"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:1 => /dev/nvidia-caps/nvidia-cap1"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:2 => /dev/nvidia-caps/nvidia-cap2"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/195:0 => /dev/nvidia0"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:3 => /dev/nvidia-caps/nvidia-cap3"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:4 => /dev/nvidia-caps/nvidia-cap4"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:5 => /dev/nvidia-caps/nvidia-cap5"
time="2023-02-10T02:45:34Z" level=info msg="Creating link /dev/char/236:6 => /dev/nvidia-caps/nvidia-cap6"
[...]

It can be conveniently integrated as a udev rule for persistence across reboots:

# /lib/udev/rules.d/71-nvidia-dev-char.rules
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/bin/nvidia-ctk system create-dev-char-symlinks --create-all"

If you’re using the Nvidia GPU Driver Container, you must also specify the –driver-root= option pointing to the directory where the driver and device nodes are created.

But..

Unfortunately, for now, we also still need to pass the devices directly into the containers to prevent the issue from happening upon systemd reloads, which arguably reduces portability:

# docker-compose.yml
version: "3.9"
 
services:
  cuda:
    image: nvidia/cuda:12.0.1-runtime-ubuntu22.04
    devices:
      - /dev/nvidia0
      - /dev/nvidiactl
      - /dev/nvidia-uvm
    command: ["nvidia-smi"]
    deploy:
      resources:
        reservations:
          devices:
            - count: all
              capabilities: [gpu]
              driver: nvidia