Introduction

A true zombie process does not keep executing code and usually does not hold GPU memory by itself. In practice, when nvidia-smi still shows memory usage after a crashed job, the real problem is often a lingering CUDA context, an unkillable process in driver state, or another live process that inherited part of the workload. The fix starts with identifying what is actually alive before reaching for a GPU reset or reboot.

First: Verify Whether It Is Really a Zombie

On Linux, a zombie process shows state Z in ps. If the parent is init or systemd, it should usually be reaped quickly.

ps -eo pid,ppid,state,cmd | grep ' Z '
ps -p 12345 -o pid,ppid,state,cmd

If the process state is really Z, the process is already dead. That means the remaining GPU memory is not being held by that zombie in the normal sense.

Check What the GPU Driver Still Sees

Use NVIDIA tools first, because OS process state and GPU driver state can diverge.

nvidia-smi
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv

If a PID still appears there, inspect it from the OS side:

ps -p 12345 -o pid,ppid,state,cmd
cat /proc/12345/status

Sometimes the process is still alive but stuck in uninterruptible sleep rather than being a zombie.

Look for Other Processes Holding the Device

The visible memory consumer may not be the original training process. Wrapper processes, subprocesses, notebook kernels, or data loaders may still hold the device.

sudo fuser -v /dev/nvidia*
sudo lsof /dev/nvidia0

These commands often reveal another process that still has the GPU device open. If that process is genuinely alive, terminate it normally first, then escalate if needed.

Kill Live Processes, Not Zombies

If the PID is still running, try graceful termination before a hard kill:

kill 12345
sleep 2
kill -9 12345

This will not help for a real zombie, because a zombie is already dead. It can help for the much more common case where the process is hung and still owns the CUDA context.

Reset the GPU If the Driver Still Holds Context

If no useful process can be cleaned up but memory is still stuck in the driver, a GPU reset may release the state.

sudo nvidia-smi --gpu-reset -i 0

This only works in some configurations. It may fail if:

• the GPU is the primary display device
• another process is still attached
• the driver or kernel module is in a bad state

On shared servers, coordinate this carefully because a reset affects all workloads on that device.

Reboot Is the Last Reliable Escape Hatch

If the driver state is corrupted and reset fails, rebooting the machine is often the only reliable cleanup. That is not elegant, but it is sometimes the correct operational answer.

Before rebooting, collect enough information to prevent recurrence:

dmesg | tail -n 100
journalctl -k -n 100
nvidia-smi -q

Driver errors, Xid events, or repeated CUDA crashes usually point to the real underlying issue.

Prevent the Problem in Future Jobs

The best fix is to reduce the chance of leaked GPU state:

1. ensure training jobs handle signals and clean up workers
2. avoid leaving notebook kernels detached from job lifecycles
3. keep driver and CUDA runtime versions aligned
4. run GPU workloads under a scheduler that tracks and cleans job processes

On multi-user systems, job managers such as Slurm make cleanup more reliable than ad hoc shell sessions.

Common Pitfalls

• Assuming a true zombie process is actively holding GPU memory.
• Using kill -9 repeatedly on a PID that is already dead.
• Ignoring helper processes that still have /dev/nvidia* open.
• Resetting a shared GPU without checking who else is using it.
• Rebooting immediately without collecting logs that explain the failure mode.

Summary

• A real zombie usually does not own GPU memory in the normal way.
• Use ps, nvidia-smi, fuser, and lsof to identify the real holder of the device state.
• Kill live processes if they still exist, but do not expect that to fix a true zombie.
• Use nvidia-smi --gpu-reset when driver state is stuck and reset is supported.
• If driver corruption persists, reboot and capture logs so the root cause can be addressed.