GPUDirect Storage

GPUDirect Storage (GDS) is an NVIDIA Magnum IO feature that sets up a direct memory access (DMA) path between storage and GPU memory, with no bounce buffer in CPU system memory. Applications reach it through the cuFile API in libcufile.so. Operating systems programming DMA engines cannot process GPU virtual addresses without help, so a kernel module, nvidia-fs.ko, supplies them, except on NVMe mounts from CUDA 12.8. NVIDIA names local NVMe and NVMe over Fabrics (NVMe-oF) as example storage.

The file has to be opened with O_DIRECT, although from CUDA Toolkit 12.2 (GDS 1.7.x) the library takes the O_DIRECT path by itself for page-aligned buffers, sizes and offsets. Where GDS cannot run, a compatibility mode stages data through CPU memory. Dynamic routing covers network file systems such as DDN EXAScaler, VAST NFS and WekaFS. NVIDIA's GDS release notes cover x86 and Grace-based platforms, and its developer page states GDS is not supported on Windows.

Check where the storage NIC sits relative to the GPU: NVIDIA notes that peer-to-peer traffic between a GPU and a storage NIC on different root ports may have higher latency and be less efficient than traffic under a PCIe switch. Run gdscheck on the target host first.

Sources

Last verified October 30, 2026.