InfiniBand vs Ethernet

InfiniBand and Ethernet are the two fabrics used to network GPU nodes into a cluster. The choice turns on how much lossless, low-latency behavior is built into the hardware. InfiniBand moves data with remote direct memory access (RDMA). Its credit-based flow control lets a sender transmit only when the receiver has advertised buffer space, so the fabric does not drop packets. Ethernet is the general-purpose standard used everywhere else. It reaches similar ground through RDMA over Converged Ethernet (RoCE), which relies on priority flow control (PFC) and explicit congestion notification (ECN).

The InfiniBand Trade Association lists InfiniBand links at up to 800 Gb/s. NVIDIA's DGX B300 SuperPOD reference architecture builds its compute fabric on XDR InfiniBand or Spectrum-X Ethernet, both at 800 Gb/s. NVIDIA rates XDR at a peak of 800 Gb/s per direction. The InfiniBand option uses Quantum-3 Q3400-RA switches. In that design, Spectrum-X pairs Spectrum-4 switches with ConnectX-8 adapters. The design's InfiniBand storage fabric uses 400 Gb/s NDR switches.

Port speed no longer separates the two fabrics. Ask which fabric carries GPU-to-GPU collective traffic in the quoted design and whether it matches the vendor's reference architecture. On Ethernet, also ask who configures PFC and ECN on the switches, since RoCE depends on both.

Sources

Last verified October 7, 2026.