InfiniBand vs Ethernet

InfiniBand and Ethernet are the two fabrics used to network GPU nodes into a cluster. The choice turns on how much lossless, low-latency behavior is built into the hardware. InfiniBand was designed for high-performance computing: credit-based flow control keeps it from dropping packets, and adaptive routing and RDMA are native to the protocol. Ethernet is the general-purpose standard used everywhere else, and it reaches similar ground through RoCE, RDMA over Converged Ethernet, plus priority flow control and careful congestion tuning.

Both run at 400 Gb/s per port on current switches, InfiniBand NDR against 400 gigabit Ethernet, with 800 Gb/s parts shipping on each side. Nvidia bought Mellanox in 2020, builds its DGX SuperPOD reference clusters on InfiniBand, and also sells Spectrum-X, an Ethernet platform aimed at the same GPU traffic. Large clusters commonly run two networks: a tuned fabric for GPU-to-GPU collectives, and ordinary Ethernet for management, provisioning, and storage.

When comparing cluster quotes, ask which fabric carries collective traffic between GPUs and how the switches behave under congestion. That answer predicts scaling efficiency better than the port speed on the datasheet.

Sources

Last verified August 29, 2026.