PCIe switch

A PCIe switch is a chip that routes traffic between one upstream PCI Express port and several downstream ports. It fans a CPU's lanes out to more GPUs, drives or cards than the CPU could connect directly. Broadcom sells them as the PEX family. Unlike bifurcation, which only divides lanes, a switch forwards packets, so devices behind it can talk to each other without going through the CPU.

Broadcom's PEX89144 is a PCIe Gen 5 switch with 144 lanes, 72 ports, 115 ns latency and 49 W typical power. Supermicro's SYS-421GE-TNRT lists a PCIe 5.0 x16 switch in a dual-root layout, 13 PCIe 5.0 x16 slots and up to 10 double-width GPUs. The SYS-421GE-TNRT3 lists a PCIe 5.0 x16 CPU-to-GPU interconnect with no switch named, 8 slots and up to 8 GPUs. Per the kernel documentation, a transaction that crosses a switch can stay inside the PCIe hierarchy and never reach the root port, depending on ACS settings.

Ask whether a multi-GPU server is switched or direct before comparing slot counts. If GPUs need to exchange data, put them behind the same switch. Where the path reaches the host bridge, the Linux kernel blocks peer-to-peer by default unless the hardware is on its allow list.

Sources

Last verified October 30, 2026.