Tensor parallelism

Tensor parallelism splits individual weight matrices across multiple GPUs, so each one holds a slice of every layer instead of a full copy. Every GPU computes its portion of each matrix multiplication, and the results sync across GPUs before the next layer runs. Pipeline parallelism does something different, assigning whole layers to different GPUs rather than splitting the math inside each one.

The price of splitting inside a layer is communication. GPUs sync several times per layer, not once per pipeline stage, so the interconnect has to keep up. Training a 70 billion parameter model commonly uses tensor parallelism across 4 or 8 GPUs inside a single server. NVLink on an H100 gives each GPU up to 900 GB/s, fast enough to hide most of that sync overhead. Spread the same split across servers on a slower network and the GPUs spend more time waiting than computing.

This is why multi-GPU servers built for training pair every GPU with a high-bandwidth link rather than leaving them on standard PCIe. If a workload uses tensor parallelism, interconnect bandwidth matters as much as GPU count.

Sources

Last verified August 29, 2026.