Cluster computing

Cluster computing links many individual servers, called nodes, into a single system that runs one workload together. It splits a job across compute nodes, while a head node, also called a login node, handles submission and scheduling. A high speed interconnect such as InfiniBand or fast Ethernet moves data between nodes quickly enough that the group behaves like one machine.

A small cluster might have 8 compute nodes, each with 8 GPUs, for 64 GPUs total. Tie them together with 400 Gb/s InfiniBand and a scheduler like Slurm, which queues jobs and assigns them to free nodes. Larger clusters run into hundreds or thousands of nodes, where interconnect bandwidth and topology matter as much as node count. Once a job spans more than a handful of machines, topology decides how much of the compute is actually usable.

When specing a cluster, the interconnect and scheduler matter as much as the compute nodes themselves. A fast GPU stalls waiting on data if the network between nodes is undersized.

Sources

Last verified August 29, 2026.