Inference

Inference is the stage where a trained model answers a prompt, as opposed to training, when its weights are still being adjusted. The model runs a forward pass over the input. For text it produces the reply one token at a time, each token conditioned on everything generated before it.

Throughput is usually bound by memory bandwidth rather than raw compute, because producing each token means streaming the weights through GPU memory at least once. A rough ceiling is memory bandwidth divided by model size. An RTX 4090 has about 1,008 GB/s of bandwidth. A 4 GB quantized model on that card tops out near 250 tokens per second before other overhead counts against it.

Memory bandwidth predicts local single-user inference speed better than a GPU's advertised TFLOPS. Check that figure on a spec sheet before the compute number.

Sources

Last verified August 29, 2026.