Transformer

A transformer is a neural network architecture built around self-attention, a mechanism that lets every token in a sequence weigh every other token directly. It replaced the recurrent networks that used to process sequences one step at a time, and it's the architecture behind GPT, Llama, and BERT.

Because self-attention computes all token relationships in parallel rather than in sequence, a transformer trains efficiently on GPUs, which are built for exactly that kind of parallel matrix math. A recurrent network's step by step dependency chain can't use a GPU's parallelism nearly as well. The tradeoff is memory. Attention cost grows with the square of sequence length, so a 128,000 token context window takes far more memory and compute than an 8,000 token window.

For hardware specing, that quadratic attention cost is why long context windows drive VRAM requirements up sharply, independent of model parameter count.

Sources

Last verified August 29, 2026.