Local LLM

A local LLM is a language model that runs inference on hardware you control, a workstation, server, or laptop, instead of a hosted API. It loads the weights into GPU VRAM and generates every token on that machine, so prompts and outputs never leave your network. The tradeoff is that you own the compute problem, and the model has to fit the hardware you actually have.

Sizing comes down to VRAM. A 7 billion parameter model needs roughly 14 GB at 16-bit precision, or about 4 GB once quantized to 4-bit. Either way it fits a single 24 GB consumer card. A 70 billion parameter model needs around 140 GB at 16-bit, dropping to roughly 35 to 40 GB quantized. That fits one 48 GB workstation GPU, or splits across two 24 GB cards. Tools like llama.cpp, Ollama, and vLLM handle the loading and quantization.

Match the model to the job before buying the hardware. A quantized 7 to 13 billion parameter model covers most coding and chat work on one high-end GPU. The largest open models need several cards and a memory budget to match.

Sources

Last verified August 29, 2026.