How much VRAM does Mistral Medium 3.5 128B need?
Mistral Medium 3.5 128B needs 72.1 GB for its weights at 4-bit, 75.9 GB with an 8K context and 84.2 GB with 32K.
- Parameters
- 128B
- Longest context
- 256K
- Released
- Mar 2026
- License
- Mistral Modified MIT
- Strongest at
- Chat, Coding, Agents, Docs
Memory at each precision
| Precision | Weights | With 8K | With 32K | With 128K |
|---|---|---|---|---|
| As released (FP8) | 124 GB | 128 GB | 137 GB | 170 GB |
| 6-bit | 97.5 GB | 101 GB | 110 GB | 143 GB |
| 5-bit | 84.6 GB | 88.4 GB | 96.7 GB | 130 GB |
| 4-bit | 72.1 GB | 75.9 GB | 84.2 GB | 117 GB |
| 3-bit | 58.1 GB | 62.0 GB | 70.2 GB | 103 GB |
| 2-bit | 44.6 GB | 48.4 GB | 56.7 GB | 89.7 GB |
| 1.6-bit | 24.5 GB | 28.4 GB | 36.6 GB | 69.6 GB |
One conversation, 16-bit context cache, runtime overhead included.
What runs it
| Machine | Best fit | Context | Speed | First reply |
|---|---|---|---|---|
| RTX 3060 12 GB | Too big | |||
| RTX 4060 Ti 16 GB | Too big | |||
| RTX 3090 | Too big | |||
| RTX 4090 | Too big | |||
| RTX 5090 | Too big | |||
| 2× RTX 3090 | Too big | |||
| RTX PRO 6000 Blackwell | 4-bit | 52K | 16 tok/s | 2.1 s |
| M4 Max 128 GB | 5-bit | 16K | 3.9 tok/s | 43 s |
| M5 Max 128 GB | 5-bit | 16K | 4.3 tok/s | 13 s |
| Ryzen AI Max+ 395 128 GB | 5-bit | 16K | 1.7 tok/s | 25 s |
| DGX Spark 128 GB | 6-bit | 44K | 1.8 tok/s | 8.7 s |
| M3 Ultra 256 GB | FP8 | 160K | 3.9 tok/s | 24 s |
| M5 Ultra 512 GB | FP8 | 256K | 5.7 tok/s | 7.6 s |
| H100 SXM | 3-bit | 48K | 35 tok/s | 1.7 s |
| DGX Station GB300 | FP8 | 256K | 34 tok/s | 0.7 s |
| 8× H200 NVL | FP8 | 256K | 24 tok/s | 0.3 s |
Best fit is the highest quality, up to 8-bit, that leaves 16K of context. First reply is the wait for a 2K prompt. Speeds are rough, plus or minus 25%.
How to run it
- Ollama
ollama run mistral-medium-3.5:128b- llama.cpp
llama-server -hf unsloth/Mistral-Medium-3.5-128B-GGUF- vLLM
vllm serve mistralai/Mistral-Medium-3.5-128B- MLX (Mac)
mlx_lm.chat --model mlx-community/Mistral-Medium-3.5-128B-4bit
Questions
How much VRAM does Mistral Medium 3.5 128B need?
Mistral Medium 3.5 128B needs 72.1 GB for its weights at 4-bit, 75.9 GB with an 8K context and 84.2 GB with 32K.
Can Mistral Medium 3.5 128B run on an RTX 4090?
Not on one card. The smallest machine here that runs it is the RTX PRO 6000 Blackwell.