Weights
Q4 uses much less memory than FP16. Metadata and quantization format mean that “4 bit” is not exactly “0.5 byte per parameter.”
Model memory made understandable
How much VRAM does a local LLM need?
The short answer: enough for weights, context cache and headroom. The longer answer helps you choose between 16, 32, 64 and 128 GB without confusing marketing metrics.
The simple formula
Weights + KV cache + runtime + headroom
Parameter count alone is not a memory figure. What matters is the number of bits per weight and everything that must remain in memory during use.
Model weightsparameters × quantization+KV cachecontext × architecture+Runtimebuffers and backend+HeadroomOS and other tools
KV cache
It stores information from the current context. Long prompts, concurrent sessions and some architectures make it grow rapidly.
Runtime
Ollama, llama.cpp, LM Studio and other backends need working buffers. Implementation and GPU offload affect the total.
Headroom
A pool that only just fits on paper is a poor buying plan. The OS, display and companion services need breathing room.
Conservative guidance
Typical needs for quantized text models
Ranges roughly assume Q4-like quantization plus ordinary runtime headroom. Long context, MoE architectures, vision encoders or concurrent users can require more.
| Model class | Approx. weights | Sensible fast-memory class | Typical use |
|---|---|---|---|
| 7B–8B | about 5–6 GB | 12–16 GB | chat, summarization, simple local helpers |
| 12B–14B | about 8–10 GB | 16 GB | better generalists, entry-level coding |
| 20B–32B | about 14–22 GB | 24–32 GB | coding, RAG, capable assistants |
| 65B–70B | about 40–45 GB | 64 GB or more | higher model quality, complex tasks |
| 100B–120B | about 65–80 GB | 96–128 GB or more | professional capacity class |
“Fits in memory” does not mean “runs fast.” Bandwidth, compute, backend and offload distribution determine speed.
Two memory paths
VRAM and unified memory are not the same
Dedicated VRAM
Memory directly attached to a separate GPU. High bandwidth makes it the fastest route while the entire model fits.
- excellent inference speed
- clear, fixed capacity limit
- offloading costs performance
Unified memory
CPU and integrated GPU share a large pool. Bigger models can fit, but the OS and applications use that same memory.
- 64 to 128 GB in compact systems
- configurable UMA allocation matters
- less bandwidth than high-end VRAM
Common questions
What buyers often confuse
Is 16 GB of VRAM enough for local AI?
Yes, for many 7B to 14B models in suitable quantization. Long context, large vision models or 32B models can exceed it.
Can I launch a 70B model with 32 GB of VRAM?
Often yes by offloading part of it to system RAM or using stronger quantization. It will not reside entirely in VRAM, and performance can drop substantially.
Does 128 GB RAM mean 128 GB for the GPU?
No. Only suitable unified-memory architectures can share a large portion. The OS and runtime need headroom, and the maximum configurable UMA allocation must be checked.
Do I need an NPU?
For large local LLMs, an NPU alone is rarely the main buying criterion. Runtime support, memory capacity, bandwidth and GPU performance matter more.
From theory to selection
Turn model size and context into a concrete direction
The PC Finder applies this memory logic and explains its recommendation.