Local LLM memory planning: weights are only the starting point
Estimate whether a local language model will fit by accounting for quantized weights, KV cache, context length, runtime buffers, GPU offload, and concurrent sessions.
Bottom line
Model-file size is a lower bound, not a complete memory estimate. The same quantized model can fit at a short context and fail at a long one because KV cache and runtime overhead grow separately.
Decision table
| Hardware | Best fit | Editorial decision |
|---|---|---|
| No discrete GPU | CPU-first experimentation | Use a quantized model sized for system RAM and accept lower throughput. Memory bandwidth often matters as much as raw compute. Leave RAM for the OS, runtime, and context cache. |
| 8 GB VRAM | Small quantized models and partial offload | A useful entry tier for compact assistants. Keep context conservative and verify actual allocation; a model file close to 8 GB leaves no safe room for cache or runtime buffers. |
| 12–16 GB | Broader 7B–14B-class quantized choices | Often enough for a useful local coding or writing setup, depending on quantization and context. Prefer a smaller model at a usable context over a larger model that constantly spills. |
| 24 GB | Larger quantized models or more context | More flexibility for full GPU offload and larger context, but architecture and KV-cache precision still matter. Concurrent users multiply cache requirements. |
| 48 GB+ | Large models, long context, or concurrency | Capacity supports larger workloads; validate quality and throughput before assuming the biggest model is the best fit for the task. |
Build the estimate in layers
Start with the quantized model file or reported weight allocation. Then add KV cache for the context you will actually use, runtime workspaces and buffers, and a safety margin. If only some layers are on the GPU, also check system RAM for the CPU-resident portion.
Do not turn this into one universal calculator constant. KV-cache size depends on architecture, layer count, hidden dimensions, cache precision, sequence length, batch/concurrency, and implementation. Measure the chosen runtime after loading and again near your target context.
Why context length changes the answer
Autoregressive generation reuses keys and values from prior tokens. The KV cache avoids recomputing all previous attention state, but it consumes memory that grows with sequence length. Hugging Face documents dynamic, static, quantized, and offloaded cache strategies, each with different memory and performance trade-offs.
A model advertised with a very large maximum context does not mean your hardware can use that context economically. Set context to the work: a short chat, a code repository, and multi-document retrieval have different needs.
Quantization and offload are trade spaces
Quantization reduces weight memory and can also be applied to cache in supported runtimes. More aggressive quantization may reduce quality for some models or tasks. Compare outputs on your own prompts rather than selecting only by file size.
llama.cpp supports broad hardware backends and CPU+GPU hybrid inference. Partial offload can make a larger model run, but transfers and CPU-resident work may reduce throughput. Ollama provides a convenient model-management layer; the underlying fit still depends on model, quantization, context, and backend.
A validation checklist before downloading
Define a target context and number of simultaneous sessions. Choose a quantization with published artifacts for the exact model. Confirm that model weight size plus planned cache and overhead sits below available VRAM or RAM, then test with the runtime’s memory reporting.
If the plan has almost no headroom at an empty prompt, it is not a stable fit. Reduce context, choose a smaller quantization or model, offload deliberately, or move up a hardware tier.
- Task and acceptable model quality
- Exact model architecture and quantization
- Target context—not the advertised maximum
- Number of parallel prompts or users
- GPU layers/offload plan and available system RAM
- Measured allocation after load and near target context
How this was researched
This guide synthesizes Hugging Face cache documentation and the upstream llama.cpp and Ollama projects. It explains planning principles and validation steps rather than claiming exact card-specific benchmarks. Runtime, backend, architecture, and context choices can change the result.
Evidence confidence: High for principles; medium for workload-specific fit. “Documented” means an upstream source states the capability. Tier advice and scenario conclusions are editorial synthesis and should be validated against your exact model, settings, and runtime.
Read our complete methodology and corrections policy →Primary sources
- [1]KV cache strategies · Hugging Face Transformers
Cache growth, implementations, quantization, and offloading.
- [2]llama.cpp · ggml-org / GitHub
Official runtime, quantization, hardware backend, and hybrid inference capabilities.
- [3]