FP16 / BF16
Approximately 2 bytes per parameter for weights alone.
LLM GPU sizing.
Start with the weights.
Use parameter count × bytes per parameter as a first-pass estimate for model weights. Then add headroom for the runtime, KV cache, context, batching and concurrency. This is a sizing aid, not a guarantee that a workload fits.
INT8
Approximately 1 byte per parameter for weights alone, before format/runtime overhead.
Add headroom
Serving memory is more than model weights. Validate with your actual runtime.
WEIGHTS-ONLY EXAMPLESDecimal parameter notation; approximate values.
ParametersPrecisionWeight memory
7BFP16/BF16~14 GB
7BINT8~7 GB
13BFP16/BF16~26 GB
34BFP16/BF16~68 GB
70BFP16/BF16~140 GB
70BINT8~70 GB
Why actual VRAM
is higher.
An inference server also needs memory for runtime allocations and, depending on architecture and serving strategy, KV cache and other buffers. Longer contexts and higher concurrency can materially increase memory use.
- Choose the exact model and precision first
- Estimate weight memory
- Add runtime and context headroom
- Benchmark at realistic concurrency