1. VRAM
Fit weights, runtime state, KV cache, batch and framework overhead with headroom.
Choose a GPU.
Start with the workload.
The best GPU is not a universal model name. It is the configuration that fits your memory requirement and delivers the latency, throughput and cost your application needs.
2. Performance
Measure the latency and throughput your actual application cares about.
3. Cost
Compare cost per useful workload result, not hourly price in isolation.
Inference
sizing.
For LLM serving, model parameters and numerical precision establish a starting memory requirement. Context length, KV cache, batching and concurrency add memory pressure. Quantization can reduce memory requirements, but the exact result depends on the model and runtime.
- Define model and precision
- Define maximum context and expected concurrency
- Set time-to-first-token and throughput targets
- Benchmark the serving stack you will actually use
A100 vs H100
Compare Ampere and Hopper classes.
H100 vs H200
Compare 80 GB and 141 GB Hopper memory classes.
H200 vs B200
Compare Hopper and Blackwell classes.
GPU benchmarks
See the methodology used for measured results.