Why bigger models cost more and run slower. The hardware story behind every cost decision.
VRAM: the threshold
VRAM (Video RAM) is the GPU's memory. It determines whether a model can fit on the GPU. It's a threshold, not a speedometer.
GPU with 24GB VRAM:
→ 7B model fits → runs at full GPU speed.
→ 70B model → doesn't fit, can't run.
GPU with 80GB VRAM:
→ 7B model fits → runs at the SAME speed as above.
→ 70B model fits → now it CAN run.
The 7B model runs at the same speed on both. Extra VRAM doesn't
make it faster, it just sits empty.
Speed comes from compute, not capacity
VRAM = "Can it fit?" (threshold)
Compute = "How fast?" (CUDA cores, tensor cores, clock)
More GPUs = "Scale it" (parallelism)
Memory bandwidth (HBM speed) matters too. For large-batch inference, bandwidth is often the bottleneck, not raw FLOPs.
What happens when it doesn't fit:
Model fits in VRAM: 50 tokens/sec
Model partially in VRAM: 5 tokens/sec (offloading kicks in)
Model offloaded to disk: 0.5 tokens/sec (catastrophic)
A bigger countertop doesn't make the chef cook faster. But if the recipe book doesn't fit on the counter, the chef keeps running to the shelf. That's slow.
Why bigger is slower: the math
7B model: ~7 billion multiplications per output token
70B model: ~70 billion multiplications per output token
Same GPU → 70B has 10x more work per word.
How production systems get both accuracy AND speed
| Technique | How it works |
|---|---|
| Throw hardware at it | Multiple GPUs in parallel. Linear cost, sub-linear speed past a point. |
| Distillation | Train a smaller "student" to mimic a larger "teacher." Cheaper inference, similar quality on bounded tasks. |
| Speculative decoding | A tiny model guesses next 5 tokens; the big model verifies all at once. 3 to 4x faster, same accuracy. |
| Model routing / cascading | Different models for different tasks. |
| Quantization | Reduce precision (16-bit → 8-bit or 4-bit). Smaller VRAM, faster, slight quality loss. |
The triangle, again: pick two
For most product workloads, you don't need the largest model. Match model size to task complexity. Reserve the biggest model for genuinely hard reasoning. Model selection is a product decision, not a prestige one.