LLM Quantization BenchWhere Quantization Actually Bites
The same model at FP16, Q8_0 and Q4_K_M measured on-device for accuracy, speed and memory, with a per-category breakdown that shows exactly which capability quantization costs you.
Headline results
- Q4 vs FP16
- 0.39× size, 2.0× speed
- Accuracy cost
- −2.5 points
- Where it lands
- All arithmetic, 40% → 30%
- Unaffected
- Factual · reading · reasoning
System architecture
Problem
Shipping a model to a laptop or a phone means quantizing it, and the guidance is a shrug: four-bit is usually fine. Usually fine is not a deployment decision. What is missing is which capability degrades, by how much, and what you get back in speed and memory.
Approach
Hold the model fixed and vary only the quantization. Qwen2.5-0.5B-Instruct at FP16, Q8_0 and Q4_K_M as llama.cpp GGUFs on Apple Silicon with Metal, scored on 40 multiple-choice questions at temperature 0 with three 128-token generations per quant for throughput. Accuracy is broken out by category, arithmetic, factual recall, reading comprehension and reasoning, so the aggregate cannot hide where the loss lands. File size, resident memory and load time are recorded alongside.
Impact
Q4_K_M runs at 0.39× the file size and 2.0× the speed of FP16 for 2.5 points of accuracy. The per-category breakdown is the finding: factual, reading and reasoning scores are identical across all three quantizations, and the entire drop is arithmetic, 40% down to 30%. So the deployment question is not whether to quantize, it is whether your workload does precise multi-step computation. Q8_0 is dominated on this model, matching Q4's accuracy at 1.4× the size.
Decisions & tradeoffs
Break accuracy out by category
A single 2.5-point drop reads as a rounding error and tells you nothing. Split by capability it becomes a clear rule: quantize freely for retrieval and comprehension, think twice for anything computational.
Report resident memory, not just file size
File size is what people quote and RSS is what fills the machine. Both are measured because a 491 MB file that resides at 392 MB is a different deployment story than the download suggests.
State the small-n caveat with the headline
Forty questions puts roughly 2.5 points on a single question, which is the same size as the headline gap. Saying so is what keeps the per-category finding, which is robust, separate from the aggregate, which is not.
Build spec
- Model
- Qwen2.5-0.5B-Instruct, llama.cpp GGUF, Metal
- f16
- 1,266 MB · 40.5 tok/s · 60.0% · arith 40%
- q8_0
- 676 MB · 67.4 tok/s · 57.5% · arith 30%
- q4_k_m
- 491 MB · 80.6 tok/s · 57.5% · arith 30%
- Q4 vs FP16
- 0.39× size · 2.0× speed · −2.5 points
System notes
- The whole accuracy loss is arithmetic; factual, reading and reasoning are identical across every quant
- Smaller is faster because inference here is bandwidth-bound, so the memory saving buys throughput too
- Q8_0 is dominated on this model, which is a useful negative for anyone reaching for it as the safe middle
- Load time drops from 10.6s to 0.50s, which matters for anything that cold-starts
What this does not show
- Forty questions puts roughly 2.5 points on a single answer, the same size as the headline gap. The per-category split is the robust finding; the aggregate is not.
Stack
llama.cpp (Metal) · GGUF · Qwen2.5 · huggingface-hub · psutil