Model DistillationWhen The Small Model Pays For Itself
Distills a real Claude teacher into a local classifier and measures the three numbers that decide whether distillation is worth doing: quality retained, the cost of an imperfect teacher, and the break-even point.
Headline results
- Quality retained
- 81% of teacher
- Labelling cost
- $0.10 one-time
- Break-even
- ≈300 documents
- Gap
- 0.054 noise + 0.12 capacity
System architecture
Problem
Paying a frontier model to classify the same narrow thing forever is a recurring bill for a solved problem. Distillation is the standard answer, and the standard write-up stops at the student's accuracy, which leaves the two decisions a team actually faces unanswered: how much quality you lose, and how many documents it takes before the swap pays.
Approach
A real Claude Haiku teacher on Bedrock labels a four-category slice of 20 Newsgroups zero-shot, with headers, footers and quotes stripped so it is genuine content classification against real gold labels. A local TF-IDF and logistic-regression student trains only on the teacher's labels. Both are scored against the held-out gold set, and a third model trained on gold labels establishes the student's own ceiling, which is what separates teacher noise from student capacity. Teacher labels are cached and committed, so the numbers reproduce for free.
Impact
The student recovers 81% of teacher accuracy, 0.733 against 0.907, for a one-time $0.10 labelling run, after which inference is free and offline. The useful part is the decomposition: a student trained on gold scores 0.787, so roughly 0.054 of the gap is the tax for learning from an imperfect teacher and roughly 0.12 is the student's own capacity ceiling. The fix is a stronger student, not cleaner labels, and break-even lands around 300 documents.
Decisions & tradeoffs
Train a third model on gold to find the ceiling
Without it, the teacher-student gap is one number and the obvious response is to buy better labels. Measuring the student on gold shows most of the gap is capacity, which points at a different fix entirely.
Commit the teacher's label cache
Distillation results that require paid API calls to reproduce are unverifiable in practice. The cache makes the headline numbers replayable by anyone, and deleting it re-runs the real calls.
Use a task with real gold labels
Scoring the student against the teacher measures imitation, not correctness. 20 Newsgroups has ground truth, so both models are graded by the same honest marker.
Build spec
- Teacher
- Claude Haiku 4.5 on Bedrock, zero-shot, 0.907 on gold
- Student
- TF-IDF + LogReg on teacher labels, 0.733
- Ceiling
- Same student on gold labels, 0.787
- Decomposition
- ≈0.054 teacher-noise tax · ≈0.12 capacity ceiling
- Economics
- $0.10 one-time labelling · break-even ≈300 docs
System notes
- The gap is decomposed into teacher noise and student capacity, which is what tells you where to spend next
- Teacher labels are cached and committed, so the exact numbers reproduce without spending anything
- Real gold labels exist for the task, so both teacher and student are scored against truth rather than each other
- Break-even is computed from measured token cost, not estimated
Stack
Claude on Bedrock · scikit-learn · 20 Newsgroups · Cost modelling