NLP ServiceDo You Even Need An LLM
A deployed fine-tuned text classifier benchmarked head to head against an LLM zero-shot baseline on accuracy, latency and cost, to answer whether the task needs an LLM at all.
Headline results
- Fine-tuned
- 100% at 0.222ms
- LLM zero-shot
- 87.8% at 540ms
- Cost per 1k
- $0.00 vs $0.68
- Speedup
- ≈2,432×
System architecture
Problem
Reaching for an LLM to classify support tickets is the expensive default. It is a defensible choice when you have no labels and low volume, and an indefensible one when you have both, but teams rarely run the comparison because the LLM works well enough on day one and the bill arrives later.
Approach
Fine-tune a small specialized model, TF-IDF and logistic regression, on labelled data for a narrow classification task, and serve it behind a FastAPI endpoint. Then benchmark it head to head against an LLM zero-shot baseline on the three axes that decide the question: accuracy, latency per prediction, and cost per thousand calls. The baseline runs as a calibrated mock by default and swaps to real Claude with an API key.
Impact
On a narrow high-volume task the specialized model wins on all three axes at once: 100% accuracy against 87.8%, 0.22ms per prediction against 540ms, and effectively zero cost against $0.68 per thousand. That is roughly 2,400× the speed for better accuracy and no per-call bill. The transferable judgment is knowing which side of that trade a task sits on, because the same comparison inverts when labels are scarce or the label set changes weekly.
Decisions & tradeoffs
Benchmark on three axes, not on accuracy
Accuracy alone would make this a coin flip. Latency and cost are what turn it into a decision, and they are the two numbers that never appear in the notebook comparison teams actually run.
State the conditions where the answer flips
The specialized model wins here because the task is narrow, high-volume, and labelled. Without labels or with a label set that changes weekly, the LLM is the right call, and a benchmark that does not say so is advocacy.
Serve it rather than score it
A 0.222ms number from a loop in a notebook is not a serving latency. Putting it behind FastAPI keeps the measurement close to what a caller would experience.
Build spec
- Fine-tuned
- 100% accuracy · 0.222ms/pred · $0.00 per 1k
- LLM zero-shot
- 87.8% accuracy · 540ms/pred · $0.68 per 1k
- Speed
- ≈2,432× lower latency
- Model
- TF-IDF + LogisticRegression
- Serving
- FastAPI /classify
System notes
- All three axes measured together, because winning on accuracy while losing on cost is not a win
- Served behind a real FastAPI endpoint rather than benchmarked in a notebook
- The LLM baseline runs as a calibrated mock by default, so the comparison reproduces without keys
- Sub-millisecond inference is what makes the specialized model viable at high volume, not the accuracy
What this does not show
- The specialized model wins because the task is narrow, high-volume and labelled. Without labels, or with a label set that changes weekly, the LLM is the right call.
Stack
TF-IDF + LogReg · FastAPI · Claude baseline · Cost/latency benchmark