Active LearningWhich Labels Are Worth Buying
A label-efficiency benchmark: margin, least-confidence and entropy query strategies against random, measured by labels-to-target-accuracy and learning-curve area on a real dataset.
Headline results
- margin
- 38% fewer labels than random
- least_confidence
- 15% worse than random
- entropy
- 38% worse than random
- Advice
- Which strategy, not whether
System architecture
Problem
Labels cost money and most of them teach the model nothing. Active learning promises the same accuracy for fewer labels by choosing which examples to annotate. The promise is usually reported as a single success story, which leaves the practical question unanswered: which selection strategy, and does it ever backfire.
Approach
A pool-based loop on scikit-learn digits. Start from two labelled examples per class, fit logistic regression, score a held-out test set, query a batch from the unlabelled pool with the chosen strategy, label it, repeat, recording the whole learning curve. Four strategies compete: random as the baseline, least-confidence, margin, and entropy. Success is measured as labels needed to reach a target accuracy plus the area under the learning curve, so a strategy cannot win by getting lucky at one point.
Impact
Margin sampling reaches 90% accuracy on 80 labels against random's 130, a 38% saving, and it also wins on learning-curve AUC and final accuracy. Least-confidence needs 150 labels and entropy needs 180, so both are meaningfully worse than labelling at random. The usable advice is not use active learning, it is use margin sampling, because two of the three popular strategies here actively lose to the baseline by fixating on ambiguous outliers.
Decisions & tradeoffs
Keep random as the baseline everything must clear
Active-learning results are usually reported against nothing. Random selection is the only honest bar, and here it beats two of the three strategies under test, which would have been invisible without it.
Score the curve, not a checkpoint
Labels-to-target rewards a strategy that happens to spike at the threshold. Reporting learning-curve AUC alongside it makes the comparison stable across where you choose to stop.
Explain the losses rather than dropping them
Least-confidence and entropy lose because pure uncertainty pulls them toward genuinely ambiguous outliers that teach the decision boundary very little. Margin is the strategy that asks a more useful question.
Build spec
- Dataset
- scikit-learn digits, 10 classes, pool 1,237, test 540
- margin
- 80 labels to 90% · +38% vs random · AUC 0.928
- random
- 130 labels to 90% · AUC 0.885
- least_confidence
- 150 labels · 15% worse than random
- entropy
- 180 labels · 38% worse than random
System notes
- Two of three popular strategies underperform random selection, which is the finding worth publishing
- Scored on labels-to-target and curve AUC together, so a single lucky checkpoint cannot decide the winner
- The seed, pool size, and test split are fixed and printed, so the run reproduces
- Uncertainty-only strategies fixate on ambiguous outliers, which is the mechanism behind the losses
Stack
scikit-learn · NumPy · Pool-based AL · Learning-curve AUC