Synthetic Data PipelineMore Data Is Not Better Data
Generate synthetic labeled data, filter it for quality, label consistency and duplication, then measure the downstream accuracy lift on a held-out set of real data.
Headline results
- Filtered set
- 5× smaller
- Accuracy
- 95.8% → 100%
- Removed
- 283 mislabeled, 803 duplicate
- Scored on
- Held-out real data
System architecture
Problem
Generating training data with an LLM is easy and the output is noisy by construction: near-duplicates, mislabels, and rows too short to carry signal. Teams generate ten thousand examples, train on all of them, and never check whether the synthetic data helped, because checking requires held-out real data and a control.
Approach
Generate labelled data, then filter it in three passes: drop low-quality rows on length and emptiness, drop rows whose text cues contradict their stated label, and drop near-duplicates by Jaccard overlap on word sets. Then train the identical downstream model on the raw set and on the filtered set, and score both on a held-out set of real data. The generator runs as a calibrated mock by default so the pipeline reproduces offline, with a flag to generate with Claude instead.
Impact
Filtering cut 1,500 raw rows to 264, removing 150 low-quality, 283 mislabelled and 803 duplicates. The filtered set is five times smaller and trains a better model: 95.8% to 100% accuracy on the real test set. The 283 mislabelled rows were actively poisoning training and the 803 duplicates added nothing, so the volume everyone optimises for was the problem rather than the goal.
Decisions & tradeoffs
Measure downstream, not the data
Data-quality metrics computed on the synthetic set itself are self-referential. Training the same model on both versions and scoring on real held-out data is the only measurement that answers whether the filtering was worth doing.
Report each filter's removals separately
One aggregate drop count hides which filter earned its place. Splitting it shows deduplication removed the most rows while label consistency removed the ones that were actually hurting accuracy.
Make the mock generator the default
The claim is about filtering, not about generation quality. A calibrated deterministic generator keeps the experiment reproducible and free, and the Claude path stays available for anyone who wants to vary it.
Build spec
- Raw
- 1,500 rows, noisy by construction
- Filtered
- 264 kept (−150 quality, −283 mislabel, −803 dup)
- Raw-trained
- 95.83% accuracy · macro-F1 0.958
- Filtered-trained
- 100% accuracy · macro-F1 1.000
- Lift
- +4.2 points on a 5× smaller set
System notes
- The filtered set is 5× smaller and scores higher, which inverts the instinct to keep everything
- Each filter reports what it removed and why, so the loss is attributable rather than a single count
- Downstream accuracy is measured on real held-out data, which is the only test that settles the question
- The default generator is a calibrated mock, so the whole pipeline runs offline and deterministically
Stack
scikit-learn · Claude generation · Jaccard dedup · Data-centric AI