Summarization Eval ToolkitMulti-Signal Evaluation for Text Summarizers
A model-agnostic toolkit that evaluates text summarizers with ROUGE, BLEU, BERTScore, perplexity, and an LLM-as-judge, logs everything to MLflow, and works as both a CLI and an importable library.
Headline results
- Metrics
- ROUGE · BLEU · BERTScore · perplexity
- Judge
- Coherence · faithfulness · relevance
- Tracking
- MLflow
- Tests
- 18 passing
System architecture
Problem
A good summary is not one thing. A summary can have high word overlap with the reference yet hallucinate a fact, paraphrase perfectly yet share almost no words, or read fluently while being irrelevant. No single number captures all of this, so a summarizer needs several complementary signals that each measure a different failure mode.
Approach
The toolkit generates summaries with any Hugging Face seq2seq model and scores them with a metric suite where each metric covers a distinct failure: ROUGE-1/2/L for lexical overlap, BLEU for n-gram precision with a brevity penalty, BERTScore for semantic similarity to the reference, reference-free perplexity for fluency under a causal LM, and an LLM-as-judge rating coherence, faithfulness, and relevance on a 1 to 5 scale. The key axis is that ROUGE and BLEU ask same words, BERTScore asks same meaning as the reference, and the judge reads the source document to catch a high-overlap summary that hallucinates. The whole pipeline is model-agnostic and driven by config.yaml, with model ids, dataset, judge, and metric toggles all configurable and nothing hardcoded to one checkpoint. Every metric is flattened into MLflow params and logged, and an optional Seq2SeqTrainer wrapper fine-tunes any T5 or BART checkpoint. It ships as both a CLI and an importable library.
Impact
Packaged as a reusable, unit-tested library plus CLI rather than a one-off notebook, it makes the disagreement between metric families the signal: when the three metric types agree you can trust the verdict, and when they disagree the disagreement is itself the finding. The lightweight lexical-metric unit tests run in-repo via pytest, while a full CNN/DailyMail pass with BERTScore and the LLM judge is GPU-bound and honestly labeled as not executed here.
Decisions & tradeoffs
Report several signals, not one number
Each metric measures a different failure mode, so the toolkit surfaces all of them side by side. When the metric families disagree, that disagreement is treated as the finding rather than noise.
Config-driven, model-agnostic pipeline
Model ids, dataset, judge, and metric toggles all come from config.yaml with nothing hardcoded to one checkpoint. The only checkpoint-specific quirk, T5's summarize prefix, is auto-applied and overridable.
Honest labeling of unmeasured runs
Only the lightweight lexical-metric unit tests run in-repo, since a full BERTScore and LLM-judge pass is GPU-bound. The sample outputs are labeled as illustrative of the format, not a measured run.
Build spec
- Reference metrics
- ROUGE-1/2/L, BLEU, BERTScore
- Reference-free
- Perplexity + LLM-as-judge (1-5)
- Tracking
- MLflow params + metrics
- Interface
- CLI + importable library
- Origin
- LUMS LLM course lab, generalized
System notes
- Metric suite where each signal catches a distinct failure mode
- LLM judge reads the source to flag hallucinated but high-overlap summaries
- Model-agnostic pipeline driven entirely by config.yaml
- MLflow tracking plus optional T5/BART fine-tune wrapper
Stack
Transformers · MLflow · ROUGE · BERTScore · LLM-judge · Python