One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa

arXiv:2608.192002026-08-21

Head-to-head test of three summarization AIs shows BART, which rewrites text from scratch, beats models that just pick existing sentences

Researchers compared three transformer-based summarization models, BERT, RoBERTa, and BART, on the CNN/DailyMail news dataset. BERT and RoBERTa were fine-tuned to pick out important sentences directly from articles (extractive summarization), while BART was used as a ready-made, already summarization-tuned model that writes new sentences (abstractive summarization). Scored with the ROUGE and BLEU metrics, BART came out clearly ahead of both, and RoBERTa outperformed BERT.

What they did

  1. BERT and RoBERTa, which select sentences one by one ('extractive summarization'), were fine-tuned for 16 epochs on 15,000 training articles from CNN/DailyMail, while BART used an already summarization-pretrained checkpoint (BART-large-CNN) without any additional fine-tuning.
  2. All experiments ran on a CPU-only machine with no GPU, forcing smaller batch sizes and fewer training epochs than a typical setup.
  3. On the ROUGE-1 score, BART scored about 0.4117 versus RoBERTa's 0.1820 and BERT's 0.1485, a gap that held up similarly across BLEU and other ROUGE variants.
  4. RoBERTa also reached a lower final training loss (1.47) than BERT (2.11), a rough sign it fit the training data better under the same setup.
  5. Because BART generates new wording, its summaries read more fluently, but manual checks found occasional factual slips like altered numbers or invented details (hallucination); BERT and RoBERTa, which just copy original sentences, stayed more factually accurate but produced less polished summaries.

Why it matters

This gives practical guidance for choosing a summarization model for real tasks like condensing news or reports. But since BART was compared in its off-the-shelf form while BERT and RoBERTa were specifically fine-tuned, the size of BART's advantage should be read with that mismatch in mind.

Terms in this paper

  • BERT · An encoder-only language model pretrained to understand text bidirectionally, reading context from both directions
  • RoBERTa · A version of BERT trained with improved methods and more data for better performance
  • BART · A model with both an understanding encoder and a text-generating decoder, well suited to writing new summary text
  • Extractive summarization · Building a summary by copying existing sentences straight from the source text
  • Abstractive summarization · Building a summary by generating new sentences that paraphrase the source
  • ROUGE / BLEU · Automated scoring metrics that measure overlap between an AI-generated summary and a human-written reference summary
  • Hallucination / factual inconsistency · When an AI confidently produces details or numbers not actually present in the source text

Original abstract (English)

Text summarization refers to the task of condensing a document into a shorter version while preserving its key information. Automatic text summarization (ATS), driven by advancements in natural language processing (NLP), has developed rapidly in recent years. ATS methods are commonly categorized by input type (such as single-document or multi-document summarization) and by output type (extractive, abstractive, and hybrid). This article presents a focused review of modern summarization techniques with an emphasis on transformer based models and large language models (LLMs), specifically BERT, RoBERTa and BART. It examines their architectures, pretraining strategies, and their suitability for extractive and abstractive summarization tasks.

Authors · Daisy Aptovska, Vinayak Elangovan

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB