Investigating the recall efficiency in abstractive summarization: an experimental based comparative study

Abstract

This study explores text summarization, a critical component of natural language processing (NLP), specifically targeting scientific documents. Traditional extractive summarization, which relies on the original wording, often results in disjointed sequences of sentences and fails to convey key ideas concisely. To address these issues and ensure comprehensive inclusion of relevant details, our research aims to improve the coherence and completeness of summaries. We employed 25 different large language models (LLMs) to evaluate their performance in generating abstractive summaries of scholarly scientific documents. A recall-oriented evaluation of the generated summaries revealed that LLMs such as 'Claude v2.1,' 'PPLX 70B Online,' and 'Mistral 7B Instruct' demonstrated exceptional performance with ROUGE-1 scores of 0.92, 0.88, and 0.85, respectively, supported by high precision and recall values from bidirectional encoder representations from transformers (BERT) scores (0.902, 0.894, and 0.888). These findings offer valuable insights for NLP researchers, laying the foundation for future advancements in LLMs for summarization. The study highlights potential improvements in text summarization techniques, benefiting various NLP applications.