A systematic evaluation and benchmarking of text summarization methods for biomedical literature: From word-frequency methods to language models

Abstract The rapid expansion of biomedical literature demands automated summarization tools that can reliably condense research articles into concise, accurate summaries. We benchmarked 62 text summarization methods, ranging from frequency-based and TextRank extractors to encoder-decoder models (EDMs) and large language models (LLMs), on 1,000 biomedical abstracts with author-generated highlights as reference summaries. Models were evaluated using a composite suite of lexical, semantic, and factual metrics, including ROUGE, BLEU, METEOR, embedding-based similarity, and factuality scores. Our results indicate that general-purpose language models (LMs) achieve the highest overall performance across lexical and semantic dimensions, outperforming both reasoning-oriented and domain-specific models. Notably, medium-sized models often outperform frontier-scale counterparts, suggesting an optimal balance between model capacity and computational efficiency. Statistical extractive methods consistently lag behind neural approaches. These findings provide a systematic reference for selecting biomedical summarization tools and highlight that broad pretraining remains more effective than narrow domain adaptation for generating high-quality scientific summaries. Competing Interest Statement K.K. is co-founder and shareholder of Delta4 GmbH (Vienna, Austria). F.B., E.B., L.F., L.G., K.K.-I., S.W., P.A., P.P. and M.L. are employees of Delta4 GmbH (Vienna, Austria). Footnotes Expanded evaluation framework from single factual consistency metric to more robust suite of methods; Added expert validation; Added data leakage analysis; Revised Figure 1, 3, 4, 5; Added Figure 6; Added Table 3, 4.