{"paper_id":"45c0756f-70e3-4183-8a91-552e004222d2","body_text":"Comparing the Performance of SOTA Text Summarization Models on AI Research Papers | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparing the Performance of SOTA Text Summarization Models on AI Research Papers Pradnya Gotmare, Sushant Nair This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8658398/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In the realm of academics and research work, efficient, accurate and privacy-focused summarization of research papers have emerged as significant areas of interest. The availability of such a tool would make the tasks like literature survey much easier and faster, enabling academics to channel their energy to other aspects of their work. In this work the best summarization model is identified by comparing the performance of four State Of The Art (SOTA) text summarization models. These models are Facebook’s Bart Large CNN, Phil Schmid’s Bart Large CNN SAMSum (Bart Large CNN fine-tuned on the SAMSum dataset), Sam Shleifer’s DistilBART CNN 12 6 and Google’s PEGASUS CNN Dailymail. The initial part of this work focuses on evaluating the summaries generated by the Vanilla models. This dataset has been obtained from the AI Arxiv2 Dataset, which contains a diverse range of information about research papers published in ArXiv under the AI domain. The generated summaries are evaluated using metrics like BLEU, ROUGE, BERTScore and METEOR. The models are compared to determine the best model for summarization. This work involves fine-tuning the Vanilla models on a separate train dataset with 1,377 examples obtained from the AI Arxiv2 Dataset, in an attempt to improve their ability to summarize text containing AI jargon and terminology. The later part focuses on evaluating the summaries generated by these fine-tuned models on the test dataset with 326 examples. Artificial Intelligence and Machine Learning Text Summarization Natural Language Processing Summarization Models BART Models Figures Figure 1 Figure 2 1. Introduction Academics and researchers face a significant task of studying existing research work before embarking on a project, in order to gain insights about the approaches used by other researchers in order to solve similar problems, and to avoid accidentally repeating the work already done before. However, most research papers span many pages, making this process challenging. Furthermore, a comprehensive literature survey would involve going through hundreds of research papers. Extracting relevant sentences for the summary is time-consuming and challenging. An automated tool would be a great boon, as it would enable researchers to get the cream of the content of hundreds of research papers quickly, conserving time and energy. The increased efficiency would also mean that the researchers would be better suited to understand the gist of the research papers and make good decisions quickly. Apart from researchers and academics, this kind of tool can also benefit students and other knowledge workers to quickly gain insights about the subject matter presented in the material. It is also important to note that the latest LLMs are often more sophisticated than required. In the matter of summarizing text, other capabilities like code generation and image generation are largely irrelevant. Although recent advances in LLM technology like sparse Mixture of Experts (MoEs) architecture mean that of the several trillion parameters, only a few hundred billion are being used to generate the answer, yet even in that case the number of parameters run into the hundreds of billions, and using them results in high financial and environmental costs [ 1 ][ 2 ][ 3 ][ 4 ][ 5 ]. On the other hand, using a model specifically meant for summarizing research papers would imply that the model is very small. The models presented in this work have been fine-tuned on existing BART and PEGASUS models, which themselves have parameters of the order of a few hundred million. This implies that these specialised models are hundred times smaller than their general-purpose LLM counterparts, which would result in much lower financial and environmental costs of utilising these models [ 6 ]. As a result, these models can be run using less-powerful GPUs and even some modern CPUs available locally. These models can also run on hosted environments like Google Colab and Kaggle, leading to an increase in privacy, security, and control over the model. Lastly, it may be mentioned that the Vanilla models have been trained on datasets which are universally open for use and are not under any copyrights. The fine-tuned models have been fine-tuned on open-access research papers from Arxiv. So therefore no copyright infringement is committed by the training, fine-tuning and use of these models. The scope of this work is to generate concise summaries of research papers and other academic content in the domain of artificial intelligence that capture key points from diverse documents. The initial part of this paper focuses on selecting four SOTA text summarization models from Hugging Face. The methodology used for carrying out this work has been mentioned in Section 3 of this work. Section 3.1 describes the reasoning employed for selecting the four aforementioned models out of the other possible models. Section 3.2 describes the setup of the project, revealing details like the reasoning behind selecting the aforementioned models out of other possible models. The project setup solved issues like creation of the dataset based on the information from the AI Arxiv 2 dataset and parallel processing of files. Section 4 pertains to the experimental results of this work. Section 4.1 describes the challenges faced while fine tuning the Vanilla models. Sections 4.2 and 4.3 describes the comparison of summary scores of the Vanilla as well as fine-tuned models. Sections 5 wrap up this work with a conclusion, followed by acknowledgements. The implementation has been made available at https://github.com/sushantnair/AIStudy . 2. Related Work Summarization is the process of taking a large document and condensing it such that the key information is preserved, while being presented in the simplest manner possible. There are two broad kinds of summarization extractive and abstractive. Extractive summarization deals with selecting the most important sentences from the document and directly using them for the summary, with no modifications. This approach can be performed using several low-compute approaches like graph-based approach and machine learning-based approach, which involves algorithms like Naive Bayes, Decision Tree, Support Vector Machine and Hidden Markov Model [ 7 ]. Abstractive summarization, on the other hand, deals with understanding the meaning of the text and generating original summary using different choice of words. This approach requires understanding the context and meaning of the paper, for which compute-intensive deep learning models and transformer models are required. A brief survey of text summarization techniques has been performed by [ 7 ]. It provides an overview of the growing need for effective text summarization methods due to the explosion of text data. It emphasizes the importance of summarization in making vast amounts of information manageable and accessible, highlighting the challenges posed by unstructured data from various sources. This survey categorizes and evaluates different automatic text summarization techniques, including extractive and abstractive methods. It discusses their effectiveness, limitations, and potential applications, serving as a foundational reference for researchers and practitioners in the field of Natural Language Processing. Apart from informing about the extractive and abstractive summarization approaches, it also reveals about the various methods for evaluating the accuracy of generated summaries [ 7 ]. Human-based evaluation is the gold standard for evaluation of a machine-generated summary. Details on how to craft good human summaries have been explained by [ 8 ]. Modern abstractive summarization techniques are based on the transformer model [ 9 ]. This paper introduces the transformer model, which relies solely on self-attention mechanisms, eliminating the need for recurrent layers in sequence modeling tasks like translation. These self-attention mechanisms enable the model to capture semantic meaning of the text, which is crucial for generating good abstractive summaries. The proposed work involves the fine-tuning of the BART and PEGASUS models. BART is highly suited for text summarization as it has a combination of BERT-like bidirectional encoder and GPT-like autoregressive (left-to-right) decoder [ 10 ]. The encoder enables the model to effectively understand the text, while the decoder helps it to express the meaning of the text in a novel and relevant manner. Like BART, PEGASUS is also trained by masking text and making it learn the completions, enabling the model to learn summarization by predicting them based on context [ 11 ]. However, the masking mechanism for both is different. While BART masks word-wise, PEGASUS masks sentence-wise. The model architecture of PEGASUS is also different, causing it to be heavier and slower than BART. Several metrics, are used to assess the quality of generated content, that include Bilingual Evaluation Understudy (BLEU) [ 12 ], Recall Oriented Understudy for Gisting Evaluation (ROUGE) [ 13 ], BERTScore [ 14 ] and Metric for Evaluation of Translation with Explicit ORdering (METEOR) [ 15 ]. Out of these, BERTScore is the best suited to overcome the limitations of traditional n-gram based scoring methods. It is better able to handle generated summaries which use synonyms or alternate sentence structure as compared to the reference summary. 3. Methodology This section describes the methodology applied to different models in this work. It describes the different library functions implemented for summary generation and evaluation. Different metrics have been used for evaluation of the models. 3.1. Model Selection HuggingFace is one of the most popular websites for sharing and collaboration of artificial intelligence models and research. Facebook’s BART Large CNN and Sam Schliefer’s DistilBART CNN 12 6 were selected as they featured among the top two in the “most downloaded” category. Google’s PEGASUS and Phil Schmidt’s BART Large CNN Samsum were selected as they featured among the top two in the “trending” category. 3.2. Libraries 3.2.1. Arxiv Extractor Library This work has implemented LinkExtractor and TextProcessor classes as part of this library. The LinkExtractor class extracts the content of a research paper given its arXiv URL. The TextProcessor class prepares the data for the model. This library was developed to suit the custom requirements of the work. 3.2.2. Summarization Tools Library This library implements the following types of Summarization: Rolling Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in a rolling manner as shown in Fig. 1 . There are two variations. Rolling summarization with static chunking and rolling summarization with dynamic chunking. In static chunking the size of the chunk and the extent of overlapping do not change with chunks. However, they decrease as the number of chunks increases in case of dynamic chunking. Apart from these, with dynamic chunking, the chunk_shrink_factor, the overlap_shrink_factor and the chunk_group_size values are also required. The chunk_ group_size specifies the number of chunks which have the same value of chunk_size and chunk_overlap. The amount of reduction of chunk_size and chunk_overlap across consecutive chunk groups is specified by the chunk_shrink_factor and overlap_shrink_factor, respectively. Overlapping Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in an overlapping manner as shown in Fig. 1 . For each input chunk, the corresponding summary is individually generated. These summaries are concatenated with each other and the resultant summary is the output. This summarization technique is suited when a detailed summary is required. Layered Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in a layered manner as shown in Fig. 2 . The branching factor and depth influence the quality and length of the summary. The length of summaries generated by this method fall between the lengths of summaries generated by the rolling and overlapping methods. It indicates that the layered method can generate the ideal kind of summaries that would be suitable for most use cases. For this method, the value of branching factor (the spread across the tree) and depth (of the tree) are most important features. 3.2.3. Metrics Calculator Library This work implements the MetricsCalculator class as part of this library, to satisfy the custom requirements. The following metrics are computed: BLEU: This metric is calculated by the calculate_bleu method, which obtains the BLEU scores considering values of n for the n-grams to be in the range 1 through 4. ROUGE: The calculate_rouge method obtains the ROUGE-1, ROUGE-2 and ROUGE-L. BERTScore: The calculate_bert_score method obtains the BERTScore. However, the transformer model chosen is based on the available CPU/GPU RAM. The default model is DistilBERT Base Uncased. METEOR: The METEOR Scores are obtained using the calculate_meteor method. 3.3. Implementation Details The code implementation consists of different sections, as follows. A_Database_Creator: This section handles the creation of the database. It creates the database by loading the HuggingFace dataset, and generating the database tables with necessary fields such as Paper URL, authors, etc. B_Dataset_Builder: This section builds the dataset of files to be input for the purpose of fine-tuning and testing. First, the PDF of a paper is downloaded. Next, these downloaded files are converted into text files. These text files are truncated and finally the truncated files are denoised. In this work, a total of 1,377 files, corresponding to 1,377 research papers have been used for fine-tuning the models, and a further 326 files have been used for testing the performance of the models. C_Summary_Generator: This section is used for generating summaries of the cleaned input documents. There are two modes of operation: first to generate only the Llama 3.1 8B summaries of the input and second, to generate summaries using the text summarization models. The first mode is performed for the 1,377 research papers meant for fine-tuning, in order to generate the fine tuning dataset of the research papers. It is also performed for the 326 research papers meant for testing, in order to generate the ground truth summaries to score the summaries generated by the Vanilla and fine-tuned models. The second mode is used to generate the summaries of the 326 research papers of the test set by the Vanilla and fine-tuned models. The summaries generated by Llama 3.1 8B are considered to be the benchmark. D_Scorer: This section is used to get the score of the summaries generated by the Vanilla and Fine-tuned text summarization models on the test dataset of 326 papers. E_Model_Comparer: This section is used to compare the Vanilla models with their fine-tuned counterparts generating various Excel file tables. 4. Results and Discussion Example 1 shows the summary generated by the Vanilla BART Large CNN Model. Example 1 Mistral 7B leverages grouped query attention (GQA) and sliding window attention (SWA) GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding. SWA is designed to handle longer sequences more effectively at a reduced computational cost. Example 2 shows the summary generated by the Fine-tuned BART Large CNN Model. Example 2 Large language models like Mistral 7B aim to balance high performance while maintaining efficient inference by leveraging grouped-query attention and sliding window attention. These attention mechanisms significantly accelerate inference speed and reduce memory requirements, making them suitable for real-time applications. The model is released under the Apache 2.0 license and has a reference implementation facilitating easy deployment on cloud platforms. These summaries have been generated for the snippet of the introduction paragraph of the Mistral 7B paper [ 16 ] by Jiang et al. The snippet is available in Appendix A. 4.1. Challenges faced while fine-tuning the Vanilla models Several challenges were encountered in the process of developing the fine-tuning script. The first version involved manually handling details like dataset processing and tokenization in pursuance of fine-grained control over the GPU utilization. However, the fine-tuned models obtained as a result produced very low quality summaries consisting of words randomly placed with no sensibility or meaning. The primary reason for this problem was an improper implementation of the tokenizer and complexities arising from manually handling the dataset. Another anomaly observed is that the fine-tuned model was generating summaries much longer than needed, with several repeated phrases. This indicated that the distribution of training data needed to be fixed. The chunking strategy implemented by script was determined to be the source of the problem. It was found that there was no coherence in the pairing of the source document chunks with the summary chunks. Further improvement was achieved by changing the evaluation metric from pure loss to Rouge. After this, the authors tried various values of the per_device_train_batch_size, per_device_eval_batch_size and generation_max_length hyper parameters. It was found that a value of 4 for the first two and a value of 256 for the last hyperparameter resulted in good summaries. 4.2. Best Summarization Type and Model for Each Test Research Paper Table 1 is the comparison of Vanilla and fine-tuned versions of BART Large CNN. The overlapping summarization type produces the best summaries for 187 out of 326 papers. The Layered Summarization is found best for 132 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type produces the best summaries for 297 out of 326 papers, i. e. 91% of the papers. Table 1 Best Summarization and Model Type for BART Large CNN Paper No. Best Summarization Type Best Model Type 1 Overlapping Fine-tuned 2 Layered Fine-tuned 3 Overlapping Fine-tuned … … … 326 Layered Fine-tuned Total Overlapping: 187; Layered: 132 Fine-tuned: 297; Vanilla: 29 Table 2 Best Summarization and Model Type for BART Large CNN SAMSum Paper No. Best Summarization Type Best Model Type 1 Overlapping Fine-tuned 2 Overlapping Vanilla 3 Overlapping Fine-tuned … … … 326 Overlapping Fine-tuned Total Overlapping: 198; Layered: 120 Fine-tuned: 287; Vanilla: 39 Table 2 is the comparison of Vanilla and fine-tuned versions of BART Large CNN SAMSum. The overlapping summarization type produces the best summaries for 198 out of 326 papers. The layered summarization produces the best summaries for 120 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type is the best, producing the best summaries for 287 out of 326 papers, or 88% of the papers. Table 3 Best Summarization and Model Type for DistilBART Paper No. Best Summarization Type Best Model Type 1 Overlapping Fine-tuned 2 Layered Fine-tuned 3 Overlapping Fine-tuned … … … 326 Overlapping Fine-tuned Total Overlapping: 182; Layered: 136 Fine-tuned: 293; Vanilla: 33 Table 3 is the comparison of Vanilla and fine-tuned versions of DistilBART. The overlapping summarization type produces the best summary for 182 out of 326 papers. The layered summarization type produces the best summary for 136 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type produces the best summary for 293 out of 326 papers, or 91% of the papers. Table 4 Best Summarization and Model Type for PEGASUS CNN DailyMail Paper No. Best Summarization Type Best Model Type 1 Overlapping Fine-tuned 2 Overlapping Fine-tuned 3 Overlapping Fine-tuned … … … 326 Overlapping Vanilla Total Overlapping: 286; Layered: 34 Fine-tuned: 300, Vanilla: 26 Table 4 is the comparison of Vanilla and fine-tuned versions of PEGASUS CNN DailyMail. The overlapping summarization type produces the best summaries for 286 out of 326 papers. Unlike other models before, it holds the absolute majority, accounting for nearly 88% of all papers. The fine-tuned model type produces the best summaries for 300 out of 326 papers, or 92% of the papers. With these observations, it can be concluded that the best summarization type is overlapping summarization. summing over the four tables, it scores 853 out of 1304, or about 65%. The next best is the layered type, scoring 422 out of 1304, or about 32%. Together they account for about 97% of the total scores. An interesting trend observed is that the more complex the summarization type, the worse is the performance. Overlapping summarization is the simplest type, while layered is more complex. Yet, overlapping has the highest scores, than the layered one. On the other hand, the layered summaries strike a balance in length between overlapping and rolling types. It can also be concluded that the best model type is fine-tuned, by a large margin. It scores 1117 out of 1304, or almost 90%. On the other hand, the Vanilla type scores just 127 out of 1304, or about 10%. 4.3. Maximum, minimum and average of metrics The maximum, minimum and average value for a metric is obtained by finding the maximum, minimum and average values respectively of that metric across all the research papers summarized by that model in the test dataset of 326 papers. Table 5 shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned BART Large CNN. It can be observed that for both the Vanilla as well as fine-tuned model, BERTScore precision has the highest maximum, minimum and average scores. Further, as compared to the vanilla model, the fine-tuned model has higher minimum and maximum values for all the three columns Max, Min as well as Avg. Table 5 Maximum, Minimum and Average of metrics for Vanilla and Fine-tuned BART Large CNN Metrics Vanilla BART Large CNN Fine-tuned BART Large CNN Max Min Avg Max Min Avg ROUGE-1 0.6134 0.026 0.3118 0.6721 0.0216 0.3524 ROUGE-2 0.3445 0 0.1154 0.4444 0 0.1616 ROUGE-L 0.4304 0.0219 0.1616 0.5495 0.0216 0.1963 BERTScore Precision 0.8761 0.5074 0.7066 0.8897 0.5422 0.755 BERTScore Recall 0.8367 0.394 0.6636 0.8794 0.414 0.6824 BERTScore F1 0.7968 0.4479 0.6834 0.8428 0.5075 0.716 METEOR 0.503 0.0085 0.2005 0.6887 0.0103 0.2308 Table 6 Maximum, Minimum and Average of metrics for Vanilla and Fine-tuned BART Large CNN SAMSum Metrics Vanilla BART Large CNN SAMSum Fine-tuned BART Large CNN SAMSum Max Min Avg Max Min Avg ROUGE-1 0.6258 0.0048 0.3207 0.7041 0.0332 0.3562 ROUGE-2 0.3752 0 0.126 0.4433 0 0.1605 ROUGE-L 0.4428 0.0048 0.1689 0.5714 0.0257 0.1958 BERTScore Precision 0.8761 0.4134 0.7148 0.8871 0.6165 0.7467 BERTScore Recall 0.8351 0.3514 0.6653 0.8798 0.4719 0.6866 BERTScore F1 0.8258 0.3821 0.6884 0.8643 0.5485 0.7148 METEOR 0.5462 0.0062 0.2039 0.6373 0.0108 0.2333 Table 6 shows the maximum, minimum and average score for each metric for Vanilla and Fine-tuned BART Large CNN SAMSum. It can be observed for both the Vanilla as well as fine-tuned model that the BERTScore Precision has the highest maximum, minimum and average scores. Further, as compared to the vanilla model, the fine-tuned model has higher minimum and maximum values for all the three columns Max, Min as well as Avg. Table 7 Maximum, minimum and average of metrics for Vanilla and fine-tuned DistilBART Metrics Vanilla DistilBART Fine-tuned DistilBART Max Min Avg Max Min Avg ROUGE-1 0.62 0.031 0.3281 0.7141 0.0341 0.3488 ROUGE-2 0.381 0 0.1267 0.5197 0 0.157 ROUGE-L 0.5155 0.0232 0.1714 0.5525 0.0272 0.1916 BERTScore Precision 0.8761 0.4711 0.7225 0.8838 0.4664 0.7513 BERTScore Recall 0.8382 0.4396 0.6725 0.8751 0.4201 0.6807 BERTScore F1 0.819 0.486 0.6957 0.8685 0.4434 0.7135 METEOR 0.5257 0.008 0.2095 0.7027 0.0116 0.2265 Table 7 shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned DistilBART. It can be observed for both the Vanilla as well as fine-tuned model that the BERTScore precision has the maximum and average scores. Also the highest minimum score for the fine-tuned model also belongs to BERTScore precision. It is also observed that for Vanilla model BERTScore F1 has the highest minimum score. Further, as compared to the vanilla model, the fine-tuned model has better minimum and maximum values for two of the three columns, Max and Avg. However, for the Min column, the Vanilla model does better. Table 8 Max, Min and Average of Metrics for Vanilla and Fine-tuned PEGASUS CNN DailyMail Metrics Vanilla PEGASUS CNN DailyMail Fine-tuned PEGASUS CNN DailyMail Max Min Avg Max Min Avg ROUGE-1 0.657 0.0204 0.2943 0.6913 0.0261 0.3245 ROUGE-2 0.3692 0 0.1117 0.4086 0 0.1401 ROUGE-L 0.3978 0.0155 0.1553 0.4198 0.0195 0.1796 BERTScore Precision 0.8556 0.5299 0.6962 0.8905 0.5781 0.7483 BERTScore Recall 0.8224 0.4084 0.6486 0.8785 0.4265 0.6621 BERTScore F1 0.7879 0.4663 0.6704 0.8226 0.4972 0.7019 METEOR 0.5389 0.0077 0.1765 0.6843 0.009 0.2013 Table 8 shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned PEGASUS CNN DailyMail. It can be observed that for both the Vanilla as well as fine-tuned model, BERTScore precision metric has the highest maximum, minimum and average scores. Further, as compared to the Vanilla model, the fine-tuned model has better minimum and maximum values for all the three columns Max, Min and Avg. Table 9 Summary of Tables 5 , 6 , 7 and 8 . Model Max Max Max Min Max Avg Vanilla BART Large CNN 0.8761 0.5074 0.7066 Fine-tuned BART Large CNN 0.8897 0.5422 0.755 Vanilla BART Large CNN SAMSum 0.8761 0.4134 0.7148 Fine-tuned BART Large CNN SAMSum 0.8871 0.6165 0.7467 Vanilla DistilBART CNN 12 6 0.8761 0.486 0.7225 Fine-tuned DistilBART CNN 12 6 0.8838 0.4664 0.7513 Vanilla PEGASUS CNN DailyMail 0.8556 0.5299 0.6962 Fine-tuned PEGASUS CNN DailyMail 0.8905 0.5781 0.7483 Table 9 has been created by taking the minimum and maximum value of the Max, Min and Avg columns of Tables 5 , 6 , 7 and 8 for the Vanilla and fine-tuned models for BERTScore precision metric. For nearly every pair of minimum values and every pair of maximum values, the fine-tuned variant of each model produces better results as compared to the Vanilla variant of that model. The model having the maximum score is the fine-tuned variant of BART Large CNN SAMSum. Section 4.3 . mentioned the best summarization type and model type (Vanilla or fine-tuned). This section mentioned the fine-tuned BART Large CNN SAMSum as the best model. 5. Conclusion This work demonstrates the performance of four fine-tuned transformer-based models namely BART Large CNN, BART Large CNN SAMSum, DistilBART and PEGASUS CNN DailyMail. DistilBART provides the best performance among Vanilla models, despite being the smallest. Fine-tuning the models results in a universal improvement in performance, with BART Large CNN SAMSum and BART Large CNN models being the best. In this scenario, DistilBART secured the best score for only the BERTScore precision metric. This work highlights improved performance achieved by fine-tuning the models. This also indicates that other vanilla models had better capacity to learn new distributions of data as compared to DistilBART. On the other hand, PEGASUS, the largest model, displayed the worst performance despite fine-tuning. It indicates that model size and performance are not necessarily correlated. Thus, it can be concluded that fine-tuned versions of the Vanilla models outperform their Vanilla counterparts, and that overlapping summarization is the best method of summarization. Overall, it has been observed that the fine-tuned BART Large CNN SAMSum is the best model for research paper summarization. Declarations Acknowledgements The authors of this paper feel grateful to KJ Somaiya School of Engineering for access to High Performance Computing Lab. The authors also express gratitude to Dr. Grishma J Sharma from KJ Somaiya School of Engineering for constructive feedback. Funding Not applicable. This research did not receive funding. Conflict of interest/ Competing interests The authors declare that they have no known competing interest Ethics approval and consent to participate Not applicable Consent for publication Data availability Data and code is available athttps://github.com/sushantnair/AIStudy Material availability Not applicable Code availability Data and code is available athttps://github.com/sushantnair/AIStudy Author contribution S. N. and P. G. have equally contributed to this work. References Samsi S et al (2023) From words to watts: Benchmarking the energy costs of large language model inference. 2023 IEEE High Performance Extreme Computing Conference (HPEC). IEEE Ren S et al (2024) Reconciling the contrasting narratives on the environmental impact of large language models. Sci Rep 14(1):26310 Khan T et al (2025) Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights. arXiv preprint arXiv:2504.06307 Li P et al (2023) Making ai less thirsty: Uncovering and addressing the secret water footprint of ai models. arXiv preprint arXiv:2304.03271 ( Chen Z et al (2022) Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.02813 Desislavov R (2021) Fernando Martínez-Plumed, and José Hernández-Orallo. Compute and energy consumption trends in deep learning inference. arXiv preprint arXiv:2109.05472 Allahyari M et al (2017) Text summarization techniques: a brief survey. arXiv preprint arXiv:1707.02268 Özdemir S (2018) The effect of summarization strategies teaching on strategy usage and narrative text summarization success Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30 Lewis M et al (2019) Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 Zhang J et al (2020) Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. International conference on machine learning. PMLR Papineni K et al (2002) Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics Lin C-Y (2004) Rouge: A package for automatic evaluation of summaries. Text summarization branches out Zhang T et al (2019) Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 Banerjee, Satanjeev, and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.2005 Jiang AQ et al Mistral 7B, pp. 1–9, [Online]. Available: https://mistral.ai/news/announcing-mistral-7b/ Roziere B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, Synnaeve G (2023) Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Scialom T (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Lample G (2023) Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 Kwon W, Li Z, Zhuang S, Sheng Y, Zheng L, Yu CH, Stoica I (2023), October Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles (pp. 611–626) Chaplot DS (2023) Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed. arXiv preprint arXiv:2310.06825, 3. Additional Declarations The authors declare no competing interests. Supplementary Files Appendix.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-8658398\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":578044959,\"identity\":\"0895b49b-487f-43b8-9b78-8a8b9c392823\",\"order_by\":0,\"name\":\"Pradnya Gotmare\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIklEQVRIiWNgGAWjYJACCQjFAyblwGQCEolbCxtEizHpWhIbEBLYtZhLH354u7DNzq5fvvfohp97bNLnt3cnfnjYZsfAz55jgE2LZV+asfXMtuTkmW18aTd7nqXlbjhzdrNEYlsyg2TPG6xaDM4wmEnztjEnGxzjMbvBc+Bw7gYJIEpsY2YwuIHdFoMz7N+AWurBWm7+OfA/XX5G7uYfiW31DPY4tfCAbDlsB9Jym+fAgQSGG7nbgLYcZjCQwOGXHp5ia55zxxMk23LMbsscSDYE+mWbRcK54zwSZ54VYA0xHvaNt3nKqu35mc+Y3XxzwE5evr13880fZdVy/O3JG7A6DEQwsqFEB0SEB5tyuBaGPwz2aOJ/cGkYBaNgFIyCEQgAIqxlFWVLWnQAAAAASUVORK5CYII=\",\"orcid\":\"https://orcid.org/0000-0001-7892-5386\",\"institution\":\"K J Somaiya School of Engineering\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Pradnya\",\"middleName\":\"\",\"lastName\":\"Gotmare\",\"suffix\":\"\"},{\"id\":578045769,\"identity\":\"7c388c0c-3708-4d3c-ba9d-7583a78c6392\",\"order_by\":1,\"name\":\"Sushant Nair\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"K J Somaiya School of Engineering\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Sushant\",\"middleName\":\"\",\"lastName\":\"Nair\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2026-01-21 10:40:18\",\"currentVersionCode\":1,\"declarations\":{\"humanSubjects\":false,\"vertebrateSubjects\":false,\"conflictsOfInterestStatement\":false,\"humanSubjectEthicalGuidelines\":false,\"humanSubjectConsent\":false,\"humanSubjectClinicalTrial\":false,\"humanSubjectCaseReport\":false,\"vertebrateSubjectEthicalGuidelines\":false},\"doi\":\"10.21203/rs.3.rs-8658398/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-8658398/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":100847084,\"identity\":\"4747d856-a9fd-444a-afe9-ce0d5cdbf1e6\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:21\",\"extension\":\"docx\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":126599,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"RevisedSNResearchPaperDec25.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/89e2ee9d3d079356c2c07fd2.docx\"},{\"id\":100847085,\"identity\":\"3c39a225-c7e0-45b7-8527-5dd520f6b479\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:21\",\"extension\":\"json\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":342,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"rs8658398.json\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/77899e91449a91127cdd8b3e.json\"},{\"id\":100847089,\"identity\":\"b67f461e-7200-44aa-b731-3ad8d5eebfef\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"xml\",\"order_by\":2,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":89528,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"rs86583980enriched.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/d7388af4b5b46efe556c8789.xml\"},{\"id\":100847087,\"identity\":\"7f21997b-0a6f-4ede-97ea-8043cbf4545b\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:21\",\"extension\":\"jpeg\",\"order_by\":3,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":244020,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"floatimage1.jpeg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/58d4695540041ef5c4da0b4d.jpeg\"},{\"id\":100847088,\"identity\":\"11c975c6-e6ac-483f-95df-c35789165982\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"jpeg\",\"order_by\":4,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":269252,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"floatimage2.jpeg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/918ef0cc1335ee40c829ccbb.jpeg\"},{\"id\":100859137,\"identity\":\"eed7ba92-0737-4383-95d6-ef3f513d4028\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 07:25:22\",\"extension\":\"png\",\"order_by\":5,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":49231,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Onlinefloatimage1.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/d176d9ce3b4f49cfbf5caddc.png\"},{\"id\":100847090,\"identity\":\"6d9319b6-312c-40c6-8321-435912a9ce67\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"png\",\"order_by\":6,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":47010,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Onlinefloatimage2.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/9d3fa942e86a3fa903510ba2.png\"},{\"id\":100847091,\"identity\":\"6644a430-1833-4498-9c3b-dfa1d000ef35\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"xml\",\"order_by\":7,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":88673,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"rs86583980structuring.xml\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/515c6dc2f0d0483d1b76f6b2.xml\"},{\"id\":100847095,\"identity\":\"41d6c46c-54aa-48cd-90df-dda9383754f8\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:23\",\"extension\":\"html\",\"order_by\":8,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"acdc-reference\",\"size\":95909,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"earlyproof.html\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/d61e07d095643d0b6d4148a9.html\"},{\"id\":100847094,\"identity\":\"6d7c5bf0-3066-4e3b-8907-081b6d48345c\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"jpg\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":69521,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eSchematic Diagram of Rolling Summarization (l) and Overlapping Summarization (r)\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure1.jpg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/ffd706f1191cc2a5fc4aad93.jpg\"},{\"id\":100847092,\"identity\":\"087f1fa5-b796-4dc9-ad58-c87cdbad536e\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:22\",\"extension\":\"jpg\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":95335,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eSchematic Diagram of Layered Summarization\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"Figure2.jpg\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/f96cb78fd9e066ce1ea7fb72.jpg\"},{\"id\":100859974,\"identity\":\"6ff37651-c154-4085-a810-084eaf79f0ce\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 07:35:19\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":1125308,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/ac6fdbaf-1ed6-459d-80a2-09669ae255a7.pdf\"},{\"id\":100847083,\"identity\":\"8559af56-2ca3-44c5-8491-b5aca1fd958c\",\"added_by\":\"auto\",\"created_at\":\"2026-01-22 04:40:21\",\"extension\":\"docx\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":15498,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Appendix.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-8658398/v1/ef838fea4abde32f0c2a7f69.docx\"}],\"financialInterests\":\"The authors declare no competing interests.\",\"formattedTitle\":\"\\u003cp\\u003e\\u003cstrong\\u003eComparing the Performance of SOTA Text Summarization Models on AI Research Papers\\u003c/strong\\u003e\\u003c/p\\u003e\",\"fulltext\":[{\"header\":\"1. Introduction\",\"content\":\"\\u003cp\\u003eAcademics and researchers face a significant task of studying existing research work before embarking on a project, in order to gain insights about the approaches used by other researchers in order to solve similar problems, and to avoid accidentally repeating the work already done before. However, most research papers span many pages, making this process challenging. Furthermore, a comprehensive literature survey would involve going through hundreds of research papers. Extracting relevant sentences for the summary is time-consuming and challenging. An automated tool would be a great boon, as it would enable researchers to get the cream of the content of hundreds of research papers quickly, conserving time and energy. The increased efficiency would also mean that the researchers would be better suited to understand the gist of the research papers and make good decisions quickly. Apart from researchers and academics, this kind of tool can also benefit students and other knowledge workers to quickly gain insights about the subject matter presented in the material.\\u003c/p\\u003e \\u003cp\\u003eIt is also important to note that the latest LLMs are often more sophisticated than required. In the matter of summarizing text, other capabilities like code generation and image generation are largely irrelevant. Although recent advances in LLM technology like sparse Mixture of Experts (MoEs) architecture mean that of the several trillion parameters, only a few hundred billion are being used to generate the answer, yet even in that case the number of parameters run into the hundreds of billions, and using them results in high financial and environmental costs [\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e][\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e][\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e][\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e][\\u003cspan citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e]. On the other hand, using a model specifically meant for summarizing research papers would imply that the model is very small. The models presented in this work have been fine-tuned on existing BART and PEGASUS models, which themselves have parameters of the order of a few hundred million. This implies that these specialised models are hundred times smaller than their general-purpose LLM counterparts, which would result in much lower financial and environmental costs of utilising these models [\\u003cspan citationid=\\\"CR6\\\" class=\\\"CitationRef\\\"\\u003e6\\u003c/span\\u003e]. As a result, these models can be run using less-powerful GPUs and even some modern CPUs available locally. These models can also run on hosted environments like Google Colab and Kaggle, leading to an increase in privacy, security, and control over the model. Lastly, it may be mentioned that the Vanilla models have been trained on datasets which are universally open for use and are not under any copyrights. The fine-tuned models have been fine-tuned on open-access research papers from Arxiv. So therefore no copyright infringement is committed by the training, fine-tuning and use of these models.\\u003c/p\\u003e \\u003cp\\u003eThe scope of this work is to generate concise summaries of research papers and other academic content in the domain of artificial intelligence that capture key points from diverse documents. The initial part of this paper focuses on selecting four SOTA text summarization models from Hugging Face. The methodology used for carrying out this work has been mentioned in Section 3 of this work. Section \\u003cspan refid=\\\"Sec4\\\" class=\\\"InternalRef\\\"\\u003e3.1\\u003c/span\\u003e describes the reasoning employed for selecting the four aforementioned models out of the other possible models. Section \\u003cspan refid=\\\"Sec5\\\" class=\\\"InternalRef\\\"\\u003e3.2\\u003c/span\\u003e describes the setup of the project, revealing details like the reasoning behind selecting the aforementioned models out of other possible models. The project setup solved issues like creation of the dataset based on the information from the AI Arxiv 2 dataset and parallel processing of files. Section 4 pertains to the experimental results of this work. Section \\u003cspan refid=\\\"Sec11\\\" class=\\\"InternalRef\\\"\\u003e4.1\\u003c/span\\u003e describes the challenges faced while fine tuning the Vanilla models. Sections \\u003cspan refid=\\\"Sec12\\\" class=\\\"InternalRef\\\"\\u003e4.2\\u003c/span\\u003e and \\u003cspan refid=\\\"Sec13\\\" class=\\\"InternalRef\\\"\\u003e4.3\\u003c/span\\u003e describes the comparison of summary scores of the Vanilla as well as fine-tuned models. Sections 5 wrap up this work with a conclusion, followed by acknowledgements. The implementation has been made available at \\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://github.com/sushantnair/AIStudy\\u003c/span\\u003e\\u003cspan address=\\\"https://github.com/sushantnair/AIStudy\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e.\\u003c/p\\u003e\"},{\"header\":\"2. Related Work\",\"content\":\"\\u003cp\\u003eSummarization is the process of taking a large document and condensing it such that the key information is preserved, while being presented in the simplest manner possible. There are two broad kinds of summarization extractive and abstractive. Extractive summarization deals with selecting the most important sentences from the document and directly using them for the summary, with no modifications. This approach can be performed using several low-compute approaches like graph-based approach and machine learning-based approach, which involves algorithms like Naive Bayes, Decision Tree, Support Vector Machine and Hidden Markov Model [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]. Abstractive summarization, on the other hand, deals with understanding the meaning of the text and generating original summary using different choice of words. This approach requires understanding the context and meaning of the paper, for which compute-intensive deep learning models and transformer models are required.\\u003c/p\\u003e \\u003cp\\u003eA brief survey of text summarization techniques has been performed by [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]. It provides an overview of the growing need for effective text summarization methods due to the explosion of text data. It emphasizes the importance of summarization in making vast amounts of information manageable and accessible, highlighting the challenges posed by unstructured data from various sources. This survey categorizes and evaluates different automatic text summarization techniques, including extractive and abstractive methods. It discusses their effectiveness, limitations, and potential applications, serving as a foundational reference for researchers and practitioners in the field of Natural Language Processing.\\u003c/p\\u003e \\u003cp\\u003eApart from informing about the extractive and abstractive summarization approaches, it also reveals about the various methods for evaluating the accuracy of generated summaries [\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e]. Human-based evaluation is the gold standard for evaluation of a machine-generated summary. Details on how to craft good human summaries have been explained by [\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e].\\u003c/p\\u003e \\u003cp\\u003eModern abstractive summarization techniques are based on the transformer model [\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e]. This paper introduces the transformer model, which relies solely on self-attention mechanisms, eliminating the need for recurrent layers in sequence modeling tasks like translation. These self-attention mechanisms enable the model to capture semantic meaning of the text, which is crucial for generating good abstractive summaries. The proposed work involves the fine-tuning of the BART and PEGASUS models. BART is highly suited for text summarization as it has a combination of BERT-like bidirectional encoder and GPT-like autoregressive (left-to-right) decoder [\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e]. The encoder enables the model to effectively understand the text, while the decoder helps it to express the meaning of the text in a novel and relevant manner. Like BART, PEGASUS is also trained by masking text and making it learn the completions, enabling the model to learn summarization by predicting them based on context [\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e]. However, the masking mechanism for both is different. While BART masks word-wise, PEGASUS masks sentence-wise. The model architecture of PEGASUS is also different, causing it to be heavier and slower than BART.\\u003c/p\\u003e \\u003cp\\u003eSeveral metrics, are used to assess the quality of generated content, that include Bilingual Evaluation Understudy (BLEU) [\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e], Recall Oriented Understudy for Gisting Evaluation (ROUGE) [\\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e], BERTScore [\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e] and Metric for Evaluation of Translation with Explicit ORdering (METEOR) [\\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e]. Out of these, BERTScore is the best suited to overcome the limitations of traditional n-gram based scoring methods. It is better able to handle generated summaries which use synonyms or alternate sentence structure as compared to the reference summary.\\u003c/p\\u003e\"},{\"header\":\"3. Methodology\",\"content\":\"\\u003cp\\u003eThis section describes the methodology applied to different models in this work. It describes the different library functions implemented for summary generation and evaluation. Different metrics have been used for evaluation of the models.\\u003c/p\\u003e \\u003cdiv id=\\\"Sec4\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.1. Model Selection\\u003c/h2\\u003e \\u003cp\\u003eHuggingFace is one of the most popular websites for sharing and collaboration of artificial intelligence models and research. Facebook\\u0026rsquo;s BART Large CNN and Sam Schliefer\\u0026rsquo;s DistilBART CNN 12 6 were selected as they featured among the top two in the \\u0026ldquo;most downloaded\\u0026rdquo; category. Google\\u0026rsquo;s PEGASUS and Phil Schmidt\\u0026rsquo;s BART Large CNN Samsum were selected as they featured among the top two in the \\u0026ldquo;trending\\u0026rdquo; category.\\u003c/p\\u003e \\u003cp\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec5\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.2. Libraries\\u003c/h2\\u003e \\u003cdiv id=\\\"Sec6\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e3.2.1. Arxiv Extractor Library\\u003c/h2\\u003e \\u003cp\\u003eThis work has implemented LinkExtractor and TextProcessor classes as part of this library. The LinkExtractor class extracts the content of a research paper given its arXiv URL. The TextProcessor class prepares the data for the model. This library was developed to suit the custom requirements of the work.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec7\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e3.2.2. Summarization Tools Library\\u003c/h2\\u003e \\u003cp\\u003eThis library implements the following types of Summarization:\\u003c/p\\u003e \\u003cp\\u003eRolling Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in a rolling manner as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e. There are two variations. Rolling summarization with static chunking and rolling summarization with dynamic chunking. In static chunking the size of the chunk and the extent of overlapping do not change with chunks. However, they decrease as the number of chunks increases in case of dynamic chunking. Apart from these, with dynamic chunking, the chunk_shrink_factor, the overlap_shrink_factor and the chunk_group_size values are also required. The chunk_ group_size specifies the number of chunks which have the same value of chunk_size and chunk_overlap. The amount of reduction of chunk_size and chunk_overlap across consecutive chunk groups is specified by the chunk_shrink_factor and overlap_shrink_factor, respectively.\\u003c/p\\u003e \\u003cp\\u003eOverlapping Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in an overlapping manner as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e. For each input chunk, the corresponding summary is individually generated. These summaries are concatenated with each other and the resultant summary is the output. This summarization technique is suited when a detailed summary is required.\\u003c/p\\u003e \\u003cp\\u003eLayered Summarization: It is implemented to generate summaries from the TS Model by feeding the text chunks in a layered manner as shown in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e. The branching factor and depth influence the quality and length of the summary. The length of summaries generated by this method fall between the lengths of summaries generated by the rolling and overlapping methods. It indicates that the layered method can generate the ideal kind of summaries that would be suitable for most use cases. For this method, the value of branching factor (the spread across the tree) and depth (of the tree) are most important features.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec8\\\" class=\\\"Section3\\\"\\u003e \\u003ch2\\u003e3.2.3. Metrics Calculator Library\\u003c/h2\\u003e \\u003cp\\u003eThis work implements the MetricsCalculator class as part of this library, to satisfy the custom requirements. The following metrics are computed:\\u003c/p\\u003e \\u003cp\\u003eBLEU: This metric is calculated by the calculate_bleu method, which obtains the BLEU scores considering values of n for the n-grams to be in the range 1 through 4.\\u003c/p\\u003e \\u003cp\\u003eROUGE: The calculate_rouge method obtains the ROUGE-1, ROUGE-2 and ROUGE-L.\\u003c/p\\u003e \\u003cp\\u003eBERTScore: The calculate_bert_score method obtains the BERTScore. However, the transformer model chosen is based on the available CPU/GPU RAM. The default model is DistilBERT Base Uncased.\\u003c/p\\u003e \\u003cp\\u003eMETEOR: The METEOR Scores are obtained using the calculate_meteor method.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec9\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e3.3. Implementation Details\\u003c/h2\\u003e \\u003cp\\u003eThe code implementation consists of different sections, as follows.\\u003c/p\\u003e \\u003cp\\u003eA_Database_Creator: This section handles the creation of the database. It creates the database by loading the HuggingFace dataset, and generating the database tables with necessary fields such as Paper URL, authors, etc.\\u003c/p\\u003e \\u003cp\\u003eB_Dataset_Builder: This section builds the dataset of files to be input for the purpose of fine-tuning and testing. First, the PDF of a paper is downloaded. Next, these downloaded files are converted into text files. These text files are truncated and finally the truncated files are denoised. In this work, a total of 1,377 files, corresponding to 1,377 research papers have been used for fine-tuning the models, and a further 326 files have been used for testing the performance of the models.\\u003c/p\\u003e \\u003cp\\u003eC_Summary_Generator: This section is used for generating summaries of the cleaned input documents. There are two modes of operation: first to generate only the Llama 3.1 8B summaries of the input and second, to generate summaries using the text summarization models. The first mode is performed for the 1,377 research papers meant for fine-tuning, in order to generate the fine tuning dataset of the research papers. It is also performed for the 326 research papers meant for testing, in order to generate the ground truth summaries to score the summaries generated by the Vanilla and fine-tuned models. The second mode is used to generate the summaries of the 326 research papers of the test set by the Vanilla and fine-tuned models. The summaries generated by Llama 3.1 8B are considered to be the benchmark.\\u003c/p\\u003e \\u003cp\\u003eD_Scorer: This section is used to get the score of the summaries generated by the Vanilla and Fine-tuned text summarization models on the test dataset of 326 papers.\\u003c/p\\u003e \\u003cp\\u003eE_Model_Comparer: This section is used to compare the Vanilla models with their fine-tuned counterparts generating various Excel file tables.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"4. Results and Discussion\",\"content\":\"\\u003cp\\u003e \\u003cstrong\\u003eExample 1\\u003c/strong\\u003e \\u003cp\\u003eshows the summary generated by the Vanilla BART Large CNN Model.\\u003c/p\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003cstrong\\u003eExample 1\\u003c/strong\\u003e \\u003cp\\u003e \\u003cem\\u003eMistral 7B leverages grouped query attention (GQA) and sliding window attention (SWA) GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding. SWA is designed to handle longer sequences more effectively at a reduced computational cost.\\u003c/em\\u003e \\u003c/p\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003cstrong\\u003eExample 2\\u003c/strong\\u003e \\u003cp\\u003eshows the summary generated by the Fine-tuned BART Large CNN Model.\\u003c/p\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003cstrong\\u003eExample 2\\u003c/strong\\u003e \\u003cp\\u003e \\u003cem\\u003eLarge language models like Mistral 7B aim to balance high performance while maintaining efficient inference by leveraging grouped-query attention and sliding window attention. These attention mechanisms significantly accelerate inference speed and reduce memory requirements, making them suitable for real-time applications. The model is released under the Apache 2.0 license and has a reference implementation facilitating easy deployment on cloud platforms.\\u003c/em\\u003e \\u003c/p\\u003e \\u003c/p\\u003e \\u003cp\\u003eThese summaries have been generated for the snippet of the introduction paragraph of the Mistral 7B paper [\\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e] by Jiang et al. The snippet is available in \\u003cspan refid=\\\"Sec15\\\" class=\\\"InternalRef\\\"\\u003eAppendix\\u003c/span\\u003e A.\\u003c/p\\u003e \\u003cdiv id=\\\"Sec11\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.1. Challenges faced while fine-tuning the Vanilla models\\u003c/h2\\u003e \\u003cp\\u003eSeveral challenges were encountered in the process of developing the fine-tuning script. The first version involved manually handling details like dataset processing and tokenization in pursuance of fine-grained control over the GPU utilization. However, the fine-tuned models obtained as a result produced very low quality summaries consisting of words randomly placed with no sensibility or meaning. The primary reason for this problem was an improper implementation of the tokenizer and complexities arising from manually handling the dataset.\\u003c/p\\u003e \\u003cp\\u003eAnother anomaly observed is that the fine-tuned model was generating summaries much longer than needed, with several repeated phrases. This indicated that the distribution of training data needed to be fixed. The chunking strategy implemented by script was determined to be the source of the problem. It was found that there was no coherence in the pairing of the source document chunks with the summary chunks. Further improvement was achieved by changing the evaluation metric from pure loss to Rouge. After this, the authors tried various values of the per_device_train_batch_size, per_device_eval_batch_size and generation_max_length hyper parameters. It was found that a value of 4 for the first two and a value of 256 for the last hyperparameter resulted in good summaries.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec12\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.2. Best Summarization Type and Model for Each Test Research Paper\\u003c/h2\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e is the comparison of Vanilla and fine-tuned versions of BART Large CNN. The overlapping summarization type produces the best summaries for 187 out of 326 papers. The Layered Summarization is found best for 132 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type produces the best summaries for 297 out of 326 papers, i. e. 91% of the papers.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab1\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 1\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eBest Summarization and Model Type for BART Large CNN\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"3\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003ePaper No.\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eBest Summarization Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eBest Model Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eLayered\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e326\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eLayered\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eTotal\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping: 187; Layered: 132\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned: 297; Vanilla: 29\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab2\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 2\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eBest Summarization and Model Type for BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"3\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003ePaper No.\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eBest Summarization Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eBest Model Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eVanilla\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e326\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eTotal\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping: 198; Layered: 120\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned: 287; Vanilla: 39\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab2\\\" class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e is the comparison of Vanilla and fine-tuned versions of BART Large CNN SAMSum. The overlapping summarization type produces the best summaries for 198 out of 326 papers. The layered summarization produces the best summaries for 120 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type is the best, producing the best summaries for 287 out of 326 papers, or 88% of the papers.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab3\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 3\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eBest Summarization and Model Type for DistilBART\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"3\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003ePaper No.\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eBest Summarization Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eBest Model Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eLayered\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e326\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eTotal\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping: 182; Layered: 136\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned: 293; Vanilla: 33\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab3\\\" class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003e is the comparison of Vanilla and fine-tuned versions of DistilBART. The overlapping summarization type produces the best summary for 182 out of 326 papers. The layered summarization type produces the best summary for 136 out of 326 papers. Together, they account for nearly 98% of all papers. The fine-tuned model type produces the best summary for 293 out of 326 papers, or 91% of the papers.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab4\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 4\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eBest Summarization and Model Type for PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"3\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003ePaper No.\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eBest Summarization Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eBest Model Type\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e3\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e\\u0026hellip;\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003e326\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eVanilla\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eTotal\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eOverlapping: 286; Layered: 34\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eFine-tuned: 300, Vanilla: 26\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab4\\\" class=\\\"InternalRef\\\"\\u003e4\\u003c/span\\u003e is the comparison of Vanilla and fine-tuned versions of PEGASUS CNN DailyMail. The overlapping summarization type produces the best summaries for 286 out of 326 papers. Unlike other models before, it holds the absolute majority, accounting for nearly 88% of all papers. The fine-tuned model type produces the best summaries for 300 out of 326 papers, or 92% of the papers.\\u003c/p\\u003e \\u003cp\\u003eWith these observations, it can be concluded that the best summarization type is overlapping summarization. summing over the four tables, it scores 853 out of 1304, or about 65%. The next best is the layered type, scoring 422 out of 1304, or about 32%. Together they account for about 97% of the total scores. An interesting trend observed is that the more complex the summarization type, the worse is the performance. Overlapping summarization is the simplest type, while layered is more complex. Yet, overlapping has the highest scores, than the layered one. On the other hand, the layered summaries strike a balance in length between overlapping and rolling types. It can also be concluded that the best model type is fine-tuned, by a large margin. It scores 1117 out of 1304, or almost 90%. On the other hand, the Vanilla type scores just 127 out of 1304, or about 10%.\\u003c/p\\u003e \\u003c/div\\u003e \\u003cdiv id=\\\"Sec13\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003e4.3. Maximum, minimum and average of metrics\\u003c/h2\\u003e \\u003cp\\u003eThe maximum, minimum and average value for a metric is obtained by finding the maximum, minimum and average values respectively of that metric across all the research papers summarized by that model in the test dataset of 326 papers.\\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned BART Large CNN. It can be observed that for both the Vanilla as well as fine-tuned model, BERTScore precision has the highest maximum, minimum and average scores. Further, as compared to the vanilla model, the fine-tuned model has higher minimum and maximum values for all the three columns Max, Min as well as Avg.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab5\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 5\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eMaximum, Minimum and Average of metrics for Vanilla and Fine-tuned BART Large CNN\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"7\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c6\\\" colnum=\\\"6\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c7\\\" colnum=\\\"7\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eMetrics\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c4\\\" namest=\\\"c2\\\"\\u003e \\u003cp\\u003eVanilla BART Large CNN\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c7\\\" namest=\\\"c5\\\"\\u003e \\u003cp\\u003eFine-tuned BART Large CNN\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.6134\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.026\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.3118\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.6721\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0216\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.3524\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.3445\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1154\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.4444\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1616\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-L\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.4304\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0219\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1616\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.5495\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0216\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1963\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Precision\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5074\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7066\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8897\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.5422\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.755\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Recall\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8367\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.394\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6636\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8794\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.414\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.6824\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore F1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.7968\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4479\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6834\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8428\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.5075\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.716\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMETEOR\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.503\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0085\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.2005\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.6887\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0103\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.2308\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab6\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 6\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eMaximum, Minimum and Average of metrics for Vanilla and Fine-tuned BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"7\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c6\\\" colnum=\\\"6\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c7\\\" colnum=\\\"7\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eMetrics\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c4\\\" namest=\\\"c2\\\"\\u003e \\u003cp\\u003eVanilla BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c7\\\" namest=\\\"c5\\\"\\u003e \\u003cp\\u003eFine-tuned BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.6258\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0048\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.3207\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.7041\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0332\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.3562\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.3752\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.126\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.4433\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1605\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-L\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.4428\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0048\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1689\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.5714\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0257\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1958\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Precision\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4134\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7148\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8871\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.6165\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7467\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Recall\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8351\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.3514\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6653\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8798\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4719\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.6866\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore F1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8258\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.3821\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6884\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8643\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.5485\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7148\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMETEOR\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.5462\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0062\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.2039\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.6373\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0108\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.2333\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab6\\\" class=\\\"InternalRef\\\"\\u003e6\\u003c/span\\u003e shows the maximum, minimum and average score for each metric for Vanilla and Fine-tuned BART Large CNN SAMSum. It can be observed for both the Vanilla as well as fine-tuned model that the BERTScore Precision has the highest maximum, minimum and average scores. Further, as compared to the vanilla model, the fine-tuned model has higher minimum and maximum values for all the three columns Max, Min as well as Avg.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab7\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 7\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eMaximum, minimum and average of metrics for Vanilla and fine-tuned DistilBART\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"7\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c6\\\" colnum=\\\"6\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c7\\\" colnum=\\\"7\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eMetrics\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c4\\\" namest=\\\"c2\\\"\\u003e \\u003cp\\u003eVanilla DistilBART\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c7\\\" namest=\\\"c5\\\"\\u003e \\u003cp\\u003eFine-tuned DistilBART\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.62\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.031\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.3281\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.7141\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0341\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.3488\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.381\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1267\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.5197\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.157\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-L\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.5155\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0232\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1714\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.5525\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0272\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1916\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Precision\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4711\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7225\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8838\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4664\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7513\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Recall\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8382\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4396\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6725\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8751\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4201\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.6807\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore F1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.819\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.486\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6957\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8685\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4434\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7135\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMETEOR\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.5257\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.008\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.2095\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.7027\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0116\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.2265\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab7\\\" class=\\\"InternalRef\\\"\\u003e7\\u003c/span\\u003e shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned DistilBART. It can be observed for both the Vanilla as well as fine-tuned model that the BERTScore precision has the maximum and average scores. Also the highest minimum score for the fine-tuned model also belongs to BERTScore precision. It is also observed that for Vanilla model BERTScore F1 has the highest minimum score. Further, as compared to the vanilla model, the fine-tuned model has better minimum and maximum values for two of the three columns, Max and Avg. However, for the Min column, the Vanilla model does better.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab8\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 8\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eMax, Min and Average of Metrics for Vanilla and Fine-tuned PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"7\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c6\\\" colnum=\\\"6\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c7\\\" colnum=\\\"7\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\" morerows=\\\"1\\\" rowspan=\\\"2\\\"\\u003e \\u003cp\\u003eMetrics\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c4\\\" namest=\\\"c2\\\"\\u003e \\u003cp\\u003eVanilla PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colspan=\\\"3\\\" nameend=\\\"c7\\\" namest=\\\"c5\\\"\\u003e \\u003cp\\u003eFine-tuned PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eMax\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eMin\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003eAvg\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.657\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0204\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.2943\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.6913\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0261\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.3245\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-2\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.3692\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1117\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.4086\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1401\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eROUGE-L\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.3978\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0155\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1553\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.4198\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.0195\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.1796\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Precision\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8556\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5299\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6962\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8905\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.5781\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7483\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore Recall\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8224\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4084\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6486\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8785\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4265\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.6621\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eBERTScore F1\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.7879\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4663\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6704\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.8226\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.4972\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.7019\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMETEOR\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.5389\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.0077\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.1765\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003e0.6843\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003e0.009\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c7\\\"\\u003e \\u003cp\\u003e0.2013\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab8\\\" class=\\\"InternalRef\\\"\\u003e8\\u003c/span\\u003e shows the maximum, minimum and average score for each metric for Vanilla and fine-tuned PEGASUS CNN DailyMail. It can be observed that for both the Vanilla as well as fine-tuned model, BERTScore precision metric has the highest maximum, minimum and average scores. Further, as compared to the Vanilla model, the fine-tuned model has better minimum and maximum values for all the three columns Max, Min and Avg.\\u003c/p\\u003e \\u003cp\\u003e \\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab9\\\" border=\\\"1\\\"\\u003e \\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 9\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eSummary of Tables\\u0026nbsp;\\u003cspan refid=\\\"Tab5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e, \\u003cspan refid=\\\"Tab6\\\" class=\\\"InternalRef\\\"\\u003e6\\u003c/span\\u003e, \\u003cspan refid=\\\"Tab7\\\" class=\\\"InternalRef\\\"\\u003e7\\u003c/span\\u003e and \\u003cspan refid=\\\"Tab8\\\" class=\\\"InternalRef\\\"\\u003e8\\u003c/span\\u003e.\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e \\u003ccolgroup cols=\\\"4\\\"\\u003e \\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e \\u003cdiv align=\\\"char\\\" char=\\\".\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e \\u003cthead\\u003e \\u003ctr\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eModel\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eMax Max\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eMax Min\\u003c/p\\u003e \\u003c/th\\u003e \\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eMax Avg\\u003c/p\\u003e \\u003c/th\\u003e \\u003c/tr\\u003e \\u003c/thead\\u003e \\u003ctbody\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eVanilla BART Large CNN\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5074\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7066\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eFine-tuned BART Large CNN\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8897\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5422\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.755\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eVanilla BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4134\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7148\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eFine-tuned BART Large CNN SAMSum\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8871\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.6165\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7467\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eVanilla DistilBART CNN 12 6\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8761\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.486\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7225\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eFine-tuned DistilBART CNN 12 6\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8838\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.4664\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7513\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eVanilla PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8556\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5299\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.6962\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003ctr\\u003e \\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eFine-tuned PEGASUS CNN DailyMail\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003e0.8905\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003e0.5781\\u003c/p\\u003e \\u003c/td\\u003e \\u003ctd align=\\\"char\\\" char=\\\".\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003e0.7483\\u003c/p\\u003e \\u003c/td\\u003e \\u003c/tr\\u003e \\u003c/tbody\\u003e \\u003c/colgroup\\u003e \\u003c/table\\u003e\\u003c/div\\u003e \\u003c/p\\u003e \\u003cp\\u003eTable\\u0026nbsp;\\u003cspan refid=\\\"Tab9\\\" class=\\\"InternalRef\\\"\\u003e9\\u003c/span\\u003e has been created by taking the minimum and maximum value of the Max, Min and Avg columns of Tables \\u003cspan refid=\\\"Tab5\\\" class=\\\"InternalRef\\\"\\u003e5\\u003c/span\\u003e, \\u003cspan refid=\\\"Tab6\\\" class=\\\"InternalRef\\\"\\u003e6\\u003c/span\\u003e, \\u003cspan refid=\\\"Tab7\\\" class=\\\"InternalRef\\\"\\u003e7\\u003c/span\\u003e and \\u003cspan refid=\\\"Tab8\\\" class=\\\"InternalRef\\\"\\u003e8\\u003c/span\\u003e for the Vanilla and fine-tuned models for BERTScore precision metric. For nearly every pair of minimum values and every pair of maximum values, the fine-tuned variant of each model produces better results as compared to the Vanilla variant of that model. The model having the maximum score is the fine-tuned variant of BART Large CNN SAMSum. Section \\u003cspan refid=\\\"Sec13\\\" class=\\\"InternalRef\\\"\\u003e4.3\\u003c/span\\u003e. mentioned the best summarization type and model type (Vanilla or fine-tuned). This section mentioned the fine-tuned BART Large CNN SAMSum as the best model.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"5. Conclusion\",\"content\":\"\\u003cp\\u003eThis work demonstrates the performance of four fine-tuned transformer-based models namely BART Large CNN, BART Large CNN SAMSum, DistilBART and PEGASUS CNN DailyMail. DistilBART provides the best performance among Vanilla models, despite being the smallest. Fine-tuning the models results in a universal improvement in performance, with BART Large CNN SAMSum and BART Large CNN models being the best. In this scenario, DistilBART secured the best score for only the BERTScore precision metric. This work highlights improved performance achieved by fine-tuning the models. This also indicates that other vanilla models had better capacity to learn new distributions of data as compared to DistilBART. On the other hand, PEGASUS, the largest model, displayed the worst performance despite fine-tuning. It indicates that model size and performance are not necessarily correlated. Thus, it can be concluded that fine-tuned versions of the Vanilla models outperform their Vanilla counterparts, and that overlapping summarization is the best method of summarization. Overall, it has been observed that the fine-tuned BART Large CNN SAMSum is the best model for research paper summarization.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003cp\\u003eAcknowledgements\\u003c/p\\u003e\\n\\u003cp\\u003eThe authors of this paper feel grateful to KJ Somaiya School of Engineering for access to High Performance Computing Lab. The authors also express gratitude to Dr. Grishma J Sharma from KJ Somaiya School of Engineering for constructive feedback.\\u0026nbsp;\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eFunding\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eNot applicable. This research did not receive funding.\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eConflict of interest/ Competing interests\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eThe authors declare that they have no known competing interest\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eEthics approval and consent to participate\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eNot applicable\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eConsent for publication\\u003c/strong\\u003e\\u003c/li\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eData availability\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eData and code is available athttps://github.com/sushantnair/AIStudy\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eMaterial availability\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eNot applicable\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eCode availability\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eData and code is available athttps://github.com/sushantnair/AIStudy\\u003c/p\\u003e\\n\\u003cul\\u003e\\n \\u003cli\\u003e\\u003cstrong\\u003eAuthor contribution\\u003c/strong\\u003e\\u003c/li\\u003e\\n\\u003c/ul\\u003e\\n\\u003cp\\u003eS. N. and P. G. have equally contributed to this work.\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\u003cli\\u003e\\u003cspan\\u003eSamsi S et al (2023) From words to watts: Benchmarking the energy costs of large language model inference. 2023 IEEE High Performance Extreme Computing Conference (HPEC). IEEE\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eRen S et al (2024) Reconciling the contrasting narratives on the environmental impact of large language models. Sci Rep 14(1):26310\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKhan T et al (2025) Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights. arXiv preprint arXiv:2504.06307\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLi P et al (2023) Making ai less thirsty: Uncovering and addressing the secret water footprint of ai models. arXiv preprint arXiv:2304.03271 (\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eChen Z et al (2022) Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.02813\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eDesislavov R (2021) Fernando Mart\\u0026iacute;nez-Plumed, and Jos\\u0026eacute; Hern\\u0026aacute;ndez-Orallo. Compute and energy consumption trends in deep learning inference. arXiv preprint arXiv:2109.05472\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eAllahyari M et al (2017) Text summarization techniques: a brief survey. arXiv preprint arXiv:1707.02268\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003e\\u0026Ouml;zdemir S (2018) The effect of summarization strategies teaching on strategy usage and narrative text summarization success\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eVaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLewis M et al (2019) Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eZhang J et al (2020) Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. International conference on machine learning. PMLR\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003ePapineni K et al (2002) Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eLin C-Y (2004) Rouge: A package for automatic evaluation of summaries. Text summarization branches out\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eZhang T et al (2019) Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eBanerjee, Satanjeev, and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.2005\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eJiang AQ et al Mistral 7B, pp. 1\\u0026ndash;9, [Online]. Available: \\u003cspan class=\\\"ExternalRef\\\"\\u003e\\u003cspan class=\\\"RefSource\\\"\\u003ehttps://mistral.ai/news/announcing-mistral-7b/\\u003c/span\\u003e\\u003cspan address=\\\"https://mistral.ai/news/announcing-mistral-7b/\\\" targettype=\\\"URL\\\" class=\\\"RefTarget\\\"\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eRoziere B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, Synnaeve G (2023) Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eTouvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Scialom T (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eTouvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Lample G (2023) Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eKwon W, Li Z, Zhuang S, Sheng Y, Zheng L, Yu CH, Stoica I (2023), October Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles (pp. 611\\u0026ndash;626)\\u003c/span\\u003e\\u003c/li\\u003e \\u003cli\\u003e\\u003cspan\\u003eChaplot DS (2023) Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, l\\u0026eacute;lio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timoth\\u0026eacute;e lacroix, william el sayed. arXiv preprint arXiv:2310.06825, 3.\\u003c/span\\u003e\\u003c/li\\u003e\\u003c/ol\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":true,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"K J Somaiya School of Engineering\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"Text Summarization, Natural Language Processing, Summarization Models, BART Models\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-8658398/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-8658398/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003eIn the realm of academics and research work, efficient, accurate and privacy-focused summarization of research papers have emerged as significant areas of interest. The availability of such a tool would make the tasks like literature survey much easier and faster, enabling academics to channel their energy to other aspects of their work. In this work the best summarization model is identified by comparing the performance of four State Of The Art (SOTA) text summarization models. These models are Facebook\\u0026rsquo;s Bart Large CNN, Phil Schmid\\u0026rsquo;s Bart Large CNN SAMSum (Bart Large CNN fine-tuned on the SAMSum dataset), Sam Shleifer\\u0026rsquo;s DistilBART CNN 12 6 and Google\\u0026rsquo;s PEGASUS CNN Dailymail. The initial part of this work focuses on evaluating the summaries generated by the Vanilla models. This dataset has been obtained from the AI Arxiv2 Dataset, which contains a diverse range of information about research papers published in ArXiv under the AI domain. The generated summaries are evaluated using metrics like BLEU, ROUGE, BERTScore and METEOR. The models are compared to determine the best model for summarization. This work involves fine-tuning the Vanilla models on a separate train dataset with 1,377 examples obtained from the AI Arxiv2 Dataset, in an attempt to improve their ability to summarize text containing AI jargon and terminology. The later part focuses on evaluating the summaries generated by these fine-tuned models on the test dataset with 326 examples.\\u003c/p\\u003e\",\"manuscriptTitle\":\"Comparing the Performance of SOTA Text Summarization Models on AI Research Papers\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2026-01-22 04:40:16\",\"doi\":\"10.21203/rs.3.rs-8658398/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"3badd52f-8e07-49b9-9922-04d710a78a1b\",\"owner\":[],\"postedDate\":\"January 22nd, 2026\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[{\"id\":61501078,\"name\":\"Artificial Intelligence and Machine Learning\"}],\"tags\":[],\"updatedAt\":\"2026-01-22T04:40:17+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2026-01-22 04:40:16\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-8658398\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-8658398\",\"identity\":\"rs-8658398\",\"version\":[\"v1\"]},\"buildId\":\"XKTyCvWXoU3ODBz1xrDgd\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}