Generation of Pull Request Description using Transformers

Generation of Pull Request Description using Transformers | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Generation of Pull Request Description using Transformers Manisha Saini, Kartik Agarwala, Aditi Singh, Shreya Rustagi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7089220/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Pull Requests (PR) is a mechanism by which project owners are notified about the changes made by a developer to merge their proposed changes into a project’s codebase. A PR is made up of several interrelated commits. To create a PR, developers need to provide a title and a description explaining what changes they made and why. These descriptions are important because they help reviewers quickly understand the purpose of the changes without having to get into the details. Developers often forget to write descriptions for their PRs, so it's helpful for them to have a way to generate these descriptions automatically. We conducted a comparative analysis of transformer-based models (T5, FLAN-T5, BART, and PLBART) for generating pull request descriptions. The evaluation utilized ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L) to measure recall, precision, and F1-score for each model's effectiveness. Among the models, BARTbase outperformed the other transformer models. We proposed a fine-tuned BART-base transformer model with hyperparameters: a learning rate of 0.00002736 and a weight decay of 0.1. From experimental analysis we inferred that our proposed approach outperformed other state-of-the-art approaches. Pull Requests Transformer Summarization PR Descriptions Full Text Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7089220","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":494987979,"identity":"669e86e7-059c-428a-8901-bec656ea6f03","order_by":0,"name":"Manisha Saini","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA90lEQVRIiWNgGAWjYJCCA4wNIApIJlQcAItIgBFxWs4QqYUBogXEaINrwQ10288ePPhzhx2DfPvhtgcP592JNjjAfPA2D4NFHi4tZmfyEg7znklmMDiT2G6QuO1Z7oYDbMnWPAwSxTi1HMgxOMzYxsxgwJDYJpG47TBQC4+ZNFBLYgMuLeffGBz82VbPIN//EKhlDkgL/zf8Wm7kGBzgbTvMwHADZEsD2BY2AlreGBzmbTvOY3ADaEvCsWe5Mw+zGVvOMcDnsBzjjz/bquXk+9OfSf6ouZPbd7z54Y03FXU4tcAAD4LJDCIMCKgfBaNgFIyCUYAXAADCnV8wDYyqhQAAAABJRU5ErkJggg==","orcid":"","institution":"BML Munjal University","correspondingAuthor":true,"prefix":"","firstName":"Manisha","middleName":"","lastName":"Saini","suffix":""},{"id":494987980,"identity":"552bb7a3-0310-42f2-84e7-8539586a3da7","order_by":1,"name":"Kartik Agarwala","email":"","orcid":"","institution":"BML Munjal University","correspondingAuthor":false,"prefix":"","firstName":"Kartik","middleName":"","lastName":"Agarwala","suffix":""},{"id":494987981,"identity":"fe618397-1cd3-44ee-97cd-31c3eb1c842d","order_by":2,"name":"Aditi Singh","email":"","orcid":"","institution":"BML Munjal University","correspondingAuthor":false,"prefix":"","firstName":"Aditi","middleName":"","lastName":"Singh","suffix":""},{"id":494987982,"identity":"58dd1514-b2e2-46f6-bff4-9cff29f0d12e","order_by":3,"name":"Shreya Rustagi","email":"","orcid":"","institution":"BML Munjal University","correspondingAuthor":false,"prefix":"","firstName":"Shreya","middleName":"","lastName":"Rustagi","suffix":""}],"badges":[],"createdAt":"2025-07-10 05:44:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7089220/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7089220/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":91098986,"identity":"43f366ad-5e32-439f-bf7d-b24e78e993bf","added_by":"auto","created_at":"2025-09-11 14:34:50","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1105175,"visible":true,"origin":"","legend":"","description":"","filename":"FPullRequestSummarizationFinaldraft.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7089220/v1_covered_371b1868-39c4-4bef-b67f-e9efd4c9a758.pdf"}],"financialInterests":"","formattedTitle":"Generation of Pull Request Description using Transformers","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Pull Requests, Transformer, Summarization, PR Descriptions","lastPublishedDoi":"10.21203/rs.3.rs-7089220/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7089220/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Pull Requests (PR) is a mechanism by which project owners are notified about the changes made by a developer to merge their proposed changes into a project’s codebase. A PR is made up of several interrelated commits. To create a PR, developers need to provide a title and a description explaining what changes they made and why. These descriptions are important because they help reviewers quickly understand the purpose of the changes without having to get into the details. Developers often forget to write descriptions for their PRs, so it's helpful for them to have a way to generate these descriptions automatically. We conducted a comparative analysis of transformer-based models (T5, FLAN-T5, BART, and PLBART) for generating pull request descriptions. The evaluation utilized ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L) to measure recall, precision, and F1-score for each model's effectiveness. Among the models, BARTbase outperformed the other transformer models. We proposed a fine-tuned BART-base transformer model with hyperparameters: a learning rate of 0.00002736 and a weight decay of 0.1. From experimental analysis we inferred that our proposed approach outperformed other state-of-the-art approaches.","manuscriptTitle":"Generation of Pull Request Description using Transformers","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-05 18:27:08","doi":"10.21203/rs.3.rs-7089220/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e7ae7777-5eed-47c7-88b1-773f257b1ca8","owner":[],"postedDate":"August 5th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-11T14:26:42+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-05 18:27:08","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7089220","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7089220","identity":"rs-7089220","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}