Post-Translational Modification Prediction via Prompt-Based Fine-Tuning of a GPT-2 Model

Post-Translational Modification Prediction via Prompt-Based Fine-Tuning of a GPT-2 Model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Post-Translational Modification Prediction via Prompt-Based Fine-Tuning of a GPT-2 Model Palistha Shrestha, Jeevan Kandel, Hilal Tayara, Kil To Chong This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4194361/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Post-translational modifications (PTMs) are pivotal in modulating protein functions, influencing key cellular processes such as signaling, localization, and protein degradation. The complexity of these biological interactions necessitates efficient predictive methodologies. In this work, we introduce PTMGPT2, an interpretable protein language model that utilizes prompt-based fine-tuning to improve its accuracy and generalizability in precisely predicting PTMs. Drawing inspiration from recent advancements in GPT-based architectures, PTMGPT2 adopts an unsupervised learning approach to identify PTMs. It utilizes a custom prompt to guide the model through the subtle linguistic patterns encoded in amino acid sequences, generating tokens indicative of PTM sites. To provide interpretability, we visualize attention profiles from the model’s final decoder layer, elucidating sequence motifs essential for molecular recognition and modification variability. Furthermore, we conducted analyses to investigate the effects of mutations at or near PTM sites, thereby offering deeper insights into protein functionality. Our analysis encompasses a comprehensive dataset comprising 3,88,084 modification sites across 19 distinct PTM types, facilitating the identification of novel PTM sites. Comparative assessments reveal that PTMGPT2 outperforms existing methods by an average 5.45% in MCC, underscoring its potential in identifying novel therapeutic strategies, disease associations, and drug targets. Bioinformatics Artificial Intelligence and Machine Learning PTM sites Unsupervised learning Prompt design Generative model Full Text Additional Declarations The authors declare no competing interests. Supplementary Files Supplementaryfile1Nature.pdf Supplementaryfile2Nature.xlsx Supplementaryfile3Nature.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4194361","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":285779517,"identity":"8c24fc96-21b2-41ef-8346-69f21ae7c5dd","order_by":0,"name":"Palistha Shrestha","email":"","orcid":"https://orcid.org/0009-0001-3660-0150","institution":"Jeonbuk National University","correspondingAuthor":false,"prefix":"","firstName":"Palistha","middleName":"","lastName":"Shrestha","suffix":""},{"id":285779518,"identity":"869fea36-547c-4828-bf24-24101e0dc27f","order_by":1,"name":"Jeevan Kandel","email":"","orcid":"https://orcid.org/0000-0001-5223-355X","institution":"Jeonbuk National University","correspondingAuthor":false,"prefix":"","firstName":"Jeevan","middleName":"","lastName":"Kandel","suffix":""},{"id":285779519,"identity":"77365cd9-8ac9-413c-9484-994f83c4bee4","order_by":2,"name":"Hilal Tayara","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+0lEQVRIiWNgGAWjYBACA2bmBoYHEDbjA8kGKKsBrxagdEICmM1sQJwWBoQWNgmYUrxazNkZGz8k/rBL7J99+FmF5Y47iQ3shx8wztyDW4tlM2OzREJCcuKMc2lmNyTPPEts4EkzYNzwDI/DDjM2ALUwJzacYQBqaTuc2MCQAwyIA3i1NP9ISKhPnH+G/VsBWAv/G4Ja2oC2HE7ccIbHjAGsRQJoywYCWiwS0o4bbzzDUywheeawcZvEM4ODM/BpOX/48I0PNtWy886wb/wsueOwbD9/8sOHPXi0wIBjA5BglgASbEBMhAYGBnsQwfiBGKWjYBSMglEw4gAAUOxaolSDVOwAAAAASUVORK5CYII=","orcid":"","institution":"Jeonbuk National University","correspondingAuthor":true,"prefix":"","firstName":"Hilal","middleName":"","lastName":"Tayara","suffix":""},{"id":285779668,"identity":"a528ec52-8ac1-4764-8c76-6a959297f123","order_by":3,"name":"Kil To Chong","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3ElEQVRIie3QPQrCMBTA8QcPOr2SNSVgr1ApOLZXqRR0dSw4GBDaKzh4C8E5pVCXVi/goAhObr2AX1W31FEw/yUhvB8kATCZfjCOAHjfMKjaI+ok+CSO/JpASzz1Gu0iTsbyZpLsg1VelyeCwAWqD1oiEFEsqnO8VtuxTxD3pZ15WtJDtNBOi3igqoEgwAiYpb/Ym/jyQWbdRLQk8OBBigjsVE+cOfqCqiLiajtylt6mn1KpJ3yXHxtKipAt6pJfkqnLaKQnr4aSq9ty+6uOl3wKgclvZ00mk+nfugJX+TiWLHeVUQAAAABJRU5ErkJggg==","orcid":"","institution":"Jeonbuk National University","correspondingAuthor":true,"prefix":"","firstName":"Kil","middleName":"To","lastName":"Chong","suffix":""}],"badges":[],"createdAt":"2024-03-31 06:09:45","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4194361/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4194361/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":53827988,"identity":"45e95f72-4826-4728-832e-7e4244504a6f","added_by":"auto","created_at":"2024-04-01 03:41:51","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2389376,"visible":true,"origin":"","legend":"","description":"","filename":"PostTranslationalModificationPredictionviaPromptBasedFineTuningofaGPT2Model.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4194361/v1_covered_a0e88892-2979-4fdd-b7ca-24f00457f05e.pdf"},{"id":53827834,"identity":"da623bb9-0a75-4c67-bf29-9d0d15749649","added_by":"auto","created_at":"2024-04-01 03:33:47","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":353933,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile1Nature.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4194361/v1/19fcd7075e30a3ad5ba0105c.pdf"},{"id":53827835,"identity":"c05d6fc7-9beb-445c-a9d2-295a93af5feb","added_by":"auto","created_at":"2024-04-01 03:33:47","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":2014753,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile2Nature.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-4194361/v1/559ff51506295cea2eb0102a.xlsx"},{"id":53827836,"identity":"0ae65f38-227a-4f8c-88cd-b323c85c6a3b","added_by":"auto","created_at":"2024-04-01 03:33:47","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":1466779,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile3Nature.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4194361/v1/df7afd101eb7d0f2ff39ca6d.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003ePost-Translational Modification Prediction via Prompt-Based Fine-Tuning of a GPT-2 Model\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Jeonbuk National University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"PTM sites, Unsupervised learning, Prompt design, Generative model","lastPublishedDoi":"10.21203/rs.3.rs-4194361/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4194361/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003ePost-translational modifications (PTMs) are pivotal in modulating protein functions, influencing key cellular processes such as signaling, localization, and protein degradation. The complexity of these biological interactions necessitates efficient predictive methodologies. In this work, we introduce PTMGPT2, an interpretable protein language model that utilizes prompt-based fine-tuning to improve its accuracy and generalizability in precisely predicting PTMs. Drawing inspiration from recent advancements in GPT-based architectures, PTMGPT2 adopts an unsupervised learning approach to identify PTMs. It utilizes a custom prompt to guide the model through the subtle linguistic patterns encoded in amino acid sequences, generating tokens indicative of PTM sites. To provide interpretability, we visualize attention profiles from the model’s final decoder layer, elucidating sequence motifs essential for molecular recognition and modification variability. Furthermore, we conducted analyses to investigate the effects of mutations at or near PTM sites, thereby offering deeper insights into protein functionality. Our analysis encompasses a comprehensive dataset comprising 3,88,084 modification sites across 19 distinct PTM types, facilitating the identification of novel PTM sites. Comparative assessments reveal that PTMGPT2 outperforms existing methods by an average 5.45% in MCC, underscoring its potential in identifying novel therapeutic strategies, disease associations, and drug targets.\u003c/p\u003e","manuscriptTitle":"Post-Translational Modification Prediction via Prompt-Based Fine-Tuning of a GPT-2 Model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-01 03:33:42","doi":"10.21203/rs.3.rs-4194361/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ca000ebb-5856-4df7-9906-6e040243b21b","owner":[],"postedDate":"April 1st, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":30081446,"name":"Bioinformatics"},{"id":30081447,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2024-04-01T03:33:42+00:00","versionOfRecord":[],"versionCreatedAt":"2024-04-01 03:33:42","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4194361","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4194361","identity":"rs-4194361","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}