PhosF3C: A Feature Fusion Architecture with Fine-Tuned Protein Language Model and Conformer for prediction of general phosphorylation site

PhosF3C: A Feature Fusion Architecture with Fine-Tuned Protein Language Model and Conformer for prediction of general phosphorylation site | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article PhosF3C: A Feature Fusion Architecture with Fine-Tuned Protein Language Model and Conformer for prediction of general phosphorylation site Yuhuan Liu, Haitian Zhong, Jixiu Zhai, Xueying Wang, Tianchi LU This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5871318/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Protein phosphorylation, a key post-translational modification (PTM), provides essential insight into protein properties, making its prediction highly significant. Using the emerging capabilities of large language models (LLMs), we apply LoRA fine-tuning to ESM2, a powerful protein large language model, to efficiently extract features with minimal computational resources, optimizing task-specific text alignment. Additionally, we integrate the conformer architecture with the Feature Coupling Unit (FCU) to enhance local and global feature exchange, further improving prediction accuracy. Our model achieves state-of-the-art (SOTA) performance, obtaining AUC scores of 79.5%, 76.3%, and 71.4% at the S, T, and Y sites of the general data sets. Based on the powerful feature extraction capabilities of LLMs, we conduct a series of analyses on protein representations, including studies on their structure, sequence, and various chemical properties (such as Hydrophobicity (GRAVY), Surface Charge, and Isoelectric Point). We propose a test method called Linear Regression Tomography (LRT) which is a top-down method using representation to explore the model's feature extraction capabilities, offering a pathway to improved interpretability. Our resources, including data and code, are publicly accessible at \url{ https://github.com/SkywalkerLuke/PhosF3C} Protein Phosphorylation Large language model LoRA Conformer Full Text Additional Declarations No competing interests reported. Supplementary Files PhosF3CSupplementaryMaterial.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5871318","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":405034125,"identity":"af9521f4-173e-4552-9aaf-7a5125897c48","order_by":0,"name":"Yuhuan Liu","email":"","orcid":"","institution":"Lanzhou University","correspondingAuthor":false,"prefix":"","firstName":"Yuhuan","middleName":"","lastName":"Liu","suffix":""},{"id":405034126,"identity":"bc6808bf-a370-48ef-81a7-919c2f707da2","order_by":1,"name":"Haitian Zhong","email":"","orcid":"","institution":"Chinese Academy of Sciences","correspondingAuthor":false,"prefix":"","firstName":"Haitian","middleName":"","lastName":"Zhong","suffix":""},{"id":405034127,"identity":"8e3b6c16-5bd6-4513-a099-f061b815d405","order_by":2,"name":"Jixiu Zhai","email":"","orcid":"","institution":"Lanzhou University","correspondingAuthor":false,"prefix":"","firstName":"Jixiu","middleName":"","lastName":"Zhai","suffix":""},{"id":405034130,"identity":"2525a489-5eb0-4087-be2d-ad2be66e1f55","order_by":3,"name":"Xueying Wang","email":"","orcid":"","institution":"City University of Hong Kong","correspondingAuthor":false,"prefix":"","firstName":"Xueying","middleName":"","lastName":"Wang","suffix":""},{"id":405034132,"identity":"480ddbf2-115b-479f-95f7-416d9fd7a1b8","order_by":4,"name":"Tianchi LU","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHUlEQVRIie2RvUoDQRCAZxm5a8Zcu8eZPMPJghr8exUloI22YhUEITaeaRV8iKsOy5GFpAnYpjM2VhYHQuCqZO9MDOgSLQX3g91idr+dmR0Ah+MvgmaJC7kGwAD5PFIFgdBm0EwhEAzitlLwB2V+CsiA9Jl4ibLv4+uoeNiiuMfiffdEQ3AV9CKCnXrKqzq2FuZtricDSfGAMTrNNEiNnlGOVMq11oG9l41IdIwyfBpjqcRYKfowZVJsVfzxh/I88rC5UCZLFJplYfZQLBSuFGthms7CxCghM4ZJdkymF9W8j1vqTpOyte93+5ksOu1GjVnkRbbdCLqPL8O38736Tf9aSdsvf+GyHM2KhPJ561S+0y43zH932eFwOP4JU1OuVZaykTwRAAAAAElFTkSuQmCC","orcid":"","institution":"City University of Hong Kong","correspondingAuthor":true,"prefix":"","firstName":"Tianchi","middleName":"","lastName":"LU","suffix":""}],"badges":[],"createdAt":"2025-01-21 07:53:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5871318/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5871318/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75705791,"identity":"2095a7b3-a685-4653-899e-ed1d9200dc79","added_by":"auto","created_at":"2025-02-07 10:17:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":7166216,"visible":true,"origin":"","legend":"","description":"","filename":"PhosF3C.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5871318/v1_covered_f0597b9c-847d-467f-b2b5-af502f33fac1.pdf"},{"id":74575550,"identity":"5920f9d9-9e4c-4e5c-a6ad-3c061533e042","added_by":"auto","created_at":"2025-01-23 14:59:46","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":5210183,"visible":true,"origin":"","legend":"","description":"","filename":"PhosF3CSupplementaryMaterial.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5871318/v1/3b9a8a3aa709461af5a7c930.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"PhosF3C: A Feature Fusion Architecture with Fine-Tuned Protein Language Model and Conformer for prediction of general phosphorylation site","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Protein Phosphorylation, Large language model, LoRA, Conformer","lastPublishedDoi":"10.21203/rs.3.rs-5871318/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5871318/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Protein phosphorylation, a key post-translational modification (PTM), provides essential insight into protein properties, making its prediction highly significant. Using the emerging capabilities of large language models (LLMs), we apply LoRA fine-tuning to ESM2, a powerful protein large language model, to efficiently extract features with minimal computational resources, optimizing task-specific text alignment. Additionally, we integrate the conformer architecture with the Feature Coupling Unit (FCU) to enhance local and global feature exchange, further improving prediction accuracy. Our model achieves state-of-the-art (SOTA) performance, obtaining AUC scores of 79.5%, 76.3%, and 71.4% at the S, T, and Y sites of the general data sets. Based on the powerful feature extraction capabilities of LLMs, we conduct a series of analyses on protein representations, including studies on their structure, sequence, and various chemical properties (such as Hydrophobicity (GRAVY), Surface Charge, and Isoelectric Point). We propose a test method called Linear Regression Tomography (LRT) which is a top-down method using representation to explore the model's feature extraction capabilities, offering a pathway to improved interpretability. Our resources, including data and code, are publicly accessible at \\url{https://github.com/SkywalkerLuke/PhosF3C}","manuscriptTitle":"PhosF3C: A Feature Fusion Architecture with Fine-Tuned Protein Language Model and Conformer for prediction of general phosphorylation site","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-23 14:51:35","doi":"10.21203/rs.3.rs-5871318/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"15b6c06d-632e-4a5e-90de-cfac2e3b4055","owner":[],"postedDate":"January 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-07T10:08:53+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-23 14:51:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5871318","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5871318","identity":"rs-5871318","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}