A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods

A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods Haifeng chen, Zhuoqi Zheng, Bo Zhang, Bozitao Zhong, Kexin Liu, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4249839/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract De novo protein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal that FrameFlow and RFdiffusion in unconditional generation and GPDL-H in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts are available at https://github.com/Immortals-33/Scaffold-Lab . Biological sciences/Computational biology and bioinformatics Biological sciences/Biophysics/Computational biophysics de novo protein design protein backbone generation evaluation benchmark Figures Figure 1 Figure 2 Figure 3 Figure 4 Full Text Additional Declarations There is NO Competing Interest. Supplementary Files ScaffoldLab0403SIFinal.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4249839","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":290266902,"identity":"2d5f7368-7c78-49c6-9392-86e151fea291","order_by":0,"name":"Haifeng chen","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAklEQVRIiWNgGAWjYBACAwbGhg8QBgPjAwYeqCABLY0zoMqYDRh4DIjRwsAI08ImAVWNX4s5++HGho87auXN2XuPVf6Q+ZPYwN68TYKh5g5OLZY9iY2NM88cN9zZcy7tNg+PQWIDz7EyCYZjz3A77EBi+2PetmMJBjdyzG4zgLRI5JhJMDYcxq3l/MPG5r8gLfffmBX+AGmRf0NAy43ExmbGthqgLTxmDGCHSfAQ0vKwsbG37YDhhjM5xtI8PMbGbTxpxRYJx/A5LP1hw8+2OnmD42cMP/7skZPtZz+88caHGtxaoACqgLEHGDsgRgIhDQwMdVD6B2Glo2AUjIJRMPIAADltWnnYO+50AAAAAElFTkSuQmCC","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":true,"prefix":"","firstName":"Haifeng","middleName":"","lastName":"chen","suffix":""},{"id":290266903,"identity":"e9835cf5-9d4a-4850-984f-500e8247691b","order_by":1,"name":"Zhuoqi Zheng","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Zhuoqi","middleName":"","lastName":"Zheng","suffix":""},{"id":290266904,"identity":"9fabcdab-56b7-4e14-9c31-414308be2340","order_by":2,"name":"Bo Zhang","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Bo","middleName":"","lastName":"Zhang","suffix":""},{"id":290266905,"identity":"66de107f-0b42-481c-a537-033200889709","order_by":3,"name":"Bozitao Zhong","email":"","orcid":"https://orcid.org/0000-0001-9363-6099","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Bozitao","middleName":"","lastName":"Zhong","suffix":""},{"id":290266906,"identity":"aa72a155-2e6b-43db-b16a-a79911030cd6","order_by":4,"name":"Kexin Liu","email":"","orcid":"https://orcid.org/0009-0005-5331-2870","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Kexin","middleName":"","lastName":"Liu","suffix":""},{"id":290266907,"identity":"3f7d6518-b1e5-4e02-a12c-6d4a1f7bcead","order_by":5,"name":"Zhengxin Li","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Zhengxin","middleName":"","lastName":"Li","suffix":""},{"id":290266908,"identity":"1cbbb9c7-792d-4da2-8d8e-89f676264101","order_by":6,"name":"Junjie Zhu","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Junjie","middleName":"","lastName":"Zhu","suffix":""},{"id":290266909,"identity":"cd8e8997-c68d-4cc5-aca7-d825a0a99cdb","order_by":7,"name":"JIngyu Yu","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"JIngyu","middleName":"","lastName":"Yu","suffix":""},{"id":290266910,"identity":"fab8e40b-545e-4f19-b72d-40251517dae9","order_by":8,"name":"Ting Wei","email":"","orcid":"","institution":"Shanghai Jiao Tong University","correspondingAuthor":false,"prefix":"","firstName":"Ting","middleName":"","lastName":"Wei","suffix":""}],"badges":[],"createdAt":"2024-04-11 02:35:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4249839/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4249839/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":56585065,"identity":"2a53f8c2-129e-45f1-947f-5f3ad3bb79a6","added_by":"auto","created_at":"2024-05-16 07:19:45","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":181272,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eThe overall workflow of Scaffold-Lab.\u003c/strong\u003e Seven state-of-the-art protein backbone generation methods were evaluated by two types of tasks throughout a refolding pipeline. Four qualitative metrics were introduced to qualitatively assess the performances of methods, namely designability, novelty, diversity and efficiency.\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1/b38bea1217ba01f3f17fec04.png"},{"id":56585067,"identity":"8e9340ea-50e4-4330-aa3e-c37b31190e02","added_by":"auto","created_at":"2024-05-16 07:19:45","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":314042,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eEvaluation results of unconditional generation.\u003c/strong\u003e \u003cstrong\u003e(A)\u003c/strong\u003eDistribution of top 3 sc-TM. The number attached to each group denotes the proportion of backbones with top 3 sc-TM ≥ 0.5. \u003cstrong\u003e(B)\u003c/strong\u003eSuccess rates of tested methods. Success rate is defined by the number of backbones passing the sc-TM threshold divided by the total number of backbones within each group, with top 3 sc-TM by solid line and best sc-TM by dashed line. The text of numbers refers to the solid lines. \u003cstrong\u003e(C)\u003c/strong\u003e Distribution of pdb-TM values. The number attached to each group denotes the median value. \u003cstrong\u003e(D)\u003c/strong\u003eDiversity performances. Diversity is defined as the number of unique clusters divided by the total number of proteins within each group using Foldseek-Cluster. \u003cstrong\u003e(E)\u003c/strong\u003e Differences between novelty calculation by whether adding the penalty term of designability. Stars on each data group indicate the differences between two types of data within the group, calculated using the T-test. *, \u003cem\u003ep\u003c/em\u003e-value \u0026lt; 0.05; **, \u003cem\u003ep\u003c/em\u003e-value \u0026lt; 0.01; ***, \u003cem\u003ep\u003c/em\u003e-value \u0026lt; 0.001. \u003cstrong\u003e(F) \u003c/strong\u003eTime usage for generating protein backbones among different methods.\u003cstrong\u003e \u003c/strong\u003eThe efficiency is calculated as the average time usage per backbone for generating 100 backbones. All tests were conducted under identical hardware resources.\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1/333589b2c6ffe0d915b09b89.png"},{"id":56585069,"identity":"2b838ef8-4f8f-44a6-9025-abab268f4cfb","added_by":"auto","created_at":"2024-05-16 07:19:45","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":201086,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eStructural analysis on generated proteins. (A)\u003c/strong\u003e Average secondary structure distribution within each method. \u003cstrong\u003e(B)\u003c/strong\u003e Distribution of secondary structure composition of each protein. All data was curated by directly calculating onto protein backbones without refolding. \u003cem\u003eGenie\u003c/em\u003e was excluded from this test as it solely generates -only backbones.\u003cstrong\u003e (C) \u003c/strong\u003eRadius of gyration (Rg) values of different methods. The points are the mean value of Rg and the error bars denote the standard deviation interval within each group. The empirical regression curve of Rg for structured proteins is shown in a dashed style with the text denoted the formula based on previous studies\u003csup\u003e90,91\u003c/sup\u003e.\u003c/p\u003e","description":"","filename":"Figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1/190a6991d5ff9a90f375f1d2.png"},{"id":56585068,"identity":"590d886d-6248-49fe-92f1-8a8bd6c12947","added_by":"auto","created_at":"2024-05-16 07:19:45","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":263600,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eMotif-scaffolding results and failure mode analysis for tested methods. (A)\u003c/strong\u003e Overall distribution of success rate and diversity across the 24-cases benchmark. The success definition follows the “Top N” strategy in unconditional generation. \u003cstrong\u003e(B)\u003c/strong\u003e, \u003cstrong\u003e(C)\u003c/strong\u003e, \u003cstrong\u003e(D)\u003c/strong\u003e The difficulty of solving motif-scaffolding problems varies across different cases. \u003cstrong\u003e(B) \u003c/strong\u003eAn illustration of \u003cem\u003eSimple\u003c/em\u003e cases represented by \u003cem\u003e2KL8\u003c/em\u003e. \u003cstrong\u003e(C)\u003c/strong\u003e An illustration of \u003cem\u003eModerate\u003c/em\u003e cases represented by \u003cem\u003e4JHW\u003c/em\u003e. \u003cstrong\u003e(D)\u003c/strong\u003e An illustration of \u003cem\u003eDifficult\u003c/em\u003e cases represented by \u003cem\u003e5WN9\u003c/em\u003e. In \u003cstrong\u003e(B)\u003c/strong\u003e, \u003cstrong\u003e(C)\u003c/strong\u003e and \u003cstrong\u003e(D)\u003c/strong\u003e, refolded structures of designs by different methods (shown in the legend on top of Figure 4F) are overlayed with crystal structures from PDB (shown in gray). Motifs are displayed as \u003cem\u003eopaque\u003c/em\u003e and scaffolds are displayed as \u003cem\u003etransparent\u003c/em\u003e. \u003cstrong\u003e(E)\u003c/strong\u003e Percentage of different outcomes of motif-scaffolding task among the 24-cases dataset. Each color denotes a unique type of outcome. \u003cstrong\u003e(F)\u003c/strong\u003e Schematic diagram of different outcomes. Each case in the “case” panel denotes a unique type of outcome and different colors denote different methods (shown in the legend on top of Figure 4F). Inside each scheme, we use \u003cem\u003edark\u003c/em\u003e colors to represent \u003cem\u003erefolded structures\u003c/em\u003e after folding and \u003cem\u003elight\u003c/em\u003e colors to denote \u003cem\u003eoriginal backbones\u003c/em\u003e designed by different methods. Experimental structures are shown in gray, motifs are displayed as \u003cem\u003eopaque\u003c/em\u003e and scaffolds are displayed as \u003cem\u003etransparent\u003c/em\u003e. The motif-RMSD is calculated between refolded structures and experimental structures from PDB.\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1/129553e263395a91b6880e07.png"},{"id":64181967,"identity":"551817d6-f25c-4851-a666-a9fa9c9f3384","added_by":"auto","created_at":"2024-09-09 15:06:37","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":860897,"visible":true,"origin":"","legend":"","description":"","filename":"ScaffoldLab0403Finaltxt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1_covered_c897c3b2-ecec-4b04-aaee-c93c2b630856.pdf"},{"id":56585066,"identity":"e2bf46c0-c701-4e08-9a7e-d43514e6122d","added_by":"auto","created_at":"2024-05-16 07:19:45","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":8745249,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"ScaffoldLab0403SIFinal.docx","url":"https://assets-eu.researchsquare.com/files/rs-4249839/v1/0b37d743f19e3ce312f17866.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"de novo protein design, protein backbone generation, evaluation, benchmark","lastPublishedDoi":"10.21203/rs.3.rs-4249839/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4249839/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e \u003cem\u003eDe novo\u003c/em\u003e protein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal that \u003cem\u003eFrameFlow\u003c/em\u003e and \u003cem\u003eRFdiffusion\u003c/em\u003e in unconditional generation and \u003cem\u003eGPDL-H\u003c/em\u003e in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts are available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/Immortals-33/Scaffold-Lab\u003c/span\u003e\u003cspan address=\"https://github.com/Immortals-33/Scaffold-Lab\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e","manuscriptTitle":"A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-16 07:19:40","doi":"10.21203/rs.3.rs-4249839/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"beb617f9-1d8d-48d2-bf6d-e5780b64c969","owner":[],"postedDate":"May 16th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":30575148,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":30575149,"name":"Biological sciences/Biophysics/Computational biophysics"}],"tags":[],"updatedAt":"2024-09-09T14:58:27+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-16 07:19:40","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4249839","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4249839","identity":"rs-4249839","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}