LabOS: The AI-XR Co-Scientist That Sees and Works With Humans

LabOS: The AI-XR Co-Scientist That Sees and Works With Humans | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Biological Sciences - Article LabOS: The AI-XR Co-Scientist That Sees and Works With Humans Le Cong, David Smerkous, Xiaotong Wang, Di Yin, Zaixi Zhang, Ruofan Jin, and 27 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8154250/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Modern science advances fastest when thought meets action. LabOS represents the first AI co-scientist that unites computational reasoning with physical experimentation through multimodal perception, self-evolving agents, and XR-enabled, embodied human-AI collaboration. To empower agentic AI with embodied intelligence, we built LabSuperVision, the first vision–language-model (VLM) benchmark spanning biomedical and materials science laboratories, designed to evaluate scientific perception and reasoning. This endowed LabOS with visual reasoning capabilities that surpass those of general-purpose models on laboratory-specific tasks. By connecting multi-model AI agents, smart glasses, and human-AI-robot collaboration, LabOS allows AI to see what scientists see, understand experimental context, and participate in real-time execution through immersive XR interaction and robotic autonomation. Across applications—from cancer immunotherapy target discovery to stem cell engineering—LabOS shows that AI can move beyond computational design to participation, turning the laboratory into an intelligent, collaborative environment where human and machine discovery evolve together. Biological sciences/Biological techniques Physical sciences/Engineering Biological sciences/Computational biology and bioinformatics Full Text Additional Declarations Yes there is potential Competing Interest. Stanford University and Princeton University have filed a patent application related to this work. Supplementary Files LabOSSupplementaryMethodsandNotes.pdf Supplementary Methods and Notes SuppVideo14DPU480.mov SuppVideo1_4D_reconstruction SuppVideo2HumanAIcobotdemo.mp4 SuppVideo2_Human-AI-cobot_demo SuppVideo3WorldModelRoboticsModule.mov SuppVideo3_WorldModel_Robotics_Module Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8154250","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Biological Sciences - Article","associatedPublications":[],"authors":[{"id":550608611,"identity":"3274e78e-c226-4a8b-aaa6-373e63ba3eb7","order_by":0,"name":"Le Cong","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA20lEQVRIiWNgGAWjYLCCBww2YPoDAwMzkEogQksCQxoDDwMD4wxStBwmQQt/e+/hFwkV5+3t2c8ebPi4x5qBnz3HAK8WiTPn0iwSztxO7OHJS2yc8SydQbLnDX4tBhI5ZgaJbbcTeBhyzB/zHDjMYHCDgC0QLf/O2fPwvzFsBmmxJ0KL8YPEhgOMPRI5EC1AEUJ+OWPGkHAsObHnxhvDxhkH0nkkzjwrwKuFv73H+MOHGjt79v4cw4YPB6zl+NuTN+DVAgRsEsg8HkLKQYD5AzGqRsEoGAWjYAQDAG+IR/zZc0yCAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-4725-8714","institution":"Stanford University","correspondingAuthor":true,"prefix":"","firstName":"Le","middleName":"","lastName":"Cong","suffix":""},{"id":550608612,"identity":"df5dc3af-e086-4121-a3de-20fd181d589b","order_by":1,"name":"David Smerkous","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Smerkous","suffix":""},{"id":550608613,"identity":"7b5c2d7e-9b33-4866-9188-366b11aa9b5e","order_by":2,"name":"Xiaotong Wang","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Xiaotong","middleName":"","lastName":"Wang","suffix":""},{"id":550608614,"identity":"94022b00-bec4-4d5a-983a-7f7cf0be0ac7","order_by":3,"name":"Di Yin","email":"","orcid":"https://orcid.org/0000-0002-3272-9643","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Di","middleName":"","lastName":"Yin","suffix":""},{"id":550608615,"identity":"27745f9b-d956-43e0-9cd2-dc0fe395b730","order_by":4,"name":"Zaixi Zhang","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Zaixi","middleName":"","lastName":"Zhang","suffix":""},{"id":550608616,"identity":"71f44c75-653b-472e-b36b-1bc3d9ef658d","order_by":5,"name":"Ruofan Jin","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Ruofan","middleName":"","lastName":"Jin","suffix":""},{"id":550608617,"identity":"509812fb-f642-4a52-97af-89ff8dc1c1db","order_by":6,"name":"Yinkai Wang","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Yinkai","middleName":"","lastName":"Wang","suffix":""},{"id":550608618,"identity":"73062aae-62db-49b1-867e-a1b9d07e37f7","order_by":7,"name":"Michal Gerasimiuk","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Michal","middleName":"","lastName":"Gerasimiuk","suffix":""},{"id":550608619,"identity":"415dd4be-12d5-41c6-9c8f-515d1912de71","order_by":8,"name":"Ravi Dinesh","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Ravi","middleName":"","lastName":"Dinesh","suffix":""},{"id":550608620,"identity":"25f6b65c-b380-49c8-86ff-4ce7c839e471","order_by":9,"name":"Alex Smerkous","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Alex","middleName":"","lastName":"Smerkous","suffix":""},{"id":550608621,"identity":"8d1c49f2-5b1e-4ddd-a6b2-c47c554f96a9","order_by":10,"name":"Lihan Shi","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Lihan","middleName":"","lastName":"Shi","suffix":""},{"id":550608622,"identity":"aff299d5-992a-4719-ae45-bf9be94b7c84","order_by":11,"name":"Zhaoyi Zheng","email":"","orcid":"https://orcid.org/0000-0001-7016-4447","institution":"Princeton University","correspondingAuthor":false,"prefix":"","firstName":"Zhaoyi","middleName":"","lastName":"Zheng","suffix":""},{"id":550608623,"identity":"f57f5e8f-58b2-488c-b2be-b150b3964eee","order_by":12,"name":"Ian Lam","email":"","orcid":"https://orcid.org/0009-0009-3694-7662","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Ian","middleName":"","lastName":"Lam","suffix":""},{"id":550608624,"identity":"7d4119fd-d1ff-4c7a-8f27-5736855cb036","order_by":13,"name":"Xuekun Wu","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Xuekun","middleName":"","lastName":"Wu","suffix":""},{"id":550608625,"identity":"1b555122-f5c9-47c3-b6d5-3f34130cc26a","order_by":14,"name":"Shilong Liu","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Shilong","middleName":"","lastName":"Liu","suffix":""},{"id":550608626,"identity":"c5a2f80a-f457-4710-8161-299cacac4e5c","order_by":15,"name":"Peishan Li","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Peishan","middleName":"","lastName":"Li","suffix":""},{"id":550608627,"identity":"dde111ca-b8db-4ea5-9ee6-7d34743421b2","order_by":16,"name":"Yi Zhu","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Yi","middleName":"","lastName":"Zhu","suffix":""},{"id":550608628,"identity":"8c863577-7a29-4ca8-8efd-dc8e7cc3caf3","order_by":17,"name":"Ning Zhao","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Ning","middleName":"","lastName":"Zhao","suffix":""},{"id":550608629,"identity":"fe27c0e3-2995-4a97-9c77-5a688a78c077","order_by":18,"name":"Meenal Parakh","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Meenal","middleName":"","lastName":"Parakh","suffix":""},{"id":550608630,"identity":"ea18ef67-5886-4ed9-b59f-d100b134c51b","order_by":19,"name":"Simran Serrao","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Simran","middleName":"","lastName":"Serrao","suffix":""},{"id":550608631,"identity":"71f9d8a7-493e-477b-a20d-e4a60fda7494","order_by":20,"name":"Imran Mohammad","email":"","orcid":"https://orcid.org/0000-0003-2288-2887","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Imran","middleName":"","lastName":"Mohammad","suffix":""},{"id":550608632,"identity":"a516125a-dbad-4c15-9cef-79d292409504","order_by":21,"name":"Chao-Yeh Chen","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Chao-Yeh","middleName":"","lastName":"Chen","suffix":""},{"id":550608633,"identity":"3429f2ba-84bb-4570-a091-612228ad9c73","order_by":22,"name":"Xiufeng Xie","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Xiufeng","middleName":"","lastName":"Xie","suffix":""},{"id":550608634,"identity":"30638718-9786-4527-9c30-1b4e7875341a","order_by":23,"name":"Tiffany Chen","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Tiffany","middleName":"","lastName":"Chen","suffix":""},{"id":550608635,"identity":"1a17cf98-d5df-4882-9c77-d790948d44ab","order_by":24,"name":"David Weinstein","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Weinstein","suffix":""},{"id":550608636,"identity":"7e2f9252-e581-47c6-b666-579b07dfe832","order_by":25,"name":"Greg Barbone","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Greg","middleName":"","lastName":"Barbone","suffix":""},{"id":550608637,"identity":"30056804-de75-4e96-99eb-1748daaa21ff","order_by":26,"name":"Belgin Caglar","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Belgin","middleName":"","lastName":"Caglar","suffix":""},{"id":550608638,"identity":"d3326ae1-ed29-4a7a-aa19-83eb565f921e","order_by":27,"name":"John Sunwoo","email":"","orcid":"https://orcid.org/0000-0002-8393-4196","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"John","middleName":"","lastName":"Sunwoo","suffix":""},{"id":550608639,"identity":"77eaa8da-63b6-47e7-9a7a-95b233634bc3","order_by":28,"name":"Fuxin Li","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Fuxin","middleName":"","lastName":"Li","suffix":""},{"id":550608640,"identity":"739679f6-1531-457e-b5dc-1c8d736adbfb","order_by":29,"name":"Jia Deng","email":"","orcid":"","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Jia","middleName":"","lastName":"Deng","suffix":""},{"id":550608641,"identity":"6bbc70ba-0e89-43b2-8ec3-90e07c19bf89","order_by":30,"name":"Joseph Wu","email":"","orcid":"https://orcid.org/0000-0002-6068-8041","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"Joseph","middleName":"","lastName":"Wu","suffix":""},{"id":550608642,"identity":"99443d56-01b8-44f5-b6fc-a5a624ba9ac1","order_by":31,"name":"Sanfeng Wu","email":"","orcid":"https://orcid.org/0000-0002-6227-6286","institution":"Princeton University","correspondingAuthor":false,"prefix":"","firstName":"Sanfeng","middleName":"","lastName":"Wu","suffix":""},{"id":550608643,"identity":"7be5745c-587b-46a2-8466-d6bbd35b51fe","order_by":32,"name":"Mengdi Wang","email":"","orcid":"https://orcid.org/0000-0002-2101-9507","institution":"Princeton University","correspondingAuthor":false,"prefix":"","firstName":"Mengdi","middleName":"","lastName":"Wang","suffix":""}],"badges":[],"createdAt":"2025-11-19 10:50:50","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8154250/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8154250/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":96917210,"identity":"82061487-fe2b-464a-8a40-46d5a7550442","added_by":"auto","created_at":"2025-11-27 14:09:22","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2262168,"visible":true,"origin":"","legend":"Article File","description":"","filename":"LabOSSubmissionv11192025.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8154250/v1_covered_5d2b8ea2-5bf9-4659-9ee9-2c673e7a2bd4.pdf"},{"id":96792486,"identity":"52b40c1c-1f45-494c-817c-2115e7ed48d2","added_by":"auto","created_at":"2025-11-26 07:04:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":81091,"visible":true,"origin":"","legend":"Supplementary Methods and Notes","description":"","filename":"LabOSSupplementaryMethodsandNotes.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8154250/v1/0e801802d914cd28157e9492.pdf"},{"id":96792487,"identity":"5efb9607-ac20-45e7-8a33-78657062a114","added_by":"auto","created_at":"2025-11-26 07:04:06","extension":"mov","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":20211728,"visible":true,"origin":"","legend":"SuppVideo1_4D_reconstruction","description":"","filename":"SuppVideo14DPU480.mov","url":"https://assets-eu.researchsquare.com/files/rs-8154250/v1/a622ad5c89ac3129b4d235be.mov"},{"id":96792489,"identity":"ea3815f9-f794-448a-abdc-1d12db697589","added_by":"auto","created_at":"2025-11-26 07:04:06","extension":"mp4","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":14519428,"visible":true,"origin":"","legend":"SuppVideo2_Human-AI-cobot_demo","description":"","filename":"SuppVideo2HumanAIcobotdemo.mp4","url":"https://assets-eu.researchsquare.com/files/rs-8154250/v1/4e994d2f5866872c9a9d753f.mp4"},{"id":96792488,"identity":"742496c2-8dac-4468-a1c6-d40203cc7de9","added_by":"auto","created_at":"2025-11-26 07:04:06","extension":"mov","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":18359004,"visible":true,"origin":"","legend":"SuppVideo3_WorldModel_Robotics_Module","description":"","filename":"SuppVideo3WorldModelRoboticsModule.mov","url":"https://assets-eu.researchsquare.com/files/rs-8154250/v1/d18f2d8e9750a179220c0a49.mov"}],"financialInterests":"\u003cb\u003eYes\u003c/b\u003e there is potential Competing Interest.\nStanford University and Princeton University have filed a patent application related to this work.","formattedTitle":"LabOS: The AI-XR Co-Scientist That Sees and Works With Humans","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8154250/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8154250/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Modern science advances fastest when thought meets action. LabOS represents the first AI co-scientist that unites computational reasoning with physical experimentation through multimodal perception, self-evolving agents, and XR-enabled, embodied human-AI collaboration. To empower agentic AI with embodied intelligence, we built LabSuperVision, the first vision–language-model (VLM) benchmark spanning biomedical and materials science laboratories, designed to evaluate scientific perception and reasoning. This endowed LabOS with visual reasoning capabilities that surpass those of general-purpose models on laboratory-specific tasks. By connecting multi-model AI agents, smart glasses, and human-AI-robot collaboration, LabOS allows AI to see what scientists see, understand experimental context, and participate in real-time execution through immersive XR interaction and robotic autonomation. Across applications—from cancer immunotherapy target discovery to stem cell engineering—LabOS shows that AI can move beyond computational design to participation, turning the laboratory into an intelligent, collaborative environment where human and machine discovery evolve together.","manuscriptTitle":"LabOS: The AI-XR Co-Scientist That Sees and Works With Humans","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-26 07:04:01","doi":"10.21203/rs.3.rs-8154250/v1","editorialEvents":[],"status":"published","journal":{"display":false,"email":"[email protected]","identity":"nature","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"nature","sideBox":"Learn more about [Nature](http://www.nature.com/nature/)","snPcode":"","submissionUrl":"","title":"Nature","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"b63d90e6-2975-4745-9166-e817c965b6b3","owner":[],"postedDate":"November 26th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":58585587,"name":"Biological sciences/Biological techniques"},{"id":58585588,"name":"Physical sciences/Engineering"},{"id":58585589,"name":"Biological sciences/Computational biology and bioinformatics"}],"tags":[],"updatedAt":"2026-01-16T15:52:34+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-26 07:04:01","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8154250","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8154250","identity":"rs-8154250","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}