Interactive robot with multimodal multitask model for early screening of multiple common adolescent mental disorders

Interactive robot with multimodal multitask model for early screening of multiple common adolescent mental disorders | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Interactive robot with multimodal multitask model for early screening of multiple common adolescent mental disorders Peiwu Qin, Zhicheng Du, Likun Zhang, Shiyao Zhai, Zhengyang Lei, and 7 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5731226/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The early detection of mental disorders in adolescents represents a significant global public health challenge. Due to the complex and subtle nature of mental disorders, making it difficult to detect abnormalities using a single factor. Additionally, the generalized multimodal C omputer- A ided S creening ( CAS) systems, incorporating interactive robots for adolescent mental health assessment, remain unavailable. In this study, we present an Android application equipped with mini-games and chat recording, deployed in a portable robot, to screen 3,783 middle school students. This system generates a multimodal screening dataset comprising facial images, physiological signals, voice recordings, and textual transcripts. We develop a model called GAME ( G eneralized Model with A ttention and M ultimodal E mbraceNet) with novel attention mechanism that integrates cross-modal features into the model. GAME evaluates adolescent mental conditions with high accuracy (73.34% – 92.77%) and F1-Score (71.32% – 91.06%) and outperforms traditional methods. Our findings reveal that each modality contributes dynamically to mental disorder detection and the identification of comorbidities across various disorders, supporting the feasibility of an explainable model. This study provides a system capable of acquiring multimodal information and constructs a generalized multimodal integration algorithm with novel attention mechanisms for the early screening of adolescent mental disorders. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Biological sciences/Biological techniques/Behavioural methods Adolescent mental disorder Mental health screening Interactive multi-sensor robot Multimodal learning Human-Computer interaction Computer-aided screening Full Text Additional Declarations There is NO Competing Interest. Supplementary Files Supplement.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5731226","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":424899748,"identity":"cd53d5ed-0c46-4079-b055-7efe9b907ac5","order_by":0,"name":"Peiwu Qin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+klEQVRIiWNgGAWjYJCCAwwVMAYIAYEEYS1ngCQbKVoYGNsgWhiI0mIukbzxcOG8OnsG+faHhwt+3Unsb2A+eJuHwS4PlxbLGWkFh2duO8zMwMZjcHhm37PEGQfYkq15GJKLcWkxuJ1jcJh32wE2oBaGw7w9hxMbDvCYSfMwHEhswKtlTh0PAxv7A7CW+Qf4vxGhpYFZAuh9g8M8Pw4nbjjAw4ZXi+X8ZwWHeY4dNmBjA+s9bLzxMJux5RyDZJxazHkOb/7MU1Nnz898/PFnnj+HZecdb354402FHW6HgREDNFrAEcQMFccFDFBl/+BWOQpGwSgYBSMXAADIn1m98xx0BgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-7829-8973","institution":"Tsinghua University","correspondingAuthor":true,"prefix":"","firstName":"Peiwu","middleName":"","lastName":"Qin","suffix":""},{"id":424899749,"identity":"fcfe4269-acc5-4e12-9d4d-0ba3d476984a","order_by":1,"name":"Zhicheng Du","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Zhicheng","middleName":"","lastName":"Du","suffix":""},{"id":424899750,"identity":"473a7a3b-491b-4caf-b2c8-d672fbad9b69","order_by":2,"name":"Likun Zhang","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Likun","middleName":"","lastName":"Zhang","suffix":""},{"id":424899751,"identity":"78abea7b-4001-4ba2-bb81-bed160d3b7c4","order_by":3,"name":"Shiyao Zhai","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Shiyao","middleName":"","lastName":"Zhai","suffix":""},{"id":424899752,"identity":"e9a93ff9-b7f1-4450-becf-40b8e681a0dd","order_by":4,"name":"Zhengyang Lei","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Zhengyang","middleName":"","lastName":"Lei","suffix":""},{"id":424899753,"identity":"daf411e0-b559-4abc-8bdc-cd2700ddcbcc","order_by":5,"name":"Yongjie Zhou","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Yongjie","middleName":"","lastName":"Zhou","suffix":""},{"id":424899754,"identity":"0184adee-e174-4c08-8695-d1b33bf2270c","order_by":6,"name":"Yu Dongmei","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Dongmei","suffix":""},{"id":424899755,"identity":"63acc0f4-7fcd-4482-a34a-b7b59f35a462","order_by":7,"name":"Chenggang Yan","email":"","orcid":"","institution":"Hangzhou Dianzi University","correspondingAuthor":false,"prefix":"","firstName":"Chenggang","middleName":"","lastName":"Yan","suffix":""},{"id":424899756,"identity":"283a8cc5-9ea4-4d71-9bc4-24afee6aea3a","order_by":8,"name":"Xi Yuan","email":"","orcid":"","institution":"Hangzhou Dianzi University","correspondingAuthor":false,"prefix":"","firstName":"Xi","middleName":"","lastName":"Yuan","suffix":""},{"id":424899757,"identity":"94bcbb05-2fec-42be-afbe-11af1581e891","order_by":9,"name":"Jiansong Ji","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Jiansong","middleName":"","lastName":"Ji","suffix":""},{"id":424899758,"identity":"e173a601-e272-47e0-9352-7025d7afd652","order_by":10,"name":"Yang Liu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Liu","suffix":""},{"id":424899759,"identity":"9ac9726b-58a6-4fbd-ad4c-f4ecf8b49d63","order_by":11,"name":"Zhenglin Chen","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Zhenglin","middleName":"","lastName":"Chen","suffix":""}],"badges":[],"createdAt":"2024-12-29 18:20:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5731226/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5731226/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":79184710,"identity":"63834ca8-3ebc-458f-9068-0ce89e7bdf81","added_by":"auto","created_at":"2025-03-25 11:19:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1193954,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5731226/v1_covered_d7eecdbc-4357-4df3-9993-2e92487da91d.pdf"},{"id":79184389,"identity":"719ea9db-22b2-4ed9-97e4-f54847638ab7","added_by":"auto","created_at":"2025-03-25 11:11:33","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":31012264,"visible":true,"origin":"","legend":"","description":"","filename":"Supplement.docx","url":"https://assets-eu.researchsquare.com/files/rs-5731226/v1/380303808b69e70ec40a3a1e.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Interactive robot with multimodal multitask model for early screening of multiple common adolescent mental disorders","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Adolescent mental disorder, Mental health screening, Interactive multi-sensor robot, Multimodal learning, Human-Computer interaction, Computer-aided screening","lastPublishedDoi":"10.21203/rs.3.rs-5731226/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5731226/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe early detection of mental disorders in adolescents represents a significant global public health challenge. Due to the complex and subtle nature of mental disorders, making it difficult to detect abnormalities using a single factor. Additionally, the generalized multimodal \u003cb\u003eC\u003c/b\u003eomputer-\u003cb\u003eA\u003c/b\u003eided \u003cb\u003eS\u003c/b\u003ecreening (\u003cb\u003eCAS)\u003c/b\u003e systems, incorporating interactive robots for adolescent mental health assessment, remain unavailable. In this study, we present an Android application equipped with mini-games and chat recording, deployed in a portable robot, to screen 3,783 middle school students. This system generates a multimodal screening dataset comprising facial images, physiological signals, voice recordings, and textual transcripts. We develop a model called \u003cb\u003eGAME\u003c/b\u003e (\u003cb\u003eG\u003c/b\u003eeneralized Model with \u003cb\u003eA\u003c/b\u003ettention and \u003cb\u003eM\u003c/b\u003eultimodal \u003cb\u003eE\u003c/b\u003embraceNet) with novel attention mechanism that integrates cross-modal features into the model. GAME evaluates adolescent mental conditions with high accuracy (73.34% \u0026ndash; 92.77%) and F1-Score (71.32% \u0026ndash; 91.06%) and outperforms traditional methods. Our findings reveal that each modality contributes dynamically to mental disorder detection and the identification of comorbidities across various disorders, supporting the feasibility of an explainable model. This study provides a system capable of acquiring multimodal information and constructs a generalized multimodal integration algorithm with novel attention mechanisms for the early screening of adolescent mental disorders.\u003c/p\u003e","manuscriptTitle":"Interactive robot with multimodal multitask model for early screening of multiple common adolescent mental disorders","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-25 11:11:27","doi":"10.21203/rs.3.rs-5731226/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ac02013a-0e06-45ec-a97f-a7f5f05673ed","owner":[],"postedDate":"March 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":45284871,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":45284872,"name":"Health sciences/Health care"},{"id":45284873,"name":"Biological sciences/Biological techniques/Behavioural methods"}],"tags":[],"updatedAt":"2025-03-25T11:11:28+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-25 11:11:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5731226","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5731226","identity":"rs-5731226","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}