Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion

Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion Chunhai Hu, Xiaonan Liu, Bin Liu This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3994065/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The 2D video analysis technology has propelled research in rodent behavioral studies towards automated and high-fidelity analysis of behavioral data. Precise posture detection serves as a crucial prerequisite for behavior quantification. However, 2D video analysis falls short in determining the three-dimensional pose of mice. Existing methods for three-dimensional pose reconstruction often rely on multi-view cameras, which are costly and lack real-time capabilities. Therefore, this paper introduces a method that integrates depth information for three-dimensional pose reconstruction of experimental mice. The approach involves constructing a skeletal model of experimental mouse poses based on multiple keypoints. Utilizing a single low-cost depth camera, the Intel RealSense D455, data is collected from a top-down perspective as RGB-D data. To address the keypoint detection challenge, this study designs and improves the YOLOv7 network model. Omni-dimensional dynamic convolutions and coordinate attention mechanisms are respectively introduced into the feature extraction and fusion layers to enhance detection accuracy. Ghost convolutions are incorporated into the feature extraction layer to achieve a lightweight network design. Through ablation experiments, this network model achieves high precision and real-time keypoint detection performance on RGB images. The network model attains an average PCK (Probability of Correct Keypoint) of 97.71$%$ for ten mouse keypoints, with a real-time frame rate of 70.47 frames. Subsequently, the depth information provided by the depth camera is optimized. Two-dimensional keypoint coordinates are employed as indices to retrieve corresponding depth values, completing the three-dimensional pose reconstruction. The reconstructed three-dimensional poses are then utilized for the analysis of mouse behavior in the vertical dimension during open field experiments. This research method offers a more comprehensive and accurate representation of experimental mouse motion, posture, and behavior, thereby contributing to a more detailed and profound understanding of behavioral studies. Experimental mice Depth camera Three-dimensional pose reconstruction YOLOv7 Vertical behavior Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3994065","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":275366662,"identity":"887193d1-0d7d-48d6-8bec-dde27c764919","order_by":0,"name":"Chunhai Hu","email":"","orcid":"","institution":"Yanshan University","correspondingAuthor":false,"prefix":"","firstName":"Chunhai","middleName":"","lastName":"Hu","suffix":""},{"id":275366663,"identity":"a4f5de2a-66d0-461f-8b79-f7d31cd3990b","order_by":1,"name":"Xiaonan Liu","email":"","orcid":"","institution":"Yanshan University","correspondingAuthor":false,"prefix":"","firstName":"Xiaonan","middleName":"","lastName":"Liu","suffix":""},{"id":275366664,"identity":"ed6f5483-4c73-44ae-b0e5-41bad38939e8","order_by":2,"name":"Bin Liu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAr0lEQVRIiWNgGAWjYDCCA8xtDB8gTANitTC2Mc4gWQszD0la+I4ntj22batLbGBv3ibBUHOHsBbJMw/bjXPbDic28Bwrk2A49oywFoMbiW3SuW0HEhskcswkGBsOE6nFEuQw+TekaAGGANAWHiK1gPxi2HPusHEbT1qxRcIxIrTwHU8+9uBHWZ1sP/vhjTc+1BChhYEhgYGBkY2BgQ3KJlILwx/ilI6CUTAKRsEIBQCW2Tyz7vFPjAAAAABJRU5ErkJggg==","orcid":"","institution":"Yanshan University","correspondingAuthor":true,"prefix":"","firstName":"Bin","middleName":"","lastName":"Liu","suffix":""}],"badges":[],"createdAt":"2024-02-27 13:52:43","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3994065/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3994065/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":53496770,"identity":"d71d31ca-09c7-419b-b698-7654405abc46","added_by":"auto","created_at":"2024-03-26 17:05:10","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2528894,"visible":true,"origin":"","legend":"","description":"","filename":"MachineVisionandApplications.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3994065/v1_covered_810d35c7-8776-44d6-9ecc-ad4ed70779e2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Experimental mice, Depth camera, Three-dimensional pose reconstruction, YOLOv7, Vertical behavior","lastPublishedDoi":"10.21203/rs.3.rs-3994065/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3994065/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The 2D video analysis technology has propelled research in rodent behavioral studies towards automated and high-fidelity analysis of behavioral data. Precise posture detection serves as a crucial prerequisite for behavior quantification. However, 2D video analysis falls short in determining the three-dimensional pose of mice. Existing methods for three-dimensional pose reconstruction often rely on multi-view cameras, which are costly and lack real-time capabilities. Therefore, this paper introduces a method that integrates depth information for three-dimensional pose reconstruction of experimental mice. The approach involves constructing a skeletal model of experimental mouse poses based on multiple keypoints. Utilizing a single low-cost depth camera, the Intel RealSense D455, data is collected from a top-down perspective as RGB-D data. To address the keypoint detection challenge, this study designs and improves the YOLOv7 network model. Omni-dimensional dynamic convolutions and coordinate attention mechanisms are respectively introduced into the feature extraction and fusion layers to enhance detection accuracy. Ghost convolutions are incorporated into the feature extraction layer to achieve a lightweight network design. Through ablation experiments, this network model achieves high precision and real-time keypoint detection performance on RGB images. The network model attains an average PCK (Probability of Correct Keypoint) of 97.71$\\%$ for ten mouse keypoints, with a real-time frame rate of 70.47 frames. Subsequently, the depth information provided by the depth camera is optimized. Two-dimensional keypoint coordinates are employed as indices to retrieve corresponding depth values, completing the three-dimensional pose reconstruction. The reconstructed three-dimensional poses are then utilized for the analysis of mouse behavior in the vertical dimension during open field experiments. This research method offers a more comprehensive and accurate representation of experimental mouse motion, posture, and behavior, thereby contributing to a more detailed and profound understanding of behavioral studies. ","manuscriptTitle":"Research on Three-Dimensional Pose Reconstruction of Experimental Mice with Depth Information Fusion","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-05 18:18:15","doi":"10.21203/rs.3.rs-3994065/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2b92bea7-cd78-4bb7-9027-62dc5447f3a3","owner":[],"postedDate":"March 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-03-26T16:56:55+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-05 18:18:15","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3994065","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3994065","identity":"rs-3994065","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}