Testing the reproducibility and effectiveness of deep learning models among clinics: sperm detection as a pilot study

Testing the reproducibility and effectiveness of deep learning models among clinics: sperm detection as a pilot study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Method Article Testing the reproducibility and effectiveness of deep learning models among clinics: sperm detection as a pilot study Jiaqi Wang, Yufei Jin, Aojun Jiang, Wenyuan Chen, Guanqiao Shan, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4008354/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Background: Deep learning has been increasingly investigated for assisting clinical in vitro fertilization (IVF). The first technical step in many tasks is to visually detect and locate sperm, oocytes, and embryos in images. For clinical deployment of such deep learning models, different clinics use different image acquisition hardware and different sample preprocessing protocols, raising the concern over whether the reported accuracy of a deep learning model by one clinic could be reproduced in another clinic. Here we aim to investigate the effect of each imaging factor on the reproducibility of object detection models, using sperm analysis as a pilot example. Methods: Ablation studies were performed using state-of-the-art models for detecting human sperm to quantitatively assess how model precision (false-positive detection) and recall (missed detection) were affected by imaging magnification, imaging mode, and sample preprocessing protocols. The results led to the hypothesis that the richness of image acquisition conditions in a training dataset deterministically affects model reproducibility. The hypothesis was tested by first enriching the training dataset with a wide range of imaging conditions, then validated through internal blind tests on new samples and external multi-center clinical validations. Results: Ablation experiments revealed that removing subsets of data from the training dataset significantly reduced model precision. Removing raw sample images from the training dataset caused the largest drop in model precision, whereas removing 20x images caused the largest drop in model recall. by incorporating different imaging and sample preprocessing conditions into a rich training dataset, the model achieved an intraclass correlation coefficient (ICC) of 0.97 (95% CI: 0.94-0.99) for precision, and an ICC of 0.97 (95% CI: 0.93-0.99) for recall. Multi-center clinical validation showed no significant differences in model precision or recall across different clinics and applications. Conclusions: The results validated the hypothesis that the richness of data in the training dataset is a key factor impacting model reproducibility. These findings highlight the importance of diversity in a training dataset for model evaluation and suggest that future deep learning models in andrology and reproductive medicine should incorporate comprehensive feature sets for enhanced reproducibility across clinics. Semen analysis Sperm detection Reproducibility Multicenter validation Deep learning Full Text Additional Declarations No competing interests reported. Supplementary Files Supplementaryfigure1.eps Supplementaryfigure2.eps Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 14 Apr, 2024 Reviews received at journal 05 Apr, 2024 Reviewers agreed at journal 26 Mar, 2024 Reviewers agreed at journal 25 Mar, 2024 Reviewers invited by journal 25 Mar, 2024 Editor assigned by journal 04 Mar, 2024 Submission checks completed at journal 04 Mar, 2024 First submitted to journal 03 Mar, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4008354","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Method Article","associatedPublications":[],"authors":[{"id":276222008,"identity":"5a16f27a-5343-41a6-9843-87f0574034a1","order_by":0,"name":"Jiaqi Wang","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Jiaqi","middleName":"","lastName":"Wang","suffix":""},{"id":276222009,"identity":"e59b246e-ae9a-4623-9b1b-ec3dfecc60f3","order_by":1,"name":"Yufei Jin","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Yufei","middleName":"","lastName":"Jin","suffix":""},{"id":276222010,"identity":"37a8d1df-3adf-47af-9316-d2f7ebc90cce","order_by":2,"name":"Aojun Jiang","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Aojun","middleName":"","lastName":"Jiang","suffix":""},{"id":276222011,"identity":"dabb5e70-342b-4163-861c-618c0fe66fb9","order_by":3,"name":"Wenyuan Chen","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Wenyuan","middleName":"","lastName":"Chen","suffix":""},{"id":276222012,"identity":"92fbed35-b54f-4a15-8ad3-4fd28ea95268","order_by":4,"name":"Guanqiao Shan","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Guanqiao","middleName":"","lastName":"Shan","suffix":""},{"id":276222013,"identity":"1d7eed87-6efa-495a-90bb-08b06c976d3d","order_by":5,"name":"Yifan Gu","email":"","orcid":"","institution":"Central South University","correspondingAuthor":false,"prefix":"","firstName":"Yifan","middleName":"","lastName":"Gu","suffix":""},{"id":276222014,"identity":"cc5fba98-e69e-407c-9f2b-aa0bdc79aa50","order_by":6,"name":"Yue Ming","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Yue","middleName":"","lastName":"Ming","suffix":""},{"id":276222015,"identity":"377e0455-5271-477d-99dc-94a7f6d06a51","order_by":7,"name":"Jichang Li","email":"","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Jichang","middleName":"","lastName":"Li","suffix":""},{"id":276222016,"identity":"7106785f-6656-4547-bb18-cd9c08ef23a6","order_by":8,"name":"Chunfeng Yue","email":"","orcid":"","institution":"Suzhou Boundless Medical Technology Ltd., Co.","correspondingAuthor":false,"prefix":"","firstName":"Chunfeng","middleName":"","lastName":"Yue","suffix":""},{"id":276222017,"identity":"4aec9178-f567-4210-98e7-c50ada227b4a","order_by":9,"name":"Zongjie Huang","email":"","orcid":"","institution":"Suzhou Boundless Medical Technology Ltd., Co.","correspondingAuthor":false,"prefix":"","firstName":"Zongjie","middleName":"","lastName":"Huang","suffix":""},{"id":276222018,"identity":"fbcea805-5285-4520-a549-688e8eac82b3","order_by":10,"name":"Clifford Librach","email":"","orcid":"","institution":"CReATe Fertility Centre","correspondingAuthor":false,"prefix":"","firstName":"Clifford","middleName":"","lastName":"Librach","suffix":""},{"id":276222019,"identity":"376e0c18-54e9-487f-8d5d-6e266139f2f2","order_by":11,"name":"Ge Lin","email":"","orcid":"","institution":"Central South University","correspondingAuthor":false,"prefix":"","firstName":"Ge","middleName":"","lastName":"Lin","suffix":""},{"id":276222020,"identity":"c84a1ffe-9f3a-4398-b8e9-cf9381d450af","order_by":12,"name":"Xibu Wang","email":"","orcid":"","institution":"The 3rd Affiliated Hospital of Shenzhen University, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Xibu","middleName":"","lastName":"Wang","suffix":""},{"id":276222021,"identity":"e8996238-7eff-4608-a237-9b6bde80b579","order_by":13,"name":"Huan Zhao","email":"","orcid":"","institution":"The 3rd Affiliated Hospital of Shenzhen University, Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Huan","middleName":"","lastName":"Zhao","suffix":""},{"id":276222022,"identity":"f3075fa8-1c94-4f37-91d1-cbc80893b6d1","order_by":14,"name":"Yu Sun","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Sun","suffix":""},{"id":276222023,"identity":"b546bdec-795d-4d0a-9d7d-1fd34de4c33a","order_by":15,"name":"Zhuoran Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/klEQVRIiWNgGAWjYDACCTB5AML5wMCQIAEXJEYL4wyStTDzEKNFfnbzs4df/txJ7Jc+/OyxTc3hPMkG5oO3eRjs8nBpYZxzzNxYhudZ4sy+NHPjnGOHi6UZ2JKteRiSi3FpYZZIMJOWkDicuOEMg5l0bsPhxHkMPGbSPAwHEhtwaGGTSP8mLWEA0sL+TdoSrIX/G14tPBI5ZpIfEkBagIYzArXMZuBhw6tFQiKnTJrhwGHjmT08ZZI9x9ITZzazGVvOMUjGqUV+Rvo2yR9/Dsv287Bvk/hRY50443jzwxtvKuxwagEHAQ8qF0QY4FEPBIw/8MuPglEwCkbBSAcAbt1UDLwpQAAAAAAASUVORK5CYII=","orcid":"","institution":"Chinese University of Hong Kong, Shenzhen","correspondingAuthor":true,"prefix":"","firstName":"Zhuoran","middleName":"","lastName":"Zhang","suffix":""}],"badges":[],"createdAt":"2024-03-03 11:04:42","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4008354/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4008354/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":52044418,"identity":"db6b3250-6062-4df6-a986-9c4039509272","added_by":"auto","created_at":"2024-03-05 19:13:57","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1028659,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4008354/v1_covered_f8aaad38-c990-43d7-b867-ff4fa7d55c0a.pdf"},{"id":52044255,"identity":"ca3e2afc-8c37-463f-9969-b18994d27710","added_by":"auto","created_at":"2024-03-05 19:05:52","extension":"eps","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":465971,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfigure1.eps","url":"https://assets-eu.researchsquare.com/files/rs-4008354/v1/35b37e556b879e0be8d10388.eps"},{"id":52044254,"identity":"c9250a06-f9d4-4d8f-a424-f1fb79dd6781","added_by":"auto","created_at":"2024-03-05 19:05:52","extension":"eps","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":185235,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfigure2.eps","url":"https://assets-eu.researchsquare.com/files/rs-4008354/v1/ffe4a039bb20f7267813c951.eps"}],"financialInterests":"No competing interests reported.","formattedTitle":"Testing the reproducibility and effectiveness of deep learning models among clinics: sperm detection as a pilot study","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"reproductive-biology-and-endocrinology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"rbej","sideBox":"Learn more about [Reproductive Biology and Endocrinology](http://rbej.biomedcentral.com)","snPcode":"12958","submissionUrl":"https://submission.nature.com/new-submission/12958/3","title":"Reproductive Biology and Endocrinology","twitterHandle":"@BioMedCentral","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Semen analysis, Sperm detection, Reproducibility, Multicenter validation, Deep learning","lastPublishedDoi":"10.21203/rs.3.rs-4008354/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4008354/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground:\u003c/h2\u003e\u003cp\u003eDeep learning has been increasingly investigated for assisting clinical in vitro fertilization (IVF). The first technical step in many tasks is to visually detect and locate sperm, oocytes, and embryos in images. For clinical deployment of such deep learning models, different clinics use different image acquisition hardware and different sample preprocessing protocols, raising the concern over whether the reported accuracy of a deep learning model by one clinic could be reproduced in another clinic. Here we aim to investigate the effect of each imaging factor on the reproducibility of object detection models, using sperm analysis as a pilot example.\u003c/p\u003e\u003ch2\u003eMethods:\u003c/h2\u003e\u003cp\u003eAblation studies were performed using state-of-the-art models for detecting human sperm to quantitatively assess how model precision (false-positive detection) and recall (missed detection) were affected by imaging magnification, imaging mode, and sample preprocessing protocols. The results led to the hypothesis that the richness of image acquisition conditions in a training dataset deterministically affects model reproducibility. The hypothesis was tested by first enriching the training dataset with a wide range of imaging conditions, then validated through internal blind tests on new samples and external multi-center clinical validations.\u003c/p\u003e\u003ch2\u003eResults:\u003c/h2\u003e\u003cp\u003eAblation experiments revealed that removing subsets of data from the training dataset significantly reduced model precision. Removing raw sample images from the training dataset caused the largest drop in model precision, whereas removing 20x images caused the largest drop in model recall. by incorporating different imaging and sample preprocessing conditions into a rich training dataset, the model achieved an intraclass correlation coefficient (ICC) of 0.97 (95% CI: 0.94-0.99) for precision, and an ICC of 0.97 (95% CI: 0.93-0.99) for recall. Multi-center clinical validation showed no significant differences in model precision or recall across different clinics and applications.\u003c/p\u003e\u003ch2\u003eConclusions:\u003c/h2\u003e\u003cp\u003eThe results validated the hypothesis that the richness of data in the training dataset is a key factor impacting model reproducibility. These findings highlight the importance of diversity in a training dataset for model evaluation and suggest that future deep learning models in andrology and reproductive medicine should incorporate comprehensive feature sets for enhanced reproducibility across clinics.\u003c/p\u003e","manuscriptTitle":"Testing the reproducibility and effectiveness of deep learning models among clinics: sperm detection as a pilot study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-05 19:05:47","doi":"10.21203/rs.3.rs-4008354/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-04-14T17:46:12+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-05T13:05:39+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"39ecf217-f19c-4a6a-a732-2c2b7c6f47fd","date":"2024-03-26T11:36:11+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"0009b35a-ec26-4c25-a538-379881ca67da","date":"2024-03-25T14:32:01+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-03-25T12:48:19+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-03-04T11:35:11+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-03-04T07:08:04+00:00","index":"","fulltext":""},{"type":"submitted","content":"Reproductive Biology and Endocrinology","date":"2024-03-03T10:57:31+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"reproductive-biology-and-endocrinology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"rbej","sideBox":"Learn more about [Reproductive Biology and Endocrinology](http://rbej.biomedcentral.com)","snPcode":"12958","submissionUrl":"https://submission.nature.com/new-submission/12958/3","title":"Reproductive Biology and Endocrinology","twitterHandle":"@BioMedCentral","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a8e4030b-c169-4831-b52d-7fc1d1f65343","owner":[],"postedDate":"March 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-05-14T16:39:41+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-05 19:05:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4008354","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4008354","identity":"rs-4008354","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}