Adaptive Heterogeneous Representation Learning for Reliable and Compact Pap Smear Screening

Adaptive Heterogeneous Representation Learning for Reliable and Compact Pap Smear Screening | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Adaptive Heterogeneous Representation Learning for Reliable and Compact Pap Smear Screening Vinitha Panicker J, G Gopakumar, Jyothisha J Nair This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9186105/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 14 You are reading this latest preprint version Abstract Cervical cancer remains a major cause of preventable mortality among women, making reliable cytology screening essential for early detection. However, manual Pap smear assessment is labour-intensive, time-consuming, and susceptible to inter-observer variability, while automated analysis is challenged by subtle inter-class differences, pronounced intra-class heterogeneity, and staining variability. This study presents a reliable and compact deep learning framework for automated Pap smear screening that addresses these challenges through adaptive fusion of complementary cytomorphological representations. The proposed model combines ResNet-50 and ConvNeXt-Base within a Dual Attention Fusion mechanism that assigns sample-specific importance to heterogeneous feature streams, enabling more discriminative integration than fixed fusion strategies. To improve efficiency, a Hierarchical Autoencoder learns compact task-relevant latent representations while preserving diagnostic performance. Evaluated on the SIPaKMeD and Herlev datasets under multiclass and binary screening settings using five-fold cross-validation, the framework achieves macro-F1 scores of 99.66% and 98.15%, with binary F1 scores of 99.65% and 99.41%, respectively. McNemar’s test indicates significant gains over single-backbone and non-adaptive fusion baselines. Grad-CAM highlights diagnostically relevant nuclear and peri-nuclear regions, while cross-dataset experiments suggest improved robustness under domain shift. These findings support the proposed framework as an interpretable and parameter-efficient approach for automated cervical cytology screening. Cervical cytology Cervical cancer screening Pap smear images Deep learning Dual attention fusion Explainable AI Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 30 Apr, 2026 Reviews received at journal 24 Apr, 2026 Reviews received at journal 20 Apr, 2026 Reviewers agreed at journal 18 Apr, 2026 Reviewers agreed at journal 12 Apr, 2026 Reviews received at journal 09 Apr, 2026 Reviewers agreed at journal 09 Apr, 2026 Reviews received at journal 07 Apr, 2026 Reviewers agreed at journal 01 Apr, 2026 Reviewers invited by journal 01 Apr, 2026 Editor invited by journal 27 Mar, 2026 Editor assigned by journal 24 Mar, 2026 Submission checks completed at journal 24 Mar, 2026 First submitted to journal 21 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9186105","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":616439213,"identity":"fc824688-9f59-499e-a557-f3abded4ea85","order_by":0,"name":"Vinitha Panicker J","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLUlEQVRIie3SP0uGQBzA8RPByXJVBH0LPghRPNBr+R0OLj7P4lIUTxeCU9AWQkFv4XFraDgRcrHduAhcnqkhG1qCyPN5AqHzmYP8Dudx8uGHfxAaG/uLyQhRifCdRPjV2pzrvVVAsh5xf4g+SHhrwhFCmPTHiIhT7GR5c3c4166j8yY5evEvr6j0/nS/v9CInDFVQPJdoFnphfpzFhnLMpwlDGQzWOm6ThVvKiBGpDo0i2VMKhxLdQwzwgCZAW2fhap75jA5w7cd+QLfbqd8cmJT7UNENLkjOV5ykhIAh4HSTXGoqgySx7gIJxWOjOQBJinD8bQlRpor7sHNb6Jopft2HJ/Mrcqvm4tTsC3m5SygC80qorp6FbzlTdDbr3+G7itvC7bfHhsbG/vXfQMxdG+GVjB95wAAAABJRU5ErkJggg==","orcid":"","institution":"Amrita Vishwa Vidyapeetham University","correspondingAuthor":true,"prefix":"","firstName":"Vinitha","middleName":"Panicker","lastName":"J","suffix":""},{"id":616439214,"identity":"114fc2fe-b967-4792-8f66-5e6fdfda518e","order_by":1,"name":"G Gopakumar","email":"","orcid":"","institution":"Amrita Vishwa Vidyapeetham University","correspondingAuthor":false,"prefix":"","firstName":"G","middleName":"","lastName":"Gopakumar","suffix":""},{"id":616439215,"identity":"1179e94e-5956-4ec8-847a-3ae685cff1ff","order_by":2,"name":"Jyothisha J Nair","email":"","orcid":"","institution":"Mahatma Gandhi University","correspondingAuthor":false,"prefix":"","firstName":"Jyothisha","middleName":"J","lastName":"Nair","suffix":""}],"badges":[],"createdAt":"2026-03-21 13:38:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9186105/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9186105/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106403851,"identity":"a030a607-eef5-4860-a2ab-03e8189778be","added_by":"auto","created_at":"2026-04-08 09:15:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2888758,"visible":true,"origin":"","legend":"","description":"","filename":"AdaptiveHeterogeneousRepresentationLearning.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9186105/v1_covered_cce98e90-d271-4a96-9136-e88c2b9c6fa2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Adaptive Heterogeneous Representation Learning for Reliable and Compact Pap Smear Screening","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Cervical cytology, Cervical cancer screening, Pap smear images, Deep learning, Dual attention fusion, Explainable AI","lastPublishedDoi":"10.21203/rs.3.rs-9186105/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9186105/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eCervical cancer remains a major cause of preventable mortality among women, making reliable cytology screening essential for early detection. However, manual Pap smear assessment is labour-intensive, time-consuming, and susceptible to inter-observer variability, while automated analysis is challenged by subtle inter-class differences, pronounced intra-class heterogeneity, and staining variability. This study presents a reliable and compact deep learning framework for automated Pap smear screening that addresses these challenges through adaptive fusion of complementary cytomorphological representations. The proposed model combines ResNet-50 and ConvNeXt-Base within a Dual Attention Fusion mechanism that assigns sample-specific importance to heterogeneous feature streams, enabling more discriminative integration than fixed fusion strategies. To improve efficiency, a Hierarchical Autoencoder learns compact task-relevant latent representations while preserving diagnostic performance. Evaluated on the SIPaKMeD and Herlev datasets under multiclass and binary screening settings using five-fold cross-validation, the framework achieves macro-F1 scores of 99.66% and 98.15%, with binary F1 scores of 99.65% and 99.41%, respectively. McNemar\u0026rsquo;s test indicates significant gains over single-backbone and non-adaptive fusion baselines. Grad-CAM highlights diagnostically relevant nuclear and peri-nuclear regions, while cross-dataset experiments suggest improved robustness under domain shift. These findings support the proposed framework as an interpretable and parameter-efficient approach for automated cervical cytology screening.\u003c/p\u003e","manuscriptTitle":"Adaptive Heterogeneous Representation Learning for Reliable and Compact Pap Smear Screening","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-06 21:50:36","doi":"10.21203/rs.3.rs-9186105/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-30T14:31:16+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-24T07:22:52+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-20T15:34:00+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"187680001751276723883748366237326932119","date":"2026-04-18T11:10:41+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"101101080033405770476927917492758030303","date":"2026-04-12T14:47:42+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-09T10:54:03+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"155296409967793772719470494720392590357","date":"2026-04-09T08:14:06+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-07T10:26:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"88380949859774691313415325416344890071","date":"2026-04-01T13:50:19+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-01T13:24:47+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-27T15:07:45+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-24T11:18:19+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-24T11:17:55+00:00","index":"","fulltext":""},{"type":"submitted","content":"Discover Artificial Intelligence","date":"2026-03-21T13:31:29+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"discover-artificial-intelligence","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"diai","sideBox":"Learn more about [Discover Artificial Intelligence](https://www.springer.com/44163)","snPcode":"","submissionUrl":"","title":"Discover Artificial Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Discover Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"36e04d4a-62a4-4249-a2cb-0b9c826308bf","owner":[],"postedDate":"April 6th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Revision requested","date":"2026-04-30T14:31:16+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-12T10:30:50+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-06 21:50:36","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9186105","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9186105","identity":"rs-9186105","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}