Evaluating Deep Learning Change Detection in Aerial Imagery: A New Multi-Metric Benchmarking Framework

Evaluating Deep Learning Change Detection in Aerial Imagery: A New Multi-Metric Benchmarking Framework | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Method Article Evaluating Deep Learning Change Detection in Aerial Imagery: A New Multi-Metric Benchmarking Framework Ahmed Alaa Abdelbaky Hassouna, Mohamed Badr Ismail, Amany Shaban Hassan, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6486635/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Change detection (CD) in aerial images is a critical task in various applications such as urban expansion monitoring, environmental assessment, and disaster response. However, the existing literature often lacks comprehensive and systematic evaluations of deep learning (DL)-based CD models, leaving gaps in understanding their generalizability, robustness, and performance trade-offs across diverse conditions. This study addresses these gaps by proposing a novel framework for benchmarking and assessing CD models, offering a detailed and quantitative evaluation of five state-of-the-art models: CSA-CDGAN, Changeformer, BIT, Tiny, and SNUNet. Our framework consists of three distinct evaluation pipelines: (1) cross-testing across diverse benchmark datasets to assess generalization, (2) sensitivity analysis to examine model performance with respect to change size and complexity, and (3) robustness analysis to evaluate resilience against image corruptions and noise. Key results demonstrate the utility of our framework in revealing the strengths and weaknesses of the evaluated models. CSA-CDGAN excels in handling high noise levels, which showed the highest precision and F1 score, and maintained strong recall across a wide noise spectrum. Changeformer outperforms others in moderately noisy conditions (30-31 dB), while Tiny excels in detecting smaller changes under severe noise (29.35-29.5 dB). Additionally, the framework highlights the challenges faced by BIT, particularly its lower performance in both precision and recall, making it less suited for high-noise environments. This comprehensive benchmarking framework provides critical insights for selecting suitable CD models based on real-world application needs, considering factors like noise levels, change sizes, and dataset variability. The results also lay the groundwork for future research, guiding the development of more robust and versatile CD models. The study establishes a new standard for model evaluation, offering a systematic approach to improve the reliability and applicability of CD models in practical scenarios. Change Detection (CD) Remote Sensing Aerial Images Deep Learning (DL) 25 Sustainable Development Benchmarking Robustness Analysis Contour Analytics Model 26 Evaluation. Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6486635","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Method Article","associatedPublications":[],"authors":[{"id":458276681,"identity":"2346b645-9211-42ab-ab80-b404cd7225cf","order_by":0,"name":"Ahmed Alaa Abdelbaky Hassouna","email":"","orcid":"","institution":"Minia University","correspondingAuthor":false,"prefix":"","firstName":"Ahmed","middleName":"Alaa Abdelbaky","lastName":"Hassouna","suffix":""},{"id":458276682,"identity":"a2883989-4370-465e-a5ce-30fd4418b1bc","order_by":1,"name":"Mohamed Badr Ismail","email":"","orcid":"","institution":"Minia University","correspondingAuthor":false,"prefix":"","firstName":"Mohamed","middleName":"Badr","lastName":"Ismail","suffix":""},{"id":458276683,"identity":"d3a7a8d6-9d4b-4fba-93ce-3661d27b2c37","order_by":2,"name":"Amany Shaban Hassan","email":"","orcid":"","institution":"Minia University","correspondingAuthor":false,"prefix":"","firstName":"Amany","middleName":"Shaban","lastName":"Hassan","suffix":""},{"id":458276684,"identity":"6f878e25-9f4a-49fb-a8f4-0e038c74dd83","order_by":3,"name":"Huthaifa Ashqar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCElEQVRIiWNgGAWjYBACAwYGNiApB2KzgQg5qAQzHi3MIC3GcC3GRGphQGhJbCCkxVwi/9iDHwUG8gzsx589eFNzL72//YzZA4YKa5heDGA5I5ndsMfAwLCBJ8fccM6x4twZZ3LMDRjOpOPUYnAjmU2Cx+APYwNDDps0D1tCbsOBHDMJxrbDeLVI/jEwsG/gf/5MmudfQrr8+TdALf/wa5HmMTBIbJBIMJPmbUtIMLgBsqUBj5Yzj82kZQwMktsk3phJzu1LMNx441m5QcKxdGOcWo4nPpN888fAtp8//ZnEm28J8nLnk7c9+FBjLYtLCxyAI4UHxk4gpBwO4FpGwSgYBaNgFCABAD+0U927hDu1AAAAAElFTkSuQmCC","orcid":"","institution":"Arab American University","correspondingAuthor":true,"prefix":"","firstName":"Huthaifa","middleName":"","lastName":"Ashqar","suffix":""},{"id":458276685,"identity":"664e7a96-a97b-4c4c-81c1-4bb76761895d","order_by":4,"name":"Anas M.R. Alsobeh","email":"","orcid":"","institution":"Southern Illinois University Carbondale","correspondingAuthor":false,"prefix":"","firstName":"Anas","middleName":"M.R.","lastName":"Alsobeh","suffix":""},{"id":458276686,"identity":"7b4cd482-7176-4166-b841-38d2b5937878","order_by":5,"name":"Abdallah A. Hassan","email":"","orcid":"","institution":"Minia University","correspondingAuthor":false,"prefix":"","firstName":"Abdallah","middleName":"A.","lastName":"Hassan","suffix":""},{"id":458276687,"identity":"ca7d94bb-8738-48d2-a2aa-c06a2ca75bb5","order_by":6,"name":"Mohammed Elhenawy","email":"","orcid":"","institution":"Queensland University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"","lastName":"Elhenawy","suffix":""}],"badges":[],"createdAt":"2025-04-19 21:23:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6486635/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6486635/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":91243045,"identity":"8eabebaa-7068-4d67-a8f0-a9be8ab3eb6a","added_by":"auto","created_at":"2025-09-13 12:46:55","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2059163,"visible":true,"origin":"","legend":"","description":"","filename":"znctjryzrzbhqbxctwhvnzdqdfmnykkn.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6486635/v1_covered_8ae57c12-6bd5-4d70-915b-9259e59b4846.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluating Deep Learning Change Detection in Aerial Imagery: A New Multi-Metric Benchmarking Framework","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Change Detection (CD), Remote Sensing, Aerial Images, Deep Learning (DL), 25 Sustainable Development, Benchmarking, Robustness Analysis, Contour Analytics, Model 26 Evaluation.","lastPublishedDoi":"10.21203/rs.3.rs-6486635/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6486635/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Change detection (CD) in aerial images is a critical task in various applications such as urban expansion monitoring, environmental assessment, and disaster response. However, the existing literature often lacks comprehensive and systematic evaluations of deep learning (DL)-based CD models, leaving gaps in understanding their generalizability, robustness, and performance trade-offs across diverse conditions. This study addresses these gaps by proposing a novel framework for benchmarking and assessing CD models, offering a detailed and quantitative evaluation of five state-of-the-art models: CSA-CDGAN, Changeformer, BIT, Tiny, and SNUNet. Our framework consists of three distinct evaluation pipelines: (1) cross-testing across diverse benchmark datasets to assess generalization, (2) sensitivity analysis to examine model performance with respect to change size and complexity, and (3) robustness analysis to evaluate resilience against image corruptions and noise. Key results demonstrate the utility of our framework in revealing the strengths and weaknesses of the evaluated models. CSA-CDGAN excels in handling high noise levels, which showed the highest precision and F1 score, and maintained strong recall across a wide noise spectrum. Changeformer outperforms others in moderately noisy conditions (30-31 dB), while Tiny excels in detecting smaller changes under severe noise (29.35-29.5 dB). Additionally, the framework highlights the challenges faced by BIT, particularly its lower performance in both precision and recall, making it less suited for high-noise environments. This comprehensive benchmarking framework provides critical insights for selecting suitable CD models based on real-world application needs, considering factors like noise levels, change sizes, and dataset variability. The results also lay the groundwork for future research, guiding the development of more robust and versatile CD models. The study establishes a new standard for model evaluation, offering a systematic approach to improve the reliability and applicability of CD models in practical scenarios.","manuscriptTitle":"Evaluating Deep Learning Change Detection in Aerial Imagery: A New Multi-Metric Benchmarking Framework","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-20 05:33:38","doi":"10.21203/rs.3.rs-6486635/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2f453f25-bca8-4b29-ab52-d4e0e299f9a1","owner":[],"postedDate":"May 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-09-13T12:38:45+00:00","versionOfRecord":[],"versionCreatedAt":"2025-05-20 05:33:38","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6486635","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6486635","identity":"rs-6486635","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}