Hard-Rejection Outlier Detection in Ensemble Kalman Filtering: A Three-Stage Robust Framework with Catastrophic Divergence Protection

Hard-Rejection Outlier Detection in Ensemble Kalman Filtering: A Three-Stage Robust Framework with Catastrophic Divergence Protection | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Hard-Rejection Outlier Detection in Ensemble Kalman Filtering: A Three-Stage Robust Framework with Catastrophic Divergence Protection Jaafar Almutawa This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9414916/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The Ensemble Kalman Filter (EnKF) is widely used for high-dimensional nonlinear state estimation, yet it remains vulnerable to outliers in observations, process dynamics, and initialization. This paper proposes the Dynamical Hybrid Least Trimmed Squares EnKF (DH-LTS-EnKF), a three-stage robust filtering framework that addresses all three contamination sources within a single sequential architecture. At each assimilation step, per-sensor standardized innovations identify and replace corrupted observations with forecast predictions (hard rejection), an adaptive innovation-norm monitor detects process disturbances and temporarily reduces observation trust, and Mahalanobis-distance trimming guards against initialization errors. In the absence of outliers, no detection triggers and the algorithm reduces exactly to the standard EnKF. Robustness characterizations are provided, including a detection guarantee for measurement outliers and a heuristic error decomposition quantifying false-positive and undetected-process contributions. Experiments on a 100-state heat conduction problem and the 40-state chaotic Lorenz-96 model (200 Monte Carlo runs each) show that DH-LTS-EnKF reduces state RMSE by up to 53% under moderate contamination and by $99.5%$ under extreme measurement outliers. At large outlier magnitudes, hard rejection achieves $81%$ lower RMSE than Huber-based soft downweighting. On Lorenz-96, standard and covariance-inflated EnKF diverge catastrophically under extreme outliers, while DH-LTS-EnKF maintains stable near-baseline tracking in all 200 runs. Ensemble Kalman Filter Least Trimmed Squares Robust state estimation Outlier detection Data assimilation Lorenz-96 Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9414916","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":630255161,"identity":"84ed0067-9782-43be-8d1a-8193fe1f6ae4","order_by":0,"name":"Jaafar Almutawa","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABPElEQVRIie3Qv0vDQBQH8BdOkqVxlEDU/AsJDlHQ+K8kBNqlFYJLQcELgXOJZG3xn4hLVHC4ENAl0LVjiqBLhnQsaDE/pCIJmQXzXe7d4z483gF06fIHw2CArCi2vxv7AAgXhQ4oLxncRJhJUbAVh4PypC2kUL+IgTcEmgm6jhZ49QQSuxOryYpog3vOJmI21s5UDiXIeqiPcPuK7b6BQsShYt8Qc/TohkSgsXl+5LAymsYNu+hq0qPAFATzARr5c4MIIaGGH/UA8aROvPel/UHhtJzyGVwN5JKsW8hkqDj5FKMgDh9EekVwG0ktZ5cKJhH71u3e+kXJd3EO42czJ6wcTetE8QZ3dkqPTzzR9JdpfCGpnBPOx5ea4c+ixavVQKp/F346MsDW5kprAECqdXKCsoaXXbp06fJ/8wU94HhuTPp1CAAAAABJRU5ErkJggg==","orcid":"","institution":"Bahrain Polytechnic","correspondingAuthor":true,"prefix":"","firstName":"Jaafar","middleName":"","lastName":"Almutawa","suffix":""}],"badges":[],"createdAt":"2026-04-14 11:39:01","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9414916/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9414916/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108491396,"identity":"527bdedc-e0c0-4eee-99a9-7f1b30cf6644","added_by":"auto","created_at":"2026-05-05 09:53:45","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":583196,"visible":true,"origin":"","legend":"","description":"","filename":"ManuscriptFiles.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9414916/v1_covered_8f8c092e-9426-4de8-bd2c-689e08eb3f14.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"\nHard-Rejection Outlier Detection in Ensemble Kalman Filtering: A Three-Stage Robust Framework with Catastrophic Divergence Protection\n","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Ensemble Kalman Filter, Least Trimmed Squares, Robust state estimation, Outlier detection, Data assimilation, Lorenz-96","lastPublishedDoi":"10.21203/rs.3.rs-9414916/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9414916/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The Ensemble Kalman Filter (EnKF) is widely used for\nhigh-dimensional nonlinear state estimation, yet it remains\nvulnerable to outliers in observations, process dynamics, and\ninitialization. This paper proposes the Dynamical Hybrid Least\nTrimmed Squares EnKF (DH-LTS-EnKF), a three-stage robust\nfiltering framework that addresses all three contamination\nsources within a single sequential architecture. At each\nassimilation step, per-sensor standardized innovations identify\nand replace corrupted observations with forecast predictions\n(hard rejection), an adaptive innovation-norm monitor detects\nprocess disturbances and temporarily reduces observation trust,\nand Mahalanobis-distance trimming guards against initialization\nerrors. In the absence of outliers, no detection triggers and the\nalgorithm reduces exactly to the standard EnKF. Robustness\ncharacterizations are provided, including a detection guarantee\nfor measurement outliers and a heuristic error decomposition\nquantifying false-positive and undetected-process contributions.\nExperiments on a 100-state heat conduction problem and the\n40-state chaotic Lorenz-96 model (200 Monte Carlo runs each)\nshow that DH-LTS-EnKF reduces state RMSE by up to 53\\%\nunder moderate contamination and by $99.5\\%$ under extreme\nmeasurement outliers. At large outlier magnitudes, hard rejection\nachieves $81\\%$ lower RMSE than Huber-based soft downweighting.\nOn Lorenz-96, standard and covariance-inflated EnKF diverge\ncatastrophically under extreme outliers, while DH-LTS-EnKF\nmaintains stable near-baseline tracking in all 200 runs.","manuscriptTitle":"Hard-Rejection Outlier Detection in Ensemble Kalman Filtering: A Three-Stage Robust Framework with Catastrophic Divergence Protection","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-30 22:28:48","doi":"10.21203/rs.3.rs-9414916/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3794567b-e3f8-426d-b62a-a42fc10616c0","owner":[],"postedDate":"April 30th, 2026","published":true,"recentEditorialEvents":[{"type":"editorInvitedReview","content":"","date":"2026-05-12T14:34:11+00:00","index":14,"fulltext":""},{"type":"reviewerAgreed","content":"45550823414481227997248133097382864423","date":"2026-05-04T12:14:45+00:00","index":13,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-30T22:28:48+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-30 22:28:48","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9414916","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9414916","identity":"rs-9414916","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}