Optimal Policy Determination for Autonomous Underwater Robots Using MDP and POMDP Frameworks

Optimal Policy Determination for Autonomous Underwater Robots Using MDP and POMDP Frameworks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Optimal Policy Determination for Autonomous Underwater Robots Using MDP and POMDP Frameworks Janakkumar Patel This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9286142/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Autonomous underwater robots operating in dynamic and uncertain ocean environments must navigate safely and efficiently toward predefined objectives while avoiding obstacles. This paper investigates the use of Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) frameworks as foundational tools for optimal policy determination in such cyber-physical systems. A two-dimensional 5x5 grid world models the seafloor environment, incorporating stochastic transitions that reflect real-world disturbances such as tides, currents, and sensor noise. Value iteration is applied to derive the MDP optimal baseline policy, which serves as an upper-bound benchmark for the POMDP solutions. Three offline POMDP solvers — QMDP, Fast Informed Bound (FIB), and Successive Approximations of the Reachable Space under Optimal Policies (SARSOP) — are evaluated under four observation model accuracy levels ranging from 70 to 100 percent. Monte Carlo simulations involving 15,000 independent trials are used to assess each solver. Results confirm that expected rewards increase with improved observability across all three methods. SARSOP consistently provides the tightest balance between computational tractability and solution quality. The analysis reveals that the greedy action selector, rather than the solution methods themselves, is the primary driver of performance variability across observation models. These findings contribute to the growing body of work on decision-making under uncertainty for autonomous marine cyber-physical systems and suggest several directions for practical enhancement. autonomous underwater vehicle Markov decision process POMDP cyber-physical systems SARSOP QMDP path planning decision under uncertainty Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9286142","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":615634267,"identity":"45a10b2f-bb00-466d-bafe-1da5a996c423","order_by":0,"name":"Janakkumar Patel","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7UlEQVRIiWNgGAWjYBACxgYQPsAgx8/MfADIl5AhWouxZHtbAkgLD5E2HWBI3HDmjAGIQ1gLc/vZgx9nnLnD2HAj5/OrGzUWPAzsh49uwGtDT16y5IYbz5gZZ+Rus845BnQYT1raDfyOyjGQfPDhMBuzRO424xw2oBYJHjP8WvrfGP8EauFhk8h5ZpzzjxgtM3LMgA47LMHDc4b5cW4bUVremFnOOPPMQIK9zYw5t0+Ch42QXwz7c4xv9hy7U7//MPPjzznf6uT42Q8fw6+lAUwdABFsEmASn3IQkGdAaGH+QEj1KBgFo2AUjEwAAHZMUQ/GycJdAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0002-3527-9331","institution":"Campbellsville University","correspondingAuthor":true,"prefix":"","firstName":"Janakkumar","middleName":"","lastName":"Patel","suffix":""}],"badges":[],"createdAt":"2026-04-01 03:53:05","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9286142/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9286142/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105967314,"identity":"70a31e24-47a8-47c7-9a9e-2a615ce89545","added_by":"auto","created_at":"2026-04-02 02:26:32","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":270605,"visible":true,"origin":"","legend":"","description":"","filename":"Manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9286142/v1_covered_37e2aff9-aa11-4fef-a139-80e34427b03e.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eOptimal Policy Determination for Autonomous Underwater Robots Using MDP and POMDP Frameworks\u003c/strong\u003e\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"autonomous underwater vehicle; Markov decision process; POMDP; cyber-physical systems; SARSOP; QMDP; path planning; decision under uncertainty","lastPublishedDoi":"10.21203/rs.3.rs-9286142/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9286142/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAutonomous underwater robots operating in dynamic and uncertain ocean environments must navigate safely and efficiently toward predefined objectives while avoiding obstacles. This paper investigates the use of Markov Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) frameworks as foundational tools for optimal policy determination in such cyber-physical systems. A two-dimensional 5x5 grid world models the seafloor environment, incorporating stochastic transitions that reflect real-world disturbances such as tides, currents, and sensor noise. Value iteration is applied to derive the MDP optimal baseline policy, which serves as an upper-bound benchmark for the POMDP solutions. Three offline POMDP solvers — QMDP, Fast Informed Bound (FIB), and Successive Approximations of the Reachable Space under Optimal Policies (SARSOP) — are evaluated under four observation model accuracy levels ranging from 70 to 100 percent. Monte Carlo simulations involving 15,000 independent trials are used to assess each solver. Results confirm that expected rewards increase with improved observability across all three methods. SARSOP consistently provides the tightest balance between computational tractability and solution quality. The analysis reveals that the greedy action selector, rather than the solution methods themselves, is the primary driver of performance variability across observation models. These findings contribute to the growing body of work on decision-making under uncertainty for autonomous marine cyber-physical systems and suggest several directions for practical enhancement.\u003c/p\u003e","manuscriptTitle":"Optimal Policy Determination for Autonomous Underwater Robots Using MDP and POMDP Frameworks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-02 02:26:26","doi":"10.21203/rs.3.rs-9286142/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"59474163-60f1-4ce3-b25c-a2bbee1e0d1e","owner":[],"postedDate":"April 2nd, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-04-02T02:26:26+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-02 02:26:26","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9286142","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9286142","identity":"rs-9286142","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}