SecuDevSLM: A Systematic Security Evaluation Framework for Small Language Models on Mobile Devices

SecuDevSLM: A Systematic Security Evaluation Framework for Small Language Models on Mobile Devices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article SecuDevSLM: A Systematic Security Evaluation Framework for Small Language Models on Mobile Devices Zijun Li, Tengfei Tu, Yuxin Lian, Zicheng Kong, Aofan Liu, Wenmin Li This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7352742/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The deployment of Small Language Models (SLMs) on edge devices introduces critical security challenges, as their constrained architectures and heterogeneous runtime environments exacerbate vulnerabilities like hallucination and jailbreaking—risks that remain understudied in resource-limited settings. To address this, we present SecuDevSLM, a fully on-device security evaluation framework for iOS and Android. Unlike traditional API-dependent approaches, SecuDevSLM eliminates dependence on external LLM APIs by leveraging a unified adversarial corpus for attack simulation, and introduces an integrated detection mechanism combining semantic noise entropy analysis and dual-modal evaluation, achieving 9% higher F1-score in threat identification compared to conventional tools. It systematically tests SLM robustness across 15 threat categories using 27 adversarial templates, generating multi-turn hallucination, jailbreak, and noise-based attacks. In total, 58 mainstream SLMs undergo tens of thousands of attack rounds. Compared to API-based methods, SecuDevSLM avoids high interaction costs and mitigates data leakage risks. Our experiments show that for sem antically aligned attacks, training corpus richness positively correlates with model robustness, though with diminishing returns. In weakly aligned attacks, the overall capacity of the model plays a larger role. Platform differences also impact stability: under hallucination, Android exhibits 23% greater character-level variance and 1.7 \((\times)\) higher fluctuation in response to text length than iOS. However, in long-text hallucinations, both perform similarly. For jailbreak attacks, Android shows 56% more output volatility, and when corpus richness is low, its attack success rate reaches 85%, versus 10% on iOS. Finally, we confirm that static indicators, such as parameter count and dataset features, are strongly linearly correlated with real-world performance, enabling developers better measure the security and performance of models before they are connected to the system. Small Language Models On-Device Security Hallucination Attacks Jailbreak Attacks Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7352742","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":539046418,"identity":"95be1e40-ac34-41aa-9f15-e2f300c9d285","order_by":0,"name":"Zijun Li","email":"","orcid":"","institution":"Beijing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Zijun","middleName":"","lastName":"Li","suffix":""},{"id":539046419,"identity":"066cbab7-2841-40de-9210-bc142a8f6daa","order_by":1,"name":"Tengfei Tu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABGklEQVRIie3Rv0vDQBTA8RcCd8vFOVJM/oUXbujQwX8lR4csTe3oIHjTZQl1jf9FN+lWCdQl7g4OSsBRAi7+COI1AVFIoqPgfW9L3oc8LgAm09/NA6IPAIKH7RO7d5a17/gn4ZpY8lcEGgIgVj+RQ/fosly8orcHRJRscRtduPHVHRxPhKTXm86vuFObny+R68VyzvAhXmdzIaGIhGTzsIeQkZPWQgFV+xnm8epmFkhL5UK6DHsIfXNSPNUkedEkwoa8DxJis2cM9WJbqDAPWyIHSFHykSMxUDaZupoE6/RRZOE24orNOglNxP0Tq9H3ExVUYZ37YxpvqupkcnBGi07SZCn4+iMYwO6qSO/8rvr7soOzJpPJ9P/6AA08WMr+6Nj2AAAAAElFTkSuQmCC","orcid":"","institution":"Beijing University of Posts and Telecommunications","correspondingAuthor":true,"prefix":"","firstName":"Tengfei","middleName":"","lastName":"Tu","suffix":""},{"id":539046420,"identity":"68e023c2-4032-46ba-8f70-192c980e36a0","order_by":2,"name":"Yuxin Lian","email":"","orcid":"","institution":"Beijing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Yuxin","middleName":"","lastName":"Lian","suffix":""},{"id":539046421,"identity":"63f2b233-35f8-400c-89c0-f69f039545e6","order_by":3,"name":"Zicheng Kong","email":"","orcid":"","institution":"Beijing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Zicheng","middleName":"","lastName":"Kong","suffix":""},{"id":539046422,"identity":"58ed475e-ef94-4f45-89d4-ee0ac02f0bb0","order_by":4,"name":"Aofan Liu","email":"","orcid":"","institution":"Peking University","correspondingAuthor":false,"prefix":"","firstName":"Aofan","middleName":"","lastName":"Liu","suffix":""},{"id":539046423,"identity":"acfae325-9948-4dee-bb2c-3f97593f7a83","order_by":5,"name":"Wenmin Li","email":"","orcid":"","institution":"Beijing University of Posts and Telecommunications","correspondingAuthor":false,"prefix":"","firstName":"Wenmin","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2025-08-12 07:38:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7352742/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7352742/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95524080,"identity":"47ee3c39-42ef-489f-93a1-c2b7587fd1c2","added_by":"auto","created_at":"2025-11-10 10:02:08","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7924,"visible":true,"origin":"","legend":"","description":"","filename":"338e50b305b245dc8cf17e3e88f30391.json","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/d65b5aedbadde5f9e6403b57.json"},{"id":95344192,"identity":"bcdbe9f3-b13c-4f8f-9bcf-7e8df793f348","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"xml","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":154936,"visible":true,"origin":"","legend":"","description":"","filename":"338e50b305b245dc8cf17e3e88f303911enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/cca9e273b118817f5ae70684.xml"},{"id":95344194,"identity":"c649b8cb-c76d-4f10-aa05-62c2e7ef40a8","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5653492,"visible":true,"origin":"","legend":"","description":"","filename":"SecuDevSLM.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/9caa93cf7bf008ea0e3cd5db.pdf"},{"id":95344193,"identity":"ea1d178d-c3e5-414c-b192-48bcd1f8d278","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1156218,"visible":true,"origin":"","legend":"","description":"","filename":"V21.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/e4464686b23ff9458aedd9fc.png"},{"id":95344190,"identity":"13af4c40-18ce-4f2f-ae33-f51feb96e53d","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":128746,"visible":true,"origin":"","legend":"","description":"","filename":"V22.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/d7964438f8130c0a2249ec4b.png"},{"id":95344202,"identity":"3f927eb6-377e-4ef8-9cb8-fa8ffbf5a363","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"pdf","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":287524,"visible":true,"origin":"","legend":"","description":"","filename":"fig10epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/96959ff39881f4b83a6cc00d.pdf"},{"id":95344198,"identity":"8b38487b-b512-4596-99c7-6880804e8288","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"eps","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5212974,"visible":true,"origin":"","legend":"","description":"","filename":"fig10.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/05863a6804ada69b07a0161b.eps"},{"id":95525047,"identity":"813fa661-345c-4919-9489-65cc49763f16","added_by":"auto","created_at":"2025-11-10 10:04:07","extension":"pdf","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":284313,"visible":true,"origin":"","legend":"","description":"","filename":"fig11epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/ed95e37a1176cb7d33a6e64c.pdf"},{"id":95344191,"identity":"fd3a4eb0-729b-4d74-9112-736ff8a271c0","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"eps","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5231326,"visible":true,"origin":"","legend":"","description":"","filename":"fig11.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/c4d11182106d806dd90e7038.eps"},{"id":95344218,"identity":"28deccc2-883f-47d1-b2ce-685002d0ac6a","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":289781,"visible":true,"origin":"","legend":"","description":"","filename":"fig12epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/e40c156f40d75cffc1c24007.pdf"},{"id":95344196,"identity":"bcd7e28e-15ff-42cb-a21c-be0226a1b4a6","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"eps","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5199325,"visible":true,"origin":"","legend":"","description":"","filename":"fig12.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/22141b8abdbef05e2c4fd4c4.eps"},{"id":95344199,"identity":"40cebbab-d07f-4fb3-96ff-52f73df975d9","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"pdf","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":240669,"visible":true,"origin":"","legend":"","description":"","filename":"fig13epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/bfd4d282475580a1fad0d77a.pdf"},{"id":95524997,"identity":"3ed9a0bb-3423-4186-ad9b-cbf58b51419a","added_by":"auto","created_at":"2025-11-10 10:04:01","extension":"eps","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6327724,"visible":true,"origin":"","legend":"","description":"","filename":"fig13.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/3024d2e87cfb39014c92ce2d.eps"},{"id":95344200,"identity":"9669bff0-b98d-4ad2-a6dd-9d4307b0d211","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"jpeg","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":115919,"visible":true,"origin":"","legend":"","description":"","filename":"fig13.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/2c066b461f4fa15ceaf38f22.jpeg"},{"id":95344203,"identity":"1401c1ed-ce37-4c2f-b504-1a8f2dbb8631","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":293438,"visible":true,"origin":"","legend":"","description":"","filename":"fig14epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/526954a47f3c004aa0577472.pdf"},{"id":95344212,"identity":"2193f709-33c4-47e6-9e13-35cb88cdf8d8","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4623658,"visible":true,"origin":"","legend":"","description":"","filename":"fig14.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/9c50803e63fed8dceed5176e.eps"},{"id":95524773,"identity":"2f721ddc-5b18-4b9c-83a8-50cde61a5479","added_by":"auto","created_at":"2025-11-10 10:03:28","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":187480,"visible":true,"origin":"","legend":"","description":"","filename":"fig14.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/ab111a22c3348a035e01b6dd.png"},{"id":95525180,"identity":"2532dd2d-5f98-4ede-9db5-5adebf11157e","added_by":"auto","created_at":"2025-11-10 10:04:26","extension":"pdf","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":257735,"visible":true,"origin":"","legend":"","description":"","filename":"fig15epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/891bda545375b009ff7c436a.pdf"},{"id":95344201,"identity":"350f0c16-5fc9-478f-acb4-853c54ff87c9","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"eps","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4620364,"visible":true,"origin":"","legend":"","description":"","filename":"fig15.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/1c9575e4017ecdc262be89e6.eps"},{"id":95523846,"identity":"53709864-71ec-483e-a443-ccac8e303fab","added_by":"auto","created_at":"2025-11-10 10:01:13","extension":"png","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":143694,"visible":true,"origin":"","legend":"","description":"","filename":"fig15.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/53bd0dc4679df3bb9307649e.png"},{"id":95344209,"identity":"4d6503e1-b60a-4e1f-95bd-2dfa4098f788","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1723391,"visible":true,"origin":"","legend":"","description":"","filename":"fig16epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/6001bc0a585c98ca03f590d7.pdf"},{"id":95524454,"identity":"d323f4da-869d-47e3-ab98-2af1cffffabf","added_by":"auto","created_at":"2025-11-10 10:02:47","extension":"eps","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6404695,"visible":true,"origin":"","legend":"","description":"","filename":"fig16.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/5c38d61dc2292ebc7b36018a.eps"},{"id":95344227,"identity":"4561efe6-8568-45b0-991b-21876fe6c354","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":275409,"visible":true,"origin":"","legend":"","description":"","filename":"fig16.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/53253565d46b73b35cf08b96.png"},{"id":95524925,"identity":"87e83640-2b7d-4f96-90d1-61b643ae60fd","added_by":"auto","created_at":"2025-11-10 10:03:48","extension":"pdf","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1812183,"visible":true,"origin":"","legend":"","description":"","filename":"fig17epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/cbe620189bd8172949de8571.pdf"},{"id":95344232,"identity":"8205fc34-e09b-4cad-9c2a-720f1f4c05bf","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":13383548,"visible":true,"origin":"","legend":"","description":"","filename":"fig17.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/e1b4cca42fa3724a5ff7d95a.eps"},{"id":95524936,"identity":"f32987d5-c0b0-405d-8012-82ba563a0659","added_by":"auto","created_at":"2025-11-10 10:03:50","extension":"jpeg","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":486603,"visible":true,"origin":"","legend":"","description":"","filename":"fig17.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/1f60c2dcee2321ca80e92edd.jpeg"},{"id":95344197,"identity":"0512279d-f663-4711-ae54-183070b2adb0","added_by":"auto","created_at":"2025-11-07 02:52:43","extension":"pdf","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":210325,"visible":true,"origin":"","legend":"","description":"","filename":"fig2epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/5c6fdb96ceae458e90aee290.pdf"},{"id":95344216,"identity":"e71c7374-cef3-4a37-8673-32bc23f07f3b","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":29,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3684402,"visible":true,"origin":"","legend":"","description":"","filename":"fig3a.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/853e9bfe5dc67143491d9c22.eps"},{"id":95524368,"identity":"432b141a-3194-4d58-ad59-6df4c66a77c5","added_by":"auto","created_at":"2025-11-10 10:02:40","extension":"png","order_by":30,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":79448,"visible":true,"origin":"","legend":"","description":"","filename":"fig3a.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/7bf45f6e313f75eb29682a00.png"},{"id":95525332,"identity":"be42df8d-f97c-4594-b2a4-98b7d30a151d","added_by":"auto","created_at":"2025-11-10 10:04:50","extension":"eps","order_by":31,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4275590,"visible":true,"origin":"","legend":"","description":"","filename":"fig3b.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/bfe7e1dc5ebc693d389ce04d.eps"},{"id":95344205,"identity":"f9ce8fd6-e1c8-4e26-a2b4-6ddbfbeed398","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":32,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":146496,"visible":true,"origin":"","legend":"","description":"","filename":"fig3b.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/4084c028cd07eaa1ecd114a0.png"},{"id":95344215,"identity":"87533bc9-1ed5-4561-ac3f-9894cc238cd9","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":33,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":223720,"visible":true,"origin":"","legend":"","description":"","filename":"fig4epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/ff70c6da492775432271d80a.pdf"},{"id":95344225,"identity":"b1b3d646-e140-433b-b73e-6ba21cb1bf3e","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":34,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7795593,"visible":true,"origin":"","legend":"","description":"","filename":"fig4.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/206cbf27ca003e41ff3a5454.eps"},{"id":95524319,"identity":"abb5b33d-a94a-4faa-90d8-76387ad75489","added_by":"auto","created_at":"2025-11-10 10:02:37","extension":"jpeg","order_by":35,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":101243,"visible":true,"origin":"","legend":"","description":"","filename":"fig4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/3e52ead8d723a67446e19feb.jpeg"},{"id":95344236,"identity":"f21e6b69-8181-461b-a6cf-94758b5f748d","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"pdf","order_by":36,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1191358,"visible":true,"origin":"","legend":"","description":"","filename":"fig5epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/c5c7ad4779b543350ee22cf1.pdf"},{"id":95344245,"identity":"6a36da59-c5f6-4788-bcf0-3301ffd6557f","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"eps","order_by":37,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5501889,"visible":true,"origin":"","legend":"","description":"","filename":"fig5.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/b6f48999126360c7fba91d8c.eps"},{"id":95344222,"identity":"a9ad115a-26e1-4d41-813d-73457daa5832","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"jpeg","order_by":38,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":285731,"visible":true,"origin":"","legend":"","description":"","filename":"fig5.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/918a6eb6b0ce7e9ea1e5166d.jpeg"},{"id":95344243,"identity":"76b22c66-8013-4051-8211-fe9f9aad4a19","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"pdf","order_by":39,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1209317,"visible":true,"origin":"","legend":"","description":"","filename":"fig6epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/97f63a1c08fe6a37351fdada.pdf"},{"id":95344229,"identity":"7ce9bc80-3d56-4a1b-887f-12b28643976f","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":40,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4605384,"visible":true,"origin":"","legend":"","description":"","filename":"fig6.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/5046a3c44be75190cc3e0118.eps"},{"id":95344237,"identity":"6d15709a-6c62-4483-bd93-8237fe622be1","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"png","order_by":41,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":229472,"visible":true,"origin":"","legend":"","description":"","filename":"fig6.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/a9162bdd4f81f634c16adc7d.png"},{"id":95344230,"identity":"f1d74921-0c4e-4266-bd8b-c79c8ea0205f","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":42,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1242022,"visible":true,"origin":"","legend":"","description":"","filename":"fig7aepsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/664bf0895c486cecdf8514ad.pdf"},{"id":95525303,"identity":"d8b12cb4-3452-4492-a78f-f684a4e7aa78","added_by":"auto","created_at":"2025-11-10 10:04:45","extension":"eps","order_by":43,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3442219,"visible":true,"origin":"","legend":"","description":"","filename":"fig7a.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/35f3c79333a3e106c19c14e3.eps"},{"id":95344248,"identity":"b3daf0c1-fb20-4a44-b9d5-b2d78fb99420","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"jpeg","order_by":44,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":131443,"visible":true,"origin":"","legend":"","description":"","filename":"fig7a.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/f9166e4b8f10cf98522f301b.jpeg"},{"id":95344241,"identity":"99794920-ecaa-4ac2-91ef-25815cf622f5","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"pdf","order_by":45,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":800131,"visible":true,"origin":"","legend":"","description":"","filename":"fig7bepsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/bbe48ff1df697dc1b65b7c40.pdf"},{"id":95344228,"identity":"9f7ffc6d-f1e8-40fe-b966-f062373cc6ba","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":46,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3502921,"visible":true,"origin":"","legend":"","description":"","filename":"fig7b.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/6376e0a874376cef81e03e1c.eps"},{"id":95524127,"identity":"4fcc5464-7fa8-45cd-9afa-602624ba1ca4","added_by":"auto","created_at":"2025-11-10 10:02:18","extension":"jpeg","order_by":47,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":82158,"visible":true,"origin":"","legend":"","description":"","filename":"fig7b.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/dd4f5ad56d60e728950d6df7.jpeg"},{"id":95344247,"identity":"7e2e3de0-e0af-429f-9c8d-448b3573360b","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"pdf","order_by":48,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":279456,"visible":true,"origin":"","legend":"","description":"","filename":"fig8epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/0549a852b05ab8b6f860b8d5.pdf"},{"id":95344210,"identity":"4b697ae1-1df7-4fa4-9e20-96ff7f030543","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"eps","order_by":49,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4803328,"visible":true,"origin":"","legend":"","description":"","filename":"fig8.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/89e38848f44113a19a4b8462.eps"},{"id":95344231,"identity":"b3d9891d-112b-4cdd-b4d2-0787c89c1c5e","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"pdf","order_by":50,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":293293,"visible":true,"origin":"","legend":"","description":"","filename":"fig9epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/2985864ae8118dc3ff5724bb.pdf"},{"id":95524971,"identity":"724b3f69-9bde-403c-a1f9-999ee7194a72","added_by":"auto","created_at":"2025-11-10 10:03:56","extension":"eps","order_by":51,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5295620,"visible":true,"origin":"","legend":"","description":"","filename":"fig9.eps","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/66b92a918c2fec60a6558487.eps"},{"id":95344221,"identity":"f5a39f9b-edbe-405c-961b-34010346f2bc","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"cls","order_by":52,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":55857,"visible":true,"origin":"","legend":"","description":"","filename":"snjnl.cls","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/59536ca7c01361c81649a42f.cls"},{"id":95344214,"identity":"582faa3a-7d92-4c66-886b-5cbbd200157a","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":53,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1082105,"visible":true,"origin":"","legend":"","description":"","filename":"v20.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/706a12b343bfaaf88a7e036c.png"},{"id":95344249,"identity":"cfb1147b-3c8c-4932-8210-b81ce04ddb72","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"pdf","order_by":54,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":613165,"visible":true,"origin":"","legend":"","description":"","filename":"v2epsconvertedto.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/71380346ac895392459c1c34.pdf"},{"id":95344207,"identity":"4b8f1425-ba87-4340-9f99-88b1b6026a96","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":56,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":839108,"visible":true,"origin":"","legend":"","description":"","filename":"OnlineV21.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/9b5871e10d109dc77d95c099.png"},{"id":95344239,"identity":"54b72aff-1e5e-433d-88d4-ff1849ed60d7","added_by":"auto","created_at":"2025-11-07 02:52:45","extension":"png","order_by":57,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":171561,"visible":true,"origin":"","legend":"","description":"","filename":"OnlineV22.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/706b278b26ec8f05039d657b.png"},{"id":95344235,"identity":"94645579-e0e1-45a2-aaf9-03088255a856","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":63,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":74168,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefig3a.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/4b921ed143f6ebeb11d9d05d.png"},{"id":95344233,"identity":"04d4bef3-0278-48c3-8c5d-14ceda5b23f4","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"png","order_by":64,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":162572,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefig3b.png","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/a12de97021f67a4f34eaa0a6.png"},{"id":95524806,"identity":"61975383-0d70-4d12-b8d2-b531133f9940","added_by":"auto","created_at":"2025-11-10 10:03:35","extension":"xml","order_by":71,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":167319,"visible":true,"origin":"","legend":"","description":"","filename":"338e50b305b245dc8cf17e3e88f303911structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/9b9b5c5fd96edb104259e216.xml"},{"id":95344223,"identity":"166144b5-0ee3-430d-ab4c-f14f7beddaa0","added_by":"auto","created_at":"2025-11-07 02:52:44","extension":"html","order_by":72,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":190123,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1/2ee635537ab8d6691dbf3d39.html"},{"id":97107570,"identity":"fb8f0c4d-c24a-468f-9bf3-be91eeee0d43","added_by":"auto","created_at":"2025-12-01 05:23:54","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4292534,"visible":true,"origin":"","legend":"","description":"","filename":"SecuDevSLM.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7352742/v1_covered_8f08577e-b0cd-4f9c-acb8-1926340cbdb7.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"SecuDevSLM: A Systematic Security Evaluation Framework for Small Language Models on Mobile Devices","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Small Language Models, On-Device Security, Hallucination Attacks, Jailbreak Attacks","lastPublishedDoi":"10.21203/rs.3.rs-7352742/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7352742/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe deployment of Small Language Models (SLMs) on edge devices introduces critical security challenges, as their constrained architectures and heterogeneous runtime environments exacerbate vulnerabilities like hallucination and jailbreaking\u0026mdash;risks that remain understudied in resource-limited settings. To address this, we present SecuDevSLM, a fully on-device security evaluation framework for iOS and Android. Unlike traditional API-dependent approaches, SecuDevSLM eliminates dependence on external LLM APIs by leveraging a unified adversarial corpus for attack simulation, and introduces an integrated detection mechanism combining semantic noise entropy analysis and dual-modal evaluation, achieving 9% higher F1-score in threat identification compared to conventional tools. It systematically tests SLM robustness across 15 threat categories using 27 adversarial templates, generating multi-turn hallucination, jailbreak, and noise-based attacks. In total, 58 mainstream SLMs undergo tens of thousands of attack rounds. Compared to API-based methods, SecuDevSLM avoids high interaction costs and mitigates data leakage risks. Our experiments show that for sem antically aligned attacks, training corpus richness positively correlates with model robustness, though with diminishing returns. In weakly aligned attacks, the overall capacity of the model plays a larger role. Platform differences also impact stability: under hallucination, Android exhibits 23% greater character-level variance and 1.7 \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\((\\times)\\)\u003c/span\u003e\u003c/span\u003e higher fluctuation in response to text length than iOS. However, in long-text hallucinations, both perform similarly. For jailbreak attacks, Android shows 56% more output volatility, and when corpus richness is low, its attack success rate reaches 85%, versus 10% on iOS. Finally, we confirm that static indicators, such as parameter count and dataset features, are strongly linearly correlated with real-world performance, enabling developers better measure the security and performance of models before they are connected to the system.\u003c/p\u003e","manuscriptTitle":"SecuDevSLM: A Systematic Security Evaluation Framework for Small Language Models on Mobile Devices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-07 02:52:37","doi":"10.21203/rs.3.rs-7352742/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6f25ccaf-3726-485e-8efb-c24fe9548c2f","owner":[],"postedDate":"November 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-12-01T05:23:35+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-07 02:52:37","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7352742","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7352742","identity":"rs-7352742","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}