Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer

Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer Suji Jose, Philip Samuel, Sumam Mary Idicula This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8405699/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Automated front-end Code Generation (CG) is required due to the rising demand for fast and error-free development of mobile applications. Existing research has overlooked semantic context-aware automated front-end code generation for mobile applications, resulting in less reliable generated interfaces. Thus, this paper proposes a novel Visual Top-k Attention Bidirectional Encoder Representations from Transformers (VisualTABERT) and Code Longformer Attention Text-To-Text Transfer Transformer (CodeLAT5+) enabled semantic context-aware automated front-end CG system. First, the mobile application screenshots are collected and preprocessed. The UI components are then segmented using a Spatial Pyramid Pooling–enhanced YOLOv8 (SPP-YOLOv8) model, followed by text extraction and attribute identification for each detected element. Similar UI elements are grouped based on the extracted text attributes and segmented UI elements employing Density Survival Function Based Spatial Clustering of Applications with Noise (DSFBSCAN). Then using a VisualTABERT based module, the semantic relationships between UI elements and their associated textual content are established. Consequently, by using Affine Polynomial Kernel Transformation (APKT), the coordinate alignment is done for the UI elements. According to the grouped similar UI elements, semantic relationships, and coordinate alignment, the structured intermediate representation is performed via Schema Mapping (SM). Finally, CodeLAT5+ generates the target source code by utilizing the structured intermediate representation obtained from the preceding transformation process. As per the outcome, the proposed model achieved a higher Mean Reciprocal Rank (MRR) (0.9256) than the conventional methods. Automated Front End Code Generation Mobile Applications Semantic Context-aware extraction User Interface (UI) elements Co-ordinate alignment Structured Intermediate Representation Optical Character Recognition (OCR) and Spatial Pyramid Pooling based You Only Look Once version 8 (SPPYOLO-v8) Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8405699","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":571050606,"identity":"ec51033a-9ef2-409a-ad42-0b61ce8de7dc","order_by":0,"name":"Suji Jose","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3ElEQVRIiWNgGAWjYHCChAMSQJIfIcDYQJwWSbC6BOK0QIDBAbgWAkB+RsLDAxYVdXLGx88Yf/j5g0Gev4G57QFew28kAB125rCx2ZkcM8meBAbDGQcY2w3wapEAapFsO5C47QaPGQNPAgPjBgbGNgkCDgNpqavfPIPH+OOfBAZ7gloYboC1MCcYSPAYSANtSSSoxeDMA7BfDGecSSuTlkmTSJ5xmJDD2nOSP0tU1Mnztx/e/PGNjY1tf3v7M/wOA3qZGUkFkMmMXz0QsB9g/EBQ0SgYBaNgFIxoAABVPke7Pw+InAAAAABJRU5ErkJggg==","orcid":"","institution":"Cochin University of Science and Technology","correspondingAuthor":true,"prefix":"","firstName":"Suji","middleName":"","lastName":"Jose","suffix":""},{"id":571050607,"identity":"e1da9c71-72dd-45ad-a0c3-318db34c85d8","order_by":1,"name":"Philip Samuel","email":"","orcid":"","institution":"Cochin University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Philip","middleName":"","lastName":"Samuel","suffix":""},{"id":571050611,"identity":"7e98abe7-a1cf-4ff9-8835-5eb001cd4f38","order_by":2,"name":"Sumam Mary Idicula","email":"","orcid":"","institution":"Cochin University of Science and Technology","correspondingAuthor":false,"prefix":"","firstName":"Sumam","middleName":"Mary","lastName":"Idicula","suffix":""}],"badges":[],"createdAt":"2025-12-19 14:23:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8405699/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8405699/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":99858342,"identity":"47d976f1-791a-4802-a763-3f193f608310","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5203,"visible":true,"origin":"","legend":"","description":"","filename":"624bb73753824a5985ff2358caf47de5.json","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/1ee13464e045564f7365a598.json"},{"id":100357260,"identity":"349a3565-9e25-4c40-8a26-e9e24b680a28","added_by":"auto","created_at":"2026-01-16 07:19:32","extension":"xml","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":71876,"visible":true,"origin":"","legend":"","description":"","filename":"624bb73753824a5985ff2358caf47de51enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/fcf35bb1495178a0f0dd5c5a.xml"},{"id":100357570,"identity":"585cd5d2-a877-4e42-9fda-bfadc9583820","added_by":"auto","created_at":"2026-01-16 07:20:04","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3927061,"visible":true,"origin":"","legend":"","description":"","filename":"ArticleTitle47.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/211bfbb30712c063e2d2441d.pdf"},{"id":100357265,"identity":"7764db44-d7e2-4af3-b5dd-b93078f2c108","added_by":"auto","created_at":"2026-01-16 07:19:33","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1060299,"visible":true,"origin":"","legend":"","description":"","filename":"Picture1.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/2048fb818600170a9243b0cb.png"},{"id":100357619,"identity":"90418c03-a27c-482f-a2f5-cebfebac6b23","added_by":"auto","created_at":"2026-01-16 07:20:06","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":594873,"visible":true,"origin":"","legend":"","description":"","filename":"Picture2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/b118e36eb68bdc5afb41287e.png"},{"id":99858343,"identity":"bcd9e556-d4e8-4224-98e7-458ea9a3885c","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"pdf","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":52282,"visible":true,"origin":"","legend":"","description":"","filename":"coverletter.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/f26cf83693f0cf0d3ce9e4b2.pdf"},{"id":99858346,"identity":"5d1a0821-e955-4808-a3c6-3833b9e3c91a","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"eps","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2890,"visible":true,"origin":"","legend":"","description":"","filename":"empty.eps","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/c3e0134326a8655a448fc4d5.eps"},{"id":100357781,"identity":"83b5a831-6242-4d3a-8577-51f66d2ec6b0","added_by":"auto","created_at":"2026-01-16 07:20:18","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":24351,"visible":true,"origin":"","legend":"","description":"","filename":"eqn2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/a8406ca10cd8a71d17700b1f.png"},{"id":99858349,"identity":"0531f37c-52f3-4127-ae50-627231020331","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":41166,"visible":true,"origin":"","legend":"","description":"","filename":"eqn4.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/3086a569fc5cb54c91df79c9.png"},{"id":100357242,"identity":"7b67daf6-e1c5-434e-b2ce-2ea55b567c22","added_by":"auto","created_at":"2026-01-16 07:19:30","extension":"eps","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91593,"visible":true,"origin":"","legend":"","description":"","filename":"fig.eps","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/3ce37d7ab6c3bded6541613d.eps"},{"id":100357503,"identity":"789d6191-3e30-4ea5-974c-1630392dc970","added_by":"auto","created_at":"2026-01-16 07:19:56","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":222595,"visible":true,"origin":"","legend":"","description":"","filename":"graph1.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/849d3338f69f998e9138d83a.png"},{"id":100357527,"identity":"29811375-1fda-4b70-a279-d819558c35b1","added_by":"auto","created_at":"2026-01-16 07:20:00","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":645391,"visible":true,"origin":"","legend":"","description":"","filename":"graph2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/5076ff4723dfe347efcc4032.png"},{"id":99858355,"identity":"a8cde787-c0cc-413a-b060-fbb34156ec0b","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":242194,"visible":true,"origin":"","legend":"","description":"","filename":"graph3.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/dd2ef84ace01fe1a5e0d5bca.png"},{"id":100357171,"identity":"56c5a4b2-7cdb-4555-baff-c4019b5f2edb","added_by":"auto","created_at":"2026-01-16 07:19:11","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":334855,"visible":true,"origin":"","legend":"","description":"","filename":"graph4.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/1693da3659e9a83f3ecc0fad.png"},{"id":100357632,"identity":"2c63a0c4-8b7c-4c2f-a738-8932023eb8f2","added_by":"auto","created_at":"2026-01-16 07:20:07","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":202981,"visible":true,"origin":"","legend":"","description":"","filename":"graph5.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/ee4a6ceb410858716ef0534a.png"},{"id":100357390,"identity":"9a9087ab-06e3-4e6e-8416-4d2beaf08d44","added_by":"auto","created_at":"2026-01-16 07:19:50","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":102543,"visible":true,"origin":"","legend":"","description":"","filename":"graph6.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/249c30a6f9cdd9aa5037ba8e.png"},{"id":99858358,"identity":"74d6ed95-6103-4f61-8ba4-42ee6af0f5a2","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":106254,"visible":true,"origin":"","legend":"","description":"","filename":"graph7.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/d97d80388cbfba504cb2782c.png"},{"id":100357247,"identity":"fccf3839-8414-4fe6-a410-9bca4083e0ca","added_by":"auto","created_at":"2026-01-16 07:19:31","extension":"bst","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":146013,"visible":true,"origin":"","legend":"","description":"","filename":"snapacite.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/c16e3a1b2c35dd406add8d6e.bst"},{"id":99858363,"identity":"59513fe8-bd89-4a82-9328-8c5d557e39ee","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"bst","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29828,"visible":true,"origin":"","legend":"","description":"","filename":"snaps.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/dfcb498bd9653813b8e9b318.bst"},{"id":99858382,"identity":"d967a6ce-a744-45a2-9a56-f65fb3f96f8f","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"pdf","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":421391,"visible":true,"origin":"","legend":"","description":"","filename":"snarticle.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/25ed186c9dc29691fbb40a2c.pdf"},{"id":99858353,"identity":"23d434fb-006e-4278-8176-3aec9ddc7eb6","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"bst","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":35515,"visible":true,"origin":"","legend":"","description":"","filename":"snbasic.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/bcae711db7764e7c921af8bf.bst"},{"id":99858383,"identity":"7bdac11f-be16-4ade-8206-4d93a5605f52","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"bst","order_by":21,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33968,"visible":true,"origin":"","legend":"","description":"","filename":"snchicago.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/58ffd0c60cddce3a8fa1e1a9.bst"},{"id":99858380,"identity":"65d32fa1-4a4a-4e77-a384-899db9996101","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"cls","order_by":22,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":55857,"visible":true,"origin":"","legend":"","description":"","filename":"snjnl.cls","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/9e188a009e7f1a8923d4a8b8.cls"},{"id":100357667,"identity":"244e8b7a-1b3e-4e9b-a80e-9a141f3e93eb","added_by":"auto","created_at":"2026-01-16 07:20:09","extension":"bst","order_by":23,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64023,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysay.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/4c5ded40603ec772f56f1759.bst"},{"id":100357762,"identity":"26fc5dc3-43a0-4890-bc20-e56f5d210242","added_by":"auto","created_at":"2026-01-16 07:20:18","extension":"bst","order_by":24,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64166,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysnum.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/04ec9003287bcb12ddc1f441.bst"},{"id":100357524,"identity":"9b7394f2-177f-4064-82b5-94ef6f564c8a","added_by":"auto","created_at":"2026-01-16 07:19:59","extension":"bst","order_by":25,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":37333,"visible":true,"origin":"","legend":"","description":"","filename":"snnature.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/850a5e84c8185a93e040e828.bst"},{"id":99858366,"identity":"cd6e0134-0b0a-4929-9135-e53edf77a8cd","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"bst","order_by":26,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39951,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouveray.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/a51f1b044b3cf54ff22d3ea0.bst"},{"id":99858359,"identity":"5b719c64-6030-4b0f-94b4-abd5691795bb","added_by":"auto","created_at":"2026-01-09 06:29:39","extension":"bst","order_by":27,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40758,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouvernum.bst","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/aacb95c5c0abd1fa99112b9b.bst"},{"id":99858391,"identity":"b5075c0b-412e-4858-923a-38e051a3d7e0","added_by":"auto","created_at":"2026-01-09 06:29:41","extension":"png","order_by":28,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5227,"visible":true,"origin":"","legend":"","description":"","filename":"tab1.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/5f1164161e07a972ff9c8323.png"},{"id":99858369,"identity":"73c6b836-7ac2-41c1-917c-7e655f041482","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":29,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7399,"visible":true,"origin":"","legend":"","description":"","filename":"tab2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/b2e4a3d4e20576e9a29de00b.png"},{"id":100357652,"identity":"45eb2385-c3bb-4d9f-bd9f-474d6aa55e02","added_by":"auto","created_at":"2026-01-16 07:20:09","extension":"png","order_by":30,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6639,"visible":true,"origin":"","legend":"","description":"","filename":"tab3.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/635b1645a88a5453508bd3b2.png"},{"id":100357164,"identity":"f7586985-e54d-4429-928b-13d7b1d15a09","added_by":"auto","created_at":"2026-01-16 07:19:10","extension":"png","order_by":31,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16237,"visible":true,"origin":"","legend":"","description":"","filename":"tab4.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/8b8fbe67150a1d82631c4fb8.png"},{"id":99858375,"identity":"62344518-d32d-4dbb-a6a6-7356d5ef4af7","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":32,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":37759,"visible":true,"origin":"","legend":"","description":"","filename":"tab5.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/261f42f4c8fad555d39cfc98.png"},{"id":99858387,"identity":"0647ebd8-0a7a-49ca-a0c4-98b989f671ac","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":33,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16549,"visible":true,"origin":"","legend":"","description":"","filename":"tab6.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/9d3285e0d186296810536494.png"},{"id":99858364,"identity":"abc5bbaa-8f78-477d-8bc2-93edb0fe3a95","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":34,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":66913,"visible":true,"origin":"","legend":"","description":"","filename":"tab7.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/1ac4c136d071113914ae3cee.png"},{"id":100357182,"identity":"b43d0774-2378-45fb-ae2f-5b22936b8e17","added_by":"auto","created_at":"2026-01-16 07:19:13","extension":"png","order_by":35,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":102753,"visible":true,"origin":"","legend":"","description":"","filename":"tab8.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/7e1b23a3bfc25ff9065c231f.png"},{"id":99858372,"identity":"6dc1834c-fa6b-40c6-9e55-257c6ea23e2f","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"pdf","order_by":36,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":418495,"visible":true,"origin":"","legend":"","description":"","filename":"usermanual.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/18cd4cb544b7af1841603ad2.pdf"},{"id":100358135,"identity":"2d99e79f-803f-4fdb-97d8-ed8c01899255","added_by":"auto","created_at":"2026-01-16 07:20:40","extension":"png","order_by":39,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":19218,"visible":true,"origin":"","legend":"","description":"","filename":"Onlineeqn2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/adc0a2021c638ab051f47f4e.png"},{"id":100357754,"identity":"bd3d455f-c862-49ce-bd6d-049215e0649e","added_by":"auto","created_at":"2026-01-16 07:20:17","extension":"png","order_by":40,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":26777,"visible":true,"origin":"","legend":"","description":"","filename":"Onlineeqn4.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/c498e00418b9ad7c7ea2df96.png"},{"id":99858368,"identity":"1fb350e0-7bc2-431a-bec5-e285094dd7a9","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":43,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":201086,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegraph3.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/b89d2a678994f258b03a860a.png"},{"id":99858378,"identity":"b37ab4ae-6901-4a20-8772-7430da5ee26b","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":45,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":168366,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegraph5.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/bdf832c17d552c275d7e0529.png"},{"id":99858384,"identity":"cf8d0b66-4e65-40a7-85ec-1fececab4ab2","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":47,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":76242,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinegraph7.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/03743446cfde63901d58dd1b.png"},{"id":100358125,"identity":"9048b73e-71be-4030-a3ea-f77fa7d284cb","added_by":"auto","created_at":"2026-01-16 07:20:39","extension":"png","order_by":48,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":3320,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab1.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/c236b3bcf48d6fd79296647f.png"},{"id":99858388,"identity":"e279dd84-734f-4637-819f-29e117e2ffe3","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":49,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5694,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab2.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/07887da19a78974bfb790d9d.png"},{"id":99858371,"identity":"0a83b8d4-b286-44e1-8194-4a870ec15ea8","added_by":"auto","created_at":"2026-01-09 06:29:40","extension":"png","order_by":50,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":4892,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab3.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/5dcc85c3b2ad78363b98a601.png"},{"id":100357158,"identity":"d5e283e1-061f-4f6d-990f-a50273679194","added_by":"auto","created_at":"2026-01-16 07:19:08","extension":"png","order_by":51,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":12064,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab4.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/8329976140bbda3b38400754.png"},{"id":100357442,"identity":"d522d0bd-4a30-4477-b929-564e256ebefa","added_by":"auto","created_at":"2026-01-16 07:19:52","extension":"png","order_by":52,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":25852,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab5.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/b71d0654f61ed2f52630ae77.png"},{"id":100357971,"identity":"2cbefb13-1c73-4d89-bcb6-2c0a47ddf336","added_by":"auto","created_at":"2026-01-16 07:20:32","extension":"png","order_by":53,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10180,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab6.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/20b02a98c8406f0722484dbd.png"},{"id":100357355,"identity":"26f53e82-f2b7-4c91-b542-92ba2f85625b","added_by":"auto","created_at":"2026-01-16 07:19:45","extension":"png","order_by":54,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49480,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab7.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/83c40d83f9f467e75795bc57.png"},{"id":100357999,"identity":"1650e3b2-640c-4f2a-ae4f-e1aa5cf36e6e","added_by":"auto","created_at":"2026-01-16 07:20:33","extension":"png","order_by":55,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":75822,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinetab8.png","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/750de99c036a9199fe2f4f2c.png"},{"id":100357285,"identity":"a94bcbe4-43aa-4f5e-a660-2315a36597d0","added_by":"auto","created_at":"2026-01-16 07:19:34","extension":"xml","order_by":56,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":81256,"visible":true,"origin":"","legend":"","description":"","filename":"624bb73753824a5985ff2358caf47de51structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/98f6637a9f7d5daa63a1a62b.xml"},{"id":100357684,"identity":"b1baca6f-9a6e-48e8-a6de-e52d34aaff28","added_by":"auto","created_at":"2026-01-16 07:20:10","extension":"html","order_by":57,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":95958,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1/b5c84af4584d59df651641ae.html"},{"id":104736668,"identity":"f1d34663-0a43-40fa-ae7a-0cecb68004ee","added_by":"auto","created_at":"2026-03-16 15:28:38","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1477966,"visible":true,"origin":"","legend":"","description":"","filename":"ArticleTitle47.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8405699/v1_covered_5c6c7c62-ae1a-472a-a9ef-9f2e340e335c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Automated Front End Code Generation, Mobile Applications, Semantic Context-aware extraction, User Interface (UI) elements, Co-ordinate alignment, Structured Intermediate Representation, Optical Character Recognition (OCR), and Spatial Pyramid Pooling based You Only Look Once version 8 (SPPYOLO-v8)","lastPublishedDoi":"10.21203/rs.3.rs-8405699/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8405699/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAutomated front-end Code Generation (CG) is required due to the rising demand for fast and error-free development of mobile applications. Existing research has overlooked semantic context-aware automated front-end code generation for mobile applications, resulting in less reliable generated interfaces. Thus, this paper proposes a novel Visual Top-k Attention Bidirectional Encoder Representations from Transformers (VisualTABERT) and Code Longformer Attention Text-To-Text Transfer Transformer (CodeLAT5+) enabled semantic context-aware automated front-end CG system. First, the mobile application screenshots are collected and preprocessed. The UI components are then segmented using a Spatial Pyramid Pooling\u0026ndash;enhanced YOLOv8 (SPP-YOLOv8) model, followed by text extraction and attribute identification for each detected element. Similar UI elements are grouped based on the extracted text attributes and segmented UI elements employing Density Survival Function Based Spatial Clustering of Applications with Noise (DSFBSCAN). Then using a VisualTABERT based module, the semantic relationships between UI elements and their associated textual content are established. Consequently, by using Affine Polynomial Kernel Transformation (APKT), the coordinate alignment is done for the UI elements. According to the grouped similar UI elements, semantic relationships, and coordinate alignment, the structured intermediate representation is performed via Schema Mapping (SM). Finally, CodeLAT5+ generates the target source code by utilizing the structured intermediate representation obtained from the preceding transformation process. As per the outcome, the proposed model achieved a higher Mean Reciprocal Rank (MRR) (0.9256) than the conventional methods.\u003c/p\u003e","manuscriptTitle":"Semantic Context-Aware Automated Front-End Code Generation for Mobile applications using a Vision Language Code Transformer","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-01-09 06:29:30","doi":"10.21203/rs.3.rs-8405699/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"600d3068-e452-40fe-8967-30bd57d8c072","owner":[],"postedDate":"January 9th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-16T15:27:55+00:00","versionOfRecord":[],"versionCreatedAt":"2026-01-09 06:29:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8405699","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8405699","identity":"rs-8405699","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}