Automating Code Generation for a New Ecosystem: Establishing Baselines with Large Language Model Based Code Generation for ArkTS and HarmonyOS

Automating Code Generation for a New Ecosystem: Establishing Baselines with Large Language Model Based Code Generation for ArkTS and HarmonyOS | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automating Code Generation for a New Ecosystem: Establishing Baselines with Large Language Model Based Code Generation for ArkTS and HarmonyOS Mehmet Cem Aytekin, Fatma Gizem Calli, Mustafa Umut Demirezen This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7362986/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 27 Feb, 2026 Read the published version in Automated Software Engineering → Version 1 posted 11 You are reading this latest preprint version Abstract Automated code generation has advanced significantly with large language models (LLMs), yet their performance in emerging domain-specific languages for practical software development remains largely untested. This gap is particularly critical for major, rapidly growing platforms like Huawei's HarmonyOS. Powering over 900 million devices, the HarmonyOS ecosystem is shifting to a native-only application environment with the HarmonyOS NEXT update. This shift makes development in its native ArkTS UI language essential, driving an urgent need for effective developer tools. Addressing this industry challenge, we conduct the first systematic evaluation of Large Language Models (LLMs) on their ability to generate valid ArkTS code to accelerate the software development lifecycle and assist developers in mastering this new language. To ground our evaluation in practical scenarios, we introduce two curated datasets: a test dataset, \textit{ArkTS-Test}, and a training dataset for fine-tuning, both derived from common UI development tasks in ArkTS. Using these datasets, we evaluate a diverse range of LLMs—from large-scale proprietary models (e.g., Claude, Gemini, DeepSeek-V3, Qwen3 Coder, from 70b to up to 671B total parameters) to smaller open-source models (7B–14B) and report our observations. Next, we propose a methodology called Iterative Compilation Feedback (ICF), which enables LLMs to autonomously correct their own code by leveraging compiler error messages. Our experiments show that ICF boosts the syntactic accuracy of large-scale LLMs to as high as 91%. Furthermore, we show that fine-tuning a small-scale LLM (GPT-4o-mini) and combining it with our ICF method yields results comparable to the best-performing large-scale models. Finally, we conclude with a detailed categorization of compilation errors, identifying which types our ICF method resolves most effectively and which persist due to model knowledge limitations. ArkTS HarmonyOS Large Language Models (LLMs) Code Generation Compiler Feedback Low-Resource Languages Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 27 Feb, 2026 Read the published version in Automated Software Engineering → Version 1 posted Editorial decision: Revision requested 01 Dec, 2025 Reviews received at journal 01 Dec, 2025 Reviewers agreed at journal 14 Nov, 2025 Reviewers agreed at journal 13 Nov, 2025 Reviews received at journal 12 Oct, 2025 Reviewers agreed at journal 29 Aug, 2025 Reviewers agreed at journal 29 Aug, 2025 Reviewers invited by journal 28 Aug, 2025 Editor assigned by journal 19 Aug, 2025 Submission checks completed at journal 19 Aug, 2025 First submitted to journal 13 Aug, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7362986","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":507958559,"identity":"10b4b93e-871b-4ada-ae60-5f363e893573","order_by":0,"name":"Mehmet Cem Aytekin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKElEQVRIiWNgGAWjYFACHlSuARsD8zEwiw2vlgQYJwGkhS2NRC1AETO8ztJt7z264eMPBnl+sQOMn3l/2BjzSfd8e8xTc1iOj4H54aMbmFrMzpxLuzkjgcFw5uwEZmmehDQzNpmz2415jh02BrrQ2DgHi5YbOWa3eRIYEgxuJzAAtRy2YZPI3SbNw3Y7sY2Bh00am5b7b8xu/wFqsb+dwPybJ+E/UEvOM2mef7frcWq5wWN2mwFki3QCG9CWA2ZALWzSvG23E9hwaTmTY3azJ03CcAbQJZZz0pKN2STSzCTn9v03bGPG4ZfjZ8xu/LCxkeefnXz4xhsbO8P5M5KfSbz5liYv39788DEWLVAgAcSMDUyoKYEZp3IEYPxBhKJRMApGwSgYeQAAYCBdMivkzNYAAAAASUVORK5CYII=","orcid":"","institution":"Huawei Turkey R\u0026D Center","correspondingAuthor":true,"prefix":"","firstName":"Mehmet","middleName":"Cem","lastName":"Aytekin","suffix":""},{"id":507958562,"identity":"c980f0d7-edb7-4b84-b4f7-68f1bb2ba334","order_by":1,"name":"Fatma Gizem Calli","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Fatma","middleName":"Gizem","lastName":"Calli","suffix":""},{"id":507958563,"identity":"b33eb6f9-ad90-4bf2-b023-584b79f3aab9","order_by":2,"name":"Mustafa Umut Demirezen","email":"","orcid":"","institution":"Huawei Turkey R\u0026D Center","correspondingAuthor":false,"prefix":"","firstName":"Mustafa","middleName":"Umut","lastName":"Demirezen","suffix":""}],"badges":[],"createdAt":"2025-08-13 09:08:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7362986/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7362986/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10515-026-00599-9","type":"published","date":"2026-02-27T15:57:24+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":103766609,"identity":"b948193a-25a0-421d-9c4d-f8570c1cafef","added_by":"auto","created_at":"2026-03-02 16:15:12","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1826675,"visible":true,"origin":"","legend":"","description":"","filename":"ArkTSMehmetCem.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7362986/v1_covered_7c8502af-4031-4a98-80be-3f6a13743cbb.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Automating Code Generation for a New Ecosystem: Establishing Baselines with Large Language Model Based Code Generation for ArkTS and HarmonyOS","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"automated-software-engineering","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ause","sideBox":"Learn more about [Automated Software Engineering](http://link.springer.com/journal/10515)","snPcode":"10515","submissionUrl":"https://submission.nature.com/new-submission/10515/3","title":"Automated Software Engineering","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"ArkTS, HarmonyOS, Large Language Models (LLMs), Code Generation, Compiler Feedback, Low-Resource Languages","lastPublishedDoi":"10.21203/rs.3.rs-7362986/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7362986/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Automated code generation has advanced significantly with large language models (LLMs), yet their performance in emerging domain-specific languages for practical software development remains largely untested. This gap is particularly critical for major, rapidly growing platforms like Huawei's HarmonyOS. Powering over 900 million devices, the HarmonyOS ecosystem is shifting to a native-only application environment with the HarmonyOS NEXT update. This shift makes development in its native ArkTS UI language essential, driving an urgent need for effective developer tools. Addressing this industry challenge, we conduct the first systematic evaluation of Large Language Models (LLMs) on their ability to generate valid ArkTS code to accelerate the software development lifecycle and assist developers in mastering this new language. To ground our evaluation in practical scenarios, we introduce two curated datasets: a test dataset, \\textit{ArkTS-Test}, and a training dataset for fine-tuning, both derived from common UI development tasks in ArkTS. Using these datasets, we evaluate a diverse range of LLMs—from large-scale proprietary models (e.g., Claude, Gemini, DeepSeek-V3, Qwen3 Coder, from 70b to up to 671B total parameters) to smaller open-source models (7B–14B) and report our observations. Next, we propose a methodology called Iterative Compilation Feedback (ICF), which enables LLMs to autonomously correct their own code by leveraging compiler error messages. Our experiments show that ICF boosts the syntactic accuracy of large-scale LLMs to as high as 91\\%. Furthermore, we show that fine-tuning a small-scale LLM (GPT-4o-mini) and combining it with our ICF method yields results comparable to the best-performing large-scale models. Finally, we conclude with a detailed categorization of compilation errors, identifying which types our ICF method resolves most effectively and which persist due to model knowledge limitations.","manuscriptTitle":"Automating Code Generation for a New Ecosystem: Establishing Baselines with Large Language Model Based Code Generation for ArkTS and HarmonyOS","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-04 18:22:03","doi":"10.21203/rs.3.rs-7362986/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-12-02T04:30:35+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-12-02T00:22:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"180825311854989129645559038908156528023","date":"2025-11-14T11:09:57+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"69152423874705241483845656665634308604","date":"2025-11-13T17:06:30+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-12T12:16:41+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"218526842529540062810976793726748890179","date":"2025-08-29T20:51:52+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"81969209309590448437216392798493915543","date":"2025-08-29T05:30:48+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-08-29T03:25:51+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-08-19T17:20:46+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-08-19T17:20:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"Automated Software Engineering","date":"2025-08-13T08:54:44+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"automated-software-engineering","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ause","sideBox":"Learn more about [Automated Software Engineering](http://link.springer.com/journal/10515)","snPcode":"10515","submissionUrl":"https://submission.nature.com/new-submission/10515/3","title":"Automated Software Engineering","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"fa49c58d-562d-47d1-895a-4f31bb4f34f7","owner":[],"postedDate":"September 4th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2026-03-02T16:14:04+00:00","versionOfRecord":{"articleIdentity":"rs-7362986","link":"https://doi.org/10.1007/s10515-026-00599-9","journal":{"identity":"automated-software-engineering","isVorOnly":false,"title":"Automated Software Engineering"},"publishedOn":"2026-02-27 15:57:24","publishedOnDateReadable":"February 27th, 2026"},"versionCreatedAt":"2025-09-04 18:22:03","video":"","vorDoi":"10.1007/s10515-026-00599-9","vorDoiUrl":"https://doi.org/10.1007/s10515-026-00599-9","workflowStages":[]},"version":"v1","identity":"rs-7362986","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7362986","identity":"rs-7362986","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}