A BLEU-Based Comparative Analysis of Human and ChatGPT 4.0 Translation in Kumpulan Lagu dan Cerita Anak- anak Dwibahasa

A BLEU-Based Comparative Analysis of Human and ChatGPT 4.0 Translation in Kumpulan Lagu dan Cerita Anak- anak Dwibahasa | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article A BLEU-Based Comparative Analysis of Human and ChatGPT 4.0 Translation in Kumpulan Lagu dan Cerita Anak- anak Dwibahasa Amon Bernabas Tenis, Adi Sytrisno This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9118811/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This study aims to compare the translation quality of human translators and ChatGPT 4.0 using the Bilingual Evaluation Understudy (BLEU) metric, focusing on twelve stories from Kumpulan Lagu dan Cerita Anak-Anak Dwibahasa. The purpose of this research is to examine how closely ChatGPT 4.0’s translations align with human translations in terms of lexical and structural similarity. The methodology includes four main stages: preparing human and machine translation outputs, performing tokenization, calculating n-gram precision, and computing the final BLEU scores based on geometric means and brevity penalties. The findings reveal that ChatGPT 4.0 consistently produced translations that were longer and more stylistically elaborated than the human references, resulting in BLEU scores ranging from 0.4859 to 0.9068. These results indicate that although ChatGPT 4.0 can generate fluent and contextually appropriate translations, its outputs do not closely match human translations at the n-gram level. The study concludes that BLEU remains effective for measuring surface-level similarity but is limited in capturing stylistic and interpretive aspects of AI-generated translation in children’s literature. Physical sciences/Engineering Physical sciences/Mathematics and computing BLEU metric Machine translation ChatGPT 4.0 Human translation Bilingual children’s literature Full Text Additional Declarations No competing interests reported. Supplementary Files Appendices.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9118811","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":609467636,"identity":"cb6269cd-fe1f-4e1e-8420-51cc68c68104","order_by":0,"name":"Amon Bernabas Tenis","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABY0lEQVRIie2Ry0rDQBRA7zCSIqR2OyVqfiFRCEIt/ZWEQLspUilIFl1EAsmyO4ko7S9UhFk3DGSVD4hMEaXgOlIE3RSTiCUWfO1EchYDc+ce5j4ASkr+KDg/ydtFBIyDu8QCECqrDGSTz5RpplQEU/WjVME/VABqoiZtuqs4FGvI389YOO9Zt0e1C2e+SKzZtuykChpReQuDUH/qglyz8WlcsHm7s+9H/T6ZhRqZRg+iwoR+45hy1U0VaYeC6k+Rc1D4kXc1qerqhh3rGgQuExWMr/g55ShX6hTQBJBbKEzOlaVujOPOIgmWTJQdUKTqJW+9K601RckVWzcmcVchgc1EYBt7aYQbmVJ/pGCsKSpvtyU/1I2rWXRCopBlvaRDDrnpYuQ0ECWmzz70ssvNUOoNdGPEvevEGrCWPGTpKge8Ofac4OaFHjaHnndfmNiXIBuL2Ubw96kF6fk32SUlJSX/lFckEIFwCCgGxQAAAABJRU5ErkJggg==","orcid":"","institution":"Gadjah Mada University","correspondingAuthor":true,"prefix":"","firstName":"Amon","middleName":"Bernabas","lastName":"Tenis","suffix":""},{"id":609467637,"identity":"7c109087-8a80-4f13-8fb6-9c1f4cdf1695","order_by":1,"name":"Adi Sytrisno","email":"","orcid":"","institution":"Gadjah Mada University","correspondingAuthor":false,"prefix":"","firstName":"Adi","middleName":"","lastName":"Sytrisno","suffix":""}],"badges":[],"createdAt":"2026-03-14 01:38:09","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9118811/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9118811/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109209559,"identity":"e8f85908-522d-4b5c-bbfa-e5421be3bf15","added_by":"auto","created_at":"2026-05-13 15:29:25","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":290155,"visible":true,"origin":"","legend":"","description":"","filename":"AnonymousArticleBLEU.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9118811/v1_covered_fef5136c-a2a0-4f9c-bff3-2a30bfef43ae.pdf"},{"id":105265850,"identity":"7dba7519-8e64-4eb3-b6ea-490268802878","added_by":"auto","created_at":"2026-03-24 07:27:38","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":73143,"visible":true,"origin":"","legend":"","description":"","filename":"Appendices.docx","url":"https://assets-eu.researchsquare.com/files/rs-9118811/v1/adee291d6219bf52b24d0902.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"A BLEU-Based Comparative Analysis of Human and ChatGPT 4.0 Translation in Kumpulan Lagu dan Cerita Anak- anak Dwibahasa","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"BLEU metric, Machine translation, ChatGPT 4.0, Human translation, Bilingual children’s literature","lastPublishedDoi":"10.21203/rs.3.rs-9118811/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9118811/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis study aims to compare the translation quality of human translators and ChatGPT 4.0 using the Bilingual Evaluation Understudy (BLEU) metric, focusing on twelve stories from Kumpulan Lagu dan Cerita Anak-Anak Dwibahasa. The purpose of this research is to examine how closely ChatGPT 4.0\u0026rsquo;s translations align with human translations in terms of lexical and structural similarity. The methodology includes four main stages: preparing human and machine translation outputs, performing tokenization, calculating n-gram precision, and computing the final BLEU scores based on geometric means and brevity penalties. The findings reveal that ChatGPT 4.0 consistently produced translations that were longer and more stylistically elaborated than the human references, resulting in BLEU scores ranging from 0.4859 to 0.9068. These results indicate that although ChatGPT 4.0 can generate fluent and contextually appropriate translations, its outputs do not closely match human translations at the n-gram level. The study concludes that BLEU remains effective for measuring surface-level similarity but is limited in capturing stylistic and interpretive aspects of AI-generated translation in children\u0026rsquo;s literature.\u003c/p\u003e","manuscriptTitle":"A BLEU-Based Comparative Analysis of Human and ChatGPT 4.0 Translation in Kumpulan Lagu dan Cerita Anak- anak Dwibahasa","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-24 07:23:51","doi":"10.21203/rs.3.rs-9118811/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"52f4d4e9-9033-44bd-a35a-765d4363eaa0","owner":[],"postedDate":"March 24th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":64854088,"name":"Physical sciences/Engineering"},{"id":64854089,"name":"Physical sciences/Mathematics and computing"}],"tags":[],"updatedAt":"2026-05-13T15:28:55+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-24 07:23:51","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9118811","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9118811","identity":"rs-9118811","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}