Accelerating Small Language Model via Quantization: A GPT-4 Guided Approach for Low-Resource Story Completion

Accelerating Small Language Model via Quantization: A GPT-4 Guided Approach for Low-Resource Story Completion | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Accelerating Small Language Model via Quantization: A GPT-4 Guided Approach for Low-Resource Story Completion Rakshit Dabral This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7914593/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 28 Oct, 2025 Read the published version in INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT → Version 1 posted You are reading this latest preprint version Abstract This paper introduces Story Completer, a sophisticated and efficient engine for real-time children's story completion, extending the foundational work of the TinyStories project. While TinyStories demonstrated that small models (< 10M parameters) can generate coherent narratives on simplified data, our work addresses the challenge of deploying high-quality, context-aware generative models on resource-constrained hardware. The core innovation is a hybrid architecture that synergizes the rich semantic knowledge of a large language model with the computational efficiency of a smaller one. We integrate pre-computed GPT-4 text-embedding-ada-002 vectors within a compact, 12-million-parameter decoder-only transformer, effectively distilling the contextual understanding of a massive model into a lightweight system. Our methodology involved training this custom model from scratch using a meticulous strategy designed to adapt the model specifically for storytelling. A key contribution of this project is the optimization of the trained model for practical deployment on consumer-grade hardware, including low-end PCs and CPUs. After initial training, we applied post-training quantization , converting the model's weights from 16-bit floating-point precision (FP16) to 8-bit unsigned integers (uint8). This optimization yielded significant performance gains without a noticeable degradation in narrative quality. Comparative analysis between the normal and quantized models demonstrates the effectiveness of this approach. The quantization process reduced the model size by 49.5% , from 1546.04 MB to 780.03 MB. Furthermore, it achieved a 55.4% speedup in average inference time , decreasing from 21.701 seconds to 9.671 seconds. This project provides strong evidence for an efficient paradigm in model design, where the distilled intelligence of larger models, combined with optimization techniques like quantization, can be leveraged to create smaller, faster, and highly capable specialized systems. Theoretical Computer Science Artificial Intelligence and Machine Learning GPT-4 Embeddings Model Compression Quantization Real-Time Text Generation Small Language Model Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Published Journal Publication published 28 Oct, 2025 Read the published version in INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7914593","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":532861775,"identity":"fc00028c-9087-4164-bb57-d42e180b3fa9","order_by":0,"name":"Rakshit Dabral","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAElEQVRIiWNgGAWjYFACNgZmBgMGxgYJIPuBgQ2QZGw8QIQWA4iWhII0kJYGIrQwwLR8OAwWw6vFvP1Y8ueCgj+y/bN7H35IMDhvt7b9MNCWGptoXFpkzqQdk55hYGA8485xY4kEg9vJ284kArUcS8ttwKFFgiG9jZnHwCCx4UYaA1iL2QGgFsaGw7i18D9v/gzSMv9GGvOPBINzyWbnHxLQIpF2QBqkZcONNDagLQfszG4QskXiWRpQi7HxxjvH2CwSDJITzG4AbUnA5xf+NOPPPH/kZOfdbmO+8eGPnb3Z+fSHDz7U2ODUggESwSoTiFUOAvakKB4Fo2AUjIKRAQCthmMlg8lIWwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0009-0004-2940-1542","institution":"Dr. Akhilesh Das Gupta Institute of Professional Studies","correspondingAuthor":true,"prefix":"","firstName":"Rakshit","middleName":"","lastName":"Dabral","suffix":""}],"badges":[],"createdAt":"2025-10-21 12:50:32","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-7914593/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7914593/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.55041/IJSREM53280","type":"published","date":"2025-10-29T00:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":94059337,"identity":"4b45f8d8-4dc3-4dcd-a6ea-9c14f23e417f","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":115193,"visible":true,"origin":"","legend":"","description":"","filename":"researchpaper.docx","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/e8a2ab553972a666448e44d3.docx"},{"id":94059338,"identity":"4ec09d5c-07a9-4530-bd63-b09218de6f15","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":342,"visible":true,"origin":"","legend":"","description":"","filename":"rs7914593.json","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/9dcb30109538d5bf2f241b63.json"},{"id":94059335,"identity":"ef1e304d-d83a-400d-8fb1-ef8cd6da15c8","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"xml","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39014,"visible":true,"origin":"","legend":"","description":"","filename":"rs79145930enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/aa47a8ff6b3e0320dcd51c42.xml"},{"id":94060102,"identity":"476d2082-3016-443f-bb1e-8653264dcfac","added_by":"auto","created_at":"2025-10-22 06:15:51","extension":"png","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":22645,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/464365c6d042eebfa8719c8b.png"},{"id":94059340,"identity":"725b0965-1b5b-487d-83fb-5f9ac71a7a74","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"png","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":23433,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/fef6cfc7bd28ed0755307413.png"},{"id":94060410,"identity":"0cca8543-9fb4-4106-a7ae-808d2efdb4b1","added_by":"auto","created_at":"2025-10-22 06:23:51","extension":"png","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17661,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/39abd3cd459f4401677acdfd.png"},{"id":94059339,"identity":"71e2d0b9-323b-4c2d-bca6-39be37ea5a5e","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"png","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":16420,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/51d91cac97f2765117001c13.png"},{"id":94060411,"identity":"9495b604-2283-4136-b1c5-ebecfc8f6efe","added_by":"auto","created_at":"2025-10-22 06:23:51","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7489,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/77c28b9bad590f3f067b4cdf.png"},{"id":94059341,"identity":"cf648947-f026-4e4d-b049-d67b05d15457","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6922,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/1e5f34f073d1d5f0ac047f25.png"},{"id":94059344,"identity":"efded320-f304-45fd-ae41-1920d520220d","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5758,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/d99af7383302fc75bf2bf52e.png"},{"id":94059343,"identity":"fe48f986-24af-45fc-bff9-32766506b169","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5468,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/5acc7c1e411357c1fc6b9d63.png"},{"id":94060104,"identity":"6295c649-9711-40ae-9711-a7b876b99b01","added_by":"auto","created_at":"2025-10-22 06:15:51","extension":"xml","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":37858,"visible":true,"origin":"","legend":"","description":"","filename":"rs79145930structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/a371743119fdfb4a0a967fdf.xml"},{"id":94059348,"identity":"2c7cd30e-c78b-4193-a87a-516e33b29c76","added_by":"auto","created_at":"2025-10-22 06:07:51","extension":"html","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":46070,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1/f7ff2887e9731c79558a9100.html"},{"id":106813486,"identity":"007722f7-540e-45f2-8e26-71cd6e722c5d","added_by":"auto","created_at":"2026-04-13 16:48:12","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":448305,"visible":true,"origin":"","legend":"","description":"","filename":"researchpaper.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7914593/v1_covered_c7916d0f-40ef-4d42-8730-6034fc25fe98.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003ch1\u003e\u003cstrong\u003eAccelerating Small Language Model via Quantization:\u003c/strong\u003e\u003c/h1\u003e\n\u003cp\u003eA GPT-4 Guided Approach for Low-Resource Story Completion\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"Dr. Akhilesh Das Gupta Institute of Professional Studies","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"GPT-4 Embeddings, Model Compression, Quantization , Real-Time Text Generation, Small Language Model","lastPublishedDoi":"10.21203/rs.3.rs-7914593/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7914593/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper introduces Story Completer, a sophisticated and efficient engine for real-time children's story completion, extending the foundational work of the TinyStories project. While TinyStories demonstrated that small models (\u0026lt;\u0026thinsp;10M parameters) can generate coherent narratives on simplified data, our work addresses the challenge of deploying high-quality, context-aware generative models on resource-constrained hardware. The core innovation is a hybrid architecture that synergizes the rich semantic knowledge of a large language model with the computational efficiency of a smaller one. We integrate pre-computed GPT-4 text-embedding-ada-002 vectors within a compact, 12-million-parameter decoder-only transformer, effectively distilling the contextual understanding of a massive model into a lightweight system. Our methodology involved training this custom model from scratch using a meticulous strategy designed to adapt the model specifically for storytelling.\u003c/p\u003e\u003cp\u003eA key contribution of this project is the optimization of the trained model for practical deployment on consumer-grade hardware, including low-end PCs and CPUs. After initial training, we applied \u003cb\u003epost-training quantization\u003c/b\u003e, converting the model's weights from 16-bit floating-point precision (FP16) to 8-bit unsigned integers (uint8). This optimization yielded significant performance gains without a noticeable degradation in narrative quality.\u003c/p\u003e\u003cp\u003eComparative analysis between the normal and quantized models demonstrates the effectiveness of this approach. The quantization process reduced the \u003cb\u003emodel size by 49.5%\u003c/b\u003e, from 1546.04 MB to 780.03 MB. Furthermore, it achieved a \u003cb\u003e55.4% speedup in average inference time\u003c/b\u003e, decreasing from 21.701 seconds to 9.671 seconds. This project provides strong evidence for an efficient paradigm in model design, where the distilled intelligence of larger models, combined with optimization techniques like quantization, can be leveraged to create smaller, faster, and highly capable specialized systems.\u003c/p\u003e","manuscriptTitle":"Accelerating Small Language Model via Quantization:\nA GPT-4 Guided Approach for Low-Resource Story Completion","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-22 06:07:46","doi":"10.21203/rs.3.rs-7914593/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"68901189-b457-4877-9469-6b963532ede4","owner":[],"postedDate":"October 22nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":56645408,"name":"Theoretical Computer Science"},{"id":56645409,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2026-04-13T16:48:08+00:00","versionOfRecord":{"articleIdentity":"rs-7914593","link":"https://doi.org/10.55041/IJSREM53280","journal":{"identity":"international-journal-of-scientific-research-in-engineering-and-management","isVorOnly":true,"title":"INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT"},"publishedOn":"2025-10-29 00:00:00","publishedOnDateReadable":"October 29th, 2025"},"versionCreatedAt":"2025-10-22 06:07:46","video":"","vorDoi":"10.55041/IJSREM53280","vorDoiUrl":"https://doi.org/10.55041/IJSREM53280","workflowStages":[]},"version":"v1","identity":"rs-7914593","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7914593","identity":"rs-7914593","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}