On the Performance of Large Language Models on Introductory Programming Assignments

On the Performance of Large Language Models on Introductory Programming Assignments | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article On the Performance of Large Language Models on Introductory Programming Assignments Nishat Raihan, Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5348871/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Recent advances in artificial intelligence (AI), machine learning (ML), and natural language processing (NLP) have led to the development of a new generation of Large Language Models (LLMs) trained on massive amounts of data. Commercial applications (e.g., ChatGPT) have made this available to the general public, enabling the use of LLMs to produce high-quality texts for academic and professional purposes. Educational institutions are increasingly aware of stu-dents’ use of AI-generated content and are researching its impact and potential misuse. Computer Science (CS) and related fields are particularly affected, as LLMs can also generate programming code in various languages. To understand the potential impact of publicly available LLMs in CS education, we extend our previously introduced CSEPrompts [1], a framework comprising hundreds of programming exercise prompts and multiple-choice questions from introductory CS and programming courses. We provide experimental results on CSEPrompts, evaluating the performance of several LLMs in generating Python code and answering basic computer science and programming questions, offering insights into the implications of this technology for CS education. Benchmark Dataset Code LLM Prompting Full Text Additional Declarations No competing interests reported. Supplementary Files CSEPromptsmain.zip Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 11 Feb, 2025 Reviews received at journal 26 Jan, 2025 Reviews received at journal 09 Jan, 2025 Reviewers agreed at journal 12 Dec, 2024 Reviewers agreed at journal 11 Dec, 2024 Reviewers invited by journal 10 Dec, 2024 Editor assigned by journal 01 Nov, 2024 Submission checks completed at journal 01 Nov, 2024 First submitted to journal 28 Oct, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5348871","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":372938657,"identity":"4353757f-f707-4361-87f2-e6508389a038","order_by":0,"name":"Nishat Raihan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAv0lEQVRIiWNgGAWjYJCCw0Asxw9hMxOvxViygRQtIGWJGw4Qq0V39uGHhwtq7jBuPr86TYKhwjqxgZAWs3NpBodnHHvGbHbj7TYJhjPpRGg5w2BwmIftMJvZjbPbJBjbDhOjhf3DYZ5/h3mMZ4C0/CNKC4/BYd62wxIG/L1ALQ3EaSk4zNt32EDiBu9mi4Rj6cbEOGzzZ55vh+v7+89uvPGhxlqWoBYEkEhgYEggXjkI8B8gTf0oGAWjYBSMHAAAfLdDzlfuA/oAAAAASUVORK5CYII=","orcid":"","institution":"George Mason University","correspondingAuthor":true,"prefix":"","firstName":"Nishat","middleName":"","lastName":"Raihan","suffix":""},{"id":372938658,"identity":"289cbeb9-c0ce-42eb-bf1a-aca0e0c97f29","order_by":1,"name":"Dhiman Goswami","email":"","orcid":"","institution":"George Mason University","correspondingAuthor":false,"prefix":"","firstName":"Dhiman","middleName":"","lastName":"Goswami","suffix":""},{"id":372938659,"identity":"6e508d78-34c8-4d18-91b7-7843125dc5ec","order_by":2,"name":"Sadiya Sayara Chowdhury Puspo","email":"","orcid":"","institution":"George Mason University","correspondingAuthor":false,"prefix":"","firstName":"Sadiya","middleName":"Sayara Chowdhury","lastName":"Puspo","suffix":""},{"id":372938660,"identity":"c297c8d9-6a50-403b-bc19-b5616cf2095d","order_by":3,"name":"Mohammed Latif Siddiq","email":"","orcid":"","institution":"University of Notre Dame","correspondingAuthor":false,"prefix":"","firstName":"Mohammed","middleName":"Latif","lastName":"Siddiq","suffix":""},{"id":372938661,"identity":"16eacd4e-adba-424c-aacd-0c141bb0553e","order_by":4,"name":"Christian Newman","email":"","orcid":"","institution":"Rochester Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Christian","middleName":"","lastName":"Newman","suffix":""},{"id":372938662,"identity":"515202dd-8ecd-48f2-b44f-c89c3be3e079","order_by":5,"name":"Tharindu Ranasinghe","email":"","orcid":"","institution":"Lancaster University","correspondingAuthor":false,"prefix":"","firstName":"Tharindu","middleName":"","lastName":"Ranasinghe","suffix":""},{"id":372938663,"identity":"6310ec28-4224-439e-a4d2-6b7d933c897e","order_by":6,"name":"Joanna C.S. Santos","email":"","orcid":"","institution":"University of Notre Dame","correspondingAuthor":false,"prefix":"","firstName":"Joanna","middleName":"C.S.","lastName":"Santos","suffix":""},{"id":372938664,"identity":"f18466e9-3ef6-4590-b5cc-d01d95608fb4","order_by":7,"name":"Marcos Zampieri","email":"","orcid":"","institution":"George Mason University","correspondingAuthor":false,"prefix":"","firstName":"Marcos","middleName":"","lastName":"Zampieri","suffix":""}],"badges":[],"createdAt":"2024-10-28 17:08:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5348871/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5348871/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":68875496,"identity":"15cdd548-2052-49dd-947a-7cb453e49c67","added_by":"auto","created_at":"2024-11-13 04:37:34","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":341726,"visible":true,"origin":"","legend":"","description":"","filename":"CSEPrompts20JIIS.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5348871/v1_covered_29ae2894-e73d-4b82-8bc6-c66be5e0fa7d.pdf"},{"id":68875126,"identity":"d7cc1e21-a531-4886-be4a-c456964e6c16","added_by":"auto","created_at":"2024-11-13 04:29:44","extension":"zip","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":387805,"visible":true,"origin":"","legend":"","description":"","filename":"CSEPromptsmain.zip","url":"https://assets-eu.researchsquare.com/files/rs-5348871/v1/6adb5583d24d90a40b35fabd.zip"}],"financialInterests":"No competing interests reported.","formattedTitle":"On the Performance of Large Language Models on Introductory Programming Assignments","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jiis","sideBox":"Learn more about [Journal of Intelligent Information Systems](http://link.springer.com/journal/10844)","snPcode":"10844","submissionUrl":"https://submission.nature.com/new-submission/10844/3","title":"Journal of Intelligent Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Benchmark Dataset, Code LLM, Prompting","lastPublishedDoi":"10.21203/rs.3.rs-5348871/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5348871/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recent advances in artificial intelligence (AI), machine learning (ML), and natural language processing (NLP) have led to the development of a new generation of Large Language Models (LLMs) trained on massive amounts of data. Commercial applications (e.g., ChatGPT) have made this available to the general public, enabling the use of LLMs to produce high-quality texts for academic and professional purposes. Educational institutions are increasingly aware of stu-dents’ use of AI-generated content and are researching its impact and potential misuse. Computer Science (CS) and related fields are particularly affected, as LLMs can also generate programming code in various languages. To understand the potential impact of publicly available LLMs in CS education, we extend our previously introduced CSEPrompts [1], a framework comprising hundreds of programming exercise prompts and multiple-choice questions from introductory CS and programming courses. We provide experimental results on CSEPrompts, evaluating the performance of several LLMs in generating Python code and answering basic computer science and programming questions, offering insights into the implications of this technology for CS education.","manuscriptTitle":"On the Performance of Large Language Models on Introductory Programming Assignments","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-13 04:29:28","doi":"10.21203/rs.3.rs-5348871/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-02-11T14:18:23+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-01-26T21:06:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-01-09T21:31:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"262357857500241991938190792695512188539","date":"2024-12-12T10:47:24+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"24337835407794898980382129346535185769","date":"2024-12-11T20:33:39+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-12-10T10:36:10+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-11-01T10:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-11-01T09:43:22+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Intelligent Information Systems","date":"2024-10-28T17:05:36+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-intelligent-information-systems","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jiis","sideBox":"Learn more about [Journal of Intelligent Information Systems](http://link.springer.com/journal/10844)","snPcode":"10844","submissionUrl":"https://submission.nature.com/new-submission/10844/3","title":"Journal of Intelligent Information Systems","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"3f8aaa95-0316-4631-a85c-a35020a6ce5e","owner":[],"postedDate":"November 13th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-07-10T11:08:39+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-13 04:29:28","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5348871","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5348871","identity":"rs-5348871","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}