Field-testing multiple-choice questions with AI examinees: English grammar items

Field-testing multiple-choice questions with AI examinees: English grammar items | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Field-testing multiple-choice questions with AI examinees: English grammar items Hotaka Maeda This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3858355/v2 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 02 Oct, 2024 Read the published version in Educational and Psychological Measurement → Version 2 posted You are reading this latest preprint version Show more versions Abstract Field-testing is a necessary but resource-intensive step in the development of high-quality educational assessments. I present an innovative method for field-testing newly written exam items by replacing human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ , and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root-mean-squared-error = 0.18). The simulation study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach still fell short of the accuracy of analyses that can be achieved with human examinee response data. If further refined, potential resource savings in transition from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1 Educational Psychology Artificial Intelligence and Machine Learning natural language processing pretesting artificial intelligence BERT question difficulty prediction Full Text Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Published Journal Publication published 02 Oct, 2024 Read the published version in Educational and Psychological Measurement → Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3858355","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267955938,"identity":"7f7b86b2-cd59-4a41-97be-d70b74442d13","order_by":0,"name":"Hotaka Maeda","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7klEQVRIiWNgGAWjYBACAyBmbICxPwBJNnaCWpgRWhhngLQwk6KFmQdEEdJizt5/8OOMGoZo/vbmjZ9tfm2T5wOa8eFjDm4tlj2HmSU3HGPInXHmWLF0bt9twzZmBmbJmdvwOOxGMoPkAzaG3IYbOQbSuT23GYFa2Jh58Wth/vngH0Pu/PtvjH9b9ty2J0YLm+TGNobcDTd4zKQZftxOJKzlzGEzy5l9Erkbz6SVWfY23E5uY2Zsxu+X442Pb/Z8s8mdd/zw5hs//ty2nd/efPDDRzxaoEACQjG2gckGguqRwB9SFI+CUTAKRsFIAQBu41Q7dJK/OAAAAABJRU5ErkJggg==","orcid":"","institution":"Smarter Balanced","correspondingAuthor":true,"prefix":"","firstName":"Hotaka","middleName":"","lastName":"Maeda","suffix":""}],"badges":[],"createdAt":"2024-01-12 22:59:10","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-3858355/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-3858355/v2","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1177/00131644241281053","type":"published","date":"2024-10-03T00:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":66205862,"identity":"655f0e10-63e7-4151-aa54-e62f9921e747","added_by":"auto","created_at":"2024-10-08 16:30:40","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1273867,"visible":true,"origin":"","legend":"","description":"","filename":"AIFieldtestingGrammarItems12preprintversion.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3858355/v2_covered_5366d35c-26a5-4940-8017-90b5058ac6cf.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eField-testing multiple-choice questions with AI examinees: English grammar items\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"natural language processing, pretesting, artificial intelligence, BERT, question difficulty prediction","lastPublishedDoi":"10.21203/rs.3.rs-3858355/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3858355/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eField-testing is a necessary but resource-intensive step in the development of high-quality educational assessments. I present an innovative method for field-testing newly written exam items by replacing human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability \u003cem\u003eθ\u003c/em\u003e, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root-mean-squared-error = 0.18). The simulation study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach still fell short of the accuracy of analyses that can be achieved with human examinee response data. If further refined, potential resource savings in transition from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: \u003ca href=\"https://github.com/hotakamaeda/ai_field_testing1\"\u003ehttps://github.com/hotakamaeda/ai_field_testing1\u003c/a\u003e\u003c/p\u003e","manuscriptTitle":"Field-testing multiple-choice questions with AI examinees: English grammar items","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2024-07-29 19:20:48","doi":"10.21203/rs.3.rs-3858355/v2","editorialEvents":[{"type":"communityComments","content":1}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2024-01-19 02:25:52","doi":"10.21203/rs.3.rs-3858355/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ef35b578-ab44-487c-88ab-d6f6aac0b22b","owner":[],"postedDate":"July 29th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":34033405,"name":"Educational Psychology"},{"id":34033406,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2024-10-08T16:30:29+00:00","versionOfRecord":{"articleIdentity":"rs-3858355","link":"https://doi.org/10.1177/00131644241281053","journal":{"identity":"educational-and-psychological-measurement","isVorOnly":true,"title":"Educational and Psychological Measurement"},"publishedOn":"2024-10-03 00:00:00","publishedOnDateReadable":"October 3rd, 2024"},"versionCreatedAt":"2024-07-29 19:20:48","video":"","vorDoi":"10.1177/00131644241281053","vorDoiUrl":"https://doi.org/10.1177/00131644241281053","workflowStages":[]},"version":"v2","identity":"rs-3858355","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3858355","identity":"rs-3858355","version":["v2"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}