Passing the Turing Test: Fine-tuned AI feedback is less detectable than human or prompt-engineered feedback

Passing the Turing Test: Fine-tuned AI feedback is less detectable than human or prompt-engineered feedback | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Passing the Turing Test: Fine-tuned AI feedback is less detectable than human or prompt-engineered feedback Peter Ruijten-Dodoiu, Manuel Oliveira This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7486768/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract As large language models (LLMs) become increasingly integrated into educational technologies, questions arise about the authenticity and pedagogical value of AI-generated feedback. This study investigates whether human participants can distinguish between feedback written by a human instructor and that generated by an AI model, and how the method of generation (prompt engineering vs. fine-tuning) affects this perception. One hundred participants completed a Turing-test-inspired task in which they evaluated feedback texts and indicated whether they believed it to be written by a human or an AI model. The AI-generated feedback was produced using either prompt engineering or a fine-tuned LLaMA 3.1 model adapted with QLoRA. The results showed that participants correctly identified the prompt engineering feedback as nonhuman in most cases, while the fine-tuned feedback identification was indistinguishable from chance. These findings suggest that fine-tuning can significantly enhance the human-likeness of AI-generated feedback, with implications for the design of scalable, trustworthy feedback systems in education. The study contributes to ongoing discussions about the role of AI in supporting reflective learning and highlights the importance of transparency, trust, and pedagogical alignment in AI-mediated educational environments. Generative AI Assessment Feedback Fine-tuning Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7486768","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":508341625,"identity":"4f331d29-5eb7-4937-9467-e9437cebc6db","order_by":0,"name":"Peter Ruijten-Dodoiu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2UlEQVRIiWNgGAWjYDADNoYDQLICzGY8QKSWw0DyjAGYQ5wWBgZmoPFtRGgxZ2B+wHSD4V5iH+P5g48L5/1J7GfgPYBXi2UDmwFzDkNxYhvDYWbjmdsMEmc28CXg1WJwgAGkJQGkhU2aF6hlwwEeAwJa2D/AtLD/5p1jkLifsBYehC3MvA1AWxgIaLFs5ik4nGOQYAzUYizNc8zYeMZhAlrM2ds3Ps6pSJCdP+Pgw888NXKy/e09hg/wOowZFAug2JCAGc2MTz1IC5zF30BA6SgYBaNgFIxYAAAGLkWmt6dtyQAAAABJRU5ErkJggg==","orcid":"","institution":"Eindhoven University of Technolog","correspondingAuthor":true,"prefix":"","firstName":"Peter","middleName":"","lastName":"Ruijten-Dodoiu","suffix":""},{"id":508341626,"identity":"f2f20635-27d2-46fd-b1b7-c80bdb757046","order_by":1,"name":"Manuel Oliveira","email":"","orcid":"","institution":"Eindhoven University of Technolog","correspondingAuthor":false,"prefix":"","firstName":"Manuel","middleName":"","lastName":"Oliveira","suffix":""}],"badges":[],"createdAt":"2025-08-29 08:53:29","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7486768/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7486768/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":105562739,"identity":"9d46c08b-896f-4abd-83b1-d314c1225026","added_by":"auto","created_at":"2026-03-27 12:44:27","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1334250,"visible":true,"origin":"","legend":"","description":"","filename":"TuringTestinSpringerNatureLaTeXTemplate.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7486768/v1_covered_6a26819d-4d05-4bc2-a1b0-372122633965.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Passing the Turing Test: Fine-tuned AI feedback is less detectable than human or prompt-engineered feedback","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Generative AI, Assessment, Feedback, Fine-tuning","lastPublishedDoi":"10.21203/rs.3.rs-7486768/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7486768/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAs large language models (LLMs) become increasingly integrated into educational technologies, questions arise about the authenticity and pedagogical value of AI-generated feedback. This study investigates whether human participants can distinguish between feedback written by a human instructor and that generated by an AI model, and how the method of generation (prompt engineering vs. fine-tuning) affects this perception. One hundred participants completed a Turing-test-inspired task in which they evaluated feedback texts and indicated whether they believed it to be written by a human or an AI model. The AI-generated feedback was produced using either prompt engineering or a fine-tuned LLaMA 3.1 model adapted with QLoRA. The results showed that participants correctly identified the prompt engineering feedback as nonhuman in most cases, while the fine-tuned feedback identification was indistinguishable from chance. These findings suggest that fine-tuning can significantly enhance the human-likeness of AI-generated feedback, with implications for the design of scalable, trustworthy feedback systems in education. The study contributes to ongoing discussions about the role of AI in supporting reflective learning and highlights the importance of transparency, trust, and pedagogical alignment in AI-mediated educational environments.\u003c/p\u003e","manuscriptTitle":"Passing the Turing Test: Fine-tuned AI feedback is less detectable than human or prompt-engineered feedback","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-03 20:54:01","doi":"10.21203/rs.3.rs-7486768/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"34c7776f-087b-4dfe-8f39-1f3a5dde2672","owner":[],"postedDate":"September 3rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-19T08:58:17+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-03 20:54:01","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7486768","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7486768","identity":"rs-7486768","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}