DEDICAITE — DEtecting AI-generated TExts in a DIdactic Context

DEDICAITE — DEtecting AI-generated TExts in a DIdactic Context | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article DEDICAITE — DEtecting AI-generated TExts in a DIdactic Context Maria Berger, Steffen Hessler, Johanna Sophie Busse, Berin Doru, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7682039/v2 This work is licensed under a CC BY 4.0 License Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Abstract The widespread use of generative AI by students poses challenges for university teachers. Recent studies showed that medical and humanities scholars familiar with student-written texts are 70% able to recognize whether a text was student-written or generated by ChatGPT. In a randomized study, we examine whether we can reproduce this hit-rate with a larger sample of teachers from all university faculties, and whether we can confirm the hypothesis that linguistic features rather than content are crucial for correct classification. Therefore, 295 university teachers received one of two samples of an academic text speaking e.g., about legal or a scientific topic, written either by a student or generated by ChatGPT-4.0. The participants were randomly assigned to two groups: one received detailed instructions on linguistic features for authorship recognition, the other did not. We then asked participants how familiar they were with the topic of the text (6-point-Likert) and whether the text was characterized by detailed argumentation, avoidance of redundancies, and a recurring theme. The detection rate was 66% and 63.8% (not significant), in both groups respectively, although only 11% had received a text with a familiar topic. For non-humanities scholars, the dedicated instructions led to a significantly higher hit rate (75% vs. 59%). In general, texts written by humans were more often correctly identified (72% vs. 58%). For the subsequent questions on text properties, the uni-variate analysis of the answers for correctly recognizing student texts, resulted in a highly significant positive agreement, and a rejection for ChatGPT-generated texts. Artificial Intelligence and Machine Learning Linguistics ChatGPT randomized controlled study student-written ChatGPT-generated targeted instructions stylistic attributes subject area-close subject area-far authorship prediction Full Text Additional Declarations The authors declare no competing interests. A project description was sent to the Ethics Committee of the Medical Faculty at the Ruhr University Bochum (IRB), prior to the study. As the study does not involve direct research on human participants or patient data, nor did participants dependent on the interviewers, the committee informed us that ethical approval was not required and was waived by the IRB. Participants' data and responses were collected and stored anonymously. Cite Share Download PDF Status: Posted Version 2 posted You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7682039","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":523151989,"identity":"d0b899af-9cab-4893-a31b-21e9ac67016b","order_by":0,"name":"Maria Berger","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/0lEQVRIie3PMUsDMRjG8ec4yPTCrRGO+gmEyEG6FD9LpBAX5+og6FSXQ9fb/Q7OKYFO8boW7FARMjlUBHEo1uO6FDG1o2D+JEvIj7wBYrE/mmnXul6eAaxduxJNe1c7EGwQS8L8Qg6uH57MB2ad7u3deIGLlIpHJ+cY9IJEuhMxKuGLfOr7FcaMZH3aFah1mBgNQ7DHFXdFumRE0pHkydCGycRjtIS9rPjkDfjkVJQtWYXJVDdfhlU8K1MkQ0GCWmK2EA+bC39YcVYguVHEHTsTqu5vGUynry/ns32e2WfgfdVpXrufLwZHQbJONJurjQP1871vZeHpY7FY7J/3BV+hUDEsRn1LAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0009-0008-6191-9623","institution":"Ruhr University Bochum","correspondingAuthor":true,"prefix":"","firstName":"Maria","middleName":"","lastName":"Berger","suffix":""},{"id":523151990,"identity":"ff0ba780-5139-4a6c-8b0b-4c63c14e275f","order_by":1,"name":"Steffen Hessler","email":"","orcid":"https://orcid.org/0009-0009-0225-6169","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Steffen","middleName":"","lastName":"Hessler","suffix":""},{"id":523151991,"identity":"9237e72f-73b7-4e05-affe-4781b964b9e1","order_by":2,"name":"Johanna Sophie Busse","email":"","orcid":"https://orcid.org/0009-0007-1923-3575","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Johanna","middleName":"Sophie","lastName":"Busse","suffix":""},{"id":523151992,"identity":"85099f30-dc38-4934-80bf-0432308e5434","order_by":3,"name":"Berin Doru","email":"","orcid":"https://orcid.org/0009-0008-1067-7468","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Berin","middleName":"","lastName":"Doru","suffix":""},{"id":523151993,"identity":"428cdd75-56bd-4752-95a5-1b4c706b4c2a","order_by":4,"name":"Stephanie Anna Christine Heimgartner","email":"","orcid":"https://orcid.org/0009-0004-1589-9423","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Stephanie","middleName":"Anna Christine","lastName":"Heimgartner","suffix":""},{"id":523151994,"identity":"a4957269-6c88-4e1b-9f7a-ac0582ae19ec","order_by":5,"name":"Judith Herzog","email":"","orcid":"","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Judith","middleName":"","lastName":"Herzog","suffix":""},{"id":523151995,"identity":"fbb69705-5d06-458f-9998-52d4845ac581","order_by":6,"name":"Judith Schönhoff","email":"","orcid":"https://orcid.org/0009-0007-5656-4816","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Judith","middleName":"","lastName":"Schönhoff","suffix":""},{"id":523151996,"identity":"ae85caa1-9ac2-4ff1-9cf9-336acf34cc90","order_by":7,"name":"Julien Stein","email":"","orcid":"https://orcid.org/0000-0002-9699-3240","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Julien","middleName":"","lastName":"Stein","suffix":""},{"id":523151997,"identity":"0f290903-1eed-4c32-8cb0-19ca05b5d96f","order_by":8,"name":"Marianne Tokic","email":"","orcid":"https://orcid.org/0000-0001-9789-575X","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Marianne","middleName":"","lastName":"Tokic","suffix":""},{"id":523151998,"identity":"323c218c-1ac9-485e-8c0b-445678f726e2","order_by":9,"name":"Christoph Maier","email":"","orcid":"https://orcid.org/0000-0003-1212-744X","institution":"Ruhr University Bochum","correspondingAuthor":false,"prefix":"","firstName":"Christoph","middleName":"","lastName":"Maier","suffix":""}],"badges":[],"createdAt":"2025-09-22 17:54:51","currentVersionCode":2,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-7682039/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-7682039/v2","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107168763,"identity":"7afcfb34-5e29-4132-8747-f0fdcd4a8f8f","added_by":"auto","created_at":"2026-04-17 14:27:06","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":549833,"visible":true,"origin":"","legend":"","description":"","filename":"DEDICAITE.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7682039/v2_covered_44955745-0d23-4d49-a66f-5033e1e2e4d1.pdf"}],"financialInterests":"\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003eA project description was sent to the Ethics Committee of the Medical Faculty at the Ruhr University Bochum (IRB), prior to the study. As the study does not involve direct research on human participants or patient data, nor did participants dependent on the interviewers, the committee informed us that ethical approval was not required and was waived by the IRB. Participants' data and responses were collected and stored anonymously.\u003c/p\u003e","formattedTitle":"\u003cp\u003eDEDICAITE — DEtecting AI-generated TExts in a DIdactic Context\u003c/p\u003e","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Ruhr University Bochum","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"ChatGPT, randomized controlled study, student-written, ChatGPT-generated, targeted instructions, stylistic attributes, subject area-close, subject area-far, authorship prediction","lastPublishedDoi":"10.21203/rs.3.rs-7682039/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7682039/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe widespread use of generative AI by students poses challenges for university teachers. Recent studies showed that medical and humanities scholars familiar with student-written texts are 70% able to recognize whether a text was student-written or generated by ChatGPT. In a randomized study, we examine whether we can reproduce this hit-rate with a larger sample of teachers from all university faculties, and whether we can confirm the hypothesis that linguistic features rather than content are crucial for correct classification. Therefore, 295 university teachers received one of two samples of an academic text speaking e.g., about legal or a scientific topic, written either by a student or generated by ChatGPT-4.0. The participants were randomly assigned to two groups: one received detailed instructions on linguistic features for authorship recognition, the other did not. We then asked participants how familiar they were with the topic of the text (6-point-Likert) and whether the text was characterized by detailed argumentation, avoidance of redundancies, and a recurring theme. The detection rate was 66% and 63.8% (not significant), in both groups respectively, although only 11% had received a text with a familiar topic. For non-humanities scholars, the dedicated instructions led to a significantly higher hit rate (75% vs. 59%). In general, texts written by humans were more often correctly identified (72% vs. 58%). For the subsequent questions on text properties, the uni-variate analysis of the answers for correctly recognizing student texts, resulted in a highly significant positive agreement, and a rejection for ChatGPT-generated texts.\u003c/p\u003e","manuscriptTitle":"DEDICAITE — DEtecting AI-generated TExts in a DIdactic Context","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2025-10-22 19:42:03","doi":"10.21203/rs.3.rs-7682039/v2","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2025-10-01 03:21:16","doi":"10.21203/rs.3.rs-7682039/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6e742874-d5b6-48d1-b472-7543ff32f718","owner":[],"postedDate":"October 22nd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":66099638,"name":"Artificial Intelligence and Machine Learning"},{"id":66099639,"name":"Linguistics"}],"tags":[],"updatedAt":"2025-10-01T03:21:16+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-22 19:42:03","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v2","identity":"rs-7682039","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7682039","identity":"rs-7682039","version":["v2"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}