Distortion by Design? Examining How Generative AI Assistance Alters Graduate Students’ Writing Style and Critical Reasoning in Written Paper Critiques

Distortion by Design? Examining How Generative AI Assistance Alters Graduate Students’ Writing Style and Critical Reasoning in Written Paper Critiques | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Distortion by Design? Examining How Generative AI Assistance Alters Graduate Students’ Writing Style and Critical Reasoning in Written Paper Critiques Alejandro H. Espera Jr. This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9612303/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Contribution: This article presents a mixed-methods study of generative AI assistance in graduate engineering paper critiques and operationalizes AI distortion as a validity problem in writing-based assessment. Background: Generative AI can improve fluency and organization, but it may also encourage overreliance, homogenized academic expression, cognitive offloading, and weakened assessment validity. Research Questions: The study asks how rubric performance, AI-writing indicators, and feedback-coded reasoning markers differ across AI-permitted and AI-prohibited critique phases. Methodology: A convergent mixed-methods design was applied to 120 anonymized critique records from a graduate sustainable-energy course. Results: AI-permitted critiques scored substantially higher than AI-prohibited critiques across all rubric domains, while qualitative feedback indicated that surface polish sometimes masked reduced methodological specificity and source fidelity. Conclusion: The results support a sociocognitive validity interpretation: AI can scaffold communication but may distort the evidentiary meaning of written critique artifacts when the intended construct is graduate-level disciplinary reasoning. generative AI large language models engineering education assessment validity written critique cognitive offloading sociocognitive writing mixed methods Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 I. Introduction Generative AI tools have changed the conditions under which students produce academic writing. In graduate engineering courses, written critiques are often used to assess comprehension of technical literature, evaluation of methods, contextualization of findings, and scholarly communication. These assessment practices are now affected by tools that can rapidly generate fluent prose, synthesize sources, and reorganize arguments, leaving the instructor uncertain about which parts of the final artifact reflect student cognition [ 1 ]-[ 7 ]. This problem is not reducible to plagiarism or detection. The more consequential issue for engineering education is validity: whether a written critique still supports the intended inference that the student understood the paper, evaluated the evidence, and developed an independent scholarly judgment. Validity theory frames this as a problem of construct representation and construct-irrelevant variance. At the same time, recent AI-detection scholarship cautions that detector outputs should not be treated as stand-alone evidence of authorship [ 8 ]-[ 14 ]. The central claim is that AI assistance may improve surface-level writing while simultaneously altering the visible traces of critical reasoning. I refer to this phenomenon as AI distortion: a mismatch between polished written performance and the underlying construct that the assessment intends to measure. This claim is consistent with emerging evidence that generative AI can shape interaction patterns in writing, homogenize style, alter scientific vocabulary, and blur the distinction between fluent academic prose and human-authored disciplinary judgment [ 15 ]-[ 21 ]. II. Theoretical Framework: Sociocognitive-Validity Model of AI Distortion The theoretical framework integrates four traditions: sociocognitive writing theory, cognitive offloading, assessment validity, and genre-based disciplinary critique. Sociocognitive writing theory treats writing as a problem-solving activity involving planning, translation, revision, and audience-oriented decision making rather than merely the production of finished text [ 22 ]-[ 24 ]. The text/style evidence dimension is also informed by writing analytics research showing that lexical diversity, cohesion, and formulaic n-gram patterns can reveal meaningful differences in written products [ 25 ]-[ 27 ]. Cognitive offloading explains why AI assistance can simultaneously help and distort. External tools can reduce cognitive burden and support expression, but they may also move planning, synthesis, and argument selection outside the student’s own reasoning process. In paper critiques, this matters because the educational target is not only a clean final essay but also the development of disciplinary judgment through reading, comparison, evaluation, and revision [ 28 ]-[ 30 ]. Assessment validity theory provides the alignment mechanism for the study. Rubric scores are interpreted as evidence of student competence only if the artifact reflects the intended construct. AI assistance introduces possible construct-irrelevant variance when language quality, organization, or generic synthesis improves without proportional improvement in comprehension, methodological critique, or contextualized reasoning [ 9 ]-[ 13 ]. Genre-based disciplinary critique theory completes the model. A graduate paper critique is a disciplinary genre in which students must identify claims, evaluate methods, compare evidence, contextualize implications, and communicate a warranted judgment. The framework aligns the four rubric criteria with three evidence streams: rubric evidence, text/style evidence, and reasoning evidence, so that the study can ask not only whether scores differ, but whether the written artifact remains valid evidence of student cognition [ 9 ], [ 23 ], [ 24 ]. We address the following research questions: RQ1: How do rubric scores for Comprehension, Critical Analysis, Contextualization, Communication, and Total differ between AI-permitted and AI-prohibited critique phases? RQ2: What patterns do Turnitin AI-writing indicators reveal during the no-AI phase, and how do these indicators align with rubric outcomes? RQ3: What qualitative feedback-coded markers indicate stylistic distortion, epistemic distortion, or authentic reasoning attempts? RQ4: How do the quantitative and qualitative strands jointly inform the validity argument for using written critiques to assess graduate students’ critical reasoning in the era of generative AI? III. Methodology The study used a convergent mixed-methods design. Quantitative rubric scores and AI-writing indicators were analyzed alongside qualitative instructor feedback. The strands were integrated through joint displays that connect score patterns with evidence of writing style, source fidelity, and reasoning depth; this follows mixed-methods guidance that integration should occur through explicit displays and meta-inferences rather than through separate quantitative and qualitative reporting alone [ 31 ]-[ 33 ]. The corpus consisted of 120 graded graduate critique submissions across nine critique assignments in a sustainable energy and materials course. Critiques 1–6 permitted limited AI support; Critiques 7–9 explicitly prohibited AI use. The nine critique topics covered photovoltaics, battery systems, emerging electrodes, hydrogen technologies, electrochemical hydrogen production, additive manufacturing, thermoelectrics, carbon capture, and advanced characterization/modeling. All participant names were removed from the manuscript reporting. Students were assigned pseudonymous identifiers (P01-P15) during analysis, and all tables report aggregate, critique-level, or policy-condition-level results rather than identifiable student records. Each critique was scored using a 100-point analytic rubric: Comprehension and Summary (20 points), Critical Analysis (40 points), Contextualization (25 points), and Communication (15 points). Instructors also provided feedback that identified strengths, weaknesses, AI overreliance patterns, source alignment problems, and revision needs. Rubric-based scoring was treated as assessment evidence, but the interpretation of scores was examined through a validity lens rather than assumed to be self-evident [ 9 ]-[ 13 ]. Quantitative analysis used descriptive statistics for each rubric domain and condition, Welch independent-samples tests for AI-permitted versus AI-prohibited phase comparisons, and Hedges g to estimate effect magnitude. Because the dataset came from an authentic course context rather than a randomized experiment, inferential statistics were used to provide descriptive evidence for the observed corpus rather than to make strong causal claims. This interpretation is consistent with evidence-focused reporting expectations for engineering education studies [ 34 ], [ 35 ]. Qualitative analysis coded instructor feedback using a hybrid deductive-inductive codebook. Deductive codes were derived from the theoretical framework: Generic Summary, Surface Polish Without Depth, Source Drift or Misalignment, AI-Policy Violation, Weak Methodological Interrogation, Authentic Voice or Personal Reasoning, and High Specificity/Source Fidelity. Inductive refinements were added when feedback patterns recurred across critiques. The coding strategy followed the guidance for reflexive thematic analysis and mixed-method integration [ 31 ]-[ 33 ]. TABLE I. Corpus Structure and Topic Context Across Critique Assignments Critique Policy condition n Assigned paper-topic context C1 AI permitted 13 Perovskite and perovskite-silicon photovoltaics: efficiency records, stability, commercialization, lifecycle risk C2 AI permitted 11 Tandem photovoltaics: all-perovskite vs. monolithic perovskite/silicon modules, interface passivation, Science breakthrough framing C3 AI permitted 14 Solid-state and next-generation batteries: interfaces, SEI/solid electrolyte barriers, manufacturability, commercialization C4 AI permitted 14 Emerging electrode/material platforms: 2D materials, MOFs, sodium-ion cathodes, sustainability tradeoffs C5 AI permitted 13 Hydrogen-based energy systems: hydrogen storage, PV/PEM integration, SOFC recirculation, system deployment C6 AI permitted 14 Hydrogen production materials and systems: perovskite PEC, PbS quantum-dot PEC, Ni-based AEM electrolysis C7 AI prohibited 14 3D printing/additive manufacturing for energy systems: printed energy devices, microbial/fungal batteries, system readiness C8 AI prohibited 14 Emerging energy materials: thermoelectrics, wearable TEGs, AgBiS2 quantum dots, COF-based carbon capture C9 AI prohibited 13 Advanced electrolyte, characterization, and computational workflows for lithium-based batteries and materials innovation TABLE II. Rubric-to-Construct Alignment for the Sociocognitive-Validity Framework Rubric domain Weight Construct evidence Distortion risk Comprehension and Summary 20 Source fidelity; accurate identification of problem, methods, findings, contribution Misrepresentation, missing assigned-paper details, source drift Critical Analysis 40 Methodological judgment; evaluation of evidence, assumptions, limitations, tradeoffs Generic critique, overconfident claims, summary without interrogation Contextualization 25 Connection to broader literature, sustainability applications, future work Broad but unsupported claims, irrelevant external framing, weak implications Communication 15 Scholarly organization, citation integrity, clarity, flow Polish without depth, AI-template language, citation/formatting issues TABLE III. Mixed-Methods Analysis Plan and Empirical Results Analysis component Procedure Principal result Descriptive rubric analytics Mean/SD scores by policy phase and critique number AI-permitted total mean = 82.65; AI-prohibited total mean = 59.78 Welch phase comparisons Independent phase comparison for each rubric domain All five outcomes differed at p = 90%; 21 at > = 50% Feedback coding Instructor comments coded for distortion/authenticity markers Generic synthesis, surface polish without depth, source drift, and AI-policy violation were recurrent Joint display integration Scores, flags, and qualitative markers interpreted together AI improved observable performance but weakened validity of final-text-only inference IV. Results A. Rubric Performance by AI Policy Phase AI-permitted critiques scored higher than AI-prohibited critiques across every rubric domain. The mean total score was 82.65 (SD = 14.67) in the AI-permitted phase and 59.78 (SD = 18.05) in the AI-prohibited phase. The difference was statistically reliable using Welch tests and large in magnitude (t = 7.00, p < .001, Hedges g = 1.43). Communication showed the largest standardized difference (g = 1.42), consistent with the interpretation that AI assistance most strongly affects surface presentation and writing polish. TABLE IV. Rubric Score Comparisons by AI Policy Condition Outcome AI permitted M (SD) AI prohibited M (SD) Welch t p Hedges g Comprehension (20) 18.33 (3.08) 13.78 (4.97) 5.35 < .001 1.18 Critical Analysis (40) 32.05 (7.05) 23.56 (6.63) 6.51 < .001 1.22 Contextualization (25) 21.01 (4.74) 15.00 (5.24) 6.15 < .001 1.22 Communication (15) 11.25 (2.44) 7.44 (3.07) 6.91 < .001 1.42 Total (100) 82.65 (14.67) 59.78 (18.05) 7.00 < .001 1.43 B. Critique-Level Trends and Topic Context The score trajectory did not simply reflect a single difficult assignment. Means were consistently higher across all six AI-permitted critique rounds and consistently lower across all three AI-prohibited rounds. The AI-prohibited phase occurred during Critiques 7–9, which focused on additive manufacturing for energy systems, emerging energy materials, and advanced battery/electrolyte/computational workflows. These topics required synthesis across heterogeneous technical domains, making source fidelity and original comparative judgment especially visible in instructor feedback. TABLE V. Critique-Level Mean Scores With Assigned Topic Context Critique Condition Total Comp. Crit. Context Comm. Topic context C1 AI permitted 87.92 18.69 34.00 21.92 13.31 Perovskite and perovskite-silicon photovoltaics: efficiency records, stability, commercialization, lifecycle risk C2 AI permitted 80.36 17.27 30.36 21.55 11.18 Tandem photovoltaics: all-perovskite vs. monolithic perovskite/silicon modules, interface passivation, Science breakthrough framing C3 AI permitted 79.14 17.86 31.57 19.07 10.64 Solid-state and next-generation batteries: interfaces, SEI/solid electrolyte barriers, manufacturability, commercialization C4 AI permitted 83.86 18.93 32.86 21.14 10.93 Emerging electrode/material platforms: 2D materials, MOFs, sodium-ion cathodes, sustainability tradeoffs C5 AI permitted 86.38 19.62 33.69 22.46 10.62 Hydrogen-based energy systems: hydrogen storage, PV/PEM integration, SOFC recirculation, system deployment C6 AI permitted 78.36 17.50 29.71 20.21 10.93 Hydrogen production materials and systems: perovskite PEC, PbS quantum-dot PEC, Ni-based AEM electrolysis C7 AI prohibited 60.57 14.29 23.29 15.43 7.57 3D printing/additive manufacturing for energy systems: printed energy devices, microbial/fungal batteries, system readiness C8 AI prohibited 61.79 13.93 25.43 15.07 7.36 Emerging energy materials: thermoelectrics, wearable TEGs, AgBiS2 quantum dots, COF-based carbon capture C9 AI prohibited 56.77 13.08 21.85 14.46 7.38 Advanced electrolyte, characterization, and computational workflows for lithium-based batteries and materials innovation C. Turnitin AI-Writing Indicators in the No-AI Phase During the AI-prohibited phase, 23 submissions had recorded Turnitin AI-writing indicators. Nine submissions were recorded at 100%, 12 at 90% or above, and 21 at 50% or above. These high indicator values clustered in the no-AI phase and were associated with severe Communication penalties and lower total scores when the instructor determined that the result violated explicit course policy. The indicators were interpreted cautiously as part of a broader evidence set, including explicit AI disclosures, generic structure, mismatched references, source drift, and instructor comments. TABLE VI. Turnitin AI-Writing Indicator Summary During AI-Prohibited Phase AI-writing indicator threshold Submissions meeting threshold Percent of no-AI submissions (n = 41) >=50% 21 51.2% >=70% 18 43.9% >=90% 12 29.3% 100% 9 22.0% D. Qualitative Distortion Markers and Mixed-Methods Integration Qualitative feedback showed that high-scoring AI-permitted critiques often displayed polished structure but were sometimes described as generic, summary-heavy, or insufficiently interrogative. In contrast, some lower-scoring no-AI critiques showed rougher prose but clearer evidence of authentic struggle, personal voice, and developmental reasoning. The most severe concerns occurred when high AI-writing indicators appeared despite explicit no-AI instructions; in these cases, feedback emphasized policy violation and reduced confidence in the originality of the submitted reasoning. TABLE VII. Feedback-Coded Distortion and Authenticity Markers Marker Operational meaning Observed pattern Validity interpretation Generic summary Broad synthesis with limited paper-specific evidence Common in AI-permitted work; also appeared in high-flag no-AI submissions Weakens Critical Analysis and Contextualization evidence Surface polish without depth Fluent prose and coherent structure with weak methodological interrogation Most visible when Communication was strong but feedback requested deeper critique Signals possible stylistic distortion Source drift or misalignment Assigned paper set replaced or supplemented by irrelevant papers, previous critique topics, or broad external framing Appeared in several lower-scoring no-AI submissions and flagged cases Threatens Comprehension and source fidelity AI-policy violation Explicit AI disclosure or high Turnitin indicator during no-AI phase Used as grading-relevant evidence only during Critiques 7-9 Threatens validity and academic integrity Authentic voice and effort Simpler prose with visible attempts at reasoning, personal technical experience, or concrete observations More visible in some no-AI submissions with low or zero AI indicators Supports validity of inference despite lower polish Strong disciplinary judgment Specific evaluation of methods, assumptions, scalability, lifecycle, and deployment evidence Observed in strongest submissions across phases Supports high Critical Analysis and Contextualization scores TABLE VIII. Joint Display of Quantitative and Qualitative Findings Evidence pattern Quantitative result Qualitative result Integrated inference AI permitted (Critiques 1–6) High scores across all domains; total M = 82.65 Often polished and organized; recurring feedback requested deeper critique or less summary AI likely functioned as writing scaffold; final text may overstate independent reasoning AI prohibited (Critiques 7–9), low/no AI indicator Lower scores but feedback often identifies genuine effort, source engagement, and developmental gaps Rougher prose, clearer individual voice, more visible reasoning difficulties Lower surface quality may provide more valid evidence of current student competence AI prohibited, high AI indicator Severe penalties, especially Communication and total score Explicit AI disclosure, generic synthesis, mismatched references, or source drift in several cases Policy violation creates construct contamination and undermines score interpretation High-performing independent cases Scores remained high despite no-AI condition for selected students Specific methods critique, topic fidelity, and strong disciplinary judgment Demonstrates that high performance is possible without AI when reasoning is visible V. Discussion The findings support the sociocognitive validity framework. AI assistance appears to function as a cognitive scaffold that improves fluency, organization, and rhetorical coherence; however, it also changes the evidentiary meaning of the final written critique. In this dataset, AI-permitted work scored higher, but feedback often identified generic synthesis and limited methodological interrogation. This pattern aligns with research on AI-supported writing, cognitive offloading, interaction patterns, and higher-order thinking: tools can increase performance in the immediate artifact while reducing the visibility or development of the learner’s own reasoning process [ 3 ]-[ 5 ], [ 15 ]-[ 19 ], [ 28 ]-[ 30 ]. The topic context clarifies why the results matter. In Critiques 1–2, students critiqued photovoltaic papers where common AI-generated discourse could plausibly reproduce familiar claims about efficiency, stability, scalability, and commercialization. In Critiques 3–6, the topics shifted to batteries, electrode materials, and hydrogen systems, requiring a more specific methodological critique. In Critiques 7–9, where AI was prohibited, students confronted additive manufacturing, thermoelectric/quantum-dot/COF materials, and advanced battery/electrolyte/computational workflows. These later topics made source fidelity and comparative reasoning more demanding, and high AI-writing indicators became especially consequential because they conflicted with explicit policy. The most important implication is that written critiques remain valuable only if assessment design distinguishes writing polish from reasoning evidence. Communication scores rose substantially in the AI-permitted phase, but the same surface qualities can mask shallow critique. Conversely, lower no-AI scores may reveal authentic struggles that are pedagogically useful. This distinction is central to assessment validity theory because the score interpretation depends on whether the observed performance adequately represents the intended construct [ 9 ]-[ 13 ]. The results also caution against treating AI detection as the only solution. Turnitin indicators were useful in the no-AI phase because they were interpreted alongside explicit policy, disclosure evidence, instructor feedback, source fidelity, and rubric performance. However, AI-writing indicators should not be treated as definitive measures of authorship or cognition because detection tools can produce accuracy-bias trade-offs and may be especially problematic for multilingual or non-native English writers [ 8 ], [ 14 ]. For engineering education, the broader contribution is a shift from detection to validity. Instead of asking only whether AI was used, instructors should ask whether the submitted critique still demonstrates the intended construct. The framework and joint displays in this paper offer a practical way to connect rubric evidence, text/style evidence, and reasoning evidence, consistent with validity arguments that require explicit warrants for score interpretation and use [ 11 ]-[ 13 ], [ 32 ], [ 33 ]. VI. Implications for Assessment Design First, rubrics should separate communication quality from reasoning depth. Communication should not compensate for a weak source-specific critique. Second, critique prompts should require evidence that is difficult to generate generically: named methods, specific results, limitations tied to figures or protocols, and explicit comparison across assigned papers. Third, if AI is permitted, students should submit an AI-use disclosure and a process memo. Finally, instructors should consider oral checks, annotated source maps, or AI-audit assignments that ask students to evaluate and correct AI-generated critiques. These recommendations are consistent with research showing that AI interaction patterns, authorial voice, and student perceptions of AI-based writing tools shape whether AI functions as a scaffold or substitute [ 15 ], [ 19 ]-[ 21 ]. VII. Limitations The study has several limitations. The dataset came from one course context, and the phase comparison was not randomized. Critiques 7–9 differed in both AI policy and technical topic, so phase effects cannot be interpreted as purely causal AI effects. The dataset contained rubric scores, instructor comments, and AI-writing indicators but did not include complete process traces, prompt histories, or validated automated measures of lexical diversity and cohesion. In addition, AI-writing indicators were treated as contextual evidence rather than definitive proof, consistent with current concerns about AI detector reliability and fairness [ 8 ], [ 14 ]. VII. Conclusion This study examined how generative AI assistance altered graduate students’ written paper critiques across nine assignments. AI-permitted critiques scored substantially higher than AI-prohibited critiques across all rubric domains, especially the Communication and Total score. Yet qualitative feedback showed that the same surface gains could coincide with generic synthesis, weaker source fidelity, and reduced methodological interrogation. These findings extend recent work on generative AI in higher education by locating the central problem in the validity of writing-based assessment rather than in detection alone [ 1 ]-[ 8 ], [ 11 ]-[ 21 ], [ 30 ]. The main contribution is a sociocognitive validity framework for interpreting AI distortion in writing-based assessment. The framework shows that the central issue is not whether AI improves writing, but whether the written artifact still provides valid evidence of student cognition. For graduate engineering education, this means assessment systems must move beyond final-text grading toward transparent, process-aware, source-specific, and validity-oriented evaluation. Declarations Acknowledgment The author acknowledges the anonymized students whose coursework and feedback records served as the basis for this classroom-based inquiry. Participant names and personally identifying details were removed from all manuscript tables, figures, and quoted feedback. Any final submission should include the appropriate institutional ethics, consent, or exemption statement before journal upload. Ethics Approval Statement This study involved analysis of anonymized course-based assessment records and instructor feedback. The study was reviewed and approved by the Research Ethics Committee. All data were anonymized prior to analysis, and no personally identifying student information is reported in the manuscript. Participant Consent Statement Participants provided consent for their anonymized coursework data, grading records, and instructor feedback to be used for research and publication purposes. All participant identifiers were removed, and the results are reported only in anonymized and aggregated form. References Kasneci E et al (2023) ChatGPT for good? On opportunities and challenges of large language models for education. Learn Individ Differ 103:102274 Cotton DRE, Cotton PA, Shipway JR (2024) Chatting and cheating: Ensuring academic integrity in the era of ChatGPT, Innov. Educ Teach Int 61(2):228–239 Deng R, Jiang M, Yu X, Lu Y, Liu S (2025) Does ChatGPT enhance student learning? A systematic review and meta-analysis. Comput Educ 227:105224 Wang J, Fan W (2025) The effect of ChatGPT on students’ learning performance, perception, and higher-order thinking: A meta-analysis, Humanit. Soc Sci Commun 12:621 Belkina M et al (2025) Implementing generative AI in higher education: A systematic review of case studies. Comput Educ : Artif Intell 8:100407 Abdallah N, Katmah R, Khalaf K, Jelinek HF (2025) Systematic review of ChatGPT in higher education. Soc Sci Humanit Open 12:101866 Zawacki-Richter O, Marin VI, Bond M, Gouverneur F (2019) Systematic review of research on AI applications in higher education. Int J Educ Technol High Educ 16:39 Liang W, Yuksekgonul M, Mao Y, Wu E, Zou J (2023) GPT detectors are biased against non-native English writers, Patterns, vol. 4, no. 7, p. 100779 Hodges TS et al (2019) Developing and examining validity evidence for a writing achievement rubric, Assessing Writing, vol. 41, pp. 26–36 Panadero E et al (2024) Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analysis. Educ Psychol Rev 36:21 Messick S (1995) Validity of psychological assessment. Am Psychol 50(9):741–749 Kane MT (2013) Validating the interpretations and uses of test scores. J Educ Meas 50(1):1–73 AERA, APA, and NCME (2014) Standards for Educational and Psychological Testing. AERA, Washington, DC, USA Pratama AR et al (2025) The accuracy-bias trade-offs in AI text detection tools. npj Sci Learn Kim J, Lee S-S, Detrick R et al (2025) Students-generative AI interaction patterns and impact on academic writing. J Comput High Educ Agarwal D, Naaman M, Vashistha A (2025) AI suggestions homogenize writing toward Western styles, Proc. CHI Conf. Human Factors Comput. Syst Kobak D et al (2025) Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Sci Adv 11(8):eadt3813 Sourati Z et al (2026) The homogenizing effect of large language models on human expression and thought. Trends Cogn Sci Liu M, Zhang LJ, Biebricher C (2024) Investigating students’ cognitive processes in generative AI-assisted composing and traditional writing. Comput Educ 211:104977 Amirjalili F et al (2024) A comparative analysis of AI-generated text and human authorship in academic writing. Front Educ 9:1347421 Boillos M et al (2025) Student perspectives on AI-based language tools in university writing. J Writ Res Vygotsky LS (1978) Mind in Society. Harvard Univ. Press, Cambridge, MA, USA Flower L, Hayes JR (1981) A cognitive process theory of writing, College Composition and Communication, vol. 32, no. 4, pp. 365–387 Bereiter C, Scardamalia M (1987) The Psychology of Written Composition. Erlbaum, Hillsdale, NJ, USA Kyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity indices using direct judgments. Lang Assess Q 18(2):154–170 Crossley SA et al (2019) The Tool for the Automatic Analysis of Cohesion 2.0. Behav Res Methods 51:14–30 Garner J, Crossley S, Kyle K (2019) N-gram measures and L2 writing proficiency, System, vol. 80, pp. 176–187 Salomon G, Perkins DN, Globerson T (1991) Partners in cognition: Extending human intelligence with intelligent technologies. Educ Res 20(3):2–9 Risko EF, Gilbert SJ (2016) Cognitive offloading. Trends Cogn Sci 20(9):676–688 Gerlich R (2025) AI tools in society: Impacts on cognitive offloading and the future of critical thinking, Societies, vol. 15, no. 1, p. 6 Byrne D (2022) A worked example of Braun and Clarke’s approach to reflexive thematic analysis, Qual—quant., vol. 56, pp. 1391–1412 Fetters MD, Tajima C (2022) Joint displays of integrated data collection in mixed methods research. Int J Qual Methods, 21 McCrudden MT et al (2021) Joint displays for mixed methods research in psychology, Methods Psychol., vol. 5, p. 100069 Froyd JE (2018) Structured abstracts for the IEEE Transactions on Education, IEEE Trans—Educ. 61(3):214–217 IEEE Education Society (2026) IEEE Transactions on Education Author Resources, [Online]. Available: IEEE Education Society website Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9612303","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":634405874,"identity":"7ee49766-c8c7-45b1-a0c0-213d17a5399f","order_by":0,"name":"Alejandro H. Espera Jr.","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIiWNgGAWjYDACCSjN2MB8gIGHDcRMAPGI0sKWQKIWBgYeA+K08M9uPvy5oOKePPPsMx8/vCk7zMDPnmPAXLgDjyV3jqVJzzhTbNjYl7tZcs65wwySPW8MmGeeweOwGzlmzLxtCYyNPbwbpHnbDjMY3ADawtuGW4f8jfzPn3n/Jdg39vA8/g3SYk9IC9BMBmnehoREoBY2iC0SBLQY3kgzk55xLCG5sYfNzHLOuXQeiTPPCg7PxKNF7kby488FNQm2G3uYH994U2Ytx9+evPFxIR4tIMAMtq4BwuEBEYfxa4BqkccQGQWjYBSMglEABQAprFHPE1BCGgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-3294-1847","institution":"Ateneo de Davao University","correspondingAuthor":true,"prefix":"","firstName":"Alejandro","middleName":"H.","lastName":"Espera","suffix":"Jr."}],"badges":[],"createdAt":"2026-05-04 22:18:10","currentVersionCode":1,"declarations":{"humanSubjects":true,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":true,"humanSubjectConsent":true,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-9612303/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9612303/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108947446,"identity":"3b96a80d-107c-4e0b-ab17-6ec0c3138875","added_by":"auto","created_at":"2026-05-11 06:29:05","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":1005855,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSociocognitive validity framework for AI distortion in written critique assessment.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/16b26b488d50ff1ce9d00602.png"},{"id":108947381,"identity":"c12fcaab-825b-4a81-b40b-646489593718","added_by":"auto","created_at":"2026-05-11 06:28:19","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":121697,"visible":true,"origin":"","legend":"\u003cp\u003eMean total rubric score across critique assignments. The dashed line marks the transition to the no-AI phase.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/073897f695ea98e4f5f1e258.png"},{"id":108947405,"identity":"d45900d9-ce93-4fcc-bd66-85615d587e63","added_by":"auto","created_at":"2026-05-11 06:28:34","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":82323,"visible":true,"origin":"","legend":"\u003cp\u003eMean rubric-domain scores by AI policy condition. Error bars show approximate 95% confidence intervals.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/c03ce41ff275f99cc5463303.png"},{"id":108947429,"identity":"3315387b-4b71-4120-b5d5-81ea1f31e40a","added_by":"auto","created_at":"2026-05-11 06:28:55","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":171509,"visible":true,"origin":"","legend":"\u003cp\u003eHeat map of mean rubric-domain performance by critique, scaled as a percentage of each domain’s maximum.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/07389a4e6eea9e246ccf74ce.png"},{"id":108947496,"identity":"5fcd09ae-b54d-4577-9e83-c213f0e12207","added_by":"auto","created_at":"2026-05-11 06:29:27","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":144064,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eTurnitin AI-writing indicator versus total rubric score in the AI-prohibited phase.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/a2f6ad0ef30899159f1fda74.png"},{"id":108947437,"identity":"2b6fb8bd-3c78-4ec0-b532-eb0d72c9e4d2","added_by":"auto","created_at":"2026-05-11 06:29:00","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":186296,"visible":true,"origin":"","legend":"\u003cp\u003eSurface polish versus reasoning-depth matrix based on Communication and combined Critical Analysis/Contextualization scores.\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/6f908cd3376e724b1b9b0085.png"},{"id":108947553,"identity":"4e81c69e-6437-487c-8de3-c43668fcdb35","added_by":"auto","created_at":"2026-05-11 06:29:35","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1879610,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9612303/v1/5b6bf33f-17a4-4121-ad1e-f66dabfb8a3b.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eDistortion by Design? Examining How Generative AI Assistance Alters Graduate Students’ Writing Style and Critical Reasoning in Written Paper Critiques\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"I. Introduction","content":"\u003cp\u003eGenerative AI tools have changed the conditions under which students produce academic writing. In graduate engineering courses, written critiques are often used to assess comprehension of technical literature, evaluation of methods, contextualization of findings, and scholarly communication. These assessment practices are now affected by tools that can rapidly generate fluent prose, synthesize sources, and reorganize arguments, leaving the instructor uncertain about which parts of the final artifact reflect student cognition [\u003cspan additionalcitationids=\"CR2 CR3 CR4 CR5 CR6\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]-[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThis problem is not reducible to plagiarism or detection. The more consequential issue for engineering education is validity: whether a written critique still supports the intended inference that the student understood the paper, evaluated the evidence, and developed an independent scholarly judgment. Validity theory frames this as a problem of construct representation and construct-irrelevant variance. At the same time, recent AI-detection scholarship cautions that detector outputs should not be treated as stand-alone evidence of authorship [\u003cspan additionalcitationids=\"CR9 CR10 CR11 CR12 CR13\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]-[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe central claim is that AI assistance may improve surface-level writing while simultaneously altering the visible traces of critical reasoning. I refer to this phenomenon as AI distortion: a mismatch between polished written performance and the underlying construct that the assessment intends to measure. This claim is consistent with emerging evidence that generative AI can shape interaction patterns in writing, homogenize style, alter scientific vocabulary, and blur the distinction between fluent academic prose and human-authored disciplinary judgment [\u003cspan additionalcitationids=\"CR16 CR17 CR18 CR19 CR20\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]-[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e"},{"header":"II. Theoretical Framework: Sociocognitive-Validity Model of AI Distortion","content":"\u003cp\u003eThe theoretical framework integrates four traditions: sociocognitive writing theory, cognitive offloading, assessment validity, and genre-based disciplinary critique. Sociocognitive writing theory treats writing as a problem-solving activity involving planning, translation, revision, and audience-oriented decision making rather than merely the production of finished text [\u003cspan additionalcitationids=\"CR23\" citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]-[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. The text/style evidence dimension is also informed by writing analytics research showing that lexical diversity, cohesion, and formulaic n-gram patterns can reveal meaningful differences in written products [\u003cspan additionalcitationids=\"CR26\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]-[\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eCognitive offloading explains why AI assistance can simultaneously help and distort. External tools can reduce cognitive burden and support expression, but they may also move planning, synthesis, and argument selection outside the student\u0026rsquo;s own reasoning process. In paper critiques, this matters because the educational target is not only a clean final essay but also the development of disciplinary judgment through reading, comparison, evaluation, and revision [\u003cspan additionalcitationids=\"CR29\" citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]-[\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAssessment validity theory provides the alignment mechanism for the study. Rubric scores are interpreted as evidence of student competence only if the artifact reflects the intended construct. AI assistance introduces possible construct-irrelevant variance when language quality, organization, or generic synthesis improves without proportional improvement in comprehension, methodological critique, or contextualized reasoning [\u003cspan additionalcitationids=\"CR10 CR11 CR12\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]-[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eGenre-based disciplinary critique theory completes the model. A graduate paper critique is a disciplinary genre in which students must identify claims, evaluate methods, compare evidence, contextualize implications, and communicate a warranted judgment. The framework aligns the four rubric criteria with three evidence streams: rubric evidence, text/style evidence, and reasoning evidence, so that the study can ask not only whether scores differ, but whether the written artifact remains valid evidence of student cognition [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWe address the following research questions:\u003c/p\u003e \u003cp\u003eRQ1: How do rubric scores for Comprehension, Critical Analysis, Contextualization, Communication, and Total differ between AI-permitted and AI-prohibited critique phases?\u003c/p\u003e \u003cp\u003eRQ2: What patterns do Turnitin AI-writing indicators reveal during the no-AI phase, and how do these indicators align with rubric outcomes?\u003c/p\u003e \u003cp\u003eRQ3: What qualitative feedback-coded markers indicate stylistic distortion, epistemic distortion, or authentic reasoning attempts?\u003c/p\u003e \u003cp\u003eRQ4: How do the quantitative and qualitative strands jointly inform the validity argument for using written critiques to assess graduate students\u0026rsquo; critical reasoning in the era of generative AI?\u003c/p\u003e"},{"header":"III. Methodology","content":"\u003cp\u003eThe study used a convergent mixed-methods design. Quantitative rubric scores and AI-writing indicators were analyzed alongside qualitative instructor feedback. The strands were integrated through joint displays that connect score patterns with evidence of writing style, source fidelity, and reasoning depth; this follows mixed-methods guidance that integration should occur through explicit displays and meta-inferences rather than through separate quantitative and qualitative reporting alone [\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e]-[\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe corpus consisted of 120 graded graduate critique submissions across nine critique assignments in a sustainable energy and materials course. Critiques 1–6 permitted limited AI support; Critiques 7–9 explicitly prohibited AI use. The nine critique topics covered photovoltaics, battery systems, emerging electrodes, hydrogen technologies, electrochemical hydrogen production, additive manufacturing, thermoelectrics, carbon capture, and advanced characterization/modeling. All participant names were removed from the manuscript reporting. Students were assigned pseudonymous identifiers (P01-P15) during analysis, and all tables report aggregate, critique-level, or policy-condition-level results rather than identifiable student records.\u003c/p\u003e \u003cp\u003eEach critique was scored using a 100-point analytic rubric: Comprehension and Summary (20 points), Critical Analysis (40 points), Contextualization (25 points), and Communication (15 points). Instructors also provided feedback that identified strengths, weaknesses, AI overreliance patterns, source alignment problems, and revision needs. Rubric-based scoring was treated as assessment evidence, but the interpretation of scores was examined through a validity lens rather than assumed to be self-evident [\u003cspan class=\"CitationRef\"\u003e9\u003c/span\u003e]-[\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eQuantitative analysis used descriptive statistics for each rubric domain and condition, Welch independent-samples tests for AI-permitted versus AI-prohibited phase comparisons, and Hedges g to estimate effect magnitude. Because the dataset came from an authentic course context rather than a randomized experiment, inferential statistics were used to provide descriptive evidence for the observed corpus rather than to make strong causal claims. This interpretation is consistent with evidence-focused reporting expectations for engineering education studies [\u003cspan class=\"CitationRef\"\u003e34\u003c/span\u003e], [\u003cspan class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eQualitative analysis coded instructor feedback using a hybrid deductive-inductive codebook. Deductive codes were derived from the theoretical framework: Generic Summary, Surface Polish Without Depth, Source Drift or Misalignment, AI-Policy Violation, Weak Methodological Interrogation, Authentic Voice or Personal Reasoning, and High Specificity/Source Fidelity. Inductive refinements were added when feedback patterns recurred across critiques. The coding strategy followed the guidance for reflexive thematic analysis and mixed-method integration [\u003cspan class=\"CitationRef\"\u003e31\u003c/span\u003e]-[\u003cspan class=\"CitationRef\"\u003e33\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003cb\u003eTABLE I.\u003c/b\u003e Corpus Structure and Topic Context Across Critique Assignments\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eCritique\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003ePolicy condition\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003en\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eAssigned paper-topic context\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePerovskite and perovskite-silicon photovoltaics: efficiency records, stability, commercialization, lifecycle risk\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eTandem photovoltaics: all-perovskite vs. monolithic perovskite/silicon modules, interface passivation, Science breakthrough framing\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSolid-state and next-generation batteries: interfaces, SEI/solid electrolyte barriers, manufacturability, commercialization\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEmerging electrode/material platforms: 2D materials, MOFs, sodium-ion cathodes, sustainability tradeoffs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHydrogen-based energy systems: hydrogen storage, PV/PEM integration, SOFC recirculation, system deployment\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHydrogen production materials and systems: perovskite PEC, PbS quantum-dot PEC, Ni-based AEM electrolysis\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e3D printing/additive manufacturing for energy systems: printed energy devices, microbial/fungal batteries, system readiness\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEmerging energy materials: thermoelectrics, wearable TEGs, AgBiS2 quantum dots, COF-based carbon capture\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAdvanced electrolyte, characterization, and computational workflows for lithium-based batteries and materials innovation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eTABLE II.\u003c/b\u003e Rubric-to-Construct Alignment for the Sociocognitive-Validity Framework\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabb\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eRubric domain\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eWeight\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eConstruct evidence\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eDistortion risk\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eComprehension and Summary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSource fidelity; accurate identification of problem, methods, findings, contribution\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eMisrepresentation, missing assigned-paper details, source drift\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCritical Analysis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eMethodological judgment; evaluation of evidence, assumptions, limitations, tradeoffs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eGeneric critique, overconfident claims, summary without interrogation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eContextualization\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eConnection to broader literature, sustainability applications, future work\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eBroad but unsupported claims, irrelevant external framing, weak implications\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCommunication\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eScholarly organization, citation integrity, clarity, flow\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePolish without depth, AI-template language, citation/formatting issues\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003cb\u003eTABLE III.\u003c/b\u003e Mixed-Methods Analysis Plan and Empirical Results\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabc\" border=\"1\"\u003e \u003ccolgroup cols=\"3\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eAnalysis component\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eProcedure\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003ePrincipal result\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eDescriptive rubric analytics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eMean/SD scores by policy phase and critique number\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI-permitted total mean = 82.65; AI-prohibited total mean = 59.78\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eWelch phase comparisons\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eIndependent phase comparison for each rubric domain\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAll five outcomes differed at p \u0026lt; .001; Total g = 1.43\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI-writing indicator analysis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eTurnitin AI percentages in no-AI phase\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e23 recorded indicators; 9 at 100%; 12 at \u0026gt; = 90%; 21 at \u0026gt; = 50%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eFeedback coding\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eInstructor comments coded for distortion/authenticity markers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eGeneric synthesis, surface polish without depth, source drift, and AI-policy violation were recurrent\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eJoint display integration\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eScores, flags, and qualitative markers interpreted together\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI improved observable performance but weakened validity of final-text-only inference\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e "},{"header":"IV. Results","content":"\u003cp\u003e \u003cb\u003eA. Rubric Performance by AI Policy Phase\u003c/b\u003e \u003c/p\u003e\u003cp\u003eAI-permitted critiques scored higher than AI-prohibited critiques across every rubric domain. The mean total score was 82.65 (SD = 14.67) in the AI-permitted phase and 59.78 (SD = 18.05) in the AI-prohibited phase. The difference was statistically reliable using Welch tests and large in magnitude (t = 7.00, p \u0026lt; .001, Hedges g = 1.43). Communication showed the largest standardized difference (g = 1.42), consistent with the interpretation that AI assistance most strongly affects surface presentation and writing polish.\u003c/p\u003e\u003cp\u003e \u003cb\u003eTABLE IV.\u003c/b\u003e Rubric Score Comparisons by AI Policy Condition\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabd\" border=\"1\"\u003e \u003ccolgroup cols=\"6\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eOutcome\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eAI permitted M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eAI prohibited M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eWelch t\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003ep\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eHedges g\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eComprehension (20)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e18.33 (3.08)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13.78 (4.97)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e5.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026lt; .001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.18\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCritical Analysis (40)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e32.05 (7.05)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e23.56 (6.63)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e6.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026lt; .001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.22\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eContextualization (25)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21.01 (4.74)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e15.00 (5.24)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e6.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026lt; .001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.22\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCommunication (15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e11.25 (2.44)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e7.44 (3.07)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e6.91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026lt; .001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eTotal (100)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e82.65 (14.67)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e59.78 (18.05)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e7.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026lt; .001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e1.43\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e\u003cp\u003e \u003cb\u003eB. Critique-Level Trends and Topic Context\u003c/b\u003e \u003c/p\u003e\u003cp\u003eThe score trajectory did not simply reflect a single difficult assignment. Means were consistently higher across all six AI-permitted critique rounds and consistently lower across all three AI-prohibited rounds. The AI-prohibited phase occurred during Critiques 7–9, which focused on additive manufacturing for energy systems, emerging energy materials, and advanced battery/electrolyte/computational workflows. These topics required synthesis across heterogeneous technical domains, making source fidelity and original comparative judgment especially visible in instructor feedback.\u003c/p\u003e\u003cp\u003e \u003cb\u003eTABLE V. Critique-Level Mean Scores With Assigned Topic Context\u003c/b\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabe\" border=\"1\"\u003e \u003ccolgroup cols=\"8\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eCritique\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eCondition\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eComp.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eCrit.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eContext\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eComm.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eTopic context\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e87.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e18.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e34.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21.92\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13.31\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePerovskite and perovskite-silicon photovoltaics: efficiency records, stability, commercialization, lifecycle risk\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e80.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e17.27\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e30.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21.55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e11.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eTandem photovoltaics: all-perovskite vs. monolithic perovskite/silicon modules, interface passivation, Science breakthrough framing\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e79.14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e17.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e31.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e19.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e10.64\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSolid-state and next-generation batteries: interfaces, SEI/solid electrolyte barriers, manufacturability, commercialization\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e83.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e18.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e32.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21.14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e10.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEmerging electrode/material platforms: 2D materials, MOFs, sodium-ion cathodes, sustainability tradeoffs\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e86.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e19.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e33.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e22.46\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e10.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHydrogen-based energy systems: hydrogen storage, PV/PEM integration, SOFC recirculation, system deployment\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e78.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e17.50\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e29.71\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e20.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e10.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHydrogen production materials and systems: perovskite PEC, PbS quantum-dot PEC, Ni-based AEM electrolysis\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e60.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e23.29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e15.43\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e7.57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e3D printing/additive manufacturing for energy systems: printed energy devices, microbial/fungal batteries, system readiness\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e61.79\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e25.43\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e15.07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e7.36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEmerging energy materials: thermoelectrics, wearable TEGs, AgBiS2 quantum dots, COF-based carbon capture\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eC9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e56.77\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e13.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21.85\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e14.46\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e7.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAdvanced electrolyte, characterization, and computational workflows for lithium-based batteries and materials innovation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e\u003cp\u003e \u003cb\u003eC. Turnitin AI-Writing Indicators in the No-AI Phase\u003c/b\u003e \u003c/p\u003e\u003cp\u003eDuring the AI-prohibited phase, 23 submissions had recorded Turnitin AI-writing indicators. Nine submissions were recorded at 100%, 12 at 90% or above, and 21 at 50% or above. These high indicator values clustered in the no-AI phase and were associated with severe Communication penalties and lower total scores when the instructor determined that the result violated explicit course policy. The indicators were interpreted cautiously as part of a broader evidence set, including explicit AI disclosures, generic structure, mismatched references, source drift, and instructor comments.\u003c/p\u003e\u003cp\u003e \u003cb\u003eTABLE VI.\u003c/b\u003e Turnitin AI-Writing Indicator Summary During AI-Prohibited Phase\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabf\" border=\"1\"\u003e \u003ccolgroup cols=\"3\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eAI-writing indicator threshold\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eSubmissions meeting threshold\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003ePercent of no-AI submissions (n = 41)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026gt;=50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e51.2%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026gt;=70%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e43.9%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u0026gt;=90%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e29.3%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\"\u003e \u003cp\u003e22.0%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e\u003cp\u003e \u003cb\u003eD. Qualitative Distortion Markers and Mixed-Methods Integration\u003c/b\u003e \u003c/p\u003e\u003cp\u003eQualitative feedback showed that high-scoring AI-permitted critiques often displayed polished structure but were sometimes described as generic, summary-heavy, or insufficiently interrogative. In contrast, some lower-scoring no-AI critiques showed rougher prose but clearer evidence of authentic struggle, personal voice, and developmental reasoning. The most severe concerns occurred when high AI-writing indicators appeared despite explicit no-AI instructions; in these cases, feedback emphasized policy violation and reduced confidence in the originality of the submitted reasoning.\u003c/p\u003e\u003cp\u003e \u003cb\u003eTABLE VII.\u003c/b\u003e Feedback-Coded Distortion and Authenticity Markers\u003c/p\u003e\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eMarker\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eOperational meaning\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eObserved pattern\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eValidity interpretation\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eGeneric summary\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eBroad synthesis with limited paper-specific evidence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eCommon in AI-permitted work; also appeared in high-flag no-AI submissions\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eWeakens Critical Analysis and Contextualization evidence\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eSurface polish without depth\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eFluent prose and coherent structure with weak methodological interrogation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eMost visible when Communication was strong but feedback requested deeper critique\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eSignals possible stylistic distortion\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eSource drift or misalignment\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eAssigned paper set replaced or supplemented by irrelevant papers, previous critique topics, or broad external framing\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eAppeared in several lower-scoring no-AI submissions and flagged cases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eThreatens Comprehension and source fidelity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eAI-policy violation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eExplicit AI disclosure or high Turnitin indicator during no-AI phase\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eUsed as grading-relevant evidence only during Critiques 7-9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eThreatens validity and academic integrity\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eAuthentic voice and effort\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eSimpler prose with visible attempts at reasoning, personal technical experience, or concrete observations\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eMore visible in some no-AI submissions with low or zero AI indicators\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eSupports validity of inference despite lower polish\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 102px;\"\u003e\n \u003cp\u003eStrong disciplinary judgment\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 227px;\"\u003e\n \u003cp\u003eSpecific evaluation of methods, assumptions, scalability, lifecycle, and deployment evidence\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 190px;\"\u003e\n \u003cp\u003eObserved in strongest submissions across phases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 173px;\"\u003e\n \u003cp\u003eSupports high Critical Analysis and Contextualization scores\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\u003cp\u003e \u003cb\u003eTABLE VIII.\u003c/b\u003e Joint Display of Quantitative and Qualitative Findings\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Tabg\" border=\"1\"\u003e \u003ccolgroup cols=\"4\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\"\u003e \u003cp\u003eEvidence pattern\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eQuantitative result\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eQualitative result\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\"\u003e \u003cp\u003eIntegrated inference\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI permitted (Critiques 1–6)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHigh scores across all domains; total M = 82.65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOften polished and organized; recurring feedback requested deeper critique or less summary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI likely functioned as writing scaffold; final text may overstate independent reasoning\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited (Critiques 7–9), low/no AI indicator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eLower scores but feedback often identifies genuine effort, source engagement, and developmental gaps\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eRougher prose, clearer individual voice, more visible reasoning difficulties\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eLower surface quality may provide more valid evidence of current student competence\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAI prohibited, high AI indicator\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSevere penalties, especially Communication and total score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eExplicit AI disclosure, generic synthesis, mismatched references, or source drift in several cases\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePolicy violation creates construct contamination and undermines score interpretation\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHigh-performing independent cases\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eScores remained high despite no-AI condition for selected students\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSpecific methods critique, topic fidelity, and strong disciplinary judgment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eDemonstrates that high performance is possible without AI when reasoning is visible\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e"},{"header":"V. Discussion","content":"\u003cp\u003eThe findings support the sociocognitive validity framework. AI assistance appears to function as a cognitive scaffold that improves fluency, organization, and rhetorical coherence; however, it also changes the evidentiary meaning of the final written critique. In this dataset, AI-permitted work scored higher, but feedback often identified generic synthesis and limited methodological interrogation. This pattern aligns with research on AI-supported writing, cognitive offloading, interaction patterns, and higher-order thinking: tools can increase performance in the immediate artifact while reducing the visibility or development of the learner\u0026rsquo;s own reasoning process [\u003cspan additionalcitationids=\"CR4\" citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]-[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], [\u003cspan additionalcitationids=\"CR16 CR17 CR18\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]-[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], [\u003cspan additionalcitationids=\"CR29\" citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]-[\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe topic context clarifies why the results matter. In Critiques 1\u0026ndash;2, students critiqued photovoltaic papers where common AI-generated discourse could plausibly reproduce familiar claims about efficiency, stability, scalability, and commercialization. In Critiques 3\u0026ndash;6, the topics shifted to batteries, electrode materials, and hydrogen systems, requiring a more specific methodological critique. In Critiques 7\u0026ndash;9, where AI was prohibited, students confronted additive manufacturing, thermoelectric/quantum-dot/COF materials, and advanced battery/electrolyte/computational workflows. These later topics made source fidelity and comparative reasoning more demanding, and high AI-writing indicators became especially consequential because they conflicted with explicit policy.\u003c/p\u003e \u003cp\u003eThe most important implication is that written critiques remain valuable only if assessment design distinguishes writing polish from reasoning evidence. Communication scores rose substantially in the AI-permitted phase, but the same surface qualities can mask shallow critique. Conversely, lower no-AI scores may reveal authentic struggles that are pedagogically useful. This distinction is central to assessment validity theory because the score interpretation depends on whether the observed performance adequately represents the intended construct [\u003cspan additionalcitationids=\"CR10 CR11 CR12\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]-[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe results also caution against treating AI detection as the only solution. Turnitin indicators were useful in the no-AI phase because they were interpreted alongside explicit policy, disclosure evidence, instructor feedback, source fidelity, and rubric performance. However, AI-writing indicators should not be treated as definitive measures of authorship or cognition because detection tools can produce accuracy-bias trade-offs and may be especially problematic for multilingual or non-native English writers [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eFor engineering education, the broader contribution is a shift from detection to validity. Instead of asking only whether AI was used, instructors should ask whether the submitted critique still demonstrates the intended construct. The framework and joint displays in this paper offer a practical way to connect rubric evidence, text/style evidence, and reasoning evidence, consistent with validity arguments that require explicit warrants for score interpretation and use [\u003cspan additionalcitationids=\"CR12\" citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]-[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e].\u003c/p\u003e"},{"header":"VI. Implications for Assessment Design","content":"\u003cp\u003eFirst, rubrics should separate communication quality from reasoning depth. Communication should not compensate for a weak source-specific critique. Second, critique prompts should require evidence that is difficult to generate generically: named methods, specific results, limitations tied to figures or protocols, and explicit comparison across assigned papers. Third, if AI is permitted, students should submit an AI-use disclosure and a process memo. Finally, instructors should consider oral checks, annotated source maps, or AI-audit assignments that ask students to evaluate and correct AI-generated critiques. These recommendations are consistent with research showing that AI interaction patterns, authorial voice, and student perceptions of AI-based writing tools shape whether AI functions as a scaffold or substitute [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e], [\u003cspan additionalcitationids=\"CR20\" citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]-[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e"},{"header":"VII. Limitations","content":"\u003cp\u003eThe study has several limitations. The dataset came from one course context, and the phase comparison was not randomized. Critiques 7\u0026ndash;9 differed in both AI policy and technical topic, so phase effects cannot be interpreted as purely causal AI effects. The dataset contained rubric scores, instructor comments, and AI-writing indicators but did not include complete process traces, prompt histories, or validated automated measures of lexical diversity and cohesion. In addition, AI-writing indicators were treated as contextual evidence rather than definitive proof, consistent with current concerns about AI detector reliability and fairness [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e"},{"header":"VII. Conclusion","content":"\u003cp\u003eThis study examined how generative AI assistance altered graduate students\u0026rsquo; written paper critiques across nine assignments. AI-permitted critiques scored substantially higher than AI-prohibited critiques across all rubric domains, especially the Communication and Total score. Yet qualitative feedback showed that the same surface gains could coincide with generic synthesis, weaker source fidelity, and reduced methodological interrogation. These findings extend recent work on generative AI in higher education by locating the central problem in the validity of writing-based assessment rather than in detection alone [\u003cspan additionalcitationids=\"CR2 CR3 CR4 CR5 CR6 CR7\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]-[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], [\u003cspan additionalcitationids=\"CR12 CR13 CR14 CR15 CR16 CR17 CR18 CR19 CR20\" citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]-[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe main contribution is a sociocognitive validity framework for interpreting AI distortion in writing-based assessment. The framework shows that the central issue is not whether AI improves writing, but whether the written artifact still provides valid evidence of student cognition. For graduate engineering education, this means assessment systems must move beyond final-text grading toward transparent, process-aware, source-specific, and validity-oriented evaluation.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgment\u003c/h2\u003e \u003cp\u003eThe author acknowledges the anonymized students whose coursework and feedback records served as the basis for this classroom-based inquiry. Participant names and personally identifying details were removed from all manuscript tables, figures, and quoted feedback. Any final submission should include the appropriate institutional ethics, consent, or exemption statement before journal upload.\u003c/p\u003e\n\u003cp\u003eEthics Approval Statement\u003c/p\u003e\n\u003cp\u003eThis study involved analysis of anonymized course-based assessment records and instructor feedback. The study was reviewed and approved by the Research Ethics Committee. All data were anonymized prior to analysis, and no personally identifying student information is reported in the manuscript.\u003c/p\u003e\n\u003cp\u003eParticipant Consent Statement\u003c/p\u003e\n\u003cp\u003eParticipants provided consent for their anonymized coursework data, grading records, and instructor feedback to be used for research and publication purposes. All participant identifiers were removed, and the results are reported only in anonymized and aggregated form.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eKasneci E et al (2023) ChatGPT for good? On opportunities and challenges of large language models for education. Learn Individ Differ 103:102274\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCotton DRE, Cotton PA, Shipway JR (2024) Chatting and cheating: Ensuring academic integrity in the era of ChatGPT, Innov. Educ Teach Int 61(2):228\u0026ndash;239\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeng R, Jiang M, Yu X, Lu Y, Liu S (2025) Does ChatGPT enhance student learning? A systematic review and meta-analysis. Comput Educ 227:105224\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang J, Fan W (2025) The effect of ChatGPT on students\u0026rsquo; learning performance, perception, and higher-order thinking: A meta-analysis, Humanit. Soc Sci Commun 12:621\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBelkina M et al (2025) Implementing generative AI in higher education: A systematic review of case studies. Comput Educ : Artif Intell 8:100407\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbdallah N, Katmah R, Khalaf K, Jelinek HF (2025) Systematic review of ChatGPT in higher education. Soc Sci Humanit Open 12:101866\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZawacki-Richter O, Marin VI, Bond M, Gouverneur F (2019) Systematic review of research on AI applications in higher education. Int J Educ Technol High Educ 16:39\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiang W, Yuksekgonul M, Mao Y, Wu E, Zou J (2023) GPT detectors are biased against non-native English writers, Patterns, vol. 4, no. 7, p. 100779\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHodges TS et al (2019) Developing and examining validity evidence for a writing achievement rubric, Assessing Writing, vol. 41, pp. 26\u0026ndash;36\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePanadero E et al (2024) Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analysis. Educ Psychol Rev 36:21\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMessick S (1995) Validity of psychological assessment. Am Psychol 50(9):741\u0026ndash;749\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKane MT (2013) Validating the interpretations and uses of test scores. J Educ Meas 50(1):1\u0026ndash;73\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAERA, APA, and NCME (2014) Standards for Educational and Psychological Testing. AERA, Washington, DC, USA\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePratama AR et al (2025) The accuracy-bias trade-offs in AI text detection tools. npj Sci Learn\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKim J, Lee S-S, Detrick R et al (2025) Students-generative AI interaction patterns and impact on academic writing. J Comput High Educ\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAgarwal D, Naaman M, Vashistha A (2025) AI suggestions homogenize writing toward Western styles, Proc. CHI Conf. Human Factors Comput. Syst\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKobak D et al (2025) Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Sci Adv 11(8):eadt3813\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSourati Z et al (2026) The homogenizing effect of large language models on human expression and thought. Trends Cogn Sci\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu M, Zhang LJ, Biebricher C (2024) Investigating students\u0026rsquo; cognitive processes in generative AI-assisted composing and traditional writing. Comput Educ 211:104977\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmirjalili F et al (2024) A comparative analysis of AI-generated text and human authorship in academic writing. Front Educ 9:1347421\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBoillos M et al (2025) Student perspectives on AI-based language tools in university writing. J Writ Res\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVygotsky LS (1978) Mind in Society. Harvard Univ. Press, Cambridge, MA, USA\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFlower L, Hayes JR (1981) A cognitive process theory of writing, College Composition and Communication, vol. 32, no. 4, pp. 365\u0026ndash;387\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBereiter C, Scardamalia M (1987) The Psychology of Written Composition. Erlbaum, Hillsdale, NJ, USA\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity indices using direct judgments. Lang Assess Q 18(2):154\u0026ndash;170\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCrossley SA et al (2019) The Tool for the Automatic Analysis of Cohesion 2.0. Behav Res Methods 51:14\u0026ndash;30\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGarner J, Crossley S, Kyle K (2019) N-gram measures and L2 writing proficiency, System, vol. 80, pp. 176\u0026ndash;187\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSalomon G, Perkins DN, Globerson T (1991) Partners in cognition: Extending human intelligence with intelligent technologies. Educ Res 20(3):2\u0026ndash;9\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRisko EF, Gilbert SJ (2016) Cognitive offloading. Trends Cogn Sci 20(9):676\u0026ndash;688\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGerlich R (2025) AI tools in society: Impacts on cognitive offloading and the future of critical thinking, Societies, vol. 15, no. 1, p. 6\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eByrne D (2022) A worked example of Braun and Clarke\u0026rsquo;s approach to reflexive thematic analysis, Qual\u0026mdash;quant., vol. 56, pp. 1391\u0026ndash;1412\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFetters MD, Tajima C (2022) Joint displays of integrated data collection in mixed methods research. Int J Qual Methods, 21\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcCrudden MT et al (2021) Joint displays for mixed methods research in psychology, Methods Psychol., vol. 5, p. 100069\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFroyd JE (2018) Structured abstracts for the IEEE Transactions on Education, IEEE Trans\u0026mdash;Educ. 61(3):214\u0026ndash;217\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIEEE Education Society (2026) IEEE Transactions on Education Author Resources, [Online]. Available: IEEE Education Society website\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Ateneo de Davao University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"generative AI, large language models, engineering education, assessment validity, written critique, cognitive offloading, sociocognitive writing, mixed methods","lastPublishedDoi":"10.21203/rs.3.rs-9612303/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9612303/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eContribution: This article presents a mixed-methods study of generative AI assistance in graduate engineering paper critiques and operationalizes AI distortion as a validity problem in writing-based assessment.\u003c/p\u003e \u003cp\u003eBackground: Generative AI can improve fluency and organization, but it may also encourage overreliance, homogenized academic expression, cognitive offloading, and weakened assessment validity.\u003c/p\u003e \u003cp\u003eResearch Questions: The study asks how rubric performance, AI-writing indicators, and feedback-coded reasoning markers differ across AI-permitted and AI-prohibited critique phases.\u003c/p\u003e \u003cp\u003eMethodology: A convergent mixed-methods design was applied to 120 anonymized critique records from a graduate sustainable-energy course.\u003c/p\u003e \u003cp\u003eResults: AI-permitted critiques scored substantially higher than AI-prohibited critiques across all rubric domains, while qualitative feedback indicated that surface polish sometimes masked reduced methodological specificity and source fidelity.\u003c/p\u003e \u003cp\u003eConclusion: The results support a sociocognitive validity interpretation: AI can scaffold communication but may distort the evidentiary meaning of written critique artifacts when the intended construct is graduate-level disciplinary reasoning.\u003c/p\u003e","manuscriptTitle":"Distortion by Design? Examining How Generative AI Assistance Alters Graduate Students’ Writing Style and Critical Reasoning in Written Paper Critiques","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-11 06:24:44","doi":"10.21203/rs.3.rs-9612303/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"40cfdd31-2d19-4c68-9175-609b16e34629","owner":[],"postedDate":"May 11th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-11T06:24:44+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-11 06:24:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9612303","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9612303","identity":"rs-9612303","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}