Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep Tomohiro Ioku, Sachihiko Kondo, Yasuhisa Watanabe This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4469443/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This study investigates the performance of generative artificial intelligence (AI) in evaluating the acceptance of generative AI technologies within higher education guidelines, reflecting on the implications for educational policy and practice. Drawing on a dataset of guidelines from top-ranked universities, we compared generative AI evaluations with human evaluations, focusing on acceptance, performance expectancy, facilitating conditions, and perceived risk. Our study revealed a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI, suggesting that generative AI can accurately reflect human judgment in this context. Further, we found positive associations between ChatGPT-rated acceptance and performance expectancy and facilitating conditions, while a negative correlation with perceived risk. These results validate generative AI evaluation, which also extends the application of the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology framework from individual to institutional perspectives. Educational Psychology artificial intelligence higher education guidelines UTAUT Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction In Philip K. Dick’s visionary novel “ Do Androids Dream of Electric Sheep? ”, the line between human and artificial intelligence (AI) blurs, prompting readers to ponder the capabilities of AI. Inspired by the great work, our study explores the performance of AI in academia. Given the widespread debates on the use of generative AI such as ChatGPT at universities worldwide, we particularly focus on the performance of generative AI, in assessing the acceptance of generative AI in higher education policies. Specifically, moving beyond the novel’s speculative fiction, our study compares generative AI evaluation with human evaluation and validates it using Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework. Performance of AI Recent research across various disciplines demonstrates that generative AI technologies perform well in passing professional and academic exams. A growing body of medical research has examined the performance of AI in passing exams. In the United States Medical Licensing Exam (USMLE), generative AI performed at or near the passing threshold of 60% accuracy (Kung et al., 2023 ). Similarly, another study found that generative AI performed at a level comparable to a third-year medical student on USMLE Step 1 and Step 2 exams (Gilson et al., 2023 ). Further, a previous study found that generative AI achieved a 79.9% correct response rate on the Japanese Medical Licensing Exam, notably outperforming the average examinee by 17% on hard questions (Takagi et al., 2023 ). Furthermore, a prior study found that generative AI outperformed the Japanese medical residents on the General Medicine In-Training Examination, particularly in areas requiring detailed medical knowledge and difficult questions (Watari et al., 2023 ). Another body of research on engineering and computer sciences has examined the performance of AI in passing exams. Computer science research found that generative AI achieved a score that just met the passing of a computer science exam focusing on algorithms and data structures (Bordt & von Luxburg, 2023 ). Engineering research found that generative AI performed well across various tasks, including theoretical questions, programming, and practical circuit design, with a cumulative grade of 73% (Elder et al., 2023 ). A recent study found that generative AI was capable of solving simple math problems and addressing undergraduate-level mathematics questions (Frieder et al., 2024 ). Management and financial research has also examined the performance of AI in passing exams. Previous research found that in the Operations MBA final exam, generative AI offered correct answers with excellent explanations for basic questions. (Terwiesch, 2023 ). An experimental study found that generative AI performed exceptionally well in economics exams, scoring higher than the average college student in both microeconomics and macroeconomics tests (Geerling et al., 2023 ). A previous study also found that generative AI performed well in solving basic finance problems in solving basic undergraduate finance problems with an 85% accuracy rate (Yang & Stivers, 2024 ). Thus, recent studies in fields such as medicine, management, and engineering, provide evidence that AI technologies exhibit impressive performance in certain task completions, but there is little research concerning the performance of AI in education policy. The opportunities and challenges of adopting AI-generated content in educational policy are widely debated, and university guidelines reflect this divide (McDonald et al., 2024 ; Moorhouse et al., 2023 ). However, quantitatively assessing the acceptance of generative AI presents a challenge. To address this gap, we investigate the performance of AI in analyzing the text of guidelines, offering a method to measure the acceptance of generative AI in higher education. Specifically, this study examines the performance of generative AI in text evaluation by exploring the following two questions. Can generative AI evaluate the acceptance of generative AI as effectively as humans? Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk? Evaluating Acceptance of Generative AI: Generative AI and Human Perspectives For the first question, we argue that generative AI is capable of evaluating the acceptance of generative AI as effectively as humans given the previous literature on the performance of Ai in text evaluation. An experimental study examined the performance of generative AI feedback on writing and its preference among English as a new language students (Escalante et al., 2023 ). The experimental group received feedback from generative AI and the control group received feedback from their human tutor. The results showed that students who received feedback from generative AI did improve their writing skills comparable to students who received feedback from their human tutors. Those students were also split fairly evenly in their preferences between generative AI and human feedback, showing that each form of feedback has its own perceived benefits. Further, another study investigates the use of generative AI for automated essay scoring in assessing TOEFL essays (Mizumoto & Eguchi, 2023 ). Mizumoto and Eguchi ( 2023 ) compared AI-generated scores to human benchmarks and explored the effect of incorporating linguistic features on scoring accuracy. The results revealed that generative AI could effectively score essays with a level of accuracy and reliability, especially when combined with analysis of linguistic features. Furthermore, prior research explored the performance of generative AI in supporting English learning as a foreign language teacher (Guo & Wang, 2023 ). Guo and Wang ( 2023 ) asked teachers to evaluate both generative AI’s feedback on student writing and human teachers’ feedback. The results showed that generative AI generated more feedback than teachers, distributing attention evenly across content, organization, and language aspects. Overall, these findings suggest that generative AI performs certain tasks including text evaluation as effectively as humans, if not more so in some aspects. Hypothesis 1 Human-rated acceptance of generative AI is positively associated with generative AI-rated acceptance of generative AI Validating Generative AI Evaluation: Expectancy, Conditions, and Risk As for the second question, we argue that we can validate its evaluation by confirming its correlations with evaluations of other aspects that research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework focuses (Abdaljaleel et al., 2024 ; Ben Arfi et al., 2021 ; Davis, 1989 ; Polyportis & Pahos, 2024 ; Venkatesh et al., 2003 ). Venkatesh et al. ( 2003 ) developed UTAUT as a thorough synthesis of previous technology acceptance studies, by reviewing the existing models, including TAM. UTAUT includes four key constructs: performance expectancy, effort expectancy, social influence, and facilitating condition. Especially, we focus on perceived risk as well as performance expectancy and facilitating conditions. Our focus was informed by recent discourse and research articles in the respective contexts, in which universities expect generative AI to improve teaching and learning activities and prepare themselves for support and resources for it, but are concerned about academic integrity (Abdaljaleel et al., 2024 ; Crompton & Burke, 2024 ; Harvard University, 2023 ; Imperial College London, 2023 ; Stanford University, 2023 ). Performance expectancy is the extent to which using a technology will provide benefits to individuals in performing certain activities (Venkatesh et al., 2003 , 2012 ). In the context of generative AI in higher educational policies, benefits involve the degree to which generative AI tools will support and improve teaching and learning activities, contributing to educational outcomes in academic research and activities. A large body of research has shown that performance expectancy can be a determinant of AI technology acceptance (Andrews et al., 2021 ; Chatterjee & Bhattacharjee, 2020 ; Guggemos et al., 2020 ; Raffaghelli et al., 2022 ). A recent survey showed that performance expectancy had a positive effect on students’ attitudes towards the generative AI (Foroughi et al., 2023 ). That is, when students believed that using the chatbot would bring them benefits such as convenience, efficiency, or effectiveness in their educational tasks, they had a positive attitude toward the chatbot. Provided that performance expectancy has a positive effect on accepting new technologies, as universities expect AI to greatly assist in personalized learning, efficient data analysis, and creative academic endeavors, they should accept generative AI. Therefore, we hypothesize, Hypothesis 2 Performance expectancy is positively associated with acceptance of generative AI Facilitating conditions are defined as individuals’ perceptions of the availability of resources and support necessary for performing activities (Brown & Venkatesh, 2005 ; Venkatesh et al., 2003 ). Within the context of generative AI in higher education policies, these resources and support specifically include the extent to which universities provide the essential technical, academic, and policy frameworks required for the effective integration of generative AI tools. Numerous studies have suggested that facilitating conditions are another determinant of individuals’ acceptance of AI technologies (Cabrera-Sánchez et al., 2021 ; Chatterjee & Bhattacharjee, 2020 ; Kwak et al., 2022 ). More recently, a survey study found that facilitating conditions had the strongest effect on students’ intentions and actual usage of generative AI (Habibi et al., 2023 ), indicating that when students had access to the necessary resources for using generative AI, such as a laptop and internet connection, they were not only more intent on using generative AI but also used it more frequently. Given these findings on facilitating conditions and technology acceptance, universities that provide the required resources and support for leveraging generative AI should endorse its acceptance. Accordingly, we hypothesize, Hypothesis 3 Facilitating conditions are positively associated with acceptance of generative AI Perceived risk refers to an individual’s subjective evaluation of the potential negative outcomes associated with a specific action (Abdaljaleel et al., 2024 ; Ben Arfi et al., 2021 ; Zhang et al., 2019 ). In the context of generative AI in higher education policies, such risk covers the degree to which a university perceives potential risks associated with the use of generative AI tools in academics such as cheating, misinformation use, and copyright violation, which undermine student learning. Some studies have suggested that perceived risk is a potential determinant of individuals’ acceptance of new technologies (Ben Arfi et al., 2021 ; Zhang et al., 2019 ). Recent research found the negative effect of perceived risk on acceptance of generative AI, which showed that when students and faculty member believed that using generative AI for answering academic queries is risky, they did not think that generative AI in higher education is good for society (Jain & Raghuram, 2024 ). A multinational study replicated this result (Abdaljaleel et al., 2024 ). If perceived risk has a negative effect on accepting new technologies, as universities perceives potential risks associated with the use of generative AI tools in higher education, they should be cautious about endorsing its acceptance. Thus, we hypothesize, Hypothesis 4 Perceived risk is negatively associated with acceptance of generative AI Present Study The present study rigorously examines the performance of generative AI, a generative AI model, in assessing the acceptance of generative AI in guidelines from top-ranked universities worldwide. We aim to understand how well generative AI evaluation corresponds with human judgments in terms of acceptance, performance expectancy, facilitating conditions, and perceived risk associated with the use of generative AI. The purpose is twofold: first, to investigate the performance of generative AI in accurately reflecting human perspectives on the adoption of AI within educational policies; and second, to examine the validity of generative AI evaluation from existing technology acceptance frameworks, including TAM and UTAUT. Method To ensure a systematic and efficient selection of guidelines, we used the Quacquarelli Symonds (QS) World University Rankings 2023 to identify universities for inclusion ( https://www.topuniversities.com/qs-world-university-rankings ). We conducted an a priori power with GPower (Faul et al., 2007 ). A minimum sample size of 26 was required to detect a large effect with 80% statistical power and an α of 5% in correlation analysis. Based on these criteria, we decided to collect guidelines from the top 50 ranked universities. The research ethics committee at the Center for International Education and Exchange of Osaka University approved the study procedure prior to data collection. Appendixes in this study can be found on the Open Science Framework site: https://osf.io/7qshb/?view_only=2de75d6f87e24a7ab3633c8272c8a058 . The datasets used in this study are available from the corresponding author on a reasonable request. Keyword-based Search This study performed a keyword-based web search of the Google search engine in private browsing mode, after log-out from personal accounts and erasure of all web cookies and history (Jobin et al., 2019 ; Piasecki et al., 2018 ). We used the following keywords during the search: [generative AI guidelines], [generative AI policies], [generative AI statementes], and [UNIVERSITY NAME]. [UNIVERSITY NAME] was entered as the name of a university in the top 50 of the QS university rankings. The Google results up to the 200th listings for each Google search were followed and screened for generative AI guidelines only. Within these link listings, we identified 35 non-duplicate documents. We continued to monitor the literature in parallel with the data analysis and until June 30, 2023, to retrieve eligible documents that were released after our search was completed. Inclusion Criteria Based on our inclusion criteria, target documents (including statements, guidelines, and notices) included in the final synthesis were (i) written in English; (ii) issued by institutional entities from universities; (iii) mentioned explicitly in their title/description to generative AI or related notions, (iv) expressed a normative stance defined as a preference for a particular course of action related to generative AI. Assessment of Acceptance of Generative AI We used a generative AI to measure the acceptance of generative AI in higher education following two steps (i.e., Chat GPT). To measure the acceptance of generative AI in higher education, we operationally defined it as extent to which an institution supports or restricts the use of generative AI technologies by students in academic settings, as reflected in the university’s official guidelines and policies. This definition was derived from previous studies that linked a university’s stance on generative AI to academic integrity (Bin-Nashwan et al., 2023 ; Cotton et al., 2024 ; Eke, 2023 ). Based on this definition, we created four levels of acceptance (See the details in Appendix 1): Level 1, strongly against, prohibits AI use without permission, treating violations as misconduct. Level 2, against, views unauthorized AI as plagiarism, requiring ethical adherence and permission. Level 3, neutral, neither endorses nor bans AI, stressing academic integrity and ethical usage with necessary permissions. Level 4, supportive, encourages responsible AI use, acknowledging its educational value, with guidelines for proper application in assessments. Next, we evaluated the acceptance of generative AI in higher education using two methods. First, for the human rating, we had three native English undergraduate students independently review 35 university guidelines (two students were male), categorizing them into predefined levels of acceptance. We assessed the reliability of their combined ratings and found it to be acceptable (α = .73). Thus, we calculated the average of these ratings to represent the human-rated acceptance of generative AI in higher education. Second, for the generative AI rating, we asked ChatGPT for classifying each university guideline into the established levels of acceptance (See an example; https://chat.openai.com/share/ae65480c-7721-42fb-9901-493c5dba542a ). To ensure accuracy, ChatGPT evaluated each guideline three times. The reliability of these ratings was also found to be high (α = .82). We used the average of these ratings as the ChatGPT-rated acceptance of generative AI in higher education. Assessment of Performance Expectancy, Facilitating Conditions, Perceived Risk To measure performance expectancy, we operationally defined it as the degree to which a university perceives potential benefits associated with the use of generative AI tools in academic settings (Chatterjee & Bhattacharjee, 2020 ; Foroughi et al., 2023 ; Raffaghelli et al., 2022 ). Based on this definition, we created four levels of performance expectancy (See the details in Appendix 2). Level 1, Low Expectancy, sees minimal benefits, remaining skeptical of significant improvements. Level 2, Cautious Expectancy, holds a cautious outlook, expecting moderate enhancements in specific areas. Level 3, Moderate Expectancy, anticipates noticeable gains in performance and efficiency, viewing AI as a supportive tool for academic tasks. At Level 4, High Expectancy, the university foresees substantial improvements in education and research, expecting AI to advance personalized learning, data analysis, and creative academic processes. To measure facilitating conditions, our study operationally defined it as the extent to which universities provide the necessary technical, academic, and policy support for the effective integration of generative AI tools in academic settings (Cabrera-Sánchez et al., 2021 ; Chatterjee & Bhattacharjee, 2020 ; Habibi et al., 2023 ). Based on this definition, we created four levels of facilitating conditions (See the details in Appendix 3). At Level 1, Minimal Facilitation, there’s hardly any support or AI resources. Level 2, Basic Support, offers basic AI tools and a general guide for use, but lacks in-depth resources and training. At Level 3, Supportive Infrastructure, the support is better, with good access to AI technology, some specialized training, and guidelines for ethical use. Level 4, Highly Facilitative, features a strong infrastructure with advanced AI labs, extensive training programs, and comprehensive support policies, ensuring widespread access to AI for all academic activities. To measure perceived risk, this study operationally defined it as the degree to which a university perceives potential risks associated with the use of generative AI tools in academic settings (Abdaljaleel et al., 2024 ; Ben Arfi et al., 2021 ; Jain & Raghuram, 2024 ). Based on this definition, we created four levels of perceived risk (See the details in Appendix 4). Level 1, Embracing with Guidance, shows the least concern for risks, focusing on ethical use and responsible AI integration with supportive guidelines. Level 2, Conditionally Accepting, balances potential risks and benefits, allowing AI use under ethical and citation standards. Level 3, Cautiously Permissive, acknowledges higher risks, implementing stricter guidelines to prevent misuse. At Level 4, Highly Cautious, the university enforces severe restrictions due to concerns like academic dishonesty, diminishing learning quality, and intellectual property issues, with strict consequences for violations. Similar to ChatGPT rating in acceptance of generative AI, we asked ChatGPT to categorize each set of guidelines into predefined levels concerning performance expectancy, facilitating conditions, and perceived risk. ChatGPT conducted this classification task three times for each guideline to ensure precision. The reliability of ChatGPT’s ratings across these evaluations was satisfactory, with .92 for performance expectancy, .71 for facilitating conditions, and .89 for perceived risk. We used the average ratings from these assessments. Results Correlation Analysis To test our hypotheses, we conducted Pearson correlation analysis between human-rated acceptance, ChatGPT-rated acceptance, performance expectancy, facilitative conditions, and perceived risk. Figure 1 presents overall correlation matrix for these variables. As Fig. 2 showed, there was a strongly positive correlation between ChatGPT-rated acceptance and human-rated acceptance ( r = .85, p < .001). This means that generative AI evaluates texts as accurately as humans. As shown in Fig. 3, ChatGPT-rated acceptance was also positively correlated with performance expectancy ( r = .53, p = .001). This indicates that as universities expect AI to enhance teaching and learning activities, they become more accepting of generative AI. Figure 4 presented a similar correlation between ChatGPT-rated acceptance and facilitating conditions ( r = .48, p = .003). That is, universities that provide the resources for using generative AI are more accepting of it. Finally, unlike those aspects of generative AI (Fig. 5), ChatGPT-rated acceptance was negatively correlated with perceived risk ( r = − .60, p < .001). This suggests that universities aware of the potential risks associated with using generative AI tools in higher education should be cautious in endorsing their acceptance. Discussion We investigated the performance of generative AI in assessing the acceptance of its own use within higher education guidelines. The results confirmed all four hypotheses: Hypothesis 1 was supported by a strong positive correlation between human-rated and ChatGPT-rated acceptance. This study supported Hypothesis 2 by finding a positive relationship between performance expectancy and acceptance. Hypothesis 3 was confirmed by the positive correlation between facilitating conditions and acceptance. Finally, Hypothesis 4 , which posited a negative correlation between perceived risk and acceptance, was supported. These findings collectively show the performance of generative AI in text evaluation. Evaluating Acceptance of Generative AI Building on a growing body of research on the performance of AI, our study addresses the first research question: Can generative AI evaluate the acceptance of generative AI as effectively as humans? Previous studies have shown that AI, including generative AI, performs impressively in various academic and professional exams, equalling or exceeding human performance across fields such as medicine, engineering, and finance (Frieder et al., 2024 ; Gilson et al., 2023 ; Kung et al., 2023 ; Takagi et al., 2023 ; Terwiesch, 2023 ; Watari et al., 2023 ; Yang & Stivers, 2024 ). Moreover, prior studies find that generative AI can effectively perform tasks similar to humans, such as providing writing feedback, evaluating essays, and supporting language learning (Escalante et al., 2023 ; Guo & Wang, 2023 ; Mizumoto & Eguchi, 2023 ). Our study first found a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI in educational policies. Thus, in response to our research question, the strong positive correlation between ChatGPT-rated and human-rated acceptance provides evidence for the notion that generative AI can evaluate the acceptance of generative AI within educational policies as effectively as human evaluators. Validating Generative AI Evaluation Our study also contributes to the extensive research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework by investigating the second research question: Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk? Previous studies have suggested that performance expectancy, facilitating conditions, and perceived risk are associated with the acceptance of AI technologies, underlining their role in technology adoption decisions (Ben Arfi et al., 2021 ; Cabrera-Sánchez et al., 2021 ; Chatterjee & Bhattacharjee, 2020 ; Guggemos et al., 2020 ; Kwak et al., 2022 ; Zhang et al., 2019 ). Recent studies indicate that performance expectancy and facilitating conditions are positively associated with students’ attitudes and usage of generative AI, while perceived risk is negatively associated with the acceptance of generative AI in education (Foroughi et al., 2023 ; Habibi et al., 2023 ; Jain & Raghuram, 2024 ). Our study found a positive correlation between ChatGPT-rated acceptance of generative AI and performance expectancy, indicating that universities’ expectations of AI enhancing teaching and learning activities are associated with greater acceptance of generative AI. Similarly, there was a positive correlation between ChatGPT-rated acceptance of generative AI and facilitating conditions, which suggests that universities equipped with the necessary resources for using generative AI are likely to accept it. Conversely, the negative correlation between ChatGPT-rated acceptance of generative AI and perceived risk shows that universities aware of the potential risks for generative AI in education are reluctant to accept it. These results expand the application of TAM or UTAUT framework from individual to institutional perspectives, indicating that generative AI can accurately evaluate and distinguish between various aspects of generative AI in educational policies. This answers our second research question, validating generative AI’s assessments against key criteria such as performance expectancy, facilitating conditions, and perceived risk. Practical Implications This study has implications for those who currently use or intend to use generative AI because it provides insights into the performance of AI technologies. As generative AI becomes more prevalent in society, expectations have risen substantially. Several articles highlight that emerging technologies such as generative AI often experience stages of initial enthusiasm, followed by disappointment, and finally, practical application, cautioning against overestimating their immediate impact (Eulerich et al., 2023 ). For some users, their expectations may not be fulfilled probably due to too much expectation. Here is a question; What can generative AI perform and to what extent? Addressing the question is crucial to avoid betraying users’ expectations. Our study answers this question in text evaluation by demonstrating that generative AI can evaluate guidelines on four levels, akin to humans, thereby contributing to the practical application of generative AI in society. Furthermore, our findings inform research practices on educational policy for generative AI. Educational policy research collected and quantitatively analyzed policy documents and guidelines about generative AI from higher education institutions (McDonald et al., 2024 ). Another research categorizes the guidelines into different nine fields, including authorship, acknowledgment, plagiarism, advice on assessment design and tasks, detection of GAI use, responsible agent, proper use, improper use, and communication with students (Moorhouse et al., 2023 ). These studies help us understand how higher education institutions are navigating the integration of generative AI technologies into teaching activities and the development of guidelines to support instructors. However, we still do not know the degree to which each higher education institution accepts generative AI. Then researchers are required to rate it from guidelines, but the process of human rating of texts can be fraught with issues that fatigue, subjectivity, and inconsistency may produce unreliable results (Hussein et al., 2019 ; Mizumoto & Eguchi, 2023 ). Generative AI provides a faster, more consistent alternative, potentially transforming rating method, as the reliability of the ChatGPT rating was higher than the reliability of the human rating in this study. Limitations We acknowledge some limitations of this study. First, this study focuses on the top 50 universities according to the QS World University Rankings, which may not fully represent the diverse range of higher education institutions globally. As such, the findings may not be applicable to smaller, less research-intensive institutions. Second, this study captures a snapshot of university policies about generative AI at a specific timeframe. Given the fast-paced evolution of both AI technology and societal norms towards it, these policies and guidelines are subject to change. Therefore, the findings might quickly become outdated, limiting their long-term relevance. Third, while our study effectively quantifies acceptance levels and correlates them with various factors, we did not focus on qualitative aspects of university guidelines and policies toward generative AI. The rich context underlying these documents could provide deeper insights into institutional stances, which our quantitative approach might not fully capture. Conclusion This study demonstrates that generative AI can effectively evaluate the acceptance of generative AI in higher education guidelines, corresponding to human judgments. The positive correlations between generative AI-rated acceptance and performance expectancy and facilitating conditions, along with its negative correlation with perceived risk, validate the generative AI evaluations within the frameworks of technology acceptance theories. These findings highlight the performance of AI in assisting policy analysis and decision-making in higher educational contexts. Declarations Acknowledgements We would like to thank our research collaborators. Funding This research received no specific grant from any funding agency. Availability of Data and Materials The datasets used in this study are available from the corresponding author on reasonable request. References Abdaljaleel, M., Barakat, M., Alsanafi, M., Salim, N. A., Abazid, H., Malaeb, D., Mohammed, A. H., Hassan, B. A. R., Wayyes, A. M., Farhan, S. S., Khatib, S. El, Rahal, M., Sahban, A., Abdelaziz, D. H., Mansour, N. O., AlZayer, R., Khalil, R., Fekih-Romdhane, F., Hallit, R., … Sallam, M. (2024). A multinational study on the factors influencing university students’ attitudes and usage of ChatGPT. Scientific Reports , 14 , 1983. https://doi.org/10.1038/s41598-024-52549-8 Andrews, J. E., Ward, H., & Yoon, J. (2021). UTAUT as a model for understanding intention to adopt AI and related technologies among librarians. Journal of Academic Librarianship , 47 , 102437. https://doi.org/https://doi.org/10.1016/j.acalib.2021.102437 Ben Arfi, W., Ben Nasr, I., Khvatova, T., & Ben Zaied, Y. (2021). Understanding acceptance of eHealthcare by IoT natives and IoT immigrants: An integrated model of UTAUT, perceived risk, and financial cost. Technological Forecasting and Social Change , 163 , 120437. https://doi.org/https://doi.org/10.1016/j.techfore.2020.120437 Bin-Nashwan, S. A., Sadallah, M., & Bouteraa, M. (2023). Use of ChatGPT in academia: Academic integrity hangs in the balance. Technology in Society , 75 , 102370. https://doi.org/https://doi.org/10.1016/j.techsoc.2023.102370 Bordt, S., & von Luxburg, U. (2023). Chatgpt participates in a computer science exam. ArXiv Preprint ArXiv. 2303.09461. Brown, S. A., & Venkatesh, V. (2005). Model of adoption of technology in households: A baseline model test and extension incorporating household life cycle. MIS Quarterly , 29 , 399–426. https://doi.org/10.2307/25148690 Cabrera-Sánchez, J.-P., Villarejo-Ramos, Á. F., Liébana-Cabanillas, F., & Shaikh, A. A. (2021). Identifying relevant segments of AI applications adopters – Expanding the UTAUT2’s variables. Telematics and Informatics , 58 , 101529. https://doi.org/https://doi.org/10.1016/j.tele.2020.101529 Chatterjee, S., & Bhattacharjee, K. K. (2020). Adoption of artificial intelligence in higher education: a quantitative analysis using structural equation modelling. Education and Information Technologies , 25 (5), 3443–3463. https://doi.org/10.1007/s10639-020-10159-7 Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International , 61 , 228–239. https://doi.org/10.1080/14703297.2023.2190148 Crompton, H., & Burke, D. (2024). The Educational Affordances and Challenges of ChatGPT: State of the Field. TechTrends , 68 , 380–392. https://doi.org/10.1007/s11528-024-00939-0 Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly , 13 , 319–340. https://doi.org/10.2307/249008 Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology , 13 , 100060. https://doi.org/https://doi.org/10.1016/j.jrt.2023.100060 Elder, C., Pozek, G., Horine, S., Tripaldelli, A., & Butka, B. (2023). Can artificial intelligence pass a sophomore level digital design laboratory? SoutheastCon 2023 , 861–868. https://doi.org/10.1109/SoutheastCon51012.2023.10115116 Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education , 20 , 57. https://doi.org/10.1186/s41239-023-00425-2 Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2023). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. SSRN Electronic Journal . http://dx.doi.org/10.2139/ssrn.4452175 Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods , 39 , 175–191. https://doi.org/10.3758/BF03193146 Foroughi, B., Senali, M. G., Iranmanesh, M., Khanfar, A., Ghobakhloo, M., Annamalai, N., & Naghmeh-Abbaspour, B. (2023). Determinants of iIntention to use ChatGPT for educational purposes: Findings from PLS-SEM and fsQCA. International Journal of Human–Computer Interaction , 1–20. https://doi.org/10.1080/10447318.2023.2226495 Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., & Berner, J. (2024). Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems , 36 , 699--744. Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? American Economist , 68 , 233–245. https://doi.org/10.1177/05694345231169654 Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education , 9 , e45312. https://doi.org/10.2196/45312 Guggemos, J., Seufert, S., & Sonderegger, S. (2020). Humanoid robots in higher education: Evaluating the acceptance of Pepper in the context of an academic writing course using the UTAUT. British Journal of Educational Technology , 51 , 1864–1883. https://doi.org/https://doi.org/10.1111/bjet.13006 Guo, K., & Wang, D. (2023). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Education and Information Technologies . https://doi.org/10.1007/s10639-023-12146-0 Habibi, A., Muhaimin, M., Danibao, B. K., Wibowo, Y. G., Wahyuni, S., & Octavia, A. (2023). ChatGPT in higher education learning: Acceptance and use. Computers and Education: Artificial Intelligence , 5 , 100190. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100190 Harvard University. (2023). Initial guidelines for using ChatGPT and other generative AI tools at Harvard. Retrieved from https://huit.harvard.edu/news/ai-guidelines (July, 13, 2023). Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science , 5 , e208. Imperial College London. (2023). Generative AI tools guidance: College guidance on the use of generative AI tools. Retrieved from https://www.imperial.ac.uk/about/leadership-and-strategy/provost/vice-provost-education/generative-ai-tools-guidance/ (March, 1, 2023). Jain, K. K., & Raghuram, J. N. V. (2024). Gen-AI integration in higher education: Predicting intentions using SEM-ANN approach. Education and Information Technologies . https://doi.org/10.1007/s10639-024-12506-4 Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence , 9 , 389–399. https://doi.org/10.1038/s42256-019-0088-2 Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health , 2 , e0000198. https://doi.org/10.1371/journal.pdig.0000198 Kwak, Y., Seo, Y. H., & Ahn, J.-W. (2022). Nursing students’ intent to use AI-based healthcare technology: Path analysis using the unified theory of acceptance and use of technology. Nurse Education Today , 119 , 105541. https://doi.org/https://doi.org/10.1016/j.nedt.2022.105541 McDonald, N., Johri, A., Ali, A., & Hingle, A. (2024). Generative artificial intelligence in higher education: Evidence from an analysis of institutional policies and guidelines. ArXiv Preprint ArXiv:2402.01659 . Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics , 2 , 100050. https://doi.org/https://doi.org/10.1016/j.rmal.2023.100050 Moorhouse, B. L., Yeo, M. A., & Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world’s top-ranking universities. Computers and Education Open , 5 , 100151. https://doi.org/https://doi.org/10.1016/j.caeo.2023.100151 Piasecki, J., Waligora, M., & Dranseika, V. (2018). Google search as an additional source in systematic reviews. Science and Engineering Ethics , 24 , 809–810. https://doi.org/10.1007/s11948-017-0010-4 Polyportis, A., & Pahos, N. (2024). Understanding students’ adoption of the ChatGPT chatbot in higher education: the role of anthropomorphism, trust, design novelty and institutional policy. Behaviour & Information Technology , 1–22. https://doi.org/10.1080/0144929X.2024.2317364 Raffaghelli, J. E., Rodríguez, M. E., Guerrero-Roldán, A.-E., & Bañeres, D. (2022). Applying the UTAUT model to explain the students’ acceptance of an early warning system in Higher Education. Computers & Education , 182 , 104468. https://doi.org/https://doi.org/10.1016/j.compedu.2022.104468 Stanford University. (2023). Generative AI policy guidance. Retrieved from https://communitystandards.stanford.edu/generative-ai-policy-guidance (February, 16, 2023). Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023). Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: Comparison study. JMIR Medical Education , 9 , e48002. https://doi.org/10.2196/48002 Terwiesch, C. (2023). Would chat GPT3 get a Wharton MBA: A prediction based on its performance in the operations management course. Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly , 27 , 425–478. https://doi.org/10.2307/30036540 Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly , 36 , 157–178. https://doi.org/10.2307/41410412 Watari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., & Tokuda, Y. (2023). Performance comparison of ChatGPT-4 and Japanese medical residents in the general medicine in-training examination: Comparison study. JMIR Medical Education , 9 , e52202. https://doi.org/10.2196/52202 Yang, C., & Stivers, A. (2024). Investigating AI languages’ ability to solve undergraduate finance problems. Journal of Education for Business , 99 , 44–51. https://doi.org/10.1080/08832323.2023.2253963 Zhang, T., Tao, D., Qu, X., Zhang, X., Lin, R., & Zhang, W. (2019). The roles of initial trust and perceived risk in public’s acceptance of automated vehicles. Transportation Research Part C: Emerging Technologies , 98 , 207–220. https://doi.org/https://doi.org/10.1016/j.trc.2018.11.018 Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4469443","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":306166894,"identity":"90e74248-1fe0-4a43-b555-36257b87416f","order_by":0,"name":"Tomohiro Ioku","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAt0lEQVRIiWNgGAWjYFACxgbGBgYbIIO5gYHhABEaeCBa0hgY2BiJ1gKyh+EwCVrsGZgbH85sOy9ncL+x8cGHMwzy/GIE9AEd1my4se22scExIGPGDQbDmbMTCGppk3zYdjtxwzHGNmmeDwwJBreJ03KOVC0b2w5AtdwgRssBkBfOJRtLHksEMs5IEPYLewP7w4c9ZXZyfIcPH3zw4ZiNPL80AS0M8g9QuBIElI+CUTAKRsEoIAoAAFynRHxbmrdsAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-5499-6470","institution":"Osaka University","correspondingAuthor":true,"prefix":"","firstName":"Tomohiro","middleName":"","lastName":"Ioku","suffix":""},{"id":306166900,"identity":"57d62938-7ca0-4b68-91ae-e98d0a15fd13","order_by":1,"name":"Sachihiko Kondo","email":"","orcid":"","institution":"Osaka University","correspondingAuthor":false,"prefix":"","firstName":"Sachihiko","middleName":"","lastName":"Kondo","suffix":""},{"id":306166903,"identity":"eabb3de3-1bb3-4aec-95f1-775498c7a068","order_by":2,"name":"Yasuhisa Watanabe","email":"","orcid":"","institution":"University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Yasuhisa","middleName":"","lastName":"Watanabe","suffix":""}],"badges":[],"createdAt":"2024-05-24 01:57:53","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4469443/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4469443/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":57296753,"identity":"d8d23561-f457-4f31-a542-f6d60c33ba08","added_by":"auto","created_at":"2024-05-28 19:59:32","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":43735,"visible":true,"origin":"","legend":"\u003cp\u003eCorrelation Matrix for Focal Variables. Note. HA = Human-rated Acceptance of Generative AI, GPTA = ChatGPT-rated Acceptance of Generative AI, PE = Performance Expectancy, FC = Facilitating Conditions, PR = Perceived Risk. The boxes show pairs of variables. The numbers inside the boxes range from -1.0 to 1.0, representing the strength and direction of the correlation between the variables.\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/8943feeeafe80bd27ed96563.png"},{"id":57296754,"identity":"05dc3e18-a943-4962-ac69-d1ba323937a2","added_by":"auto","created_at":"2024-05-28 19:59:32","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":136467,"visible":true,"origin":"","legend":"\u003cp\u003eScatter Plot for ChatGPT-rated Acceptance and Human-rated Acceptance of Generative AI\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/1913cff508df21988f8cc620.png"},{"id":57297093,"identity":"a4ef67a5-be0e-4814-a29b-d261232c00ba","added_by":"auto","created_at":"2024-05-28 20:07:32","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":149048,"visible":true,"origin":"","legend":"\u003cp\u003eScatter Plot for ChatGPT-rated Acceptance and Performance Expectancy of Generative AI\u003c/p\u003e","description":"","filename":"Figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/7e2b9d827d739597bbacff54.png"},{"id":57297092,"identity":"4ea68062-91e7-47e8-ad44-3baf7e58fd52","added_by":"auto","created_at":"2024-05-28 20:07:32","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":146191,"visible":true,"origin":"","legend":"\u003cp\u003eScatter Plot for ChatGPT-rated Acceptance and Facilitating Conditions of Generative AI\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/fd865289bbcdcc4fe6162300.png"},{"id":57296756,"identity":"1e400df8-f2de-4a42-979a-f9a9c281d1da","added_by":"auto","created_at":"2024-05-28 19:59:32","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":4493,"visible":true,"origin":"","legend":"\u003cp\u003eLegend not included with this version.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/027930a0ea1e51b39e510b50.png"},{"id":57297094,"identity":"d1841b4d-912a-4378-9b8e-b397bb1a8586","added_by":"auto","created_at":"2024-05-28 20:07:37","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":912932,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4469443/v1/82678636-1533-4234-8ab8-2d9fe65d5a80.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003ePerformance of Artificial Intelligence: Does artificial intelligence dream of electric sheep\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn Philip K. Dick\u0026rsquo;s visionary novel \u0026ldquo;\u003cem\u003eDo Androids Dream of Electric Sheep?\u003c/em\u003e\u0026rdquo;, the line between human and artificial intelligence (AI) blurs, prompting readers to ponder the capabilities of AI. Inspired by the great work, our study explores the performance of AI in academia. Given the widespread debates on the use of generative AI such as ChatGPT at universities worldwide, we particularly focus on the performance of generative AI, in assessing the acceptance of generative AI in higher education policies. Specifically, moving beyond the novel\u0026rsquo;s speculative fiction, our study compares generative AI evaluation with human evaluation and validates it using \u003cem\u003eTechnology Acceptance Model\u003c/em\u003e (TAM) or the \u003cem\u003eUnified Theory of Acceptance and Use of Technology\u003c/em\u003e (UTAUT) framework.\u003c/p\u003e\n\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e\n\u003ch3\u003ePerformance of AI\u003c/h3\u003e\n\u003cp\u003eRecent research across various disciplines demonstrates that generative AI technologies perform well in passing professional and academic exams. A growing body of medical research has examined the performance of AI in passing exams. In the United States Medical Licensing Exam (USMLE), generative AI performed at or near the passing threshold of 60% accuracy (Kung et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Similarly, another study found that generative AI performed at a level comparable to a third-year medical student on USMLE Step 1 and Step 2 exams (Gilson et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Further, a previous study found that generative AI achieved a 79.9% correct response rate on the Japanese Medical Licensing Exam, notably outperforming the average examinee by 17% on hard questions (Takagi et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Furthermore, a prior study found that generative AI outperformed the Japanese medical residents on the General Medicine In-Training Examination, particularly in areas requiring detailed medical knowledge and difficult questions (Watari et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eAnother body of research on engineering and computer sciences has examined the performance of AI in passing exams. Computer science research found that generative AI achieved a score that just met the passing of a computer science exam focusing on algorithms and data structures (Bordt \u0026amp; von Luxburg, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Engineering research found that generative AI performed well across various tasks, including theoretical questions, programming, and practical circuit design, with a cumulative grade of 73% (Elder et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). A recent study found that generative AI was capable of solving simple math problems and addressing undergraduate-level mathematics questions (Frieder et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eManagement and financial research has also examined the performance of AI in passing exams. Previous research found that in the Operations MBA final exam, generative AI offered correct answers with excellent explanations for basic questions. (Terwiesch, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). An experimental study found that generative AI performed exceptionally well in economics exams, scoring higher than the average college student in both microeconomics and macroeconomics tests (Geerling et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). A previous study also found that generative AI performed well in solving basic finance problems in solving basic undergraduate finance problems with an 85% accuracy rate (Yang \u0026amp; Stivers, \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eThus, recent studies in fields such as medicine, management, and engineering, provide evidence that AI technologies exhibit impressive performance in certain task completions, but there is little research concerning the performance of AI in education policy. The opportunities and challenges of adopting AI-generated content in educational policy are widely debated, and university guidelines reflect this divide (McDonald et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Moorhouse et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). However, quantitatively assessing the acceptance of generative AI presents a challenge. To address this gap, we investigate the performance of AI in analyzing the text of guidelines, offering a method to measure the acceptance of generative AI in higher education. Specifically, this study examines the performance of generative AI in text evaluation by exploring the following two questions. Can generative AI evaluate the acceptance of generative AI as effectively as humans? Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk?\u003c/p\u003e\n\u003c/div\u003e\n\u003ch3\u003eEvaluating Acceptance of Generative AI: Generative AI and Human Perspectives\u003c/h3\u003e\n\u003cp\u003eFor the first question, we argue that generative AI is capable of evaluating the acceptance of generative AI as effectively as humans given the previous literature on the performance of Ai in text evaluation. An experimental study examined the performance of generative AI feedback on writing and its preference among English as a new language students (Escalante et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). The experimental group received feedback from generative AI and the control group received feedback from their human tutor. The results showed that students who received feedback from generative AI did improve their writing skills comparable to students who received feedback from their human tutors. Those students were also split fairly evenly in their preferences between generative AI and human feedback, showing that each form of feedback has its own perceived benefits. Further, another study investigates the use of generative AI for automated essay scoring in assessing TOEFL essays (Mizumoto \u0026amp; Eguchi, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Mizumoto and Eguchi (\u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e) compared AI-generated scores to human benchmarks and explored the effect of incorporating linguistic features on scoring accuracy. The results revealed that generative AI could effectively score essays with a level of accuracy and reliability, especially when combined with analysis of linguistic features. Furthermore, prior research explored the performance of generative AI in supporting English learning as a foreign language teacher (Guo \u0026amp; Wang, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Guo and Wang (\u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e) asked teachers to evaluate both generative AI\u0026rsquo;s feedback on student writing and human teachers\u0026rsquo; feedback. The results showed that generative AI generated more feedback than teachers, distributing attention evenly across content, organization, and language aspects. Overall, these findings suggest that generative AI performs certain tasks including text evaluation as effectively as humans, if not more so in some aspects.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHypothesis 1\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHuman-rated acceptance of generative AI is positively associated with generative AI-rated acceptance of generative AI\u003c/p\u003e\n\u003ch3\u003eValidating Generative AI Evaluation: Expectancy, Conditions, and Risk\u003c/h3\u003e\n\u003cp\u003eAs for the second question, we argue that we can validate its evaluation by confirming its correlations with evaluations of other aspects that research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework focuses (Abdaljaleel et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Ben Arfi et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e; Davis, \u003cspan class=\"CitationRef\"\u003e1989\u003c/span\u003e; Polyportis \u0026amp; Pahos, \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Venkatesh et al., \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e). Venkatesh et al. (\u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e) developed UTAUT as a thorough synthesis of previous technology acceptance studies, by reviewing the existing models, including TAM. UTAUT includes four key constructs: performance expectancy, effort expectancy, social influence, and facilitating condition. Especially, we focus on perceived risk as well as performance expectancy and facilitating conditions. Our focus was informed by recent discourse and research articles in the respective contexts, in which universities expect generative AI to improve teaching and learning activities and prepare themselves for support and resources for it, but are concerned about academic integrity (Abdaljaleel et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Crompton \u0026amp; Burke, \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Harvard University, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e; Imperial College London, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e; Stanford University, \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003ePerformance expectancy is the extent to which using a technology will provide benefits to individuals in performing certain activities (Venkatesh et al., \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e2012\u003c/span\u003e). In the context of generative AI in higher educational policies, benefits involve the degree to which generative AI tools will support and improve teaching and learning activities, contributing to educational outcomes in academic research and activities. A large body of research has shown that performance expectancy can be a determinant of AI technology acceptance (Andrews et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e; Chatterjee \u0026amp; Bhattacharjee, \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e; Guggemos et al., \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e; Raffaghelli et al., \u003cspan class=\"CitationRef\"\u003e2022\u003c/span\u003e). A recent survey showed that performance expectancy had a positive effect on students\u0026rsquo; attitudes towards the generative AI (Foroughi et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). That is, when students believed that using the chatbot would bring them benefits such as convenience, efficiency, or effectiveness in their educational tasks, they had a positive attitude toward the chatbot. Provided that performance expectancy has a positive effect on accepting new technologies, as universities expect AI to greatly assist in personalized learning, efficient data analysis, and creative academic endeavors, they should accept generative AI. Therefore, we hypothesize,\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHypothesis 2\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePerformance expectancy is positively associated with acceptance of generative AI\u003c/p\u003e\n\u003cp\u003eFacilitating conditions are defined as individuals\u0026rsquo; perceptions of the availability of resources and support necessary for performing activities (Brown \u0026amp; Venkatesh, \u003cspan class=\"CitationRef\"\u003e2005\u003c/span\u003e; Venkatesh et al., \u003cspan class=\"CitationRef\"\u003e2003\u003c/span\u003e). Within the context of generative AI in higher education policies, these resources and support specifically include the extent to which universities provide the essential technical, academic, and policy frameworks required for the effective integration of generative AI tools. Numerous studies have suggested that facilitating conditions are another determinant of individuals\u0026rsquo; acceptance of AI technologies (Cabrera-S\u0026aacute;nchez et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e; Chatterjee \u0026amp; Bhattacharjee, \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e; Kwak et al., \u003cspan class=\"CitationRef\"\u003e2022\u003c/span\u003e). More recently, a survey study found that facilitating conditions had the strongest effect on students\u0026rsquo; intentions and actual usage of generative AI (Habibi et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e), indicating that when students had access to the necessary resources for using generative AI, such as a laptop and internet connection, they were not only more intent on using generative AI but also used it more frequently. Given these findings on facilitating conditions and technology acceptance, universities that provide the required resources and support for leveraging generative AI should endorse its acceptance. Accordingly, we hypothesize,\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHypothesis 3\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFacilitating conditions are positively associated with acceptance of generative AI\u003c/p\u003e\n\u003cp\u003ePerceived risk refers to an individual\u0026rsquo;s subjective evaluation of the potential negative outcomes associated with a specific action (Abdaljaleel et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e; Ben Arfi et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e; Zhang et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). In the context of generative AI in higher education policies, such risk covers the degree to which a university perceives potential risks associated with the use of generative AI tools in academics such as cheating, misinformation use, and copyright violation, which undermine student learning. Some studies have suggested that perceived risk is a potential determinant of individuals\u0026rsquo; acceptance of new technologies (Ben Arfi et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e; Zhang et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). Recent research found the negative effect of perceived risk on acceptance of generative AI, which showed that when students and faculty member believed that using generative AI for answering academic queries is risky, they did not think that generative AI in higher education is good for society (Jain \u0026amp; Raghuram, \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e). A multinational study replicated this result (Abdaljaleel et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e). If perceived risk has a negative effect on accepting new technologies, as universities perceives potential risks associated with the use of generative AI tools in higher education, they should be cautious about endorsing its acceptance. Thus, we hypothesize,\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHypothesis 4\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePerceived risk is negatively associated with acceptance of generative AI\u003c/p\u003e\n\u003ch3\u003ePresent Study\u003c/h3\u003e\n\u003cp\u003eThe present study rigorously examines the performance of generative AI, a generative AI model, in assessing the acceptance of generative AI in guidelines from top-ranked universities worldwide. We aim to understand how well generative AI evaluation corresponds with human judgments in terms of acceptance, performance expectancy, facilitating conditions, and perceived risk associated with the use of generative AI. The purpose is twofold: first, to investigate the performance of generative AI in accurately reflecting human perspectives on the adoption of AI within educational policies; and second, to examine the validity of generative AI evaluation from existing technology acceptance frameworks, including TAM and UTAUT.\u003c/p\u003e"},{"header":"Method","content":"\u003cp\u003eTo ensure a systematic and efficient selection of guidelines, we used the Quacquarelli Symonds (QS) World University Rankings 2023 to identify universities for inclusion (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.topuniversities.com/qs-world-university-rankings\u003c/span\u003e\u003cspan address=\"https://www.topuniversities.com/qs-world-university-rankings\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). We conducted an a priori power with GPower (Faul et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). A minimum sample size of 26 was required to detect a large effect with 80% statistical power and an α of 5% in correlation analysis. Based on these criteria, we decided to collect guidelines from the top 50 ranked universities. The research ethics committee at the Center for International Education and Exchange of Osaka University approved the study procedure prior to data collection. Appendixes in this study can be found on the Open Science Framework site: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://osf.io/7qshb/?view_only=2de75d6f87e24a7ab3633c8272c8a058\u003c/span\u003e\u003cspan address=\"https://osf.io/7qshb/?view_only=2de75d6f87e24a7ab3633c8272c8a058\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. The datasets used in this study are available from the corresponding author on a reasonable request.\u003c/p\u003e \u003cp\u003e \u003cb\u003eKeyword-based Search\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThis study performed a keyword-based web search of the Google search engine in private browsing mode, after log-out from personal accounts and erasure of all web cookies and history (Jobin et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Piasecki et al., \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). We used the following keywords during the search: [generative AI guidelines], [generative AI policies], [generative AI statementes], and [UNIVERSITY NAME]. [UNIVERSITY NAME] was entered as the name of a university in the top 50 of the QS university rankings. The Google results up to the 200th listings for each Google search were followed and screened for generative AI guidelines only. Within these link listings, we identified 35 non-duplicate documents. We continued to monitor the literature in parallel with the data analysis and until June 30, 2023, to retrieve eligible documents that were released after our search was completed.\u003c/p\u003e\n\u003ch3\u003eInclusion Criteria\u003c/h3\u003e\n\u003cp\u003e Based on our inclusion criteria, target documents (including statements, guidelines, and notices) included in the final synthesis were (i) written in English; (ii) issued by institutional entities from universities; (iii) mentioned explicitly in their title/description to generative AI or related notions, (iv) expressed a normative stance defined as a preference for a particular course of action related to generative AI.\u003c/p\u003e\n\u003ch3\u003eAssessment of Acceptance of Generative AI\u003c/h3\u003e\n\u003cp\u003eWe used a generative AI to measure the acceptance of generative AI in higher education following two steps (i.e., Chat GPT). To measure the acceptance of generative AI in higher education, we operationally defined it as extent to which an institution supports or restricts the use of generative AI technologies by students in academic settings, as reflected in the university\u0026rsquo;s official guidelines and policies. This definition was derived from previous studies that linked a university\u0026rsquo;s stance on generative AI to academic integrity (Bin-Nashwan et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Cotton et al., \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Eke, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Based on this definition, we created four levels of acceptance (See the details in Appendix 1): Level 1, strongly against, prohibits AI use without permission, treating violations as misconduct. Level 2, against, views unauthorized AI as plagiarism, requiring ethical adherence and permission. Level 3, neutral, neither endorses nor bans AI, stressing academic integrity and ethical usage with necessary permissions. Level 4, supportive, encourages responsible AI use, acknowledging its educational value, with guidelines for proper application in assessments.\u003c/p\u003e \u003cp\u003eNext, we evaluated the acceptance of generative AI in higher education using two methods. First, for the human rating, we had three native English undergraduate students independently review 35 university guidelines (two students were male), categorizing them into predefined levels of acceptance. We assessed the reliability of their combined ratings and found it to be acceptable (α\u0026thinsp;=\u0026thinsp;.73). Thus, we calculated the average of these ratings to represent the human-rated acceptance of generative AI in higher education. Second, for the generative AI rating, we asked ChatGPT for classifying each university guideline into the established levels of acceptance (See an example; \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://chat.openai.com/share/ae65480c-7721-42fb-9901-493c5dba542a\u003c/span\u003e\u003cspan address=\"https://chat.openai.com/share/ae65480c-7721-42fb-9901-493c5dba542a\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). To ensure accuracy, ChatGPT evaluated each guideline three times. The reliability of these ratings was also found to be high (α\u0026thinsp;=\u0026thinsp;.82). We used the average of these ratings as the ChatGPT-rated acceptance of generative AI in higher education.\u003c/p\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eAssessment of Performance Expectancy, Facilitating Conditions, Perceived Risk\u003c/h2\u003e \u003cp\u003eTo measure performance expectancy, we operationally defined it as the degree to which a university perceives potential benefits associated with the use of generative AI tools in academic settings (Chatterjee \u0026amp; Bhattacharjee, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Foroughi et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Raffaghelli et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Based on this definition, we created four levels of performance expectancy (See the details in Appendix 2). Level 1, Low Expectancy, sees minimal benefits, remaining skeptical of significant improvements. Level 2, Cautious Expectancy, holds a cautious outlook, expecting moderate enhancements in specific areas. Level 3, Moderate Expectancy, anticipates noticeable gains in performance and efficiency, viewing AI as a supportive tool for academic tasks. At Level 4, High Expectancy, the university foresees substantial improvements in education and research, expecting AI to advance personalized learning, data analysis, and creative academic processes.\u003c/p\u003e \u003cp\u003eTo measure facilitating conditions, our study operationally defined it as the extent to which universities provide the necessary technical, academic, and policy support for the effective integration of generative AI tools in academic settings (Cabrera-S\u0026aacute;nchez et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Chatterjee \u0026amp; Bhattacharjee, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Habibi et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Based on this definition, we created four levels of facilitating conditions (See the details in Appendix 3). At Level 1, Minimal Facilitation, there\u0026rsquo;s hardly any support or AI resources. Level 2, Basic Support, offers basic AI tools and a general guide for use, but lacks in-depth resources and training. At Level 3, Supportive Infrastructure, the support is better, with good access to AI technology, some specialized training, and guidelines for ethical use. Level 4, Highly Facilitative, features a strong infrastructure with advanced AI labs, extensive training programs, and comprehensive support policies, ensuring widespread access to AI for all academic activities.\u003c/p\u003e \u003cp\u003eTo measure perceived risk, this study operationally defined it as the degree to which a university perceives potential risks associated with the use of generative AI tools in academic settings (Abdaljaleel et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Ben Arfi et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Jain \u0026amp; Raghuram, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Based on this definition, we created four levels of perceived risk (See the details in Appendix 4). Level 1, Embracing with Guidance, shows the least concern for risks, focusing on ethical use and responsible AI integration with supportive guidelines. Level 2, Conditionally Accepting, balances potential risks and benefits, allowing AI use under ethical and citation standards. Level 3, Cautiously Permissive, acknowledges higher risks, implementing stricter guidelines to prevent misuse. At Level 4, Highly Cautious, the university enforces severe restrictions due to concerns like academic dishonesty, diminishing learning quality, and intellectual property issues, with strict consequences for violations.\u003c/p\u003e \u003cp\u003e Similar to ChatGPT rating in acceptance of generative AI, we asked ChatGPT to categorize each set of guidelines into predefined levels concerning performance expectancy, facilitating conditions, and perceived risk. ChatGPT conducted this classification task three times for each guideline to ensure precision. The reliability of ChatGPT\u0026rsquo;s ratings across these evaluations was satisfactory, with .92 for performance expectancy, .71 for facilitating conditions, and .89 for perceived risk. We used the average ratings from these assessments.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eCorrelation Analysis\u003c/h2\u003e \u003cp\u003eTo test our hypotheses, we conducted Pearson correlation analysis between human-rated acceptance, ChatGPT-rated acceptance, performance expectancy, facilitative conditions, and perceived risk. Figure\u0026nbsp;1 presents overall correlation matrix for these variables. As Fig.\u0026nbsp;2 showed, there was a strongly positive correlation between ChatGPT-rated acceptance and human-rated acceptance (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.85, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001). This means that generative AI evaluates texts as accurately as humans. As shown in Fig.\u0026nbsp;3, ChatGPT-rated acceptance was also positively correlated with performance expectancy (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.53, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.001). This indicates that as universities expect AI to enhance teaching and learning activities, they become more accepting of generative AI. Figure\u0026nbsp;4 presented a similar correlation between ChatGPT-rated acceptance and facilitating conditions (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.48, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;=\u0026thinsp;.003). That is, universities that provide the resources for using generative AI are more accepting of it. Finally, unlike those aspects of generative AI (Fig.\u0026nbsp;5), ChatGPT-rated acceptance was negatively correlated with perceived risk (\u003cem\u003er\u003c/em\u003e\u0026thinsp;=\u0026thinsp;\u0026minus;\u0026thinsp;.60, \u003cem\u003ep\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;.001). This suggests that universities aware of the potential risks associated with using generative AI tools in higher education should be cautious in endorsing their acceptance.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003e We investigated the performance of generative AI in assessing the acceptance of its own use within higher education guidelines. The results confirmed all four hypotheses: Hypothesis \u003cspan refid=\"FPar1\" class=\"InternalRef\"\u003e1\u003c/span\u003e was supported by a strong positive correlation between human-rated and ChatGPT-rated acceptance. This study supported Hypothesis \u003cspan refid=\"FPar2\" class=\"InternalRef\"\u003e2\u003c/span\u003e by finding a positive relationship between performance expectancy and acceptance. Hypothesis \u003cspan refid=\"FPar3\" class=\"InternalRef\"\u003e3\u003c/span\u003e was confirmed by the positive correlation between facilitating conditions and acceptance. Finally, Hypothesis \u003cspan refid=\"FPar4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, which posited a negative correlation between perceived risk and acceptance, was supported. These findings collectively show the performance of generative AI in text evaluation.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eEvaluating Acceptance of Generative AI\u003c/h2\u003e \u003cp\u003eBuilding on a growing body of research on the performance of AI, our study addresses the first research question: Can generative AI evaluate the acceptance of generative AI as effectively as humans? Previous studies have shown that AI, including generative AI, performs impressively in various academic and professional exams, equalling or exceeding human performance across fields such as medicine, engineering, and finance (Frieder et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Gilson et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Kung et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Takagi et al., \u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Terwiesch, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Watari et al., \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Yang \u0026amp; Stivers, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Moreover, prior studies find that generative AI can effectively perform tasks similar to humans, such as providing writing feedback, evaluating essays, and supporting language learning (Escalante et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Guo \u0026amp; Wang, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Mizumoto \u0026amp; Eguchi, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Our study first found a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI in educational policies. Thus, in response to our research question, the strong positive correlation between ChatGPT-rated and human-rated acceptance provides evidence for the notion that generative AI can evaluate the acceptance of generative AI within educational policies as effectively as human evaluators.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eValidating Generative AI Evaluation\u003c/h2\u003e \u003cp\u003eOur study also contributes to the extensive research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework by investigating the second research question: Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk? Previous studies have suggested that performance expectancy, facilitating conditions, and perceived risk are associated with the acceptance of AI technologies, underlining their role in technology adoption decisions (Ben Arfi et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Cabrera-S\u0026aacute;nchez et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Chatterjee \u0026amp; Bhattacharjee, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Guggemos et al., \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Kwak et al., \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Zhang et al., \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Recent studies indicate that performance expectancy and facilitating conditions are positively associated with students\u0026rsquo; attitudes and usage of generative AI, while perceived risk is negatively associated with the acceptance of generative AI in education (Foroughi et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Habibi et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Jain \u0026amp; Raghuram, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Our study found a positive correlation between ChatGPT-rated acceptance of generative AI and performance expectancy, indicating that universities\u0026rsquo; expectations of AI enhancing teaching and learning activities are associated with greater acceptance of generative AI. Similarly, there was a positive correlation between ChatGPT-rated acceptance of generative AI and facilitating conditions, which suggests that universities equipped with the necessary resources for using generative AI are likely to accept it. Conversely, the negative correlation between ChatGPT-rated acceptance of generative AI and perceived risk shows that universities aware of the potential risks for generative AI in education are reluctant to accept it. These results expand the application of TAM or UTAUT framework from individual to institutional perspectives, indicating that generative AI can accurately evaluate and distinguish between various aspects of generative AI in educational policies. This answers our second research question, validating generative AI\u0026rsquo;s assessments against key criteria such as performance expectancy, facilitating conditions, and perceived risk.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003ePractical Implications\u003c/h2\u003e \u003cp\u003eThis study has implications for those who currently use or intend to use generative AI because it provides insights into the performance of AI technologies. As generative AI becomes more prevalent in society, expectations have risen substantially. Several articles highlight that emerging technologies such as generative AI often experience stages of initial enthusiasm, followed by disappointment, and finally, practical application, cautioning against overestimating their immediate impact (Eulerich et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). For some users, their expectations may not be fulfilled probably due to too much expectation. Here is a question; What can generative AI perform and to what extent? Addressing the question is crucial to avoid betraying users\u0026rsquo; expectations. Our study answers this question in text evaluation by demonstrating that generative AI can evaluate guidelines on four levels, akin to humans, thereby contributing to the practical application of generative AI in society.\u003c/p\u003e \u003cp\u003eFurthermore, our findings inform research practices on educational policy for generative AI. Educational policy research collected and quantitatively analyzed policy documents and guidelines about generative AI from higher education institutions (McDonald et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Another research categorizes the guidelines into different nine fields, including authorship, acknowledgment, plagiarism, advice on assessment design and tasks, detection of GAI use, responsible agent, proper use, improper use, and communication with students (Moorhouse et al., \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). These studies help us understand how higher education institutions are navigating the integration of generative AI technologies into teaching activities and the development of guidelines to support instructors. However, we still do not know the degree to which each higher education institution accepts generative AI. Then researchers are required to rate it from guidelines, but the process of human rating of texts can be fraught with issues that fatigue, subjectivity, and inconsistency may produce unreliable results (Hussein et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Mizumoto \u0026amp; Eguchi, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Generative AI provides a faster, more consistent alternative, potentially transforming rating method, as the reliability of the ChatGPT rating was higher than the reliability of the human rating in this study.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eWe acknowledge some limitations of this study. First, this study focuses on the top 50 universities according to the QS World University Rankings, which may not fully represent the diverse range of higher education institutions globally. As such, the findings may not be applicable to smaller, less research-intensive institutions. Second, this study captures a snapshot of university policies about generative AI at a specific timeframe. Given the fast-paced evolution of both AI technology and societal norms towards it, these policies and guidelines are subject to change. Therefore, the findings might quickly become outdated, limiting their long-term relevance. Third, while our study effectively quantifies acceptance levels and correlates them with various factors, we did not focus on qualitative aspects of university guidelines and policies toward generative AI. The rich context underlying these documents could provide deeper insights into institutional stances, which our quantitative approach might not fully capture.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003e This study demonstrates that generative AI can effectively evaluate the acceptance of generative AI in higher education guidelines, corresponding to human judgments. The positive correlations between generative AI-rated acceptance and performance expectancy and facilitating conditions, along with its negative correlation with perceived risk, validate the generative AI evaluations within the frameworks of technology acceptance theories. These findings highlight the performance of AI in assisting policy analysis and decision-making in higher educational contexts.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgements\u003c/h2\u003e \u003cp\u003eWe would like to thank our research collaborators.\u003c/p\u003e\u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eThis research received no specific grant from any funding agency.\u003c/p\u003e\u003ch2\u003eAvailability of Data and Materials\u003c/h2\u003e \u003cp\u003eThe datasets used in this study are available from the corresponding author on reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAbdaljaleel, M., Barakat, M., Alsanafi, M., Salim, N. A., Abazid, H., Malaeb, D., Mohammed, A. H., Hassan, B. A. R., Wayyes, A. M., Farhan, S. S., Khatib, S. El, Rahal, M., Sahban, A., Abdelaziz, D. H., Mansour, N. O., AlZayer, R., Khalil, R., Fekih-Romdhane, F., Hallit, R., \u0026hellip; Sallam, M. (2024). A multinational study on the factors influencing university students\u0026rsquo; attitudes and usage of ChatGPT. \u003cem\u003eScientific Reports\u003c/em\u003e, \u003cem\u003e14\u003c/em\u003e, 1983. https://doi.org/10.1038/s41598-024-52549-8\u003c/li\u003e\n\u003cli\u003eAndrews, J. E., Ward, H., \u0026amp; Yoon, J. (2021). UTAUT as a model for understanding intention to adopt AI and related technologies among librarians. \u003cem\u003eJournal of Academic Librarianship\u003c/em\u003e, \u003cem\u003e47\u003c/em\u003e, 102437. https://doi.org/https://doi.org/10.1016/j.acalib.2021.102437\u003c/li\u003e\n\u003cli\u003eBen Arfi, W., Ben Nasr, I., Khvatova, T., \u0026amp; Ben Zaied, Y. (2021). Understanding acceptance of eHealthcare by IoT natives and IoT immigrants: An integrated model of UTAUT, perceived risk, and financial cost. \u003cem\u003eTechnological Forecasting and Social Change\u003c/em\u003e, \u003cem\u003e163\u003c/em\u003e, 120437. https://doi.org/https://doi.org/10.1016/j.techfore.2020.120437\u003c/li\u003e\n\u003cli\u003eBin-Nashwan, S. A., Sadallah, M., \u0026amp; Bouteraa, M. (2023). Use of ChatGPT in academia: Academic integrity hangs in the balance. \u003cem\u003eTechnology in Society\u003c/em\u003e, \u003cem\u003e75\u003c/em\u003e, 102370. https://doi.org/https://doi.org/10.1016/j.techsoc.2023.102370\u003c/li\u003e\n\u003cli\u003eBordt, S., \u0026amp; von Luxburg, U. (2023). Chatgpt participates in a computer science exam. \u003cem\u003eArXiv Preprint ArXiv. \u003c/em\u003e2303.09461.\u003c/li\u003e\n\u003cli\u003eBrown, S. A., \u0026amp; Venkatesh, V. (2005). Model of adoption of technology in households: A baseline model test and extension incorporating household life cycle. \u003cem\u003eMIS Quarterly\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e, 399\u0026ndash;426. https://doi.org/10.2307/25148690\u003c/li\u003e\n\u003cli\u003eCabrera-S\u0026aacute;nchez, J.-P., Villarejo-Ramos, \u0026Aacute;. F., Li\u0026eacute;bana-Cabanillas, F., \u0026amp; Shaikh, A. A. (2021). Identifying relevant segments of AI applications adopters \u0026ndash; Expanding the UTAUT2\u0026rsquo;s variables. \u003cem\u003eTelematics and Informatics\u003c/em\u003e, \u003cem\u003e58\u003c/em\u003e, 101529. https://doi.org/https://doi.org/10.1016/j.tele.2020.101529\u003c/li\u003e\n\u003cli\u003eChatterjee, S., \u0026amp; Bhattacharjee, K. K. (2020). Adoption of artificial intelligence in higher education: a quantitative analysis using structural equation modelling. \u003cem\u003eEducation and Information Technologies\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(5), 3443\u0026ndash;3463. https://doi.org/10.1007/s10639-020-10159-7\u003c/li\u003e\n\u003cli\u003eCotton, D. R. E., Cotton, P. A., \u0026amp; Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. \u003cem\u003eInnovations in Education and Teaching International\u003c/em\u003e, \u003cem\u003e61\u003c/em\u003e, 228\u0026ndash;239. https://doi.org/10.1080/14703297.2023.2190148\u003c/li\u003e\n\u003cli\u003eCrompton, H., \u0026amp; Burke, D. (2024). The Educational Affordances and Challenges of ChatGPT: State of the Field. \u003cem\u003eTechTrends\u003c/em\u003e, \u003cem\u003e68\u003c/em\u003e, 380\u0026ndash;392. https://doi.org/10.1007/s11528-024-00939-0\u003c/li\u003e\n\u003cli\u003eDavis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. \u003cem\u003eMIS Quarterly\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e, 319\u0026ndash;340. https://doi.org/10.2307/249008\u003c/li\u003e\n\u003cli\u003eEke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? \u003cem\u003eJournal of Responsible Technology\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e, 100060. https://doi.org/https://doi.org/10.1016/j.jrt.2023.100060\u003c/li\u003e\n\u003cli\u003eElder, C., Pozek, G., Horine, S., Tripaldelli, A., \u0026amp; Butka, B. (2023). Can artificial intelligence pass a sophomore level digital design laboratory? \u003cem\u003eSoutheastCon 2023\u003c/em\u003e, 861\u0026ndash;868. https://doi.org/10.1109/SoutheastCon51012.2023.10115116\u003c/li\u003e\n\u003cli\u003eEscalante, J., Pack, A., \u0026amp; Barrett, A. (2023). AI-generated feedback on writing: insights into efficacy and ENL student preference. \u003cem\u003eInternational Journal of Educational Technology in Higher Education\u003c/em\u003e, \u003cem\u003e20\u003c/em\u003e, 57. https://doi.org/10.1186/s41239-023-00425-2\u003c/li\u003e\n\u003cli\u003eEulerich, M., Sanatizadeh, A., Vakilzadeh, H., \u0026amp; Wood, D. A. (2023). Is it all hype? ChatGPT\u0026rsquo;s performance and disruptive potential in the accounting and auditing industries. \u003cem\u003eSSRN Electronic Journal\u003c/em\u003e. http://dx.doi.org/10.2139/ssrn.4452175\u003c/li\u003e\n\u003cli\u003eFaul, F., Erdfelder, E., Lang, A.-G., \u0026amp; Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. \u003cem\u003eBehavior Research Methods\u003c/em\u003e, \u003cem\u003e39\u003c/em\u003e, 175\u0026ndash;191. https://doi.org/10.3758/BF03193146\u003c/li\u003e\n\u003cli\u003eForoughi, B., Senali, M. G., Iranmanesh, M., Khanfar, A., Ghobakhloo, M., Annamalai, N., \u0026amp; Naghmeh-Abbaspour, B. (2023). Determinants of iIntention to use ChatGPT for educational purposes: Findings from PLS-SEM and fsQCA. \u003cem\u003eInternational Journal of Human\u0026ndash;Computer Interaction\u003c/em\u003e, 1\u0026ndash;20. https://doi.org/10.1080/10447318.2023.2226495\u003c/li\u003e\n\u003cli\u003eFrieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., \u0026amp; Berner, J. (2024). Mathematical capabilities of chatgpt. \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e, \u003cem\u003e36\u003c/em\u003e, 699--744.\u003c/li\u003e\n\u003cli\u003eGeerling, W., Mateer, G. D., Wooten, J., \u0026amp; Damodaran, N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? \u003cem\u003eAmerican Economist\u003c/em\u003e, \u003cem\u003e68\u003c/em\u003e, 233\u0026ndash;245. https://doi.org/10.1177/05694345231169654\u003c/li\u003e\n\u003cli\u003eGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., \u0026amp; Chartash, D. (2023). How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. \u003cem\u003eJMIR Medical Education\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e, e45312. https://doi.org/10.2196/45312\u003c/li\u003e\n\u003cli\u003eGuggemos, J., Seufert, S., \u0026amp; Sonderegger, S. (2020). Humanoid robots in higher education: Evaluating the acceptance of Pepper in the context of an academic writing course using the UTAUT. \u003cem\u003eBritish Journal of Educational Technology\u003c/em\u003e, \u003cem\u003e51\u003c/em\u003e, 1864\u0026ndash;1883. https://doi.org/https://doi.org/10.1111/bjet.13006\u003c/li\u003e\n\u003cli\u003eGuo, K., \u0026amp; Wang, D. (2023). To resist it or to embrace it? Examining ChatGPT\u0026rsquo;s potential to support teacher feedback in EFL writing. \u003cem\u003eEducation and Information Technologies\u003c/em\u003e. https://doi.org/10.1007/s10639-023-12146-0\u003c/li\u003e\n\u003cli\u003eHabibi, A., Muhaimin, M., Danibao, B. K., Wibowo, Y. G., Wahyuni, S., \u0026amp; Octavia, A. (2023). ChatGPT in higher education learning: Acceptance and use. \u003cem\u003eComputers and Education: Artificial Intelligence\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e, 100190. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100190\u003c/li\u003e\n\u003cli\u003eHarvard University. (2023). Initial guidelines for using ChatGPT and other generative AI tools at Harvard. Retrieved from https://huit.harvard.edu/news/ai-guidelines (July, 13, 2023).\u003c/li\u003e\n\u003cli\u003eHussein, M. A., Hassan, H., \u0026amp; Nassef, M. (2019). Automated language essay scoring systems: A literature review. \u003cem\u003ePeerJ Computer Science\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e, e208.\u003c/li\u003e\n\u003cli\u003eImperial College London. (2023). Generative AI tools guidance: College guidance on the use of generative AI tools. Retrieved from https://www.imperial.ac.uk/about/leadership-and-strategy/provost/vice-provost-education/generative-ai-tools-guidance/ (March, 1, 2023).\u003c/li\u003e\n\u003cli\u003eJain, K. K., \u0026amp; Raghuram, J. N. V. (2024). Gen-AI integration in higher education: Predicting intentions using SEM-ANN approach. \u003cem\u003eEducation and Information Technologies\u003c/em\u003e. https://doi.org/10.1007/s10639-024-12506-4\u003c/li\u003e\n\u003cli\u003eJobin, A., Ienca, M., \u0026amp; Vayena, E. (2019). The global landscape of AI ethics guidelines. \u003cem\u003eNature Machine Intelligence\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e, 389\u0026ndash;399. https://doi.org/10.1038/s42256-019-0088-2\u003c/li\u003e\n\u003cli\u003eKung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepa\u0026ntilde;o, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., \u0026amp; Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. \u003cem\u003ePLOS Digital Health\u003c/em\u003e, \u003cem\u003e2\u003c/em\u003e, e0000198. https://doi.org/10.1371/journal.pdig.0000198\u003c/li\u003e\n\u003cli\u003eKwak, Y., Seo, Y. H., \u0026amp; Ahn, J.-W. (2022). Nursing students\u0026rsquo; intent to use AI-based healthcare technology: Path analysis using the unified theory of acceptance and use of technology. \u003cem\u003eNurse Education Today\u003c/em\u003e, \u003cem\u003e119\u003c/em\u003e, 105541. https://doi.org/https://doi.org/10.1016/j.nedt.2022.105541\u003c/li\u003e\n\u003cli\u003eMcDonald, N., Johri, A., Ali, A., \u0026amp; Hingle, A. (2024). Generative artificial intelligence in higher education: Evidence from an analysis of institutional policies and guidelines. \u003cem\u003eArXiv Preprint ArXiv:2402.01659\u003c/em\u003e.\u003c/li\u003e\n\u003cli\u003eMizumoto, A., \u0026amp; Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. \u003cem\u003eResearch Methods in Applied Linguistics\u003c/em\u003e, \u003cem\u003e2\u003c/em\u003e, 100050. https://doi.org/https://doi.org/10.1016/j.rmal.2023.100050\u003c/li\u003e\n\u003cli\u003eMoorhouse, B. L., Yeo, M. A., \u0026amp; Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world\u0026rsquo;s top-ranking universities. \u003cem\u003eComputers and Education Open\u003c/em\u003e, \u003cem\u003e5\u003c/em\u003e, 100151. https://doi.org/https://doi.org/10.1016/j.caeo.2023.100151\u003c/li\u003e\n\u003cli\u003ePiasecki, J., Waligora, M., \u0026amp; Dranseika, V. (2018). Google search as an additional source in systematic reviews. \u003cem\u003eScience and Engineering Ethics\u003c/em\u003e, \u003cem\u003e24\u003c/em\u003e, 809\u0026ndash;810. https://doi.org/10.1007/s11948-017-0010-4\u003c/li\u003e\n\u003cli\u003ePolyportis, A., \u0026amp; Pahos, N. (2024). Understanding students\u0026rsquo; adoption of the ChatGPT chatbot in higher education: the role of anthropomorphism, trust, design novelty and institutional policy. \u003cem\u003eBehaviour \u0026amp; Information Technology\u003c/em\u003e, 1\u0026ndash;22. https://doi.org/10.1080/0144929X.2024.2317364\u003c/li\u003e\n\u003cli\u003eRaffaghelli, J. E., Rodr\u0026iacute;guez, M. E., Guerrero-Rold\u0026aacute;n, A.-E., \u0026amp; Ba\u0026ntilde;eres, D. (2022). Applying the UTAUT model to explain the students\u0026rsquo; acceptance of an early warning system in Higher Education. \u003cem\u003eComputers \u0026amp; Education\u003c/em\u003e, \u003cem\u003e182\u003c/em\u003e, 104468. https://doi.org/https://doi.org/10.1016/j.compedu.2022.104468\u003c/li\u003e\n\u003cli\u003eStanford University. (2023). Generative AI policy guidance. Retrieved from https://communitystandards.stanford.edu/generative-ai-policy-guidance (February, 16, 2023).\u003c/li\u003e\n\u003cli\u003eTakagi, S., Watari, T., Erabi, A., \u0026amp; Sakaguchi, K. (2023). Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: Comparison study. \u003cem\u003eJMIR Medical Education\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e, e48002. https://doi.org/10.2196/48002\u003c/li\u003e\n\u003cli\u003eTerwiesch, C. (2023). \u003cem\u003eWould chat GPT3 get a Wharton MBA: A prediction based on its performance in the operations management course. Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania.\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003eVenkatesh, V., Morris, M. G., Davis, G. B., \u0026amp; Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. \u003cem\u003eMIS Quarterly\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e, 425\u0026ndash;478. https://doi.org/10.2307/30036540\u003c/li\u003e\n\u003cli\u003eVenkatesh, V., Thong, J. Y. L., \u0026amp; Xu, X. (2012). Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. \u003cem\u003eMIS Quarterly\u003c/em\u003e, \u003cem\u003e36\u003c/em\u003e, 157\u0026ndash;178. https://doi.org/10.2307/41410412\u003c/li\u003e\n\u003cli\u003eWatari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., \u0026amp; Tokuda, Y. (2023). Performance comparison of ChatGPT-4 and Japanese medical residents in the general medicine in-training examination: Comparison study. \u003cem\u003eJMIR Medical Education\u003c/em\u003e, \u003cem\u003e9\u003c/em\u003e, e52202. https://doi.org/10.2196/52202\u003c/li\u003e\n\u003cli\u003eYang, C., \u0026amp; Stivers, A. (2024). Investigating AI languages\u0026rsquo; ability to solve undergraduate finance problems. \u003cem\u003eJournal of Education for Business\u003c/em\u003e, \u003cem\u003e99\u003c/em\u003e, 44\u0026ndash;51. https://doi.org/10.1080/08832323.2023.2253963\u003c/li\u003e\n\u003cli\u003eZhang, T., Tao, D., Qu, X., Zhang, X., Lin, R., \u0026amp; Zhang, W. (2019). The roles of initial trust and perceived risk in public\u0026rsquo;s acceptance of automated vehicles. \u003cem\u003eTransportation Research Part C: Emerging Technologies\u003c/em\u003e, \u003cem\u003e98\u003c/em\u003e, 207\u0026ndash;220. https://doi.org/https://doi.org/10.1016/j.trc.2018.11.018\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Osaka University","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"artificial intelligence, higher education, guidelines, UTAUT","lastPublishedDoi":"10.21203/rs.3.rs-4469443/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4469443/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis study investigates the performance of generative artificial intelligence (AI) in evaluating the acceptance of generative AI technologies within higher education guidelines, reflecting on the implications for educational policy and practice. Drawing on a dataset of guidelines from top-ranked universities, we compared generative AI evaluations with human evaluations, focusing on acceptance, performance expectancy, facilitating conditions, and perceived risk. Our study revealed a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI, suggesting that generative AI can accurately reflect human judgment in this context. Further, we found positive associations between ChatGPT-rated acceptance and performance expectancy and facilitating conditions, while a negative correlation with perceived risk. These results validate generative AI evaluation, which also extends the application of the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology framework from individual to institutional perspectives.\u003c/p\u003e","manuscriptTitle":"Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-28 19:59:27","doi":"10.21203/rs.3.rs-4469443/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"70d10f77-2aa4-469d-b7f6-7a2bcba1eb6f","owner":[],"postedDate":"May 28th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":32336233,"name":"Educational Psychology"}],"tags":[],"updatedAt":"2024-05-28T19:59:27+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-28 19:59:27","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4469443","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4469443","identity":"rs-4469443","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}