Enhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT)

Enhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT) | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Enhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT) Baharuddin Baharuddin, Lilis Handayani, Rusli Rusli This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3859753/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background In medical education, particularly in biochemistry, crafting high-quality assessment questions is a primary challenge. Each item necessitates thorough evaluation, and precise identification of student abilities is crucial for maximally reflecting learning achievement. Objective This study aims to enhance assessment quality in biochemistry medical education by implementing Item Response Theory (IRT). This approach addresses Classical Test Theory (CTT) limitations. Recognizing the critical role of question quality in the learning process, the study investigates how IRT can more holistically and equitably assess student abilities. It includes a comparative analysis of student scores before and after IRT implementation. Methods Employing a mixed-method research approach, this study combines comparative quantitative analysis with qualitative ICC curve analysis in a pre-post experimental design. It focuses on biochemistry exam data from medical students (n = 89). IRT is used to measure the probability of student responses to questions, using parameters such as discrimination, difficulty level, and guessing probability. Jamovi software supports this analysis by accelerating computational processes. Results Significant improvements were observed in both question quality and student scores. Prior to IRT implementation, the average initial exam score was 56.1, which increased to 74.1 in the subsequent exam. The IRT evaluation indicated that the exam questions achieved a more effective differentiation between students of varying abilities. This improvement was evident from the increased person reliability and through Wright Map visualizations, which helped identify highly difficult questions via the Item Characteristic Curve (ICC). Conclusion The study advocates for integrating IRT as a standard method in biochemistry medical assessments. It highlights the necessity of assessments that are sensitive to individual student capabilities, providing more precise feedback for enhancing the quality of learning. These findings are crucial for evolving evaluation methodologies and advancing medical education standards. solution item response theory assessment biochemistry medical education Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction In medical education, particularly in biochemistry, creating high-quality assessment questions represents a significant challenge. Each item requires detailed evaluation through item analysis. Many students encounter difficulties in biochemistry learning, especially in the preclinical phase [ 1 ]–[ 3 ]. Attempts to improve this using Classical Test Theory (CTT) have been limited, as the analysis focuses only on the items without considering student ability and the probability of guessing. In short, CTT does not account for item-weighted assessment [ 4 ]. Therefore, a more comprehensive method is required, one of which is Item Response Theory (IRT). This method refines the CTT approach. However, research on the application of IRT in medical biochemistry is scarce, yet its findings could provide solutions for medical education units, especially those in charge of biochemistry, to improve question quality. The uniqueness of the IRT model approach lies in its sensitivity to the individual's (subject's) response to each item [ 5 ], even including calculations for the probability of guessing for a population on an item. This approach utilizes the logit function and logistic parameters, such as discrimination (α), difficulty level (β), and pseudo-guessing parameter (c), to depict the relationship between items and various levels of a latent trait [ 6 ]. This method is particularly vital in examination systems like medical biochemistry, especially to enhance the quality of evaluation. To measure the completeness of learning in medical biochemistry, an assessment-based evaluation is necessary. However, challenges often arise in evaluating exam results. Evaluations are sometimes solely based on student performance scores. Often, there is no item analysis evaluation, leading to situations where students are blamed for low scores, when in fact, the exam questions might be flawed in diction or too difficult compared to the taught material. The application process of this method is straightforward, especially when using a single parameter (dichotomous), known as the Rasch Model. With technological advancements, this can be quickly executed using software like Jamovi. Implementing this method will assist in improving distractors in Multiple Choice Questions (MCQ) [ 7 ]. The research questions formulated in this study are: What is the range of theta performance values for assessment and students? Is there a significant change in the average score after implementing the revised questions? Does the theta value of items change when distractor modifications are made? Methods This study utilizes exam data to evaluate question performance [ 8 ]. The analytical approach consists of pre and post assessments following the implementation of Item Response Theory (IRT) to evaluate student performance scores. The evaluation was conducted on student assessments (n = 89) using two approaches. The first approach involved an analysis of average scores, and the second, a comparative quantitative analysis of item performance. The latter also included a qualitative study of the Item Characteristic Curve (ICC). Items subjected to IRT were tabulated and visualized, presenting theta (θ) scores and their graphical changes in the ICC. To expedite the computational process in this research, Jamovi software with an open-access license was used. To eliminate subjectivity, this study was conducted after grades were published and no further grade disputes from students were present. Results and Discussion The evaluation was conducted using two approaches. The initial results showed a significant difference in average scores before and after the implementation of Item Response Theory (IRT). The items evaluated and improved using the IRT approach were visualized as follows. In this study, the overall reliability of the measurement tool was assessed, with a person reliability value of 0.780 for the first exam and 0.724 for the second exam. The analysis using IRT on exam data indicated a Person Reliability of 0.724 on a predetermined scale, suggesting that the exam had a good level of reliability in measuring the latent abilities or attributes of the participants. There was a significant increase in scores before and after the implementation, with a p-value < 0.001 at an alpha of 0.05. The average score for the first exam was 56.1, and for the second exam, it was 74.1. Based on the score distribution, it was also observed that the exam questions effectively differentiated students (Fig. 1 ). Student abilities (scores) were well-identified across the low-middle-high spectrum, with the central tendency located in the middle area of the histogram curve distribution. These findings align with the study by Dorner et al. [ 9 ]. We identified items with very high difficulty levels. This identification was easily accomplished using the Wright Map distribution of the items. This visualization is adequately representative of item difficulty, as it computationally matches the items with the respondents' latent traits (ability to answer). It was evident that there are 11 particularly difficult items, numbered 20, 8, 46, 5, 39, 50, 7, 9, 29, 34, and 44. These items will subsequently be analyzed and improved upon after examining their Item Characteristic Curves (ICC). ICC analysis is essential for examining three aspects. First, to ascertain the difficulty level of the items. Second, to assess the performance of the items in terms of discrimination. Does the item effectively separate based on the abilities of medical students in biochemistry? Third, to observe the detailed ability of students in responding to the items. A more detailed view of each item's ICC is presented in Fig. 2 . Our findings indicate that there should be a spike (increase in ability) in student responses to items within the 1–3 ability range, as students in this range possess above-average abilities. However, in reality, students with high-order thinking skills did not respond effectively (unable to answer correctly), especially noticeable in items number 20, 8, and 46 (see the vertical red boxes). Their response abilities were relatively similar to those of students with low-order thinking skills. A notable increase was observed in items number 5, 7, 9, and 34 (see the vertical high green boxes). From this, we gather crucial information about which items require intervention and evaluation. This section also highlights the need for internal consolidation in the study. This is a form of reflection for educators on the delivered material. It's important to avoid presenting items that are outside the core competencies formulated. Therefore, the role of the coordinator is crucial, as they must evaluate and consolidate immediately upon identifying any discrepancies. The change in theta (θ) scores in biochemistry items indicates that the implementation of IRT is effective. This change can be observed in the ICC shown in Fig. 3 . It's crucial to examine the relationship between student abilities and item difficulty levels within the framework of latent variables, a possibility afforded by the mathematical model system in IRT [ 10 ]. This IRT technique also aims to enhance accuracy in performance classification [ 11 ], [ 12 ]. A slight yet noticeable improvement in response ability was observed, as depicted in the ICC of the three revised items, numbers 20, 8, and 46 (see the short vertical green boxes). Although the increase is modest, the trend line shows an upward trajectory. Moreover, the ability performance no longer shows depression at the baseline. Item number 5 (Q5) also demonstrates an improvement in baseline ability (see the tall vertical green box) as illustrated in Fig. 4 . Key Findings A significant observation in this study is that ICC can be used to analyze repeated test items. There's a common belief that reused items are of lower quality due to memorization by students. This is not entirely true or false, as the primary goal is to identify and select items that can effectively separate abilities. Utilizing ICC enhances precision in this selection. This study demonstrates that items like numbers 9 and 7 (Q9 and Q7) show clear ability separation despite being reused. ICC also aids in filtering out inconsistent items in terms of separation, such as item number 50 (Q50) (indicated by a long vertical red box). For item 50, it was observed that students with lower abilities had a high probability of guessing correctly. The ICC analysis also revealed transitions from difficult to easy items, as seen in items 39 and 44 (Q39 and Q44). Once high-quality items are identified, the next step is to create an item bank. IRT can be adapted to produce more effective exam responses. A study by Ballis et al. found that exam frame design could enhance student responses in MCQ exams [ 13 ]. Conclusion There was a significant change in average scores before and after IRT implementation, with the initial exam score being 56.1 and the second exam score 74.1 (p < 0.01). Item analysis revealed changes in theta scores for highly difficult items. An increase in baseline ability performance was observed in each item improved through the IRT approach, as indicated by the ICC curve patterns. These findings facilitate a more efficient and precise process of item identification and evaluation, particularly for biochemistry assessments in medical education. Limitations This study, while providing insightful findings, does have its limitations. The sample size, restricted to 89 medical students, may limit the generalizability of the results. Additionally, the study focuses exclusively on biochemistry assessments, which might not reflect the applicability of the findings to other subjects. Future research could benefit from a more diverse sample and a broader range of subjects to strengthen the generalizability of these findings. Declarations Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Conflict of Interest The authors declare that there is no conflict of interest regarding the publication of this paper. References Z. Elhousni, J. LAAMECH, R. ZERHANE, and R. JANATI-IDRISSI, “Difficulties in learning biochemistry: Case of 1st year medical students, Tangier,” Journal for Educators, Teachers and Trainers , vol. 14, no. 1, pp. 63–74, Mar. 2023, doi: 10.47750/JETT.2023.14.01.006. M. FMMT, W. KNH, L. SD, P. MYW, and P. PAJ, “Evaluation of the Teaching Approaches of Biochemistry for Medical Students: A Sri Lankan Case Study,” Journal of Community Medicine & Health Education 2015 5:4 , vol. 5, no. 4, pp. 1–4, Aug. 2015, doi: 10.4172/2161-0711.1000359. E. Wood, “Biochemistry is a difficult subject for both student and teacher,” Biochem Educ , vol. 18, no. 4, pp. 170–172, Oct. 1990, doi: 10.1016/0307-4412(90)90123-6. Steven E. Stemler and Adam Naples, “Rasch Measurement v. Item Response Theory: Knowing When to Cross the Line,” Practical Assessment, Research & Evaluation , vol. 26, no. 11, pp. 1–6, 2021, doi: 10.7275/v2gd-4441. P. F. M. Krabbe, “Item Response Theory,” The Measurement of Health and Health Status , pp. 171–195, Jan. 2017, doi: 10.1016/B978-0-12-801504-9.00010-6. D. Zarate, B. A. Hobson, E. March, M. D. Griffiths, and V. Stavropoulos, “Psychometric properties of the Bergen Social Media Addiction Scale: An analysis using item response theory,” Addictive Behaviors Reports , vol. 17, p. 100473, Jun. 2023, doi: 10.1016/J.ABREP.2022.100473. A. P. Kumar, A. Nayak, K. Manjula Shenoy, S. Goyal, and Chaitanya, “A novel approach to generate distractors for Multiple Choice Questions,” Expert Syst Appl , vol. 225, p. 120022, Sep. 2023, doi: 10.1016/J.ESWA.2023.120022. M. Al-A’Ali, “IRT-Item Response Theory Assessment for an Adaptive Teaching Assessment System,” in 10th WSEAS Interbational Conference on APPLIED MATHEMATICS , Dallas: Applied Mathematics, 2006, pp. 1–5. Accessed: Oct. 10, 2023. [Online]. Available: https://www.researchgate.net/publication/240639436_IRT-Item_Response_Theory_Assessment_for_an_Adaptive_Teaching_Assessment_System M. A. Dorner, P. Sadler, and B. Alters, “Still a private universe? Community college students’ understanding of evolution,” Evolution: Education and Outreach , vol. 16, no. 1, Dec. 2023, doi: 10.1186/S12052-022-00178-Y. S. P. Reise and T. M. Moore, “Item Response Theory,” APA handbook of research methods in psychology: Foundations, planning, measures, and psychometrics (Vol. 1) (2nd ed.). , pp. 809–835, Jun. 2023, doi: 10.1037/0000318-037. A. E. Wyse and S. Hao, “An Evaluation of Item Response Theory Classification Accuracy and Consistency Indices,” Appl Psychol Meas , vol. 36, no. 7, pp. 602–624, Oct. 2012, doi: 10.1177/0146621612451522. L. M. Rudner, “Expected Classification Accuracy,” Practical Assessment, Research, and Evaluation , vol. 10, no. 1, p. 13, Nov. 2019, doi: https://doi.org/10.7275/56a5-6b14. B. Ballis, L. Lusher, and P. Martorell, “The effects of exam frames on student effort and performance,” Econ Educ Rev , vol. 90, p. 102286, Oct. 2022, doi: 10.1016/J.ECONEDUREV.2022.102286. Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3859753","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":266849793,"identity":"bd7f1926-b92c-4d35-9157-fcb0f9f41881","order_by":0,"name":"Baharuddin Baharuddin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKklEQVRIie2RMUvDQBTH3xlIl5OsL8TqVzgJpIYqfhVLoC6XjqJQJEW4LiWuWcSv4NT5QqBbdK2jdA0SCBRFKSa2k6ZSN5H7wXHH8X68978DUCj+IAxotWlEgCbLw64By5sKEuDPin4CEmwz2FgpK9lXBaBOaTXSuLgQ7WZojYoi7zME9OOC9mHPCLTBtEZxRz3PTMWpLXbSMcpJpfQ8i05gP5Lk6qBuMMmZORBJR6A/Bqm/XwJyZlEdyB0QUZvlIbNflwqf5XJRdeH2G13A8Vplyp1VFw4Yi0/FsbYFdNYpbpQ5bnBfZsGug2nIUKfPXfcmRC9K6rO0DG4/Bmft5m3kzfLzOUOj4SfTbH54dD0cPtW92Hf0cm3R6ke0jepXkJffVCsUCsV/5wPRBmGKHZ7MggAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-7079-5748","institution":"Faculty of Medicine, Universitas Surabaya, Surabaya, Indonesia","correspondingAuthor":true,"prefix":"","firstName":"Baharuddin","middleName":"","lastName":"Baharuddin","suffix":""},{"id":266849794,"identity":"36bfc556-7556-40e0-84f6-94867c34e334","order_by":1,"name":"Lilis Handayani","email":"","orcid":"https://orcid.org/0000-0001-7210-1677","institution":"Graduate School of Natural Science and Technology, Kanazawa University, Kanazawa, Japan","correspondingAuthor":false,"prefix":"","firstName":"Lilis","middleName":"","lastName":"Handayani","suffix":""},{"id":266849795,"identity":"b9202af2-6727-4edb-8aca-a6c32d922251","order_by":2,"name":"Rusli Rusli","email":"","orcid":"https://orcid.org/0000-0001-5149-6438","institution":"SUPM Negeri Sorong/Politeknik Kelautan dan Perikanan Sorong, Sorong, Indonesia","correspondingAuthor":false,"prefix":"","firstName":"Rusli","middleName":"","lastName":"Rusli","suffix":""}],"badges":[],"createdAt":"2024-01-13 10:08:31","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-3859753/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3859753/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49713740,"identity":"dc837316-2b61-45a9-9249-16ba6cd986c1","added_by":"auto","created_at":"2024-01-16 20:43:22","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":176447,"visible":true,"origin":"","legend":"\u003cp\u003eWright Map illustrating the identification of difficult items in the assessment.\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-3859753/v1/0496c6a16c0bb077701778d7.jpg"},{"id":49714458,"identity":"f985f6f3-a143-4659-b995-bbe145eebd51","added_by":"auto","created_at":"2024-01-16 20:51:22","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":162420,"visible":true,"origin":"","legend":"\u003cp\u003eWright Map to assist in the identification of difficult items in the assessment.\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-3859753/v1/e8b134e63ba45608431ca284.jpg"},{"id":49713742,"identity":"37fe82c4-8695-47a6-a5f2-bff6a19e1e04","added_by":"auto","created_at":"2024-01-16 20:43:22","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":355677,"visible":true,"origin":"","legend":"\u003cp\u003eICC of Items with High Difficulty Levels in the First Examination (Round 1).\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-3859753/v1/0fd1e3f5849aa2853a7230b8.jpg"},{"id":49713743,"identity":"c53e1be7-0530-4b97-b9b0-b9e4cc453564","added_by":"auto","created_at":"2024-01-16 20:43:22","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":351872,"visible":true,"origin":"","legend":"\u003cp\u003eICC for Items with High Difficulty Levels in the Second Examination (Round 2).\u003c/p\u003e","description":"","filename":"4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-3859753/v1/03843418bdf454546bce68d7.jpg"},{"id":49714479,"identity":"fd53891e-1de5-43a5-906e-9b303d313b19","added_by":"auto","created_at":"2024-01-16 20:51:27","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":498721,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3859753/v1/c23b73a7-6f26-4e30-b70a-424bd248b425.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eEnhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT)\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn medical education, particularly in biochemistry, creating high-quality assessment questions represents a significant challenge. Each item requires detailed evaluation through item analysis. Many students encounter difficulties in biochemistry learning, especially in the preclinical phase [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]\u0026ndash;[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Attempts to improve this using Classical Test Theory (CTT) have been limited, as the analysis focuses only on the items without considering student ability and the probability of guessing. In short, CTT does not account for item-weighted assessment [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Therefore, a more comprehensive method is required, one of which is Item Response Theory (IRT). This method refines the CTT approach. However, research on the application of IRT in medical biochemistry is scarce, yet its findings could provide solutions for medical education units, especially those in charge of biochemistry, to improve question quality.\u003c/p\u003e \u003cp\u003eThe uniqueness of the IRT model approach lies in its sensitivity to the individual's (subject's) response to each item [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], even including calculations for the probability of guessing for a population on an item. This approach utilizes the logit function and logistic parameters, such as discrimination (α), difficulty level (β), and pseudo-guessing parameter (c), to depict the relationship between items and various levels of a latent trait [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. This method is particularly vital in examination systems like medical biochemistry, especially to enhance the quality of evaluation.\u003c/p\u003e \u003cp\u003eTo measure the completeness of learning in medical biochemistry, an assessment-based evaluation is necessary. However, challenges often arise in evaluating exam results. Evaluations are sometimes solely based on student performance scores. Often, there is no item analysis evaluation, leading to situations where students are blamed for low scores, when in fact, the exam questions might be flawed in diction or too difficult compared to the taught material.\u003c/p\u003e \u003cp\u003eThe application process of this method is straightforward, especially when using a single parameter (dichotomous), known as the Rasch Model. With technological advancements, this can be quickly executed using software like Jamovi. Implementing this method will assist in improving distractors in Multiple Choice Questions (MCQ) [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe research questions formulated in this study are:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eWhat is the range of theta performance values for assessment and students?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIs there a significant change in the average score after implementing the revised questions?\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDoes the theta value of items change when distractor modifications are made?\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eThis study utilizes exam data to evaluate question performance [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. The analytical approach consists of pre and post assessments following the implementation of Item Response Theory (IRT) to evaluate student performance scores. The evaluation was conducted on student assessments (n\u0026thinsp;=\u0026thinsp;89) using two approaches. The first approach involved an analysis of average scores, and the second, a comparative quantitative analysis of item performance. The latter also included a qualitative study of the Item Characteristic Curve (ICC).\u003c/p\u003e \u003cp\u003eItems subjected to IRT were tabulated and visualized, presenting theta (θ) scores and their graphical changes in the ICC. To expedite the computational process in this research, Jamovi software with an open-access license was used. To eliminate subjectivity, this study was conducted after grades were published and no further grade disputes from students were present.\u003c/p\u003e"},{"header":"Results and Discussion","content":"\u003cp\u003eThe evaluation was conducted using two approaches. The initial results showed a significant difference in average scores before and after the implementation of Item Response Theory (IRT). The items evaluated and improved using the IRT approach were visualized as follows. In this study, the overall reliability of the measurement tool was assessed, with a person reliability value of 0.780 for the first exam and 0.724 for the second exam. The analysis using IRT on exam data indicated a Person Reliability of 0.724 on a predetermined scale, suggesting that the exam had a good level of reliability in measuring the latent abilities or attributes of the participants.\u003c/p\u003e \u003cp\u003eThere was a significant increase in scores before and after the implementation, with a p-value\u0026thinsp;\u0026lt;\u0026thinsp;0.001 at an alpha of 0.05. The average score for the first exam was 56.1, and for the second exam, it was 74.1. Based on the score distribution, it was also observed that the exam questions effectively differentiated students (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Student abilities (scores) were well-identified across the low-middle-high spectrum, with the central tendency located in the middle area of the histogram curve distribution. These findings align with the study by Dorner et al. [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eWe identified items with very high difficulty levels. This identification was easily accomplished using the Wright Map distribution of the items. This visualization is adequately representative of item difficulty, as it computationally matches the items with the respondents' latent traits (ability to answer). It was evident that there are 11 particularly difficult items, numbered 20, 8, 46, 5, 39, 50, 7, 9, 29, 34, and 44. These items will subsequently be analyzed and improved upon after examining their Item Characteristic Curves (ICC).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eICC analysis is essential for examining three aspects. First, to ascertain the difficulty level of the items. Second, to assess the performance of the items in terms of discrimination. Does the item effectively separate based on the abilities of medical students in biochemistry? Third, to observe the detailed ability of students in responding to the items. A more detailed view of each item's ICC is presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Our findings indicate that there should be a spike (increase in ability) in student responses to items within the 1\u0026ndash;3 ability range, as students in this range possess above-average abilities. However, in reality, students with high-order thinking skills did not respond effectively (unable to answer correctly), especially noticeable in items number 20, 8, and 46 (see the vertical red boxes). Their response abilities were relatively similar to those of students with low-order thinking skills. A notable increase was observed in items number 5, 7, 9, and 34 (see the vertical high green boxes). From this, we gather crucial information about which items require intervention and evaluation.\u003c/p\u003e \u003cp\u003eThis section also highlights the need for internal consolidation in the study. This is a form of reflection for educators on the delivered material. It's important to avoid presenting items that are outside the core competencies formulated. Therefore, the role of the coordinator is crucial, as they must evaluate and consolidate immediately upon identifying any discrepancies.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe change in theta (θ) scores in biochemistry items indicates that the implementation of IRT is effective. This change can be observed in the ICC shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. It's crucial to examine the relationship between student abilities and item difficulty levels within the framework of latent variables, a possibility afforded by the mathematical model system in IRT [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. This IRT technique also aims to enhance accuracy in performance classification [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e], [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA slight yet noticeable improvement in response ability was observed, as depicted in the ICC of the three revised items, numbers 20, 8, and 46 (see the short vertical green boxes). Although the increase is modest, the trend line shows an upward trajectory. Moreover, the ability performance no longer shows depression at the baseline. Item number 5 (Q5) also demonstrates an improvement in baseline ability (see the tall vertical green box) as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003eKey Findings\u003c/h3\u003e\n\u003cp\u003eA significant observation in this study is that ICC can be used to analyze repeated test items. There's a common belief that reused items are of lower quality due to memorization by students. This is not entirely true or false, as the primary goal is to identify and select items that can effectively separate abilities. Utilizing ICC enhances precision in this selection. This study demonstrates that items like numbers 9 and 7 (Q9 and Q7) show clear ability separation despite being reused. ICC also aids in filtering out inconsistent items in terms of separation, such as item number 50 (Q50) (indicated by a long vertical red box). For item 50, it was observed that students with lower abilities had a high probability of guessing correctly. The ICC analysis also revealed transitions from difficult to easy items, as seen in items 39 and 44 (Q39 and Q44). Once high-quality items are identified, the next step is to create an item bank.\u003c/p\u003e \u003cp\u003eIRT can be adapted to produce more effective exam responses. A study by Ballis et al. found that exam frame design could enhance student responses in MCQ exams [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThere was a significant change in average scores before and after IRT implementation, with the initial exam score being 56.1 and the second exam score 74.1 (p\u0026thinsp;\u0026lt;\u0026thinsp;0.01). Item analysis revealed changes in theta scores for highly difficult items. An increase in baseline ability performance was observed in each item improved through the IRT approach, as indicated by the ICC curve patterns. These findings facilitate a more efficient and precise process of item identification and evaluation, particularly for biochemistry assessments in medical education.\u003c/p\u003e"},{"header":"Limitations","content":"\u003cp\u003eThis study, while providing insightful findings, does have its limitations. The sample size, restricted to 89 medical students, may limit the generalizability of the results. Additionally, the study focuses exclusively on biochemistry assessments, which might not reflect the applicability of the findings to other subjects. Future research could benefit from a more diverse sample and a broader range of subjects to strengthen the generalizability of these findings.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.\u003c/p\u003e \u003ch2\u003eConflict of Interest\u003c/h2\u003e \u003cp\u003eThe authors declare that there is no conflict of interest regarding the publication of this paper.\u003c/p\u003e "},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eZ. Elhousni, J. LAAMECH, R. ZERHANE, and R. JANATI-IDRISSI, \u0026ldquo;Difficulties in learning biochemistry: Case of 1st year medical students, Tangier,\u0026rdquo; \u003cem\u003eJournal for Educators, Teachers and Trainers\u003c/em\u003e, vol. 14, no. 1, pp. 63\u0026ndash;74, Mar. 2023, doi: 10.47750/JETT.2023.14.01.006.\u003c/li\u003e\n\u003cli\u003eM. FMMT, W. KNH, L. SD, P. MYW, and P. PAJ, \u0026ldquo;Evaluation of the Teaching Approaches of Biochemistry for Medical Students: A Sri Lankan Case Study,\u0026rdquo; \u003cem\u003eJournal of Community Medicine \u0026amp; Health Education 2015 5:4\u003c/em\u003e, vol. 5, no. 4, pp. 1\u0026ndash;4, Aug. 2015, doi: 10.4172/2161-0711.1000359.\u003c/li\u003e\n\u003cli\u003eE. Wood, \u0026ldquo;Biochemistry is a difficult subject for both student and teacher,\u0026rdquo; \u003cem\u003eBiochem Educ\u003c/em\u003e, vol. 18, no. 4, pp. 170\u0026ndash;172, Oct. 1990, doi: 10.1016/0307-4412(90)90123-6.\u003c/li\u003e\n\u003cli\u003eSteven E. Stemler and Adam Naples, \u0026ldquo;Rasch Measurement v. Item Response Theory: Knowing When to Cross the Line,\u0026rdquo; \u003cem\u003ePractical Assessment, Research \u0026amp; Evaluation\u003c/em\u003e, vol. 26, no. 11, pp. 1\u0026ndash;6, 2021, doi: 10.7275/v2gd-4441.\u003c/li\u003e\n\u003cli\u003eP. F. M. Krabbe, \u0026ldquo;Item Response Theory,\u0026rdquo; \u003cem\u003eThe Measurement of Health and Health Status\u003c/em\u003e, pp. 171\u0026ndash;195, Jan. 2017, doi: 10.1016/B978-0-12-801504-9.00010-6.\u003c/li\u003e\n\u003cli\u003eD. Zarate, B. A. Hobson, E. March, M. D. Griffiths, and V. Stavropoulos, \u0026ldquo;Psychometric properties of the Bergen Social Media Addiction Scale: An analysis using item response theory,\u0026rdquo; \u003cem\u003eAddictive Behaviors Reports\u003c/em\u003e, vol. 17, p. 100473, Jun. 2023, doi: 10.1016/J.ABREP.2022.100473.\u003c/li\u003e\n\u003cli\u003eA. P. Kumar, A. Nayak, K. Manjula Shenoy, S. Goyal, and Chaitanya, \u0026ldquo;A novel approach to generate distractors for Multiple Choice Questions,\u0026rdquo; \u003cem\u003eExpert Syst Appl\u003c/em\u003e, vol. 225, p. 120022, Sep. 2023, doi: 10.1016/J.ESWA.2023.120022.\u003c/li\u003e\n\u003cli\u003eM. Al-A\u0026rsquo;Ali, \u0026ldquo;IRT-Item Response Theory Assessment for an Adaptive Teaching Assessment System,\u0026rdquo; in \u003cem\u003e10th WSEAS Interbational Conference on APPLIED MATHEMATICS\u003c/em\u003e, Dallas: Applied Mathematics, 2006, pp. 1\u0026ndash;5. Accessed: Oct. 10, 2023. [Online]. Available: https://www.researchgate.net/publication/240639436_IRT-Item_Response_Theory_Assessment_for_an_Adaptive_Teaching_Assessment_System\u003c/li\u003e\n\u003cli\u003eM. A. Dorner, P. Sadler, and B. Alters, \u0026ldquo;Still a private universe? Community college students\u0026rsquo; understanding of evolution,\u0026rdquo; \u003cem\u003eEvolution: Education and Outreach\u003c/em\u003e, vol. 16, no. 1, Dec. 2023, doi: 10.1186/S12052-022-00178-Y.\u003c/li\u003e\n\u003cli\u003eS. P. Reise and T. M. Moore, \u0026ldquo;Item Response Theory,\u0026rdquo; \u003cem\u003eAPA handbook of research methods in psychology: Foundations, planning, measures, and psychometrics (Vol. 1) (2nd ed.).\u003c/em\u003e, pp. 809\u0026ndash;835, Jun. 2023, doi: 10.1037/0000318-037.\u003c/li\u003e\n\u003cli\u003eA. E. Wyse and S. Hao, \u0026ldquo;An Evaluation of Item Response Theory Classification Accuracy and Consistency Indices,\u0026rdquo; \u003cem\u003eAppl Psychol Meas\u003c/em\u003e, vol. 36, no. 7, pp. 602\u0026ndash;624, Oct. 2012, doi: 10.1177/0146621612451522.\u003c/li\u003e\n\u003cli\u003eL. M. Rudner, \u0026ldquo;Expected Classification Accuracy,\u0026rdquo; \u003cem\u003ePractical Assessment, Research, and Evaluation\u003c/em\u003e, vol. 10, no. 1, p. 13, Nov. 2019, doi: https://doi.org/10.7275/56a5-6b14.\u003c/li\u003e\n\u003cli\u003eB. Ballis, L. Lusher, and P. Martorell, \u0026ldquo;The effects of exam frames on student effort and performance,\u0026rdquo; \u003cem\u003eEcon Educ Rev\u003c/em\u003e, vol. 90, p. 102286, Oct. 2022, doi: 10.1016/J.ECONEDUREV.2022.102286.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"solution, item response theory, assessment, biochemistry, medical education","lastPublishedDoi":"10.21203/rs.3.rs-3859753/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3859753/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eIn medical education, particularly in biochemistry, crafting high-quality assessment questions is a primary challenge. Each item necessitates thorough evaluation, and precise identification of student abilities is crucial for maximally reflecting learning achievement.\u003c/p\u003e\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eThis study aims to enhance assessment quality in biochemistry medical education by implementing Item Response Theory (IRT). This approach addresses Classical Test Theory (CTT) limitations. Recognizing the critical role of question quality in the learning process, the study investigates how IRT can more holistically and equitably assess student abilities. It includes a comparative analysis of student scores before and after IRT implementation.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eEmploying a mixed-method research approach, this study combines comparative quantitative analysis with qualitative ICC curve analysis in a pre-post experimental design. It focuses on biochemistry exam data from medical students (n\u0026thinsp;=\u0026thinsp;89). IRT is used to measure the probability of student responses to questions, using parameters such as discrimination, difficulty level, and guessing probability. Jamovi software supports this analysis by accelerating computational processes.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eSignificant improvements were observed in both question quality and student scores. Prior to IRT implementation, the average initial exam score was 56.1, which increased to 74.1 in the subsequent exam. The IRT evaluation indicated that the exam questions achieved a more effective differentiation between students of varying abilities. This improvement was evident from the increased person reliability and through Wright Map visualizations, which helped identify highly difficult questions via the Item Characteristic Curve (ICC).\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eThe study advocates for integrating IRT as a standard method in biochemistry medical assessments. It highlights the necessity of assessments that are sensitive to individual student capabilities, providing more precise feedback for enhancing the quality of learning. These findings are crucial for evolving evaluation methodologies and advancing medical education standards.\u003c/p\u003e","manuscriptTitle":"Enhancing Biochemistry Assessment Quality in Medical Education Through Item Response Theory (IRT)","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-16 20:43:17","doi":"10.21203/rs.3.rs-3859753/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c8a7a2d6-5ae8-4349-b121-2844dbc7ad73","owner":[],"postedDate":"January 16th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-05-27T15:59:10+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-16 20:43:17","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3859753","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3859753","identity":"rs-3859753","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}