Cognitive Level Coding as a Predictor of Item Discrimination: An Empirical Study in Anesthetic Pharmacology Education

Cognitive Level Coding as a Predictor of Item Discrimination: An Empirical Study in Anesthetic Pharmacology Education | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Cognitive Level Coding as a Predictor of Item Discrimination: An Empirical Study in Anesthetic Pharmacology Education Qingsong Jiang, Xiaoli Li, Jingyuan Wan, Weiying Zhou, Limei Ma, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8963083/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 12 You are reading this latest preprint version Abstract Background Bloom's taxonomy is widely used to classify examination items by cognitive level, yet its predictive validity for item discrimination remains understudied in pharmacology education. This study investigated whether cognitive level coding predicts item discrimination in an anesthetic pharmacology examination. Methods This retrospective document analysis examined 58 items from a final anesthetic pharmacology examination completed by 194 undergraduate students. Two independent raters coded items using a modified three-level Bloom's taxonomy (recall, interpretation, problem-solving). Item difficulty (P) and discrimination (D) indices were extracted from the official test analysis report. Predictive validity was assessed using Spearman correlation, linear regression, and one‑way ANOVA. Items with D < 0.10 underwent distractor analysis and cognitive attribution. Results The examination demonstrated excellent reliability (Cronbach's α = 0.91), moderate difficulty (P = 0.68), and good overall discrimination (D = 0.45). Cognitive level was positively correlated with item discrimination (rₛ = 0.42, 95% CI [0.18, 0.61], P < 0.01), explaining 16.8% of the variance (R² = 0.168). Problem‑solving items exhibited higher discrimination (0.49 ± 0.18) than recall items (0.34 ± 0.22) (η² = 0.14, P = 0.03). Five items (8.6%) showed low discrimination (D < 0.10), predominantly at the recall level, and were compromised by ceiling effects, dysfunctional distractors, or strong‑distractor‑induced empirical error. Conclusions Cognitive level coding significantly predicts item discrimination in anesthetic pharmacology assessments, with problem‑solving items demonstrating superior discriminatory power. However, predictive validity depends on item‑writing quality. Integrating cognitive blueprints into examination design and refining distractors using empirical data are recommended to enhance assessment quality in pharmacology education. Clinical trial number Not applicable. Pharmacology education Educational measurement Predictive validity Discrimination Bloom’s taxonomy Figures Figure 1 Introduction Anesthetic pharmacology serves as a cornerstone course for undergraduates anesthesia students, bridging foundational pharmacological principles with perioperative clinical practice [ 1 ]. Effectively evaluating this course requires striking a balance between consolidating factual knowledge and cultivating advanced clinical reasoning capabilities, a challenge central to pharmacy and health professions education [ 2 ]. Among the various evaluation metrics, item discrimination (D), defined as an item’s capacity to differentiate between high- and low-achieving students, is a key indicator of assessment quality [ 3 ]. However, despite the widespread use of cognitive level coding, grounded in Bloom's taxonomy, in curriculum design, the extent to which this coding predicts item discrimination in pharmacology education remains unclear. Bloom's revised taxonomy categorizes educational objectives and assessment items into six cognitive levels: remember, understand, apply, analyze, evaluate, and create [ 4 ]. In medical education, a simplified three-tier framework, which covers recall, interpretation, and problem-solving, is routinely adopted to align assessment items with levels of cognitive complexity [ 5 , 6 ]. Recall-level items asses factual retention, interpretation items evaluate the application of fundamental concepts, and problem-solving items require the integration of knowledge across multiple domains to underpin clinical decision-making [ 7 ]. The core premise is that higher-level items, by assessing clinical reasoning, should demonstrate a stronger capacity to differentiate between students with varying competency levels. If supported by empirical evidence, this proposition would furnish predictive validity evidence for cognitive level classification. Predictive validity, is understood as the extent to which a measure predicts a criterion, represents a core component of validity evidence in educational assessment [ 8 ]. In this context, cognitive level coding acts as the predictor, while item discrimination (D) functions as the criterion. Despite the widespread application of Bloom’s taxonomy in health professions education, empirical evidence supporting its predictive value remains limited and contentious. Recently, Ray et al. have called into question the deeply entrenched dependence on Bloom's taxonomy, pointing to its resource-intensive implementation and the absence of robust empirical data linking it to improved educational outcomes [ 9 ]. Liu et al. further challenge the validity of claims that MCQs can adequately assess higher-order cognition, emphasizing the inherent challenges in defining higher-order thinking and verifying whether assessment tasks truly align with the targeted cognitive competencies [ 10 ]. While positive correlations between cognitive level and item discrimination have been documented in fields such as veterinary medicine [ 11 ], evidence-based medicine [ 12 ], and undergraduate biology education [ 13 ], no such evidence exists in pharmacology education, especially within the specialized field of anesthetic pharmacology. This evidentiary gap highlights the urgent need for empirical research to determine whether cognitive level coding can reliably predict item discrimination in this specific context. Given the growing emphasis on competency-based assessment in pharmacy education, this study addresses critical gaps in understanding how cognitive taxonomies are operationalized into assessment instruments, while upholding rigorous validity standards. To realize this aim, the study examines the predictive validity of cognitive level coding, which spans recall, interpretation, and problem-solving, for item discrimination within an anesthetic pharmacology examination. Furthermore, it compares the discriminatory efficacy of items across these three cognitive levels and identifies the cognitive attributes and item-writing shortcomings that contribute to low item discrimination. Drawing on 194 authentic examination papers and comprehensive item-level response data, this research offers the first empirical assessment of the predictive validity of cognitive level coding for item discrimination in pharmacology education. The findings directly respond to recent calls for empirical evidence and put forward targeted recommendations for improving examination quality across pharmacology curricula. METHODS Study design and participants This retrospective document analysis examined the final closed-book, computer-based Anesthetic Pharmacology examination, which was administered in January 2026 to the 2023 cohort of anesthesia undergraduates at a Chinese medical university. The 120-minute examination was aligned with the 2025 Anesthetic Pharmacology Syllabus, and Pharmacology (10th edition) and Anesthetic Pharmacology (4th edition) were designated as core textbooks [ 1 , 14 ]. A total of 194 students completed the examination. De-identified score data, together with the official test paper analysis report containing item difficulty, discrimination, and option selection rates, were retrieved from the university's standard Test Analysis System (2.0). The study was exempt from ethics review, given that it relied on routinely collected educational data. Test paper structure The assessment comprised 58 items, with a total score of 100 points. It consisted of 40 A1-type items (1 point per item), which were single-best-answer multiple-choice questions designed to assess basic knowledge; 10 A2-type items (1 point per item), presented as single-best-answer clinical vignettes; 5 short-answer questions (4 points per item), aimed at evaluating the clear expression of core concepts; 2 essay questions (10 points per item), developed to test knowledge integration and analytical competencies; and 1 case analysis (10 points), focused on clinical decision-making and the rationale for underlying mechanisms. Cognitive level coding Two independent raters, a pharmacology educator and an anesthesiologist, coded all 58 items using a modified three-level Bloom’s taxonomy [ 15 , 16 ]. The coding scheme, operational definitions, and illustrative items from this examination are presented in Table 1 . Prior to full coding process, 10 items (17%) were randomly selected for pilot coding, which demonstrated excellent inter-rater reliability (Cohen’s κ = 0.86). Disagreements were resolved through consensus discussion. Table 1 Cognitive level coding scheme with operational definitions and examples Cognitive level Operational definition Typical item stem features Example from this examination (item number, difficulty, discrimination) Recall Direct retrieval of facts, definitions, or classifications; no inference required “Which of the following belongs to class X?” “What is the mechanism of action of Y?” Item 18: Long-term use of which drug causes irreversible tooth discoloration? (tetracycline, P = 0.97, D = 0.11) Interpretation Understanding concepts; simple inference in a single clinical context “A patient develops symptom Z after taking drug W; the most likely drug is…” Item 44: A patient on anti-tuberculosis therapy develops optic neuritis; the most likely drug is… (ethambutol, P = 0.44, D = 0.74) Problem-solving Integration of multiple knowledge domains; clinical decision-making, differential diagnosis, or justification Requires synthesis of pathophysiology, pharmacology, and clinical context Item 41: Heart transplant patient develops acute rejection after adding rifampicin; the most likely pharmacological cause is… (CYP enzyme induction, P = 0.84, D = 0.36) For predictive validity analysis, cognitive levels were converted to numerical values Recall = 1, Interpretation = 2, Problem-solving = 3. Item statistics and data sources The difficulty index (P) was defined as the ratio of the mean score to the maximum score, with the following interpretations: P > 0.7 denotes an easy item; 0.3 ≤ P ≤ 0.7 denotes a moderate item; and P < 0.3 denotes a difficult item. The discrimination index (D) was calculated using the extreme groups method: students were ranked by total scores, with the top 27% forming the high-performing group and the bottom 27% forming the low-performing group. D was computed as the difference between the mean scores of the high-performing and low-performing groups, divided by the maximum score. The interpretation thresholds were: D ≥ 0.3 indicates good discrimination; 0.2 ≤ D < 0.3 indicates fair discrimination; and D < 0.2 indicates poor discrimination. Reliability was evaluated using Cronbach's α [ 17 ]. All item-level statistics and option selection percentages were sourced from the official test paper analysis report generated by the university's standard Test Analysis System (2.0) and were cross-verified through manual calculation for a randomly selected 10% sample. Predictive validity analysis Spearman rank correlation analysis was employed to assess the bivariate relationship between cognitive level (coded as 1, 2, 3) and item discrimination (D). Simple linear regression was performed, taking cognitive level as the independent variable and D as the dependent variable. The standardized regression coefficient (β), coefficient of determination (R²), and 95% confidence intervals (CIs) were reported. One-way analysis of variance (ANOVA) was used to compare the mean D values across the three cognitive levels, with the effect size quantified as eta-squared (η²). Post-hoc pairwise comparisons were conducted using the least significant difference (LSD) test. The significance threshold was set at α = 0.05. All statistical analyses were conducted using SPSS version 26.0. Cognitive attribution of low-discrimination items Items with a discrimination index (D) below 0.10 were classified as “low-discrimination” and subjected to a comprehensive cognitive attribution analysis. For each such item, the official item analysis report, which detailed the percentage of students selecting each response option, was carefully examined. The identified deficiencies were categorized according to established frameworks [ 11 , 18 ], and encompassed: (1) ceiling effect (P > 0.9, D < 0.10); (2) dysfunctional distractors (one or more options selected by fewer than 5% or more than 95% of students, or the correct option not being the most frequently chosen); (3) strong distractor-induced empirical error (a plausible yet incorrect option that equally attracts high- and low-performing students, leading to a near-zero D value); and (4) zero discrimination (D = 0.00). The cognitive attribution analysis established links between the identified deficiencies and the mismatch between the intended cognitive level and the actual cognitive process elicited by the item. RESULTS Overall examination quality The assessment demonstrated robust internal consistency, with a Cronbach’s α coefficient of 0.91. The mean score was 67.56 ± 18.36, with a median of 71.5 and a range from 14.5 to 99. The overall difficulty level was moderate (P = 0.68), while the overall discrimination performance was favorable (D = 0.45). Table 2 details the difficulty and discrimination indices stratified by item type. Table 2 Difficulty and discrimination indices by item type Item type Number of items Total points Mean score Difficulty (P) Discrimination (D) A1 (single best answer) 40 40 28.81 0.72 0.41 A2 (clinical vignette) 10 10 6.93 0.69 0.40 Short answer 5 20 11.25 0.56 0.57 Essay 2 20 14.38 0.72 0.39 Case analysis 1 10 6.20 0.62 0.42 Whole test 58 100 67.56 0.68 0.45 Distribution of cognitive levels Of the 58 items, 28 (48.3%) were categorized as recall, 18 (31.0%) as interpretation, and 12 (20.7%) as problem-solving. Although the three cognitive levels contributed similar proportions of total points (34%, 33%, and 33%, respectively), recall items made up nearly half of all items, while problem-solving items account for only one-fifth. Predictive validity of cognitive level coding for item discrimination Spearman correlation analysis revealed a positive association between cognitive level and item discrimination (r s = 0.42, 95% CI [0.18, 0.61], P < 0.01). Simple linear regression (Fig. 1 ) showed that cognitive level significantly predicted item discrimination (β = 0.41, t = 3.37, P < 0.01), with the regression equation formulated as D = 0.20 + 0.09 × cognitive level. Cognitive level explained 16.8% of the variance in item discrimination (R² = 0.168, adjusted R² = 0.153). One-way ANOVA (Table 3 ) indicated a statistically significant difference in the mean D value across the three cognitive levels (F 2, 55 = 4.57, P = 0.03, η² = 0.14, representing a medium effect size). Post-hoc LSD tests demonstrated that problem-solving items had higher discrimination (0.49 ± 0.18) than recall items (0.34 ± 0.22), with a mean difference of 0.15 (95% CI [0.02, 0.28], P = 0.03). Interpretation items (0.38 ± 0.21) showed no significant differences from either recall items ( P = 0.48) or problem-solving items ( P = 0.12). No significant differences in difficulty were found across the three cognitive levels (F 2, 55 = 1.96, P = 0.15), confirming that the observed differences in discrimination were not confounded by item difficulty. Table 3 Difficulty and discrimination of items by cognitive level Cognitive level N (%) Points (%) Difficulty (P)(Mean ± SD) Discrimination (D) (Mean ± SD) Student score rate (%) Recall 28 (48.3) 34 (34) 0.72 ± 0.19 0.34 ± 0.22 72 Interpretation 18 (31.0) 33 (33) 0.67 ± 0.17 0.38 ± 0.21 67 Problem-solving 12 (20.7) 33 (33) 0.65 ± 0.15 0.49 ± 0.18 65 F (P) – – 1.96 (0.15) 4.57 (0.03) – η² (effect size) – – – 0.14 – Note: SD = standard deviation. Effect size η² = 0.14 indicates a medium effect. Cognitive attribution of low-discrimination items Five items, accounting for 8.6% of the total, had discrimination indices below 0.10, contributing 5 points (representing 5% of the total score). Among these items, three (60%) was classified as recall-level, and one each fell into the interpretation and problem-solving categories. Table 4 provides a detailed breakdown of their cognitive levels, discrimination indices, and defect classifications. Table 4 Low-discrimination items (D < 0.10): cognitive level, difficulty, discrimination, and defect attribution Item No. Type Cognitive level Difficulty (P) Discrimination (D) Defect attribution (based on option selection percentages) 1 A1 Recall 0.94 0.08 Ceiling effect: 94.3% answered correctly, leaving virtually no variance to discriminate. 2 A1 Interpretation 0.35 0.02 Strong-distractor-induced empirical error: 60.8% of students selected B (“double the regular first dose”), misled by the common clinical heuristic “double first dose.” High- and low-performers chose B at similar rates (61.2% vs. 60.3%). The intended cognitive level (interpretation) was not realized; the item functioned as disguised recall. 3 A1 Recall 0.61 0.06 Dysfunctional distractors: Distractors A, C and E attracted 10.8%, 11.9% and 14.4% of students respectively; the correct answer B was chosen by only 61.3% – barely above chance. 37 A1 Recall 0.42 0.00 Zero discrimination: High- and low-performers selected the four distractors in almost identical proportions. The item contributed no information about student ability. 47 A2 Problem-solving 0.69 0.08 Concept-migration error: 26.3% of students erroneously selected B (“inhibition of TXA 2 synthesis”), incorrectly transferring the antiplatelet mechanism of aspirin to the context of acute gastric mucosal injury. In-depth case analysis – Item 2: When a “clinical heuristic” becomes a dysfunctional distractor Item 2 presented the following question: “For a drug exhibiting first-order elimination and conforming to a one-compartment model, which dosing regimen is optimal for rapidly achieving steady-state concentrations via a conventional oral schedule?” The correct answer(Option D) was “administering a loading dose, followed by maintenance doses.” Notably, 60.8% of students chose Option B (“doubling the initial standard dose”), a pattern observed consistently across both high- and low-performing cohorts. As a result, the item demonstrated negligible discrimination (D = 0.02). Students had memorized the clinical heuristic of a “double first dose” but lacked an understanding of the pharmacokinetic rationale behind loading dose calculation. The item stem’s mention of a “conventional oral regimen,” paired with option B (“double the regular first dose”), created a strong semantic link that triggered retrieval of the memorized heuristic instead of first-principles reasoning. Although categorized at the interpretation cognitive level, the actual cognitive process evoked was merely the recall of a rote-memorized rule. This gap between the intended and actual cognitive engagement completely compromised the item's predictive validity. Therefore, the predictive validity of cognitive level coding is conditional, not absolute; it fundamentally depends on how accurately the item presentation elicits the intended cognitive process. Items designed to evaluate higher-order cognition may fail if distractors inadvertently encourage reliance on rote memorization. Discussion This study offers the first empirical evidence in pharmacology education, showing that cognitive level classifications (recall, interpretation, and problem-solving) serve as a significant predictor of item discrimination. Problem-solving items demonstrated substantially higher discriminatory efficacy compared with recall items, and cognitive level explained 16.8% of the variation in discrimination. Further examination of low-discrimination items revealed that predictive validity hinges on item-writing quality, highlighting the necessity of rigorous item-development protocols. Predictive validity of cognitive level coding: Evidence and boundary conditions This study provides novel empirical insights for pharmacology education, showing that cognitive level coding (recall, interpretation, and problem-solving) acts as a robust predictor of item discrimination. The positive Spearman correlation (rₛ = 0.42), moderate regression effect size (R² = 0.168), and the ANOVA comparison between recall and problem-solving items jointly confirm the predictive validity of this coding scheme. These results are consistent with the findings of Buljan et al. [ 12 ], who noted that items classified at the “apply” and “analyze” levels in an evidence-based medicine assessment had higher median discrimination than “remember”-level items, thereby supporting the view that increased cognitive demands strengthen an item's capacity to differentiate between students of varying proficiency. Likewise, Javaid et al. found that greater item complexity defined by higher cognitive demand and the integration of multiple concepts correlated with improved discrimination across various health sciences examinations. The effect size in this study (η² = 0.14) is comparable to that reported in veterinary education [ 11 ], where items requiring higher-order thinking also demonstrated superior discriminatory power. Collectively, these findings emphasize consistency with transformative pedagogical practices, underscoring the efficacy of cognitive frameworks in enhancing assessment validity. Notably, predictive validity hinges critically on item construction quality. Item 2 serves as a boundary case: its intended cognitive demand (interpretation) was compromised, as superficial cues (e.g., “double the regular first dose”) triggered a low-level retrieval strategy. This phenomenon, termed “pseudo-interpretation” or “stem-matching cue,” has been identified as a substantial threat to the validity of multiple-choice items [ 19 , 20 ]. Despite a reasonable difficulty index (P = 0.35), the item's negligible discrimination reveals that cognitive level coding alone is insufficient; items must be designed to compel examinees to engage at the targeted cognitive levels. Consequently, predictive validity relies on both the coding framework and the rigor of item design. This perspective directly responds to the critique from Ray et al. [ 9 ] that “mapping alone may not improve outcomes.” Their scoping review of 24 studies on the association between Bloom's taxonomy mapping and educational outcomes in health professions education found that, despite widespread adoption, empirical support for aligning examination questions with learning taxonomies remains limited, highlighting the need for rigorous item construction to ensure that intended cognitive levels are authentically enacted. Structural Imbalance: Scarce but superior problem‑solving items Problem-solving items constituted merely one-fifth of the examination but contributed one-third of the total points. This “high-weight, low-count” pattern is common in medical education assessments [ 21 ]. Although each problem-solving item produced considerable score variance, their limited number restricted the test’s overall discriminatory power and magnified the influence of individual defective items (such as item 47). Hence, raising the proportion of problem-solving items is advisable to stabilize the total score distribution. For example, shifting from the existing 48:31:21 allocation (recall:interpretation:problem-solving) to a more balanced 40:30:30 or even 30:40:30 distribution would leverage the enhanced discriminatory capability of higher-level items while preventing excessive weighting of individual items. Cognitive attribution of low‑discrimination items: A diagnostic tool for item improvement Our cognitive attribution analysis reveals that low-discrimination items are systematic rather than random; they cluster at the recall level and display identifiable, correctable flaws. Ceiling effects, with item 1 as a typical example, can be mitigated by upgrading items to the interpretation level—for instance, by incorporating simple clinical scenarios. Dysfunctional distractors, such as those in item 3, can be eliminated or substituted based on empirical distractor analysis [ 18 ]. Most significantly, empirical errors induced by potent distractors (as observed in item 2) present unique opportunity, as they accurately pinpoint where students’ mental models deviate from expert comprehension [ 22 ]. Instead of discarding these potent distractors, item stems should be refined to remove superficial cues while preserving the diagnostic value of the distractors. For example, item 2 can be improved by modifying the stem to specify “based exclusively on pharmacokinetic principles” and restructuring option B as “administer an initial dose that exceeds the standard maintenance dose,” thus eliminating direct dependence on the “double dose” heuristic. Item 47, the sole problem-solving item exhibiting low discrimination, revealed a concept-transfer error: students erroneously extended the antiplatelet mechanism (driven by thromboxane A₂ reduction) to the mechanism underlying gastric mucosal injury (driven by prostaglandin E₂ reduction). This constitutes a classic, deeply entrenched misconception that demands targeted instructional intervention. Consequently, pharmacology educators should explicitly juxtapose these two mechanisms when teaching nonsteroidal anti-inflammatory drugs. Implications for pharmacology education Although grounded in anesthetic pharmacology, a specialized disciplinary application, the cognitive level coding framework and its proven predictive validity for item discrimination demonstrate broad applicability across disciplinary boundaries. Pharmacology curricula in pharmacy and health professions programs face similar assessment challenges, particularly in integrating factual mastery with clinical reasoning. Thus, our findings offer three actionable, transferable recommendations for pharmacology educators aiming to strengthen assessment rigor. First, educators should integrate a cognitive level blueprint into the pre-examination item vetting process. Predetermined ratios for recall, interpretation, and problem-solving items (for example, 40:30:30) should be set prior to exam development. Item writers must specify the intended cognitive level of each question and ensure that correct answers cannot be obtained solely through lower order strategies. Second, ongoing distractor refinement, guided by real student response data, is essential. After examination administration, distractor analysis must be performed on items whose discrimination indices are below 0.20. When a distractor triggers empirical errors in both high- and low-achieving examinees, the stem should be modified to eliminate superficial cues while preserving the distractor’s diagnostic utility. Institutions are urged to build local “pharmacology misconception repository” to support future item development and instructional planning. Third, educators should raise the proportion of problem-solving items to capitalize on their superior discriminatory capability. Effective problem-solving items generally entail synthesizing at least two pharmacological principles, integrate clinical decision points, and include distractors that mirror genuine clinical reasoning errors. Integrating these recommendations, we put forward a three-step quality assurance protocol for item development: (1) pre-testing to validate cognitive alignment, ensuring items are targeted at the intended cognitive levels; (2) empirical distractor assessment grounded in student response data; and (3) sustaining a balanced distribution of items across cognitive levels. These guidelines are in line with current competency-based assessment requirements in medical education [ 23 ] and can be readily implementable within core pharmacology courses. Limitations and future research This study involves several methodological limitations that call for cautious interpretation. First, the analysis was restricted to a single institution and a single examination, which constrains generalizability; thus, future research should cover multiple institutions, diverse pharmacology curricula, and varied assessment approaches to confirm broader applicability. Second, although cognitive level coding showed high inter-rater reliability (Cohen's κ = 0.86), it is still open to subjective judgment. Assigning equal numerical intervals (1, 2, 3) in regression analysis, a common practice in educational measurement, presupposes equidistance among cognitive levels, and this presumption may not accurately mirror the underlying construct. Subsequent research could tackle this limitation by applying ordinal logistic regression or item response theory (IRT) models, treating cognitive level as a categorical predictor without enforcing linear constraints. Third, the study examined only item discrimination as an indicator of predictive validity; the predictive value of cognitive level coding for other critical educational outcomes, such as clinical competence, long-term knowledge retention, and performance in licensure examinations, remains uncharted. Addressing these limitations outlines several crucial directions for future research. First, prospective intervention studies are needed to assess examination quality before and after the adoption of cognitive level blueprints, generating causal evidence of their effectiveness. Second, multi-center studies across diverse pharmacology courses and institutions are vital to confirm the generalizability of the predictive validity observed in this single-center study. Third, analyses based on IRT can clarify whether cognitive level influences item information functions, offering detailed psychometric insights into item performance across varying ability levels. Fourth, qualitative think-aloud studies are essential to verify the actual cognitive processes students use when answering items classified at different cognitive levels, thereby validating the alignment between intended cognitive demands and students' response strategies. Together, advancing these research paths will reinforce the evidence foundation for using cognitive level coding to enhance assessment rigor in pharmacology education. Conclusions This study offers the first empirical evidence in pharmacology education that cognitive level coding significantly predicts item discrimination. Problem-solving items exhibit greater discriminatory power than recall items, verifying that higher cognitive demand elevates assessment quality. Nevertheless, this predictive validity is contingent on item writing quality—superficial cues can compromise even properly classified items. These findings advocate for incorporating cognitive blueprints into examination design and leveraging empirical data to refine distractors, thereby propelling competency-based assessment forward in pharmacy education. Declarations Ethics approval and consent to participate This study complied with the ethical standards set forth in the Declaration of Helsinki (2013 revision). The research utilized fully anonymized routine educational assessment data (de-identified student examination scores and item-level response data) with no direct interaction with participants, no collection of personal identifiable information, and no intervention in students’ learning or assessment processes. In accordance with the Research Ethics Review Guidelines for Human Biological Samples and Data (2023) issued by the National Health Commission of the People’s Republic of China and the ethical review regulations of Chongqing Medical University’s Institutional Review Board (IRB), this type of retrospective educational research based on anonymized routine data is exempt from formal ethics committee approval. The Chongqing Medical University IRB has confirmed the waiver of ethical review for this study (no formal IRB approval number is required for exempted research per institutional rules). Informed consent from participants was also unnecessary due to the anonymized nature of the data and the non-interventional study design. Consent for publication Not applicable. Data availability The datasets generated and analyzed during the current study are based on routine student data collection related to university programming and are not publicly available. Competing interests The authors declare no competing interests. Fundings This study was supported by Chongqing Association of Higher Education 2025-2026 Annual Higher Education Scientific Research Project (No. cqgj25031C), 2025 Medical Education Research Planning Project of Chongqing Medical University (No. J0625017), and 2025 Chongqing Higher Education Teaching Reform Research Project of Chongqing Medical University (No. 253075). Authors’ contributions Study design: Qingsong Jiang, Ke Wu and Hongmei Qiu. Data analysis: Xiaoli Li. Data collection: Jingyuan Wan, Weiying Zhou, Limei Ma and Qingzhu Ren. Writing of the manuscript: Qingsong Jiang, Ke Wu and Hongmei Qiu. All authors read and approved the final version of the manuscript and agreed to its submission. Acknowledgements The authors would like to thank all the students who agreed to complete the final examination. References Yu T, Wang GL. Anesthetic Pharmacology. 4th ed. Beijing: People's Medical Publishing House; 2024. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7. Sim SM, Rasiah RI. Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singap. 2006;35(2):67–71. Adams NE. Bloom's taxonomy of cognitive learning objectives. J Med Libr Assoc. 2015;103(3):152–3. Tai H, Kovarik C. ChatGPT-4's level of dermatological knowledge based on board examination review questions and Bloom's Taxonomy. JMIR Dermatol. 2025;8:e74085. Tran T, Le U, Phan V. Evaluating the accuracy and educational potential of generative AI models in pharmacy education: a comparative analysis of ChatGPT and Gemini across Bloom's Taxonomy. Pharm (Basel). 2025;14(1):1. Cohen Aubart F, Lhote R, Hertig A, Noel N, Costedoat-Chalumeau N, Cariou A, Meyer G. Progressive clinical case-based multiple-choice questions: an innovative way to evaluate and rank undergraduate medical students. Rev Med Interne. 2021;42(5):302–9. American Educational Research Association, American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014. Ray ME, Rudolph MJ, Daugherty KK. Bloom's taxonomy in health professions education: associations with exam scores, clinical reasoning, and instructional effectiveness: a scoping review. Curr Pharm Teach Learn. 2025;17(11):102444. Liu Q, Wald N, Daskon C, Harland T. Multiple-choice questions (MCQs) for higher-order cognition: perspectives of university teachers. Innov Educ Teach Int. 2024;61(4):802–14. Rush BR, Rankin DC, White BJ. The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Med Educ. 2016;16(1):250. Buljan I, Marušić M, Tokalić R, Viđak M, Peričić TP, Hren D, Marušićet A. Cognitive levels in testing knowledge in evidence-based medicine: a cross sectional study. BMC Med Educ. 2021;21(1):25. Hillsley K. Question format is the best predictor of item discrimination: a multivariable analysis. J Microbiol Biol Educ. 2025;26(3):e0020525. Yang BF, Chen JG. Pharmacology. 10th ed. Beijing: People's Medical Publishing House; 2024. Anderson LW, Krathwohl DR. A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. New York: Longman; 2001. Crowe A, Dirks C, Wenderoth MP. Biology in bloom: implementing Bloom's taxonomy to enhance student learning in biology. CBE Life Sci Educ. 2008;7(4):368–81. Wang Z, Zheng XL, He X, Liu J. Optimizing pathophysiology instruction in dental school: examination paper analysis and strategic reflections. BMC Med Educ. 2025;25(1):754. Costello E, Holland JC, Kirwan C. Evaluation of MCQs from MOOCs for common item writing flaws. BMC Res Notes. 2018;11(1):849. Monrad SU, Bibler Zaidi NL, Grob KL, Kurtz JB, Tai AW, Hortsch M, et al. What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy. Med Teach. 2021;43(5):575–82. Tariq S, Tariq S, Maqsood S, Jawed S, Baiget M. Evaluation of cognitive levels and item writing flaws in medical pharmacology internal assessment examinations. Pak J Med Sci. 2017;33(4):866–70. Davies DJ, McLean PF, Kemp PR, Liddle AD, Morrell MJ, Halse O, et al. Assessment of factual recall and higher-order cognitive domains in an open-book medical school examination. Adv Health Sci Educ Theory Pract. 2022;27(1):147–65. National Research Council. How people learn: brain, mind, experience, and school. Expanded ed. Washington, DC: National Academies; 2000. Ennab F, Abdulkareem RH, Kaddoura R, Hegazy A, Lootah M, Banerjee Y, et al. Exploring undergraduate medical students' perception of an integrated longitudinal research curriculum within a competency-based framework. PLoS ONE. 2026;21(2):e0343409. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviews received at journal 21 Apr, 2026 Reviews received at journal 21 Apr, 2026 Reviews received at journal 14 Apr, 2026 Reviewers agreed at journal 12 Apr, 2026 Reviewers agreed at journal 10 Apr, 2026 Reviewers agreed at journal 07 Apr, 2026 Reviewers agreed at journal 03 Apr, 2026 Reviewers invited by journal 03 Apr, 2026 Editor assigned by journal 31 Mar, 2026 Editor invited by journal 10 Mar, 2026 Submission checks completed at journal 08 Mar, 2026 First submitted to journal 08 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8963083","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":619442079,"identity":"53d85377-6447-4620-b04b-22e820de46e9","order_by":0,"name":"Qingsong Jiang","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Qingsong","middleName":"","lastName":"Jiang","suffix":""},{"id":619442080,"identity":"11bdd4e1-0c2f-4019-9198-d222417b075d","order_by":1,"name":"Xiaoli Li","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Xiaoli","middleName":"","lastName":"Li","suffix":""},{"id":619442081,"identity":"34489d79-fc01-4ece-99f1-e2d8366ac24d","order_by":2,"name":"Jingyuan Wan","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Jingyuan","middleName":"","lastName":"Wan","suffix":""},{"id":619442082,"identity":"c64958e2-6563-4206-bcfa-62bcd8ddfd88","order_by":3,"name":"Weiying Zhou","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Weiying","middleName":"","lastName":"Zhou","suffix":""},{"id":619442083,"identity":"d81a816e-6753-44d2-a3f8-a7c756b625ab","order_by":4,"name":"Limei Ma","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Limei","middleName":"","lastName":"Ma","suffix":""},{"id":619442084,"identity":"5c0a32ac-757c-439d-bb11-c25b020fd7ca","order_by":5,"name":"Qingzhu Ren","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Qingzhu","middleName":"","lastName":"Ren","suffix":""},{"id":619442085,"identity":"3264f43e-4369-42d8-b777-8e495d5d3191","order_by":6,"name":"Ke Wu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAArUlEQVRIiWNgGAWjYFACNgaGD1CmBNFaGGeQrIWZhyQtBjfSEh/b1FhHGxxgPnibh8Eujxgth41zjqXnbjjAlmzNw5BcTFCL2Y30NunchsNALTxm0jwMBxIbiNDS/tsSrIX/G7Fa0o4xM0JsYSNOi/2ZZ8mSPUC/zDzMZmw5xyCZsBbJ9jTDDz9qrHP7jjc/vPGmwo6wFihgBiNgABKpngGqfhSMglEwCkYBdgAApQA7nWWqjvEAAAAASUVORK5CYII=","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":true,"prefix":"","firstName":"Ke","middleName":"","lastName":"Wu","suffix":""},{"id":619442086,"identity":"7d054ace-74b4-41c5-b1b9-9f0a3257da2b","order_by":7,"name":"Hongmei Qiu","email":"","orcid":"","institution":"Chongqing Medical University","correspondingAuthor":false,"prefix":"","firstName":"Hongmei","middleName":"","lastName":"Qiu","suffix":""}],"badges":[],"createdAt":"2026-02-25 04:38:46","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8963083/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8963083/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":106574369,"identity":"57a5fd37-e218-4b7d-9a3e-48407f0aa780","added_by":"auto","created_at":"2026-04-10 04:48:28","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":133452,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eScatter plot with regression line: Cognitive level predicts item discrimination.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Figure1CognitiveLevelDiscrimination1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-8963083/v1/641992d0ea589a6b1ae3cc7c.jpg"},{"id":106725187,"identity":"1363594a-ae55-412b-9a01-91f1c370b820","added_by":"auto","created_at":"2026-04-12 18:31:45","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1238309,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8963083/v1/1bf2bf52-85d3-47a0-bd53-08ab4e32eb2b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Cognitive Level Coding as a Predictor of Item Discrimination: An Empirical Study in Anesthetic Pharmacology Education","fulltext":[{"header":"Introduction","content":"\u003cp\u003eAnesthetic pharmacology serves as a cornerstone course for undergraduates anesthesia students, bridging foundational pharmacological principles with perioperative clinical practice [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Effectively evaluating this course requires striking a balance between consolidating factual knowledge and cultivating advanced clinical reasoning capabilities, a challenge central to pharmacy and health professions education [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Among the various evaluation metrics, item discrimination (D), defined as an item\u0026rsquo;s capacity to differentiate between high- and low-achieving students, is a key indicator of assessment quality [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. However, despite the widespread use of cognitive level coding, grounded in Bloom's taxonomy, in curriculum design, the extent to which this coding predicts item discrimination in pharmacology education remains unclear.\u003c/p\u003e \u003cp\u003eBloom's revised taxonomy categorizes educational objectives and assessment items into six cognitive levels: remember, understand, apply, analyze, evaluate, and create [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. In medical education, a simplified three-tier framework, which covers recall, interpretation, and problem-solving, is routinely adopted to align assessment items with levels of cognitive complexity [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Recall-level items asses factual retention, interpretation items evaluate the application of fundamental concepts, and problem-solving items require the integration of knowledge across multiple domains to underpin clinical decision-making [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. The core premise is that higher-level items, by assessing clinical reasoning, should demonstrate a stronger capacity to differentiate between students with varying competency levels. If supported by empirical evidence, this proposition would furnish predictive validity evidence for cognitive level classification.\u003c/p\u003e \u003cp\u003ePredictive validity, is understood as the extent to which a measure predicts a criterion, represents a core component of validity evidence in educational assessment [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. In this context, cognitive level coding acts as the predictor, while item discrimination (D) functions as the criterion. Despite the widespread application of Bloom\u0026rsquo;s taxonomy in health professions education, empirical evidence supporting its predictive value remains limited and contentious. Recently, Ray et al. have called into question the deeply entrenched dependence on Bloom's taxonomy, pointing to its resource-intensive implementation and the absence of robust empirical data linking it to improved educational outcomes [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Liu et al. further challenge the validity of claims that MCQs can adequately assess higher-order cognition, emphasizing the inherent challenges in defining higher-order thinking and verifying whether assessment tasks truly align with the targeted cognitive competencies [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. While positive correlations between cognitive level and item discrimination have been documented in fields such as veterinary medicine [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e], evidence-based medicine [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], and undergraduate biology education [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], no such evidence exists in pharmacology education, especially within the specialized field of anesthetic pharmacology. This evidentiary gap highlights the urgent need for empirical research to determine whether cognitive level coding can reliably predict item discrimination in this specific context.\u003c/p\u003e \u003cp\u003eGiven the growing emphasis on competency-based assessment in pharmacy education, this study addresses critical gaps in understanding how cognitive taxonomies are operationalized into assessment instruments, while upholding rigorous validity standards. To realize this aim, the study examines the predictive validity of cognitive level coding, which spans recall, interpretation, and problem-solving, for item discrimination within an anesthetic pharmacology examination. Furthermore, it compares the discriminatory efficacy of items across these three cognitive levels and identifies the cognitive attributes and item-writing shortcomings that contribute to low item discrimination. Drawing on 194 authentic examination papers and comprehensive item-level response data, this research offers the first empirical assessment of the predictive validity of cognitive level coding for item discrimination in pharmacology education. The findings directly respond to recent calls for empirical evidence and put forward targeted recommendations for improving examination quality across pharmacology curricula.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eStudy design and participants\u003c/h2\u003e \u003cp\u003eThis retrospective document analysis examined the final closed-book, computer-based Anesthetic Pharmacology examination, which was administered in January 2026 to the 2023 cohort of anesthesia undergraduates at a Chinese medical university. The 120-minute examination was aligned with the 2025 Anesthetic Pharmacology Syllabus, and \u003cem\u003ePharmacology\u003c/em\u003e (10th edition) and \u003cem\u003eAnesthetic Pharmacology\u003c/em\u003e (4th edition) were designated as core textbooks [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eA total of 194 students completed the examination. De-identified score data, together with the official test paper analysis report containing item difficulty, discrimination, and option selection rates, were retrieved from the university's standard Test Analysis System (2.0). The study was exempt from ethics review, given that it relied on routinely collected educational data.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eTest paper structure\u003c/h3\u003e\n\u003cp\u003eThe assessment comprised 58 items, with a total score of 100 points. It consisted of 40 A1-type items (1 point per item), which were single-best-answer multiple-choice questions designed to assess basic knowledge; 10 A2-type items (1 point per item), presented as single-best-answer clinical vignettes; 5 short-answer questions (4 points per item), aimed at evaluating the clear expression of core concepts; 2 essay questions (10 points per item), developed to test knowledge integration and analytical competencies; and 1 case analysis (10 points), focused on clinical decision-making and the rationale for underlying mechanisms.\u003c/p\u003e\n\u003ch3\u003eCognitive level coding\u003c/h3\u003e\n\u003cp\u003eTwo independent raters, a pharmacology educator and an anesthesiologist, coded all 58 items using a modified three-level Bloom\u0026rsquo;s taxonomy [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. The coding scheme, operational definitions, and illustrative items from this examination are presented in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Prior to full coding process, 10 items (17%) were randomly selected for pilot coding, which demonstrated excellent inter-rater reliability (Cohen\u0026rsquo;s κ\u0026thinsp;=\u0026thinsp;0.86). Disagreements were resolved through consensus discussion.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCognitive level coding scheme with operational definitions and examples\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCognitive level\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOperational definition\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTypical item stem features\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExample from this examination (item number, difficulty, discrimination)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDirect retrieval of facts, definitions, or classifications; no inference required\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026ldquo;Which of the following belongs to class X?\u0026rdquo; \u0026ldquo;What is the mechanism of action of Y?\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eItem 18: Long-term use of which drug causes irreversible tooth discoloration? (tetracycline, P\u0026thinsp;=\u0026thinsp;0.97, D\u0026thinsp;=\u0026thinsp;0.11)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eUnderstanding concepts; simple inference in a single clinical context\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026ldquo;A patient develops symptom Z after taking drug W; the most likely drug is\u0026hellip;\u0026rdquo;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eItem 44: A patient on anti-tuberculosis therapy develops optic neuritis; the most likely drug is\u0026hellip; (ethambutol, P\u0026thinsp;=\u0026thinsp;0.44, D\u0026thinsp;=\u0026thinsp;0.74)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProblem-solving\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIntegration of multiple knowledge domains; clinical decision-making, differential diagnosis, or justification\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRequires synthesis of pathophysiology, pharmacology, and clinical context\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eItem 41: Heart transplant patient develops acute rejection after adding rifampicin; the most likely pharmacological cause is\u0026hellip; (CYP enzyme induction, P\u0026thinsp;=\u0026thinsp;0.84, D\u0026thinsp;=\u0026thinsp;0.36)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eFor predictive validity analysis, cognitive levels were converted to numerical values\u003c/strong\u003e \u003cp\u003eRecall\u0026thinsp;=\u0026thinsp;1, Interpretation\u0026thinsp;=\u0026thinsp;2, Problem-solving\u0026thinsp;=\u0026thinsp;3.\u003c/p\u003e \u003c/p\u003e\n\u003ch3\u003eItem statistics and data sources\u003c/h3\u003e\n\u003cp\u003eThe difficulty index (P) was defined as the ratio of the mean score to the maximum score, with the following interpretations: P\u0026thinsp;\u0026gt;\u0026thinsp;0.7 denotes an easy item; 0.3\u0026thinsp;\u0026le;\u0026thinsp;P\u0026thinsp;\u0026le;\u0026thinsp;0.7 denotes a moderate item; and P\u0026thinsp;\u0026lt;\u0026thinsp;0.3 denotes a difficult item. The discrimination index (D) was calculated using the extreme groups method: students were ranked by total scores, with the top 27% forming the high-performing group and the bottom 27% forming the low-performing group. D was computed as the difference between the mean scores of the high-performing and low-performing groups, divided by the maximum score. The interpretation thresholds were: D\u0026thinsp;\u0026ge;\u0026thinsp;0.3 indicates good discrimination; 0.2\u0026thinsp;\u0026le;\u0026thinsp;D\u0026thinsp;\u0026lt;\u0026thinsp;0.3 indicates fair discrimination; and D\u0026thinsp;\u0026lt;\u0026thinsp;0.2 indicates poor discrimination. Reliability was evaluated using Cronbach's α [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. All item-level statistics and option selection percentages were sourced from the official test paper analysis report generated by the university's standard Test Analysis System (2.0) and were cross-verified through manual calculation for a randomly selected 10% sample.\u003c/p\u003e\n\u003ch3\u003ePredictive validity analysis\u003c/h3\u003e\n\u003cp\u003eSpearman rank correlation analysis was employed to assess the bivariate relationship between cognitive level (coded as 1, 2, 3) and item discrimination (D). Simple linear regression was performed, taking cognitive level as the independent variable and D as the dependent variable. The standardized regression coefficient (β), coefficient of determination (R\u0026sup2;), and 95% confidence intervals (CIs) were reported. One-way analysis of variance (ANOVA) was used to compare the mean D values across the three cognitive levels, with the effect size quantified as eta-squared (η\u0026sup2;). Post-hoc pairwise comparisons were conducted using the least significant difference (LSD) test. The significance threshold was set at \u003cem\u003eα\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.05. All statistical analyses were conducted using SPSS version 26.0.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eCognitive attribution of low-discrimination items\u003c/h2\u003e \u003cp\u003eItems with a discrimination index (D) below 0.10 were classified as \u0026ldquo;low-discrimination\u0026rdquo; and subjected to a comprehensive cognitive attribution analysis. For each such item, the official item analysis report, which detailed the percentage of students selecting each response option, was carefully examined. The identified deficiencies were categorized according to established frameworks [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], and encompassed: (1) ceiling effect (P\u0026thinsp;\u0026gt;\u0026thinsp;0.9, D\u0026thinsp;\u0026lt;\u0026thinsp;0.10); (2) dysfunctional distractors (one or more options selected by fewer than 5% or more than 95% of students, or the correct option not being the most frequently chosen); (3) strong distractor-induced empirical error (a plausible yet incorrect option that equally attracts high- and low-performing students, leading to a near-zero D value); and (4) zero discrimination (D\u0026thinsp;=\u0026thinsp;0.00). The cognitive attribution analysis established links between the identified deficiencies and the mismatch between the intended cognitive level and the actual cognitive process elicited by the item.\u003c/p\u003e \u003c/div\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eOverall examination quality\u003c/h2\u003e \u003cp\u003eThe assessment demonstrated robust internal consistency, with a Cronbach\u0026rsquo;s α coefficient of 0.91. The mean score was 67.56\u0026thinsp;\u0026plusmn;\u0026thinsp;18.36, with a median of 71.5 and a range from 14.5 to 99. The overall difficulty level was moderate (P\u0026thinsp;=\u0026thinsp;0.68), while the overall discrimination performance was favorable (D\u0026thinsp;=\u0026thinsp;0.45). Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e details the difficulty and discrimination indices stratified by item type.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDifficulty and discrimination indices by item type\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItem type\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNumber of items\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTotal points\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMean score\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDifficulty (P)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eDiscrimination (D)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA1 (single best answer)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e28.81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.41\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA2 (clinical vignette)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.40\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eShort answer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e11.25\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.57\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEssay\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e14.38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.72\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.39\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCase analysis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e6.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.62\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWhole test\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e58\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e100\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e67.56\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.68\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0.45\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eDistribution of cognitive levels\u003c/h2\u003e \u003cp\u003eOf the 58 items, 28 (48.3%) were categorized as recall, 18 (31.0%) as interpretation, and 12 (20.7%) as problem-solving. Although the three cognitive levels contributed similar proportions of total points (34%, 33%, and 33%, respectively), recall items made up nearly half of all items, while problem-solving items account for only one-fifth.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003ePredictive validity of cognitive level coding for item discrimination\u003c/h2\u003e \u003cp\u003eSpearman correlation analysis revealed a positive association between cognitive level and item discrimination (r\u003csub\u003es\u003c/sub\u003e = 0.42, 95% CI [0.18, 0.61], \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.01). Simple linear regression (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) showed that cognitive level significantly predicted item discrimination (β\u0026thinsp;=\u0026thinsp;0.41, t\u0026thinsp;=\u0026thinsp;3.37, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.01), with the regression equation formulated as D\u0026thinsp;=\u0026thinsp;0.20\u0026thinsp;+\u0026thinsp;0.09 \u0026times; cognitive level. Cognitive level explained 16.8% of the variance in item discrimination (R\u0026sup2; = 0.168, adjusted R\u0026sup2; = 0.153). One-way ANOVA (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e) indicated a statistically significant difference in the mean D value across the three cognitive levels (F\u003csub\u003e2, 55\u003c/sub\u003e = 4.57, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.03, η\u0026sup2; = 0.14, representing a medium effect size). Post-hoc LSD tests demonstrated that problem-solving items had higher discrimination (0.49\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18) than recall items (0.34\u0026thinsp;\u0026plusmn;\u0026thinsp;0.22), with a mean difference of 0.15 (95% CI [0.02, 0.28], \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.03). Interpretation items (0.38\u0026thinsp;\u0026plusmn;\u0026thinsp;0.21) showed no significant differences from either recall items (\u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.48) or problem-solving items (\u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.12). No significant differences in difficulty were found across the three cognitive levels (F\u003csub\u003e2, 55\u003c/sub\u003e = 1.96, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.15), confirming that the observed differences in discrimination were not confounded by item difficulty.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDifficulty and discrimination of items by cognitive level\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCognitive level\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eN (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePoints (%)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDifficulty (P)(Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDiscrimination (D) (Mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eStudent score rate (%)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e28 (48.3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e34 (34)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.72\u0026thinsp;\u0026plusmn;\u0026thinsp;0.19\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.34\u0026thinsp;\u0026plusmn;\u0026thinsp;0.22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e72\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18 (31.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e33 (33)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.67\u0026thinsp;\u0026plusmn;\u0026thinsp;0.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.38\u0026thinsp;\u0026plusmn;\u0026thinsp;0.21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e67\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eProblem-solving\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e12 (20.7)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e33 (33)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.65\u0026thinsp;\u0026plusmn;\u0026thinsp;0.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.49\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF (P)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1.96 (0.15)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4.57 (0.03)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eη\u0026sup2; (effect size)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e\u0026ndash;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"6\"\u003eNote: SD\u0026thinsp;=\u0026thinsp;standard deviation. Effect size η\u0026sup2; = 0.14 indicates a medium effect.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eCognitive attribution of low-discrimination items\u003c/h2\u003e \u003cp\u003eFive items, accounting for 8.6% of the total, had discrimination indices below 0.10, contributing 5 points (representing 5% of the total score). Among these items, three (60%) was classified as recall-level, and one each fell into the interpretation and problem-solving categories. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e provides a detailed breakdown of their cognitive levels, discrimination indices, and defect classifications.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLow-discrimination items (D\u0026thinsp;\u0026lt;\u0026thinsp;0.10): cognitive level, difficulty, discrimination, and defect attribution\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItem No.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eType\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCognitive level\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDifficulty (P)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eDiscrimination (D)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eDefect attribution (based on option selection percentages)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCeiling effect: 94.3% answered correctly, leaving virtually no variance to discriminate.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eInterpretation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eStrong-distractor-induced empirical error: 60.8% of students selected B (\u0026ldquo;double the regular first dose\u0026rdquo;), misled by the common clinical heuristic \u0026ldquo;double first dose.\u0026rdquo; High- and low-performers chose B at similar rates (61.2% vs. 60.3%). The intended cognitive level (interpretation) was not realized; the item functioned as disguised recall.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eDysfunctional distractors: Distractors A, C and E attracted 10.8%, 11.9% and 14.4% of students respectively; the correct answer B was chosen by only 61.3% \u0026ndash; barely above chance.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.42\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eZero discrimination: High- and low-performers selected the four distractors in almost identical proportions. The item contributed no information about student ability.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProblem-solving\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.69\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0.08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eConcept-migration error: 26.3% of students erroneously selected B (\u0026ldquo;inhibition of TXA\u003csub\u003e2\u003c/sub\u003e synthesis\u0026rdquo;), incorrectly transferring the antiplatelet mechanism of aspirin to the context of acute gastric mucosal injury.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eIn-depth case analysis \u0026ndash; Item 2: When a \u0026ldquo;clinical heuristic\u0026rdquo; becomes a dysfunctional distractor\u003c/h2\u003e \u003cp\u003eItem 2 presented the following question: \u0026ldquo;For a drug exhibiting first-order elimination and conforming to a one-compartment model, which dosing regimen is optimal for rapidly achieving steady-state concentrations via a conventional oral schedule?\u0026rdquo; The correct answer(Option D) was \u0026ldquo;administering a loading dose, followed by maintenance doses.\u0026rdquo; Notably, 60.8% of students chose Option B (\u0026ldquo;doubling the initial standard dose\u0026rdquo;), a pattern observed consistently across both high- and low-performing cohorts. As a result, the item demonstrated negligible discrimination (D\u0026thinsp;=\u0026thinsp;0.02).\u003c/p\u003e \u003cp\u003eStudents had memorized the clinical heuristic of a \u0026ldquo;double first dose\u0026rdquo; but lacked an understanding of the pharmacokinetic rationale behind loading dose calculation. The item stem\u0026rsquo;s mention of a \u0026ldquo;conventional oral regimen,\u0026rdquo; paired with option B (\u0026ldquo;double the regular first dose\u0026rdquo;), created a strong semantic link that triggered retrieval of the memorized heuristic instead of first-principles reasoning. Although categorized at the interpretation cognitive level, the actual cognitive process evoked was merely the recall of a rote-memorized rule. This gap between the intended and actual cognitive engagement completely compromised the item's predictive validity. Therefore, the predictive validity of cognitive level coding is conditional, not absolute; it fundamentally depends on how accurately the item presentation elicits the intended cognitive process. Items designed to evaluate higher-order cognition may fail if distractors inadvertently encourage reliance on rote memorization.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study offers the first empirical evidence in pharmacology education, showing that cognitive level classifications (recall, interpretation, and problem-solving) serve as a significant predictor of item discrimination. Problem-solving items demonstrated substantially higher discriminatory efficacy compared with recall items, and cognitive level explained 16.8% of the variation in discrimination. Further examination of low-discrimination items revealed that predictive validity hinges on item-writing quality, highlighting the necessity of rigorous item-development protocols.\u003c/p\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003ePredictive validity of cognitive level coding: Evidence and boundary conditions\u003c/h2\u003e \u003cp\u003eThis study provides novel empirical insights for pharmacology education, showing that cognitive level coding (recall, interpretation, and problem-solving) acts as a robust predictor of item discrimination. The positive Spearman correlation (rₛ = 0.42), moderate regression effect size (R\u0026sup2; = 0.168), and the ANOVA comparison between recall and problem-solving items jointly confirm the predictive validity of this coding scheme. These results are consistent with the findings of Buljan et al. [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], who noted that items classified at the \u0026ldquo;apply\u0026rdquo; and \u0026ldquo;analyze\u0026rdquo; levels in an evidence-based medicine assessment had higher median discrimination than \u0026ldquo;remember\u0026rdquo;-level items, thereby supporting the view that increased cognitive demands strengthen an item's capacity to differentiate between students of varying proficiency. Likewise, Javaid et al. found that greater item complexity defined by higher cognitive demand and the integration of multiple concepts correlated with improved discrimination across various health sciences examinations. The effect size in this study (η\u0026sup2; = 0.14) is comparable to that reported in veterinary education [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e], where items requiring higher-order thinking also demonstrated superior discriminatory power. Collectively, these findings emphasize consistency with transformative pedagogical practices, underscoring the efficacy of cognitive frameworks in enhancing assessment validity.\u003c/p\u003e \u003cp\u003eNotably, predictive validity hinges critically on item construction quality. Item 2 serves as a boundary case: its intended cognitive demand (interpretation) was compromised, as superficial cues (e.g., \u0026ldquo;double the regular first dose\u0026rdquo;) triggered a low-level retrieval strategy. This phenomenon, termed \u0026ldquo;pseudo-interpretation\u0026rdquo; or \u0026ldquo;stem-matching cue,\u0026rdquo; has been identified as a substantial threat to the validity of multiple-choice items [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Despite a reasonable difficulty index (P\u0026thinsp;=\u0026thinsp;0.35), the item's negligible discrimination reveals that cognitive level coding alone is insufficient; items must be designed to compel examinees to engage at the targeted cognitive levels. Consequently, predictive validity relies on both the coding framework and the rigor of item design. This perspective directly responds to the critique from Ray et al. [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] that \u0026ldquo;mapping alone may not improve outcomes.\u0026rdquo; Their scoping review of 24 studies on the association between Bloom's taxonomy mapping and educational outcomes in health professions education found that, despite widespread adoption, empirical support for aligning examination questions with learning taxonomies remains limited, highlighting the need for rigorous item construction to ensure that intended cognitive levels are authentically enacted.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eStructural Imbalance: Scarce but superior problem‑solving items\u003c/h2\u003e \u003cp\u003eProblem-solving items constituted merely one-fifth of the examination but contributed one-third of the total points. This \u0026ldquo;high-weight, low-count\u0026rdquo; pattern is common in medical education assessments [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Although each problem-solving item produced considerable score variance, their limited number restricted the test\u0026rsquo;s overall discriminatory power and magnified the influence of individual defective items (such as item 47). Hence, raising the proportion of problem-solving items is advisable to stabilize the total score distribution. For example, shifting from the existing 48:31:21 allocation (recall:interpretation:problem-solving) to a more balanced 40:30:30 or even 30:40:30 distribution would leverage the enhanced discriminatory capability of higher-level items while preventing excessive weighting of individual items.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003eCognitive attribution of low‑discrimination items: A diagnostic tool for item improvement\u003c/h2\u003e \u003cp\u003eOur cognitive attribution analysis reveals that low-discrimination items are systematic rather than random; they cluster at the recall level and display identifiable, correctable flaws. Ceiling effects, with item 1 as a typical example, can be mitigated by upgrading items to the interpretation level\u0026mdash;for instance, by incorporating simple clinical scenarios. Dysfunctional distractors, such as those in item 3, can be eliminated or substituted based on empirical distractor analysis [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Most significantly, empirical errors induced by potent distractors (as observed in item 2) present unique opportunity, as they accurately pinpoint where students\u0026rsquo; mental models deviate from expert comprehension [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Instead of discarding these potent distractors, item stems should be refined to remove superficial cues while preserving the diagnostic value of the distractors. For example, item 2 can be improved by modifying the stem to specify \u0026ldquo;based exclusively on pharmacokinetic principles\u0026rdquo; and restructuring option B as \u0026ldquo;administer an initial dose that exceeds the standard maintenance dose,\u0026rdquo; thus eliminating direct dependence on the \u0026ldquo;double dose\u0026rdquo; heuristic.\u003c/p\u003e \u003cp\u003eItem 47, the sole problem-solving item exhibiting low discrimination, revealed a concept-transfer error: students erroneously extended the antiplatelet mechanism (driven by thromboxane A₂ reduction) to the mechanism underlying gastric mucosal injury (driven by prostaglandin E₂ reduction). This constitutes a classic, deeply entrenched misconception that demands targeted instructional intervention. Consequently, pharmacology educators should explicitly juxtapose these two mechanisms when teaching nonsteroidal anti-inflammatory drugs.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eImplications for pharmacology education\u003c/h2\u003e \u003cp\u003eAlthough grounded in anesthetic pharmacology, a specialized disciplinary application, the cognitive level coding framework and its proven predictive validity for item discrimination demonstrate broad applicability across disciplinary boundaries. Pharmacology curricula in pharmacy and health professions programs face similar assessment challenges, particularly in integrating factual mastery with clinical reasoning. Thus, our findings offer three actionable, transferable recommendations for pharmacology educators aiming to strengthen assessment rigor.\u003c/p\u003e \u003cp\u003eFirst, educators should integrate a cognitive level blueprint into the pre-examination item vetting process. Predetermined ratios for recall, interpretation, and problem-solving items (for example, 40:30:30) should be set prior to exam development. Item writers must specify the intended cognitive level of each question and ensure that correct answers cannot be obtained solely through lower order strategies.\u003c/p\u003e \u003cp\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eSecond, ongoing distractor refinement, guided by real student response data, is essential. After examination administration, distractor analysis must be performed on items whose discrimination indices are below 0.20. When a distractor triggers empirical errors in both high- and low-achieving examinees, the stem should be modified to eliminate superficial cues while preserving the distractor\u0026rsquo;s diagnostic utility. Institutions are urged to build local \u0026ldquo;pharmacology misconception repository\u0026rdquo; to support future item development and instructional planning.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eThird, educators should raise the proportion of problem-solving items to capitalize on their superior discriminatory capability. Effective problem-solving items generally entail synthesizing at least two pharmacological principles, integrate clinical decision points, and include distractors that mirror genuine clinical reasoning errors.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eIntegrating these recommendations, we put forward a three-step quality assurance protocol for item development: (1) pre-testing to validate cognitive alignment, ensuring items are targeted at the intended cognitive levels; (2) empirical distractor assessment grounded in student response data; and (3) sustaining a balanced distribution of items across cognitive levels. These guidelines are in line with current competency-based assessment requirements in medical education [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] and can be readily implementable within core pharmacology courses.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003eLimitations and future research\u003c/h2\u003e \u003cp\u003eThis study involves several methodological limitations that call for cautious interpretation. First, the analysis was restricted to a single institution and a single examination, which constrains generalizability; thus, future research should cover multiple institutions, diverse pharmacology curricula, and varied assessment approaches to confirm broader applicability. Second, although cognitive level coding showed high inter-rater reliability (Cohen's κ\u0026thinsp;=\u0026thinsp;0.86), it is still open to subjective judgment. Assigning equal numerical intervals (1, 2, 3) in regression analysis, a common practice in educational measurement, presupposes equidistance among cognitive levels, and this presumption may not accurately mirror the underlying construct. Subsequent research could tackle this limitation by applying ordinal logistic regression or item response theory (IRT) models, treating cognitive level as a categorical predictor without enforcing linear constraints. Third, the study examined only item discrimination as an indicator of predictive validity; the predictive value of cognitive level coding for other critical educational outcomes, such as clinical competence, long-term knowledge retention, and performance in licensure examinations, remains uncharted.\u003c/p\u003e \u003cp\u003eAddressing these limitations outlines several crucial directions for future research. First, prospective intervention studies are needed to assess examination quality before and after the adoption of cognitive level blueprints, generating causal evidence of their effectiveness. Second, multi-center studies across diverse pharmacology courses and institutions are vital to confirm the generalizability of the predictive validity observed in this single-center study. Third, analyses based on IRT can clarify whether cognitive level influences item information functions, offering detailed psychometric insights into item performance across varying ability levels. Fourth, qualitative think-aloud studies are essential to verify the actual cognitive processes students use when answering items classified at different cognitive levels, thereby validating the alignment between intended cognitive demands and students' response strategies. Together, advancing these research paths will reinforce the evidence foundation for using cognitive level coding to enhance assessment rigor in pharmacology education.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusions","content":"\u003cp\u003eThis study offers the first empirical evidence in pharmacology education that cognitive level coding significantly predicts item discrimination. Problem-solving items exhibit greater discriminatory power than recall items, verifying that higher cognitive demand elevates assessment quality. Nevertheless, this predictive validity is contingent on item writing quality\u0026mdash;superficial cues can compromise even properly classified items. These findings advocate for incorporating cognitive blueprints into examination design and leveraging empirical data to refine distractors, thereby propelling competency-based assessment forward in pharmacy education.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis study complied with the ethical standards set forth in the Declaration of Helsinki (2013 revision). The research utilized fully anonymized routine educational assessment data (de-identified student examination scores and item-level response data) with no direct interaction with participants, no collection of personal identifiable information, and no intervention in students\u0026rsquo; learning or assessment processes. In accordance with the Research Ethics Review Guidelines for Human Biological Samples and Data (2023) issued by the National Health Commission of the People\u0026rsquo;s Republic of China and the ethical review regulations of Chongqing Medical University\u0026rsquo;s Institutional Review Board (IRB), this type of retrospective educational research based on anonymized routine data is exempt from formal ethics committee approval. The Chongqing Medical University IRB has confirmed the waiver of ethical review for this study (no formal IRB approval number is required for exempted research per institutional rules). Informed consent from participants was also unnecessary due to the anonymized nature of the data and the non-interventional study design.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and analyzed during the current study are based on routine student data collection related to university programming and are not publicly available.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFundings\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study was supported by Chongqing Association of Higher Education 2025-2026 Annual Higher Education Scientific Research Project (No. cqgj25031C), 2025 Medical Education Research Planning Project of Chongqing Medical University (No. J0625017), and 2025 Chongqing Higher Education Teaching Reform Research Project of Chongqing Medical University (No. 253075).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eStudy design: Qingsong Jiang, Ke Wu and Hongmei Qiu. Data analysis: Xiaoli Li. Data collection: Jingyuan Wan, Weiying Zhou, Limei Ma and Qingzhu Ren. Writing of the manuscript: Qingsong Jiang, Ke Wu and Hongmei Qiu. All authors read and approved the final version of the manuscript and agreed to its submission.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors would like to thank all the students who agreed to complete the final examination.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eYu T, Wang GL. Anesthetic Pharmacology. 4th ed. Beijing: People's Medical Publishing House; 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDowning SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSim SM, Rasiah RI. Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singap. 2006;35(2):67\u0026ndash;71.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAdams NE. Bloom's taxonomy of cognitive learning objectives. J Med Libr Assoc. 2015;103(3):152\u0026ndash;3.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTai H, Kovarik C. ChatGPT-4's level of dermatological knowledge based on board examination review questions and Bloom's Taxonomy. JMIR Dermatol. 2025;8:e74085.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTran T, Le U, Phan V. Evaluating the accuracy and educational potential of generative AI models in pharmacy education: a comparative analysis of ChatGPT and Gemini across Bloom's Taxonomy. Pharm (Basel). 2025;14(1):1.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCohen Aubart F, Lhote R, Hertig A, Noel N, Costedoat-Chalumeau N, Cariou A, Meyer G. Progressive clinical case-based multiple-choice questions: an innovative way to evaluate and rank undergraduate medical students. Rev Med Interne. 2021;42(5):302\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmerican Educational Research Association, American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRay ME, Rudolph MJ, Daugherty KK. Bloom's taxonomy in health professions education: associations with exam scores, clinical reasoning, and instructional effectiveness: a scoping review. Curr Pharm Teach Learn. 2025;17(11):102444.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu Q, Wald N, Daskon C, Harland T. Multiple-choice questions (MCQs) for higher-order cognition: perspectives of university teachers. Innov Educ Teach Int. 2024;61(4):802\u0026ndash;14.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRush BR, Rankin DC, White BJ. The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Med Educ. 2016;16(1):250.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBuljan I, Marušić M, Tokalić R, Viđak M, Peričić TP, Hren D, Marušićet A. Cognitive levels in testing knowledge in evidence-based medicine: a cross sectional study. BMC Med Educ. 2021;21(1):25.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHillsley K. Question format is the best predictor of item discrimination: a multivariable analysis. J Microbiol Biol Educ. 2025;26(3):e0020525.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang BF, Chen JG. Pharmacology. 10th ed. Beijing: People's Medical Publishing House; 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnderson LW, Krathwohl DR. A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. New York: Longman; 2001.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCrowe A, Dirks C, Wenderoth MP. Biology in bloom: implementing Bloom's taxonomy to enhance student learning in biology. CBE Life Sci Educ. 2008;7(4):368\u0026ndash;81.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Z, Zheng XL, He X, Liu J. Optimizing pathophysiology instruction in dental school: examination paper analysis and strategic reflections. BMC Med Educ. 2025;25(1):754.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCostello E, Holland JC, Kirwan C. Evaluation of MCQs from MOOCs for common item writing flaws. BMC Res Notes. 2018;11(1):849.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMonrad SU, Bibler Zaidi NL, Grob KL, Kurtz JB, Tai AW, Hortsch M, et al. What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy. Med Teach. 2021;43(5):575\u0026ndash;82.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTariq S, Tariq S, Maqsood S, Jawed S, Baiget M. Evaluation of cognitive levels and item writing flaws in medical pharmacology internal assessment examinations. Pak J Med Sci. 2017;33(4):866\u0026ndash;70.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDavies DJ, McLean PF, Kemp PR, Liddle AD, Morrell MJ, Halse O, et al. Assessment of factual recall and higher-order cognitive domains in an open-book medical school examination. Adv Health Sci Educ Theory Pract. 2022;27(1):147\u0026ndash;65.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNational Research Council. How people learn: brain, mind, experience, and school. Expanded ed. Washington, DC: National Academies; 2000.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEnnab F, Abdulkareem RH, Kaddoura R, Hegazy A, Lootah M, Banerjee Y, et al. Exploring undergraduate medical students' perception of an integrated longitudinal research curriculum within a competency-based framework. PLoS ONE. 2026;21(2):e0343409.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Pharmacology education, Educational measurement, Predictive validity, Discrimination, Bloom’s taxonomy","lastPublishedDoi":"10.21203/rs.3.rs-8963083/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8963083/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eBloom's taxonomy is widely used to classify examination items by cognitive level, yet its predictive validity for item discrimination remains understudied in pharmacology education. This study investigated whether cognitive level coding predicts item discrimination in an anesthetic pharmacology examination.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThis retrospective document analysis examined 58 items from a final anesthetic pharmacology examination completed by 194 undergraduate students. Two independent raters coded items using a modified three-level Bloom's taxonomy (recall, interpretation, problem-solving). Item difficulty (P) and discrimination (D) indices were extracted from the official test analysis report. Predictive validity was assessed using Spearman correlation, linear regression, and one‑way ANOVA. Items with D\u0026thinsp;\u0026lt;\u0026thinsp;0.10 underwent distractor analysis and cognitive attribution.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eThe examination demonstrated excellent reliability (Cronbach's α\u0026thinsp;=\u0026thinsp;0.91), moderate difficulty (P\u0026thinsp;=\u0026thinsp;0.68), and good overall discrimination (D\u0026thinsp;=\u0026thinsp;0.45). Cognitive level was positively correlated with item discrimination (rₛ = 0.42, 95% CI [0.18, 0.61], \u003cem\u003eP\u003c/em\u003e\u0026thinsp;\u0026lt;\u0026thinsp;0.01), explaining 16.8% of the variance (R\u0026sup2; = 0.168). Problem‑solving items exhibited higher discrimination (0.49\u0026thinsp;\u0026plusmn;\u0026thinsp;0.18) than recall items (0.34\u0026thinsp;\u0026plusmn;\u0026thinsp;0.22) (η\u0026sup2; = 0.14, \u003cem\u003eP\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.03). Five items (8.6%) showed low discrimination (D\u0026thinsp;\u0026lt;\u0026thinsp;0.10), predominantly at the recall level, and were compromised by ceiling effects, dysfunctional distractors, or strong‑distractor‑induced empirical error.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eCognitive level coding significantly predicts item discrimination in anesthetic pharmacology assessments, with problem‑solving items demonstrating superior discriminatory power. However, predictive validity depends on item‑writing quality. Integrating cognitive blueprints into examination design and refining distractors using empirical data are recommended to enhance assessment quality in pharmacology education.\u003c/p\u003e\u003ch2\u003eClinical trial number\u003c/h2\u003e \u003cp\u003eNot applicable.\u003c/p\u003e","manuscriptTitle":"Cognitive Level Coding as a Predictor of Item Discrimination: An Empirical Study in Anesthetic Pharmacology Education","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-10 04:48:24","doi":"10.21203/rs.3.rs-8963083/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2026-04-21T17:25:03+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-21T14:53:49+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-14T20:23:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"123860393180091555350566711627809536223","date":"2026-04-12T12:17:12+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"330233569422462426924589994067676862172","date":"2026-04-10T17:14:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"29310724676891944626533999635985364032","date":"2026-04-07T14:59:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"17692576162159711464286752820930653203","date":"2026-04-03T15:47:29+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-03T11:31:05+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-31T07:18:48+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-10T10:44:19+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-08T08:24:28+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Education","date":"2026-03-08T08:19:40+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"402f712b-2351-4fd8-9c3d-1b97a81e4940","owner":[],"postedDate":"April 10th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-10T04:48:25+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-10 04:48:24","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8963083","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8963083","identity":"rs-8963083","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}