Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups

Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups Ting Khee HO, Lucy O’MALLEY, Reza Vahid ROUDSARI This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7347939/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Introduction Criterion-referenced standard setting is commonly employed to establish passing marks in healthcare professional examinations, its application in Malaysian dental schools is limited. This study aimed to examine the judges’ feedback and thought processes when utilising the modified Angoff and Ebel methods in the final-year multidisciplinary exam, involving two panels of judges at a dental faculty in Malaysia. Methods A convergent mixed-methods approach was used, combining quantitative feedback questionnaires and qualitative focus group discussions (FGDs) over a three-day standard setting workshop at The National University of Malaysia. One-best answer items and short answer questions were standardised using the modified Angoff and Ebel methods. The questionnaires evaluated the judges’ understanding, confidence and preference for the defensibility of the standard setting methods. The FGD explored the judges’ experiences and thought processes, analysed through inductive thematic analysis. Results Twelve judges from various specialties participated. The feedback questionnaires showed no significant differences between the modified Angoff and Ebel methods or between Groups A and B. The judges reported enhanced understanding and confidence levels from Rounds 1 to 3. Judges’ preferences for each method varied. The themes included the following: confidence in standard setting, continuous vs. category scoring, the role of diverse expertise in decision-making, alignment with course learning outcomes, barriers to effective standard setting and the impact of training as a catalyst for effective standard setting. Conclusion Groups A and B shared similar feedback, experiences, and perspectives regarding the standard setting activities. The modified Angoff method was straightforward and easy; it gave judges more control, but it provided a subjective framework, relying on intuition and experience in setting the Angoff ratings. In contrast, the Ebel method offered a structured, objective framework, but judges felt they had less control, found the process more time-consuming, and some referred to the probability that borderline candidates would answer correctly within each category in Ebel grid to determine the difficulty level. These findings provide insights into standard setting in dental education and emphasis the importance of structured training to enhance procedural validity. Dental Education Education Measurement Standard Setting Angoff Ebel Passing Score Passing Mark Malaysia Mixed-Methods Figures Figure 1 Figure 2 Figure 3 Background Standard setting has gained significant importance in dental education assessments in recent years as it plays a crucial role in determining the cut score representing the minimum acceptable academic performance in examinations. The purpose of this process in high-stakes examination is to determine the passing scores that correspond with competence levels, ensuring graduating healthcare professionals achieve the minimal competency level to provide safe and effective patient care [ 1 , 2 ] while preventing unfair failure of competent candidates, which could lead to demotivation, anxiety, a reduced workforce and increased financial and time burdens for both the candidates and institutions conducting remedial programmes [ 3 ]. The validity criteria of the standard setting process are essential for fair and defensible outcomes. While much research focuses on refining standard setting procedures to achieve more reliable results, reliability alone is insufficient to ensure the appropriateness of performance standards. Validity evidence in standard setting should encompass procedural, internal, external, and consequential criteria [ 4 – 8 ]. Procedural evidence is recognised as one of the main considerations when evaluating the validity of passing scores as a measure of performance standards. According to Kane’s validity framework, procedural criteria validity includes the selection of appropriate standard setting methods and ensuring their effective implementation, encompassing judge selection and training, defining performance standards, data collection, and obtaining judge feedback [ 4 , 5 ]. To support the process validity in making pass-fail decisions, quantitative feedback is routinely collected from judges as part of the standard setting procedure [ 7 , 9 , 10 ]. Qualitative studies in other non-dental disciplines have examined judges’ feedback and thought processes, offering procedural validity evidence and insights into the mental processes involved in standard setting practices [ 11 – 15 ]. Giraud and Impara (2005) are among the first researchers to conduct qualitative research in the standard setting process for an English test using the modified Angoff method in a public school district. Judges struggled to align district expectations with the established standards, expressing concerns that the cut-scores were manipulated for political and social reasons [ 16 ]. Similarly, a qualitative study of high school Angoff standard setting meetings found that judges experienced political and social pressures, especially in subjects with high failure rates, and felt pressure to conform their decisions to others [ 17 ]. They also struggle to conceptualise borderline students and distinguish prediction from value judgement. McGinty (2005) suggested that understanding these influences could improve future training programmes, ultimately enhancing the validity of standard setting. Both McGinty (2005) and Giraud and Impara (2005) show potential researcher bias due to unsystematic data collection and subjective analysis [ 16 , 17 ]. McGinty relied solely on observational notes without a structured coding framework for data analysis [ 17 ] while Giraud and Impara lacked a clear description of the framework used to analyse data from observations, interviews and discussions [ 16 ]. Building on these weaknesses, other researchers adopted more structured qualitative methods using written questionnaires and open-ended questions to explore the judges’ thought processes during the standard setting process [ 11 , 12 ]. Skorupski and Hambleton’s (2005) applied item mapping standard setting method was used to set cut-scores for the Grade 5 and 6 English Test, with judges responding to Likert scale and open-ended questions at various stages during the standard setting process. The study found varied interpretations of the performance level descriptors and differing perspectives on standard setting procedures throughout the process. Judges felt time pressure in the final round, potentially leading to inconsistencies. Additionally, inconsistencies emerged between the questionnaire responses and the written comments regarding confidence in the standards. While most claimed confidence in their ratings, many still mentioned concerns in their ratings, suggesting some underlying uncertainty [ 11 ]. The triangulation of data in this research provides more credible insights into the judges’ thought processes. Meanwhile, Ferdous and Buckendahl’s (2013) utilising a similar approach as the study by Skorupski and Hambleton (2005), exploring factors influencing judges’ decision and their confidence levels using the Yes/No Angoff standard setting method in Grade 5 English and Mathematics examinations [ 12 ]. In contrast to previous research, the judges expressed high confidence in their understanding of the performance level descriptors and the standard setting process after training. They primarily relied on performance level descriptors rather than candidates' performance data or judges feedback data [ 12 ]. This criterion-referenced approach ensures that the standards are set according to desired performance expectations for each cut-score rather than relative ranking. Hein and Skaggs (2009) adopted focus group discussion (FGD) rather than traditional observations, questionnaires or interviews to collect participants’ experiences with two variations of the Bookmark method [ 18 ]. They audio-recorded and transcribed the FGD, then coded it to create categories in the result. Results showed that judges found selecting an item for a bookmark is cognitive challenge and often relied on their professional experiences and beliefs during judgement, deviating from the prescribed bookmark procedure. Numerous studies on standard setting in dental assessments have focused on judges’ consistency, the replicability of the results across different methods, and the correlations between ratings and student performance data [ 19 – 23 ]. In Malaysia, where criterion-referenced standard setting is still emerging, Ho, O'Malley et al. (2025) conducted a workshop to explore the reproducibility of modified Angoff and Ebel methods across two judge panels at a Malaysian public dental institute [ 24 ]. This study builds on previous research that aimed to investigate the experiences and thought processes of newly trained judges applying the modified Angoff and Ebel standards. It is the first study to use a mixed-methods approach to thoroughly examine judges’ feedback, experiences and perspectives on the application of these standard setting methods among dental academics. The quantitative component compared feedback from two independent judge panels and examined differences between the modified Angoff and Ebel methods, providing evidence of procedural validity. The qualitative component explored judges’ experiences, thought processes, and challenges to identify sources of variability and factors influencing scoring confidence in applying the modified Angoff and Ebel methods to knowledge test items. Findings aim to inform improvements in standard setting training thus strengthening the validity of the standard setting outcomes. These data provide evidence of procedural validity in the standard setting process. The qualitative component examined judges’ experiences and thought processes to understand their beliefs and attitudes when making standard setting judgements. The analysis aimed to identify potential sources of variability, factors influencing confidence in scoring and challenges encountered in applying the modified Angoff and Ebel methods to knowledge test items. The findings are intended to provide valuable insights for enhancing future standard setting training, thus strengthening the validity of the standard setting outcomes. Methods Study design and setting This research utilised a convergent mixed-methods research design, collecting quantitative data from feedback questionnaires and qualitative data from open-ended surveys and FGDs [25, 26]. This study was conducted alongside another study [24] that compared the reproducibility of the passing scores of modified Angoff and Ebel methods across judges, conducted in the Faculty of Dentistry, The National University of Malaysia (Universiti Kebangsaan Malaysia, UKM), a public university, in March 2024. The theory paper, consisting of one-best answer (OBA) and short-answer questions (SAQs) format for the final-year Final Professional Examination in the Comprehensive Dental Care (CDC) course conducted in 2022, was selected in this standard setting study. The CDC course is multidisciplinary, covering endodontics, operative, oral medicine and oral pathology, oral and maxillofacial surgery, public dental health, prosthodontics and periodontics. Study participants and sampling The participants, also called judges in the standard setting meetings, were selected based on the inclusion and exclusion criteria outlined in the previous study (Detailed in Ho et al., 2005 [24]). Purposive sampling method used in this study aimed to achieve maximum variation among the participants. Twelve judges participated in the standard setting meetings and were purposively sampled into two groups of six (Groups A and B), ensuring a matched gender, ethnic, years of experience, dental discipline, and academic position. Before the start of the study, the objectives and details were explained and informed consent was obtained from all participants. Procedure This study was conducted during a three-day workshop, where, on the first day, judges received lectures on the theory of standard setting. They participated in practical exercises with example items to apply the modified Angoff [27-29] and Ebel [30, 31] methods. These exercises aimed to familiarise the judges with the standard setting method and processes. The workshop was led by the first author, TKH, a dental academic pursuing a PhD in dental education; NLAK, a language educationist and psychometrician; and MNAB, a medical educationist. NLAK and MNAB both possess extensive experience in standard setting protocols and meetings relevant to their respective fields. Following the lectures and practice sessions, judges conducted standard setting meetings using the modified Angoff and Ebel methods in separate discussion rooms. For each method, judges completed two rounds of individual ratings (R1, R2), with feedback on ratings (mean, median, SD and overall passing score) reviewed before R2. A third round (R3) involved reviewing R2 ratings and participating in group discussion before finalising ratings. Judges also completed online feedback questionnaires at various stages and the workshop concluded with a focus group discussion (FGD). Figure 1 illustrates the study flow. (Insert Figure 1) Quantitative questionnaire study The feedback questionnaires were adapted and modified from previous standard setting literature and studies designed to assess process validity and the thought processes of judges during meetings [7, 12, 32, 33]. The self-reported questionnaires comprised four parts aligned with the study’s objectives and were administered at different stages of the standard setting meeting. The questionnaires were created using Qualtrics XM (Provo, UT) and the online link was distributed to the judges during the meeting. They included a five-point Likert scale (strongly agree = 5, agree = 4, neutral = 3, disagree = 2 and strongly disagree = 1), multiple-choice questions, and open-ended questions. A summary of the feedback questionnaire content is provided in Supplementary File 1 . The study comprised four parts: Part 1 collected demographic data and feedback on standard setting lectures. Parts 2 and 3 gathered identical feedback on the modified Angoff and Ebel methods after each rating round, using pseudonyms. On the final day, groups reviewed passing scores across methods. Part 4 collected final insights on the workshop and judges’ experiences with the methods used before focus group discussions. Face and content validation of the questionnaires and FGD questions was conducted by two healthcare educationists and a psychometrician (NLAK) with expertise in conducting standard setting meetings and questionnaire-related education research. They provided suggestions to enhance clarity and ensure alignment with the study objectives. A condensed workshop was pilot tested with dental academics at another institution, and their feedback further improved the questionnaires' readability and clarity. Qualitative interview study All judges were invited to participate in face-to-face FGDs to share their insights and experiences from the standard setting meetings. Since the standard setting procedure was conducted in a group setting, collecting data through FGDs allowed for interactive discussions, enabling judges to elaborate on their perspectives. This method also facilitated the collection of diverse viewpoints within a shorter timeframe. A semi-structured interview guide (Supplementary File 2) was developed by authors in accordance with the guidelines for conducting [34]. It was then reviewed by the same experts who validated the feedback questionnaire. Minor amendments were made based on their comments to ensure that the questions aligned with the study objectives. The FGDs, conducted in English, were led by the first author (TKH), a PhD student trained in dental education and quantitative and qualitative research design, and a medical educationist (MNAB) with extensive educational qualitative research experience. The discussions were held in closed rooms with semicircle seating arrangements and lasted approximately 60 minutes. The semi-structured interview guide reviewed by the moderators beforehand to ensure consistency and standardisation across FGDs. Judges were assigned pseudonyms for confidentiality. Each FGDs began with a PowerPoint presentation summarising feedback on passing scores and responses from Questionnaire Part 4c. The discussions explored judges’ confidence, thought processes during rating, and challenges encountered during decision-making at the standard setting meeting. Sessions were audio-recorded with encrypted devices, transcribed verbatim by a university-approved service, and verified by the moderators. Data analysis Feedback from Parts 1 and 4 was analysed descriptively using frequencies and percentages for categorical variables, with bar charts generated in Microsoft Excel to visualise the data. The feedback questionnaires, which utilised a 5-point Likert scale (from 1 = strongly disagree to 5 = strongly agree), had their mean and standard deviation (SD) calculated. The non-parametric Mann–Whitney U test was used to investigate the differences across the groups, whereas the Wilcoxon signed ranks test was employed to evaluate the differences across the standard setting methods using IBM SPSS (version 29; IBM Corp., Armonk, NY, USA). Non-parametric statistical tests were selected due to the small sample size and violation of the normality assumption. A p-value of <0.05 was considered statistically significant. Additionally, free-text comments from the feedback questionnaires were analysed descriptively. The authors organised, managed and analysed the FGD data using a shared Microsoft Excel spreadsheet [35]. For the qualitative analysis, they adhered to the six-phase framework of inductive thematic analysis, as outlined by Braun and Clarke and DeJonckheere et al. [36, 37]. The first author began by manually removing irrelevant data and focusing on the research objectives while becoming familiar with the data. The identified quotes and codes were transferred to Microsoft Excel spreadsheets for further analysis. The codes were grouped into categories based on the shared meaning or content. To ensure reliability and accuracy, the second and third authors independently reviewed the transcription and coding. Any discrepancies in coding were addressed through iterative discussions, leading to the development of themes through consensus. Throughout the analysis, the researchers continuously reflected on potential personal or external biases and assumptions to minimise their influence, thereby ensuring the rigour and trustworthiness of the qualitative analysis. Ethical considerations Ethics approval for this project was granted by the University of Manchester Research Ethics Committee (2023-17408-31608) and UKM Research Ethics Committee (JEP-2023-2040). Participants were informed about the project three months in advance, allowing time to arrange prior commitments. Written informed consent was obtained, and pseudonyms were used to ensure confidentiality. Participation was voluntary, with assurances that feedback would not affect their professional evaluation or future career prospects. Results Demographic data A total of 12 participants were recruited as judges for the standard setting study and were evenly divided into two groups. The demographic characteristics of the participants (e.g., gender, ethnicity, years of teaching experience, discipline, and academic position) were identical to those reported in our earlier study [ 24 ]. Questionnaire data All 12 judges participated in and completed the feedback questionnaires. Figure 2 presents the general feedback evaluation of the standard setting workshop. Overall, judges expressed positive feedback about the lectures and the workshop. The overall mean (SD) scores were 4.24 (0.54) for Questionnaire Part 1 (feedback after lectures) and 4.47 (0.36) for Part 4a (feedback after meetings). Item-level means (SD) for Groups A and B are provided in Supplementary File 3 . Mann–Whitney U-test revealed no significant difference between Groups A and B for any item (p > 0.05). The high values of Cronbach’s alpha (after lectures, alpha = 0.96; after meetings, alpha = 0.75) indicated strong reliability and internal consistency of the questionnaire items [ 38 ]. (Insert Fig. 2 ) The mean (SD) ratings of judges’ responses to the standard setting process using the modified Angoff (Part 2) and Ebel (Part 3) methods after each round of ratings are documented in Supplementary File 4 . Wilcoxon signed-rank tests comparing feedback evaluations for each item from R1–R3 between the two methods revealed no significant differences. Similarly, the Mann–Whitney U-test demonstrated no differences in judges’ responses across Group A and B for all items from R1–R3 using either method. For the statement ‘I understood how to complete my rating’, the mean ratings increased from R1 to R3 for both modified Angoff (R1 = 4.25, R2 = 4.50 and R3 = 4.67) and Ebel (R1 = 4.33, R2 = 4.42 and R3 = 4.50) methods. Additionally, judges reported progressively greater confidence from R1 to R3. For the modified Angoff method, the mean confidence ratings increased to 3.58, 4.42 and 4.67 in R1, R2 and R3, respectively, while for the Ebel method, they increased to 3.58, 3.93 and 4.42 in R1, R2, and R3, respectively. The results from Questionnaire 4b on judges’ reliance on information sources during standard setting are presented in Table 1 . Throughout the standard setting process, almost all judges relied moderately to heavily on the descriptions of the borderline groups (91.7%) and small group discussion (100%) for their judgement. All judges in Group A considered normative ratings from other judges when making their decisions. However, it was observed that two judges from Group B minimally relied on the normative rating from others in their evaluations. Table 1 Judges' responses regarding the reliance on the source of information during standard setting process (Part 4b) Group N Response, n (%) Not at all Slightly Moderately Heavily 1. My previous experience taking the test A 6 2 (33.3) 0 3 (50.0) 1 (16.7) B 6 1 (16.7) 2 (33.3) 2 (33.3) 1 (16.7) Total 12 3 (25.0) 2 (16.7) 5 (41.7) 2 (16.7) 4. My own experience with student cohort A 6 0 0 4 (66.7) 2 (33.3) B 6 1 (16.7) 1 (16.7) 2 (33.3) 2 (33.3) Total 12 1 (8.3) 1 (8.3) 6 (50.0) 4 (33.3) 7. The descriptions of the borderline groups A 6 0 1 (16.7) 1 (16.7) 4 (66.7) B 6 0 0 4 (66.7) 2 (33.3) Total 12 0 1 (8.3) 5 (41.7) 6 (50.0) 10. Small group discussion A 6 0 0 3 (50.0) 3 (50.0) B 6 0 0 3 (50.0) 3 (50.0) Total 12 0 0 6 (50.0) 6 (50.0) 13. The normative information from other judges (eg: the ratings of other judges) A 6 0 0 4 (66.7) 2 (33.3) B 6 0 2 (33.3) 3 (50.0) 1 (16.7) Total 12 0 2 (16.7) 7 (58.3) 3 (25.0) (Insert Table 1 ) Table 2 presents the judges’ feedback comparing the modified Angoff and Ebel methods in terms of defensibility, confidence to practise, group rating and cognitive demands (Part 4c). The judges also expressed comparable confidence in the defensibility of the passing score for both methods in the OBA and SAQ papers, with mean scores ranging from 3.67 to 3.92. In the OBA paper, the mean scores in confidence in the practice of modified Angoff and Ebel methods were 4.00 and 3.92, respectively. Conversely, in the SAQ paper, the mean scores for the modified Angoff and Ebel methods were 3.83 and 3.92, respectively. Regarding preferences in the OBA paper, seven judges (58.3%) preferred the Ebel method, four judges (33.3%) favoured the modified Angoff method and one judge had no preference (Fig. 3 ). Conversely, preferences for the SAQ paper were evenly split, with 50% of the judges favouring the modified Angoff method and 50% favouring the Ebel method. Overall, the Mann–Whitney U-test demonstrated no differences in judges’ responses across Group A and B for all items in Part 4c. Table 2 Judges’ responses on items comparing the modified Angoff and Ebel standard setting decision (Part 4c) Statement Group n Response, n (%) Mean Stand. Deviation p-value a SD D N A SA 1. I am confident to practise the Modified Angoff method in OBA. A 6 0 0 1 (16.7) 4 (66.7) 1 (16.7) 4.00 .632 B 6 0 0 1 (16.7) 4 (66.7) 1 (16.7) 4.00 .632 Total 12 0 0 2 (16.7) 8 (66.7) 2 (16.7) 4.00 .603 1.000 4. I am confident to practise the Modified Angoff method in SAQ. A 6 0 0 2 (33.3) 2 (33.3) 2 (33.3) 4.00 .894 B 6 0 0 2 (33.3) 4 (66.7) 0 3.67 .516 Total 12 0 0 4 (33.3) 6 (50.0) 2 (16.7) 3.83 .718 .484 7. I am confident to practise the Ebel method in OBA. A 6 0 0 1 (16.7) 3 (50.0) 2 (33.3) 4.17 .753 B 6 0 0 2 (33.3) 4 (66.7) 0 3.67 .516 Total 12 0 0 3 (25.0) 7 (58.3) 2 (16.7) 3.92 .669 .206 10. I am confident to practise the Ebel method in SAQ. A 6 0 0 1 (16.7) 5 (83.3) 0 3.83 .408 B 6 0 0 1 (16.7) 4 (66.7) 1 (16.7) 4.00 .632 Total 12 0 0 2 (16.7) 9 (75.0) 1 (8.3) 3.92 .515 .598 13. I am convinced of the defensibility of the passing score using Modified Angoff method in OBA. A 6 0 0 0 6 (100.0) 0 4.00 .000 B 6 0 0 2 (33.3) 3 (50.0) 1 (16.7) 3.83 .753 Total 12 0 0 2 (16.7) 9 (75.0) 1 (8.3) 3.92 .515 .527 16. I am convinced of the defensibility of the passing score using Modified Angoff method in SAQ. A 6 0 1 (16.7) 0 4 (66.7) 1 (16.7) 3.83 .983 B 6 0 0 3 (50.0) 3 (50.0) 0 3.50 .548 Total 12 0 1 (8.3) 3 (25.0) 7 (58.3) 1 (8.3) 3.67 .778 .280 19. I am convinced of the defensibility of the passing score using Ebel method in OBA. A 6 0 0 0 4 (66.7) 2 (33.3) 4.33 .516 B 6 0 0 3 (50.0) 3 (50.0) 0 3.50 .548 Total 12 0 0 3 (25.0) 7 (58.3) 2 (16.7) 3.92 .669 .056 22. I am convinced of the defensibility of the passing score using Ebel method in SAQ. A 6 0 0 2 (33.3) 4 (66.7) 0 3.67 .516 B 6 0 0 2 (33.3) 3 (50.0) 1 (16.7) 3.83 .753 Total 12 0 0 4 (33.3) 7 (58.3) 1 (8.3) 3.75 .622 .715 25. I prefer individual rating to group rating in Modified Angoff method. A 6 0 1 (16.7) 3 (50.0) 2 (33.3) 0 3.17 .753 B 6 0 2 (33.3) 2 (33.3) 2 (33.3) 0 3.00 .894 Total 12 0 3 (25.0) 5 (44.7) 4 (33.3) 0 3.08 .793 .733 28. I prefer individual rating to group rating in Ebel method. A 6 0 2 (33.3) 3 (50.0) 0 1 (16.7) 3.00 1.095 B 6 0 3 (50.0) 2 (33.3) 1 (16.7) 0 2.67 .816 Total 12 0 5 (44.7) 5 (44.7) 1 (8.3) 1 (8.3) 2.83 .937 .604 31. I feel Modified Angoff requires more cognitive thinking (thinking and reasoning process) than Ebel. A 6 0 4 (66.7) 1 (16.7) 1 (16.7) 0 2.50 .837 B 6 0 2 (33.3) 1 (16.7) 1 (16.7) 2 (33.3) 3.50 1.378 Total 12 0 6 (50.0) 2 (16.7) 2 (16.7) 2 (16.7) 3.00 1.206 .169 Likert’s Scale : 1 = Strongly disagree(SD), 2 = Disagree(D), 3 = Neutral(N), 4 = Agree(A), 5 = Strongly Agree(SA) a -Mann-Whitney U (*) indicates a statistical significance (p < 0.05) (Insert Table 2 ) (Insert Fig. 3 ) Qualitative data All judges attended the entire FGD sessions. The analyses in this section describe the data collected from the open-ended questions in the questionnaires and FGDs. The results suggest that much of the data from Group A was similar to those of Group B. Table 3 presents the themes and subthemes related to the judges’ experiences and thought processes during the modified Angoff and Ebel standard setting meetings. Table 3 Identified themes and subthemes related to judges’ experiences and thought processes during modified Angoff and Ebel standard setting meetings Themes Subthemes Confidence in standard setting 1. Continuous vs. category scoring 2. Cognitive challenge The role of diverse subject matter experts in decision-making 1. Expertise and content knowledge 2. Years of experience 3. Enhancing standard setting through group discussion Understanding and familiarity in Course Learning Outcomes Barriers and facilitator in the standard setting process 1. Insufficient faculty resources and workload pressures 2. Training as a catalyst for effective standard setting (Insert Table 3 ) Theme 1: Confidence in the standard setting Continuous against category scoring. The modified Angoff method involves continuous scoring requiring judges to estimate the probability of a borderline student answering correctly, with percentage ranging from 0 to 100. Judges described the percentage method in the modified Angoff approach as simplified, focused on the item-level, straightforward and readily comprehensible. “Easier to come up with the percentage” (P02). “The decision-making process (modified Angoff) is much simplified and easier.” (P12). The judges expressed greater confidence in the modified Angoff ratings because they allowed them more control over probability decisions compared to the fixed probabilities in the Ebel method. They also found the continuous ratings in the modified Angoff method to be more practical and usable than the categorical ratings in the Ebel method. “With modified Angoff, the percentage and probability were decided by the panel, whereas in Ebel, it is already fixed…so I think the panel has more control over the probability in modified Angoff over Ebel, as Ebel already fixed” (P05). “…continuous rating can be more representative of what you want to give rather than categorical rating. Ebel is categorised, it may not truly reflect the marks or the percentage that you want to give” (P13). For polytomous items (SAQs), many judges believed that the modified Angoff method permits individual ratings for each sub-question, effectively accounting for their varying levels of difficulty. “(modified Angoff in SAQ) It actually reads each sub-question depending on the complexity… one sub-question might be simple and then another sub-question maybe difficult question, in Ebel, you can’t really rate as the complexity of each sub question.” (P12) “(modified Angoff in SAQ) We can actually assess the sub-questions and then we give ratings for the three sub questions then composite rating.” (P13) In the Ebel method, the judges conducted category judgements at the item-level, evaluating each item based on its difficulty and relevance to the test [ 30 ]. Judges expressed confidence in the Ebel method because its structured framework made the decision-making process more objective. They elaborated that the Ebel method provides clear guidance for categorising items based on the levels of difficulty and relevance within the 3x3 Ebel grid. This guideline allows judges to accurately determine the relevance and difficulty of test items in alignment with the course learning outcomes (CLOs). “Ebel you identify the difficulty and then the relevancy and then it’s more objective, you know in which category you are actually rating it. I prefer Ebel in OBA” (P13). “Ebel requires less effort thought to it, it gives us some direction rather than Angoff” (P14). In the context of SAQs, judges preferred the Ebel method due to its clear and structured guidance. They found the process of quantifying ratings with numerical values to be particularly challenging. “In SAQ questions, I personally chose Ebel because it’s kind of hard for me to rate the number because, you know, if the questions have different difficulty and I don’t know how to sum up with the number. So that’s why I actually feel that Ebel gives me some direction and sort of helps me to give the rating that I am supposed to give based on the three questions” (P14). Cognitive challenges. Several judges found the broad range of continuous scoring in the modified Angoff method to be subjective, perceiving it as largely based on individual perceptions. They also found it challenging to accurately assign probability. “Angoff is an arbitrary number from zero to hundred…more of a guesswork that relies on personal perception" (P06). “Because we don’t have any guidance for the modified Angoff when we were to give the percentage for the borderline student, and we just based it on our gut feeling. Like, you know, all this thing maybe is 75% of the students can answer, things like that” (P11). “(modified Angoff) based on perceived marks, so it’s very difficult to like rate the passing marks” (P12). In the Ebel method, ratings are determined using a 3x3 Ebel grid that categorises difficulty and relevancy. However, judges mentioned their confusion in determining the relevancy category and noted inconsistencies in their ratings across nine options during the rating process. A judge referred to the percentage of borderline candidates passing in each grid when making decisions in the Ebe method. “(Ebel) It is difficult to decide the relevancy of the question within the framework of the course learning outcome” (P05). “(Ebel) You have to think quite far and deep to decide on that. So I think it takes more time to decide. From the exercises, I found that I couldn’t be consistent in my decisions. We were more or less agreeable on the level of difficulty, but we were all in conflict when it came to the relevance category, essential and important” (P16). “(Ebel) When I come to a question that I want to decide if it’s an easy, moderate or difficult question, sometimes I cannot say. So I select that first, then I look at the percentage in that category to guide me in my selection” (P15). Theme 2: The role of diverse subject matter experts in decision-making Expertise and content knowledge. Some judges observed a tendency to establish higher standards for their own subject areas within the modified Angoff framework. Conversely, others expressed confidence in gauging the level of borderline students on items related to their specialties. These judges argued that active engagement in teaching those subjects significantly enhances their ability to make fair and informed judgements. “I tend to set higher percentage when it comes to my subject, even though I try, because I’m teaching a very easy subject, it’s a very easy standard for me… for example, I put 90% and the rest of the group only puts 70, 75. So to me, that was a challenge when it comes to your own question” (P04). “In modified Angoff, it’s my discipline so I will expect the students to be able to answer so the scoring will be higher” (P11). “But if I were to do it in my own speciality, I feel that I am more confident to practise.” (P13) “…especially if that question related to my specialty…I can actually gauge the probability of the Angoff rating because we actually do the supervision in the clinic… based on our experience, we can make the judgement on the score where it should be.” (P05) From the FGDs, judges acknowledged that they encountered challenges in making judgements on unfamiliar topics, especially when they were not directly involved in the teaching sessions for those subjects. This lack of direct teaching engagement with the subject matter introduced uncertainty in their judgement on the items. Therefore, the judges suggested including a mix of disciplines in the panel to ensure fairer judgement. “I’m struggling to decide the relevancy of the subject and the difficulty level of that particular question, if it’s outside my specialty. I am not sure the depth of the subjects that have been taught; how much emphasis has been put forward to the students about the subject. For example, oral cancer, and we think it’s a very important subject but I don’t have any information for how many lectures have been taught in that particular subject and even though we can see from the course learning outcome but it’s a very vague statement” (P05). “…in short, I think it is good to have a various background of specialty in the group” (P01). Years of experience. Judges believed that senior staff members, particularly those who have experience in item writing and marking papers, may find the modified Angoff method relatively straightforward. However, some judges emphasised the value of having a variety of levels of experience for a more balanced approach to standard setting. “It plays a big role if you already have many years mark the student’s paper. So you can gauge which questions that borderline students can answer, the senior lecturer will probably think that modified Angoff will work out well” (P05). “If all the panels are senior, it’s more valid to use modified Angoff but if we have like two groups of panels, like junior and senior, I think Ebel is more accurate as it is more defined on the level of the relevancy and difficulty” (P03). “Good to have different combination of panels, however having said even same panel, same year we have different perceptions of rating.” (P14) Enhancing standard setting through group discussion. The judges concurred that holding a group discussion before finalising their judgement provided an opportunity to gain insights from other judges’ perspectives and understand the rationale behind their ratings. This process reinforced the standard for borderline students and helped refine their own scoring. “Good to listen to other judges’ justification as to why they place the relevancy. Hearing other judges’ justification for their score helps in my scoring. Keep on having to tell myself that I have to think as borderline student” (P02). “The group discussion helps to understand better the feedback of other judges' scores and this affects the next ratings” (P12). Theme 3: Understanding and familiarity in CLO Many judges emphasised the importance of understanding the CLO and aligning their judgements with it to ensure fairness and reduce variations among judges. However, in a multidisciplinary course like this, some judges face cognitive challenges due to the extensive learning outcomes. “And I feel that it’s not about the years of experience, it’s not about your specialty, but I think the more important bit is the understanding of the course learning outcome or programme learning outcome…if you understand it and you have a good grasp of what is it that we want to achieve in the programme or in the course, I think that it will reduce the different” (P16). “The course that has so many learning outcomes, like CDC, sometimes I might overload some of the learning outcomes, so I tend to be a little bit difficult at certain point, to cover all, then I have to go back to the learning outcomes” (P03). “We go through the course learning outcome first, so that we are aware of the learning outcomes, so that easier when we want to decide whether it’s essential or important or even when we want to decide in modified Angoff, which question is in which CLO and what is the level, you know. Some of us, I myself I also cannot remember, like for the CDC course, we have 16 learning outcomes” (P11). Theme 4: Barriers and facilitator in the standard setting process Insufficient faculty resources and workload pressures. Judges identified several challenges and practical issues in implementing the modified Angoff or Ebel standard setting in the dental faculty, including a limited number of teaching academics, time constraints and increased workload. They acknowledged that having sufficient judges is crucial for reliable results. However, some judges reported feeling stressed due to the iteration process and the added burden of their existing responsibilities. “Looking at the number of specialists here in UKM, I think some of the specialties maybe hard to get that good combination of panel, because we have a limited number like oral medicine and pathology” (P14). “Practically how it is going to be carried out, as we already took so many hours just to discuss 20 questions, and so many days, who will sit on the panels practically in terms of time, as ideal as this. I don’t think we can get all 5 or 4 periodontists to sit together from Monday to Friday to set standard, as everyone has clinic. When I am free you are not free, that is how the timetable goes” (P06). “It is a very tiring process, as you have to go through many rounds to get to, not really consensus, but at least not so much of standard deviation” (P02). “Even without standard setting, we are already quite burdened” (P04). Training as a catalyst for effective standard setting . The judges expressed that their understanding and confidence in applying the ratings increased with each round of the process. They also emphasised the importance of adequate training and practice before the actual standard setting process to boost confidence and effectiveness. “It is clearer now after the second round. My thoughts are clearer…I'm more confident after Round 3” (P13). “If all of your methods are well trained, I don’t think there’s any difference between Modified Angoff and Ebel” (P13). “If the panels are well trained it will overcome that problem and making Angoff more defensible in overall rating as it has more flexibility…the training is important to make sure that everyone understands the process.” (P14) Discussion and integrated consideration of the findings This mixed-methods study examined judges’ feedback, experiences and perspectives on applying the performance standard across different methods. The feedback evaluation from judges regarding the workshop and meetings showed positive responses towards the standard setting procedure (Fig. 2 ), which showed similarity with other studies in healthcare professionals' education [ 9 , 39 – 42 ]. Several authors have previously suggested formats for feedback forms that judges can complete after the meetings to evaluate the effectiveness of the workshop and meetings[ 8 , 32 ]. However, feedback forms are typically modified as needed to include the questions deemed necessary. In this study, in addition to general feedback, the judges were asked to complete a feedback form after each round of ratings. The comparison of the standard setting process in R1, R2, and R3 across the modified Angoff and Ebel methods within the same group and between Groups A and B showed no significant differences. Judges in both groups exhibited similar levels of understanding and confidence in making standard setting ratings and in the defensibility of the passing scores using both the modified Angoff and Ebel methods. The qualitative data revealed that both groups have similar experiences and perspectives regarding the standard setting meetings. However, their thought processes in applying the modified Angoff and Ebel methods differed. These findings served as an internal validation of procedural validity in setting the passing scores for the examination [ 32 ]. The quality and duration of the training have been identified as factors affecting judges’ confidence and reliability levels [ 10 , 11 , 43 ]. In this study, judges rated time allocation highly, with mean scores above 4.0 for both methods. They received a full day of lectures and one hour per round to rate 30 items. In contrast, a Canadian Ebel study provided only 20 minutes of training on standard setting and three hours to rate 203 MCQs, yielding low agreement (Fleiss’ kappa = 0.11) and suggesting that time constraints undermined procedural validity [ 44 ], as judges were under significant pressure to complete the ratings within the allotted time. Although responses on the feedback questionnaires meetings (Part 2 & Part 3) from R1 to R3 for the modified Angoff and Ebel methods were not significantly different, FGDs revealed significant differences in judges’ thought processes when using the two methods. In FGDs, judges described the numerical ratings in the modified Angoff method as straightforward and easy. Nonetheless, some judges criticised the numerical ratings as being perception-based and reliant on guesswork. Judges have faced cognitive challenges in accurately estimating borderline candidates using the Angoff method, a finding also reported in previous studies [ 13 , 45 ]. Some judges appreciated the structured framework of relevancy and difficulty levels in the Ebel method, which provided valuable guidance and boosted their confidence in the defensibility of the ratings. However, a few judges felt they had less control over the final percentage outcome compared to the modified Angoff, as it was predetermined in the Ebel grid. An earlier study by Hudges (1983) in medical examinations reported similar findings, noting that judges considered the Angoff, Guerin, and National Board of Medical Examination (Rasch model) methods feasible, but not the Ebel method. They lacked confidence in Ebel due to difficulties judging success rates from hypothetical relevancy and difficulty levels and uncertainty about how changes in these factors affected their overall judgement of minimal competent candidates [ 46 ]. Dual process theory may be used to suggest how judges think during the standard setting process [ 47 ]. Type 1 processes are intuitive and automatic in nature, associated with fast, parallel, and biased responses. These processes are automatic and experience-based in decision-making. In contrast, Type 2 processes are deliberative and analytical, relating to slower, serial, normative responses that are controlled and consequential in nature [ 47 ]. The modified Angoff method is claimed to be easy, yet subjective, and participants have been observed to shift toward Type 1 thinking by simply assigning a percentage after viewing the item. In contrast, the categorical framework in the Ebel method may guide judges to engage in Type 2 processes during their judgments, as they frequently refer to CLOs to determine the relevancy category. The purposive sampling method aimed to ensure diversity among the judges so they were broadly representative of various backgrounds, thereby making the recommended passing standard reasonable and valid [ 6 , 48 – 50 ]. Although the content was at the undergraduate dental degree level and all judges were appropriately qualified subject matter experts, FGDs revealed that some struggled to judge topics outside their direct teaching areas. Judges also differed on how their specialty influenced standard setting, some believed they set higher standards in their own field, while others felt their expertise enabled more accurate evaluation of borderline students within their specialty. Previous studies have shown that judges’ expertise in specific subject areas did not significantly influence standard setting outcomes using the Angoff method, regardless of the content subspecialty [ 46 , 51 , 52 ]. However, many studies showed that judges tended to set higher standards for the items that they knew the correct answers to and lower standards for those they did not [ 53 – 55 ]. Further FGD data in this study revealed that judges found it challenging in rating unfamiliar items. Clauser et al. (2017) found that judges tended to rate unfamiliar items as more difficult, leading to lower Angoff ratings and ultimately lower passing scores [ 56 ]. The impact of unfamiliar items on the final passing scores depends on the ratio of unfamiliar items in the exam. Excluding unfamiliar items from the standard setting process may help reduce systematic errors, nevertheless, if judges from a particular panel are more likely to mark items as unfamiliar, omitting their input may compromise the intended diversity and variation among the panels. The authors suggested that alternative approaches, such as the Bookmark method or an Item Response Theory (IRT)-based model should be considered depending on the impact of unfamiliar items and the context of the particular exam [ 56 ]. To reduce the ratings’ variability and gap between judges, group discussions before the final round were aimed to share opinions and justification for their ratings. Feedback questionnaires indicated that most judges preferred group ratings over individual ratings, with similar preferences in Groups A and B. FGDs revealed that judges found group discussions helpful for refining their ratings. Yeates et al. (2019) observed that judges often conform to group ratings to minimise variations. However, some resist conformity and, instead of engaging in debate, adjust their ratings away from the group mean when they disagree [ 57 ]. In standard setting, there is no definitive gold standard for judging the accuracy and validity of the outcomes, multiple sources of validity evidence must be considered. While high inter-judge agreement is desirable but such agreement without critical evaluation of opinions maybe is a sign of groupthink. The Groupthink Model was first described by Irving Janis in 1972 as a framework for understanding decision-making in highly cohesive groups. In such groups, members prioritise agreement and harmony over critical evaluation, which can potentially lead to poor decision-making [ 58 ]. Groupthink is typically characterised by overconfidence in decisions, a stereo-typed perception of opposing views, ignoring alternatives and pressure towards uniformity by marginalising dissenting opinions. The FGD and feedback questionnaire revealed that judges acknowledged the importance of aligning the CLO with the test’s purpose when making standard setting judgements. The feedback questionnaire indicated that judges referred to the CLO during the standard setting process. Nevertheless, some judges reported difficulties in recalling and appropriately applying the CLO when assigning ratings. According to the Standards for Educational and Psychological Testing , the process of establishing performance standards in an assessment should be closely aligned with content standards [ 59 ]. Learning outcomes are measurable achievements defined by content standards, indicating what students should know and be able to do. These outcomes need to be clear and specific to effectively guide teachers and students in their teaching, learning and assessment activities. The alignment of performance standards with the CLO is crucial for the reliability of the ratings, as it helps judges set passing scores based on the expected performance standards. The majority (91.7%) of our judges relied moderately to heavily on the descriptions of borderline groups when making judgments. A mixed-methods study by Ferdous and Buckendahl (2013) found that understanding on the performance level descriptors helped judges make consistent item ratings in the English and Mathematics examinations [ 12 ]. In contrast, judges from Giraud and Impara (2005) and McGinty (2005) admitted that their cut-score estimations were influenced by external factors. They considered political, social, and economic factors when establishing the standards for the English test and high school graduation test [ 16 , 17 ]. The use of performance level descriptors is an important indicator of a criterion-referenced approach in high-stakes examinations for healthcare professionals, as incorrect pass-fail decision can have significant implications for patient safety. Judges noted several barriers to implementing standard setting practices in the dental faculty, including a limited number of staff in a single discipline, time constraints and a high workload. Participants demonstrated improved mean ratings for understanding and confidence with each round of rating. They emphasised that adequate training in standard setting practices can enhance the reliability of the proposed passing scores. This finding aligns with other studies on the implementation of standard setting practices for national assessments in the Republic of Trinidad and Tobago [ 13 ]. Resources such as appointing subject matter experts, providing expertise to conduct standard setting meetings, and offering adequate training for staff members are important factors to consider when implementing standard setting practise. Implications and recommendations Our research is the first study to investigate the differences in the standard setting process from R1 to R3 between the modified Angoff and Ebel methods in dental education. It also explored the thought processes and factors that boosted their confidence of dental academics in the decision-making in these methods. The findings have several implications for standard setting practise. The standard setting process and feedback questionnaires provide guidelines on documenting procedural validity evidence for future standard setting studies in dental education. Besides, judges provided their perspectives and the challenges they face when making ratings. These considerations should be carefully addressed in the planning and choice of methods used for future standard setting workshops and meetings in dental education. For future standard setting workshops, we suggest appointing judges from varied backgrounds, focusing on the borderline descriptors for each CLO and promoting group discussions to achieve more credible results. Judges’ insights can improve the efficiency of future workshops and produce more credible ratings. The standard setting protocol and locally gathered data can serve as an evidence-based evaluation of standard setting activities, offering recommendations for dental assessment policy implementation. Collaborating with various stakeholders and pooling resources from multiple dental schools could facilitate the implementation of a standard setting for national dental licencing examinations. Limitations of this study This mixed-methods study combines extensive feedback evaluation forms from each round of ratings and FGDs to provide an in-depth evaluation of the procedural validity and the thought process of judges. However, this study focused on judges from one institution in Malaysia, limiting its applicability to other settings. Along with FDG, other data collection methods, such as observational studies, could be applied to examine group dynamics, specifically behaviours and communication in achieving group consensus. Individual in-depth interviews may provide more insights into other factors that influence judges to conform to the ratings. Conclusion The standard setting workshop received positive feedback and responses from judges regarding the standard setting process. Judges reported increased confidence and understanding following iterative rounds and group discussions. This study highlighted the strengths and challenges of the modified Angoff and Ebel methods in the standard setting for theory-based dental assessments. The modified Angoff method was straightforward and easy; it gave judges more control, but it provided a subjective framework, relying on intuition and experience in setting the Angoff ratings. In contrast, the Ebel method offered a structured, objective framework, but judges felt they had less control, found the process more time-consuming, and some referred to the probability that borderline candidates would answer correctly within each category in the Ebel grid to determine the difficulty level. These findings provide insights into standard setting in dental education and emphasis the importance of structured training to enhance procedural validity. Abbreviations FGD: focus group discussion; SAQ: short answer questions; OBA: one-best answer; CLO: course learning outcome Declarations Clinical trial number Not applicable. Acknowledgments We would like to thank Professor Dr Noor Lide Abu Kassim (NLAB, a language educationist and psychometrician) from Kulliyyah of Education, International Islamic University Malaysia, and Dr Mohd Nasri Bin Awang Besar (MNAB, a medical educationist) from Medical Education Department, the Faculty of Medicine, Universiti Kebangsaan Malaysia, for their expertise and contributions to the standard setting workshop. We also want to thank all the participants for their time and involvement in this research. Author contribution TKH contributed to the conception of the research idea, design, collection of data, performing quantitative and qualitative data analysis and writing the manuscript. LO contributed to formulating the research idea, interpreting quantitative and qualitative data, and critically reviewing and editing the manuscript. RVR contributed to formulating the research idea, interpreting quantitative and qualitative data, and critically reviewing and editing the manuscript. All authors read and approved the final manuscript. Funding This study is part of the research conducted for TKH’s PhD project at The University of Manchester, which is supported by student research fees. Data availability The transcripts and datasets collected and/or analysed during the current study are not publicly available due to the confidentiality of the data but are available from the first author, Ting Khee Ho on request. Declarations Ethical approval The study adhered to the relevant guidelines and regulations as detailed in the Declaration of Helsinki. Study approval was granted jointly by the ethics committees of The University of Manchester Research Ethics Committee (2023-17408-31608) the National University of Malaysia (Universiti Kebangsaan Malaysia)(JEP-2023-2040). Consent to participate All participants have provided informed consent. Consent for publication All participants have consented to publication and using anonymised data without reference to individuals. Competing interests The authors declare no competing interests Conflict of interests The authors declare no conflict of interest. References Yudkowsky R, Park YS, Downing SM. Assessment in Health Professions Education 2nd edn: Routledge New York, NY; 2019. Yudkowsky R, Downing SM, Tekian A. Standard setting. In: Assessment in Health Professions Education. edn. Edited by Yudkowsky R, Park YS, Downing SM: Routledge New York, NY; 2019: 119-48. Garcia FF, Casarin M, Silva FH, Muniz F. Association between history of academic failure during dental school and self-reported depression, anxiety, and stress: A retrospective cohort. Journal of Dental Education 2023; 87(11):1502-11. Kane MT. Validating the performance standards associated with passing scores. Review of Educational Research 1994; 64(3):425-61. Pant HA, Rupp AA, Tiffin-Richards SP, Köller O. Validity issues in standard-setting studies. Studies in Educational Evaluation 2009; 35(2):95-101. Pitoniak MJ, Morgan DL. Setting and validating cut scores for tests. In: Handbook on measurement, assessment, and evaluation in higher education. 2nd edn. Edited by Secolsky C, Denison DB. New York: Routledge; 2012: 343-65. Pitoniak MJ. Standard setting methods for complex licensure examinations. Doctoral Dissertations University of Massachusetts Amherst; 2003. Hambleton R, Pitoniak MJ, Copella JM. Essential steps in setting performance standards on educational tests and strategies for assessing the reliability of results. In: Setting Performance Standards : Foundations, Methods, and Innovations. 2nd edition edn. Edited by Cizek GJ. New York & UK: Routledge; 2012: 47-76. Yim MK, Shin S. Using the angoff method to set a standard on mock exams for the korean nursing licensing examination. Journal of Educational Evaluation for Health Professions 2020; 17:14. Kampa N, Wagner H, Köller O. The standard setting process: Validating interpretations of stakeholders. Large-scale Assessments in Education 2019; 7(1):3. Skorupski WP, Hambleton RK. What are panelists thinking when they participate in standard-setting studies? Applied Measurement in Education 2005; 18(3):233-56. Ferdous AA, Buckendahl CW. Evaluating panelists' standard setting perceptions in a developing nation. International Journal of Testing 2013; 13(1):4-18. De Lisle J. Installing a system of performance standards for national assessments in the republic of trinidad and tobago: Issues and challenges. Applied Measurement in Education 2015; 28(4):308-29. Giraud G, Impara JC, Plake BS. Teachers' conceptions of the target examinee in angoff standard setting. Applied Measurement in Education 2005; 18(3):223-32. Hein SF, Skaggs G. Conceptualizing the classroom of target students: A qualitative investigation of panelists' experiences during standard setting. Educational Measurement, Issues and Practice 2010; 29(2):36-44. Giraud G, Impara JC. Making the cut: The cut score setting process in a public school district. Applied Measurement in Education 2005; 18(3):289-312. McGinty D. Illuminating the" black box" of standard setting: An exploratory qualitative study. Applied Measurement in Education 2005; 18(3):269-87. Hein SF, Skaggs GE. A qualitative investigation of panelists' experiences of standard setting using two variations of the bookmark method. Applied Measurement in Education 2009; 22(3):207-28. Schoonheim-Klein M, Muijtjens A, Habets L, Manogue M, Van Der Vleuten C, Van Der Velden U. Who will pass the dental osce? Comparison of the angoff and the borderline regression standard setting methods. European Journal of Dental Education 2009; 13(3):162-71. Dalum J, Christidis N, Myrberg IH, Karlgren K, Leanderson C, Englund GS. Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists. European Journal of Dental Education 2023; 27(3):640-49. Abd-Rahman ANA, Baharuddin IH, Abu-Hassan MI, Davies SJ. A comparison of different standard-setting methods for professional qualifying dental examination. Journal of Dental Education 2021; 85(7):1210-16. Yousef MK, Alshawwa LA, Farsi JM, Tekian A, Soo PY. Determining defensible cut-off scores for dental courses. European Journal of Dental Education 2020; 24(2):186-92. Kim J, Yang JS. How to improve reliability of cut‐off scores in dental competency exam: A comparison of rating methods in standard setting. European Journal of Dental Education 2020; 24(4):734-40. Ho TK, Abu Kassim NL, O'Malley L, Roudsari RV. Standard setting for dental knowledge tests: Reproducibility of the modified angoff and ebel method across judges 2025; [Manuscript submitted for publication]. Younas A, Fàbregues S, Creswell JW. Generating metainferences in mixed methods research: A worked example in convergent mixed methods designs. Methodological Innovations 2023; 16(3):276-91. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research, Third Edition. edn. Thousand Oaks, California: SAGE; 2018. Cizek, Bunch. The angoff method and angoff variations. In: Standard setting. edn. Edited by Cizek GJ, Bunch MB. Thousand Oaks, California: SAGE Publications, Inc.; 2007. Angoff WH. Norms, scales, and equivalent scores, vol. 2, 2nd edn: Washington, DC: American Council on Education; 1971. Hambleton RK, Plake BS. Using an extended angoff procedure to set standards on complex performance assessments. Applied Measurement in Education 1995; 8(1):41-55. Ebel RL. Essentials of educational measurement. Englewood Cliffs, N.J.: Prentice-Hall; 1972. Homer M, Darling J, Pell G. Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality. Assessment and Evaluation in Higher Education 2012; 37(7):787-804. Cizek GJ, Bunch MB, Koons H. Setting performance standards: Contemporary methods. Educational Measurement: Issues and Practice 2005; 23(4):31-31. Cizek GJ. The forms and functions of evaluations in the standard setting process. In: Setting Performance Standards : Foundations, Methods, and Innovations. 2nd edn. Edited by Cizek GJ. New York & UK: Routledge; 2012: 165-78. Krueger RA, Casey MA. Focus Groups : A Practical Guide for Applied Research, 5th edition. edn. Thousand Oaks, California: SAGE; 2015. Brookfield C. Using Microsoft Excel for Social Research. Los Angeles: SAGE; 2021. Braun V, Clarke V. Thematic Analysis: A Practical Guide: SAGE Publications; 2021. DeJonckheere M, Vaughn LM, James TG, Schondelmeyer AC. Qualitative thematic analysis in a mixed methods study: Guidelines and considerations for integration. Journal of Mixed Methods Research 2024; 18(3):258-69. Taber KS. The use of cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education 2017; 48(6):1273-96. Yim M. Comparison of results between modified-angoff and bookmark methods for estimating cut score of the Korean medical licensing examination. Korean J Med Educ 2018; 30(4):347-57. Yousefi Afrashteh M. Comparison of the validity of bookmark and angoff standard setting methods in medical performance tests. BMC Medical Education 2021; 21(1). Kang Y. Evaluating the cutoff score of the advanced practice nurse certification examination in korea. Nurse Educ Pract 2022; 63:103407. Park J, Yim MK, Kim NJ, Ahn DS, Kim YM. Similarity of the cut score in test sets with different item amounts using the modified angoff, modified ebel, and hofstee standard-setting methods for the Korean medical licensing examination. Journal of Educational Evaluation for Health Professions 2020; 17:28. Plake BS, Impara JC, Irwin PM. Consistency of angoff-based predictions of item performance: Evidence of technical quality of results from the angoff standard setting method. Journal of Educational Measurement 2000; 37(4):347-55. Bourque J, Skinner H, Dupré J, Bacchus M, Ainslie M, Ma IWY, Cole G. Performance of the ebel standard-setting method in spring 2019 royal college of physicians and surgeons of canada internal medicine certification examination consisted of multiple-choice questions. Journal of Educational Evaluation for Health Professions 2020; 17:12. Impara JC, Plake BS. Teachers' ability to estimate item difficulty: A test of the assumptions in the angoff standard setting method. Journal of Educational Measurement 1998; 35(1):69-81. Hughes FP. Comparing four estimates of the criterion-referenced standard for a written test 1983: 1-32. Evans JS, Stanovich KE. Dual-process theories of higher cognition: Advancing the debate. Perspect Psychol Sci 2013; 8(3):223-41. Bunch MB. Practical issues in standard setting. In: Setting performance standards: Foundations, methods, and innovations. 2nd edition edn. Edited by Cizek GJ. New York & UK: Routledge; 2012. Shulruf B, Wilkinson T, Weller J, Jones P, Poole P. Insights into the angoff method: Results from a simulation study. BMC Med Educ 2016; 16:134. Rezigalla AA. Angoff's method: The impact of raters' selection. Saudi J Med Med Sci 2015; 3(3):220-25. Plake B, Impara J, Potenza M. Content specificity of expert judgments in a standard-setting study. Journal of Educational Measurement 1994; 31(4):339-47. Norcini JJ, Shea JA, Kanya DT. The effect of various factors on standard setting. Journal of Educational Measurement 1988; 25(1):57-65. Peabody MR, Wind SA. Exploring the influence of judge proficiency on standard‐setting judgments. Journal of Educational Measurement 2019; 56(1):101-20. Verheggen MM, Muijtjens AM, Van Os J, Schuwirth LW. Is an angoff standard an indication of minimal competence of examinees or of judges? Advances in Health Sciences Education : Theory and Practice 2008; 13(2):203-11. Chang L, Dziuban C, Hynes MC, Olson AH. Does a standard reflect minimal competency of examinees or judge competency. Applied Measurement in Education 1996; 9:161-73. Clauser JC, Hambleton R, Baldwin P. The effect of rating unfamiliar items on angoff passing scores. Educational and Psychological Measurement 2017; 77:901 - 16. Yeates P, Cope N, Luksaite E, Hassell A, Dikomitis L. Exploring differences in individual and group judgements in standard setting. Medical Education 2019; 53(9):941-52. Hart Pt. Irving l. Janis' victims of groupthink. Political Psychology 1991; 12(2):247-78. American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME). Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014. Additional Declarations No competing interests reported. Supplementary Files Supplementaryfile1.docx Supplementary file 1: Summary on the Feedback Questionnaire for judges Supplementaryfile2.docx Supplementary file 2: Focus Group Discussion (FGD) Guide Supplementaryfile3.docx Supplementary file 3: Mean and standard deviation of judges’ responses to each statement after attending lectures (Part 1) and after attending all standard setting meetings (Part 4a) Supplementaryfile4.docx Supplementary file 4: Mean and standard deviation of judges’ responses to each statement after Round 1, 2 and 3 of standard setting meetings (Questionnaire Part 2 & Part 3) Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7347939","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":517232536,"identity":"9ee764e3-1597-46ec-b3a5-9f1a1222b0a4","order_by":0,"name":"Ting Khee HO","email":"","orcid":"","institution":"University of Manchester","correspondingAuthor":false,"prefix":"","firstName":"Ting","middleName":"Khee","lastName":"HO","suffix":""},{"id":517232539,"identity":"89bd23b8-854b-4b7b-9ef7-11ca1e400b39","order_by":1,"name":"Lucy O’MALLEY","email":"","orcid":"","institution":"University of Manchester","correspondingAuthor":false,"prefix":"","firstName":"Lucy","middleName":"","lastName":"O’MALLEY","suffix":""},{"id":517232540,"identity":"95d0c426-f140-4905-9926-76f00680d1c3","order_by":2,"name":"Reza Vahid ROUDSARI","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA40lEQVRIiWNgGAWjYLCCDyBCAkmAsYGADsYZMC0HiNXCzEOSFvP29oefbXPs8vhnNx/+/KHiHgN/+wE2yRl4tMicOZAsnbstuVjizrE0iQNnihkkziSwSW7Ao0VCIuEAUAtz4gaJHDOGg20JDAw3GNgkH+DTIv+w+bfltnqglvzPHw7+S2CQJ6hFgplNmnHbYZAtDBIHGxIYDEBa8DqMJ43Nsnfb8cQZN9LMJM4cS+AxPJPYbInP+xLsxx/f+LmtOrF/RvLjDxU1CXJyxw8fvNmDRwsG4CEiIkfBKBgFo2AUEAIAP05NsIhQ7WcAAAAASUVORK5CYII=","orcid":"","institution":"University of Manchester","correspondingAuthor":true,"prefix":"","firstName":"Reza","middleName":"Vahid","lastName":"ROUDSARI","suffix":""}],"badges":[],"createdAt":"2025-08-11 15:38:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7347939/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7347939/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":92081629,"identity":"0c447bf9-b4e9-4f16-82b4-9f185a6ae65b","added_by":"auto","created_at":"2025-09-24 11:59:08","extension":"jpg","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":133058,"visible":true,"origin":"","legend":"","description":"","filename":"Fig1Flowchart.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/842d618adf1008e2eeb808e8.jpg"},{"id":92082709,"identity":"0f3833fd-9465-4e86-a3cf-b59aceeaa58c","added_by":"auto","created_at":"2025-09-24 12:15:09","extension":"jpg","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":171152,"visible":true,"origin":"","legend":"","description":"","filename":"Fig2generalfeedback.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/168418230d895006447456d9.jpg"},{"id":92081925,"identity":"3d03a38c-5927-4d61-90d6-4b1825e365b1","added_by":"auto","created_at":"2025-09-24 12:07:08","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":129878,"visible":true,"origin":"","legend":"","description":"","filename":"MainmanuscriptFeedbackandFocusgroup250829.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/77d52877ed9cee36f9255d52.docx"},{"id":92081926,"identity":"70a9aeb6-0690-407a-90fd-a7ffce0be92f","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"jpg","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":43535,"visible":true,"origin":"","legend":"","description":"","filename":"Fig3Preference.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/e6e13962e4a545b30da3c8b1.jpg"},{"id":92081633,"identity":"dfff1380-2ba4-4fdc-8858-75dc9df85acb","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29620,"visible":true,"origin":"","legend":"","description":"","filename":"Table.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/aa278901a267efd6aae7b38c.docx"},{"id":92081936,"identity":"3997cafb-c7e7-4a1c-985a-584116bc4f5d","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"json","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7048,"visible":true,"origin":"","legend":"","description":"","filename":"871de04304394ae78efeeade3d46adb1.json","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/ee83339a4e1862b012d72760.json"},{"id":92081639,"identity":"4eba710c-d095-461b-b180-10825d32cbfe","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17650,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile1.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/a2aa3f5de68e19304a70f338.docx"},{"id":92081641,"identity":"8f97bf5e-690f-4c75-a021-9ce7ed699261","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":25094,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile2.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/e9ee0794384bfdf6283bd5c7.docx"},{"id":92081644,"identity":"d92c3f9c-fa0a-4542-8e1b-257a911d2360","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":24418,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile3.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/584f7bfcfeece7b612daf1d2.docx"},{"id":92082711,"identity":"80abd146-fba5-4b7f-ad9a-5af0e0046053","added_by":"auto","created_at":"2025-09-24 12:15:09","extension":"docx","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":38815,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryfile4.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/7eb7360088acbf71b6e7fedf.docx"},{"id":92081652,"identity":"45662303-5d2e-454c-bbea-545f12c36dec","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"xml","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":179072,"visible":true,"origin":"","legend":"","description":"","filename":"871de04304394ae78efeeade3d46adb11enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/bfa83a0e06f425a3208200ac.xml"},{"id":92081930,"identity":"dded0061-647b-4bc3-92f0-55bc6edfcd24","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"jpg","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":133058,"visible":true,"origin":"","legend":"","description":"","filename":"Fig1Flowchart.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/b6dc428ff0a09432f67c55a2.jpg"},{"id":92081650,"identity":"d127ee79-f066-4ed2-8a55-e941895026a8","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"jpg","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":171152,"visible":true,"origin":"","legend":"","description":"","filename":"Fig2generalfeedback.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/4b8d921aa438d1b1201c1f02.jpg"},{"id":92081933,"identity":"a2eb8663-d919-4221-b567-822309f8e60e","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"jpg","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":43535,"visible":true,"origin":"","legend":"","description":"","filename":"Fig3Preference.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/acf7003d40340f5bc69df78b.jpg"},{"id":92081932,"identity":"0efb00bb-0dc5-4495-aee9-3439f10a54c1","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":57706,"visible":true,"origin":"","legend":"","description":"","filename":"OnlineFig1Flowchart.png","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/1067ddb7159369dbc1ad13da.png"},{"id":92081649,"identity":"74e18f0d-d513-4482-a2b3-95f24d3af4da","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":44886,"visible":true,"origin":"","legend":"","description":"","filename":"OnlineFig2generalfeedback.png","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/cd4e7f96390449b961373d0c.png"},{"id":92082712,"identity":"ed92e173-2c7f-4437-85ea-e5d6a382c6f1","added_by":"auto","created_at":"2025-09-24 12:15:09","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":12953,"visible":true,"origin":"","legend":"","description":"","filename":"OnlineFig3Preference.png","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/49175384ee2e96aa9749cab1.png"},{"id":92081653,"identity":"78e8caba-e86b-43e9-bb35-2b3e06099d8a","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"xml","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":176113,"visible":true,"origin":"","legend":"","description":"","filename":"871de04304394ae78efeeade3d46adb11structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/0f4895e86fbf6eb4b083ef3f.xml"},{"id":92081651,"identity":"d8254158-8a2a-4336-915a-b7429d7365bb","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"html","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":190564,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/2affb5cf59ea3fd45486dfe5.html"},{"id":92081628,"identity":"517e1af9-26ca-4156-8065-a612aceda699","added_by":"auto","created_at":"2025-09-24 11:59:08","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":133058,"visible":true,"origin":"","legend":"\u003cp\u003eFlowchart of research protocol\u003c/p\u003e","description":"","filename":"Fig1Flowchart.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/f1e0c9318842049d98ec1872.jpg"},{"id":92082710,"identity":"c088aa1f-7f54-4cd1-bfd9-09b1e020e03e","added_by":"auto","created_at":"2025-09-24 12:15:09","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":171152,"visible":true,"origin":"","legend":"\u003cp\u003eFeedback evaluation from judges after attending lectures (Part 1) and after completing the standard setting meetings (Part 4a)\u003c/p\u003e","description":"","filename":"Fig2generalfeedback.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/3ec17b65c17d86388de5b567.jpg"},{"id":92081631,"identity":"caa8f561-26ef-4c8c-9ed9-60316c062c69","added_by":"auto","created_at":"2025-09-24 11:59:08","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":43535,"visible":true,"origin":"","legend":"\u003cp\u003eJudges’ preference on standard setting methods (Part 4d)\u003c/p\u003e","description":"","filename":"Fig3Preference.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/34ac954856f6849ce4ebba90.jpg"},{"id":95229152,"identity":"fbb4fbfa-8108-4012-92ef-47732f48b136","added_by":"auto","created_at":"2025-11-05 16:34:31","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1643005,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/24af8875-5c25-4fb5-aace-f32441d87a83.pdf"},{"id":92082708,"identity":"3db9b1e2-a1cb-4d77-9332-42d319b6d189","added_by":"auto","created_at":"2025-09-24 12:15:08","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":17650,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary file 1: Summary on the Feedback Questionnaire for judges\u003c/p\u003e","description":"","filename":"Supplementaryfile1.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/c28211d667ecea440f5a1991.docx"},{"id":92081636,"identity":"ec3d4fec-d18c-47a2-8bfe-f3c72e6faaae","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":25094,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary file 2: Focus Group Discussion (FGD) Guide\u003c/p\u003e","description":"","filename":"Supplementaryfile2.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/43f237a5e0b60b92e28c5b3f.docx"},{"id":92081927,"identity":"e2cd41fa-00e3-44c5-8bcc-60001fbe8778","added_by":"auto","created_at":"2025-09-24 12:07:09","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":24418,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary file 3: Mean and standard deviation of judges’ responses to each statement after attending lectures (Part 1) and after attending all standard setting meetings (Part 4a)\u003c/p\u003e","description":"","filename":"Supplementaryfile3.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/eaa6f914ee0c586790ab6fe2.docx"},{"id":92081648,"identity":"d28e8294-9678-42de-83ac-6a01e6b76378","added_by":"auto","created_at":"2025-09-24 11:59:09","extension":"docx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":38815,"visible":true,"origin":"","legend":"\u003cp\u003eSupplementary file 4: Mean and standard deviation of judges’ responses to each statement after Round 1, 2 and 3 of standard setting meetings (Questionnaire Part 2 \u0026amp; Part 3)\u003c/p\u003e","description":"","filename":"Supplementaryfile4.docx","url":"https://assets-eu.researchsquare.com/files/rs-7347939/v1/7ea42a5f4028ff4e377a5bb7.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups","fulltext":[{"header":"Background","content":"\u003cp\u003eStandard setting has gained significant importance in dental education assessments in recent years as it plays a crucial role in determining the cut score representing the minimum acceptable academic performance in examinations. The purpose of this process in high-stakes examination is to determine the passing scores that correspond with competence levels, ensuring graduating healthcare professionals achieve the minimal competency level to provide safe and effective patient care [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] while preventing unfair failure of competent candidates, which could lead to demotivation, anxiety, a reduced workforce and increased financial and time burdens for both the candidates and institutions conducting remedial programmes [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eThe validity criteria of the standard setting process are essential for fair and defensible outcomes. While much research focuses on refining standard setting procedures to achieve more reliable results, reliability alone is insufficient to ensure the appropriateness of performance standards. Validity evidence in standard setting should encompass procedural, internal, external, and consequential criteria [\u003cspan additionalcitationids=\"CR5 CR6 CR7\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Procedural evidence is recognised as one of the main considerations when evaluating the validity of passing scores as a measure of performance standards. According to Kane\u0026rsquo;s validity framework, procedural criteria validity includes the selection of appropriate standard setting methods and ensuring their effective implementation, encompassing judge selection and training, defining performance standards, data collection, and obtaining judge feedback [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. To support the process validity in making pass-fail decisions, quantitative feedback is routinely collected from judges as part of the standard setting procedure [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Qualitative studies in other non-dental disciplines have examined judges\u0026rsquo; feedback and thought processes, offering procedural validity evidence and insights into the mental processes involved in standard setting practices [\u003cspan additionalcitationids=\"CR12 CR13 CR14\" citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eGiraud and Impara (2005) are among the first researchers to conduct qualitative research in the standard setting process for an English test using the modified Angoff method in a public school district. Judges struggled to align district expectations with the established standards, expressing concerns that the cut-scores were manipulated for political and social reasons [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Similarly, a qualitative study of high school Angoff standard setting meetings found that judges experienced political and social pressures, especially in subjects with high failure rates, and felt pressure to conform their decisions to others [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. They also struggle to conceptualise borderline students and distinguish prediction from value judgement. McGinty (2005) suggested that understanding these influences could improve future training programmes, ultimately enhancing the validity of standard setting. Both McGinty (2005) and Giraud and Impara (2005) show potential researcher bias due to unsystematic data collection and subjective analysis [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. McGinty relied solely on observational notes without a structured coding framework for data analysis [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e] while Giraud and Impara lacked a clear description of the framework used to analyse data from observations, interviews and discussions [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eBuilding on these weaknesses, other researchers adopted more structured qualitative methods using written questionnaires and open-ended questions to explore the judges\u0026rsquo; thought processes during the standard setting process [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Skorupski and Hambleton\u0026rsquo;s (2005) applied item mapping standard setting method was used to set cut-scores for the Grade 5 and 6 English Test, with judges responding to Likert scale and open-ended questions at various stages during the standard setting process. The study found varied interpretations of the performance level descriptors and differing perspectives on standard setting procedures throughout the process. Judges felt time pressure in the final round, potentially leading to inconsistencies. Additionally, inconsistencies emerged between the questionnaire responses and the written comments regarding confidence in the standards. While most claimed confidence in their ratings, many still mentioned concerns in their ratings, suggesting some underlying uncertainty [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. The triangulation of data in this research provides more credible insights into the judges\u0026rsquo; thought processes. Meanwhile, Ferdous and Buckendahl\u0026rsquo;s (2013) utilising a similar approach as the study by Skorupski and Hambleton (2005), exploring factors influencing judges\u0026rsquo; decision and their confidence levels using the Yes/No Angoff standard setting method in Grade 5 English and Mathematics examinations [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. In contrast to previous research, the judges expressed high confidence in their understanding of the performance level descriptors and the standard setting process after training. They primarily relied on performance level descriptors rather than candidates' performance data or judges feedback data [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. This criterion-referenced approach ensures that the standards are set according to desired performance expectations for each cut-score rather than relative ranking.\u003c/p\u003e\u003cp\u003eHein and Skaggs (2009) adopted focus group discussion (FGD) rather than traditional observations, questionnaires or interviews to collect participants\u0026rsquo; experiences with two variations of the Bookmark method [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. They audio-recorded and transcribed the FGD, then coded it to create categories in the result. Results showed that judges found selecting an item for a bookmark is cognitive challenge and often relied on their professional experiences and beliefs during judgement, deviating from the prescribed bookmark procedure.\u003c/p\u003e\u003cp\u003eNumerous studies on standard setting in dental assessments have focused on judges\u0026rsquo; consistency, the replicability of the results across different methods, and the correlations between ratings and student performance data [\u003cspan additionalcitationids=\"CR20 CR21 CR22\" citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. In Malaysia, where criterion-referenced standard setting is still emerging, Ho, O'Malley et al. (2025) conducted a workshop to explore the reproducibility of modified Angoff and Ebel methods across two judge panels at a Malaysian public dental institute [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. This study builds on previous research that aimed to investigate the experiences and thought processes of newly trained judges applying the modified Angoff and Ebel standards. It is the first study to use a mixed-methods approach to thoroughly examine judges\u0026rsquo; feedback, experiences and perspectives on the application of these standard setting methods among dental academics.\u003c/p\u003e\u003cp\u003eThe quantitative component compared feedback from two independent judge panels and examined differences between the modified Angoff and Ebel methods, providing evidence of procedural validity. The qualitative component explored judges\u0026rsquo; experiences, thought processes, and challenges to identify sources of variability and factors influencing scoring confidence in applying the modified Angoff and Ebel methods to knowledge test items. Findings aim to inform improvements in standard setting training thus strengthening the validity of the standard setting outcomes.\u003c/p\u003e\u003cp\u003eThese data provide evidence of procedural validity in the standard setting process. The qualitative component examined judges\u0026rsquo; experiences and thought processes to understand their beliefs and attitudes when making standard setting judgements. The analysis aimed to identify potential sources of variability, factors influencing confidence in scoring and challenges encountered in applying the modified Angoff and Ebel methods to knowledge test items. The findings are intended to provide valuable insights for enhancing future standard setting training, thus strengthening the validity of the standard setting outcomes.\u003c/p\u003e"},{"header":"Methods","content":"\u003ch2\u003eStudy design and setting\u003c/h2\u003e\n\u003cp\u003eThis research utilised a convergent mixed-methods research design, collecting quantitative data from feedback questionnaires and qualitative data from open-ended surveys and FGDs [25, 26]. This study was conducted alongside another study [24] that compared the reproducibility of the passing scores of modified Angoff and Ebel methods across judges, conducted in the Faculty of Dentistry, The National University of Malaysia (Universiti Kebangsaan Malaysia, UKM), a public university, in March 2024. The theory paper, consisting of one-best answer (OBA) and short-answer questions (SAQs) format for the final-year Final Professional Examination in the Comprehensive Dental Care (CDC) course conducted in 2022, was selected in this standard setting study. The CDC course is multidisciplinary, covering endodontics, operative, oral medicine and oral pathology, oral and maxillofacial surgery, public dental health, prosthodontics and periodontics.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eStudy participants and sampling\u003c/h2\u003e\n\u003cp\u003eThe participants, also called judges in the standard setting meetings, were selected based on the inclusion and exclusion criteria outlined in the previous study (Detailed in Ho et al., 2005 [24]). Purposive sampling method used in this study aimed to achieve maximum variation among the participants. Twelve judges participated in the standard setting meetings and were purposively sampled into two groups of six (Groups A and B), ensuring a matched gender, ethnic, years of experience, dental discipline, and academic position. Before the start of the study, the objectives and details were explained and informed consent was obtained from all participants.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eProcedure\u003c/h2\u003e\n\u003cp\u003eThis study was conducted during a three-day workshop, where, on the first day, judges received lectures on the theory of standard setting. They participated in practical exercises with example items to apply the modified Angoff [27-29] and Ebel [30, 31] methods. These exercises aimed to familiarise the judges with the standard setting method and processes. The workshop was led by the first author, TKH, a dental academic pursuing a PhD in dental education; NLAK, a language educationist and psychometrician; and MNAB, a medical educationist. NLAK and MNAB both possess extensive experience in standard setting protocols and meetings relevant to their respective fields.\u003c/p\u003e\n\u003cp\u003eFollowing the lectures and practice sessions, judges conducted standard setting meetings using the modified Angoff and Ebel methods in separate discussion rooms. For each method, judges completed two rounds of individual ratings (R1, R2), with feedback on ratings (mean, median, SD and overall passing score) reviewed before R2. A third round (R3) involved reviewing R2 ratings and participating in group discussion before finalising ratings. Judges also completed online feedback questionnaires at various stages and the workshop concluded with a focus group discussion (FGD). Figure 1 illustrates the study flow.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e(Insert Figure 1)\u003c/p\u003e\n\u003ch2\u003eQuantitative questionnaire study\u003c/h2\u003e\n\u003cp\u003eThe feedback questionnaires were adapted and modified from previous standard setting literature and studies designed to assess process validity and the thought processes of judges during meetings [7, 12, 32, 33]. The self-reported questionnaires comprised four parts aligned with the study\u0026rsquo;s objectives and were administered at different stages of the standard setting meeting. The questionnaires were created using Qualtrics XM (Provo, UT) and the online link was distributed to the judges during the meeting. They included a five-point Likert scale (strongly agree = 5, agree = 4, neutral = 3, disagree = 2 and strongly disagree = 1), multiple-choice questions, and open-ended questions. A summary of the feedback questionnaire content is provided in \u003cu\u003eSupplementary File 1\u003c/u\u003e.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe study comprised four parts: Part 1 collected demographic data and feedback on standard setting lectures. Parts 2 and 3 gathered identical feedback on the modified Angoff and Ebel methods after each rating round, using pseudonyms. On the final day, groups reviewed passing scores across methods. Part 4 collected final insights on the workshop and judges\u0026rsquo; experiences with the methods used before focus group discussions.\u003c/p\u003e\n\u003cp\u003eFace and content validation of the questionnaires and FGD questions was conducted by two healthcare educationists and a psychometrician (NLAK) with expertise in conducting standard setting meetings and questionnaire-related education research. They provided suggestions to enhance clarity and ensure alignment with the study objectives. A condensed workshop was pilot tested with dental academics at another institution, and their feedback further improved the questionnaires\u0026apos; readability and clarity.\u003c/p\u003e\n\u003ch2\u003eQualitative interview study\u003c/h2\u003e\n\u003cp\u003eAll judges were invited to participate in face-to-face FGDs to share their insights and experiences from the standard setting meetings. Since the standard setting procedure was conducted in a group setting, collecting data through FGDs allowed for interactive discussions, enabling judges to elaborate on their perspectives. This method also facilitated the collection of diverse viewpoints within a shorter timeframe. A semi-structured interview guide \u003cu\u003e(Supplementary File 2)\u0026nbsp;\u003c/u\u003ewas developed by authors in accordance with the guidelines for conducting [34]. It was then reviewed by the same experts who validated the feedback questionnaire. Minor amendments were made based on their comments to ensure that the questions aligned with the study objectives. The FGDs, conducted in English, were led by the first author (TKH), a PhD student trained in dental education and quantitative and qualitative research design, and a medical educationist (MNAB) with extensive educational qualitative research experience. The discussions were held in closed rooms with semicircle seating arrangements and lasted approximately 60 minutes. The semi-structured interview guide reviewed by the moderators beforehand to ensure consistency and standardisation across FGDs. Judges were assigned pseudonyms for confidentiality. Each FGDs began with a PowerPoint presentation summarising feedback on passing scores and responses from Questionnaire Part 4c. The discussions explored judges\u0026rsquo; confidence, thought processes during rating, and challenges encountered during decision-making at the standard setting meeting. Sessions were audio-recorded with encrypted devices, transcribed verbatim by a university-approved service, and verified by the moderators.\u003c/p\u003e\n\u003ch2\u003eData analysis\u003c/h2\u003e\n\u003cp\u003eFeedback from Parts 1 and 4 was analysed descriptively using frequencies and percentages for categorical variables, with bar charts generated in Microsoft Excel to visualise the data. The feedback questionnaires, which utilised a 5-point Likert scale (from 1 = strongly disagree to 5 = strongly agree), had their mean and standard deviation (SD) calculated. The non-parametric Mann\u0026ndash;Whitney U test was used to investigate the differences across the groups, whereas the Wilcoxon signed ranks test was employed to evaluate the differences across the standard setting methods using IBM SPSS (version 29; IBM Corp., Armonk, NY, USA). Non-parametric statistical tests were selected due to the small sample size and violation of the normality assumption. A p-value of \u0026lt;0.05 was considered statistically significant. Additionally, free-text comments from the feedback questionnaires were analysed descriptively.\u003c/p\u003e\n\u003cp\u003eThe authors organised, managed and analysed the FGD data using a shared Microsoft Excel spreadsheet [35]. For the qualitative analysis, they adhered to the six-phase framework of inductive thematic analysis, as outlined by Braun and Clarke and DeJonckheere et al. [36, 37]. The first author began by manually removing irrelevant data and focusing on the research objectives while becoming familiar with the data. The identified quotes and codes were transferred to Microsoft Excel spreadsheets for further analysis. The codes were grouped into categories based on the shared meaning or content. To ensure reliability and accuracy, the second and third authors independently reviewed the transcription and coding. Any discrepancies in coding were addressed through iterative discussions, leading to the development of themes through consensus. Throughout the analysis, the researchers continuously reflected on potential personal or external biases and assumptions to minimise their influence, thereby ensuring the rigour and trustworthiness of the qualitative analysis.\u003c/p\u003e\n\u003ch2\u003eEthical considerations\u003c/h2\u003e\n\u003cp\u003eEthics approval for this project was granted by the University of Manchester Research Ethics Committee (2023-17408-31608) and UKM Research Ethics Committee (JEP-2023-2040).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eParticipants were informed about the project three months in advance, allowing time to arrange prior commitments. Written informed consent was obtained, and pseudonyms were used to ensure confidentiality. Participation was voluntary, with assurances that feedback would not affect their professional evaluation or future career prospects.\u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n \u003ch2\u003eDemographic data\u003c/h2\u003e\n \u003cp\u003eA total of 12 participants were recruited as judges for the standard setting study and were evenly divided into two groups. The demographic characteristics of the participants (e.g., gender, ethnicity, years of teaching experience, discipline, and academic position) were identical to those reported in our earlier study [\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e].\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\n \u003ch2\u003eQuestionnaire data\u003c/h2\u003e\n \u003cp\u003eAll 12 judges participated in and completed the feedback questionnaires. Figure \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e presents the general feedback evaluation of the standard setting workshop. Overall, judges expressed positive feedback about the lectures and the workshop. The overall mean (SD) scores were 4.24 (0.54) for Questionnaire Part 1 (feedback after lectures) and 4.47 (0.36) for Part 4a (feedback after meetings). Item-level means (SD) for Groups A and B are provided in \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eSupplementary File 3\u003c/span\u003e. Mann\u0026ndash;Whitney U-test revealed no significant difference between Groups A and B for any item (p\u0026thinsp;\u0026gt;\u0026thinsp;0.05). The high values of Cronbach\u0026rsquo;s alpha (after lectures, alpha\u0026thinsp;=\u0026thinsp;0.96; after meetings, alpha\u0026thinsp;=\u0026thinsp;0.75) indicated strong reliability and internal consistency of the questionnaire items [\u003cspan class=\"CitationRef\"\u003e38\u003c/span\u003e].\u003c/p\u003e\n \u003cp\u003e(Insert Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e )\u003c/p\u003e\n \u003cp\u003eThe mean (SD) ratings of judges\u0026rsquo; responses to the standard setting process using the modified Angoff (Part 2) and Ebel (Part 3) methods after each round of ratings are documented in \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eSupplementary File 4\u003c/span\u003e. Wilcoxon signed-rank tests comparing feedback evaluations for each item from R1\u0026ndash;R3 between the two methods revealed no significant differences. Similarly, the Mann\u0026ndash;Whitney U-test demonstrated no differences in judges\u0026rsquo; responses across Group A and B for all items from R1\u0026ndash;R3 using either method. For the statement \u0026lsquo;I understood how to complete my rating\u0026rsquo;, the mean ratings increased from R1 to R3 for both modified Angoff (R1\u0026thinsp;=\u0026thinsp;4.25, R2\u0026thinsp;=\u0026thinsp;4.50 and R3\u0026thinsp;=\u0026thinsp;4.67) and Ebel (R1\u0026thinsp;=\u0026thinsp;4.33, R2\u0026thinsp;=\u0026thinsp;4.42 and R3\u0026thinsp;=\u0026thinsp;4.50) methods. Additionally, judges reported progressively greater confidence from R1 to R3. For the modified Angoff method, the mean confidence ratings increased to 3.58, 4.42 and 4.67 in R1, R2 and R3, respectively, while for the Ebel method, they increased to 3.58, 3.93 and 4.42 in R1, R2, and R3, respectively.\u003c/p\u003e\n \u003cp\u003eThe results from Questionnaire 4b on judges\u0026rsquo; reliance on information sources during standard setting are presented in Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e. Throughout the standard setting process, almost all judges relied moderately to heavily on the descriptions of the borderline groups (91.7%) and small group discussion (100%) for their judgement. All judges in Group A considered normative ratings from other judges when making their decisions. However, it was observed that two judges from Group B minimally relied on the normative rating from others in their evaluations.\u003c/p\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eJudges\u0026apos; responses regarding the reliance on the source of information during standard setting process (Part 4b)\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003ctd align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eGroup\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\" colspan=\"4\"\u003e\n \u003cp\u003eResponse, n (%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eNot at all\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eSlightly\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eModerately\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eHeavily\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e1. My previous experience taking the test\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (41.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e4. My own experience with student cohort\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e7. The descriptions of the borderline groups\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (41.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e10. Small group discussion\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e13. The normative information from other judges (eg: the ratings of other judges)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 (58.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003e(Insert Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e)\u003c/p\u003e\n \u003cp\u003eTable \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e presents the judges\u0026rsquo; feedback comparing the modified Angoff and Ebel methods in terms of defensibility, confidence to practise, group rating and cognitive demands (Part 4c). The judges also expressed comparable confidence in the defensibility of the passing score for both methods in the OBA and SAQ papers, with mean scores ranging from 3.67 to 3.92. In the OBA paper, the mean scores in confidence in the practice of modified Angoff and Ebel methods were 4.00 and 3.92, respectively. Conversely, in the SAQ paper, the mean scores for the modified Angoff and Ebel methods were 3.83 and 3.92, respectively. Regarding preferences in the OBA paper, seven judges (58.3%) preferred the Ebel method, four judges (33.3%) favoured the modified Angoff method and one judge had no preference (Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e). Conversely, preferences for the SAQ paper were evenly split, with 50% of the judges favouring the modified Angoff method and 50% favouring the Ebel method. Overall, the Mann\u0026ndash;Whitney U-test demonstrated no differences in judges\u0026rsquo; responses across Group A and B for all items in Part 4c.\u003c/p\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eJudges\u0026rsquo; responses on items comparing the modified Angoff and Ebel standard setting decision (Part 4c)\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eStatement\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eGroup\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003en\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"5\"\u003e\n \u003cp\u003eResponse, n (%)\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eMean\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003eStand.\u003c/p\u003e\n \u003cp\u003eDeviation\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" rowspan=\"2\"\u003e\n \u003cp\u003ep-value \u003csup\u003ea\u003c/sup\u003e\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eN\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eA\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSA\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e1. I am confident to practise the Modified Angoff method in OBA.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.632\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.632\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e8 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.603\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1.000\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e4. I am confident to practise the Modified Angoff method in SAQ.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.894\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.516\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.718\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.484\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e7. I am confident to practise the Ebel method in OBA.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.753\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.516\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 (58.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.669\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.206\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e10. I am confident to practise the Ebel method in SAQ.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (83.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.408\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.632\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e9 (75.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.515\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.598\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e13. I am convinced of the defensibility of the passing score using Modified Angoff method in OBA.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (100.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.000\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.753\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e9 (75.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.515\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.527\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e16. I am convinced of the defensibility of the passing score using Modified Angoff method in SAQ.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.983\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.548\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 (58.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.778\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.280\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e19. I am convinced of the defensibility of the passing score using Ebel method in OBA.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4.33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.516\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.548\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 (58.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.669\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.056\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e22. I am convinced of the defensibility of the passing score using Ebel method in SAQ.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.516\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.753\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e7 (58.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.622\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.715\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e25. I prefer individual rating to group rating in Modified Angoff method.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.753\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.894\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (25.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (44.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.08\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.793\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.733\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e28. I prefer individual rating to group rating in Ebel method.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1.095\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2.67\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.816\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (44.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e5 (44.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (8.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.937\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.604\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" rowspan=\"3\"\u003e\n \u003cp\u003e31. I feel Modified Angoff requires more cognitive thinking (thinking and reasoning process) than Ebel.\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eA\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e4 (66.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.837\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eB\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (33.3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1.378\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6 (50.0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e2 (16.7)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e3.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1.206\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e.169\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\" colspan=\"12\" style=\"width: 99.866%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eLikert\u0026rsquo;s Scale\u003c/strong\u003e: 1\u0026thinsp;=\u0026thinsp;Strongly disagree(SD), 2\u0026thinsp;=\u0026thinsp;Disagree(D), 3\u0026thinsp;=\u0026thinsp;Neutral(N), 4\u0026thinsp;=\u0026thinsp;Agree(A), 5\u0026thinsp;=\u0026thinsp;Strongly Agree(SA)\u003c/p\u003e\n \u003cp\u003e\u003csup\u003e\u003cstrong\u003ea\u003c/strong\u003e\u003c/sup\u003e\u003cstrong\u003e-Mann-Whitney U\u003c/strong\u003e (*) indicates a statistical significance (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05)\u0026nbsp;\u003c/p\u003e\u0026nbsp;\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003e(Insert Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e)\u003c/p\u003e\n \u003cp\u003e(Insert Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e)\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\n \u003ch2\u003eQualitative data\u003c/h2\u003e\n \u003cp\u003eAll judges attended the entire FGD sessions. The analyses in this section describe the data collected from the open-ended questions in the questionnaires and FGDs. The results suggest that much of the data from Group A was similar to those of Group B. Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e presents the themes and subthemes related to the judges\u0026rsquo; experiences and thought processes during the modified Angoff and Ebel standard setting meetings.\u003c/p\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003ctable id=\"Tab3\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eIdentified themes and subthemes related to judges\u0026rsquo; experiences and thought processes during modified Angoff and Ebel standard setting meetings\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eThemes\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eSubthemes\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eConfidence in standard setting\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1. Continuous vs. category scoring\u003c/p\u003e\n \u003cp\u003e2. Cognitive challenge\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eThe role of diverse subject matter experts in decision-making\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1. Expertise and content knowledge\u003c/p\u003e\n \u003cp\u003e2. Years of experience\u003c/p\u003e\n \u003cp\u003e3. Enhancing standard setting through group discussion\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eUnderstanding and familiarity in Course Learning Outcomes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eBarriers and facilitator in the standard setting process\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e1. Insufficient faculty resources and workload pressures\u003c/p\u003e\n \u003cp\u003e2. Training as a catalyst for effective standard setting\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n \u003cp\u003e\u003c/p\u003e\n \u003cp\u003e(Insert Table \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e)\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e\n \u003ch2\u003eTheme 1: Confidence in the standard setting\u003c/h2\u003e\n \u003cp\u003e\u003cstrong\u003eContinuous against category scoring.\u003c/strong\u003e The modified Angoff method involves continuous scoring requiring judges to estimate the probability of a borderline student answering correctly, with percentage ranging from 0 to 100. Judges described the percentage method in the modified Angoff approach as simplified, focused on the item-level, straightforward and readily comprehensible.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Easier to come up with the percentage\u0026rdquo;\u003c/em\u003e (P02).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;The decision-making\u003c/em\u003e process (modified Angoff) is \u003cem\u003emuch simplified and easier.\u0026rdquo;\u003c/em\u003e (P12).\u003c/p\u003e\n \u003cp\u003eThe judges expressed greater confidence in the modified Angoff ratings because they allowed them more control over probability decisions compared to the fixed probabilities in the Ebel method. They also found the continuous ratings in the modified Angoff method to be more practical and usable than the categorical ratings in the Ebel method.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;With modified Angoff, the percentage and probability were decided by the panel, whereas in Ebel, it is already fixed\u0026hellip;so I think the panel has more control over the probability in modified Angoff over Ebel, as Ebel already fixed\u0026rdquo;\u003c/em\u003e (P05).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;\u0026hellip;continuous rating can be more representative of what you want to give rather than categorical rating. Ebel is categorised, it may not truly reflect the marks or the percentage that you want to give\u0026rdquo;\u003c/em\u003e (P13).\u003c/p\u003e\n \u003cp\u003eFor polytomous items (SAQs), many judges believed that the modified Angoff method permits individual ratings for each sub-question, effectively accounting for their varying levels of difficulty.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(modified Angoff in SAQ) It actually reads each sub-question depending on the complexity\u0026hellip; one sub-question might be simple and then another sub-question maybe difficult question, in Ebel, you can\u0026rsquo;t really rate as the complexity of each sub question.\u0026rdquo;\u003c/em\u003e (P12)\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(modified Angoff in SAQ) We can actually assess the sub-questions and then we give ratings for the three sub questions then composite rating.\u0026rdquo;\u003c/em\u003e (P13)\u003c/p\u003e\n \u003cp\u003eIn the Ebel method, the judges conducted category judgements at the item-level, evaluating each item based on its difficulty and relevance to the test [\u003cspan class=\"CitationRef\"\u003e30\u003c/span\u003e]. Judges expressed confidence in the Ebel method because its structured framework made the decision-making process more objective. They elaborated that the Ebel method provides clear guidance for categorising items based on the levels of difficulty and relevance within the 3x3 Ebel grid. This guideline allows judges to accurately determine the relevance and difficulty of test items in alignment with the course learning outcomes (CLOs).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Ebel you identify the difficulty and then the relevancy and then it\u0026rsquo;s more objective, you know in which category you are actually rating it. I prefer Ebel in OBA\u0026rdquo;\u003c/em\u003e (P13).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Ebel requires less effort thought to it, it gives us some direction rather than Angoff\u0026rdquo;\u003c/em\u003e (P14).\u003c/p\u003e\n \u003cp\u003eIn the context of SAQs, judges preferred the Ebel method due to its clear and structured guidance. They found the process of quantifying ratings with numerical values to be particularly challenging.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;In SAQ questions, I personally chose Ebel because it\u0026rsquo;s kind of hard for me to rate the number because, you know, if the questions have different difficulty and I don\u0026rsquo;t know how to sum up with the number. So that\u0026rsquo;s why I actually feel that Ebel gives me some direction and sort of helps me to give the rating that I am supposed to give based on the three questions\u0026rdquo;\u003c/em\u003e (P14).\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003eCognitive challenges.\u003c/strong\u003e Several judges found the broad range of continuous scoring in the modified Angoff method to be subjective, perceiving it as largely based on individual perceptions. They also found it challenging to accurately assign probability.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Angoff is an arbitrary number from zero to hundred\u0026hellip;more of a guesswork that relies on personal perception\u0026quot;\u003c/em\u003e (P06).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Because we don\u0026rsquo;t have any guidance for the modified Angoff when we were to give the percentage for the borderline student, and we just based it on our gut feeling. Like, you know, all this thing maybe is 75% of the students can answer, things like that\u0026rdquo;\u003c/em\u003e (P11).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(modified Angoff) based on perceived marks, so it\u0026rsquo;s very difficult to like rate the passing marks\u0026rdquo;\u003c/em\u003e (P12).\u003c/p\u003e\n \u003cp\u003eIn the Ebel method, ratings are determined using a 3x3 Ebel grid that categorises difficulty and relevancy. However, judges mentioned their confusion in determining the relevancy category and noted inconsistencies in their ratings across nine options during the rating process. A judge referred to the percentage of borderline candidates passing in each grid when making decisions in the Ebe method.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(Ebel) It is difficult to decide the relevancy of the question within the framework of the course learning outcome\u0026rdquo;\u003c/em\u003e (P05).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(Ebel) You have to think quite far and deep to decide on that. So I think it takes more time to decide. From the exercises, I found that I couldn\u0026rsquo;t be consistent in my decisions. We were more or less agreeable on the level of difficulty, but we were all in conflict when it came to the relevance category, essential and important\u0026rdquo;\u003c/em\u003e (P16).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;(Ebel) When I come to a question that I want to decide if it\u0026rsquo;s an easy, moderate or difficult question, sometimes I cannot say. So I select that first, then I look at the percentage in that category to guide me in my selection\u0026rdquo;\u003c/em\u003e (P15).\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e\n \u003ch2\u003eTheme 2: The role of diverse subject matter experts in decision-making\u003c/h2\u003e\n \u003cp\u003e\u003cstrong\u003eExpertise and content knowledge.\u003c/strong\u003e Some judges observed a tendency to establish higher standards for their own subject areas within the modified Angoff framework. Conversely, others expressed confidence in gauging the level of borderline students on items related to their specialties. These judges argued that active engagement in teaching those subjects significantly enhances their ability to make fair and informed judgements.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;I tend to set higher percentage when it comes to my subject, even though I try, because I\u0026rsquo;m teaching a very easy subject, it\u0026rsquo;s a very easy standard for me\u0026hellip; for example, I put 90% and the rest of the group only puts 70, 75. So to me, that was a challenge when it comes to your own question\u0026rdquo;\u003c/em\u003e (P04).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;In modified Angoff, it\u0026rsquo;s my discipline so I will expect the students to be able to answer so the scoring will be higher\u0026rdquo;\u003c/em\u003e (P11).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;But if I were to do it in my own speciality, I feel that I am more confident to practise.\u0026rdquo;\u003c/em\u003e (P13)\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;\u0026hellip;especially if that question related to my specialty\u0026hellip;I can actually gauge the probability of the Angoff rating because we actually do the supervision in the clinic\u0026hellip; based on our experience, we can make the judgement on the score where it should be.\u0026rdquo;\u003c/em\u003e (P05)\u003c/p\u003e\n \u003c/div\u003e\n \u003cp\u003eFrom the FGDs, judges acknowledged that they encountered challenges in making judgements on unfamiliar topics, especially when they were not directly involved in the teaching sessions for those subjects. This lack of direct teaching engagement with the subject matter introduced uncertainty in their judgement on the items. Therefore, the judges suggested including a mix of disciplines in the panel to ensure fairer judgement.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;I\u0026rsquo;m struggling to decide the relevancy of the subject and the difficulty level of that particular question, if it\u0026rsquo;s outside my specialty. I am not sure the depth of the subjects that have been taught; how much emphasis has been put forward to the students about the subject. For example, oral cancer, and we think it\u0026rsquo;s a very important subject but I don\u0026rsquo;t have any information for how many lectures have been taught in that particular subject and even though we can see from the course learning outcome but it\u0026rsquo;s a very vague statement\u0026rdquo;\u003c/em\u003e (P05).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;\u0026hellip;in short, I think it is good to have a various background of specialty in the group\u0026rdquo;\u003c/em\u003e (P01).\u003c/p\u003e\n \u003c/div\u003e\n \u003cp\u003e\u003cstrong\u003eYears of experience.\u003c/strong\u003e Judges believed that senior staff members, particularly those who have experience in item writing and marking papers, may find the modified Angoff method relatively straightforward. However, some judges emphasised the value of having a variety of levels of experience for a more balanced approach to standard setting.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;It plays a big role if you already have many years mark the student\u0026rsquo;s paper. So you can gauge which questions that borderline students can answer, the senior lecturer will probably think that modified Angoff will work out well\u0026rdquo;\u003c/em\u003e (P05).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;If all the panels are senior, it\u0026rsquo;s more valid to use modified Angoff but if we have like two groups of panels, like junior and senior, I think Ebel is more accurate as it is more defined on the level of the relevancy and difficulty\u0026rdquo;\u003c/em\u003e (P03).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Good to have different combination of panels, however having said even same panel, same year we have different perceptions of rating.\u0026rdquo;\u003c/em\u003e (P14)\u003c/p\u003e\n \u003c/div\u003e\n \u003cp\u003e\u003cstrong\u003eEnhancing standard setting through group discussion.\u003c/strong\u003e The judges concurred that holding a group discussion before finalising their judgement provided an opportunity to gain insights from other judges\u0026rsquo; perspectives and understand the rationale behind their ratings. This process reinforced the standard for borderline students and helped refine their own scoring.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Good to listen to other judges\u0026rsquo; justification as to why they place the relevancy. Hearing other judges\u0026rsquo; justification for their score helps in my scoring. Keep on having to tell myself that I have to think as borderline student\u0026rdquo;\u003c/em\u003e (P02).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;The group discussion helps to understand better the feedback of other judges\u0026apos; scores and this affects the next ratings\u0026rdquo;\u003c/em\u003e (P12).\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec16\" class=\"Section2\"\u003e\n \u003ch2\u003eTheme 3: Understanding and familiarity in CLO\u003c/h2\u003e\n \u003cp\u003eMany judges emphasised the importance of understanding the CLO and aligning their judgements with it to ensure fairness and reduce variations among judges. However, in a multidisciplinary course like this, some judges face cognitive challenges due to the extensive learning outcomes.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;And I feel that it\u0026rsquo;s not about the years of experience, it\u0026rsquo;s not about your specialty, but I think the more important bit is the understanding of the course learning outcome or programme learning outcome\u0026hellip;if you understand it and you have a good grasp of what is it that we want to achieve in the programme or in the course, I think that it will reduce the different\u0026rdquo;\u003c/em\u003e (P16).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;The course that has so many learning outcomes, like CDC, sometimes I might overload some of the learning outcomes, so I tend to be a little bit difficult at certain point, to cover all, then I have to go back to the learning outcomes\u0026rdquo;\u003c/em\u003e (P03).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;We go through the course learning outcome first, so that we are aware of the learning outcomes, so that easier when we want to decide whether it\u0026rsquo;s essential or important or even when we want to decide in modified Angoff, which question is in which CLO and what is the level, you know. Some of us, I myself I also cannot remember, like for the CDC course, we have 16 learning outcomes\u0026rdquo;\u003c/em\u003e (P11).\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec17\" class=\"Section2\"\u003e\n \u003ch2\u003eTheme 4: Barriers and facilitator in the standard setting process\u003c/h2\u003e\n \u003cp\u003e\u003cstrong\u003eInsufficient faculty resources and workload pressures.\u003c/strong\u003e Judges identified several challenges and practical issues in implementing the modified Angoff or Ebel standard setting in the dental faculty, including a limited number of teaching academics, time constraints and increased workload. They acknowledged that having sufficient judges is crucial for reliable results. However, some judges reported feeling stressed due to the iteration process and the added burden of their existing responsibilities.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Looking at the number of specialists here in UKM, I think some of the specialties maybe hard to get that good combination of panel, because we have a limited number like oral medicine and pathology\u0026rdquo;\u003c/em\u003e (P14).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Practically how it is going to be carried out, as we already took so many hours just to discuss 20 questions, and so many days, who will sit on the panels practically in terms of time, as ideal as this. I don\u0026rsquo;t think we can get all 5 or 4 periodontists to sit together from Monday to Friday to set standard, as everyone has clinic. When I am free you are not free, that is how the timetable goes\u0026rdquo;\u003c/em\u003e (P06).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;It is a very tiring process, as you have to go through many rounds to get to, not really consensus, but at least not so much of standard deviation\u0026rdquo;\u003c/em\u003e (P02).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;Even without standard setting, we are already quite burdened\u0026rdquo;\u003c/em\u003e (P04).\u003c/p\u003e\n \u003c/div\u003e\n \u003cp\u003e\u003cstrong\u003eTraining as a catalyst for effective standard setting\u003c/strong\u003e. The judges expressed that their understanding and confidence in applying the ratings increased with each round of the process. They also emphasised the importance of adequate training and practice before the actual standard setting process to boost confidence and effectiveness.\u003c/p\u003e\n \u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;It is clearer now after the second round. My thoughts are clearer\u0026hellip;I\u0026apos;m more confident after Round 3\u0026rdquo;\u003c/em\u003e (P13).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;If all of your methods are well trained, I don\u0026rsquo;t think there\u0026rsquo;s any difference between Modified Angoff and Ebel\u0026rdquo;\u003c/em\u003e (P13).\u003c/p\u003e\n \u003cp\u003e\u003cem\u003e\u0026ldquo;If the panels are well trained it will overcome that problem and making Angoff more defensible in overall rating as it has more flexibility\u0026hellip;the training is important to make sure that everyone understands the process.\u0026rdquo;\u003c/em\u003e (P14)\u003c/p\u003e\n \u003c/div\u003e\n\u003c/div\u003e"},{"header":"Discussion and integrated consideration of the findings","content":"\u003cp\u003eThis mixed-methods study examined judges’ feedback, experiences and perspectives on applying the performance standard across different methods. The feedback evaluation from judges regarding the workshop and meetings showed positive responses towards the standard setting procedure (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e), which showed similarity with other studies in healthcare professionals' education [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan additionalcitationids=\"CR40 CR41\" citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e–\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e]. Several authors have previously suggested formats for feedback forms that judges can complete after the meetings to evaluate the effectiveness of the workshop and meetings[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. However, feedback forms are typically modified as needed to include the questions deemed necessary. In this study, in addition to general feedback, the judges were asked to complete a feedback form after each round of ratings. The comparison of the standard setting process in R1, R2, and R3 across the modified Angoff and Ebel methods within the same group and between Groups A and B showed no significant differences. Judges in both groups exhibited similar levels of understanding and confidence in making standard setting ratings and in the defensibility of the passing scores using both the modified Angoff and Ebel methods. The qualitative data revealed that both groups have similar experiences and perspectives regarding the standard setting meetings. However, their thought processes in applying the modified Angoff and Ebel methods differed.\u003c/p\u003e\u003cp\u003eThese findings served as an internal validation of procedural validity in setting the passing scores for the examination [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. The quality and duration of the training have been identified as factors affecting judges’ confidence and reliability levels [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e]. In this study, judges rated time allocation highly, with mean scores above 4.0 for both methods. They received a full day of lectures and one hour per round to rate 30 items. In contrast, a Canadian Ebel study provided only 20 minutes of training on standard setting and three hours to rate 203 MCQs, yielding low agreement (Fleiss’ kappa = 0.11) and suggesting that time constraints undermined procedural validity [\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e], as judges were under significant pressure to complete the ratings within the allotted time.\u003c/p\u003e\u003cp\u003eAlthough responses on the feedback questionnaires meetings (Part 2 \u0026amp; Part 3) from R1 to R3 for the modified Angoff and Ebel methods were not significantly different, FGDs revealed significant differences in judges’ thought processes when using the two methods. In FGDs, judges described the numerical ratings in the modified Angoff method as straightforward and easy. Nonetheless, some judges criticised the numerical ratings as being perception-based and reliant on guesswork. Judges have faced cognitive challenges in accurately estimating borderline candidates using the Angoff method, a finding also reported in previous studies [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e]. Some judges appreciated the structured framework of relevancy and difficulty levels in the Ebel method, which provided valuable guidance and boosted their confidence in the defensibility of the ratings. However, a few judges felt they had less control over the final percentage outcome compared to the modified Angoff, as it was predetermined in the Ebel grid. An earlier study by Hudges (1983) in medical examinations reported similar findings, noting that judges considered the Angoff, Guerin, and National Board of Medical Examination (Rasch model) methods feasible, but not the Ebel method. They lacked confidence in Ebel due to difficulties judging success rates from hypothetical relevancy and difficulty levels and uncertainty about how changes in these factors affected their overall judgement of minimal competent candidates [\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e]. Dual process theory may be used to suggest how judges think during the standard setting process [\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e]. Type 1 processes are intuitive and automatic in nature, associated with fast, parallel, and biased responses. These processes are automatic and experience-based in decision-making. In contrast, Type 2 processes are deliberative and analytical, relating to slower, serial, normative responses that are controlled and consequential in nature [\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e]. The modified Angoff method is claimed to be easy, yet subjective, and participants have been observed to shift toward Type 1 thinking by simply assigning a percentage after viewing the item. In contrast, the categorical framework in the Ebel method may guide judges to engage in Type 2 processes during their judgments, as they frequently refer to CLOs to determine the relevancy category.\u003c/p\u003e\u003cp\u003eThe purposive sampling method aimed to ensure diversity among the judges so they were broadly representative of various backgrounds, thereby making the recommended passing standard reasonable and valid [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan additionalcitationids=\"CR49\" citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e–\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e]. Although the content was at the undergraduate dental degree level and all judges were appropriately qualified subject matter experts, FGDs revealed that some struggled to judge topics outside their direct teaching areas. Judges also differed on how their specialty influenced standard setting, some believed they set higher standards in their own field, while others felt their expertise enabled more accurate evaluation of borderline students within their specialty. Previous studies have shown that judges’ expertise in specific subject areas did not significantly influence standard setting outcomes using the Angoff method, regardless of the content subspecialty [\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e, \u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e, \u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e]. However, many studies showed that judges tended to set higher standards for the items that they knew the correct answers to and lower standards for those they did not [\u003cspan additionalcitationids=\"CR54\" citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e–\u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e55\u003c/span\u003e]. Further FGD data in this study revealed that judges found it challenging in rating unfamiliar items. Clauser et al. (2017) found that judges tended to rate unfamiliar items as more difficult, leading to lower Angoff ratings and ultimately lower passing scores [\u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e]. The impact of unfamiliar items on the final passing scores depends on the ratio of unfamiliar items in the exam. Excluding unfamiliar items from the standard setting process may help reduce systematic errors, nevertheless, if judges from a particular panel are more likely to mark items as unfamiliar, omitting their input may compromise the intended diversity and variation among the panels. The authors suggested that alternative approaches, such as the Bookmark method or an Item Response Theory (IRT)-based model should be considered depending on the impact of unfamiliar items and the context of the particular exam [\u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eTo reduce the ratings’ variability and gap between judges, group discussions before the final round were aimed to share opinions and justification for their ratings. Feedback questionnaires indicated that most judges preferred group ratings over individual ratings, with similar preferences in Groups A and B. FGDs revealed that judges found group discussions helpful for refining their ratings. Yeates et al. (2019) observed that judges often conform to group ratings to minimise variations. However, some resist conformity and, instead of engaging in debate, adjust their ratings away from the group mean when they disagree [\u003cspan citationid=\"CR57\" class=\"CitationRef\"\u003e57\u003c/span\u003e]. In standard setting, there is no definitive gold standard for judging the accuracy and validity of the outcomes, multiple sources of validity evidence must be considered. While high inter-judge agreement is desirable but such agreement without critical evaluation of opinions maybe is a sign of groupthink. The Groupthink Model was first described by Irving Janis in 1972 as a framework for understanding decision-making in highly cohesive groups. In such groups, members prioritise agreement and harmony over critical evaluation, which can potentially lead to poor decision-making [\u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e58\u003c/span\u003e]. Groupthink is typically characterised by overconfidence in decisions, a stereo-typed perception of opposing views, ignoring alternatives and pressure towards uniformity by marginalising dissenting opinions.\u003c/p\u003e\u003cp\u003eThe FGD and feedback questionnaire revealed that judges acknowledged the importance of aligning the CLO with the test’s purpose when making standard setting judgements. The feedback questionnaire indicated that judges referred to the CLO during the standard setting process. Nevertheless, some judges reported difficulties in recalling and appropriately applying the CLO when assigning ratings. According to the \u003cem\u003eStandards for Educational and Psychological Testing\u003c/em\u003e, the process of establishing performance standards in an assessment should be closely aligned with content standards [\u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e59\u003c/span\u003e]. Learning outcomes are measurable achievements defined by content standards, indicating what students should know and be able to do. These outcomes need to be clear and specific to effectively guide teachers and students in their teaching, learning and assessment activities. The alignment of performance standards with the CLO is crucial for the reliability of the ratings, as it helps judges set passing scores based on the expected performance standards. The majority (91.7%) of our judges relied moderately to heavily on the descriptions of borderline groups when making judgments. A mixed-methods study by Ferdous and Buckendahl (2013) found that understanding on the performance level descriptors helped judges make consistent item ratings in the English and Mathematics examinations [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. In contrast, judges from Giraud and Impara (2005) and McGinty (2005) admitted that their cut-score estimations were influenced by external factors. They considered political, social, and economic factors when establishing the standards for the English test and high school graduation test [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. The use of performance level descriptors is an important indicator of a criterion-referenced approach in high-stakes examinations for healthcare professionals, as incorrect pass-fail decision can have significant implications for patient safety.\u003c/p\u003e\u003cp\u003eJudges noted several barriers to implementing standard setting practices in the dental faculty, including a limited number of staff in a single discipline, time constraints and a high workload. Participants demonstrated improved mean ratings for understanding and confidence with each round of rating. They emphasised that adequate training in standard setting practices can enhance the reliability of the proposed passing scores. This finding aligns with other studies on the implementation of standard setting practices for national assessments in the Republic of Trinidad and Tobago [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Resources such as appointing subject matter experts, providing expertise to conduct standard setting meetings, and offering adequate training for staff members are important factors to consider when implementing standard setting practise.\u003c/p\u003e\u003ch2\u003eImplications and recommendations\u003c/h2\u003e\u003cp\u003eOur research is the first study to investigate the differences in the standard setting process from R1 to R3 between the modified Angoff and Ebel methods in dental education. It also explored the thought processes and factors that boosted their confidence of dental academics in the decision-making in these methods. The findings have several implications for standard setting practise. The standard setting process and feedback questionnaires provide guidelines on documenting procedural validity evidence for future standard setting studies in dental education. Besides, judges provided their perspectives and the challenges they face when making ratings. These considerations should be carefully addressed in the planning and choice of methods used for future standard setting workshops and meetings in dental education.\u003c/p\u003e\u003cp\u003eFor future standard setting workshops, we suggest appointing judges from varied backgrounds, focusing on the borderline descriptors for each CLO and promoting group discussions to achieve more credible results. Judges’ insights can improve the efficiency of future workshops and produce more credible ratings. The standard setting protocol and locally gathered data can serve as an evidence-based evaluation of standard setting activities, offering recommendations for dental assessment policy implementation. Collaborating with various stakeholders and pooling resources from multiple dental schools could facilitate the implementation of a standard setting for national dental licencing examinations.\u003c/p\u003e\u003ch2\u003eLimitations of this study\u003c/h2\u003e\u003cp\u003eThis mixed-methods study combines extensive feedback evaluation forms from each round of ratings and FGDs to provide an in-depth evaluation of the procedural validity and the thought process of judges. However, this study focused on judges from one institution in Malaysia, limiting its applicability to other settings. Along with FDG, other data collection methods, such as observational studies, could be applied to examine group dynamics, specifically behaviours and communication in achieving group consensus. Individual in-depth interviews may provide more insights into other factors that influence judges to conform to the ratings.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe standard setting workshop received positive feedback and responses from judges regarding the standard setting process. Judges reported increased confidence and understanding following iterative rounds and group discussions. This study highlighted the strengths and challenges of the modified Angoff and Ebel methods in the standard setting for theory-based dental assessments. The modified Angoff method was straightforward and easy; it gave judges more control, but it provided a subjective framework, relying on intuition and experience in setting the Angoff ratings. In contrast, the Ebel method offered a structured, objective framework, but judges felt they had less control, found the process more time-consuming, and some referred to the probability that borderline candidates would answer correctly within each category in the Ebel grid to determine the difficulty level. These findings provide insights into standard setting in dental education and emphasis the importance of structured training to enhance procedural validity.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eFGD: focus group discussion; SAQ: short answer questions; OBA: one-best answer; CLO: course learning outcome\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eClinical trial number\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003eAcknowledgments\u003c/p\u003e\n\u003cp\u003eWe would like to thank Professor Dr Noor Lide Abu Kassim (NLAB, a language educationist and psychometrician) from Kulliyyah of Education, International Islamic University Malaysia, and Dr Mohd Nasri Bin Awang Besar (MNAB, a medical educationist)\u0026nbsp;from Medical\u0026nbsp;Education Department, the Faculty of Medicine, Universiti Kebangsaan Malaysia, for their expertise and contributions to the standard setting workshop. We also want to thank all the participants for their time and involvement in this research.\u003c/p\u003e\n\u003cp\u003eAuthor contribution\u003c/p\u003e\n\u003cp\u003eTKH contributed to the conception of the research idea, design, collection of data, performing quantitative and qualitative data analysis and writing the manuscript.\u003c/p\u003e\n\u003cp\u003eLO contributed to formulating the research idea, interpreting quantitative and qualitative data, and critically reviewing and editing the manuscript.\u003c/p\u003e\n\u003cp\u003eRVR contributed to formulating the research idea, interpreting quantitative and qualitative data, and critically reviewing and editing the manuscript.\u003c/p\u003e\n\u003cp\u003eAll authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis study is part of the research conducted for TKH\u0026rsquo;s PhD project at The University of Manchester, which is supported by student research fees.\u003c/p\u003e\n\u003cp\u003eData availability\u003c/p\u003e\n\u003cp\u003eThe transcripts and datasets collected and/or analysed during the current study are not publicly available due to the confidentiality of the data but are available from the first author, Ting Khee Ho on request.\u003c/p\u003e\n\u003cp\u003eDeclarations\u003c/p\u003e\n\u003cp\u003eEthical approval\u003c/p\u003e\n\u003cp\u003eThe study adhered to the relevant guidelines and regulations as detailed in the Declaration of Helsinki. Study approval was granted jointly by the ethics committees of The University of Manchester Research Ethics Committee (2023-17408-31608) the National University of Malaysia (Universiti Kebangsaan Malaysia)(JEP-2023-2040).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent to participate\u003c/p\u003e\n\u003cp\u003eAll participants have provided informed consent.\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003eAll participants have consented to publication and using anonymised data without reference to individuals.\u003c/p\u003e\n\u003cp\u003eCompeting interests\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConflict of interests\u003c/p\u003e\n\u003cp\u003eThe authors declare no conflict of interest.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eYudkowsky R, Park YS, Downing SM. Assessment in Health Professions Education 2nd edn: Routledge New York, NY; 2019.\u003c/li\u003e\n\u003cli\u003eYudkowsky R, Downing SM, Tekian A. Standard setting. In: Assessment in Health Professions Education. edn. Edited by Yudkowsky R, Park YS, Downing SM: Routledge New York, NY; 2019: 119-48.\u003c/li\u003e\n\u003cli\u003eGarcia FF, Casarin M, Silva FH, Muniz F. Association between history of academic failure during dental school and self-reported depression, anxiety, and stress: A retrospective cohort. Journal of Dental Education 2023; 87(11):1502-11.\u003c/li\u003e\n\u003cli\u003eKane MT. Validating the performance standards associated with passing scores. Review of Educational Research 1994; 64(3):425-61.\u003c/li\u003e\n\u003cli\u003ePant HA, Rupp AA, Tiffin-Richards SP, K\u0026ouml;ller O. Validity issues in standard-setting studies. Studies in Educational Evaluation 2009; 35(2):95-101.\u003c/li\u003e\n\u003cli\u003ePitoniak MJ, Morgan DL. Setting and validating cut scores for tests. In: Handbook on measurement, assessment, and evaluation in higher education. 2nd edn. Edited by Secolsky C, Denison DB. New York: Routledge; 2012: 343-65.\u003c/li\u003e\n\u003cli\u003ePitoniak MJ. Standard setting methods for complex licensure examinations. Doctoral Dissertations University of Massachusetts Amherst; 2003.\u003c/li\u003e\n\u003cli\u003eHambleton R, Pitoniak MJ, Copella JM. Essential steps in setting performance standards on educational tests and strategies for assessing the reliability of results. In: Setting Performance Standards : Foundations, Methods, and Innovations. 2nd edition edn. Edited by Cizek GJ. New York \u0026amp; UK: Routledge; 2012: 47-76.\u003c/li\u003e\n\u003cli\u003eYim MK, Shin S. Using the angoff method to set a standard on mock exams for the korean nursing licensing examination. Journal of Educational Evaluation for Health Professions 2020; 17:14.\u003c/li\u003e\n\u003cli\u003eKampa N, Wagner H, K\u0026ouml;ller O. The standard setting process: Validating interpretations of stakeholders. Large-scale Assessments in Education 2019; 7(1):3.\u003c/li\u003e\n\u003cli\u003eSkorupski WP, Hambleton RK. What are panelists thinking when they participate in standard-setting studies? Applied Measurement in Education 2005; 18(3):233-56.\u003c/li\u003e\n\u003cli\u003eFerdous AA, Buckendahl CW. Evaluating panelists\u0026apos; standard setting perceptions in a developing nation. International Journal of Testing 2013; 13(1):4-18.\u003c/li\u003e\n\u003cli\u003eDe Lisle J. Installing a system of performance standards for national assessments in the republic of trinidad and tobago: Issues and challenges. Applied Measurement in Education 2015; 28(4):308-29.\u003c/li\u003e\n\u003cli\u003eGiraud G, Impara JC, Plake BS. Teachers\u0026apos; conceptions of the target examinee in angoff standard setting. Applied Measurement in Education 2005; 18(3):223-32.\u003c/li\u003e\n\u003cli\u003eHein SF, Skaggs G. Conceptualizing the classroom of target students: A qualitative investigation of panelists\u0026apos; experiences during standard setting. Educational Measurement, Issues and Practice 2010; 29(2):36-44.\u003c/li\u003e\n\u003cli\u003eGiraud G, Impara JC. Making the cut: The cut score setting process in a public school district. Applied Measurement in Education 2005; 18(3):289-312.\u003c/li\u003e\n\u003cli\u003eMcGinty D. Illuminating the\u0026quot; black box\u0026quot; of standard setting: An exploratory qualitative study. Applied Measurement in Education 2005; 18(3):269-87.\u003c/li\u003e\n\u003cli\u003eHein SF, Skaggs GE. A qualitative investigation of panelists\u0026apos; experiences of standard setting using two variations of the bookmark method. Applied Measurement in Education 2009; 22(3):207-28.\u003c/li\u003e\n\u003cli\u003eSchoonheim-Klein M, Muijtjens A, Habets L, Manogue M, Van Der Vleuten C, Van Der Velden U. Who will pass the dental osce? Comparison of the angoff and the borderline regression standard setting methods. European Journal of Dental Education 2009; 13(3):162-71.\u003c/li\u003e\n\u003cli\u003eDalum J, Christidis N, Myrberg IH, Karlgren K, Leanderson C, Englund GS. Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists. European Journal of Dental Education 2023; 27(3):640-49.\u003c/li\u003e\n\u003cli\u003eAbd-Rahman ANA, Baharuddin IH, Abu-Hassan MI, Davies SJ. A comparison of different standard-setting methods for professional qualifying dental examination. Journal of Dental Education 2021; 85(7):1210-16.\u003c/li\u003e\n\u003cli\u003eYousef MK, Alshawwa LA, Farsi JM, Tekian A, Soo PY. Determining defensible cut-off scores for dental courses. European Journal of Dental Education 2020; 24(2):186-92.\u003c/li\u003e\n\u003cli\u003eKim J, Yang JS. How to improve reliability of cut‐off scores in dental competency exam: A comparison of rating methods in standard setting. European Journal of Dental Education 2020; 24(4):734-40.\u003c/li\u003e\n\u003cli\u003eHo TK, Abu Kassim NL, O\u0026apos;Malley L, Roudsari RV. Standard setting for dental knowledge tests: Reproducibility of the modified angoff and ebel method across judges 2025; [Manuscript submitted for publication].\u003c/li\u003e\n\u003cli\u003eYounas A, F\u0026agrave;bregues S, Creswell JW. Generating metainferences in mixed methods research: A worked example in convergent mixed methods designs. Methodological Innovations 2023; 16(3):276-91.\u003c/li\u003e\n\u003cli\u003eCreswell JW, Plano Clark VL. Designing and conducting mixed methods research, Third Edition. edn. Thousand Oaks, California: SAGE; 2018.\u003c/li\u003e\n\u003cli\u003eCizek, Bunch. The angoff method and angoff variations. In: Standard setting. edn. Edited by Cizek GJ, Bunch MB. Thousand Oaks, California: SAGE Publications, Inc.; 2007.\u003c/li\u003e\n\u003cli\u003eAngoff WH. Norms, scales, and equivalent scores, vol. 2, 2nd edn: Washington, DC: American Council on Education; 1971.\u003c/li\u003e\n\u003cli\u003eHambleton RK, Plake BS. Using an extended angoff procedure to set standards on complex performance assessments. Applied Measurement in Education 1995; 8(1):41-55.\u003c/li\u003e\n\u003cli\u003eEbel RL. Essentials of educational measurement. Englewood Cliffs, N.J.: Prentice-Hall; 1972.\u003c/li\u003e\n\u003cli\u003eHomer M, Darling J, Pell G. Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality. Assessment and Evaluation in Higher Education 2012; 37(7):787-804.\u003c/li\u003e\n\u003cli\u003eCizek GJ, Bunch MB, Koons H. Setting performance standards: Contemporary methods. Educational Measurement: Issues and Practice 2005; 23(4):31-31.\u003c/li\u003e\n\u003cli\u003eCizek GJ. The forms and functions of evaluations in the standard setting process. In: Setting Performance Standards : Foundations, Methods, and Innovations. 2nd edn. Edited by Cizek GJ. New York \u0026amp; UK: Routledge; 2012: 165-78.\u003c/li\u003e\n\u003cli\u003eKrueger RA, Casey MA. Focus Groups : A Practical Guide for Applied Research, 5th edition. edn. Thousand Oaks, California: SAGE; 2015.\u003c/li\u003e\n\u003cli\u003eBrookfield C. Using Microsoft Excel for Social Research. Los Angeles: SAGE; 2021.\u003c/li\u003e\n\u003cli\u003eBraun V, Clarke V. Thematic Analysis: A Practical Guide: SAGE Publications; 2021.\u003c/li\u003e\n\u003cli\u003eDeJonckheere M, Vaughn LM, James TG, Schondelmeyer AC. Qualitative thematic analysis in a mixed methods study: Guidelines and considerations for integration. Journal of Mixed Methods Research 2024; 18(3):258-69.\u003c/li\u003e\n\u003cli\u003eTaber KS. The use of cronbach\u0026rsquo;s alpha when developing and reporting research instruments in science education. Research in Science Education 2017; 48(6):1273-96.\u003c/li\u003e\n\u003cli\u003eYim M. Comparison of results between modified-angoff and bookmark methods for estimating cut score of the Korean medical licensing examination. Korean J Med Educ 2018; 30(4):347-57.\u003c/li\u003e\n\u003cli\u003eYousefi Afrashteh M. Comparison of the validity of bookmark and angoff standard setting methods in medical performance tests. BMC Medical Education 2021; 21(1).\u003c/li\u003e\n\u003cli\u003eKang Y. Evaluating the cutoff score of the advanced practice nurse certification examination in korea. Nurse Educ Pract 2022; 63:103407.\u003c/li\u003e\n\u003cli\u003ePark J, Yim MK, Kim NJ, Ahn DS, Kim YM. Similarity of the cut score in test sets with different item amounts using the modified angoff, modified ebel, and hofstee standard-setting methods for the Korean medical licensing examination. Journal of Educational Evaluation for Health Professions 2020; 17:28.\u003c/li\u003e\n\u003cli\u003ePlake BS, Impara JC, Irwin PM. Consistency of angoff-based predictions of item performance: Evidence of technical quality of results from the angoff standard setting method. Journal of Educational Measurement 2000; 37(4):347-55.\u003c/li\u003e\n\u003cli\u003eBourque J, Skinner H, Dupr\u0026eacute; J, Bacchus M, Ainslie M, Ma IWY, Cole G. Performance of the ebel standard-setting method in spring 2019 royal college of physicians and surgeons of canada internal medicine certification examination consisted of multiple-choice questions. Journal of Educational Evaluation for Health Professions 2020; 17:12.\u003c/li\u003e\n\u003cli\u003eImpara JC, Plake BS. Teachers\u0026apos; ability to estimate item difficulty: A test of the assumptions in the angoff standard setting method. Journal of Educational Measurement 1998; 35(1):69-81.\u003c/li\u003e\n\u003cli\u003eHughes FP. Comparing four estimates of the criterion-referenced standard for a written test 1983: 1-32.\u003c/li\u003e\n\u003cli\u003eEvans JS, Stanovich KE. Dual-process theories of higher cognition: Advancing the debate. Perspect Psychol Sci 2013; 8(3):223-41.\u003c/li\u003e\n\u003cli\u003eBunch MB. Practical issues in standard setting. In: Setting performance standards: Foundations, methods, and innovations. 2nd edition edn. Edited by Cizek GJ. New York \u0026amp; UK: Routledge; 2012.\u003c/li\u003e\n\u003cli\u003eShulruf B, Wilkinson T, Weller J, Jones P, Poole P. Insights into the angoff method: Results from a simulation study. BMC Med Educ 2016; 16:134.\u003c/li\u003e\n\u003cli\u003eRezigalla AA. Angoff\u0026apos;s method: The impact of raters\u0026apos; selection. Saudi J Med Med Sci 2015; 3(3):220-25.\u003c/li\u003e\n\u003cli\u003ePlake B, Impara J, Potenza M. Content specificity of expert judgments in a standard-setting study. Journal of Educational Measurement 1994; 31(4):339-47.\u003c/li\u003e\n\u003cli\u003eNorcini JJ, Shea JA, Kanya DT. The effect of various factors on standard setting. Journal of Educational Measurement 1988; 25(1):57-65.\u003c/li\u003e\n\u003cli\u003ePeabody MR, Wind SA. Exploring the influence of judge proficiency on standard‐setting judgments. Journal of Educational Measurement 2019; 56(1):101-20.\u003c/li\u003e\n\u003cli\u003eVerheggen MM, Muijtjens AM, Van Os J, Schuwirth LW. Is an angoff standard an indication of minimal competence of examinees or of judges? Advances in Health Sciences Education : Theory and Practice 2008; 13(2):203-11.\u003c/li\u003e\n\u003cli\u003eChang L, Dziuban C, Hynes MC, Olson AH. Does a standard reflect minimal competency of examinees or judge competency. Applied Measurement in Education 1996; 9:161-73.\u003c/li\u003e\n\u003cli\u003eClauser JC, Hambleton R, Baldwin P. The effect of rating unfamiliar items on angoff passing scores. Educational and Psychological Measurement 2017; 77:901 - 16.\u003c/li\u003e\n\u003cli\u003eYeates P, Cope N, Luksaite E, Hassell A, Dikomitis L. Exploring differences in individual and group judgements in standard setting. Medical Education 2019; 53(9):941-52.\u003c/li\u003e\n\u003cli\u003eHart Pt. Irving l. Janis\u0026apos; victims of groupthink. Political Psychology 1991; 12(2):247-78.\u003c/li\u003e\n\u003cli\u003eAmerican Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME). Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Dental Education, Education Measurement, Standard Setting, Angoff, Ebel, Passing Score, Passing Mark, Malaysia, Mixed-Methods","lastPublishedDoi":"10.21203/rs.3.rs-7347939/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7347939/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eIntroduction\u003c/h2\u003e\u003cp\u003eCriterion-referenced standard setting is commonly employed to establish passing marks in healthcare professional examinations, its application in Malaysian dental schools is limited. This study aimed to examine the judges\u0026rsquo; feedback and thought processes when utilising the modified Angoff and Ebel methods in the final-year multidisciplinary exam, involving two panels of judges at a dental faculty in Malaysia.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eA convergent mixed-methods approach was used, combining quantitative feedback questionnaires and qualitative focus group discussions (FGDs) over a three-day standard setting workshop at The National University of Malaysia. One-best answer items and short answer questions were standardised using the modified Angoff and Ebel methods. The questionnaires evaluated the judges\u0026rsquo; understanding, confidence and preference for the defensibility of the standard setting methods. The FGD explored the judges\u0026rsquo; experiences and thought processes, analysed through inductive thematic analysis.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eTwelve judges from various specialties participated. The feedback questionnaires showed no significant differences between the modified Angoff and Ebel methods or between Groups A and B. The judges reported enhanced understanding and confidence levels from Rounds 1 to 3. Judges\u0026rsquo; preferences for each method varied. The themes included the following: confidence in standard setting, continuous vs. category scoring, the role of diverse expertise in decision-making, alignment with course learning outcomes, barriers to effective standard setting and the impact of training as a catalyst for effective standard setting.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eGroups A and B shared similar feedback, experiences, and perspectives regarding the standard setting activities. The modified Angoff method was straightforward and easy; it gave judges more control, but it provided a subjective framework, relying on intuition and experience in setting the Angoff ratings. In contrast, the Ebel method offered a structured, objective framework, but judges felt they had less control, found the process more time-consuming, and some referred to the probability that borderline candidates would answer correctly within each category in Ebel grid to determine the difficulty level. These findings provide insights into standard setting in dental education and emphasis the importance of structured training to enhance procedural validity.\u003c/p\u003e","manuscriptTitle":"Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-09-24 11:59:04","doi":"10.21203/rs.3.rs-7347939/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5e1a2c88-d00f-4613-946a-0f5c4ee19536","owner":[],"postedDate":"September 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-11-05T12:39:00+00:00","versionOfRecord":[],"versionCreatedAt":"2025-09-24 11:59:04","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7347939","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7347939","identity":"rs-7347939","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}