A new bedside clinical evaluation tool (ECALM) for first-year residents rotating in the emergency department: a pilot study

A new bedside clinical evaluation tool (ECALM) for first-year residents rotating in the emergency department: a pilot study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A new bedside clinical evaluation tool (ECALM) for first-year residents rotating in the emergency department: a pilot study Siiri Astudillo Kunnas, Lilia Rotari, Farès Moustafa, Daniel Aiham Ghazali This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9098453/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background The first year of residency is a critical period for developing clinical reasoning and professional reflexes. However, learners' difficulties are rarely objectively identified due to a lack of standardized formative assessment tools adapted to the complexity of medical practice. Objective is to describe the development and pilot testing of the Evaluation Clinique Au Lit du Malade (ECALM: “bedside clinical evaluation” in French)) programme for residents on rotation in an emergency department (ED). Methods The ECALM programme was developed in three stages: (i) identification of frequent reasons for ED visits in France; (ii) creation of standardized clinical assessment scenarios via a modified Delphi method involving experienced practitioners; and (iii) final validation in a training context. Six clinical scenarios were selected: chest pain, abdominal pain, limb trauma, syncope, fever, and confusion. A pilot study with 12 residents was conducted at Amiens University Hospital's ED. Each assessment consisted of briefing, observation, and debriefing phases. Assessments were performed by two independent evaluators, and the level of inter-rater reliability was categorized as very insufficient, insufficient, good, or very good. Results In the needs assessment, 97% of senior physicians considered that residents were insufficiently prepared for history-taking and clinical examination. During the pilot study, 12 assessments were completed. The mean total duration of an ECALM session was 43 minutes. Inter-rater reliability was good, with an intraclass correlation coefficient of 0.76. All 12 residents were satisfied with the ECALM programme. The residents' post-test feedback highlighted a sense of increased psychological safety and perception of the programme as a "one-on-one tutorial." Ten of the 12 residents (83.3%) reported acquiring new knowledge during the debriefing. Conclusion The ECALM programme is an innovative tool with a high level of internal validity for the formative assessment of residents. By shifting assessment toward direct bedside observation within a non-punitive framework, it enables the early identification of knowledge gaps. Pilot testing revealed excellent inter-rater agreement and significant learning value. Emergency medicine Formative assessment Clinical supervision Internship and residency Teaching at the bedside ECALM Introduction First-year medical residents rotate through their hospital's emergency department (ED) on a regular basis. The first year of residency establishes the fundamental pillars of clinical instinct, methodology, and reasoning that will structure the residents' future practice. During this period in their medical education, residents must achieve complex learning objectives, and some residents will face greater challenges than others. However, the lack of appropriate assessment tools may often prevent these difficulties from being identified objectively. In fact, assessment should be a central part of teaching-learning dynamics; in the absence of assessment, training is reductive and incomplete. Although the residents' responsibilities are limited, their scope of practice depends on the levels of trust and independence granted by senior physicians during the rotation. To enhance clinical apprenticeship in the acquisition of history-taking, physical examination, and case presentation skills, we developed the ECALM formative assessment programme for first-year residents in the ED (the acronym ECALM is derived from the French for "bedside clinical evaluation": " évaluation clinique au lit du malade "). The ECALM programme was designed to assess not only the learners' knowledge but also their clinical competencies within a real-life care environment. ECALM has been designed to serve as a comprehensive tool for supporting clinical supervision and rotation-end evaluations more exhaustively than existing instruments do. The three main categories evaluated at present are history-taking, physical examination, and interpersonal communication skills [ 1 – 5 ]. These evaluations are primarily based on observation, and the literature data suggest that the use of clinical performance instruments significantly improves the quality of direct observations [ 6 ]. Today's tools for scoring clinical competencies include observation checklists and performance scales. Most of these scales include non-explicit criteria, such as "expected level" [ 2 ] or "average level" [ 3 ]. Despite the potential for discrepancies in scoring from one evaluator to another, the tools' degree of inter-rater reliability appears to be relatively high [ 1 , 5 , 7 ]. Several researchers have developed assessment lists [ 2 – 4 ]. However, the fact that these lists contain general or universal elements only makes poorly suited to the complexity of medical practice and the diversity of medical conditions encountered. Although general assessments provide for summative or diagnostic evaluations by highlighting the acquisition of broad themes (e.g. interprofessional relationships, physical examination, or patient interviews), they are often unsuitable for use in medical education. An assessment tool with subjective scales and non-exhaustive lists lacks the precision needed to identify specific student weaknesses and thereby fails to provide the adjustments needed to help students train successfully. La Mantia et al.[ 5 ] analyzed the differences between two distinct types of instrument: global rating scales (where observers draw direct conclusions about performance) vs. checklist-based scales (from which judgments are subsequently derived). The study's results indicated that inter-rater disagreements are not necessarily linked to the subjective nature of the scale, or the domains assessed but can involve more fundamental divergences in the rater's ability to observe specific behaviours. These differences in observational skills must be considered when designing an assessment tool. Furthermore, several studies have corroborated the need to train evaluators in the use of tools prior to deployment [ 1 , 5 , 7 ]. Although the evaluators in the previously cited studies were faculty professors, the evidence suggests that academic qualification is not a prerequisite for reliability when evaluators are pre-trained. For example, Shayne et al.[ 8 ] did not find any significant differences in reliability as a function of rater experience, academic rank, clinical setting (a university hospital vs. a non-university hospital), or prior use of the tool. By taking account of the literature findings, we sought to create an assessment programme that provides a reproducible, formative evaluation and can be used by any trained evaluator. We also sought to develop evaluation criteria that are as exhaustive as possible, in order to reflect the inherent complexity of medical practice. Methods Needs assessment Prior to developing the ECALM formative assessment program, we conducted two needs assessment surveys. These surveys were developed for this study by two emergency physicians and reviewed independently by the two Emergency Medicine Coordinator for the Amiens region and Clermont-Ferrand region. The first survey addressed ED physicians in four university hospitals and eight regional hospitals in France (Appendix 1). The second was sent to first-year residents who completed a rotation in Amiens University Hospital's ED between May 2024 and May 2025 (Appendix 2). The surveys were hosted on a free, online platform (Framaforms®, https://framaforms.org/ ) from April 15–30, 2025. The physician survey consisted of six single-choice questions on their perception of the need for an enhanced competency assessment for residents during ED rotations. The resident survey consisted of four single-choice questions. Content validity The ECALM programme was built in a stepwise manner. We first identified the primary reasons for attending an ED in France [ 9 ]. We then used a simplified Delphi method to develop standardized patient assessment sections. Firstly, two physicians independently created ECALM grids for six clinical scenarios frequently encountered in the ED. The scenarios were then reviewed independently by two other experienced practitioners. The feedback was merged in a consolidation phase, and final validation was provided by the Emergency Medicine Coordinator for the Amiens region. All the personnel involved in the development of ECALM had formal training in medical education and 4 to 20 years of experience in emergency medicine. Response process Next, we sought to identify errors, inaccuracies, and redundancies in the scale. To evaluate the learning tool's internal validity, ECALM was piloted with 12 residents on rotation in Amiens University Hospital's ED. Although the tool was designed for use by a single evaluator in a formative context, assessments were performed in pairs; this enables the results to be compared. The tool was tested by three volunteer physicians (two of whom had prior training in medical education) with four years of experience and one professor of emergency medicine with 15 years of experience in both academic and practical, hospital-based medical education. During a 45-minute briefing session, the evaluators familiarized themselves with the six assessment sections. We also defined the expected objectives of the debriefing phase with the resident, by emphasizing the importance of clarifying clinical reasoning. The residents' verbal feedback on the tool's perceived utility was collected at the end of each ECALM session, and the evaluators' opinions of the tool were collected at the end of the pilot study. Statistical analysis Quantitative variables were described as the mean ± standard deviation (SD) and/or the median [interquartile range (IQR)]. Categorical variables were summarized as the frequency (percentage). The inter-observer reliability of the residents' performance scores was used to assess construct validity by calculation of the intraclass correlation coefficient (ICC). ICC values below 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and above 0.9 were considered to indicate poor, moderate, good and excellent reliability, respectively.[ 10 ]. Results Needs assessment A total of 258 attending physicians were contacted, and 70 (27.1%) completed the survey. Of these, 30 (43%) worked solely in a university hospital, 14 (20%) worked in both a university hospital and a general hospital (GH), and 26 (37%) worked solely exclusively in a general hospital. Thirteen (18.5%) practitioners stated that they did not assess their residents' competencies, 27 (38.5%) reported subjectively assessing their residents' competence or trustworthiness, and the remaining 30 (43%) stated that they always observed each resident's history-taking and clinical examination skills at least once. Sixty-eight (97%) of the surveyed physicians believed that residents are insufficiently prepared for patient interviews, clinical examinations, and case presentation during their ED rotation. Sixty-four (91.4%) physicians considered that assessment of first-year residents bedside skills in the ED was essential. Regarding the level of trust placed to residents, 48 (68.6%) of the physicians stated they would feel more comfortable and confident with a resident who had successfully passed a bedside clinical evaluation, while 22 (31.2%) stated that this would not change their perception. This needs assessment of attending physicians confirmed the value of a programme like ECALM designed to evaluate the competencies of first-year residents during their ED rotation. The resident needs assessment survey was sent to 23 residents (14 women and 9 men), 12 (52%) of whom (7 women and 5 men) replied. All respondents were first-year residents who had completed their ED rotation at Amiens University Hospital. Ten residents (83.3%) felt "somewhat prepared" to interview a patient about their medical history, perform a physical examination, and present a case, while only two (16.7%) felt "insufficiently prepared". of their, Nine (75%) of the residents considered that a skills assessment was necessary and useful for better targeting potential gaps, while three (25%) did not perceive an assessment to be useful. Development of the ECALM program All the evaluators were volunteer senior ED physicians who had been trained in the use of the assessment tools. The patients participating in the programme were real patients attending the ED on the day of the assessment. The evaluators obtained verbal consent from the patients before participation in the assessment. Patients with known cognitive impairments or those under legal guardianship were excluded. Six standardized patient assessment sections were developed using a modified Delphi method. Each section addressed a clinical scenario frequently encountered in adult EDs. Based on recent data on the reasons for ED consultations in France (11) (notably trauma (59%), pain (43%), fever (9%), and syncope/malaise (9%)), we selected six standardized themes: chest pain, abdominal pain (Appendix 3), limb trauma, syncope, fever, and (given the increasing prevalence of elderly patients in the ED [ 12 ]) confusion. Given that the ECALM programme is intended to ensure the acquisition of core competencies for all first-year residents; we opted not to strictly follow existing specialty-specific competency frameworks [ 13 , 14 ] because these lack a year-by-year, level-based approach. However, to ensure alignment with educational programmes in France, we cross-referenced the French government's official learning objectives for first-year residents [ 15 ]. Each section was subdivided into assessments of socio-affective (soft) skills, general and theme-specific history-taking, physical examination, and case presentation. The primary objectives were to ensure mastery of competencies in these areas and thus to allow residents to pursue their education under conditions that were safe for both patients and care teams. On the day of the bedside clinical evaluation, the attending physician selected a patient from those present in the ED and obtain their explicit consent. Once the patient had been selected, the resident was told which of the six clinical scenarios would be assessed. The resident then filled out a pre-test self-questionnaire by scoring (on a Likert scale) their perceived competence in managing the specific clinical theme. The ECALM session began when the resident and the evaluators entered the examination room. The evaluators observed the encounter and did not interact with the patient or the resident. In order to mimic real-life conditions as closely as possible, no time limit was imposed. This first phase ended when the resident informed the evaluators that they had completed the history-taking, physical examination, and documentation (in the patient's paper-based or electronic medical records). The resident and evaluators then moved immediately to a private, quiet room for the second phase: case presentation. This phase concluded with constructive feedback from the evaluators, based on their observations. The evaluators were able to (i) ask questions to assess the resident's clinical reasoning, and (ii) correct the clinical reasoning if necessary. At the end of the ECALM session, the resident completed the post-test self-questionnaire, which was identical to the pre-test questionnaire. The specific content of the assessment sections was not disclosed to the residents, to ensure they did not perceive the ECALM session to be a high-stakes summative examination or, in contrast, a mere "grading" exercise. Internal Validity of the ECALM Assessment Tool Twelve residents (four (33.3%) men and eight (66.7%) women) were each evaluated for one of the six standardized clinical scenarios. The mean ± SD age was 25 ± 1 years. The mean duration of history-taking and physical examination was 15 ± 6 minutes, and the mean duration of the ECALM session as a whole was 43 ± 12 minutes (Table 1 ). Table 1 Time spent by each resident on history-taking and physical examination, and the total duration of the ECALM session. Resident Duration of history-taking & physical examination (minutes) Total ECALM duration (minutes) Resident 1 10 37 Resident 2 12 40 Resident 3 15 42 Resident 4 9 38 Resident 5 32 80 Resident 6 13 41 Resident 7 14 40 Resident 8 16 47 Resident 9 10 36 Resident 10 15 41 Resident 11 17 39 Resident 12 13 35 Mean 15 43 Standard Deviation 6 12 The evaluators checked whether the student performed the expected actions or asked the necessary questions. When an item was not applicable to the specific clinical situation, the evaluator noted "NA" (not applicable). The percentage of applicable items checked by the evaluator corresponded to a four-level performance scale: very insufficient (0%-25%), insufficient (25%-50%), good (50%-75%), and very good (75%-100%). The evaluators' respective ratings were then compared (Table 2 ). Table 2 The residents' performance levels, according to each evaluator. Resident Evaluator 1 Evaluator 2 Resident 1 50–75% 50–75% Resident 2 50–75% 50–75% Resident 3 50–75% 50–75% Resident 4 25–50% 25–50% Resident 5 75–100% 75–100% Resident 6 50–75% 50–75% Resident 7 50–75% 50–75% Resident 8 25–50% 50–75% Resident 9 50–75% 50–75% Resident 10 50–75% 50–75% Resident 11 75–100% 75–100% Resident 12 75–100% 50–75% The ICC was 0.76, which corresponds to good inter-rater reliability. Feedback from the Residents and Evaluators The pre-/post-test questionnaires showed that all the residents were satisfied with the ECALM assessment tool. In interviews conducted at the end of the ECALM sessions (Table 3 ), the residents reported perceptions of a "one-on-one tutorial", a "greater sense of safety", and a "greater perceived interest in the resident's learning process". Five of the twelve residents (41.6%) began the ECALM programme by stating they did not feel comfortable with the clinical theme being assessed, and ten (83.3%) spontaneously reported that they had learned something new during the debriefing with the evaluators. Table 3 The residents' verbatim comments on the ECALM assessment program. Resident Verbatim Resident 3 "It felt like a one-on-one tutorial. I'm pleased because when I see patients on my own, I don't necessarily realize that I could do things differently". Resident 4 "Sometimes senior physicians come and see patients with me but they usually interact with the patient at the same time. While it's good to see another way of doing things, I realize there were things that I wasn't noticing because they weren't explained to me. Now I understand certain things better". Resident 9 "I had never experienced this type of evaluation. I was stressed at first but I realized ultimately that it was organized to help us learn and not to judge us. I felt there was a genuine interest in our learning, and I felt valued". Resident 11 "I think it's a good thing to have this kind of evaluation during this rotation because we usually see lots of patients on our own. It’s an extra layer of safety for the patients and for us. I definitely feel safer now". Resident 12 "At first, I thought that this evaluation was unnecessary because I didn't feel like I'm struggling particularly; I think my supervisors are generally happy with me. However, the debriefing made me realize I was doing some things systematically without really understanding why. Ultimately, I think it's a very interesting exercise". The evaluators' feedback (Table 4 ) was encouraging but highlighted occasional difficulties in finding "eligible" patients whose reasons for consultation matched one of ECALM's six standardized categories. The flexibility of the assessment sections with regard to the expected items was much appreciated because it enabled the assessment to be adapted to suit each individual consultation. Table 4 The evaluators' verbatim comments on the ECALM program. Evaluator Verbatim Evaluator 1 "It's easier to use ECALM in the morning when the ED is less crowded. But it's sometimes hard to find eligible patients who match one of the six standardized categories. This requires organizational flexibility. I think it's best to do this evaluation when neither the evaluator nor the resident is primarily assigned to active clinical care". Evaluator 2 "The sections are easy to get used to. I like being able to remove an item when I feel, as a senior physician, that it isn't appropriate for the patient in front of us; after all, every patient is unique. The tool allows us to assess the resident on the basis of coherent expectations. For example, looking for signs of portal hypertension in a confused patient with no history of relevant gastrointestinal signs won't result in a lower score on the performance scale". Evaluator 3 "In my opinion, the debriefing phase is the most interesting part of the assessment because it ensures that the residents understand what they are doing. We sometimes have surprises, with residents explaining that they perform certain actions or examinations "'by default" because they are copying older residents or attending physicians. This phase allows us to explain things again and show the resident that learning involves questioning what we take for granted. Furthermore, the tool is easy to use, and I would like to use it to track my residents' progress during their rotations". Discussion The ECALM programme's novelty lies in its rejection of binary or punitive grading. By avoiding the disclosure of a numerical score to the student and removing the "not done equals zero" rule, the assessment is transformed into a powerful lever for learning. Unlike simulation sessions, first-year residents – supported by six years of theory-based learning – are confronted with the actual complexity of the field and real (i.e. non-simulated) patients. Evaluation with real patients reduces the performative or theatrical dimension that may arise when clinical encounters are simulated or role-played. This approach enables evaluation of the student's ability to establish a genuine physician-patient relationship. Although lack of consent from the patient being examined was a potential obstacle, we did not record any such refusals to participate. The ECALM programme is in line with constructivist and cognitivist frameworks and draws notably on Kolb's experiential learning cycle [ 16 ], where "active participation" is a key factor in effective learning. To evaluate the ECALM programme itself, we utilized two complementary models: Stufflebeam's model gives a global vision centred on adjusting or extending the programme, and Kirkpatrick's model measures the training's effects on learners by focusing on the first three levels (reaction, learning, and behaviour). One of ECALM's strengths is its grounding in a preliminary reproducibility analysis with 12 residents. This work confirmed the tool's content validity and reliability. Our results suggest that the level of reproducibility is good because the two evaluators' observations of a given resident were similar. Although ECALM was designed for use by a single evaluator in a training context, our initial decision to have a resident's performance assessed by two independent evaluators enabled internal validation of the tool. Inter-observer bias was mitigated by standardizing the disease categories and training the evaluators prior to deployment. Although the ECALM took less than an hour to administer, finding this time during a training programme was sometimes challenging. Post-assessment debriefing is at the heart of the ECALM programme; it helps to clear up ambiguities (e.g., a simple mistake in transcription vs. a true lack of knowledge) and highlight elements of which the resident was potentially unaware (e.g. misgendering, speech rate, and posture). By jointly evaluating "content" (medical knowledge) and "form" (soft skills), ECALM sessions meet the residents' social expectations and rebalance the sometimes-excessive focus on technical performance. By shifting the assessment from a theory-based exam to the bedside and by combining direct observations with structured dialogue, ECALM reinforces the sense of clinical apprenticeship. This non-critical framework fosters intrinsic motivation and enables difficulties to be identified early; this is crucial because although a lack of mastery is expected at the start of residency, it becomes more complex to admit to and correct as the residency progresses. ECALM therefore enables the early implementation of a targeted, remedial learning process between the supervisor and the resident. Lastly, the clarification of clinical reasoning encourages the development of the residents' self-assessment skills and allows the residents to identify areas for improvement more precisely. All these features help to anchor knowledge durably, provide senior physicians with objective feedback on their residents' skills, and thus enhance patient safety directly. However, deployment in the field faces some practical constraints. Given the dependence on patient flow and the need to avoid delaying emergency care, the morning is the best time for assessment – provided that patients with reasons for consultation corresponding to the standardized disease categories are available. The mean assessment time (43 minutes) meant that the patient flow could be managed by the team on duty. Ideally, a resident should be evaluated while not primarily assigned to clinical duties; however, persuading a resident to come into the hospital on a day off remains problematic. Next, variability in the number of available evaluators necessitated rigorous upstream organization. It was sometimes difficult to bring together two trained evaluators at the same time; in the future, operating ECALM with only one evaluator might be relevant. Furthermore, ECALM assessments could be led by practitioners with dedicated teaching time (e.g. clinical fellows). We feel that the six current patient assessment sections are enough to evaluate progress; increasing the number of sections might not add any additional learning value. The encouraging results of the pilot study conducted at Amiens University Hospital suggest that an assessment of the ECALM programme's external validity in a large, multicentre study is now warranted. Limitations This study had several limitations. First, it was conducted at a single centre. The clinical scenarios and the assessment framework were developed by physicians from the same establishment. This might have limited the generalizability of the findings because educational practices and training environments vary from one establishment to another. However, the single-centre design enabled the programme to be implemented and tested in a controlled educational setting and facilitated close collaboration between the educators involved in the tools' development and delivery. Hence, our pilot study was designed to assess the feasibility and acceptability of this educational program. Our findings provide preliminary insights that will help to refine the scenarios and assessment tools and underpin the design of future multicentre studies of the ECALM programme's external validity and broader applicability. Conclusion To the best of our knowledge, the ECALM programme is the first standardized, comprehensive, formative assessment framework designed specifically for first-year residents in the ED. The tool was developed after a rigorous needs assessment process and, to ensure clinical relevance, has been designed to target the medical conditions most frequently encountered in the ED. As part of the operational deployment, the program's internal validity was tested in the ED at Amiens University Hospital. The results of the present pilot study are encouraging with regard to feasibility, internal validity, and reliability. This type of assessment is essential during the first year of residency; it is during this pivotal period that the mental frameworks and clinical instinct that will structure the practitioner’s entire career are established. We now plan to investigate ECALM's external validity (and thus confirm that the model is transferable to and effective in other EDs) in a multicentre study. Declarations Conflict of interest None declared. Consent for publication: not applicable Publication statement The manuscript has not been previously published and is not under consideration in the same or substantially similar form in any other journal. Funding None declared. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Author Contribution SAK: design of the work; acquisition; interpretation of data; have drafted the workLR: acquisition; substantively revised the manuscriptFM: acquisition; substantively revised the manuscript**DAG: conception; analysis, interpretation of data; have drafted the work**All the authors have approved the submitted version (and any substantially modified version that involves the author's contribution to the study);All the authors have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. Acknowledgement The authors thank all the residents and physicians who participated in the study and also thank David Fraser (Biotech Communication SARL, Ploudalmézeau, France) for editorial assistance.The authors thank Dr Martin Petitprez and Dr Sylvain Leclere for reviewing the ECALM sheets and contributing to their revision. Data Availability All data analyzed during this study are included in the manuscript. For more details, please contact the corresponding author. References Rosenzweig S, Brigham TP, Snyder RD, Xu G, McDonald AJ. Assessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability. J Emerg Med. 1999;17(2):355–61. 10.1016/s0736-4679 . (98)00181-4 PubMed PMID: 10195505. Jouriles NJ, Emerman CL, Cydulka RK. Direct observation for assessing emergency medicine core competencies: interpersonal skills. Acad Emerg Med nov. 2002;9(11):1338–41. 10.1111/j.1553-2712.2002.tb . 01598.x PubMed PMID: 12414492. Cydulka RK, Emerman CL, Jouriles NJ. Evaluation of resident performance and intensive bedside teaching during direct observation. Acad Emerg Med avr. 1996;3(4):345–51. 10.1111/j.1553-2712.1996.tb . 03448.x PubMed PMID: 8881544. LaMantia J, Kane B, Yarris L, Tadros A, Ward MF, Lesser M, et al. Real-time inter-rater reliability of the Council of Emergency Medicine residency directors standardized direct observation assessment tool. Acad Emerg Med déc. 2009;16(Suppl 2):S51–57. 10.1111/j.1553-2712.2009.00593 . .x PubMed PMID: 20053212. LaMantia J, Rennie W, Risucci DA, Cydulka R, Spillane L, Graff L, et al. Interobserver Variability among Faculty in Evaluations of Residents Clinical Skills. Acad Emerg Med. 1999;6(1):38–44. 10.1111/j.1553-2712.1999.tb00092.x . Ross R. A clinical-performance biopsy instrument. Acad Med mars. 2002;77(3):268. doi:10.1097/00001888-200203000-00021 PubMed PMID: 11891170. Herbers JE, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4(3):202–8. 10.1007. /BF02599524 PubMed PMID: 2723833. Shayne P, Gallahue F, Rinnert S, Anderson CL, Hern G, Katz E. Reliability of a Core Competency Checklist Assessment in the Emergency Department: The Standardized Direct Observation Assessment Tool. Acad Emerg Med. 2006;13(7):727–32. 10.1197/j.aem.2006.01.030 . Motifs. et trajectoires de recours aux urgences hospitalières | Direction de la recherche, des études, de l’évaluation et des statistiques. https://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0 Guyader FP, Violeau M, Guenezan J, Guechi Y, Breque C, Betoulle-Masset P, et al. Development and validation of an assessment tool for adult simulated ultrasound-guided fascia iliaca block: a prospective monocentric study. Emerg Med J 28 mai. 2024;41(6):354–60. 10.1136/emermed-2023-213123 . PubMed PMID: 38521512. Motifs et trajectoires de recours aux urgences hospitalières. Direction de la recherche, des études, de l’évaluation et des statistiques. https://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0 Ukkonen M, Jämsen E, Zeitlin R, Pauniaho SL. Emergency department visits in older patients: a population-based survey. BMC Emerg Med. 27 févr. 2019;19(1):20. 10.1186/s12873-019-0236-3 PubMed PMID: 30813898; PubMed Central PMCID: PMC6391758. Andronikof M, Coudert B, Ellrodt A, Pateron D, Platonoff S, Raphaël V, Van der Linden T. Société Française de Médecine d’Urgence, Référentiels, « Les compétences du médecin urgentiste », 2004. Battistoni J, Bertois P, Cadiou O, Chevallier F, de Beco A, May E, Onaisi R, Saynac Y, Villebrun F. Référentiel professionnel de la médecine générale, 2025. Légifrance Arrêté. du 21 avril 2017 relatif aux connaissances, aux compétences et aux maquettes de formation des diplômes d'études spécialisées et fixant la liste de ces diplômes et des options et formations spécialisées transversales du troisième cycle des études de médecine. JORF n°0100 du 28 avril 2017. Kolb AY, Kolb DA. The Learning Way: Meta-cognitive Aspects of Experiential Learning. Simul Gaming juin. 2009;40(3):297–327. 10.1177/1046878108325713 . Additional Declarations No competing interests reported. Supplementary Files Appendix1Needsassessmentdoctors.docx Appendix2Needsassessmentresidents.docx Appendix3ECALMV2.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9098453","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":625899826,"identity":"fa0903f2-75f4-420c-956e-62c66fddb4e5","order_by":0,"name":"Siiri Astudillo Kunnas","email":"","orcid":"","institution":"Amiens University Hospital","correspondingAuthor":false,"prefix":"","firstName":"Siiri","middleName":"Astudillo","lastName":"Kunnas","suffix":""},{"id":625899827,"identity":"3819a2c3-d915-460c-b670-ba8d79ef2485","order_by":1,"name":"Lilia Rotari","email":"","orcid":"","institution":"Amiens University Hospital","correspondingAuthor":false,"prefix":"","firstName":"Lilia","middleName":"","lastName":"Rotari","suffix":""},{"id":625899828,"identity":"1832cef0-565e-4cf7-943a-969caed961a1","order_by":2,"name":"Farès Moustafa","email":"","orcid":"","institution":"Clermont-Ferrand University Hospital","correspondingAuthor":false,"prefix":"","firstName":"Farès","middleName":"","lastName":"Moustafa","suffix":""},{"id":625899829,"identity":"1b8946c4-e724-4987-a1e9-0f5c52024b7f","order_by":3,"name":"Daniel Aiham Ghazali","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABUUlEQVRIie2QMUvDQBTHXwjU5dquF1qar/BChyqKfhCXFsEsiRS6FIQ2UohLukr8GFIIjgkBO1jJWomD0rWFiEuLGbxEhSaNu8P9eDyOx/34vzsADuc/QpPWZd1NqgsEAYRXwUjG371aqOCvgqki4rYiGYUKpEp6YFWiwta1vCKfjhbRGvehMn3y3AiP6629x/P+5t6H6o1/tejGQFtZR3l5QMVii0mzi7Zn4xk5sHTneTzzgYadUfPWBFp3s4rdxjZhCroa+gRFgq7uhILpD42wY9bKBgwo5BQ18uJECZboxzgkGKycHlNADjvXn4QtllNkqimjNGXOUiAJmuuOmCjIUkRS2lGQaj2xjpRI8yV6Fk6ZsppIY1MFhb2lVjbpToqtTt6X/UGjEmjNaN2/PMFAv4s25iE0QvXtg8RHOyk/v0HgL2h+IBsFtzgcDoeT4QunAnezf3fVFQAAAABJRU5ErkJggg==","orcid":"","institution":"Amiens University Hospital","correspondingAuthor":true,"prefix":"","firstName":"Daniel","middleName":"Aiham","lastName":"Ghazali","suffix":""}],"badges":[],"createdAt":"2026-03-11 23:08:21","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9098453/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9098453/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109045838,"identity":"35868806-957b-4342-a1d3-0e7af8bc526c","added_by":"auto","created_at":"2026-05-12 05:27:14","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":254107,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9098453/v1/b6f97f58-aa2a-444d-a8fb-a8253426e283.pdf"},{"id":107697994,"identity":"651c35ec-1575-4948-92bb-c2b9e84ab754","added_by":"auto","created_at":"2026-04-24 07:29:13","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":18034,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix1Needsassessmentdoctors.docx","url":"https://assets-eu.researchsquare.com/files/rs-9098453/v1/05616337bc445211f832451a.docx"},{"id":107708054,"identity":"f22429be-b650-40e2-aef6-0d87a4b87b3f","added_by":"auto","created_at":"2026-04-24 09:21:46","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":16542,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix2Needsassessmentresidents.docx","url":"https://assets-eu.researchsquare.com/files/rs-9098453/v1/d39430964c411d677e7841ce.docx"},{"id":107697996,"identity":"cbaf9f3c-3f1d-44ae-abe3-024268369fec","added_by":"auto","created_at":"2026-04-24 07:29:13","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":23574,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix3ECALMV2.docx","url":"https://assets-eu.researchsquare.com/files/rs-9098453/v1/d9b883b50a25d0313dde7cbf.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"A new bedside clinical evaluation tool (ECALM) for first-year residents rotating in the emergency department: a pilot study","fulltext":[{"header":"Introduction","content":"\u003cp\u003eFirst-year medical residents rotate through their hospital's emergency department (ED) on a regular basis. The first year of residency establishes the fundamental pillars of clinical instinct, methodology, and reasoning that will structure the residents' future practice. During this period in their medical education, residents must achieve complex learning objectives, and some residents will face greater challenges than others. However, the lack of appropriate assessment tools may often prevent these difficulties from being identified objectively. In fact, assessment should be a central part of teaching-learning dynamics; in the absence of assessment, training is reductive and incomplete. Although the residents' responsibilities are limited, their scope of practice depends on the levels of trust and independence granted by senior physicians during the rotation. To enhance clinical apprenticeship in the acquisition of history-taking, physical examination, and case presentation skills, we developed the ECALM formative assessment programme for first-year residents in the ED (the acronym ECALM is derived from the French for \"bedside clinical evaluation\": \"\u003cem\u003e\u0026eacute;valuation clinique au lit du malade\u003c/em\u003e\"). The ECALM programme was designed to assess not only the learners' knowledge but also their clinical competencies within a real-life care environment. ECALM has been designed to serve as a comprehensive tool for supporting clinical supervision and rotation-end evaluations more exhaustively than existing instruments do. The three main categories evaluated at present are history-taking, physical examination, and interpersonal communication skills [\u003cspan additionalcitationids=\"CR2 CR3 CR4\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. These evaluations are primarily based on observation, and the literature data suggest that the use of clinical performance instruments significantly improves the quality of direct observations [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eToday's tools for scoring clinical competencies include observation checklists and performance scales. Most of these scales include non-explicit criteria, such as \"expected level\" [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] or \"average level\" [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Despite the potential for discrepancies in scoring from one evaluator to another, the tools' degree of inter-rater reliability appears to be relatively high [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Several researchers have developed assessment lists [\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. However, the fact that these lists contain general or universal elements only makes poorly suited to the complexity of medical practice and the diversity of medical conditions encountered. Although general assessments provide for summative or diagnostic evaluations by highlighting the acquisition of broad themes (e.g. interprofessional relationships, physical examination, or patient interviews), they are often unsuitable for use in medical education. An assessment tool with subjective scales and non-exhaustive lists lacks the precision needed to identify specific student weaknesses and thereby fails to provide the adjustments needed to help students train successfully.\u003c/p\u003e \u003cp\u003eLa Mantia et al.[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] analyzed the differences between two distinct types of instrument: global rating scales (where observers draw direct conclusions about performance) vs. checklist-based scales (from which judgments are subsequently derived). The study's results indicated that inter-rater disagreements are not necessarily linked to the subjective nature of the scale, or the domains assessed but can involve more fundamental divergences in the rater's ability to observe specific behaviours. These differences in observational skills must be considered when designing an assessment tool. Furthermore, several studies have corroborated the need to train evaluators in the use of tools prior to deployment [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Although the evaluators in the previously cited studies were faculty professors, the evidence suggests that academic qualification is not a prerequisite for reliability when evaluators are pre-trained. For example, Shayne et al.[\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] did not find any significant differences in reliability as a function of rater experience, academic rank, clinical setting (a university hospital vs. a non-university hospital), or prior use of the tool.\u003c/p\u003e \u003cp\u003eBy taking account of the literature findings, we sought to create an assessment programme that provides a reproducible, formative evaluation and can be used by any trained evaluator. We also sought to develop evaluation criteria that are as exhaustive as possible, in order to reflect the inherent complexity of medical practice.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eNeeds assessment\u003c/h2\u003e \u003cp\u003ePrior to developing the ECALM formative assessment program, we conducted two needs assessment surveys. These surveys were developed for this study by two emergency physicians and reviewed independently by the two Emergency Medicine Coordinator for the Amiens region and Clermont-Ferrand region. The first survey addressed ED physicians in four university hospitals and eight regional hospitals in France (Appendix 1). The second was sent to first-year residents who completed a rotation in Amiens University Hospital's ED between May 2024 and May 2025 (Appendix 2). The surveys were hosted on a free, online platform (Framaforms\u0026reg;, \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://framaforms.org/\u003c/span\u003e\u003cspan address=\"https://framaforms.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e) from April 15\u0026ndash;30, 2025. The physician survey consisted of six single-choice questions on their perception of the need for an enhanced competency assessment for residents during ED rotations. The resident survey consisted of four single-choice questions.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eContent validity\u003c/h3\u003e\n\u003cp\u003eThe ECALM programme was built in a stepwise manner. We first identified the primary reasons for attending an ED in France [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. We then used a simplified Delphi method to develop standardized patient assessment sections. Firstly, two physicians independently created ECALM grids for six clinical scenarios frequently encountered in the ED. The scenarios were then reviewed independently by two other experienced practitioners. The feedback was merged in a consolidation phase, and final validation was provided by the Emergency Medicine Coordinator for the Amiens region. All the personnel involved in the development of ECALM had formal training in medical education and 4 to 20 years of experience in emergency medicine.\u003c/p\u003e\n\u003ch3\u003eResponse process\u003c/h3\u003e\n\u003cp\u003eNext, we sought to identify errors, inaccuracies, and redundancies in the scale. To evaluate the learning tool's internal validity, ECALM was piloted with 12 residents on rotation in Amiens University Hospital's ED. Although the tool was designed for use by a single evaluator in a formative context, assessments were performed in pairs; this enables the results to be compared. The tool was tested by three volunteer physicians (two of whom had prior training in medical education) with four years of experience and one professor of emergency medicine with 15 years of experience in both academic and practical, hospital-based medical education. During a 45-minute briefing session, the evaluators familiarized themselves with the six assessment sections. We also defined the expected objectives of the debriefing phase with the resident, by emphasizing the importance of clarifying clinical reasoning. The residents' verbal feedback on the tool's perceived utility was collected at the end of each ECALM session, and the evaluators' opinions of the tool were collected at the end of the pilot study.\u003c/p\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eStatistical analysis\u003c/h2\u003e \u003cp\u003eQuantitative variables were described as the mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard deviation (SD) and/or the median [interquartile range (IQR)]. Categorical variables were summarized as the frequency (percentage). The inter-observer reliability of the residents' performance scores was used to assess construct validity by calculation of the intraclass correlation coefficient (ICC). ICC values below 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and above 0.9 were considered to indicate poor, moderate, good and excellent reliability, respectively.[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eNeeds assessment\u003c/h2\u003e \u003cp\u003eA total of 258 attending physicians were contacted, and 70 (27.1%) completed the survey. Of these, 30 (43%) worked solely in a university hospital, 14 (20%) worked in both a university hospital and a general hospital (GH), and 26 (37%) worked solely exclusively in a general hospital. Thirteen (18.5%) practitioners stated that they did not assess their residents' competencies, 27 (38.5%) reported subjectively assessing their residents' competence or trustworthiness, and the remaining 30 (43%) stated that they always observed each resident's history-taking and clinical examination skills at least once. Sixty-eight (97%) of the surveyed physicians believed that residents are insufficiently prepared for patient interviews, clinical examinations, and case presentation during their ED rotation. Sixty-four (91.4%) physicians considered that assessment of first-year residents bedside skills in the ED was essential. Regarding the level of trust placed to residents, 48 (68.6%) of the physicians stated they would feel more comfortable and confident with a resident who had successfully passed a bedside clinical evaluation, while 22 (31.2%) stated that this would not change their perception. This needs assessment of attending physicians confirmed the value of a programme like ECALM designed to evaluate the competencies of first-year residents during their ED rotation.\u003c/p\u003e \u003cp\u003eThe resident needs assessment survey was sent to 23 residents (14 women and 9 men), 12 (52%) of whom (7 women and 5 men) replied. All respondents were first-year residents who had completed their ED rotation at Amiens University Hospital. Ten residents (83.3%) felt \"somewhat prepared\" to interview a patient about their medical history, perform a physical examination, and present a case, while only two (16.7%) felt \"insufficiently prepared\". of their, Nine (75%) of the residents considered that a skills assessment was necessary and useful for better targeting potential gaps, while three (25%) did not perceive an assessment to be useful.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eDevelopment of the ECALM program\u003c/h3\u003e\n\u003cp\u003eAll the evaluators were volunteer senior ED physicians who had been trained in the use of the assessment tools. The patients participating in the programme were real patients attending the ED on the day of the assessment. The evaluators obtained verbal consent from the patients before participation in the assessment. Patients with known cognitive impairments or those under legal guardianship were excluded.\u003c/p\u003e \u003cp\u003eSix standardized patient assessment sections were developed using a modified Delphi method. Each section addressed a clinical scenario frequently encountered in adult EDs. Based on recent data on the reasons for ED consultations in France (11) (notably trauma (59%), pain (43%), fever (9%), and syncope/malaise (9%)), we selected six standardized themes: chest pain, abdominal pain (Appendix 3), limb trauma, syncope, fever, and (given the increasing prevalence of elderly patients in the ED [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]) confusion. Given that the ECALM programme is intended to ensure the acquisition of core competencies for all first-year residents; we opted not to strictly follow existing specialty-specific competency frameworks [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] because these lack a year-by-year, level-based approach. However, to ensure alignment with educational programmes in France, we cross-referenced the French government's official learning objectives for first-year residents [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Each section was subdivided into assessments of socio-affective (soft) skills, general and theme-specific history-taking, physical examination, and case presentation. The primary objectives were to ensure mastery of competencies in these areas and thus to allow residents to pursue their education under conditions that were safe for both patients and care teams.\u003c/p\u003e \u003cp\u003eOn the day of the bedside clinical evaluation, the attending physician selected a patient from those present in the ED and obtain their explicit consent. Once the patient had been selected, the resident was told which of the six clinical scenarios would be assessed. The resident then filled out a pre-test self-questionnaire by scoring (on a Likert scale) their perceived competence in managing the specific clinical theme. The ECALM session began when the resident and the evaluators entered the examination room. The evaluators observed the encounter and did not interact with the patient or the resident. In order to mimic real-life conditions as closely as possible, no time limit was imposed. This first phase ended when the resident informed the evaluators that they had completed the history-taking, physical examination, and documentation (in the patient's paper-based or electronic medical records).\u003c/p\u003e \u003cp\u003eThe resident and evaluators then moved immediately to a private, quiet room for the second phase: case presentation. This phase concluded with constructive feedback from the evaluators, based on their observations. The evaluators were able to (i) ask questions to assess the resident's clinical reasoning, and (ii) correct the clinical reasoning if necessary. At the end of the ECALM session, the resident completed the post-test self-questionnaire, which was identical to the pre-test questionnaire. The specific content of the assessment sections was not disclosed to the residents, to ensure they did not perceive the ECALM session to be a high-stakes summative examination or, in contrast, a mere \"grading\" exercise.\u003c/p\u003e\n\u003ch3\u003eInternal Validity of the ECALM Assessment Tool\u003c/h3\u003e\n \u003cp\u003eTwelve residents (four (33.3%) men and eight (66.7%) women) were each evaluated for one of the six standardized clinical scenarios. The mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD age was 25\u0026thinsp;\u0026plusmn;\u0026thinsp;1 years. The mean duration of history-taking and physical examination was 15\u0026thinsp;\u0026plusmn;\u0026thinsp;6 minutes, and the mean duration of the ECALM session as a whole was 43\u0026thinsp;\u0026plusmn;\u0026thinsp;12 minutes (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eTime spent by each resident on history-taking and physical examination, and the total duration of the ECALM session.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDuration of history-taking \u0026amp; physical examination (minutes)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTotal ECALM duration (minutes)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e42\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e38\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e80\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e41\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e14\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e47\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e41\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e39\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e43\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStandard Deviation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe evaluators checked whether the student performed the expected actions or asked the necessary questions. When an item was not applicable to the specific clinical situation, the evaluator noted \"NA\" (not applicable). The percentage of applicable items checked by the evaluator corresponded to a four-level performance scale: very insufficient (0%-25%), insufficient (25%-50%), good (50%-75%), and very good (75%-100%). The evaluators' respective ratings were then compared (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe residents' performance levels, according to each evaluator.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eResident\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eEvaluator 1\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eEvaluator 2\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25\u0026ndash;50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e25\u0026ndash;50%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e75\u0026ndash;100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e75\u0026ndash;100%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e25\u0026ndash;50%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e75\u0026ndash;100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e75\u0026ndash;100%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e75\u0026ndash;100%\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e50\u0026ndash;75%\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe ICC was 0.76, which corresponds to good inter-rater reliability.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eFeedback from the Residents and Evaluators\u003c/h2\u003e \u003cp\u003eThe pre-/post-test questionnaires showed that all the residents were satisfied with the ECALM assessment tool. In interviews conducted at the end of the ECALM sessions (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e), the residents reported perceptions of a \"one-on-one tutorial\", a \"greater sense of safety\", and a \"greater perceived interest in the resident's learning process\". Five of the twelve residents (41.6%) began the ECALM programme by stating they did not feel comfortable with the clinical theme being assessed, and ten (83.3%) spontaneously reported that they had learned something new during the debriefing with the evaluators.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe residents' verbatim comments on the ECALM assessment program.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eResident\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eVerbatim\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"It felt like a one-on-one tutorial. I'm pleased because when I see patients on my own, I don't necessarily realize that I could do things differently\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"Sometimes senior physicians come and see patients with me but they usually interact with the patient at the same time. While it's good to see another way of doing things, I realize there were things that I wasn't noticing because they weren't explained to me. Now I understand certain things better\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"I had never experienced this type of evaluation. I was stressed at first but I realized ultimately that it was organized to help us learn and not to judge us. I felt there was a genuine interest in our learning, and I felt valued\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"I think it's a good thing to have this kind of evaluation during this rotation because we usually see lots of patients on our own. It\u0026rsquo;s an extra layer of safety for the patients and for us. I definitely feel safer now\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eResident 12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"At first, I thought that this evaluation was unnecessary because I didn't feel like I'm struggling particularly; I think my supervisors are generally happy with me. However, the debriefing made me realize I was doing some things systematically without really understanding why. Ultimately, I think it's a very interesting exercise\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe evaluators' feedback (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e) was encouraging but highlighted occasional difficulties in finding \"eligible\" patients whose reasons for consultation matched one of ECALM's six standardized categories. The flexibility of the assessment sections with regard to the expected items was much appreciated because it enabled the assessment to be adapted to suit each individual consultation.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe evaluators' verbatim comments on the ECALM program.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eEvaluator\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eVerbatim\u003c/span\u003e\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvaluator 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"It's easier to use ECALM in the morning when the ED is less crowded. But it's sometimes hard to find eligible patients who match one of the six standardized categories. This requires organizational flexibility. I think it's best to do this evaluation when neither the evaluator nor the resident is primarily assigned to active clinical care\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvaluator 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"The sections are easy to get used to. I like being able to remove an item when I feel, as a senior physician, that it isn't appropriate for the patient in front of us; after all, every patient is unique. The tool allows us to assess the resident on the basis of coherent expectations. For example, looking for signs of portal hypertension in a confused patient with no history of relevant gastrointestinal signs won't result in a lower score on the performance scale\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvaluator 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\"In my opinion, the debriefing phase is the most interesting part of the assessment because it ensures that the residents understand what they are doing. We sometimes have surprises, with residents explaining that they perform certain actions or examinations \"'by default\" because they are copying older residents or attending physicians. This phase allows us to explain things again and show the resident that learning involves questioning what we take for granted. Furthermore, the tool is easy to use, and I would like to use it to track my residents' progress during their rotations\".\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe ECALM programme's novelty lies in its rejection of binary or punitive grading. By avoiding the disclosure of a numerical score to the student and removing the \"not done equals zero\" rule, the assessment is transformed into a powerful lever for learning. Unlike simulation sessions, first-year residents \u0026ndash; supported by six years of theory-based learning \u0026ndash; are confronted with the actual complexity of the field and real (i.e. non-simulated) patients. Evaluation with real patients reduces the performative or theatrical dimension that may arise when clinical encounters are simulated or role-played. This approach enables evaluation of the student's ability to establish a genuine physician-patient relationship. Although lack of consent from the patient being examined was a potential obstacle, we did not record any such refusals to participate.\u003c/p\u003e \u003cp\u003eThe ECALM programme is in line with constructivist and cognitivist frameworks and draws notably on Kolb's experiential learning cycle [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], where \"active participation\" is a key factor in effective learning. To evaluate the ECALM programme itself, we utilized two complementary models: Stufflebeam's model gives a global vision centred on adjusting or extending the programme, and Kirkpatrick's model measures the training's effects on learners by focusing on the first three levels (reaction, learning, and behaviour). One of ECALM's strengths is its grounding in a preliminary reproducibility analysis with 12 residents. This work confirmed the tool's content validity and reliability. Our results suggest that the level of reproducibility is good because the two evaluators' observations of a given resident were similar. Although ECALM was designed for use by a single evaluator in a training context, our initial decision to have a resident's performance assessed by two independent evaluators enabled internal validation of the tool. Inter-observer bias was mitigated by standardizing the disease categories and training the evaluators prior to deployment. Although the ECALM took less than an hour to administer, finding this time during a training programme was sometimes challenging.\u003c/p\u003e \u003cp\u003ePost-assessment debriefing is at the heart of the ECALM programme; it helps to clear up ambiguities (e.g., a simple mistake in transcription vs. a true lack of knowledge) and highlight elements of which the resident was potentially unaware (e.g. misgendering, speech rate, and posture). By jointly evaluating \"content\" (medical knowledge) and \"form\" (soft skills), ECALM sessions meet the residents' social expectations and rebalance the sometimes-excessive focus on technical performance. By shifting the assessment from a theory-based exam to the bedside and by combining direct observations with structured dialogue, ECALM reinforces the sense of clinical apprenticeship. This non-critical framework fosters intrinsic motivation and enables difficulties to be identified early; this is crucial because although a lack of mastery is expected at the start of residency, it becomes more complex to admit to and correct as the residency progresses. ECALM therefore enables the early implementation of a targeted, remedial learning process between the supervisor and the resident. Lastly, the clarification of clinical reasoning encourages the development of the residents' self-assessment skills and allows the residents to identify areas for improvement more precisely. All these features help to anchor knowledge durably, provide senior physicians with objective feedback on their residents' skills, and thus enhance patient safety directly.\u003c/p\u003e \u003cp\u003eHowever, deployment in the field faces some practical constraints. Given the dependence on patient flow and the need to avoid delaying emergency care, the morning is the best time for assessment \u0026ndash; provided that patients with reasons for consultation corresponding to the standardized disease categories are available. The mean assessment time (43 minutes) meant that the patient flow could be managed by the team on duty. Ideally, a resident should be evaluated while not primarily assigned to clinical duties; however, persuading a resident to come into the hospital on a day off remains problematic. Next, variability in the number of available evaluators necessitated rigorous upstream organization. It was sometimes difficult to bring together two trained evaluators at the same time; in the future, operating ECALM with only one evaluator might be relevant. Furthermore, ECALM assessments could be led by practitioners with dedicated teaching time (e.g. clinical fellows). We feel that the six current patient assessment sections are enough to evaluate progress; increasing the number of sections might not add any additional learning value. The encouraging results of the pilot study conducted at Amiens University Hospital suggest that an assessment of the ECALM programme's external validity in a large, multicentre study is now warranted.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eThis study had several limitations. First, it was conducted at a single centre. The clinical scenarios and the assessment framework were developed by physicians from the same establishment. This might have limited the generalizability of the findings because educational practices and training environments vary from one establishment to another. However, the single-centre design enabled the programme to be implemented and tested in a controlled educational setting and facilitated close collaboration between the educators involved in the tools' development and delivery. Hence, our pilot study was designed to assess the feasibility and acceptability of this educational program. Our findings provide preliminary insights that will help to refine the scenarios and assessment tools and underpin the design of future multicentre studies of the ECALM programme's external validity and broader applicability.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eTo the best of our knowledge, the ECALM programme is the first standardized, comprehensive, formative assessment framework designed specifically for first-year residents in the ED. The tool was developed after a rigorous needs assessment process and, to ensure clinical relevance, has been designed to target the medical conditions most frequently encountered in the ED. As part of the operational deployment, the program's internal validity was tested in the ED at Amiens University Hospital. The results of the present pilot study are encouraging with regard to feasibility, internal validity, and reliability. This type of assessment is essential during the first year of residency; it is during this pivotal period that the mental frameworks and clinical instinct that will structure the practitioner\u0026rsquo;s entire career are established. We now plan to investigate ECALM's external validity (and thus confirm that the model is transferable to and effective in other EDs) in a multicentre study.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003cstrong\u003eConflict of interest\u003c/strong\u003e \u003cp\u003eNone declared.\u003c/p\u003e \u003ch2\u003eConsent for publication:\u003c/h2\u003e \u003cp\u003e \u003cb\u003enot applicable\u003c/b\u003e \u003c/p\u003e \u003ch2\u003ePublication statement\u003c/h2\u003e \u003cp\u003eThe manuscript has not been previously published and is not under consideration in the same or substantially similar form in any other journal.\u003c/p\u003e \u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eNone declared. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eSAK: design of the work; acquisition; interpretation of data; have drafted the workLR: acquisition; substantively revised the manuscriptFM: acquisition; substantively revised the manuscript**DAG: conception; analysis, interpretation of data; have drafted the work**All the authors have approved the submitted version (and any substantially modified version that involves the author's contribution to the study);All the authors have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003e The authors thank all the residents and physicians who participated in the study and also thank David Fraser (Biotech Communication SARL, Ploudalm\u0026eacute;zeau, France) for editorial assistance.The authors thank Dr Martin Petitprez and Dr Sylvain Leclere for reviewing the ECALM sheets and contributing to their revision.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eAll data analyzed during this study are included in the manuscript. For more details, please contact the corresponding author.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eRosenzweig S, Brigham TP, Snyder RD, Xu G, McDonald AJ. Assessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability. J Emerg Med. 1999;17(2):355\u0026ndash;61. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/s0736-4679\u003c/span\u003e\u003cspan address=\"10.1016/s0736-4679\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. (98)00181-4 PubMed PMID: 10195505.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJouriles NJ, Emerman CL, Cydulka RK. Direct observation for assessing emergency medicine core competencies: interpersonal skills. Acad Emerg Med nov. 2002;9(11):1338\u0026ndash;41. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1553-2712.2002.tb\u003c/span\u003e\u003cspan address=\"10.1111/j.1553-2712.2002.tb\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. 01598.x PubMed PMID: 12414492.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCydulka RK, Emerman CL, Jouriles NJ. Evaluation of resident performance and intensive bedside teaching during direct observation. Acad Emerg Med avr. 1996;3(4):345\u0026ndash;51. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1553-2712.1996.tb\u003c/span\u003e\u003cspan address=\"10.1111/j.1553-2712.1996.tb\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. 03448.x PubMed PMID: 8881544.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLaMantia J, Kane B, Yarris L, Tadros A, Ward MF, Lesser M, et al. Real-time inter-rater reliability of the Council of Emergency Medicine residency directors standardized direct observation assessment tool. Acad Emerg Med d\u0026eacute;c. 2009;16(Suppl 2):S51\u0026ndash;57. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1553-2712.2009.00593\u003c/span\u003e\u003cspan address=\"10.1111/j.1553-2712.2009.00593\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. .x PubMed PMID: 20053212.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLaMantia J, Rennie W, Risucci DA, Cydulka R, Spillane L, Graff L, et al. Interobserver Variability among Faculty in Evaluations of Residents Clinical Skills. Acad Emerg Med. 1999;6(1):38\u0026ndash;44. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.1553-2712.1999.tb00092.x\u003c/span\u003e\u003cspan address=\"10.1111/j.1553-2712.1999.tb00092.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoss R. A clinical-performance biopsy instrument. Acad Med mars. 2002;77(3):268. doi:10.1097/00001888-200203000-00021 PubMed PMID: 11891170.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHerbers JE, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4(3):202\u0026ndash;8. 10.1007. /BF02599524 PubMed PMID: 2723833.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShayne P, Gallahue F, Rinnert S, Anderson CL, Hern G, Katz E. Reliability of a Core Competency Checklist Assessment in the Emergency Department: The Standardized Direct Observation Assessment Tool. Acad Emerg Med. 2006;13(7):727\u0026ndash;32. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1197/j.aem.2006.01.030\u003c/span\u003e\u003cspan address=\"10.1197/j.aem.2006.01.030\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMotifs. et trajectoires de recours aux urgences hospitali\u0026egrave;res | Direction de la recherche, des \u0026eacute;tudes, de l\u0026rsquo;\u0026eacute;valuation et des statistiques. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0\u003c/span\u003e\u003cspan address=\"https://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuyader FP, Violeau M, Guenezan J, Guechi Y, Breque C, Betoulle-Masset P, et al. Development and validation of an assessment tool for adult simulated ultrasound-guided fascia iliaca block: a prospective monocentric study. Emerg Med J 28 mai. 2024;41(6):354\u0026ndash;60. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/emermed-2023-213123\u003c/span\u003e\u003cspan address=\"10.1136/emermed-2023-213123\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. PubMed PMID: 38521512.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMotifs et trajectoires de recours aux urgences hospitali\u0026egrave;res. Direction de la recherche, des \u0026eacute;tudes, de l\u0026rsquo;\u0026eacute;valuation et des statistiques. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0\u003c/span\u003e\u003cspan address=\"https://drees.solidarites-sante.gouv.fr/publications/etudes-et-resultats/motifs-et-trajectoires-de-recours-aux-urgences-hospitalieres-0\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUkkonen M, J\u0026auml;msen E, Zeitlin R, Pauniaho SL. Emergency department visits in older patients: a population-based survey. BMC Emerg Med. 27 f\u0026eacute;vr. 2019;19(1):20. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s12873-019-0236-3\u003c/span\u003e\u003cspan address=\"10.1186/s12873-019-0236-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e PubMed PMID: 30813898; PubMed Central PMCID: PMC6391758.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAndronikof M, Coudert B, Ellrodt A, Pateron D, Platonoff S, Rapha\u0026euml;l V, Van der Linden T. Soci\u0026eacute;t\u0026eacute; Fran\u0026ccedil;aise de M\u0026eacute;decine d\u0026rsquo;Urgence, R\u0026eacute;f\u0026eacute;rentiels, \u0026laquo; Les comp\u0026eacute;tences du m\u0026eacute;decin urgentiste \u0026raquo;, 2004.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBattistoni J, Bertois P, Cadiou O, Chevallier F, de Beco A, May E, Onaisi R, Saynac Y, Villebrun F. R\u0026eacute;f\u0026eacute;rentiel professionnel de la m\u0026eacute;decine g\u0026eacute;n\u0026eacute;rale, 2025.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eL\u0026eacute;gifrance Arr\u0026ecirc;t\u0026eacute;. du 21 avril 2017 relatif aux connaissances, aux comp\u0026eacute;tences et aux maquettes de formation des dipl\u0026ocirc;mes d'\u0026eacute;tudes sp\u0026eacute;cialis\u0026eacute;es et fixant la liste de ces dipl\u0026ocirc;mes et des options et formations sp\u0026eacute;cialis\u0026eacute;es transversales du troisi\u0026egrave;me cycle des \u0026eacute;tudes de m\u0026eacute;decine. JORF n\u0026deg;0100 du 28 avril 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKolb AY, Kolb DA. The Learning Way: Meta-cognitive Aspects of Experiential Learning. Simul Gaming juin. 2009;40(3):297\u0026ndash;327. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/1046878108325713\u003c/span\u003e\u003cspan address=\"10.1177/1046878108325713\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Emergency medicine, Formative assessment, Clinical supervision, Internship and residency, Teaching at the bedside, ECALM","lastPublishedDoi":"10.21203/rs.3.rs-9098453/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9098453/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eThe first year of residency is a critical period for developing clinical reasoning and professional reflexes. However, learners' difficulties are rarely objectively identified due to a lack of standardized formative assessment tools adapted to the complexity of medical practice. Objective is to describe the development and pilot testing of the \u003cem\u003eEvaluation Clinique Au Lit du Malade\u003c/em\u003e (ECALM: \u0026ldquo;bedside clinical evaluation\u0026rdquo; in French)) programme for residents on rotation in an emergency department (ED).\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eThe ECALM programme was developed in three stages: (i) identification of frequent reasons for ED visits in France; (ii) creation of standardized clinical assessment scenarios via a modified Delphi method involving experienced practitioners; and (iii) final validation in a training context. Six clinical scenarios were selected: chest pain, abdominal pain, limb trauma, syncope, fever, and confusion. A pilot study with 12 residents was conducted at Amiens University Hospital's ED. Each assessment consisted of briefing, observation, and debriefing phases. Assessments were performed by two independent evaluators, and the level of inter-rater reliability was categorized as very insufficient, insufficient, good, or very good.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eIn the needs assessment, 97% of senior physicians considered that residents were insufficiently prepared for history-taking and clinical examination. During the pilot study, 12 assessments were completed. The mean total duration of an ECALM session was 43 minutes. Inter-rater reliability was good, with an intraclass correlation coefficient of 0.76. All 12 residents were satisfied with the ECALM programme. The residents' post-test feedback highlighted a sense of increased psychological safety and perception of the programme as a \"one-on-one tutorial.\" Ten of the 12 residents (83.3%) reported acquiring new knowledge during the debriefing.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eThe ECALM programme is an innovative tool with a high level of internal validity for the formative assessment of residents. By shifting assessment toward direct bedside observation within a non-punitive framework, it enables the early identification of knowledge gaps. Pilot testing revealed excellent inter-rater agreement and significant learning value.\u003c/p\u003e","manuscriptTitle":"A new bedside clinical evaluation tool (ECALM) for first-year residents rotating in the emergency department: a pilot study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-24 07:29:09","doi":"10.21203/rs.3.rs-9098453/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0a155cb4-04ee-43d8-a168-e73b8e6c5eda","owner":[],"postedDate":"April 24th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-12T05:15:11+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-05-08T18:04:23+00:00","index":34,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-12T05:26:17+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-24 07:29:09","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9098453","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9098453","identity":"rs-9098453","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}