Quality of Written Feedback given to Medical Students after Introduction of Real-Time Audio Monitoring of Clinical Encounters

Quality of Written Feedback given to Medical Students after Introduction of Real-Time Audio Monitoring of Clinical Encounters | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research article Quality of Written Feedback given to Medical Students after Introduction of Real-Time Audio Monitoring of Clinical Encounters Michael Sanatani, Kylea Potvin, Henry Conter, Andrew Warner, Kimberly Trudgeon This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-21365/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 25 Jul, 2020 Read the published version in BMC Medical Education → Version 1 posted 8 You are reading this latest preprint version Abstract Background Direct observation is necessary for specific and actionable feedback, however clinicians often struggle to integrate observation into their practice. Remotely audio-monitoring trainees for periods of time may improve the quality of written feedback given to them, and may be a minimally disruptive task for a consultant to perform in a busy clinic. Methods Selected faculty used a wireless audio receiver during the second half of students' oncology rotations to listen to encounters during clinic in real time. They then gave written feedback as per usual practice, as did faculty who did not use the listening-in intervention. Feedback was de-identified and rated, using a rubric, as strong/medium/weak according to consensus of 2/3 rating investigators. Results Monitoring faculty indicated that audio monitoring made the feedback process easier and increased confidence in 95% of encounters. Most students (19/21 respondents) felt monitoring contributed positively to their learning, and included more useful comments. 101 written evaluations were completed by 7 monitoring and 19 non-monitoring faculty. 22/23 (96%) of feedback after monitoring was rated as high quality, compared to 16/37 (43%) (p<0.001) for monitoring faculty before using the equipment (and 20/78 (26%) without monitoring for all consultants (p<0.001)). Conclusions Using live audio monitoring improved the quality of written feedback given to trainees, as judged by the trainees themselves and also using an exploratory grading rubric. The method was well received by both faculty and trainees. Although there are limitations compared to in-the-room observation (body language), the benefits of easy integration into clinical practice and a more natural patient encounter without the observer physically present lead the authors to now use this method routinely while teaching oncology students. Scientific Communication Health Policy Figures Figure 1 Background Direct observation in medical education is a necessary prerequisite activity for the provision of clinical formative feedback ( 1 )( 2 ), and it is thought that the main contribution to enhanced learning from direct observation occurs based on its facilitation of constructive and valid feedback, or “coaching” The combination of initially observing a trainee perform a clinical task, and then having a faculty-trainee dialogue about the current performance, comparing it to an ideally attainable goal and exploring ways to move towards that level, can be called coaching and is a commonly used conceptual framework of clinical and bedside teaching ( 3 ). It is an integral part of competency based medical education (CBME) and introducing it to clinical teaching is one of the main challenges in the transition to CBME. Feedback in the context of clinical coaching has thus been defined as “specific information about the comparison between a trainee’s observed performance and a standard, given with the intent to improve the trainee’s performance.”( 4 ) Several authors have attempted to establish characteristics of feedback that would make it effective in facilitating learning, with several common themes emerging( 5 )( 6 )( 7 ). A systematic review of feedback confirmed that it is effective in promoting physician development, but highlighted that it is not only the content but also aspects of delivery, coach credibility, and receptivity of the trainee that inform effectiveness( 8 ). Nevertheless, certain content characteristics of clinical feedback, whether delivered written or verbal, appear to be universally reported as promoting effectiveness of coaching. Task-oriented feedback identifying very specific and actionable areas of improvement, relevant to agreed-upon learning goals, is felt to be stronger than feedback containing general or value statements focused on the individual’s characteristics( 7 ). Feedback based on actual observation of a trainee performing in a workplace environment is one of the most valued and powerful influences on a trainee( 9 ). Finally, feedback should be delivered in a timely fashion ( 7 ). The role of the quality of feedback has been demonstrated in several studies. Boehler et al demonstrated that knot-tying skills improved more with effective feedback than with praise( 10 ), and Engerer and colleagues showed that communication skills improved more with specific feedback than with general comments ( 11 ). Trainees also value feedback of high quality, yet surveys consistently demonstrate a low rate of feedback felt to be of good quality according to the framework outlined above. Translating the evidence for high quality feedback into daily clinical and teaching practice means overcoming several challenges. Kogan et al recently summarized the evidence guiding direct observation for the purposes of promoting clinical skills in trainees ( 12 ). Most relevant to our study are five statements: first, the observation should be of an authentic clinical encounter. Direct observation, as Kogan et al outline, allows assessment at the “does” level of Miller’s pyramid ( 12 ). We would argue, although this is somewhat controversial ( 13 ), that this holds especially for situations where the trainee is not (or minimally) aware that they are being observed in that moment, as the performance would then be as authentic as possible. Moreover, the presence of an observer in the room may give trainees the impression that assessment was in progress, even if the intended purpose is formative. ( 14 ) Others have tried to diminish this observer effect. Sehgal et al ( 15 ) used a one-way mirror, and trainees indeed indicated they felt their patient interaction was then not adversely affected by the evaluative process. The second guideline statement of particular relevance to our study is “Do assess important clinical skills via direct observation rather than using proxy information”. In order for trainees to find feedback credible, it should be based on direct observation ( 16 ), yet time pressures in clinical practice often discourage faculty from sitting through a complete patient encounter, so as a consequence no observation is done at all. Our study aims to find a way for observation to be possible despite clinical pressures. Third, feedback should not be limited to quantitative ratings ( 12 ). This is especially important for written feedback, which is presented to competency committees and informs decisions about trainee progression through training. At our centre the near exclusive use of numerical ratings without comment limited qualitative assessments of the trainees under the CBME paradigm. Finally, two “don’t knows” are relevant to our study design: “What is the impact of cognitive load during direct observation and what are approaches to mitigate it?” and “What is the optimal duration for direct observation of different clinical skills?”( 12 ) While our study does not address these questions directly, we wished to explore an observation method that would not impose much additional cognitive or time load. By first listening to a trainee exclusively for education, the consultant’s full attention could be focused on the trainee, while deferring patient management thought processes to the traditional case presentation subsequent to the observation. As it is not clear how long an observation duration is “sufficient” for provision of high quality feedback, we would leave this at the discretion of the consultant in the interest of encouraging actual implementation in a busy clinic. While some may choose to listen to the entire encounter, even those just listening for several minutes may observe enough to move beyond the first impression judgment which often informs evaluation ( 17 ). For our study, we introduced remote wireless audio monitoring of third year medical students by using portable microphones and receivers (audio-only monitors) which can be readily introduced to outpatient clinics without any technical infrastructure changes. We had two objectives. First, we wished to examine whether this method would be a feasible way to introduce limited live “observation snapshots” into a busy clinic without disrupting patient care, as determined by faculty. Second, we wished to explore the effect of such observation on the quality of written feedback given to the directly observed trainees, based on the premise that higher quality written feedback promotes learning. Many students on the oncology rotation over the past years had indicated they did not find the currently provided written feedback comments helpful. We theorized that a lack of actual observations was contributing to written feedback felt to be generally ineffective, and that providing even a limited amount of live observation via an audio feed might lead to improvement without imposing too much of a cognitive and time load on faculty. Methods Approval for the study was obtained from the Human Subjects Research Ethics Board at Western University. Written consent was obtained from all consultants taking part in audio monitoring (“monitoring consultants”) as well as from participating third year medical students at the beginning of their oncology clinical rotation. Consultant assignment to the monitoring arm of the study was by volunteering rather than by randomization of all faculty, as there were at the outset insufficient faculty willing to take part in the study and potentially be randomized to the intervention arm. Verbal consent was obtained from any patient being seen in a room in which the microphone was present, by the clinic nurse putting the patient in the room. Eligible trainees were in their third year of medical training at the Schulich School of Medicine and Dentistry at Western University, and enrolled in a two-week clinical oncology selective block. Typically students rotate through outpatient oncology clinics of 3–5 medical and radiation oncology consultants during this time. Information about the study was included in the routine orientation on the first day of the rotation by the rotation supervisors. Students were told that any patient visit during the second week of their rotation might be live monitored via audio feed from the patient room if they worked in a monitoring consultant’s clinics, and they were aware of which consultants were monitoring consultants Monitoring was done via a Williams Sound PPA T46 FM transmitter, with the battery-powered microphone and sending unit hidden in a tissue box in the patient clinic room, and the monitoring consultant clipping the receiver to their clothing or lab coat and plugging in headphones when they wanted to listen in from the workstation or hallways outside the room. Patients were made aware of the microphone in the tissue box by the nurse accompanying them into the clinic room, and instructed on how to turn it off if desired. Students were not made aware that a specific room contained the microphone that day. Monitoring consultants, during the monitored week with the student, were free to use the device as they chose – either to listen to a full encounter or part of it, while physically present at their workstation outside the patient room. In our centre there are often several occasions in a busy clinic when a consultant has a few minutes of inaction waiting for a blood draw, or for a room to become available. Time spent listening to students via the audio monitor thus could be taken from previously unused minutes waiting for a clinic room. Written rotation feedback was completed online as per usual practice, with the addition of an initial first feedback documentation after one week of working with a monitoring consultant, before monitoring started during the second week. This was done to allow for longitudinal sequential comparison of feedback given by the same consultant to one student without and with the benefit of using the audio monitor. Trainees were told which written feedback comments were based on a monitored clinic experience at their end-of-rotation evaluation meeting, and asked to comment on the perceived usefulness of the feedback they received after monitored encounters, compared to that received as per usual practice. Monitoring consultants and students completed a survey rating their satisfaction with the audio listening process and the giving or receipt of feedback, respectively. Written feedback comments were de-identified and rated by two investigators (MS and HC), one of whom was not clinically involved, as strong, weak, or neither strong nor weak based on an exploratory rubric modified from Nesbitt et al. with permission( 18 ), considering also Lefroy et al’s definitions of “DO’s” and “DON’Ts” of feedback (Table 1 ). A third blinded investigator who was not a monitoring consultant (KP) resolved any evaluations that were discordant between rating investigators (MS and HC). Descriptive statistics were generated for all medical students (n = 20), all monitoring consultants (n = 7) and for all evaluations (n = 101). Results were stratified by ( 1 ) audio monitored vs. not audio monitored and ( 2 ) encounter type (audio monitored; not audio monitored but with a monitoring consultant; not audio monitored and a non-monitoring consultant), and compared using the Chi-square test, or Fisher’s Exact test as appropriate. All statistical analysis was performed using SAS version 9.4 software (SAS Institute, Cary NC, USA) using two-sided statistical testing at the 0.05 significance level. Table 1. Exploratory rating framework for weak vs strong written feedback – Feedback Quality Evaluation Form. ♦ Weak ♣ Lacking performance content altogether o E.g.“Saw many cases of lung cancer” ♣ Blank ♣ Nonspecific o E.g. “ A pleasure to work with”, “Functions at PGY-1 level” ♣ Irrelevant o E.g.“Spends much time studying after clinic” ♣ Based on second-hand information o E.g. “Caused pt in Dr. X’s clinic to cry” ♣ Predominantly evaluative without specific aim of performance improvement o E.g.“above average level of knowledge” ♦ Neither weak nor strong ♣ Mentions points of good performance in general o E.g.“Good communicator” ♣ Mentions areas for improvement in general o E.g. “Should take more time with patients” ♦ Strong ♣ Specific areas for improvement o E.g.“Explore symptoms in more depth during review of systems” ♣ Based on direct observation o E.g.“Did not respond to patient comments about anxieties/worries on several occasions – work on questioning further to validate and explore patient concerns” ♣ Relevant to course goals o When presenting case history, lung cases were better presented than prostate cases. Review prognostic features of CA prostate important in initial consultation discussion ♣ Explains the gap [between observed performance and explicit standard] o At this point in training would be expected to develop a differential diagnosis of at least 3 conditions or etiologies underlying a presenting symptom. Tends to focus only on the most likely cause – encouraged to think of other potential causes as well Results Between September 2016 and April 2018, 19 non-monitoring consultants and 7 monitoring consultants completed 101 written evaluations (23 monitored, 78 not monitored) for a total of 20 medical students completing their oncology selective at the London Regional Cancer Program. User acceptance and feasibility Initially, there was some faculty reluctance to consider taking part in the study. However, those consultants who did become monitoring consultants overwhelmingly found the audio monitoring helpful. No monitoring consultants felt that the audio monitoring was detrimental to their teaching activity. On the contrary, consultants appreciated observing behaviour they could give feedback on: I specifically addressed details in his patient interview which I never would have known about without listening in. (Consultant 2) Listening to the interaction allowed one to evaluate multiple things one could not do without the audio monitoring … Tone of voice - appropriate or not… Organization of questions and responses in patient interaction…Responses of trainee to patients answers and comments - were they responding appropriately and directly, would they say 'I'm not sure' if they did not know the answer, used language and tone conveying empathy/compassion/understanding… (Consultant 4) In addition, monitoring consultants emphasized how listening actually saved them time. Also, having already heard what the patient had reported, they had time to plan patient care steps even before the student had presented the case formally: I was able to think about how the trainee was interacting with the patient and formulate my feedback as she was speaking, rather than having to think about it afterward. (Consultant 7) Amongst the monitoring consultants, out of 21 monitored evaluations for which a post-evaluation survey was completed, 20 (95%) indicated that audio monitoring made the feedback process easier and increased confidence, compared to one (5%) who indicated the audio monitoring of that encounter did not significantly increase confidence in evaluation of the student. Student perception From the students’ perspective, out of 21 encounters for which post-encounter surveys were available, the audio monitoring was reported as contributing positively to their learning experience in 19 (95%) of the monitored encounters, not adding anything significant in one (5%), and detrimental in one (5%) case (due to the student reporting feeling more nervous knowing he was being potentially observed). Most students forgot in the moment that they were possibly being observed, which they found beneficial: I was unaware that I was being monitored, therefore I was able to focus more on the patient interview than being evaluated, which was of great benefit. (Student 16) Several students commented on the specificity of the feedback being helpful to their learning, and attributed it directly to the fact that the consultant had credible first-hand knowledge of what happened in the room: I found that when I wasn't monitored feedback was very general, such as case presentations were well organized or read around cases/medications. When monitored, the feedback was much more specific, such as 'I enjoyed that you explained what you were doing during your physical exam' or you used leading questions like 'you aren't having any pain?' instead of 'are you having any pain?'. The physician was able to hear exactly how I was asking questions and interacting with the patients which allowed for more specific and useful feedback of how to improve my patient encounters. (Student 19) The biggest difference for me was the observed encounter allowed my preceptor to get a better idea of where I encounter pitfalls and go astray in my interviews. Previously, they could possibly gather that from my presentation and try to help me correct it, but in this situation, they knew exactly where my mistakes were most egregious and could give me advice on how to better approach these situations and what questions are more appropriate in different scenarios. Knowing what questions to ask is such a struggle at my training level, this is great advice to get. (Student 15) Feedback from live-monitoring was far more detailed and provided specific areas for improvement with appropriate and useful examples. Pragmatically this is very helpful, but the feedback also feels less generic and genuine, I feel more invested in improving these areas than with the feedback without live-monitoring. (Student 8) Feedback quality Feedback comments in unmonitored encounters were generally nonspecific, generally evaluative and with only limited explanation of what trainees could do to improve their skills. In contrast, the vast majority of feedback based on using audio monitoring was qualitatively much more constructive and specific. Representative examples are shown below: Performed well for level of training. She had an organized approach to performing a clinical assessment and asked appropriate follow-up questions based on patient answers. Her interactions with patients were empathetic and friendly. Synthesis of information could be improved by keeping in mind a differential with no more than 3 considerations in order to focus follow-up questions . (feedback rated as “strong” - monitored) Average performance. Some gaps in knowledge base consistent with level . (rated “weak”) Good approach. Detailed review of clinical applicable items. Thorough. Continue to build up on the present base of knowledge . (rated as “neither weak nor strong”) In an effort to quantify this evident qualitative difference, comments were rated according to an exploratory rubric (Table 1 ). For evaluations with included audio monitoring of the trainee, 22/23 (96%) evaluations were rated as high quality, as compared to 20/78 (26%) for all evaluations completed without the consultant having listened in via the audio monitor (p < 0.001) (Fig. 1 ). Considering just consultants who had volunteered to be monitoring consultants, this group gave high quality feedback in 16/37 (43%) of unmonitored evaluations compared to monitored evaluations (p < 0.001). The blinded rating investigators agreed in 70/101 (weighted Kappa 0.69) of evaluations. Discussion To date, studies using recording of trainee-patient interactions have primarily focused on video recording of interactions and providing the recording to the trainee after the encounter for discussion ( 19 )( 20 ). Direct observation with the consultant in the room has been studied in this regard and found to be of benefit in improving comfort with patient care skills, presumably due to the improved feedback and teaching given( 21 ). However, it does require additional time commitment from the consultant and possibly does not represent the way the trainee usually interacts with patients. Our study demonstrated that most of the written feedback provided to students on their oncology rotations in the absence of direct observation by the majority of faculty is likely of low educational value. Student comments reflect a qualitative difference in the written feedback they received after audio monitoring compared to what they were used to previously or with non-monitoring consultants, with half the students mentioning feedback specificity as a noticeable improvement. This report from students qualitatively matches the findings from the external feedback ratings according to the feedback rubric. Patients, surveyed informally, also had no concerns and welcomed the fact that the consultant was “present” and hearing what they were saying and several mentioned being reassured as the trainee was being actively supervised and monitored. Feasibility barriers included cost ($1200 Canadian for one set of transmitter/receiver) and occasional static interference from wireless computer mice and keyboards. Our host hospital mandated compatibility with telemetry equipment, which very few other more affordable portable transmitters (such as baby monitors) can demonstrate. Limitations While the comments and ratings by both faculty and trainees appear to give a clearly supportive picture, limitations in regards to the precise value of the numerical feedback ratings need to be acknowledged as this was not a formal randomized study. As volunteers, monitoring consultants may have been more invested in education and thus give better feedback. In fact, unmonitored/baseline evaluations differed significantly between monitoring consultants and the control group (Fig. 1 ), if the results are stratified by consultant type, indicating that monitoring consultants were able to provide higher quality feedback even without audio equipment. It is unknown whether mandating use of audio monitoring would improve feedback quality given by all consultants, or whether just training in feedback techniques would be more effective. That being said, even monitoring consultants had difficulty giving truly specific feedback without observation of the trainee, as can be expected( 22 ). Another limitation is that we only examined written feedback. Verbal in-the-moment feedback may have been provided to the trainees, and arguably would represent a more effective formative feedback process than the written feedback. Visual cues were also not assessed, for example, body language. The Completed Clinical Evaluation Report Rating (CCERR) instrument, in contrast to our rubric, has been previously validated in terms of assessing clinical assessment, however the absence of graduated numerical ratings in our standard medical student evaluation forms would have made it difficult to apply the CCERR( 23 ). Finally, as Pelgrim et al. ( 24 ) point out, many studies of assessment tools and procedures examine learner attitudes towards the assessment procedure rather than their effect on learning outcomes, and our study is no exception. The question of the impact of audio monitoring not just on feedback quality but also on actual patient-trainee communication skill development must also be addressed. Overall however, based on our experience, we would encourage other centres to consider using live audio monitoring to introduce at least some of the benefits of direct observation to outpatient medicine. This may simultaneously address faculty time burden concerns regarding in-room observation and pave the way to greater acceptance of observation as an integral part of clinical education, especially during the current phase of transition to competency-based medical education. Declarations Ethics approval and consent to participate Approval for the study was obtained from the Human Subjects Research Ethics Board at Western University. Consent for publication Not applicable – no identifiable data Availability of data and materials The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Competing interests The authors declare that they have no competing interests Funding This study was unfunded Authors' contributions MS, KP, and AW were involved in writing the manuscript. AW performed data analysis and reporting. HC edited the manuscript. MS, KP, and HC were blinded raters. KT obtained consent from the students, collected and prepared the data forms, and was involved in study design. All authors read and approved the final manuscript. References http://doi.wiley.com/10.1002/msj.20123 Fromme HB, Karani R, Downing SM. Direct Observation in Medical Education: A Review of the Literature and Evidence for Validity. Mt Sinai J Med A J Transl Pers Med [Internet]. 2009 Aug 1 [cited 2019 May 3];76(4):365–71. Available from: . Awad Z, Hayden L, Muthuswamy K, Ziprin P, Darzi A, Tolley NS. Does direct observation of procedural skills reflect trainee’s progress in otolaryngology? Clin Otolaryngol. 2014;39(3):169–73. 10.1111/medu.13818 Armson H, Lockyer JM, Zetkulic M, Könings KD, Sargeant J. Identifying coaching skills to improve feedback use in postgraduate medical education. Med Educ [Internet]. 2019 May 18 [cited 2019 May 3];53(5):477–93. Available from: . http://doi.wiley.com/10.1111/j.1365-2923.2007.02973.x Van De Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OTJ. What is feedback in clinical education? Med Educ [Internet]. 2008 Jan 22 [cited 2019 May 3];42(2):189–97. Available from: . Bartlett M, Crossley J, McKinley R. Improving the quality of written feedback using written feedback. Educ Prim Care. 2017;28(1):16–22. Lefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ [Internet]. 2015 [cited 2019 May 7];4(6):284–99. Available from: . 10.1111/j.1365-2923.2009.03546.x Archer JC. State of the science in health professional education: effective feedback. Med Educ [Internet]. 2010 Jan 1 [cited 2019 May 3];44(1):101–8. Available from: . Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DW. Systematic review of the literature on assessment, feedback and physicians’ clinical performance*: BEME Guide No. 7. Vol. 28, Medical Teacher. 2006. p. 117–28. Norcini J. The power of feedback. Med Educ [Internet]. 2010 Jan [cited 2019 May 24];44(1):16–7. Available from: . http://doi.wiley.com/10.1111/j.1365-2929.2006.02503.x Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, et al. An investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ [Internet]. 2006 Aug 1 [cited 2019 May 7];40(8):746–9. Available from: . 10.1186/s12909-019-1470-9 Engerer C, Berberat PO, Dinkel A, Rudolph B, Sattel H, Wuensch A. Specific feedback makes medical students better communicators. BMC Med Educ [Internet]. 2019 Dec 8 [cited 2019 May 3];19(1):51. Available from: . 10.1007/s40037-017-0376-7 Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines. The do’s, don’ts and don’t knows of direct observation of clinical skills in medical education. Perspect Med Educ [Internet]. 2017 Oct 27 [cited 2019 May 3];6(5):286–305. Available from: . LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. Staging a performance: learners’ perceptions about direct observation during residency. Med Educ [Internet]. 2017 May [cited 2019 May 3];51(5):498–510. Available from: . http://doi.wiley.com/10.1111/medu.13232 LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. Staging a performance: learners’ perceptions about direct observation during residency. Med Educ [Internet]. 2017 May 1 [cited 2019 May 3];51(5):498–510. Available from: . http://doi.wiley.com/10.1111/tct.12140 Sehgal R, Hardman J, Haney E. Observing trainee encounters using a one-way mirror. Clin Teach [Internet]. 2014 Jul [cited 2019 May 3];11(4):247–50. Available from: . Watling C, Driessen E, van der Vleuten CPM, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements. Med Educ [Internet]. 2012 Feb [cited 2019 May 10];46(2):192–200. Available from: . Wood TJ. Exploring the role of first impressions in rater-based assessments. Adv Heal Sci Educ. 2014;19(3):409–27. Nesbitt A, Pitcher A, James L, Sturrock A, Griffin A. Written feedback on supervised learning events. Clin Teach [Internet]. 2014 Jul [cited 2019 Mar 29];11(4):279–83. Available from: . Hawkins SC, Osborne A, Schofield SJ, Pournaras DJ, Chester JF. Improving the accuracy of self-assessment of practical clinical skills using video feedback – The importance of including benchmarks. Med Teach [Internet]. 2012 Apr 28 [cited 2019 May 24];34(4):279–84. Available from: . Zick A, Granieri M, Makoul G. First-year medical students’ assessment of their own communication skills: A video-based, open-ended approach. Patient Educ Couns [Internet]. 2007 Oct [cited 2019 May 24];68(2):161–6. Available from: . Shelesky G, D’Amico F, Marfatia R, Munshi A, Wilson SA. Does weekly direct observation and formal feedback improve intern patient care skills development? A randomized controlled trial. Fam Med [Internet]. [cited 2019 May 24];44(7):486–92. Available from: . Kogan JR, Holmboe ES, Hauer KE. Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees. JAMA [Internet]. 2009 Sep 23 [cited 2019 May 3];302(12):1316. Available from: .2009.1365. Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisors’ completed clinical evaluation reports. Med Educ [Internet]. 2008 [cited 2020 Apr 2];42(8):816–22. Available from: 9834-1fe3-418d-9223-58d8db53c677%40sdc-v-sessmgr02. Pelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol RPTM, van der Vleuten CPM. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract [Internet]. 2011 Mar [cited 2019 May 24];16(1):131–42. Available from: . Cite Share Download PDF Status: Published Journal Publication published 25 Jul, 2020 Read the published version in BMC Medical Education → Version 1 posted Editorial decision: Major revision 04 May, 2020 Review # 1 received at journal 21 Apr, 2020 Editor assigned by journal 16 Apr, 2020 Reviewers invited by journal 16 Apr, 2020 Reviewer # 1 agreed at journal 16 Apr, 2020 Submission checks completed at journal 07 Apr, 2020 Editor invited by journal 07 Apr, 2020 First submitted to journal 03 Apr, 2020 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-21365","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research article","associatedPublications":[],"authors":[{"id":475000,"identity":"798da525-0259-4464-a853-c2d42ed538e2","order_by":1,"name":"Michael Sanatani","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEElEQVRIie3RMWuDQBTA8QfCuTyb9Q5D8wkKVw6uGYR+Fik4SSh0cchQCOgi9av0IxgepEtCV4cOmZwchEBxaegVS4egpHTqcP9BuJOf9/AAbLb/mAPMPOc/a5y4+CvC+0VpiFidI3BCQNIZcpU59b5LOMyy3frQLt+mijxqOwgW4NJ+iGhiN9f5loPcLu54ualR00UkcogeACM5TJBxLzUEYslLRoagdgAofPzaHCHi42gGKxrVlUdCtUJ16InbjhHfMx+EKtZ8nRJKB6XfExw5hWl/ujFvq0bPd0+E3AwmchmFKcb3g+SVatEsg8tZEasqeafbSZartkuCsHBfngf/8nendyH7+7LZbDbb3/oE5fxVh3/zgpsAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-2423-7171","institution":"Western University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Sanatani","suffix":""},{"id":475001,"identity":"7696c991-1852-4599-8d29-97f7e8597dd1","order_by":2,"name":"Kylea Potvin","email":"","orcid":"","institution":"Western University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kylea","middleName":"","lastName":"Potvin","suffix":""},{"id":475002,"identity":"081c18c4-0dc2-4007-9a30-bcf376f4177f","order_by":3,"name":"Henry Conter","email":"","orcid":"","institution":"Western University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Henry","middleName":"","lastName":"Conter","suffix":""},{"id":482317,"identity":"3932d168-6319-47f0-8d36-98c8d375aeaf","order_by":4,"name":"Andrew Warner","email":"","orcid":"","institution":"Western University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Andrew","middleName":"","lastName":"Warner","suffix":""},{"id":475003,"identity":"71224110-045f-4cae-8719-aaadccbcfbb3","order_by":5,"name":"Kimberly Trudgeon","email":"","orcid":"","institution":"Western University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kimberly","middleName":"","lastName":"Trudgeon","suffix":""}],"badges":[],"createdAt":"2020-04-04 12:02:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-21365/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-21365/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12909-020-02158-6","type":"published","date":"2020-07-25T12:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":877727,"identity":"195c378e-e37d-409a-89b8-0ca6c1371907","added_by":"auto","created_at":"2020-04-09 22:03:28","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":498287,"visible":true,"origin":"","legend":"Comparison of written feedback quality after audio-monitored and non-monitored outpatient oncology rotation weeks for (A) monitoring consultants using (n=23 ratings) versus not using (n=37 ratings) audio equipment and (B) non-monitoring consultants (n= 41 ratings) versus monitoring consultants using audio equipment. Asterisk (*) indicates significant difference (p \u003c 0.001)","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-21365/v1/1.png"},{"id":13497934,"identity":"062f6b8f-666c-4dbd-bb69-7d86300f5f26","added_by":"auto","created_at":"2021-09-16 22:55:53","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":595317,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-21365/v1/32be9d5b-25db-4b2c-98a3-933a990211eb.pdf"}],"financialInterests":"","formattedTitle":"Quality of Written Feedback given to Medical Students after Introduction of Real-Time Audio Monitoring of Clinical Encounters","fulltext":[{"header":"Background","content":" \u003cp\u003eDirect observation in medical education is a necessary prerequisite activity for the provision of clinical formative feedback (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e)(\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e), and it is thought that the main contribution to enhanced learning from direct observation occurs based on its facilitation of constructive and valid feedback, or \u0026ldquo;coaching\u0026rdquo; The combination of initially observing a trainee perform a clinical task, and then having a faculty-trainee dialogue about the current performance, comparing it to an ideally attainable goal and exploring ways to move towards that level, can be called coaching and is a commonly used conceptual framework of clinical and bedside teaching (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). It is an integral part of competency based medical education (CBME) and introducing it to clinical teaching is one of the main challenges in the transition to CBME.\u003c/p\u003e \u003cp\u003eFeedback in the context of clinical coaching has thus been defined as \u0026ldquo;specific information about the comparison between a trainee\u0026rsquo;s observed performance and a standard, given with the intent to improve the trainee\u0026rsquo;s performance.\u0026rdquo;(\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e) Several authors have attempted to establish characteristics of feedback that would make it effective in facilitating learning, with several common themes emerging(\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e)(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e)(\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e). A systematic review of feedback confirmed that it is effective in promoting physician development, but highlighted that it is not only the content but also aspects of delivery, coach credibility, and receptivity of the trainee that inform effectiveness(\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e). Nevertheless, certain content characteristics of clinical feedback, whether delivered written or verbal, appear to be universally reported as promoting effectiveness of coaching. Task-oriented feedback identifying very specific and actionable areas of improvement, relevant to agreed-upon learning goals, is felt to be stronger than feedback containing general or value statements focused on the individual\u0026rsquo;s characteristics(\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e). Feedback based on actual observation of a trainee performing in a workplace environment is one of the most valued and powerful influences on a trainee(\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e). Finally, feedback should be delivered in a timely fashion (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e). The role of the quality of feedback has been demonstrated in several studies. Boehler et al demonstrated that knot-tying skills improved more with effective feedback than with praise(\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e), and Engerer and colleagues showed that communication skills improved more with specific feedback than with general comments (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). Trainees also value feedback of high quality, yet surveys consistently demonstrate a low rate of feedback felt to be of good quality according to the framework outlined above.\u003c/p\u003e \u003cp\u003eTranslating the evidence for high quality feedback into daily clinical and teaching practice means overcoming several challenges. Kogan et al recently summarized the evidence guiding direct observation for the purposes of promoting clinical skills in trainees (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). Most relevant to our study are five statements: first, the observation should be of an authentic clinical encounter. Direct observation, as Kogan et al outline, allows assessment at the \u0026ldquo;does\u0026rdquo; level of Miller\u0026rsquo;s pyramid (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). We would argue, although this is somewhat controversial (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e), that this holds especially for situations where the trainee is not (or minimally) aware that they are being observed in that moment, as the performance would then be as authentic as possible. Moreover, the presence of an observer in the room may give trainees the impression that assessment was in progress, even if the intended purpose is formative. (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e) Others have tried to diminish this observer effect. Sehgal et al (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e) used a one-way mirror, and trainees indeed indicated they felt their patient interaction was then not adversely affected by the evaluative process. The second guideline statement of particular relevance to our study is \u0026ldquo;Do assess important clinical skills via direct observation rather than using proxy information\u0026rdquo;. In order for trainees to find feedback credible, it should be based on direct observation (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e), yet time pressures in clinical practice often discourage faculty from sitting through a complete patient encounter, so as a consequence no observation is done at all. Our study aims to find a way for observation to be possible despite clinical pressures. Third, feedback should not be limited to quantitative ratings (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e). This is especially important for written feedback, which is presented to competency committees and informs decisions about trainee progression through training. At our centre the near exclusive use of numerical ratings without comment limited qualitative assessments of the trainees under the CBME paradigm. Finally, two \u0026ldquo;don\u0026rsquo;t knows\u0026rdquo; are relevant to our study design: \u0026ldquo;What is the impact of cognitive load during direct observation and what are approaches to mitigate it?\u0026rdquo; and \u0026ldquo;What is the optimal duration for direct observation of different clinical skills?\u0026rdquo;(\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e) While our study does not address these questions directly, we wished to explore an observation method that would not impose much additional cognitive or time load. By first listening to a trainee exclusively for education, the consultant\u0026rsquo;s full attention could be focused on the trainee, while deferring patient management thought processes to the traditional case presentation subsequent to the observation. As it is not clear how long an observation duration is \u0026ldquo;sufficient\u0026rdquo; for provision of high quality feedback, we would leave this at the discretion of the consultant in the interest of encouraging actual implementation in a busy clinic. While some may choose to listen to the entire encounter, even those just listening for several minutes may observe enough to move beyond the first impression judgment which often informs evaluation (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eFor our study, we introduced remote wireless audio monitoring of third year medical students by using portable microphones and receivers (audio-only monitors) which can be readily introduced to outpatient clinics without any technical infrastructure changes. We had two objectives. First, we wished to examine whether this method would be a feasible way to introduce limited live \u0026ldquo;observation snapshots\u0026rdquo; into a busy clinic without disrupting patient care, as determined by faculty. Second, we wished to explore the effect of such observation on the quality of written feedback given to the directly observed trainees, based on the premise that higher quality written feedback promotes learning. Many students on the oncology rotation over the past years had indicated they did not find the currently provided written feedback comments helpful. We theorized that a lack of actual observations was contributing to written feedback felt to be generally ineffective, and that providing even a limited amount of live observation via an audio feed might lead to improvement without imposing too much of a cognitive and time load on faculty.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eApproval for the study was obtained from the Human Subjects Research Ethics Board at Western University. Written consent was obtained from all consultants taking part in audio monitoring (\u0026ldquo;monitoring consultants\u0026rdquo;) as well as from participating third year medical students at the beginning of their oncology clinical rotation. Consultant assignment to the monitoring arm of the study was by volunteering rather than by randomization of all faculty, as there were at the outset insufficient faculty willing to take part in the study and potentially be randomized to the intervention arm.\u003c/p\u003e\n\u003cp\u003eVerbal consent was obtained from any patient being seen in a room in which the microphone was present, by the clinic nurse putting the patient in the room.\u003c/p\u003e\n\u003cp\u003eEligible trainees were in their third year of medical training at the Schulich School of Medicine and Dentistry at Western University, and enrolled in a two-week clinical oncology selective block. Typically students rotate through outpatient oncology clinics of 3\u0026ndash;5 medical and radiation oncology consultants during this time. Information about the study was included in the routine orientation on the first day of the rotation by the rotation supervisors. Students were told that any patient visit during the second week of their rotation might be live monitored via audio feed from the patient room if they worked in a monitoring consultant\u0026rsquo;s clinics, and they were aware of which consultants were monitoring consultants\u003c/p\u003e\n\u003cp\u003eMonitoring was done via a Williams Sound PPA T46 FM transmitter, with the battery-powered microphone and sending unit hidden in a tissue box in the patient clinic room, and the monitoring consultant clipping the receiver to their clothing or lab coat and plugging in headphones when they wanted to listen in from the workstation or hallways outside the room. Patients were made aware of the microphone in the tissue box by the nurse accompanying them into the clinic room, and instructed on how to turn it off if desired. Students were not made aware that a specific room contained the microphone that day. Monitoring consultants, during the monitored week with the student, were free to use the device as they chose \u0026ndash; either to listen to a full encounter or part of it, while physically present at their workstation outside the patient room. In our centre there are often several occasions in a busy clinic when a consultant has a few minutes of inaction waiting for a blood draw, or for a room to become available. Time spent listening to students via the audio monitor thus could be taken from previously unused minutes waiting for a clinic room.\u003c/p\u003e\n\u003cp\u003eWritten rotation feedback was completed online as per usual practice, with the addition of an initial first feedback documentation after one week of working with a monitoring consultant, before monitoring started during the second week. This was done to allow for longitudinal sequential comparison of feedback given by the same consultant to one student without and with the benefit of using the audio monitor.\u003c/p\u003e\n\u003cp\u003eTrainees were told which written feedback comments were based on a monitored clinic experience at their end-of-rotation evaluation meeting, and asked to comment on the perceived usefulness of the feedback they received after monitored encounters, compared to that received as per usual practice. Monitoring consultants and students completed a survey rating their satisfaction with the audio listening process and the giving or receipt of feedback, respectively.\u003c/p\u003e\n\u003cp\u003eWritten feedback comments were de-identified and rated by two investigators (MS and HC), one of whom was not clinically involved, as strong, weak, or neither strong nor weak based on an exploratory rubric modified from Nesbitt et al. with permission(\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e), considering also Lefroy et al\u0026rsquo;s definitions of \u0026ldquo;DO\u0026rsquo;s\u0026rdquo; and \u0026ldquo;DON\u0026rsquo;Ts\u0026rdquo; of feedback (Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). A third blinded investigator who was not a monitoring consultant (KP) resolved any evaluations that were discordant between rating investigators (MS and HC).\u003c/p\u003e\n\u003cp\u003eDescriptive statistics were generated for all medical students (n\u0026thinsp;=\u0026thinsp;20), all monitoring consultants (n\u0026thinsp;=\u0026thinsp;7) and for all evaluations (n\u0026thinsp;=\u0026thinsp;101). Results were stratified by (\u003cspan class=\"CitationRef\"\u003e1\u003c/span\u003e) audio monitored vs. not audio monitored and (\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e) encounter type (audio monitored; not audio monitored but with a monitoring consultant; not audio monitored and a non-monitoring consultant), and compared using the Chi-square test, or Fisher\u0026rsquo;s Exact test as appropriate. All statistical analysis was performed using SAS version 9.4 software (SAS Institute, Cary NC, USA) using two-sided statistical testing at the 0.05 significance level.\u003c/p\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 1. Exploratory rating framework for weak vs strong written feedback \u0026ndash; Feedback Quality Evaluation Form.\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab1\" border=\"1\"\u003e\u003ccaption\u003e\u0026nbsp;\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026diams; Weak\u003c/div\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Lacking performance content altogether\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;Saw many cases of lung cancer\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Blank\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Nonspecific\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo E.g. \u0026ldquo;\u003cem\u003eA pleasure to work with\u0026rdquo;, \u0026ldquo;Functions at PGY-1 level\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Irrelevant\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;Spends much time studying after clinic\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Based on second-hand information\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g. \u0026ldquo;Caused pt in Dr. X\u0026rsquo;s clinic to cry\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Predominantly evaluative without specific aim of performance improvement\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;above average level of knowledge\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026diams; \u003cstrong\u003eNeither weak nor strong\u003c/strong\u003e\u003c/div\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Mentions points of good performance in general\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;Good communicator\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Mentions areas for improvement in general\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g. \u0026ldquo;Should take more time with patients\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026diams; \u003cstrong\u003eStrong\u003c/strong\u003e\u003c/div\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; \u003cstrong\u003eSpecific areas\u003c/strong\u003e for improvement\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;Explore symptoms in more depth during review of systems\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Based on \u003cstrong\u003edirect observation\u003c/strong\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003eo \u003cem\u003eE.g.\u0026ldquo;Did not respond to patient comments about anxieties/worries on several occasions \u0026ndash; work on questioning further to validate and explore patient concerns\u0026rdquo;\u003c/em\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; Relevant to \u003cstrong\u003ecourse goals\u003c/strong\u003e\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\n\u003cdiv class=\"Quote\"\u003eo When presenting case history, lung cases were better presented than prostate cases. Review prognostic features of CA prostate important in initial consultation discussion\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\u0026clubs; \u003cstrong\u003eExplains the gap\u003c/strong\u003e [between observed performance and explicit standard]\u003c/div\u003e\n\u003cdiv class=\"SimplePara\"\u003e\n\u003cdiv class=\"Quote\"\u003eo At this point in training would be expected to develop a differential diagnosis of at least 3 conditions or etiologies underlying a presenting symptom. Tends to focus only on the most likely cause \u0026ndash; encouraged to think of other potential causes as well\u003c/div\u003e\n\u003c/div\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e"},{"header":"Results","content":" \u003cp\u003eBetween September 2016 and April 2018, 19 non-monitoring consultants and 7 monitoring consultants completed 101 written evaluations (23 monitored, 78 not monitored) for a total of 20 medical students completing their oncology selective at the London Regional Cancer Program.\u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eUser acceptance and feasibility\u003c/h2\u003e \u003cp\u003eInitially, there was some faculty reluctance to consider taking part in the study. However, those consultants who did become monitoring consultants overwhelmingly found the audio monitoring helpful. No monitoring consultants felt that the audio monitoring was detrimental to their teaching activity. On the contrary, consultants appreciated observing behaviour they could give feedback on:\u003c/p\u003e \u003cp\u003e \u003cem\u003eI specifically addressed details in his patient interview which I never would have known about without listening in. (Consultant 2)\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eListening to the interaction allowed one to evaluate multiple things one could not do without the audio monitoring \u0026hellip; Tone of voice - appropriate or not\u0026hellip; Organization of questions and responses in patient interaction\u0026hellip;Responses of trainee to patients answers and comments - were they responding appropriately and directly, would they say 'I'm not sure' if they did not know the answer, used language and tone conveying empathy/compassion/understanding\u0026hellip; (Consultant 4)\u003c/em\u003e \u003c/p\u003e \u003cp\u003eIn addition, monitoring consultants emphasized how listening actually saved them time. Also, having already heard what the patient had reported, they had time to plan patient care steps even before the student had presented the case formally:\u003c/p\u003e \u003cp\u003e \u003cem\u003eI was able to think about how the trainee was interacting with the patient and formulate my feedback as she was speaking, rather than having to think about it afterward. (Consultant 7)\u003c/em\u003e \u003c/p\u003e \u003cp\u003eAmongst the monitoring consultants, out of 21 monitored evaluations for which a post-evaluation survey was completed, 20 (95%) indicated that audio monitoring made the feedback process easier and increased confidence, compared to one (5%) who indicated the audio monitoring of that encounter did not significantly increase confidence in evaluation of the student.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eStudent perception\u003c/h2\u003e \u003cp\u003eFrom the students\u0026rsquo; perspective, out of 21 encounters for which post-encounter surveys were available, the audio monitoring was reported as contributing positively to their learning experience in 19 (95%) of the monitored encounters, not adding anything significant in one (5%), and detrimental in one (5%) case (due to the student reporting feeling more nervous knowing he was being potentially observed). Most students forgot in the moment that they were possibly being observed, which they found beneficial:\u003c/p\u003e \u003cp\u003e \u003cem\u003eI was unaware that I was being monitored, therefore I was able to focus more on the patient interview than being evaluated, which was of great benefit. (Student 16)\u003c/em\u003e \u003c/p\u003e \u003cp\u003eSeveral students commented on the specificity of the feedback being helpful to their learning, and attributed it directly to the fact that the consultant had credible first-hand knowledge of what happened in the room:\u003c/p\u003e \u003cp\u003e \u003cem\u003eI found that when I wasn't monitored feedback was very general, such as case presentations were well organized or read around cases/medications. When monitored, the feedback was much more specific, such as 'I enjoyed that you explained what you were doing during your physical exam' or you used leading questions like 'you aren't having any pain?' instead of 'are you having any pain?'. The physician was able to hear exactly how I was asking questions and interacting with the patients which allowed for more specific and useful feedback of how to improve my patient encounters. (Student 19)\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eThe biggest difference for me was the observed encounter allowed my preceptor to get a better idea of where I encounter pitfalls and go astray in my interviews. Previously, they could possibly gather that from my presentation and try to help me correct it, but in this situation, they knew exactly where my mistakes were most egregious and could give me advice on how to better approach these situations and what questions are more appropriate in different scenarios. Knowing what questions to ask is such a struggle at my training level, this is great advice to get. (Student 15)\u003c/em\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eFeedback from live-monitoring was far more detailed and provided specific areas for improvement with appropriate and useful examples. Pragmatically this is very helpful, but the feedback also feels less generic and genuine, I feel more invested in improving these areas than with the feedback without live-monitoring. (Student 8)\u003c/em\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eFeedback quality\u003c/h2\u003e \u003cp\u003eFeedback comments in unmonitored encounters were generally nonspecific, generally evaluative and with only limited explanation of what trainees could do to improve their skills. In contrast, the vast majority of feedback based on using audio monitoring was qualitatively much more constructive and specific. Representative examples are shown below:\u003c/p\u003e \u003cp\u003e \u003cem\u003ePerformed well for level of training. She had an organized approach to performing a clinical assessment and asked appropriate follow-up questions based on patient answers. Her interactions with patients were empathetic and friendly. Synthesis of information could be improved by keeping in mind a differential with no more than 3 considerations in order to focus follow-up questions\u003c/em\u003e. (feedback rated as \u0026ldquo;strong\u0026rdquo; - monitored)\u003c/p\u003e \u003cp\u003e \u003cem\u003eAverage performance. Some gaps in knowledge base consistent with level\u003c/em\u003e. (rated \u0026ldquo;weak\u0026rdquo;)\u003c/p\u003e \u003cp\u003e \u003cem\u003eGood approach. Detailed review of clinical applicable items. Thorough. Continue to build up on the present base of knowledge\u003c/em\u003e. (rated as \u0026ldquo;neither weak nor strong\u0026rdquo;)\u003c/p\u003e \u003cp\u003eIn an effort to quantify this evident qualitative difference, comments were rated according to an exploratory rubric (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). For evaluations with included audio monitoring of the trainee, 22/23 (96%) evaluations were rated as high quality, as compared to 20/78 (26%) for all evaluations completed without the consultant having listened in via the audio monitor (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Considering just consultants who had volunteered to be monitoring consultants, this group gave high quality feedback in 16/37 (43%) of unmonitored evaluations compared to monitored evaluations (p\u0026thinsp;\u0026lt;\u0026thinsp;0.001). The blinded rating investigators agreed in 70/101 (weighted Kappa 0.69) of evaluations.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e "},{"header":"Discussion","content":"\u003cp\u003eTo date, studies using recording of trainee-patient interactions have primarily focused on video recording of interactions and providing the recording to the trainee after the encounter for discussion (\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e)(\u003cspan class=\"CitationRef\"\u003e20\u003c/span\u003e). Direct observation with the consultant in the room has been studied in this regard and found to be of benefit in improving comfort with patient care skills, presumably due to the improved feedback and teaching given(\u003cspan class=\"CitationRef\"\u003e21\u003c/span\u003e). However, it does require additional time commitment from the consultant and possibly does not represent the way the trainee usually interacts with patients.\u003c/p\u003e\n\u003cp\u003eOur study demonstrated that most of the written feedback provided to students on their oncology rotations in the absence of direct observation by the majority of faculty is likely of low educational value. Student comments reflect a qualitative difference in the written feedback they received after audio monitoring compared to what they were used to previously or with non-monitoring consultants, with half the students mentioning feedback \u003cem\u003especificity\u003c/em\u003e as a noticeable improvement. This report from students qualitatively matches the findings from the external feedback ratings according to the feedback rubric.\u003c/p\u003e\n\u003cp\u003ePatients, surveyed informally, also had no concerns and welcomed the fact that the consultant was \u0026ldquo;present\u0026rdquo; and hearing what they were saying and several mentioned being reassured as the trainee was being actively supervised and monitored. Feasibility barriers included cost ($1200 Canadian for one set of transmitter/receiver) and occasional static interference from wireless computer mice and keyboards. Our host hospital mandated compatibility with telemetry equipment, which very few other more affordable portable transmitters (such as baby monitors) can demonstrate.\u003c/p\u003e\n\u003cp\u003eLimitations\u003c/p\u003e\n\u003cp\u003eWhile the comments and ratings by both faculty and trainees appear to give a clearly supportive picture, limitations in regards to the precise value of the numerical feedback ratings need to be acknowledged as this was not a formal randomized study. As volunteers, monitoring consultants may have been more invested in education and thus give better feedback. In fact, unmonitored/baseline evaluations differed significantly between monitoring consultants and the control group (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e), if the results are stratified by consultant type, indicating that monitoring consultants were able to provide higher quality feedback even without audio equipment. It is unknown whether mandating use of audio monitoring would improve feedback quality given by all consultants, or whether just training in feedback techniques would be more effective. That being said, even monitoring consultants had difficulty giving truly specific feedback without observation of the trainee, as can be expected(\u003cspan class=\"CitationRef\"\u003e22\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eAnother limitation is that we only examined written feedback. Verbal in-the-moment feedback may have been provided to the trainees, and arguably would represent a more effective formative feedback process than the written feedback. Visual cues were also not assessed, for example, body language. The Completed Clinical Evaluation Report Rating (CCERR) instrument, in contrast to our rubric, has been previously validated in terms of assessing clinical assessment, however the absence of graduated numerical ratings in our standard medical student evaluation forms would have made it difficult to apply the CCERR(\u003cspan class=\"CitationRef\"\u003e23\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eFinally, as Pelgrim et al. (\u003cspan class=\"CitationRef\"\u003e24\u003c/span\u003e) point out, many studies of assessment tools and procedures examine learner attitudes towards the assessment procedure rather than their effect on learning outcomes, and our study is no exception. The question of the impact of audio monitoring not just on feedback quality but also on actual patient-trainee communication skill development must also be addressed.\u003c/p\u003e\n\u003cp\u003eOverall however, based on our experience, we would encourage other centres to consider using live audio monitoring to introduce at least some of the benefits of direct observation to outpatient medicine. This may simultaneously address faculty time burden concerns regarding in-room observation and pave the way to greater acceptance of observation as an integral part of clinical education, especially during the current phase of transition to competency-based medical education.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003eEthics approval and consent to participate\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eApproval for the study was obtained from the Human Subjects Research Ethics Board at Western University.\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eNot applicable \u0026ndash; no identifiable data\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eThe datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eThe authors declare that they have no competing interests\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eThis study was unfunded\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAuthors' contributions\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eMS, KP, and AW were involved in writing the manuscript.\u0026nbsp; AW performed data analysis and reporting. HC edited the manuscript. MS, KP, and HC were blinded raters. KT obtained consent from the students, collected and prepared the data forms, and was involved in study design. All authors read and approved the final manuscript.\u003c/em\u003e\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003ehttp://doi.wiley.com/10.1002/msj.20123\u003c/div\u003e \u003cspan\u003eFromme HB, Karani R, Downing SM. Direct Observation in Medical Education: A Review of the Literature and Evidence for Validity. Mt Sinai J Med A J Transl Pers Med [Internet]. 2009 Aug 1 [cited 2019 May 3];76(4):365\u0026ndash;71. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eAwad Z, Hayden L, Muthuswamy K, Ziprin P, Darzi A, Tolley NS. Does direct observation of procedural skills reflect trainee\u0026rsquo;s progress in otolaryngology? Clin Otolaryngol. 2014;39(3):169\u0026ndash;73.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003e10.1111/medu.13818\u003c/div\u003e \u003cspan\u003eArmson H, Lockyer JM, Zetkulic M, K\u0026ouml;nings KD, Sargeant J. Identifying coaching skills to improve feedback use in postgraduate medical education. Med Educ [Internet]. 2019 May 18 [cited 2019 May 3];53(5):477\u0026ndash;93. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003ehttp://doi.wiley.com/10.1111/j.1365-2923.2007.02973.x\u003c/div\u003e \u003cspan\u003eVan De Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OTJ. What is feedback in clinical education? Med Educ [Internet]. 2008 Jan 22 [cited 2019 May 3];42(2):189\u0026ndash;97. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eBartlett M, Crossley J, McKinley R. Improving the quality of written feedback using written feedback. Educ Prim Care. 2017;28(1):16\u0026ndash;22.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eLefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do\u0026rsquo;s, don\u0026rsquo;ts and don\u0026rsquo;t knows of feedback for clinical education. Perspect Med Educ [Internet]. 2015 [cited 2019 May 7];4(6):284\u0026ndash;99. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003e10.1111/j.1365-2923.2009.03546.x\u003c/div\u003e \u003cspan\u003eArcher JC. State of the science in health professional education: effective feedback. Med Educ [Internet]. 2010 Jan 1 [cited 2019 May 3];44(1):101\u0026ndash;8. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eVeloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DW. Systematic review of the literature on assessment, feedback and physicians\u0026rsquo; clinical performance*: BEME Guide No. 7. Vol.\u0026nbsp;28, Medical Teacher. 2006. p.\u0026nbsp;117\u0026ndash;28.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eNorcini J. The power of feedback. Med Educ [Internet]. 2010 Jan [cited 2019 May 24];44(1):16\u0026ndash;7. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003ehttp://doi.wiley.com/10.1111/j.1365-2929.2006.02503.x\u003c/div\u003e \u003cspan\u003eBoehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, et al. An investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ [Internet]. 2006 Aug 1 [cited 2019 May 7];40(8):746\u0026ndash;9. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003e10.1186/s12909-019-1470-9\u003c/div\u003e \u003cspan\u003eEngerer C, Berberat PO, Dinkel A, Rudolph B, Sattel H, Wuensch A. Specific feedback makes medical students better communicators. BMC Med Educ [Internet]. 2019 Dec 8 [cited 2019 May 3];19(1):51. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003e10.1007/s40037-017-0376-7\u003c/div\u003e \u003cspan\u003eKogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines. The do\u0026rsquo;s, don\u0026rsquo;ts and don\u0026rsquo;t knows of direct observation of clinical skills in medical education. Perspect Med Educ [Internet]. 2017 Oct 27 [cited 2019 May 3];6(5):286\u0026ndash;305. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eLaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. Staging a performance: learners\u0026rsquo; perceptions about direct observation during residency. Med Educ [Internet]. 2017 May [cited 2019 May 3];51(5):498\u0026ndash;510. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003ehttp://doi.wiley.com/10.1111/medu.13232\u003c/div\u003e \u003cspan\u003eLaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. Staging a performance: learners\u0026rsquo; perceptions about direct observation during residency. Med Educ [Internet]. 2017 May 1 [cited 2019 May 3];51(5):498\u0026ndash;510. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cdiv class=\"BibBookDOI\"\u003ehttp://doi.wiley.com/10.1111/tct.12140\u003c/div\u003e \u003cspan\u003eSehgal R, Hardman J, Haney E. Observing trainee encounters using a one-way mirror. Clin Teach [Internet]. 2014 Jul [cited 2019 May 3];11(4):247\u0026ndash;50. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWatling C, Driessen E, van der Vleuten CPM, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements. Med Educ [Internet]. 2012 Feb [cited 2019 May 10];46(2):192\u0026ndash;200. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eWood TJ. Exploring the role of first impressions in rater-based assessments. Adv Heal Sci Educ. 2014;19(3):409\u0026ndash;27.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eNesbitt A, Pitcher A, James L, Sturrock A, Griffin A. Written feedback on supervised learning events. Clin Teach [Internet]. 2014 Jul [cited 2019 Mar 29];11(4):279\u0026ndash;83. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eHawkins SC, Osborne A, Schofield SJ, Pournaras DJ, Chester JF. Improving the accuracy of self-assessment of practical clinical skills using video feedback \u0026ndash; The importance of including benchmarks. Med Teach [Internet]. 2012 Apr 28 [cited 2019 May 24];34(4):279\u0026ndash;84. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eZick A, Granieri M, Makoul G. First-year medical students\u0026rsquo; assessment of their own communication skills: A video-based, open-ended approach. Patient Educ Couns [Internet]. 2007 Oct [cited 2019 May 24];68(2):161\u0026ndash;6. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eShelesky G, D\u0026rsquo;Amico F, Marfatia R, Munshi A, Wilson SA. Does weekly direct observation and formal feedback improve intern patient care skills development? A randomized controlled trial. Fam Med [Internet]. [cited 2019 May 24];44(7):486\u0026ndash;92. Available from: .\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eKogan JR, Holmboe ES, Hauer KE. Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees. JAMA [Internet]. 2009 Sep 23 [cited 2019 May 3];302(12):1316. Available from: .2009.1365.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003eDudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisors\u0026rsquo; completed clinical evaluation reports. Med Educ [Internet]. 2008 [cited 2020 Apr 2];42(8):816\u0026ndash;22. Available from: 9834-1fe3-418d-9223-58d8db53c677%40sdc-v-sessmgr02.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e \u003cspan\u003ePelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol RPTM, van der Vleuten CPM. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract [Internet]. 2011 Mar [cited 2019 May 24];16(1):131\u0026ndash;42. Available from: .\u003c/span\u003e \u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-21365/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-21365/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBackground Direct observation is necessary for specific and actionable feedback, however clinicians often struggle to integrate observation into their practice. Remotely audio-monitoring trainees for periods of time may improve the quality of written feedback given to them, and may be a minimally disruptive task for a consultant to perform in a busy clinic.\u003c/p\u003e\u003cp\u003eMethods Selected faculty used a wireless audio receiver during the second half of students' oncology rotations to listen to encounters during clinic in real time. They then gave written feedback as per usual practice, as did faculty who did not use the listening-in intervention. Feedback was de-identified and rated, using a rubric, as strong/medium/weak according to consensus of 2/3 rating investigators.\u003c/p\u003e\u003cp\u003eResults Monitoring faculty indicated that audio monitoring made the feedback process easier and increased confidence in 95% of encounters.\u0026nbsp;Most students (19/21 respondents) felt monitoring contributed positively to their learning, and included more useful comments. 101 written evaluations were completed by 7 monitoring and 19 non-monitoring faculty.\u0026nbsp;22/23 (96%) of feedback after monitoring was rated as high quality, compared to 16/37 (43%) (p\u0026lt;0.001) for monitoring faculty before using the equipment (and 20/78 (26%) without monitoring for all consultants (p\u0026lt;0.001)).\u003c/p\u003e\u003cp\u003eConclusions Using live audio monitoring improved the quality of written feedback given to trainees, as judged by the trainees themselves and also using an exploratory grading rubric.\u0026nbsp;The method was well received by both faculty and trainees.\u0026nbsp;Although there are limitations compared to in-the-room observation (body language), the benefits of easy integration into clinical practice and a more natural patient encounter without the observer physically present lead the authors to now use this method routinely while teaching oncology students.\u003c/p\u003e","manuscriptTitle":"Quality of Written Feedback given to Medical Students after Introduction of Real-Time Audio Monitoring of Clinical Encounters","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2020-04-09 22:03:26","doi":"10.21203/rs.3.rs-21365/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major revision","date":"2020-05-04T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-04-21T12:00:00+00:00","index":1,"fulltext":"Recommendation: Major revisions required\nForm responses:\n---\n\nComments to Author:\n---\nThank you for inviting me to review this manuscript. It speaks to an important issue, ensuring observation so that meaningful feedback can be provided to learners. The study while small adds to the growing knowledge about strategies that may be helpful to both faculty and learners. It is innovative and should be disseminated. It does require some work to bring it up to a standard for a peer review publication. Do accept that while the comments below may appear to be critical, they are designed to help the researchers get the manuscript (m/s) so that it conforms to the style and structure of articles in the medical education literature as well as those in BMC Medical Education. Do review a few more articles in BMC Medical Education to see the structure. Reorganization of content would be helpful.\nAbstract. The abstract is well aligned with the content of the m/s. Edits may be needed as edits are made.\nBackground. The content and references in the introduction identify the problem and provide a rationale for the study. The m/s is well supported by references. It is clear that the researchers are well versed in the literature. There is some editing that could be done to make the introduction more focused. The background should describe the literature and rationale supporting the study. It should not mention the specifics of your study until the last paragraph which should identify the actual research questions. To that end, it would be clearer if the authors did some editing of page 4, para 1 (beginning with Translating…). Focus on what the literature recommends without mentioning your study. Content such as 'Our study aims..' and 'The second guideline statement', would not be included. Edit the final paragraph so that it provides a succinct set of objectives (preferably research questions). There should be a direct link between your research questions, the data collected for each, how the data were analyzed, the results and the discussion. When I read the results section, it appears to me that you had three (not two) objectives pertaining to feasibility, student perception, and feedback quality. I would not reference your own environment in the research question section; that might be provided in the methods or discussion.\nMethods. Ethics approval is generally given at the end and consists of a statement related to institutional approval. Can I recommend a re-ordering of content with specific headers for each section. Suggestions for headings include: Design and Study Context, Intervention, Participants (or recruitment), Data Collection, Data Analysis, and Ethics. Be sure you mention consent in the data collection section. Be clear in the data analysis section how you ensured quality and rigor in your analysis of the qualitative data. If you did a thematic analysis, be sure to include references to support that approach (e.g., references by V Braun \u0026 V Clarke).\nResults. Depending on how you present the quantitative and qualitive aspects of data collection and analysis in the methods section, present the findings in the same way. Generally quantitative data is presented first followed by qualitative. Figure 1 should present actual numbers and be complete in itself. I realize they are in the material provided at the top of Page 15 but it was difficult to follow this. Using percentages instead of numbers (as one might in a table) also makes it more challenging to follow what this is trying to illustrate.\nDiscussion. The first paragraph should summarize the key findings for each of the objectives/research questions along with what was unique about this study. The subsequent paragraphs should focus on the objectives/research questions drawing on how your research confirms or is different from the broader literature. It should not introduce new findings (i.e., costs, patient perceptions). The limitations section should be succinct and not introduce new findings/literature. Generally it consists of one paragraph that tells readers that this was a one site study, it involved volunteers, and focused only on written feedback. I would add a paragraph on implications for both practice (i.e., adoption of this in future) and for research.\nConclusion. There should be a conclusion section (1 paragraph) that summarizes what you learned in the study.\nReferences. There is an inconsistency re the presentation of references. Do check the journal's expectations by reviewing other articles (or author guidelines).\nI hope this is helpful.\n* Are the methods appropriate and well described?: **No**\n* Does the work include the necessary controls?: **Yes**\n* Are the conclusions drawn adequately supported by the data shown?: **No**\n* Are you able to assess any statistics in the manuscript or would you recommend an additional statistical review?: **I am able to assess the statistics**\n* Quality of written English: **Acceptable**\n* Declaration of competing interests: **I declare I have no competing interests.**\n* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please do not publish my name with my report. (default)**\n"},{"type":"editorAssigned","content":"","date":"2020-04-16T12:00:00+00:00","index":"","fulltext":""},{"type":"reviewersInvited","content":"","date":"2020-04-16T12:00:00+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-04-16T12:00:00+00:00","index":1,"fulltext":""},{"type":"checksComplete","content":"","date":"2020-04-07T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-04-07T12:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"","date":"2020-04-03T12:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0011af6e-6ce6-485e-9b72-03c65b9e6b1f","owner":[],"postedDate":"April 9th, 2020","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":81630,"name":"Scientific Communication"},{"id":81631,"name":"Health Policy"}],"tags":[],"updatedAt":"2020-07-26T15:05:17+00:00","versionOfRecord":{"articleIdentity":"rs-21365","link":"https://doi.org/10.1186/s12909-020-02158-6","journal":{"identity":"bmc-medical-education","isVorOnly":false,"title":"BMC Medical Education"},"publishedOn":"2020-07-25 12:00:00","publishedOnDateReadable":"July 25th, 2020"},"versionCreatedAt":"2020-04-09 22:03:26","video":"","vorDoi":"10.1186/s12909-020-02158-6","vorDoiUrl":"https://doi.org/10.1186/s12909-020-02158-6","workflowStages":[]},"version":"v1","identity":"rs-21365","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-21365","identity":"rs-21365","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}