Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study

Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study Mike Breunig, Ryan Kingsley, Tiffany Galush, Taryn Ragaisis, Laura Boldenow, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7011397/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 10 You are reading this latest preprint version Abstract Background In the context of competency-based medical education, more research is needed to define point-of-care ultrasound (POCUS) competency and validate learner assessment. To address noted gaps in the currently validated assessment instruments, we developed the Observed Structured POCUS Readiness Examination (OSPRE). The purpose of this study is to demonstrate evidence of validity and reliability of the OSPRE in Physician Assistant (PA) students. Methods A prospective cohort study was designed to demonstrate evidence of validity and reliability for the OSPRE instrument at one PA program. Students completed a hands-on skills assessment utilizing a simulated patient scenario involving a traumatic injury. Student performance was assessed in real time by one reviewer and asynchronously by a second reviewer blinded to the scoring of the initial reviewer. Inter-rater reliability was calculated for overall score and for each criterion of the OSPRE. Results Intraclass correlation coefficient for the final percentage score between the independent reviewers utilizing the OSPRE instrument was noted to be 0.86 (95% CI 0.56–0.95, p < 0.001), demonstrating overall good reliability. Level of agreement for inter-rater reliability for the specific individual criterion ranged between fair and substantial. Conclusion The OSPRE instrument provides a valid and reliable assessment of POCUS skills in PA students. When compared to previously validated assessment instruments, scores on the OSPRE had a wider range, suggesting that the OSPRE might have a greater discriminatory value for POCUS skillset. Point-of-Care Ultrasound Bedside Ultrasound Medical education Evaluation competency-based medical education Figures Figure 1 Figure 2 Figure 3 Introduction Point-of-care ultrasound (POCUS) is a goal-directed ultrasound performed by a clinician to answer a specific diagnostic or therapeutic question[ 1 ]. POCUS use in medicine and medical education programs is increasingly common. Professional medical societies, such as the American College of Emergency Physicians (ACEP) and the Society of Hospital Medicine (SHM), provide some guidance for teaching and evaluating POCUS skills [ 1 , 2 ]. Both societies recommend a multi-modal assessment to ensure competency, however, definitions of competency are not provided. Currently, the Accreditation Council of Graduate Medical Education (ACGME) requires inclusion of POCUS training for emergency medicine and critical care fellowships; however, clear guidance on best practices for education and learner evaluation are not provided [ 3 , 4 ]. In PA education, longitudinal POCUS education has been successfully incorporated, but similarly, there is little published validity evidence of competency assessments [ 5 ]. A 2017 systematic review of ultrasound education in medical school identified that further definition of POCUS competency is required [ 6 ]. A recent systematic review of POCUS psychomotor learning curves corroborated this finding and identified significant heterogeneity in how training programs assess POCUS skill [ 7 ]. In the context of competency-based medical education, more research is needed to define competency and validate learner assessment. Validity of an assessment instrument is defined as “the extent to which we measure what we purport to measure” [ 8 ]. Validity exists on a continuum, and the greater the evidence of validity, the more likely the assessment truly assesses what it intends to. Furthermore, validity is context dependent. An instrument that has undergone validity testing is valid in the context under which it was assessed. Validity can further be classified as construct validity, content validity, and criterion validity [ 8 ]. A construct is an abstract concept that explains a given phenomenon, such as “leadership” or in this case, “POCUS competency.” Content validity is “the degree to which an instrument is representative of the process.” Content validity operationalizes the construct by defining in a measurable way through the development of specific criterion to assess the construct [ 8 ]. Criterion validity is established through the development of a scale with normative anchors and descriptions, to more accurately discriminate learner performance on a specific criterion [ 8 ]. Several POCUS assessment tools have been reported in the literature. ACEP recommends use of a 5-point Likert scale assessing image quality and an assessment of interpretation accuracy; however, this does not evaluate learner skill in the process of obtaining images [ 1 ]. The Assessment of Competency in Thoracic Sonography (ACTS) scale utilizes a similar structure to evaluate competency for thoracic ultrasound but does not assess competency outside of psychomotor skills and is specific to one POCUS application [ 9 ]. The Ultrasound Competency Assessment Tool (UCAT) was developed in 2020 by Bell et al. employing a modified Delphi method, building consensus of sixty members of the Canadian Association for Emergency Physicians [ 10 ]. The UCAT utilizes four items, “preparation,” “image acquisition,” “image optimization,” and “clinical integration;” assessed using a 3-point Likert scale. However, the UCAT also does not evaluate learner skill in the process of obtaining images, nor does it evaluate for other vital aspects of POCUS competency, such as indication for the examination or interpretation of the images. The most thoroughly validated tool in the literature is the Observed Structured Assessment of Ultrasound Skill (OSAUS) [ 11 ]. OSAUS was developed through a Delphi consensus method and assesses learner competency with seven criteria: 1. Indication for the examination, 2. Applied knowledge of ultrasound equipment, 3. Image optimization, 4. Systematic examination, 5. Interpretation of images, 6. Documentation of examination, and 7. Medical decision making [ 12 ]. However, some advocate for a more thorough assessment of learners’ determination of appropriate depth, tomographic axis, and gain as these skills develop at differing rates [ 2 , 13 ]. Additionally, OSAUS does not assess other components of POCUS competency, such as accurate identification of structures. As POCUS is increasingly integrated into medical curricula, further work is needed to develop validated assessment tools that incorporate all the key aspects defining competency. We developed a comprehensive instrument for assessing POCUS competency called the Observed Structured POCUS Readiness Examination (OSPRE) and sought to demonstrate evidence of its validity in a cohort of physician assistant (PA) students. Methods Instrument Development Specific measures were taken to ensure evidence of validity for the OSPRE instrument. For the OSPRE, evidence of construct and content validity were demonstrated by completing a thorough review of the literature regarding POCUS competency, previously validated instruments to assess POCUS skill level, and POCUS learning. To create and test the OSPRE instrument, a panel of seven individuals were recruited, including those with expertise in PA education, evaluation instrument development, and POCUS. The panel included both PAs and physicians. The previously proposed instruments for assessing POCUS skills, including various procedural checklists, ACTS, UCAT, and OSAUS were reviewed to clarify this list. Additionally, literature regarding the process of POCUS learning and psychomotor learning theories were reviewed [ 9 , 10 , 12 , 14 – 17 ]. Based on the literature review, several important criteria were identified as missing from previously developed POCUS assessment instruments. Specifically, Simpson’s taxonomy for psychomotor skill development outlines a progression of psychomotor skills development, which is underrepresented in previously validated instruments [ 18 ]. This led to the development of a criterion (“Purposeful Examination”) aiming to identify a learner’s progress in developing movements higher order learning. Simpson outlines various observable taxonomical stages, representing the stages of skill development, which were incorporated into the descriptors of performance. Additionally, recent work on POCUS learning curves for novice PA students demonstrated that individuals obtain the unique skillsets of determining the appropriate depth, axis, gain, and overall image quality at variable rates. The previously proposed instruments do not assess these different components of image acquisitions, despite their unique As such, individual criterion for depth, axis, gain, and overall image quality were added to the OSPRE based on this new understanding. This process led to the development of the OSPRE with the following criteria: Indication, Ultrasound Device Utilization, Purposeful Examination, Depth, Axis, Gain, Overall Image Quality, Structure Identification, Interpretation, and Application (Fig. 1 ). The criteria were further defined on a five-point Likert scale, consistent with previously published instruments and with ACEP guidelines [ 1 , 12 ]. Normative anchors (“very poor” through “exemplary”) were established, as well as descriptors of expected performance. The OSPRE instrument was piloted twice on PA students prior to initiation of the study, with iterative changes to descriptors of expected performance. Study Design Following instrument development, a prospective cohort study was designed to demonstrate evidence of validity and reliability for the OSPRE at one PA program in the Midwest region of the United States. The research proposal was deemed exempt by the Mayo Clinic institutional review board (24-004937). The PA Program involved in this research study has an integrated POCUS curriculum including a multimodal evaluation process, including multiple choice quizzes/tests, hands-on skills assessments, and a portfolio of images to assess student learning. Hand-on skills assessments, where students are required to demonstrate obtaining, interpreting, and integrating POCUS findings into clinical scenarios occur at three times during the curriculum. During these sessions, students are evaluated by faculty with expertise in POCUS. Twenty-four PA students were prospectively offered enrollment in the study. Students were invited to opt out of the exempt research project by emailing a study coordinator. Participation in the study was blinded to PA program faculty, and participation or lack thereof did not influence their standing in the program. Informed consent to participate was obtained from all of the participants. Piloting of the OSPRE instrument occurred at the first and second hands-on skills assessments, with iterative changes occurring thereafter. For the third hands-on skills assessment, a simulated patient scenario was developed involving a traumatic injury prompting use of the extended focused assessment with sonography in trauma (eFAST) protocol. The scenario described a hemodynamically unstable male patient brought to the emergency department after crashing a motorized scooter, with resultant traumatic intraabdominal bleeding. Students were given twenty minutes to evaluate the patient. They required to outline the POCUS examination they wished to pursue including the indication for this examination, obtain high-quality images on the standardized patient, identify anatomical structures for each organ system on still images, interpret recorded normal and pathological POCUS videos in the context of the scenario, and determine next steps in treatment of the standardized patient. All sessions were recorded for asynchronous review, and all images generated were stored in an online archive. Four independent reviewers were selected for the research project. All reviewers were clinically practicing PAs in the division of Hospital Internal Medicine at the affiliated academic medical center. All reviewers had received extensive POCUS training, completing the Society of Hospital Medicine’s POCUS Certificate of Completion program [ 19 ]. Each reviewer evaluated the performance of eight students in-person utilizing the OSPRE. The same three reviewers independently and asynchronously evaluated the recordings of an additional eight students utilizing the OSPRE, blinded to the original reviewers’ scoring. Asynchronous reviewers were able to review a recording of the standardized patient encounter, and the stored images obtained during the encounter. A fourth reviewer, blinded to the outcomes of the original reviewers, assessed student performance in the scenario utilizing the OSAUS tool. Evaluations on the OSPRE and OSAUS instruments were submitted via an online survey tool and exported for statistical analysis. Statistical Analysis Inter-rater reliability was assessed in several ways. The final scores on all assessments by the two independent reviewers were converted to a percentage. Intraclass correlation coefficient (ICC) was calculated for the students’ overall percentage score on the OSPRE for the simulated scenario. Inter-rater reliability for each of the ten criteria on the OSPRE was assessed through the calculation of weighted kappa values. As one reviewer evaluated student performance in-person, in real-time, and the other evaluated student performance based on a recording, asynchronously, a Bland-Altman Plot was calculated to assess for potential biases between the two methods. To assess for convergent validity, the scores from both blinded reviewers on the OSPRE were compared to a third blinded reviewer scoring performance on the OSAUS. Descriptive statistics for mean, medium, maximum scores, minimum scores, and range were calculated. Correlation between the average of the two OSPRE scores and the OSAUS scores was calculated using the Pearson’s correlation coefficient. Results All twenty-four PA students were enrolled in the study, resulting in forty-eight OSPRE assessments and twenty-four OSAUS assessments. ICC for the final percentage score between the two independent reviewers utilizing the OSPRE was noted to be 0.86 (95% CI 0.56–0.95, p < 0.001), demonstrating overall good reliability. The Bland-Altman plot demonstrated a mean difference of 2.03 (95% CI, -5.15–9.23), with higher scores noted by the in-person reviewer (Fig. 2 ). Notably, a higher degree of variance was noted for the students with the lowest scores on the OSPRE. Inter-rater reliability for each of the ten criteria on the OSPRE was assessed through the calculation of weighted kappa values (Table 1 ). Level of agreement for each criterion ranged between fair and substantial. Substantial agreement was noted between the “Ultrasound device utilization” (weighted kappa 0.63, 95% CI 0.36–0.89) and “Purposeful examination” (weighted kappa 0.64, 95% CI 0.43–0.84) criteria. Moderate agreement was noted for “Indication” (weighted kappa 0.53, 95% CI -0.05–1), “Axis” (weighted kappa 0.52, 95% CI 0.34–0.70), “Structure identification” (weighted kappa 0.48, 95% CI 0.29–0.66), “Interpretation” (weighted kappa 0.44, 95% CI 0.21–0.68), and “Application” (weighted kappa 0.46, 95% CI 0.14–0.73). Fair agreement was noted for “Depth” (weighted kappa 0.38, 95% CI 0.17–0.59), “Gain” (weighted kappa 0.22, 95% CI -0.57–0.54), and “Overall image quality” (weighted kappa 0.39, 95% CI 0.23–0.56). Table 1 Inter-rater reliability for Individual Criterion on the Observed Structured POCUS Reliability Exam (OSPRE) Instrument Criterion Weighted Kappa 95% CI Level of Agreement Indication 0.54 -0.05–1 Moderate Agreement Ultrasound Device Utilization 0.63 0.36–0.89 Substantial Agreement Purposeful Examination 0.64 0.43–0.84 Substantial Agreement Depth 0.38 0.17–0.59 Fair Agreement Axis 0.52 0.34–0.70 Moderate Agreement Gain 0.22 -0.57–0.54 Fair Agreement Overall Image Quality 0.39 0.23–0.56 Fair Agreement Structure Identification 0.48 0.29–0.66 Moderate Agreement Interpretation 0.44 0.21–0.68 Moderate Agreement Application 0.46 0.14–0.73 Moderate Agreement Pearson’s correlation coefficient between the average OSPRE score and OSAUS score was calculated to be 0.83 (p < 0.0001) (Fig. 3 ). When compared to OSAUS performance, the scores on OSPRE assessments had a similar mean score (93.7% for OSPRE vs 94% for OSAUS), lower minimum scores (81.5% for OSPRE vs 87% for OSAUS), and larger range of scores (17.1% for OSPRE vs 11.5% for OSAUS) (Table 2 ). Table 2 Descriptive statistics for scores on the Observed Structured POCUS Reliability Exam (OSPRE) instrument and The Observed Structured Assessment of Ultrasound Skills (OSAUS) instruments OSPRE Reviewer #1 OSPRE Reviewer #2 OSPRE Average OSAUS Mean 94.70 92.66 93.68 94 Median 95.85 93.41 94.63 94.5 Max 99.02 98.54 98.54 100 Min 84.39 81.46 87.07 85 Range 14.63 17.07 11.46 15 Discussion The OSPRE was developed through thorough literature review of previously validated instruments and literature on POCUS learning to provide a more accurate and comprehensive assessment of POCUS competency. Through the development process, evidence of construct, content, and criterion validity were established, indicating an overall valid instrument for learner assessment. Further evidence of construct validity was provided by assessment of convergent validity, comparing student performance on the OSPRE to student performance on the OSAUS for the same scenario. While overall correlation was strong, supporting construct validity, the OSPRE scores had a slightly lower mean, lower minimum score, and higher range. Scores on the OSAUS were clustered more at the high-end of grading, when compared to OSPRE. This suggests that the OSPRE might have a greater discriminatory value for POCUS competency. Overall inter-rater reliability was noted to be good. A slight bias towards higher scores were noted on the in-person assessment than the virtual, asynchronous assessment. Interestingly, this bias was exacerbated for students who struggled on the assessment, relative to their peers. While the cause of this is unknown, the authors hypothesize that evaluators might be biased towards being more lenient while in person. While remote clinical assessments have been found to be feasible with acceptable intra-rater reliability, some studies have demonstrated similar trends [ 20 , 21 ]. Further research is needed to identify the cause of these differences. Inter-rater reliability for the specific criteria on the OSPRE were noted to have variable levels of agreement, however, given the good overall inter-rater reliability, the effects on overall scoring appear to be low. Limited availability of proficient raters precluded random sampling, which is desirable for generalizability. Therefore, we used a two-way mixed-effects model with purposeful rater selection [ 22 ]. Despite this, further piloting and reconsideration of the provided descriptors should be considered for the criterion with the lowest levels of agreement, including depth, gain, and overall image quality. The generalizability of this data is limited given the small sample size, all subjects being from one cohort, use of a two-way mixed-effects model, and the specific learning population of PA students. Further studies are needed to determine if the OSPRE might be useful for other learner populations. Additionally, this validation and reliability study occurred while assessing students on a single clinical scenario, trauma requiring the use of the eFAST protocol. Further studies should look at assessing evidence of validity and reliability of the OSPRE for different POCUS applications. Conclusion The OSPRE provides a valid and reliable assessment of POCUS competency in PA students. Further study is needed to expand the evidence of validity and reliability to different learner populations and clinical scenarios. Abbreviations American College of Emergency Physicians; ACEP Assessment of Competency in Thoracic Sonography; ACTS Observed Structured Assessment of Ultrasound Skill; OSAUS Observed Structured POCUS Readiness Examination; OSPRE Physician assistant; PA Point-of-care ultrasound; POCUS Society of Hospital Medicine; SHM Ultrasound Competency Assessment Tool; UCAT Declarations Ethics approval and consent to participate: The research proposal was deemed exempt by the Mayo Clinic institutional review board (24-004937) in accordance with the Declaration of Helsinki. Informed consent to participate was obtained from all of the participants. Consent for publication: Not Applicable Availability of data and materials: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Competing interests: The authors report no financial and non-financial competing interests. Funding: Not Applicable. Authors' contributions: MB completed the literature review, developed the instrument, completed the statistical analysis, and wrote the manuscript. RK assisted with study design, instrument development, and reviewed the manuscript. LB, TG, TR assisted with learner assessment and reviewed the manuscript. WS assisted with instrument review and reviewed the manuscript. DK assisted with study design, statistical analysis approach, and reviewed the manuscript. Acknowledgements: The authors would like to acknowledge the Mayo Clinic Rochester Hospital Internal Medicine Research and Education Committees, as well as Dr. Sagar Dugani and Dr. Caroline Burton, for their support for this project. References Physicians ACE. In: Vivek S, Tayal VSCR, editors. Ultrasound Guidelines: Emergency, Point-of-care, and Clinical Ultrasound Guidelines in Medicine. C.C., editor.; 2023. p. 47. Soni NJ, Schnobrich D, Mathews BK, Tierney DM, Jensen TP, Dancel R, et al. Point-of-Care Ultrasound for Hospitalists: A Position Statement of the Society of Hospital Medicine. J Hosp Med. 2019;14:1553–5606. (Electronic)). Education ACfGM. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. 2023. Education ACfGM. ACGME Program Requirements for Graduate Medical Education in Critical Care Medicine. 2024. Breunig M, Huckabee M, Rieck K. An Integrated Point-of-Care Ultrasound Curriculum: An Evidence-Based Approach. JPAE. 2021. Dickson R, Duncanson K, Shepherd S. The path to ultrasound proficiency: A systematic review of ultrasound education and training programmes for junior medical practitioners. Australas J Ultrasound Med. 2017;20(1):5–17. Breunig M, Chelf C, Kashiwagi D. Point-of-Care Ultrasound Psychomotor Learning Curves: A Systematic Review of the Literature. J Ultrasound Med. 2024;Forthcoming. Colton D, Covert RW. Designing and constructing instruments for social research and evaluation. San Francisco, CA: Jossey-Bass; 2007. Millington SJ, Arntfield RT, Guo RJ, Koenig S, Kory P, Noble V et al. The Assessment of Competency in Thoracic Sonography (ACTS) scale: validation of a tool for point-of-care ultrasound. Crit Ultrasound J. 2017;9(1). Bell CA-O, Wagner N, Hall A, Newbigging J, Rang L, McKaigney C. The ultrasound competency assessment tool for four-view cardiac POCUS. Ultrasound J. 2021;13(1):42. Tolsgaard MG, Todsen T, Fau - Sorensen JL, Sorensen Jl Fau -, Ringsted C, Ringsted C, Fau - Lorentzen T, Lorentzen T, Fau - Ottesen B, Ottesen B, Fau - Tabor A et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. (1932–6203 (Electronic)). Tolsgaard MG, Todsen T, Fau - Sorensen JL, Sorensen Jl Fau -, Ringsted C, Ringsted C, Fau - Lorentzen T, Lorentzen T, Fau - Ottesen B, Ottesen B, Fau - Tabor A, et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. PLoS ONE. 2013;8(2):e57687. Breunig M, Hanson A, Huckabee M. Learning curves for point-of-care ultrasound image acquisition for novice learners in a longitudinal curriculum. Ultrasound J. 2023;15(1):31. Adamson R, Morris AE, Sun Woan J, Ma IWY, Schnobrich D, Soni NJ. Development of a Focused Cardiac Ultrasound Image Acquisition Assessment Tool. ATS Scholar. 2020;1(3):260–77. Ma IWY, Desy J, Woo MY, Kirkpatrick AW, Noble VE. Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills. J Graduate Med Educ. 2020;12(2):176–84. Millington SJ, Hewak M, Arntfield RT, Beaulieu Y, Hibbert B, Koenig S, et al. Outcomes from extensive training in critical care echocardiography: Identifying the optimal number of practice studies required to achieve competency. J Crit Care. 2017;40:99–102. Skaarup S, Laursen CB, Bjerrum AS, Hilberg O. Objective and Structured Assessment of Lung Ultrasound Competence. A Multispecialty Delphi Consensus and Construct Validity Study. Annals Am Thorac Soc. 2017;14(4):555–60. Simpson E. The classification of educational objectives, psychomotor domain.1966. Medicine SoH. POCUS Certificate of Completion [Available from: https://www.hospitalmedicine.org/clinical-topics/ultrasound/pocus-certificate-of-completion/ Kunutsor SK, Metcalf EP, Westacott R, Revell L, Blythe A. Are remote clinical assessments a feasible and acceptable method of assessment? A systematic review. Med Teach. 2022;44(3):300–8. Moroz S, Andrade R, Walsh L, Richard CL. Student Performance on an Objective Structured Clinical Exam Delivered Both Virtually and In-Person. Am J Pharm Educ. 2023;87(7):100088. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2017;16(4):155–63. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 14 Oct, 2025 Reviews received at journal 31 Jul, 2025 Reviews received at journal 29 Jul, 2025 Reviewers agreed at journal 25 Jul, 2025 Reviewers agreed at journal 25 Jul, 2025 Reviewers agreed at journal 18 Jul, 2025 Reviewers invited by journal 16 Jul, 2025 Editor assigned by journal 16 Jul, 2025 Submission checks completed at journal 14 Jul, 2025 First submitted to journal 14 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7011397","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":487617946,"identity":"6f7dcf72-5759-4283-8f79-dc0fdc28cc46","order_by":0,"name":"Mike Breunig","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/0lEQVRIiWNgGAWjYBACAzjJzMAGJC3koBLMRGuRMCZSCwSAtSQ2ENJiLn34mHRBQS2Dbjvzswc/KiTS17afMXzAUGEN04sBLPvS0qRnGBxnMDvMZm7Yc0Yid9uZHGMDhjPpOLUYnOExk+YxOAbUwsMmwdsG1HIgx0yCse0wHi383+BaJP/+k0g3O/8GqOUfPi08bEAtNWAt0rwNEglmN0C2NODWYtnDZmzNY3CAB+gXM2mZYxKG2248KzZIOJZujEuLOQ/zw9s8f+rkzM4ffib5psZG3ux88sYHH2qsZXFpgYLDPEgcDgOGBPzKQaAOmcP+gLCGUTAKRsEoGEkAANIhUMrl26bJAAAAAElFTkSuQmCC","orcid":"","institution":"Mayo Clinic","correspondingAuthor":true,"prefix":"","firstName":"Mike","middleName":"","lastName":"Breunig","suffix":""},{"id":487617947,"identity":"a3ce04e2-c0c4-499a-8881-b779ff83bedf","order_by":1,"name":"Ryan Kingsley","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Ryan","middleName":"","lastName":"Kingsley","suffix":""},{"id":487617948,"identity":"ae3ab1d4-e2c4-41d4-9d24-fe0c18c3bd4f","order_by":2,"name":"Tiffany Galush","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Tiffany","middleName":"","lastName":"Galush","suffix":""},{"id":487617951,"identity":"71888377-2299-4349-bbd9-52d6cfd93dc7","order_by":3,"name":"Taryn Ragaisis","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Taryn","middleName":"","lastName":"Ragaisis","suffix":""},{"id":487617952,"identity":"8aa071a7-7330-4f87-aca0-59d9fa246676","order_by":4,"name":"Laura Boldenow","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"","lastName":"Boldenow","suffix":""},{"id":487617954,"identity":"b5338cc3-2af7-42b8-94a4-470ea3848b18","order_by":5,"name":"Will M. Schouten","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Will","middleName":"M.","lastName":"Schouten","suffix":""},{"id":487617955,"identity":"69adce1c-83bc-47b0-9477-c7eaaa2872ca","order_by":6,"name":"Deanne Kashiwagi","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Deanne","middleName":"","lastName":"Kashiwagi","suffix":""}],"badges":[],"createdAt":"2025-06-30 14:08:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7011397/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7011397/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":87225795,"identity":"09285ccf-c78b-4bc2-9285-8460066364b8","added_by":"auto","created_at":"2025-07-21 17:18:42","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":230995,"visible":true,"origin":"","legend":"\u003cp\u003eThe\u003cstrong\u003e \u003c/strong\u003eObserved Structured POCUS Readiness Examination (OSPRE) Instrument\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7011397/v1/4db77924694ccb4f6e0e2976.png"},{"id":87225497,"identity":"7484cc6a-c753-4f8f-b7dd-341eda1b7d29","added_by":"auto","created_at":"2025-07-21 17:10:42","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":29724,"visible":true,"origin":"","legend":"\u003cp\u003eThe Bland-Altman Plot to Assess Inter-Rater Reliability\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7011397/v1/a30d7f5a02f46ac07a95192f.jpg"},{"id":87225495,"identity":"605757eb-27a4-4e71-886a-e7966d8418a7","added_by":"auto","created_at":"2025-07-21 17:10:42","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":25772,"visible":true,"origin":"","legend":"\u003cp\u003eCorrelation between student scores on the Observed Structured POCUS Readiness Examination (OSPRE) instrument and The Observed Structured Assessment of Ultrasound Skills (OSAUS) instrument\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-7011397/v1/50471aedc7f54ac78ce99d42.jpg"},{"id":87226843,"identity":"5b7dc05b-87d6-45b5-8024-d6cf0257549c","added_by":"auto","created_at":"2025-07-21 17:34:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":677581,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7011397/v1/c27676a7-1c98-4247-a3a5-d5ab8105052b.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study","fulltext":[{"header":"Introduction","content":"\u003cp\u003ePoint-of-care ultrasound (POCUS) is a goal-directed ultrasound performed by a clinician to answer a specific diagnostic or therapeutic question[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. POCUS use in medicine and medical education programs is increasingly common. Professional medical societies, such as the American College of Emergency Physicians (ACEP) and the Society of Hospital Medicine (SHM), provide some guidance for teaching and evaluating POCUS skills [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Both societies recommend a multi-modal assessment to ensure competency, however, definitions of competency are not provided. Currently, the Accreditation Council of Graduate Medical Education (ACGME) requires inclusion of POCUS training for emergency medicine and critical care fellowships; however, clear guidance on best practices for education and learner evaluation are not provided [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. In PA education, longitudinal POCUS education has been successfully incorporated, but similarly, there is little published validity evidence of competency assessments [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eA 2017 systematic review of ultrasound education in medical school identified that further definition of POCUS competency is required [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. A recent systematic review of POCUS psychomotor learning curves corroborated this finding and identified significant heterogeneity in how training programs assess POCUS skill [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. In the context of competency-based medical education, more research is needed to define competency and validate learner assessment.\u003c/p\u003e\u003cp\u003eValidity of an assessment instrument is defined as \u0026ldquo;the extent to which we measure what we purport to measure\u0026rdquo; [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Validity exists on a continuum, and the greater the evidence of validity, the more likely the assessment truly assesses what it intends to. Furthermore, validity is context dependent. An instrument that has undergone validity testing is valid in the context under which it was assessed. Validity can further be classified as construct validity, content validity, and criterion validity [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. A construct is an abstract concept that explains a given phenomenon, such as \u0026ldquo;leadership\u0026rdquo; or in this case, \u0026ldquo;POCUS competency.\u0026rdquo; Content validity is \u0026ldquo;the degree to which an instrument is representative of the process.\u0026rdquo; Content validity operationalizes the construct by defining in a measurable way through the development of specific criterion to assess the construct [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Criterion validity is established through the development of a scale with normative anchors and descriptions, to more accurately discriminate learner performance on a specific criterion [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eSeveral POCUS assessment tools have been reported in the literature. ACEP recommends use of a 5-point Likert scale assessing image quality and an assessment of interpretation accuracy; however, this does not evaluate learner skill in the process of obtaining images [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. The Assessment of Competency in Thoracic Sonography (ACTS) scale utilizes a similar structure to evaluate competency for thoracic ultrasound but does not assess competency outside of psychomotor skills and is specific to one POCUS application [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The Ultrasound Competency Assessment Tool (UCAT) was developed in 2020 by Bell et al. employing a modified Delphi method, building consensus of sixty members of the Canadian Association for Emergency Physicians [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. The UCAT utilizes four items, \u0026ldquo;preparation,\u0026rdquo; \u0026ldquo;image acquisition,\u0026rdquo; \u0026ldquo;image optimization,\u0026rdquo; and \u0026ldquo;clinical integration;\u0026rdquo; assessed using a 3-point Likert scale. However, the UCAT also does not evaluate learner skill in the process of obtaining images, nor does it evaluate for other vital aspects of POCUS competency, such as indication for the examination or interpretation of the images. The most thoroughly validated tool in the literature is the Observed Structured Assessment of Ultrasound Skill (OSAUS) [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. OSAUS was developed through a Delphi consensus method and assesses learner competency with seven criteria: 1. Indication for the examination, 2. Applied knowledge of ultrasound equipment, 3. Image optimization, 4. Systematic examination, 5. Interpretation of images, 6. Documentation of examination, and 7. Medical decision making [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. However, some advocate for a more thorough assessment of learners\u0026rsquo; determination of appropriate depth, tomographic axis, and gain as these skills develop at differing rates [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Additionally, OSAUS does not assess other components of POCUS competency, such as accurate identification of structures.\u003c/p\u003e\u003cp\u003eAs POCUS is increasingly integrated into medical curricula, further work is needed to develop validated assessment tools that incorporate all the key aspects defining competency. We developed a comprehensive instrument for assessing POCUS competency called the Observed Structured POCUS Readiness Examination (OSPRE) and sought to demonstrate evidence of its validity in a cohort of physician assistant (PA) students.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cb\u003eInstrument Development\u003c/b\u003e\u003c/p\u003e\u003cp\u003eSpecific measures were taken to ensure evidence of validity for the OSPRE instrument. For the OSPRE, evidence of construct and content validity were demonstrated by completing a thorough review of the literature regarding POCUS competency, previously validated instruments to assess POCUS skill level, and POCUS learning. To create and test the OSPRE instrument, a panel of seven individuals were recruited, including those with expertise in PA education, evaluation instrument development, and POCUS. The panel included both PAs and physicians. The previously proposed instruments for assessing POCUS skills, including various procedural checklists, ACTS, UCAT, and OSAUS were reviewed to clarify this list. Additionally, literature regarding the process of POCUS learning and psychomotor learning theories were reviewed [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan additionalcitationids=\"CR15 CR16\" citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eBased on the literature review, several important criteria were identified as missing from previously developed POCUS assessment instruments. Specifically, Simpson\u0026rsquo;s taxonomy for psychomotor skill development outlines a progression of psychomotor skills development, which is underrepresented in previously validated instruments [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. This led to the development of a criterion (\u0026ldquo;Purposeful Examination\u0026rdquo;) aiming to identify a learner\u0026rsquo;s progress in developing movements higher order learning. Simpson outlines various observable taxonomical stages, representing the stages of skill development, which were incorporated into the descriptors of performance. Additionally, recent work on POCUS learning curves for novice PA students demonstrated that individuals obtain the unique skillsets of determining the appropriate depth, axis, gain, and overall image quality at variable rates. The previously proposed instruments do not assess these different components of image acquisitions, despite their unique As such, individual criterion for depth, axis, gain, and overall image quality were added to the OSPRE based on this new understanding.\u003c/p\u003e\u003cp\u003eThis process led to the development of the OSPRE with the following criteria: Indication, Ultrasound Device Utilization, Purposeful Examination, Depth, Axis, Gain, Overall Image Quality, Structure Identification, Interpretation, and Application (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The criteria were further defined on a five-point Likert scale, consistent with previously published instruments and with ACEP guidelines [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Normative anchors (\u0026ldquo;very poor\u0026rdquo; through \u0026ldquo;exemplary\u0026rdquo;) were established, as well as descriptors of expected performance. The OSPRE instrument was piloted twice on PA students prior to initiation of the study, with iterative changes to descriptors of expected performance.\u003c/p\u003e\u003cp\u003e\u003cb\u003eStudy Design\u003c/b\u003e\u003c/p\u003e\u003cp\u003eFollowing instrument development, a prospective cohort study was designed to demonstrate evidence of validity and reliability for the OSPRE at one PA program in the Midwest region of the United States. The research proposal was deemed exempt by the Mayo Clinic institutional review board (24-004937). The PA Program involved in this research study has an integrated POCUS curriculum including a multimodal evaluation process, including multiple choice quizzes/tests, hands-on skills assessments, and a portfolio of images to assess student learning. Hand-on skills assessments, where students are required to demonstrate obtaining, interpreting, and integrating POCUS findings into clinical scenarios occur at three times during the curriculum. During these sessions, students are evaluated by faculty with expertise in POCUS.\u003c/p\u003e\u003cp\u003eTwenty-four PA students were prospectively offered enrollment in the study. Students were invited to opt out of the exempt research project by emailing a study coordinator. Participation in the study was blinded to PA program faculty, and participation or lack thereof did not influence their standing in the program. Informed consent to participate was obtained from all of the participants. Piloting of the OSPRE instrument occurred at the first and second hands-on skills assessments, with iterative changes occurring thereafter. For the third hands-on skills assessment, a simulated patient scenario was developed involving a traumatic injury prompting use of the extended focused assessment with sonography in trauma (eFAST) protocol. The scenario described a hemodynamically unstable male patient brought to the emergency department after crashing a motorized scooter, with resultant traumatic intraabdominal bleeding. Students were given twenty minutes to evaluate the patient. They required to outline the POCUS examination they wished to pursue including the indication for this examination, obtain high-quality images on the standardized patient, identify anatomical structures for each organ system on still images, interpret recorded normal and pathological POCUS videos in the context of the scenario, and determine next steps in treatment of the standardized patient. All sessions were recorded for asynchronous review, and all images generated were stored in an online archive.\u003c/p\u003e\u003cp\u003eFour independent reviewers were selected for the research project. All reviewers were clinically practicing PAs in the division of Hospital Internal Medicine at the affiliated academic medical center. All reviewers had received extensive POCUS training, completing the Society of Hospital Medicine\u0026rsquo;s POCUS Certificate of Completion program [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Each reviewer evaluated the performance of eight students in-person utilizing the OSPRE. The same three reviewers independently and asynchronously evaluated the recordings of an additional eight students utilizing the OSPRE, blinded to the original reviewers\u0026rsquo; scoring. Asynchronous reviewers were able to review a recording of the standardized patient encounter, and the stored images obtained during the encounter. A fourth reviewer, blinded to the outcomes of the original reviewers, assessed student performance in the scenario utilizing the OSAUS tool. Evaluations on the OSPRE and OSAUS instruments were submitted via an online survey tool and exported for statistical analysis.\u003c/p\u003e\u003cdiv id=\"Sec2\" class=\"Section2\"\u003e\u003ch2\u003eStatistical Analysis\u003c/h2\u003e\u003cp\u003eInter-rater reliability was assessed in several ways. The final scores on all assessments by the two independent reviewers were converted to a percentage. Intraclass correlation coefficient (ICC) was calculated for the students\u0026rsquo; overall percentage score on the OSPRE for the simulated scenario. Inter-rater reliability for each of the ten criteria on the OSPRE was assessed through the calculation of weighted kappa values. As one reviewer evaluated student performance in-person, in real-time, and the other evaluated student performance based on a recording, asynchronously, a Bland-Altman Plot was calculated to assess for potential biases between the two methods.\u003c/p\u003e\u003cp\u003eTo assess for convergent validity, the scores from both blinded reviewers on the OSPRE were compared to a third blinded reviewer scoring performance on the OSAUS. Descriptive statistics for mean, medium, maximum scores, minimum scores, and range were calculated. Correlation between the average of the two OSPRE scores and the OSAUS scores was calculated using the Pearson\u0026rsquo;s correlation coefficient.\u003c/p\u003e\u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eAll twenty-four PA students were enrolled in the study, resulting in forty-eight OSPRE assessments and twenty-four OSAUS assessments. ICC for the final percentage score between the two independent reviewers utilizing the OSPRE was noted to be 0.86 (95% CI 0.56\u0026ndash;0.95, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), demonstrating overall good reliability. The Bland-Altman plot demonstrated a mean difference of 2.03 (95% CI, -5.15\u0026ndash;9.23), with higher scores noted by the in-person reviewer (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). Notably, a higher degree of variance was noted for the students with the lowest scores on the OSPRE.\u003c/p\u003e\u003cp\u003eInter-rater reliability for each of the ten criteria on the OSPRE was assessed through the calculation of weighted kappa values (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Level of agreement for each criterion ranged between fair and substantial. Substantial agreement was noted between the \u0026ldquo;Ultrasound device utilization\u0026rdquo; (weighted kappa 0.63, 95% CI 0.36\u0026ndash;0.89) and \u0026ldquo;Purposeful examination\u0026rdquo; (weighted kappa 0.64, 95% CI 0.43\u0026ndash;0.84) criteria. Moderate agreement was noted for \u0026ldquo;Indication\u0026rdquo; (weighted kappa 0.53, 95% CI -0.05\u0026ndash;1), \u0026ldquo;Axis\u0026rdquo; (weighted kappa 0.52, 95% CI 0.34\u0026ndash;0.70), \u0026ldquo;Structure identification\u0026rdquo; (weighted kappa 0.48, 95% CI 0.29\u0026ndash;0.66), \u0026ldquo;Interpretation\u0026rdquo; (weighted kappa 0.44, 95% CI 0.21\u0026ndash;0.68), and \u0026ldquo;Application\u0026rdquo; (weighted kappa 0.46, 95% CI 0.14\u0026ndash;0.73). Fair agreement was noted for \u0026ldquo;Depth\u0026rdquo; (weighted kappa 0.38, 95% CI 0.17\u0026ndash;0.59), \u0026ldquo;Gain\u0026rdquo; (weighted kappa 0.22, 95% CI -0.57\u0026ndash;0.54), and \u0026ldquo;Overall image quality\u0026rdquo; (weighted kappa 0.39, 95% CI 0.23\u0026ndash;0.56).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eInter-rater reliability for Individual Criterion on the Observed Structured POCUS Reliability Exam (OSPRE) Instrument\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"4\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eCriterion\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eWeighted Kappa\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003e95% CI\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eLevel of Agreement\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eIndication\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.54\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e-0.05\u0026ndash;1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eModerate Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUltrasound Device Utilization\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.63\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.36\u0026ndash;0.89\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eSubstantial Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003ePurposeful Examination\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.64\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.43\u0026ndash;0.84\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eSubstantial Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eDepth\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.38\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.17\u0026ndash;0.59\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eFair Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAxis\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.52\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.34\u0026ndash;0.70\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eModerate Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGain\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.22\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e-0.57\u0026ndash;0.54\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eFair Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eOverall Image Quality\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.39\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.23\u0026ndash;0.56\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eFair Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eStructure Identification\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.48\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.29\u0026ndash;0.66\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eModerate Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eInterpretation\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.44\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.21\u0026ndash;0.68\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eModerate Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eApplication\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e0.46\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e0.14\u0026ndash;0.73\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eModerate Agreement\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003ePearson\u0026rsquo;s correlation coefficient between the average OSPRE score and OSAUS score was calculated to be 0.83 (p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). When compared to OSAUS performance, the scores on OSPRE assessments had a similar mean score (93.7% for OSPRE vs 94% for OSAUS), lower minimum scores (81.5% for OSPRE vs 87% for OSAUS), and larger range of scores (17.1% for OSPRE vs 11.5% for OSAUS) (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eDescriptive statistics for scores on the Observed Structured POCUS Reliability Exam (OSPRE) instrument and The Observed Structured Assessment of Ultrasound Skills (OSAUS) instruments\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eOSPRE Reviewer #1\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eOSPRE Reviewer #2\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eOSPRE Average\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eOSAUS\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMean\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e94.70\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e92.66\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e93.68\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e94\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMedian\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e95.85\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e93.41\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e94.63\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e94.5\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMax\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e99.02\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e98.54\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e98.54\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e100\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMin\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e84.39\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e81.46\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e87.07\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e85\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eRange\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e\u003cp\u003e14.63\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e\u003cp\u003e17.07\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e\u003cp\u003e11.46\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e15\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe OSPRE was developed through thorough literature review of previously validated instruments and literature on POCUS learning to provide a more accurate and comprehensive assessment of POCUS competency. Through the development process, evidence of construct, content, and criterion validity were established, indicating an overall valid instrument for learner assessment. Further evidence of construct validity was provided by assessment of convergent validity, comparing student performance on the OSPRE to student performance on the OSAUS for the same scenario. While overall correlation was strong, supporting construct validity, the OSPRE scores had a slightly lower mean, lower minimum score, and higher range. Scores on the OSAUS were clustered more at the high-end of grading, when compared to OSPRE. This suggests that the OSPRE might have a greater discriminatory value for POCUS competency.\u003c/p\u003e\u003cp\u003eOverall inter-rater reliability was noted to be good. A slight bias towards higher scores were noted on the in-person assessment than the virtual, asynchronous assessment. Interestingly, this bias was exacerbated for students who struggled on the assessment, relative to their peers. While the cause of this is unknown, the authors hypothesize that evaluators might be biased towards being more lenient while in person. While remote clinical assessments have been found to be feasible with acceptable intra-rater reliability, some studies have demonstrated similar trends [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Further research is needed to identify the cause of these differences.\u003c/p\u003e\u003cp\u003eInter-rater reliability for the specific criteria on the OSPRE were noted to have variable levels of agreement, however, given the good overall inter-rater reliability, the effects on overall scoring appear to be low. Limited availability of proficient raters precluded random sampling, which is desirable for generalizability. Therefore, we used a two-way mixed-effects model with purposeful rater selection [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Despite this, further piloting and reconsideration of the provided descriptors should be considered for the criterion with the lowest levels of agreement, including depth, gain, and overall image quality.\u003c/p\u003e\u003cp\u003eThe generalizability of this data is limited given the small sample size, all subjects being from one cohort, use of a two-way mixed-effects model, and the specific learning population of PA students. Further studies are needed to determine if the OSPRE might be useful for other learner populations. Additionally, this validation and reliability study occurred while assessing students on a single clinical scenario, trauma requiring the use of the eFAST protocol. Further studies should look at assessing evidence of validity and reliability of the OSPRE for different POCUS applications.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe OSPRE provides a valid and reliable assessment of POCUS competency in PA students. Further study is needed to expand the evidence of validity and reliability to different learner populations and clinical scenarios.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eAmerican College of Emergency Physicians; ACEP\u003c/p\u003e\n\u003cp\u003eAssessment of Competency in Thoracic Sonography; ACTS\u003c/p\u003e\n\u003cp\u003eObserved Structured Assessment of Ultrasound Skill; OSAUS\u003c/p\u003e\n\u003cp\u003eObserved Structured POCUS Readiness Examination; OSPRE\u003c/p\u003e\n\u003cp\u003ePhysician assistant; PA\u003c/p\u003e\n\u003cp\u003ePoint-of-care ultrasound; POCUS\u003c/p\u003e\n\u003cp\u003eSociety of Hospital Medicine; SHM\u003c/p\u003e\n\u003cp\u003eUltrasound Competency Assessment Tool; UCAT\u0026nbsp;\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics approval and consent to participate: The research proposal was deemed exempt by the Mayo Clinic institutional review board (24-004937)\u0026nbsp;in accordance with the Declaration of Helsinki.\u0026nbsp; Informed consent to participate was obtained from all of the participants.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent for publication: Not Applicable\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003eCompeting interests: The authors report no financial and non-financial competing interests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFunding: Not Applicable.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAuthors' contributions: MB completed the literature review, developed the instrument, completed the statistical analysis, and wrote the manuscript. RK assisted with study design, instrument development, and reviewed the manuscript. LB, TG, TR assisted with learner assessment and reviewed the manuscript. WS assisted with instrument review and reviewed the manuscript. \u0026nbsp; DK assisted with study design, statistical analysis approach, and reviewed the manuscript.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAcknowledgements: The authors would like to acknowledge the Mayo Clinic Rochester Hospital Internal Medicine Research and Education Committees, as well as Dr. Sagar Dugani and Dr. Caroline Burton, for their support for this project.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003ePhysicians ACE. In: Vivek S, Tayal VSCR, editors. Ultrasound Guidelines: Emergency, Point-of-care, and Clinical Ultrasound Guidelines in Medicine. C.C., editor.; 2023. p. 47.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSoni NJ, Schnobrich D, Mathews BK, Tierney DM, Jensen TP, Dancel R, et al. Point-of-Care Ultrasound for Hospitalists: A Position Statement of the Society of Hospital Medicine. J Hosp Med. 2019;14:1553\u0026ndash;5606. (Electronic)).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eEducation ACfGM. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. 2023.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eEducation ACfGM. ACGME Program Requirements for Graduate Medical Education in Critical Care Medicine. 2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBreunig M, Huckabee M, Rieck K. An Integrated Point-of-Care Ultrasound Curriculum: An Evidence-Based Approach. JPAE. 2021.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eDickson R, Duncanson K, Shepherd S. The path to ultrasound proficiency: A systematic review of ultrasound education and training programmes for junior medical practitioners. Australas J Ultrasound Med. 2017;20(1):5\u0026ndash;17.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBreunig M, Chelf C, Kashiwagi D. Point-of-Care Ultrasound Psychomotor Learning Curves: A Systematic Review of the Literature. J Ultrasound Med. 2024;Forthcoming.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eColton D, Covert RW. Designing and constructing instruments for social research and evaluation. San Francisco, CA: Jossey-Bass; 2007.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMillington SJ, Arntfield RT, Guo RJ, Koenig S, Kory P, Noble V et al. The Assessment of Competency in Thoracic Sonography (ACTS) scale: validation of a tool for point-of-care ultrasound. Crit Ultrasound J. 2017;9(1).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBell CA-O, Wagner N, Hall A, Newbigging J, Rang L, McKaigney C. The ultrasound competency assessment tool for four-view cardiac POCUS. Ultrasound J. 2021;13(1):42.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTolsgaard MG, Todsen T, Fau - Sorensen JL, Sorensen Jl Fau -, Ringsted C, Ringsted C, Fau - Lorentzen T, Lorentzen T, Fau - Ottesen B, Ottesen B, Fau - Tabor A et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. (1932\u0026ndash;6203 (Electronic)).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eTolsgaard MG, Todsen T, Fau - Sorensen JL, Sorensen Jl Fau -, Ringsted C, Ringsted C, Fau - Lorentzen T, Lorentzen T, Fau - Ottesen B, Ottesen B, Fau - Tabor A, et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. PLoS ONE. 2013;8(2):e57687.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eBreunig M, Hanson A, Huckabee M. Learning curves for point-of-care ultrasound image acquisition for novice learners in a longitudinal curriculum. Ultrasound J. 2023;15(1):31.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eAdamson R, Morris AE, Sun Woan J, Ma IWY, Schnobrich D, Soni NJ. Development of a Focused Cardiac Ultrasound Image Acquisition Assessment Tool. ATS Scholar. 2020;1(3):260\u0026ndash;77.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMa IWY, Desy J, Woo MY, Kirkpatrick AW, Noble VE. Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills. J Graduate Med Educ. 2020;12(2):176\u0026ndash;84.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMillington SJ, Hewak M, Arntfield RT, Beaulieu Y, Hibbert B, Koenig S, et al. Outcomes from extensive training in critical care echocardiography: Identifying the optimal number of practice studies required to achieve competency. J Crit Care. 2017;40:99\u0026ndash;102.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSkaarup S, Laursen CB, Bjerrum AS, Hilberg O. Objective and Structured Assessment of Lung Ultrasound Competence. A Multispecialty Delphi Consensus and Construct Validity Study. Annals Am Thorac Soc. 2017;14(4):555\u0026ndash;60.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSimpson E. The classification of educational objectives, psychomotor domain.1966.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMedicine SoH. POCUS Certificate of Completion [Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.hospitalmedicine.org/clinical-topics/ultrasound/pocus-certificate-of-completion/\u003c/span\u003e\u003cspan address=\"https://www.hospitalmedicine.org/clinical-topics/ultrasound/pocus-certificate-of-completion/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKunutsor SK, Metcalf EP, Westacott R, Revell L, Blythe A. Are remote clinical assessments a feasible and acceptable method of assessment? A systematic review. Med Teach. 2022;44(3):300\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMoroz S, Andrade R, Walsh L, Richard CL. Student Performance on an Objective Structured Clinical Exam Delivered Both Virtually and In-Person. Am J Pharm Educ. 2023;87(7):100088.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eKoo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2017;16(4):155\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Point-of-Care Ultrasound, Bedside Ultrasound, Medical education, Evaluation, competency-based medical education","lastPublishedDoi":"10.21203/rs.3.rs-7011397/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7011397/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e\u003cp\u003eIn the context of competency-based medical education, more research is needed to define point-of-care ultrasound (POCUS) competency and validate learner assessment. To address noted gaps in the currently validated assessment instruments, we developed the Observed Structured POCUS Readiness Examination (OSPRE). The purpose of this study is to demonstrate evidence of validity and reliability of the OSPRE in Physician Assistant (PA) students.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eA prospective cohort study was designed to demonstrate evidence of validity and reliability for the OSPRE instrument at one PA program. Students completed a hands-on skills assessment utilizing a simulated patient scenario involving a traumatic injury. Student performance was assessed in real time by one reviewer and asynchronously by a second reviewer blinded to the scoring of the initial reviewer. Inter-rater reliability was calculated for overall score and for each criterion of the OSPRE.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eIntraclass correlation coefficient for the final percentage score between the independent reviewers utilizing the OSPRE instrument was noted to be 0.86 (95% CI 0.56\u0026ndash;0.95, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001), demonstrating overall good reliability. Level of agreement for inter-rater reliability for the specific individual criterion ranged between fair and substantial.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eThe OSPRE instrument provides a valid and reliable assessment of POCUS skills in PA students. When compared to previously validated assessment instruments, scores on the OSPRE had a wider range, suggesting that the OSPRE might have a greater discriminatory value for POCUS skillset.\u003c/p\u003e","manuscriptTitle":"Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-21 17:10:37","doi":"10.21203/rs.3.rs-7011397/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-14T12:30:12+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-31T14:42:56+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-29T15:37:05+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"120420531145099340839102564351786059559","date":"2025-07-25T20:17:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"242254911094894176921163591678710455939","date":"2025-07-25T14:44:15+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"318647079331647071025263238486966683395","date":"2025-07-18T19:56:21+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-16T19:48:26+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-16T16:38:56+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-14T13:05:50+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Education","date":"2025-07-14T13:03:11+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"meed","sideBox":"Learn more about [BMC Medical Education](http://bmcmededuc.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/meed/default.aspx","title":"BMC Medical Education","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"ae845d6e-6c64-4d8a-b894-8289647a9a70","owner":[],"postedDate":"July 21st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2025-11-19T07:55:02+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-21 17:10:37","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7011397","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7011397","identity":"rs-7011397","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}