Defining Diagnostic Error: A Scoping Review to Assess the Impact of the National Academies' Report Improving Diagnosis in Health Care

We used Arksey and O’Malley’s framework 15 for scoping reviews to explore how authors have operationalized the NASEM definition. 16 – 19 The framework recommends a comprehensive search strategy across several literature sources, including electronic databases, reference lists of relevant literature, hand-searching key journals, networks, and relevant organizations. 15 – 17 For the purpose of this study, we focused on peer-reviewed literature and consultation with subject matter experts (SMEs) in the field of diagnostic safety. The study received an exempt determination by the Single IRB for the project (MHRI: 00001338). Research Question : “How are authors operationalizing the NASEM committee’s definition of diagnostic error in published peer-reviewed diagnostic error research?” A clinical library scientist codesigned the search strategy and supported electronic database searches. We performed the final search using Medline and Google Scholar around a broad list of terms for defining diagnostic errors (Appendix A, http://links.lww.com/JPS/A472 ). The strategy was tested against core readings in the field (Appendix B, http://links.lww.com/JPS/A472 ). Medical subject heading terms from these articles were indexed and informed the final search strategy. The medical subject heading terms along with text word terms and phrases were then mapped to Google Scholar. The nonindexed portion of Medline was searched separately with a text-only variant of the search. We modified the initial search strategy in Medline to eliminate literature on medical error, preventable harms, or patient safety issues unrelated to diagnostic error, and to remove subheadings under the Diagnostic Error subject heading that related to errors in laboratory results, such as false-positive and false-negative results (Appendix C, http://links.lww.com/JPS/A472 ). Database searches were limited to English and non-English language articles with abstracts in English, published between October 1, 2015, and November 1, 2020. This date range was informed by the NASEM report publication date. 1 An additional informal PubMed and Google Scholar (key terms: diagnostic error, diagnostic safety) search was conducted to capture any new published literature between November 2020 and February 2021. Finally, the library scientist conducted an initial deduplication process. This list of candidate articles was screened for inclusion. Our initial search yielded a large number of abstracts for screening (see PRISMA diagram, Fig. 1 ). Articles were screened for inclusion using an initial set of criteria tested by 3 investigators and refined to yield a final set of criteria outlined in Table 1 . A team of abstractors experienced in scoping review methodology then applied the screening criteria to identify articles to undergo full-text screening and subsequent review. PRISMA flow diagram. Systematic Review Inclusion and Exclusion Criteria We designed a broad abstraction tool to collect descriptive characteristics such as general citation information (e.g., title, abstract, universal record locator, unique record identifier, resource type) to support the initial screen. Secondary screening required review of the full-text articles. Our team made an a priori decision that operationalization of the definition would require 2 things: (1) referencing of the NASEM report and (2) reference to the NASEM committee’s definition of diagnostic error. Articles that did not meet these criteria were excluded. The secondary and tertiary screening abstraction forms were designed to elicit information on if and where the NASEM definition was reported. Tertiary abstraction forms were designed to summarize if and how the definitions were operationalized for use and to assess article quality for literature undergoing full-text review. Consensus for consideration of each article was completed during a final team meeting. The first author (T.D.G.), a qualitative methodologist, conducted semistructured interviews with 9 SMEs selected based on scholarship (e.g., number of publications, national/international reputation) or diversity in research topic (e.g., clinical education, reporting, patient perspective, cognitive psychology). Subject matter experts were asked to discuss (1) perception of the definition, (2) experience operationalizing the definition, and (3) impact of the NASEM definition on research. Interviews lasted between 30 and 60 minutes, and SMEs were provided a $150 stipend for their time. Interviews were audio recorded and transcribed verbatim and coded using an inductive/deductive content analysis. The first author (T.D.G.) became familiar with all transcripts and created an initial codebook. Initial codes were created deductively based on the questions asked (e.g., opinion, impact on field, operationalizing). Inductive codes were open coded and added to the codebook as they emerged. Coding was discussed with the team, and any discrepancies were resolved through consensus.

Initially, 1077 unique peer-reviewed articles were screened after exclusions and deduplication. Abstract screening yielded further exclusions including publication type (n = 536), being published or accepted for publication before NASEM report publication (n = 51), and for including nonhuman subjects (n = 109). The most common publication type excluded was case study/case reports (n = 258 [48.1%]) followed by commentary or editorials (n = 84 [15.7%]). These exclusions were not mutually exclusive, resulting in 469 articles requiring full-text secondary screening. Of these, 241 articles cited the NASEM report and 53 mentioned the committee’s definition for diagnostic error. Most articles referenced the NASEM definition in the introduction (61.5%), methods (15.4%), or discussion (19.2%). Upon full-text screening, additional articles were excluded based on publication type (n = 66; e.g., narrative reviews, opinion, viewpoints, or editorials), research not focused on diagnostic error (n = 112), or not referencing the NASEM definition (n = 29). Full-text review of the remaining 34 articles that directly referenced the NASEM committee’s definition was completed independently by 2 authors (T.D.G., K.M.S.). Upon full-text review, an additional 8 articles were excluded where the NASEM diagnostic error definition was included, but the concept of diagnostic error was not used or was not under evaluation in their study (e.g., diagnostic safety, uncertainty) and another 10 articles were excluded for publication type. Sixteen studies were included in the final analysis and abstracted to determine how diagnostic error was operationalized (i.e., what is being measured; Table 2 ). Of those included, 13 were U.S. studies, 2 were in Japan, and 1 was in Germany. Nine studies 20 , 23 , 24 , 26 , 29 , 30 , 32 , 33 , 35 indicated using the NASEM definition, 5 of those 20 , 23 , 24 , 30 , 32 operationalized it using a definition proposed before the NASEM report (see Table 3 for list of definitions). Three studies 21 , 31 , 34 operationalized error using existing definitions, and 4 studies 22 , 25 , 27 , 28 operationalized components of the NASEM definition (i.e., accuracy, timeliness, communication) for the purpose of the study and did not cite existing definitions. To capture content focus, we grouped studies according to the area of focus for which the definition was used: epidemiology, patient perspectives, measurement/surveillance, and clinician perspectives. Articles Included in the Final Analysis With Corresponding Definition of Diagnostic Error Used EHR, electronic health record; MOD, missed opportunities in diagnosis. List of Definitions of Diagnostic Error Most Commonly Cited *Indicates an adapted definition after correspondence from the author. Six studies focused on epidemiology—incidence, frequency, outcome, and contributory factors—and explored accuracy and timeliness concepts. Of the studies in this category, 3 used malpractice claims, 20 – 22 2 used retrospective data, 23 , 24 and 1 used root cause analysis reports. 25 Both Newman-Toker et al 20 and Lee et al 23 indicated using the NASEM definition of error. However, both studies operationalized error using a combination of pre-NASEM definitions—wrong and missed/delayed diagnoses. 36 Missed and delayed diagnoses were not considered distinct concepts. 10 Watari et al, 21 looking at clinical outcomes, operationalized errors using the concept of delayed, wrong, missed diagnosis by Graber et al. 9 Gupta et al 22 operationalized diagnostic error by selecting from existing categories of malpractice claims: failure to diagnose, delay in diagnoses, wrong diagnosis, and other—consistent with the definition used by Graber et al. 9 Rinke et al 24 examined the frequency of missed elevated blood pressure (BP), abnormal laboratory values, and adolescent depression in primary care pediatrics using the NASEM definition. They operationalized the definition through use of children/adolescent clinical guidelines for diagnosing high BP and by evaluating follow-up of abnormal test results. For depression, Rinke et al 24 used the Singh 11 missed opportunity definition (e.g., missed opportunity in making a correct or timely diagnosis) that implied that an evaluation for a diagnosis of adolescent depression was not pursued. Finally, Dadlez et al, 25 looking at diagnostic process errors (i.e., failure points and contributing factors), evaluated missed diagnosis of adolescent depression, missed elevated BP, and missed actionable laboratory values in the pediatric ambulatory setting. None of the studies in this category focused on the concept of communication to patient. Five studies explored patient reports of diagnostic errors. Aoki and Watanuki, 27 and Bontempo and Mikesell 28 asked patients specifically about their error experiences related to accuracy. Aoki and Watanuki 27 asked primary care patients whether a doctor had made a wrong diagnosis or misdiagnosis. Bontempo and Mikesell 28 focused on diagnosis of endometriosis, asking patients whether their endometriosis had ever been misdiagnosed by another physician and calculated diagnostic delay. Only Sacco et al 26 operationalized each component of the NASEM definition from the patient perspective: (1) accuracy, wrong explanation for a health care problem; (2) communication, confusion about the explanation of the health care problem; and (3) timeliness, it took too long to receive an explanation for a health care problem. The remaining 2 studies (Giardina et al 29 , 30 ) analyzed existing data sets of patient reports of adverse events and patient complaints, respectively. Giardina et al 29 conducted a secondary data analysis of a subset of patient-reported errors and analyzed open-ended narratives using the NASEM definition for inclusion without further operationalization of accuracy, timeliness, and communication. Giardina et al 30 analyzed a subset of patient complaint categories and used the NASEM definition to identify complaints “concerning” for diagnostic error—operationalized as any language about a diagnosis, mention of a potential patient safety issue, and/or any clinician behaviors related to communication. These cases were further reviewed using the Singh 11 missed opportunity definition to determine the presence/absence of diagnostic error using the lens of both accuracy and timeliness. Three studies focused on measurement and surveillance of diagnostic errors. The prospective observational study by Soleimani et al, 31 focused on accuracy and timeliness, used medical record review that applied diagnostic criteria (e.g., a new diagnostic label within 24 hours after rapid response team and whether any features, indicative of that diagnosis, were present for greater than 6 hours before the first documentation of that new diagnosis). Where there was disagreement, a second review was conducted using the taxonomy delineating stages in the diagnostic process used in the study by Schiff et al 12 : history, physical exam, testing, assessment, referral/consultation, and follow-up. Jayaprakash et al, 32 also using a retrospective medical record review of patients who experienced a rapid response team call, used the NASEM report to operationalize error into 2 parts called diagnostic fidelity—diagnostic error and diagnostic delay—to capture accuracy and timeliness. Diagnostic error was defined as failure to establish an accurate diagnosis or failure to communicate the diagnosis in medical records, and diagnostic delay was the failure to establish a timely explanation of the patient’s health problem and communicate it in the medical records. Perry et al 33 conducted a quality improvement project to implement a methodology to identify and measure diagnostic error using the concepts of accuracy and communication across a single pediatric academic center using medical record review. The authors determined the presence of an error when there was a deviation from generally accepted performance standards, if the diagnosis could have reasonably been made based on available information at the time of presentation, and if any diagnostic uncertainty was discussed with the patient or family. Two surveys on clinicians’ perceptions of diagnostic errors were identified. Matulis et al 35 conducted a survey of perceptions of outpatient internal medicine clinicians and included the NASEM definition on the survey itself, presumably including accuracy, timeliness, and communication. The primary care physicians’ survey by Donner-Banzhoff et al’s 34 operationalized error as cases where the original diagnosis later turned out to be wrong (e.g., a case in which an undesirable diagnostic outcome had occurred as defined by Olson et al 37 ) and treatment was delayed. Qualitative analysis of the SME interviews revealed an agreement that the NASEM definition resulted in a fundamental shift in diagnostic safety to prioritize patient and family perspectives. “There is no diagnostic error work without it being patient centered…the whole point of diagnostic error work is to improve the care of patients.” (Participant [P] 1004). However, not all SMEs agreed that lack of communication to the patient should be considered a diagnostic error. Some suggested that it may alternately be an indicator of diagnostic safety. “I do think it’s an important part of the diagnostic process. I think it’s part of diagnostic quality, but I don’t see it…as part of the definition.” (P1001). Overall SMEs indicated that the NASEM definition functions as a baseline for researchers to conceptualize diagnostic error. Furthermore, the publication of the NASEM definition legitimized and has drawn attention to diagnostic error as a distinct issue within the larger context of patient safety. The act of citing the report may highlight agreement across researchers about the relevance of this change and its impact on the field. “Now…we use the National Academy of Medicine definition and then people leave us alone. So, in that sense for researchers around the definition, for those of us who like the definition and use it, that’s a change. I don’t know that it’s a substantive change from the field standpoint, but it’s an important change…that facilitates a certain amount of cohesion, which I think was the intent of the definition.” (P1008). There was no consensus on operationalizing the definition among SMEs. Most acknowledged that researchers cite NASEM but continue to use their preferred definition. “I would say I actually have a much harder time operationalizing that definition into some of the work that I have done.” (P1005). Although most of the SMEs expressed positive views of the definition, there was some frustration conveyed about ambiguity of concepts, especially timeliness and communication, which may limit the ability to compare outcomes across studies.

We found that, in the 6 years since the publication of the NASEM committee’s report, Improving Diagnosis in Healthcare , many studies mention the NASEM committee’s definition of diagnostic error, whereas few studies actually operationalize the definition. Most authors focused on accuracy and timeliness, with only 3 studies exploring communication to the patient. Future efforts should bring together established experts and emerging scientists in the field of diagnostic safety to formulate a set of common approaches for operationalizing each component within the NASEM report definition in various contexts. In line with the patient-centered focus of the NASEM report, patients and families should also take a lead role in defining the construct of “communication to the patient.”

This scoping review yielded several important findings to advance research related to measurement and reduction of diagnostic error. First, although many of the research articles we reviewed referenced the NASEM report as an anchor for their work, we only identified 16 studies that focused on diagnostic error and that both cited the NASEM report and included the NASEM definition in the text of the article. Second, the NASEM report has significantly influenced patient-centeredness concepts in discussions of diagnostic error definitions. Much of this progress has been from the inclusion of the patient perspective and the concept of communication to the patient within the definition. Third, many researchers reference the NASEM report and definition, yet continue to use pre-NASEM or thought leaders’ definitions to operationalize accuracy (misdiagnosis, wrong diagnosis) and timeliness (missed/delayed diagnosis); few address communication at all. This is likely because these definitions more applicable to their research context or the disease under study and the concept of communication of a diagnosis to a patient was not necessarily considered a diagnostic error before the NASEM report publication. The review identifies a small but growing body of literature focused on inclusion of patients’ perspectives. Although only 1 article operationalized all 3 components of the NASEM definition, 1 the increasing focus on patients’ experiences reflects the SMEs’ consensus that the NASEM report legitimized the patient’s role in the diagnostic process. However, SMEs did not all agree that failure to communicate a diagnosis to a patient should be considered an error. That ambivalence is also reflected in the literature. For example, we did not find concepts related to communication of a diagnosis to patients well established in our review, and only 1 study 26 specifically attempted to measure diagnostic communication (e.g., confusion about the explanation of the health care problem). Patient-physician communication has been well studied, 38 including how communication contributes to pitfalls in the diagnostic process. 28 , 29 , 39 Although concepts related to communication of a diagnosis to the patient comparatively have not been as well studied or operationalized, there is a growing body of literature related to timely communication of abnormal test results. For instance, a large number of studies have identified lack of timely communication of abnormal test results to patients and provided specific metrics with which to measure and improve the process. 40 – 48 Additional work is needed to study how factors related to information, timing, method, and behaviors play an integral role in communication in the context of the diagnostic process. 28 , 29 An increased emphasis on patient-partnered or patient-oriented diagnostic research and the application of interdisciplinary research from fields outside of medicine can provide knowledge on how to measure and improve diagnostic communication. Given that the components of accuracy (e.g., missed, wrong, misdiagnosis) and timeliness (e.g., delayed diagnosis) were often cited and debated pre-NASEM definition (as early as 2005), it is not surprising that these concepts were addressed most often in our review (n = 16 and n = 13, respectively). Researchers may be more comfortable focusing on these more well-established components and tend to select pragmatic definitions that have been applied in prior work. Use of such existing definitions may allow for greater generalizability of results and easier comparisons between studies, particularly in a field where science is nascent. Another area ripe for exploration is how definitions can account for concepts related to both diagnostic processes 49 such as missed opportunities 11 and outcomes such as clinical endpoints (e.g., harm, morbidity, and mortality 50 ), given that diagnosis is an intermediate outcome. This will enable better generalization of results and comparison across studies, and advance the field of diagnostic safety. Although the NASEM report highlighted the role of patients in the diagnostic process, our review shows that more paradigm shifting results are still awaited. More than one-third of the studies (n = 6) operationalized diagnostic error, whether the authors indicated using the NASEM definition or not, in ways that were specific to their research questions rather than using an existing operationalized definition (Table 3 ). Because some of these studies included citations from leaders in the field, it is less likely to be an issue of awareness of existing definitions but an issue of fit. Uncertainty is ubiquitous in diagnosis, and definitional concepts related to timeliness and accuracy may also vary by conditions being studied. Presumably, the NASEM definition is intentionally broad, allowing researchers more flexibility within the core components to be creative and thoughtful about how errors are identified and measured—especially considering the significance placed on patients’ experiences. Given the inherent difficulty in creating an absolute definition to identify error, perhaps the field may need to acknowledge and encourage diversity and innovation in safety measurement as long as the goal is to reduce patient harm. 14 The publication of Improving Diagnosis in Health Care consolidated existing diagnostic error knowledge and, in doing so, has articulated the importance of accuracy, timeliness, and communication. These concepts were chosen based on pioneering research published between 2005 and 2015, providing a foundation upon which future researchers can build their work. This approach should accelerate research and enhance generalizability of research findings. Of the 240 articles that underwent full-text screening, we identified 29 articles that referenced the NASEM report and focused on diagnostic error, but that did not include or use the NASEM definition. Although these articles fell outside the scope of our research question, this may be indicative of the difficulty in operationalizing the NASEM definition. It may also be in line with our SME’s suggestion that researchers cite the report to legitimize the study of diagnostic safety. Perhaps citing the report implies agreement with the definition—especially because much of the literature used the pre-NASEM definitional concepts (e.g., wrong, missed/delayed) that were foundational for the report. In the articles that did include the NASEM definition (n = 52), most mentions occurred in the introduction or discussion sections of the paper suggesting that, although authors believed the definition to be important and relevant, few directly attempted to operationalize the definition or even apply it to their own research. The institutional authority of NASEM has provided credibility to the diagnostic safety movement, and diagnostic errors have rapidly emerged as a leading patient safety issue in the United States. 51 While acknowledging the disagreement among thought leaders, NASEM has effectively produced a common understanding of diagnostic error—accuracy, timeliness, and communication. In our interviews, SMEs acknowledged this and suggested that citation of the NASEM report provides credibility and helps to move toward consensus. Subject matter experts also indicated that, despite any remaining disagreements about measurement of the concepts, the definition unified the field. Even when researchers do not specifically operationalize the NASEM definition and/or cite the report while using previous definitions, the work often falls within the core concepts of the NASEM definition. Application of definitional concepts may also vary by the context or disease or setting in which the specific study is being undertaken. For example, timeliness of diagnosis in cancer would be operationalized differently from that in appendicitis. Thus, additional work can help inform a set of common approaches for operationalizing each component within the NASEM definition. Our review has several limitations. Although we attempted to conduct an exhaustive search of the diagnostic error literature, it is possible that some studies were not included. We attempted to address this by conducting a broad search in multiple databases by a librarian scientist as well as including purposive hand searching of reference lists, table of contents of domain-specific journals, and key author searches. In addition, it is possible that some of the studies were conceptualized and designed before the report publication. We attempted to address this by excluding articles accepted before NASEM report publication or that indicated within the methods section that the study was initiated before the NASEM report. We cannot fully eliminate bias in our analysis because our team itself is made up of diagnostic safety researchers. Methodological limitations may also exist because only English language studies were included. Finally, our study did not address other important conceptual issues such as process versus outcome aspect of diagnosis and validity and usefulness of the various components of the NASEM definition. 10 , 11 , 49