A framework for defining diagnostically challenging conditions identifiable through electronic algorithms.

While e-triggers are helpful in research and identifying process breakdowns, their utility in improving diagnosis in specific clinical situations is limited; there needs to be more real-time detection of patients with specific conditions at risk for diagnostic errors. DCCs can evolve over months or years, are complex, and thus require modification from previously published approaches to detect missed test results or care escalations. Clear, replicable, and standardized criteria are needed to define and then identify specific DCCs more amenable to electronic detection. This would also allow translation of the approach across contexts and conditions. The proposed DCC framework can identify clinical situations that evolve over prolonged periods and have a high likelihood of diagnostic delays and where a timely diagnosis can improve patient outcomes. This proposal extends a previously proposed framework to define undesirable diagnostic events (UDE).( 19 ) While the UDE framework focuses on specific undesirable events , the DCC framework focuses on specific conditions where patients are at great risk of being missed or undiagnosed before these events occur . Most importantly, the framework aims for more proactive identification, whereas UDEs were designed to identify and measure diagnostic errors in retrospect. Researchers, clinicians, and quality improvers can use this DCC framework to proactively identify patients with potential conditions at high risk for diagnostic error. Electronic triggers can be developed based on this proposed framework to help monitor patients in their diagnostic journeys and identify when there is a need to alter the course of the diagnostic process. Figure 1 illustrates the conceptualization of this approach by comparing a patient’s diagnostic journey with no triggers, retrospective triggers, and prospective triggers. The goal of this framework is to help inform future research and development of e-trigger algorithms to proactively identify patients with certain DCCs in a timely manner. Identifying patients earlier in their diagnostic journeys can result in fewer missed or delayed diagnoses and improved patient outcomes. Next, we will provide a case study on a condition that is an exemplar for this proposed framework and then present the DCC criteria.

Table 1 presents the DCC criteria for identifying conditions that are difficult to diagnose and have a diagnostic process amenable to algorithm development. Table 2 provides examples of both good and poor DCC candidates. Candidate DCC conditions must have a high enough prevalence ; this is to maximize benefit but also ensure enough data is available for research and practice improvement. For simplicity, we limit our definition of intermediate-to-high in prevalence to conditions that have a prevalence greater than 0.06%, the definition of a rare disease within the United States. Conditions that fall under the umbrella of DCC should be prone to diagnostic error , according to the literature or expert opinion. This ensures that there is room for improvement in current practices used to pursue the diagnosis. These conditions often result in preventable harm, and many are susceptible to “diagnostic pitfalls,” defined as “clinical situations and scenarios that are vulnerable to errors that may lead to missed, delayed, or wrong diagnoses”.( 28 ) One way to identify DCCs is to consider whether there is a prolonged lag time from symptom onset to diagnosis . Importantly, the lag time and time to diagnosis varies across conditions; some conditions may be rapidly evolving, whereas others may be more slowly evolving. For example, sepsis is a serious and rapidly evolving condition where early recognition and treatment are crucial. ( 29 ) On the other hand, many conditions have a much longer timescale for lag time to diagnosis, spanning from months to years. For example, ankylosing spondylitis typically takes 5–10 years to be diagnosed. From a research standpoint, there are more opportunities to develop algorithms and direct interventions to improve patient outcomes for slowly evolving conditions . Instead of real time alerts in conditions such as sepsis, we focus on identifying conditions that progress over months or years and leverage vast amounts of EHR data in ambulatory care settings to provide information to a health system or care team to intervene. Furthermore, DCCs are often conditions that are associated with red flags , previously defined as “a constellation of symptoms, signs, clinical data or circumstances, conceptualized by an individual clinician, that should lead to heightened suspicion for a serious condition and trigger additional evaluation”.( 30 ) That is, while conditions may have subtle and/or atypical presentations, we posit that DCC will often have signs or symptoms with higher specificity to aid diagnosis. Another criterion is to prioritize conditions where a timelier diagnosis will positively impact management, prognosis, and/or the patient’s overall quality of life . Prioritization should be given to conditions that, when caught early, can have the greatest positive impact on patients’ lives. The case study above provides an example of a condition where a timely diagnosis can be life-changing. Because this proposed framework focuses on research and eventual algorithm development, the final two criteria points are crucial. First, it is important that the conditions have an established and commonly available pathway for diagnosis . It would be difficult or impossible to build an algorithm for conditions with no established pathways. For example, this would exclude diagnosis by exclusion or serendipity. Thus, the diagnosis should be able to be made by most clinicians in most settings and not just by a “genius diagnostician”; we should normalize excellence rather than encourage diagnostic heroism. Relatedly, conditions within our DCC framework should be objectively verifiable . In other words, to be amenable to trigger development, conditions must have an objective, time-bound, valid, and nearly universally available reference standard for verification of diagnosis; the process of defining either the presence or absence of the disease should be unambiguous and readily available in the EHR.

John is a 64-year-old man with a history of tobacco use with cough and shortness of breath for several months. He was seen by his primary care physician, who ordered a chest radiograph that showed prominent basilar reticulations. John was empirically started on inhalers for possible chronic obstructive pulmonary disease (COPD). Pulmonary function tests obtained 6 months later demonstrated restrictive physiology. John returned to primary care due to lack of improvement with inhalers. A cardiac stress test was ordered but he developed acute worsening of dyspnea and presented to the emergency department. Computed tomography (CT) of chest was concerning for progression of fibrotic lung disease. He was subsequently referred to a pulmonologist and diagnosed with idiopathic pulmonary fibrosis 1.5 years after onset of symptoms. He was prescribed antifibrotic therapy, which modulates disease progression but does not reverse fibrosis already present. A more timely diagnosis could have attenuated the rate of progression earlier. Fibrotic interstitial lung disease (fILD) is an umbrella term used to describe a group of disorders that cause progressive lung scarring. Although subtypes of fILD, such as idiopathic pulmonary fibrosis as described in the case above, are diagnosed by pulmonologists, the umbrella diagnosis of fILD can be made based on a chest CT alone by an internist or a radiologist. Notable, without treatment, median survival of the most severe subtypes ranges from 2–5 years. Fibrosis is irreversible once present; thus, the goal of current therapeutic approaches is to prevent new fibrosis formation through timely diagnosis and treatment. Although diagnosis is time-sensitive, on average, patients with fibrotic interstitial lung disease face a 2-year diagnostic delay.( 20 , 21 ) Factors that contribute to delays include misdiagnosis (e.g., attributing the dyspnea to COPD or cardiac causes), lack of disease awareness among patients and health care team members, delays in obtaining imaging, underrecognition of fibrosis on imaging (especially during early stages of disease), and delays in referral to pulmonary subspecialists.( 22 – 25 ) In this vignette, the patient experienced several missed opportunities for diagnosis. Patients are often not diagnosed until more advanced stages of a progressive disease or at time of acute decompensation necessitating inpatient hospitalization. Researchers have begun leveraging EHR data to create electronic phenotypes of disease.( 26 ) Others have focused on reducing diagnostic delays caused by underreporting fibrosis on chest imaging (the cornerstone of diagnosis) through novel AI-based quantitative CT measures. These measures objectively quantify parenchymal lung abnormalities, supplement traditional qualitative imaging reports, and make ILD more amenable to identification using the concept of DCC and e-triggers.( 27 ) E-triggers could be applied to detect abnormal tests that are diagnostic of disease (e.g., chest CT with fibrosis; pulmonary function tests that show restriction) but have not yet been followed up, multiple visits to primary care with refractory dyspnea and cough, and future use of large language models to identify common misdiagnoses and/or red flag symptoms in clinical information.

Large data sets can identify DCCs through signals that indicate potential missed opportunities in a patient’s diagnostic journey. The concept of triggers has been used in healthcare settings to help measure and monitor patient safety concerns. Specifically, electronic triggers (or e-triggers) offer a promising way to use electronic health record (EHR) data to identify patient safety events and have been applied to identify certain types of diagnostic errors and process breakdowns. The Safer Dx Trigger Tools Framework,( 13 ) adapted from the Safer Dx Framework,( 10 ) can help healthcare systems develop and implement e-triggers to systematically measure diagnostic errors using EHR data. The tools can effectively and efficiently identify patients at high risk for errors or delays in care, especially compared to alternative methods like voluntary reporting or manual chart reviews, which are labor-intensive, retrospective, and time-consuming.( 13 , 14 ) Several diagnosis-specific e-triggers have been developed and validated. For example, missed test result e-triggers (e.g., related to diagnostic evaluation of prostate cancer,( 15 ) colon cancer,( 16 , 17 ) and thyroid dysfunction( 18 ) alert clinicians if a patient has an abnormal test result without timely follow-up in a defined time period. Care escalation e-triggers identify patients whose level and/or intensity of care was escalated unexpectedly, potentially related to a missed diagnostic opportunity. E-triggers can help identify where missed opportunities may have occurred and ensure adequate follow-up and care. They can also identify where process breakdowns occur, providing learning opportunities.

We propose a novel DCC framework to identify difficult-to-diagnose conditions likely to be associated with missed opportunities and where electronic algorithms can assist in identifying missed opportunities in real-time. Use of this proposed framework can allow development and implementation of algorithms to identify patients with DCCs proactively and prevent delays in care.

One limitation of this proposed framework is the intentionally narrow focus. This restrictive approach limits generalizability, particularly for conditions that are not objectively verifiable. For example, difficult-to-diagnose conditions of many psychiatric and rheumatologic conditions likely do not fit within the scope of our framework. Additionally, the need for longitudinal access to patient history and data is crucial for the development, implementation, and success of e-triggers. Thus, e-trigger accuracy may be limited in patients who do not receive most care within a single system and/or when the EHR data is fragmented. In addition, there is potential that the implementation of e-triggers could lead to increased and unnecessary testing in certain populations, leading to waste and overtesting. Real-world studies should evaluate such downstream effects. Finally, triggers and diagnostic approaches must continually be updated as more precise and accurate tests become available in practice. The next steps for piloting and evaluating this proposed framework for identifying difficult-to-diagnose conditions include designing exact condition-specific examples based on real-world EHR data and then validating the use of the derived condition-specific criteria in retrospective datasets. Once derived and validated, prospective studies of implementation should be pursued in health systems that measure the effectiveness of the approach at improving diagnostic and patient outcomes, including balancing measures such as test utilization and overtreatment. While it is likely most pragmatic to perform pilot studies in systems in relatively “closed” systems such as the VA, it will be imperative to evaluate the use of DCC criteria-based e-triggers in other representative health care settings.

A diagnostic error is “the failure to establish an accurate and timely explanation of the patient’s health problem(s) or communicate that explanation to the patient”.( 1 ) Diagnostic errors affect at least 5% of outpatients annually in the U.S., ( 2 ) with similarly high rates of errors in inpatients.( 3 ) While delays in diagnosis are common, defining the timeliness of diagnosis is challenging, given the lack of standard definitions. Many patients spend years undergoing diagnostic evaluation, experiencing diagnostic “odysseys”.( 4 ) Their odysseys include seeking care from disconnected healthcare providers and systems, resource expenditures, and loss of trust, in addition to the delays in appropriate and effective treatment of their diagnostically challenging condition (DCC). The diagnostic process is dynamic and evolves over time and space, making diagnostic errors and successes difficult to identify and analyze.( 5 – 7 ) Many sociotechnical factors have been found to contribute to missed opportunities for diagnosis.( 8 , 9 ) The Safer Dx Framework highlights several dimensions where breakdowns may occur in the diagnostic process (e.g., patient-provider encounter, diagnostic test interpretation, follow-up, and referral issues).( 10 ) Breakdowns in these diagnostic processes can result in diagnostic errors and patient harm. Systematic use of strategies to identify patients who may be experiencing missed opportunities for diagnosis can support clinicians to pursue alternative diagnoses in a timely manner. Methods are needed to systematically identify patients with DCCs who are experiencing diagnostic odysseys and potential missed opportunities in their diagnosis.( 11 , 12 ) The wide availability of longitudinal EHR data to map a patient’s journey makes such methods possible. We propose a new framework to proactively identify patients with DCCs using EHR data to improve the timeliness of diagnosis and thus health outcomes.