Comparison of Diagnostic Recommendations from Individual Physicians versus the Collective Intelligence of Multiple Physicians in Ambulatory Cases Referred for Specialist Consultation.

The Human Diagnosis Project (Human Dx) platform is a mobile application that allows individuals with any level of medical training (medical student, resident/fellow, attending physician) to both: 1) submit their own clinical case to elicit feedback on the diagnosis and plan; and 2) contribute feedback on diagnoses and plans for any case submitted by other Human Dx users. For this study, Human Dx users responded by using free text to submit a ranked list of differential diagnoses and suggested next steps for the plan of care. Submitted cases include a one-line summary of the case, a clinical question, and relevant history, physical exam, and diagnostic tests (e.g., laboratory or imaging results). ( Appendix 1: Example Case ) This study was approved by our institution’s institutional review board. At the time of this study, Human Dx used a 1/n proportionally weighted algorithm based on individual user responses to produce a CI for the case, as previously described 20 (see Appendix 2: Collective Intelligence Rule ). In brief, for a submitted case, the Human Dx algorithm creates a CI output composed of respondents’ ranked list of diagnoses (collective differential) and recommendations or next steps (collective plan). (See Appendix 3: Sample Collective Intelligence Output .) This CI output reflects both how frequently a diagnosis or plan appears among all responses and its ranking on each respondent’s ordered list (e.g., top diagnosis versus fifth diagnosis), but this automated process does not account for alternative spellings of the same recommendation (e.g., blood pressure measurement or BP measurement). To acquire a sample of diverse real-life cases to submit to the Human Dx platform, an investigator (EK) reviewed actual specialist consultation requests in endocrinology, gynecology, and neurology cases submitted by clinicians at an integrated healthcare system from 2015 to 2017, using the healthcare system’s existing electronic consultation platform (e-consult). 25 , 26 These three specialties were chosen because they are areas in which primary care clinicians have some knowledge, and within this healthcare system, a specialist is required to review e-consult requests prior to scheduling an in-person specialty clinic appointment. As a result, the specialist consultants often provide recommendations to the referring clinician to advance patient care prior to scheduling a patient for an appointment. For each of the three specialties, an investigator (EK) selected ~15-20 cases for which there were clear diagnostic steps recommended by the e-consult specialist. Most cases were early in the diagnostic process, and no diagnosis was provided through the e-consult communication. We focused on cases where recommendations were provided about next steps for evaluation of the patient (i.e., the plan) rather than diagnosis. Cases were selected to ensure no chief complaint was represented more than twice. Few cases had guideline-recommended approaches for working up the patient’s complaint. Therefore, to assess the appropriateness of the recommended approach in the study cases, the study team used agreement between two specialist physicians as indicative of a reasonable standard of care and the basis for comparison (i.e., specialist-consensus recommendations), per an established approach drawn from the patient safety literature 27 , 28 ( Figure 1 ). The first specialist (specialist A in figure 1 : step 1) was the initial specialist within the integrated healthcare system that responded to the e-consult. To acquire the recommendations of a second specialist, an investigator (ECK) entered case information onto the Human Dx platform from August 2017 to March 2018. From board-certified specialist users on Human Dx, the study team recruited an endocrinologist, neurologist, and gynecologist to respond to the study cases entered on Human Dx in their specialty (specialist B in figure 1 : step 2); each of these specialists responded to the case by submitting a differential diagnosis and planned next steps from September 2017 to April 2018. The Human Dx specialists had access to the exact same information as the other Human Dx users who later responded to the case ( Figure 1 : step 4). If the e-consult specialist and Human Dx specialist agreed on at least one recommended next step in a submitted case, we included that case in this analysis (N=35). The two specialists reached agreement on at least one recommendation in 12/14 (86%) endocrinology, 13/19 (68%) gynecology, and 10/19 (53%) neurology cases. We designated recommendations that both specialists agreed upon as “specialist-consensus recommendations” and established this as the gold standard against which to assess our outcomes. Each of the 35 cases had one to six specialist-consensus recommendations. Appendix 4 contains one-line summaries of cases included in this study with their specialist-consensus recommendations. We solicited responses to the 35 cases with specialist-consensus recommendations from Human Dx users from August 2017 to November 2018 until we had a minimum of three respondents, which is the number after which the accuracy of CI plateaus. 20 We only included respondents with medical degrees who practiced within the United States (due to differences in practice patterns and available resources among countries). The physician respondents could be trained in any specialty, including endocrinology, gynecology, or neurology. The CI output for each case was derived from the responses of all physician respondents within the US (excluding the designated specialist B for each case). We designated the first three US-based physician respondents to each case (who could be trained in any specialty) to serve as our comparison cohort of “independent individual physicians” ( Figure 1 : step 6). Designating these physicians and their individual recommendations was meant to serve as a proxy for a clinician practicing independently in the community without specialty access. The responses from these three respondents were also included in the CI output. There were variable numbers of respondents to each case, and we wished to avoid any one case from contributing more than any other case in comparing independent individual physician respondents’ recommendations against the CI output. Accordingly, by capping at three those included in the “independent individual physician” cohort, we ensured that each case had an equal contribution to the outcome. We collected respondents’ level of training, location, and specialty based on self-reported information provided when individuals registered on the Human Dx platform. Our primary outcome was the appropriateness of recommended diagnostic next steps, based on agreement with specialist-consensus recommendations. We report this outcome separately for: (a) the CI output, and (b) the first three individual physician respondents (the independent individual physician cohort), which is consistent with the approach used in a prior CI study on the same platform. 20 We defined appropriateness at four different levels (from most to least appropriate): Strict appropriateness: all specialist-consensus recommendations, regardless of the number of recommendations, appear at the top of the ranked list from the CI output or an individual. (If there were five specialist-consensus recommendations, the top five recommendations in the CI output or provided by an individual were the five specialist-consensus recommendations.) Moderate appropriateness: did not meet strict level criteria but all the specialist-consensus recommendations were ranked highly within the CI output or the individual’s recommendations. Specifically, if there were X number of specialist-consensus recommendations, all of them appeared within the top 10 or the top 3 * X number of recommendations (whichever was lower). We used two measurement criteria because the number of recommendations varied across cases from one to six. If a case had only two specialist-consensus recommendations, then both recommendations would need to appear in the top six CI output’s recommendations or an individual’s recommendations (3 * 2). Alternatively, if a case had five recommendations, we looked only at the top ten, rather than the top 15 (3 * 5) recommendations. Lenient appropriateness: at least one but not all specialist-consensus recommendations appeared within the top 10 or top 3 * X recommendations from the CI output or an individual. Not appropriate: none of the specialist-consensus recommendations appeared within the top 10 or 3 * X recommendations from the CI output or an individual. The secondary outcome was the potential harm of recommendations. We assessed the potential for meaningful harm based on a scale previously employed to classify the harm of errors. 7 , 29 For each recommendation, we used a binary outcome that focused on potential for at least moderate meaningful harm, which included: initiation or cessation of medications without indication or with contraindications; invasive testing; exposure to unnecessary radiation beyond a plain radiograph; and any other actions determined by two physician investigators (ECK, SSN) to have potential to result in at least moderate harm. We only assessed harm for recommendations that were not specialist-consultant recommendations. For both the CI output recommendations and the independent individual physician cohort recommendations, we report recommendations that were identified as having potential for at least moderate harm that appeared among the top 10 or top 3* X recommendations (whichever was lower, as per the primary outcome). Two investigators (ECK, SSN) independently assessed the primary and secondary outcomes, and manually eliminated duplicate recommendations (e.g., a list of eight recommendations with one set of duplicates [“EKG” and “ECG” each appeared] would be treated as a list of seven recommendations), then reached agreement on appropriateness or harm of all recommendations in all cases. We used descriptive statistics to report characteristics of respondents and all outcomes. We used chi-squared testing to determine differences in the appropriateness and harm of the CI output versus the individual physician cohort. For the CI output, we report both outcomes at a case level (out of 35 total cases). For the independent individual physician cohort, we report the outcomes at the individual level; therefore, the overall assessment of individual physicians is out of 105 physicians (35 cases * 3 physician respondents per case). Our funding source had no role in this study.

A total of 177 physicians responded to the 35 cases (12 endocrinology, 13 gynecology,10 neurology) on Human Dx. A median of 7 physicians (interquartile range [IQR]: 5-10) responded to each case. Table 1 shows characteristics of respondents. As shown in figure 2 and detailed in Appendix 5 , when combining all levels of appropriateness, the CI output performed better than the independent individual physicians, respectively, in each specialty and overall: endocrinology (7/12) 58% versus (12/36) 33%, gynecology (10/13) 77% versus (18/39) 46%, neurology (7/10) 70% versus (17/30) 57%, and overall (24/35) 69% versus (47/105) 45%. These differences in appropriateness were statistically different for the cases overall (69% vs 45%, X 2 = 5.95, p=0.015) but not within any specialty. Figure 2 displays the level of appropriateness (strict, moderate, or lenient) of the CI output recommendations compared with the independent individual physicians. (Data also shown in Appendix 5 .) Unlike the results for any level of appropriateness, the CI output did not consistently perform better than individual physicians when considering only strict appropriateness. CI achieved strict appropriateness for seven cases (20%) overall and for none of the endocrinology cases. In contrast, individual physicians achieved strict appropriateness in 22% of cases overall and at least one of three individual physicians achieved strict appropriateness in each of the three specialties. Among all cases, strict appropriateness was the most common level of appropriateness that individuals achieved (22%) vs moderate (9%) or lenient (14%) appropriateness, whereas the CI achieved higher rates of moderate (23%) and lenient (26%) appropriateness. Some individual physicians achieved strict appropriateness in cases where the CI did not. Specifically, three physicians provided strictly appropriate recommendations for an endocrinology case while the CI did not provide strictly appropriate recommendations for any endocrinology cases. When evaluating the top 10 or top 3* X recommendations, one or more recommendations from the CI output for 19 (54%) of the cases (6 endocrinology, 7 gynecology, and 6 neurology; or 54%) had potential for meaningful harm ( Table 2 ). In most cases, one or fewer recommendations in the CI output had potential for harm: endocrinology (median 0.5, IQR 0-2); gynecology (median 1, IQR 0-1); neurology (median 1, IQR 0-2). Among the individual independent physician cohort, 41% of 105 respondents (43 total: 13 endocrinology; 16 gynecology; 14 neurology) also submitted at least one recommendation with potential for meaningful harm. Of the 105 individuals, the majority provided recommendations with no potential for moderate harm (median 0, IQR 0-1). There were no differences in the number of harmful recommendations recommended by the individual physician cohort when comparing specialties (median 0, IQR 0-1 for all three specialties). These recommendations with potential for harm suggested by 43 individual physicians were distributed across 80% of the cases (n=28; 8 endocrinology, 11 gynecology, and 9 neurology). The potential for harm was not statistically significantly different when comparing the CI output in the 35 cases to the recommendations submitted by the 105 physicians in the independent individual physician cohort: 54% (19/35) vs 41% (41/105, X 2 = 2.49, p=0.11).

In this study, we assessed the performance of the collective intelligence of multiple physicians versus individual physicians in providing appropriate diagnostic steps for a plan of care across three specialties. Although the CI recommendations matched the gold standard specialist recommendations more frequently than most independent individual physicians, the CI recommendations only aligned with specialist recommendations ~70% of the time. Moreover, at least one of the recommendations advised by the CI had potential for at least moderate harm in approximately half of the cases. Our findings are consistent with prior studies demonstrating that CI outperforms individual physicians in diagnostic accuracy 16 , 17 , 19 , 20 and at a rate of a ~30% improvement in accuracy. 19 , 20 However, our study expands the literature in an important way. By focusing on actual cases for which a clinician consulted specialists for advice, we provide evidence that CI provides value early in the diagnostic process (such as during the information gathering stage) prior to reaching a definitive final diagnosis. Feedback may be particularly important during these earlier steps in the diagnostic process. 24 , 30 Specifically, the CI recommendations suggest paths forward in the diagnostic process that may not be explored by a clinician practicing independently. These findings also suggest that CI tools may be beneficial in more complex clinical scenarios, beyond straightforward cases with a known diagnosis. Despite the potential benefits of a CI tool for these types of e-consult cases, we found that there were instances when individual physicians performed better than the collective. Prior studies have had conflicting results as to whether the CI is better than the best individual physician with comparable training 16 or just better than the average physician with comparable training. 19 , 20 Absent a methodology to reliably predict if a specific individual is going to perform better than the CI, clinicians may rely on their own perceptions for when to ignore the wisdom of the crowd. Studies have shown that clinicians are overconfident in their diagnostic ability; in particular, clinicians do a poor job of calibrating their diagnostic accuracy in cases with high uncertainty. 31 , 32 Since cases referred to a specialist are more likely to have higher diagnostic uncertainty, clinicians’ overconfidence in these situations may pose a barrier to adoption of CI tools. Although our findings support the potential for CI to improve the diagnostic work-up in cases when clinicians may request advice from a specialist, we did find potential for harm in over half of cases. However, there was a wide range in the type of harm: from inappropriate initiation of prescription medications, to radiation exposure from unnecessary imaging, to invasive diagnostic testing. The harm of an invasive diagnostic test (e.g., diagnostic laparoscopy) is higher than initiating an inappropriate prescription medication (e.g., gabapentin). Our prior studies suggest that clinicians would not blindly follow all recommendations of the CI output. 33 In particular, we previously identified trust in the source of the recommendation as an important consideration that factored into how clinicians would behave after receiving information from a clinical decision support tool. 33 It is important to also note that even if a collective intelligence recommendation seems relatively benign (additional unnecessary laboratory test), studies increasingly show that overtesting is not only wasteful but can result in patient harm. 34 – 37 Thus, while collective intelligence helps address some of the most common causes of diagnostic errors, such as failure in hypothesis generation or failure to order a necessary test, 2 , 38 , 39 this must be weighed against the harm of pursuing unnecessary tests. Of note, potential harm was also present in nearly half of individuals’ recommendations as well, suggesting that the potential harm of CI recommendations may be similar to harm from an inappropriate/inaccurate individual physician care recommendation. Although there was no statistically significant difference in harm between the CI output and independent individual physicians, this issue warrants further exploration in a larger study. Our study has several important limitations. We used a single CI platform and algorithm. However, prior studies comparing different algorithms used to generate a CI output have demonstrated that algorithmic differences have limited impact on the benefit of CI and that the results of CI collected from one platform are likely generalizable. 16 , 19 , 20 The users of Human Dx may not be representative of all physicians, but by removing medical students and international practitioners from our analysis it is more likely these findings are generalizable to practicing primary care clinicians in the US. We collected our real-world patient cases from a single healthcare system and prioritized cases where specialists provided a clear recommendation. This may result in a selection of less complex cases and skew specialist-consensus recommendations to be compatible with local practice patterns, but by requiring that two specialists agreed on a recommendation, we increased the likelihood that recommendations would be considered appropriate in multiple settings. Nonetheless, use of specialist-consensus recommendations as the gold standard has its limitations, since among specialists there will also be disagreement. Despite these limitations, our study adds to the literature by demonstrating the potential for CI to assist primary care clinicians in identifying appropriate evaluative steps during the diagnostic process, not just for determination of a diagnosis. Our findings suggest several areas for further exploration. For clinicians without adequate or timely access to specialty advice, these findings suggest that access to the collective intelligence of multiple clinicians can provide useful recommendations to advance the diagnostic process. This is consistent with prior recommendations that feedback improves diagnosis. 1 , 24 , 40 , 41 Although the collective intelligence tools currently available do not replace the need for timely access to specialty care, they may help improve the diagnostic process in settings with inadequate specialty access. Given the high rate of inappropriate recommendations, clinicians must be judicious in their acceptance of recommendations. For CI tools, providing users with “quality assurance” data on those providing feedback (e.g., information about expertise, clinical training) is crucial to their acceptance of recommendations. 33 Developers and users of collective intelligence tools should collaboratively explore how to increase the benefits of CI tools (e.g., ensuring necessary diagnostic tests are ordered, expanding the differential diagnosis) while mitigating or providing transparency on how to evaluate the risks from potentially harmful recommendations. Methodologies to help differentiate high quality recommendations from inappropriate recommendations and more highly skilled from less expert contributors may increase the value and uptake of CI tools in actual clinical practice. This is particularly true when considering the growth and improvement in other clinical decision support tools. 42 This work also provides a pathway toward operationalizing provision of feedback to clinicians. Feedback has been identified as a necessary step to increase diagnostic calibration, a concept that describes when clinicians’ confidence in their diagnostic decision making aligns with their actual diagnostic accuracy. Well-calibrated clinicians are better able to identify the correct balance between undertesting (failing to explore a broad enough array of potential diagnoses) and overtesting (exposing patients to the costs and harm of unnecessary tests). 30 , 32 , 40 , 41 , 43 Tools like the one tested in this study provide an approach for clinicians to acquire real-time feedback on their clinical decision making, which may help facilitate diagnostic improvement. Adoption could be incentivized by providing Continuous Medical Education credits for using these feedback tools.

The collective intelligence of multiple physicians provides more appropriate recommendations than individual physicians when using board-certified specialist recommendations as a gold standard for next steps in the diagnostic process. This suggests that a CI tool may provide useful evaluation recommendations even before a specialist weighs in, thereby improving timely and accurate diagnoses in settings where access to specialty care might be nonexistent, sparse, or delayed. Recommendations provided by a CI tool should not be blindly followed, as some have potential for meaningful harm. Moreover, clinicians should be wary of higher-risk diagnostic tests/therapies suggested by the collective intelligence. Future work is needed to explore how best to leverage CI (and digital tools to facilitate collaboration) to address gaps in the diagnostic process without exposing patients to additional unnecessary harm. There is also promise in evaluating the use of CI tools to facilitate more timely feedback to clinicians on their medical decision making.

Diagnostic errors – defined by the National Academy of Medicine as “the failure to establish an accurate and timely explanation of a patient’s health problem or communicate that explanation to the patient” 1 – are frequent and have a significant impact on patient morbidity and mortality. 2 – 7 Prior studies have shown that five percent of US adults who seek outpatient care annually experience a diagnostic error, 5 and that diagnostic errors may contribute to approximately ten percent of patient deaths. 8 , 9 Less is known about diagnostic error than other areas of patient safety. 10 – 12 To advance the field, in 2015, the National Academy of Medicine published Improving Diagnosis in Health Care . 13 This report provided eight key suggestions to improve diagnosis and reduce diagnostic errors, including the recommendation to facilitate more effective teamwork among healthcare professionals. This recommendation has driven patient safety advocates to explore collective intelligence (CI) tools to address diagnostic error and delay. Collective intelligence is a shared intelligence that emerges from a group of individuals acting independently or collectively on the same task. It leverages the fact that a group is likely to outperform an individual in cognitive tasks across a variety of fields. 14 , 15 Within medicine, CI has shown promise in areas of visual diagnosis (radiology, dermatology), 16 , 17 where CI has reliably outperformed individual physicians in detecting malignancies. Few studies have explored the benefit of CI for general medical diagnosis, and prior studies in general diagnosis have focused on simulated cases. 18 A study of medical students’ abilities to accurately diagnose simulated emergency medicine cases found that the CI outperformed individual students. 19 Another recent study found that the CI of multiple physicians had greater diagnostic accuracy than individual physicians in 1500+ simulated cases written for general practitioners. 20 The development of digital tools that facilitate collaboration and communication among physicians 21 – 23 provide new opportunities to investigate and leverage the potential of CI, particularly for physicians who practice in isolated settings. A CI platform open to all healthcare practitioners is the Human Diagnosis Project (Human Dx), a multinational effort in which physicians and medical students both submit and solve clinical cases, and was the platform used in a prior CI study. 20 Although CI has shown promise to improve diagnostic accuracy, there is limited literature assessing whether the CI of multiple physicians has utility in the diagnostic process, prior to reaching a definitive final diagnosis. Most literature has focused on diagnostic accuracy and diagnosis as if clinicians reach a diagnosis in a one-step process. However, as noted by the National Academy of Medicine, 1 the diagnostic process is complex and iterative. It involves a repeating cycle of information gathering, information integration and interpretation, and a working diagnosis until the diagnosis is communicated to the patient. Reducing diagnostic errors will require interventions at each stage of the diagnostic process. In recognition of the multi-step, iterative nature of the diagnostic process in real clinical care, we sought to determine if CI would provide value during earlier stages of the process. Specifically, when a clinician refers a patient to a specialist, the referring clinician is often seeking guidance on next steps for evaluation and care (rather than an immediate diagnosis) – a stage when feedback can most impact diagnostic accuracy. 24 Ideally, all general practitioners would have adequate access to specialty expertise when making decisions outside their clinical expertise, but specialty access is limited in many settings due to time, cost, or availability. Accordingly, our study aimed to: (1) assess the appropriateness of diagnostic steps advised by the CI of multiple physicians versus individual physicians collected on a digital CI platform at an earlier stage of the diagnostic process when cases were referred to a specialist, and (2) describe the potential harm of inaccurate recommendations from the CI output and individual physicians. We hypothesized that the CI output from multiple physicians would provide more appropriate diagnostic recommendations than individual physicians measured against specialist recommendations as the gold standard.