Multi-institutional Assessment of Pathologist scoring HER2 Immunohistochemistry

Abstract The HercepTest was approved 20 + years ago as the companion diagnostic test for trastuzumab in HER2 amplified/overexpressing breast cancers. Subsequent HER2 immunohistochemistry (IHC) assays followed, including the now most common Ventana 4B5 assay. While this IHC assay has become the clinical standard, its reliability, reproducibility, and accuracy have largely been approved and accepted based on concordance between small numbers of pathologists without validation in a real-world setting. In this study, we evaluate the concordance and inter-rater reliability of scoring HER2 IHC in 170 breast cancer biopsies by 18 breast cancer-specialized pathologists from 15 institutions. We used the ONEST (Observers Needed to Evaluate Subjective Tests) method to determine the plateau of concordance and the minimum number of pathologists needed to estimate inter-rater agreement values for large numbers of raters, as seen in the real-world setting. We report substantial discordance within the intermediate categories (< 1% agreement for 1 + and 3.6% agreement for 2+) in the four-category HER2 IHC scoring system. The discordance within the IHC 0 cases is also substantial with an overall percent agreement (OPA) of only 25% and poor inter-rater reliability metrics (0.49 Fleiss’ kappa, 0.55 intraclass correlation coefficient). This discordance can be partially reduced by using a three-category system (28.8% vs. 46.5% OPA for four and three-category scoring systems respectively). ONEST plots suggest that the OPA for the task of determining a HER2 IHC score 0 from not 0 plateaus statistically around 59.4% at 10 raters. Conversely, at the task of scoring HER2 IHC as 3 + or not 3 + pathologists’ concordance was much higher with an OPA that plateaus at 87.1% with 6 raters. This suggests that legacy HER2 IHC remains valuable for finding HER2 gene amplified patients, but unacceptably discordant in assigning HER2-low or negative status for emerging HER2-low therapies.