Quantitation of cancer treatment response by 2-[18F]FDG PET/CT: multi-center assessment of measurement variability using AUTO-PERCIST™

Quantitation of cancer treatment response by 2-[18F]FDG PET/CT: multi-center assessment of measurement variability using AUTO-PERCIST™ | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Original research Quantitation of cancer treatment response by 2-[ 18 F]FDG PET/CT: multi-center assessment of measurement variability using AUTO-PERCIST™ JooHyun O, Su Jin Lim, Hao Wang, Jeffrey P. Leal, Hui-Kuo G. Shu, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-93282/v2 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 12 Feb, 2021 Read the published version in EJNMMI Research → Version 2 posted 11 You are reading this latest preprint version Show more versions Abstract Background : The aim of this study was to assess the reader variability in quantitatively assessing pre- and post-treatment 2-deoxy-2-[ 18 F]fluoro-D-glucose positron emission tomography/computed tomography ([ 18 F]FDG PET/CT) scans in a defined set of images of cancer patients using the same semi-automated analytical software (Auto-PERCIST™), which identifies tumor peak standard uptake value corrected for lean body mass (SUL peak ) to determine [ 18 F]FDG PET quantitative parameters. Methods: Paired pre- and post-treatment [ 18 F]FDG PET/CT images from 30 oncologic patients and Auto-PERCIST™ semi-automated software were distributed to 13 readers across US and international sites. One reader was aware of the relevant medical history of the patients (read reference ), whereas the 12 other readers were blinded to history but had access to the correlative images. Auto-PERCIST™ was set up to first automatically identify the liver and compute the threshold for tumor measurability (1.5 x liver mean) + (2 x liver standard deviation [SD]), and then detect all sites with SUL peak greater than the threshold. Next, the readers selected sites they believed to represent tumor lesions. The main performance metric assessed was the percent change in the SUL peak (%ΔSUL peak ) of the hottest tumor identified on the baseline and follow up images. Results: The intra-class correlation coefficient (ICC) for the %ΔSUL peak of the hottest tumor was 0.87 (95%CI: [0.78, 0.92]) when all reads were included (n=297). Including only the measurements that selected the same target tumor as the read reference (n=224), the ICC for %ΔSUL peak was 1.00 (95%CI: [1.00, 1.00]). The Krippendorff alpha coefficient for response (complete or partial metabolic response, versus stable or progressive metabolic disease on PET Response Criteria in Solid Tumors 1.0) was 0.91 for all reads (n=380), and 1.00 including for reads with the same target tumor selection (n=270). Conclusion: Quantitative tumor [ 18 F]FDG SUL peak changes measured across multiple global sites and readers utilizing Auto-PERCIST™ show very high correlation. Harmonization of methods to single software, Auto-PERCIST™, resulted in virtually identical extraction of quantitative tumor response data from [ 18 F]FDG PET images when the readers select the same target tumor. Bioinformatics [18F]FDG PET/CT response assessment quantification Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction 2-deoxy-2-[ 18 F]fluoro-D-glucose positron emission tomography/computed tomography ([ 18 F]FDG PET/CT) is increasingly applied in monitoring treatment response in patients with cancer. While PET is intrinsically a quantitative imaging technique, many PET assessments of cancer response are qualitative, as for example in lymphoma where quantitative PET data are converted into a 5 point qualitative scale which is practical and highly useful ( 1,2 ). Quantitative PET assessments of response have been deployed in many research imaging studies, especially in examining early treatment response related changes in metabolism including breast cancer where these changes can predict much later pathological outcomes ( 3,4 ). The PET Response Criteria in Solid Tumors 1.0 (PERCIST 1.0) were proposed in 2009 as a method to standardize the assessment of tumor response on [ 18 F]FDG PET and emphasized use of the peak standard uptake value corrected for lean body mass (SUL peak ) in contrast to the maximum standardized uptake value (SUV max ) ( 5,6 ). While the SUV max is reasonably easy to determine with many forms of software though the SUL peak is more challenging to measure ( 7 ). Thus despite its attractiveness, quantitative PET utilizing PERCIST is not routinely performed for assessing response to therapy in patients with cancer in the clinic or many clinical trial settings, contrary to the routinely utilized Response Evaluation Criteria in Solid Tumors for assessment of anatomical imaging. One way to expand the use of quantitative [ 18 F]FDG PET/CT in clinical trials and clinical practice is to reduce reader variability of SUV measurements and make the measurements rapid and automated. In a previous multi-center, multi-reader study we conducted, multiple sites assessed the same paired pre- and post-treatment [ 18 F]FDG PET/CT images in cancer patients. The intra-class correlation coefficient (ICC) of percent change in SUV max was 0.89 (95% confidence interval (CI): [0.81, 0.94]) across multiple performance sites using a variety of analytical software tools. The ICC for the SUL peak was lower at 0.70 (95% CI: [0.54, 0.80]). SUL peak is, in principle, the more statistically sound of the PET parameters and it is the suggested metric in PERCIST ( 7 ). However, if there is considerable variability among sites in how SUL peak is generated and measured, then the PERCIST metric potentially may introduce variability into assessments of treatment response, as opposed to reducing variability ( 8 ). The aim of the present study was to determine if the utilization of Auto-PERCIST™, a semi-automated software system for the quantitative assessment [ 18 F]FDG PET images, could lower the reader variability in quantitatively assessing pre- and post-treatment [ 18 F]FDG PET/CT studies for response in a multi-center, multi-reader, multi-national study assessing identical images. Materials And Methods Pre- and post-treatment [ 18 F]FDG PET/CT images of 30 oncologic patients selected from a group of tumor types having representative patterns of FDG-avidity contained a mix of single and multiple tumors on the pretreatment scan (1 tumor, n=6; >1 but < 10 tumors, n=19; ≥ 10 tumors, n=5), and a mix of the four major response categories using PERCIST (complete metabolic response, n=6; partial metabolic response, n=11; stable metabolic disease, n=4; and progressive metabolic disease, n=9). Sites both with National Cancer Institute Quantitative Imaging Network affiliation and without which did not participate in the previous study with the same data set were recruited by email and conference calls. The dataset was the based on a previous study of reader variability ( 9 ). Thirty anonymized cases of pre- and post-treatment [ 18 F]FDG PET/CT studies (total 60 studies) were distributed along with directions for installing and utilizing the Auto-PERCIST™ software. Approval from the Johns Hopkins Institutional Review Board was obtained, and the need for patient informed consent was waived for this study of anonymized image data. Measurement Individual measurements from coupled pre- and post-treatment [ 18 F]FDG PET/CT images from one patient were counted as a read . The coupled pre- and post-treatment measurements for all 30 cases from a single reader were counted as a set of reads . One reader from the central site (reader 1) had full knowledge of the primary tumors, treatment histories and subsequent follow-up results, but all other readers had no knowledge of the patients’ medical histories as the reader is often intentionally blinded in the setting of multicenter trials. For statistical purpose, the measurements by reader 1 were considered as the reference standard for comparison (read reference ). Each reader determined which tumor to measure. The Auto-PERCIST™ loads the PET images and automatically obtains liver measurements from a 3 cm diameter sphere in the right side of the liver to compute the threshold for lesion detection. The default setting is 1.5 x liver mean + 2 standard deviations (SD) at baseline to ensure the decline in [ 18 F]FDG uptake is less likely due to chance and to minimize overestimation of response or progression. For follow up images, the default setting is lower at 1.0 x liver mean + 2SD, to allow detection of lesions with lower SULpeak. If a lesion was perceptible visually but not detected using the default threshold settings, the reader had the choice to manually lower the threshold for detection. The Auto-PERCIST™ would detect all sites with SUL peak higher than the threshold (Figure 1). It was up to the readers to determine whether the detected sites were true tumor lesions or not. The reader could also separate a detected focus of [ 18 F]FDG uptake into separate smaller lesions when needed – to exclude adjacent physiologic [ 18 F]FDG uptake or break down a large conglomeration of tumors into smaller separate lesions. The reader could also add smaller [ 18 F]FDG uptake lesions to make them a single lesion if the reader decided the separate [ 18 F]FDG uptakes were parts of a larger single lesion. The readers were instructed to select up to 5 of the hottest tumors for cases with multiple lesions. The readers could view the PET/CT images on any reading software they preferred, but the measurements came only from the Auto-PERCIST™. The measurements from Auto-PERCIST™ included SUL peak , maximum and mean SUL, number of counts, geometric mean, exposure, kurtosis, skewness, and metabolic volume. After the readers selected and quantified the lesions, the measurements were saved as text files and sent for central compilation and analysis to the Image Response Assessment Core at Johns Hopkins University. Statistical analysis The primary study metric was the percentage change in SUL peak (%ΔSUL peak ) from baseline to follow-up. Percentage change was defined as [(follow-up measurement – baseline measurement) / (baseline measurement)] x 100. For assessment of up to 5 lesions, the percentage change was computed from the sum of the lesions. Treating both case and site as random-effects, a linear random-effects model was fit via the restricted maximum likelihood estimation method, which estimated variance components of the random-effects in the model. As a measure of inter-rater agreement, the intra-class correlation coefficient (ICC) was computed using the variance components of the random-effects. The ICC was computed as [inter-subject variance / (inter-subject variance + intra-subject variance + residual variance)]. The bias-corrected and accelerated bootstrap method was implemented with 1,000 bootstrap replicates to construct the 95% confidence interval of the computed ICC. The sampling unit was a read . To assess agreement between the reference reader (read reference ) and another reader, the ICC was computed for each pair of the reference reader and 12 other readers. The mean of these ICCs and its range (minimum, maximum) were reported. Krippendorff alpha reliability coefficient was computed as a measure of agreement between multiple readers for response outcome, which was classified into four ordered major response categories using PERCIST 1.0 as: complete metabolic response (CMR), partial metabolic response (PMR), stable metabolic disease (SMD), and progressive metabolic disease (PMD). The measurements were classified: PMD for SUL peak increase ³ 30% (and 0.8 units) or new lesions; SMD for SUL peak increase or decrease < 30% (or 0.8 units); PMR for SUL peak decrease ³ 30% (and 0.8 units); and CMR for no perceptible tumor lesion. Additionally, Krippendorff coefficient was computed with the response categories being dichotomized into two levels: clinical benefit (CMR/PMR/SMD) and no benefit (PMD); or response (CMR/PMR) and no-response (SMD/PMD). Krippendorff suggests 0.8 as a threshold for satisfactory reliability, but if tentative conclusions are acceptable, 0.667 is the lowest conceivable threshold ( 10 ). Results All reads R eads were received from 13 different sites from January to September of 2018. A single reader (nuclear medicine physician/radiologist/radiological scientist) at each site measured all 30 cases. Measurements were treated as missing when a reader did not submit data. Among a total of 390 possible reads by 13 readers, 347 baseline reads and 329 follow-up reads were reported, of which 297 reads were complete baseline and follow-up pairs. Such reads were used to compute the ICC with all readers and agreement with read reference for the baseline, follow-up and percentage change in SUL peak , respectively. The ICC for %ΔSUL peak was 0.87 (95% CI: [0.78, 0.92]), and agreement with read reference was 0.88 (Range: [0.61, 1.00]). The ICC and agreement with read reference of other metrics are in Table 1. The overall within-subject coefficient of variance (COV; overall SD / average of the case means) for %ΔSULpeak change was computed as 2.293. The Bland-Altman plot of the %ΔSUL peak is shown in Figure 2. TABLE 1. ICC and agreement for single or up-to-5 SUL peak selections ICC (95% CI) Agreement with read reference ICC (Min, Max) Baseline Follow-up Percentage Change Baseline Follow-up Percentage Change 1 SUL peak (All reads) 0. 90 (0. 82 , 0.9 4 ) 0. 75 (0. 64 , 0 .82 ) 0. 87 (0. 78 , 0. 92 ) 0.95 (0.72, 1.00) 0.78 (0.37, 1.00) 0.88 (0.61, 1.00) 1 SUL peak (Reads with same tumor selected) 1.00 (0.49, 1.00) 1.00 (1.00, 1.00) 1.00 (1.00, 1.00) 0.95 (0.43, 1.00) 1.00 (1.00, 1.00) 1.00 (1.00, 1.00) Sum of up to 5 SUL peak (All reads) 0.93 (0.85, 0.96) 0.85 (0.67, 0.92) 0.77 (0.61, 0.85) 0.95 (0.84, 0.99) 0.89 (0.46, 0.99) 0.80 (0.39, 0.95) Sum of up to 5 SUL peak (Reads with same tumors selected) 0.98 (0.94, 0.98) 0.98 (0.94, 0.98) 0.96 (0.92, 0.98) 0.97 (0.95, 0.99) 0.98 (0.97, 1.00) 0.93 (0.87, 1.00) ICC - Intra-class correlation coefficient CI - Confidence interval SUL peak – Peak standardized uptake value corrected for lean body mass Agreement with read reference - ICC between reference reader and each of the other 12 readers Reads with same target tumor Among the 360 possible reads from the 12 readers, the readers selected a different lesion compared to the read reference in 46 reads at baseline, 43 reads at follow-up, and 29 reads at both baseline and follow-up. The 241 reads agreeing on target selection with the read reference were used to compute the ICCs with all readers and agreement with read reference . The ICC for %ΔSUL peak among all readers was 1.00 (95% CI: [1.00, 1.00]), and agreement with read reference was 1.00 (Range: [1.00, 1.00]). The ICC and agreement with read reference of other metrics are in Table 1. The overall within-subject COV for %ΔSULpeak change was computed as 0.007. The Bland-Altman plot is shown in Figure 3. Sum of up to 5 SUL peak In addition to the SUL peak measurement of a single lesion, the sum of SUL peak measurements of up to 5 of the selected lesions was used to compute the ICC and agreement with read reference for all reads and reads with the same target lesion (Table 1). Even when the same lesions were selected, the ICCs and agreement with read reference were not a perfect 1.00 due to (a) differences in the manual thresholds used for lesion detection and (b) utilization of the ‘erosion option’ for breaking up [ 18 F]FDG uptake volumes by the individual readers. TABLE 2. Inter-rater reliability of readers on response assessment Krippendorff alpha coefficient of 13 readers on response All reads (n=380*) Reads with same target (n=270*) Response vs. no response (CMR/PMR vs. SMD/PMD) 0.91 1.00 Clinical benefit vs. no benefit (CMR/PMR/SMD vs. PMD) 0.81 1.00 Response categories (ordinal scale) CMR vs. PMR vs. SMD vs. PMD 0.86 1.00 * R eads with missing response were excluded (10 for all reads, and 1 for reads with same target) Inter-rater reliability of readers on responses Among the 390 reads for all reads, 380 reads reported response categories. Among the 271 reads agreeing on target selection with the read reference , 270 reported response categories. The Krippendorff alpha coefficient of 13 readers for binary response measure (response (CMR/PMR) versus no-response (SMD/PMD)) was 0.91 for all reads , and 1.00 for only the reads with the same target lesion selection. When assessing clinical benefit (SMD/PMR/CMR representing clinical benefit versus PMD representing no benefit), the Krippendorff alpha coefficient was 0.81 for all reads and 1.00 for only the reads with the same target selection. With the four response categories treated in an ordinal scale, the Krippendorff alpha coefficient was 0.86 for all reads and 1.00 for only the reads with the same target selection (Table 2). Discussion Variability in measurements across readers and sites is an often cited hurdle to broader utilization of quantitative [ 18 F]FDG PET/CT for response assessment of cancer treatment ( 11 ). Test-retest studies have demonstrated high repeatability of [ 18 F]FDG and other radiopharmaceutical PET parameters ( 12-15 ). The variance of SUVs could be greater in clinical practice compared to ideal study setting ( 16 ). In the clinical setting, measurement of SUV max was demonstrated to have high agreement in our previous paper, while the statistically more robust SUL peak showed suboptimal agreement ( 9 ). We wanted to know if using uniform software could eliminate the variability associated with the computation differences for SUL peak across multiple vendors and software. The localization of the liver, SUL measurements from the liver, computation of a threshold for lesion detection, and identification of candidate lesions were all performed automatically on Auto-PERCIST™. Following detection of all sites with SUL peak higher than the set threshold, various [ 18 F]FDG uptake intensity or pattern measurements and textural features for each of the detected sites were also performed automatically. When the readers chose the same single target tumor, the measurements were identical, as could be expected. For up to 5 hottest lesions measurements, the agreement was near perfect. However, agreement was not a perfect 1.00 even when the readers chose the same tumors because the readers had the option to break down a single volume of [ 18 F]FDG uptake to separate parts, or add up two or more [ 18 F]FDG uptake sites to a single volume as they determined appropriate. Some readers chose to break down a lesion detected on Auto-PERCIST™ to avoid including physiologic [ 18 F]FDG uptake, or to separate a conglomeration of multiple tumors lesions. And some readers intentionally chose a detection threshold lower than the default software setting to include lesions with relatively low [ 18 F]FDG uptake for assessment on the follow-up PET images. The agreement was lower for follow-up images for the all-reads assessment. The readers disagreed more often on what was tumor and what was physiologic or inflammatory response on the follow-up images. A previous paper that showed excellent correlation between two different vendor software for SUL peak had the tumor sites predefined by the readers to exclude interpretive error ( 13 ). Determining which [ 18 F]FDG uptake site is true tumor remained a challenge even for experienced readers. In the outlier case in Figure 2 showing an average difference greater than 100%, some readers considered an intense [ 18 F]FDG uptake in the colon on the follow-up image to be new tumor lesion, while the read reference considered it physiologic in nature. Of the 360 non-reference baseline reads (including missing measurements) in this study, only 241 reads (67%) chose the same lesion and went on to make the same measurements as the read reference at both baseline and follow-up. Among the 30 cases, the target lesion (hottest tumor) on the post-therapy scan was different from the target lesion noted on the pre-therapy scan in 11 cases. For example in one case, the target lesion was in a mediastinal node on pre-therapy scan, and then a lung lesion became the hottest tumor in the post-therapy scan. In 3 cases, nodes in different stations were the target lesions at different time points. Among patients with multiple bone or lung metastases, different lesions in the same organ could be observed becoming the target tumors at different time points. As seen in inter-observer agreement studies of [ 18 F]FDG PET/CT performed in patients with lymphoma after therapy, even experienced readers do not always agree on what is tumor [ 18 F]FDG uptake and what is physiologic [ 18 F]FDG uptake ( 17,18 ). Rather than relying solely on the reading experience of the local site, discussions and consensus meetings and better training methods are necessary to implement [ 18 F]FDG PET/CT to its full potential. It almost certainly is the case that the availability of more relevant patient history would result in better accuracy and consistency in tumor detection. While PERCIST 1.0 is quantitative, the category of CMR is dependent on the reader’s judgement, and software quantification alone could not determine the response to be CMR. There were 6 cases considered to have reached CMR by the read reference . The twelve other readers categorized the case correctly as CMR in 44 reads out of 72 (12 readers x 6 cases), PMR was designated in 21 reads , SMD in 5 reads and PMD in 1 read , with 1 missing read. Thus, in addition to selection of different target tumor from the read reference , the reader’s decision between CMR and PMR leaves room for variability in response categorization, even if quantitation produces identical results. Detailed definition or consensus on findings compatible with the CMR category, or addition of quantitative threshold to clarify the CMR category is necessary for use in trials and in the clinical setting. A lesion could be considered present and thus not CMR even with very low SUL peak for example in the lungs, or a lesion could be considered resolved and thus CMR even with relatively high SULpeak for example in tonsils. The threshold computed from liver measurements (liver SUL mean + 2SD) was viewed by the readers as too high a cutoff for CMR in this study as could be inferred by how the readers manually lowered the threshold on the follow-up images. Revealing a potential limitation in the software, and of the PERCIST criteria, there was a small tumor with clearly perceptible [ 18 F]FDG uptake visually, which was not detectable by Auto-PERCIST™ due to the volume below the PERCIST definition of SUL peak sphere of 1 cubic centimeter (Figure 4). More mundane limitation of applying PERCIST includes the need to measure the patient’s height. That many of the referring physicians and radiologist are not familiar with the SUL peak parameters is another limitation to overcome. When there are multiple lesions showing intense [ 18 F]FDG uptake, the lesion with the worst response may not be the target lesion, and PERCIST needs to specify how to address such poorly behaving lesions for categorizing the overall response. Auto-PERCIST TM has the ability to automatically detect potentially new lesions for co-registered studies based on the location of the classified lesions. Auto-PERCIST TM also computed additional PET parameters representing tumor features, such as metabolic tumor volume, geometric mean, exposure, kurtosis and skewness, which have been reported as prognostic markers and diagnostic tools ( 19-22 ). Discordance among readers was minimal for the additional PET parameters, and the cause for any variance arose when the reader manually changed the tumor boundary. Even with the addition of several PET parameters, the measurement took seconds to at the longest and a few minutes for cases with many lesions. In addition to reducing variability in measurement, the software reduced the measurement time radically. Auto-PERCIST TM may become adjunct reading software the way myocardial perfusion and metabolism studies utilize cardiac image analysis software. Auto-PERCIST is available to academic researchers who register their interest with the Johns Hopkins Technology Transfer office. Conclusion Harmonization of methods to single software Auto-PERCIST TM resulted in virtually identical extraction of quantitative data including the SUL peak when the readers selected the same target tumor, and should promote greater use of [ 18 F]FDG PET/CT for response assessment in cancer treatment. Nonetheless, the findings show caution remains in order as lesion selection still results on qualitative assessments of whether a lesion is tumor or physiological uptake. Abbreviations [ 18 F]FDG: 2-deoxy-2-[ 18 F]fluoro-D-glucose; PET/CT: positron-emission tomography/computed tomography; SULpeak : peak standard uptake value corrected for lean body mass; %ΔSULpeak: percent change in the SULpeak; PERCIST 1.0: PET Response Criteria in Solid Tumors 1.0; SUVmax: maximum standardized uptake value; CI: confidence interval; SD: standard deviation; ICC: intraclass correlation coefficient; CMR: complete metabolic response; PMR: partial metabolic response; SMD: stable metabolic disease; PMD: progressive metabolic disease; COV: coefficient of variance Declarations Ethics approval and consent to participate This retrospective study was approved by our Institutional Review Board. Informed consent was waived. Consent for publication Not applicable Availability of data and material The datasets used in this study are available from the corresponding author on reasonable request. Competing interests No relevant conflicts of interest were identified except two of the authors, JL and RW, are co-inventors on a patent underlying the Auto-PERCISTä software. Funding Supported in part by grants awarded by the Radiological Society of North America, the Quantitative Imaging Biomarkers Alliance, National Cancer Institute (5U01CA140204-04), National Institutes of Health (NCI CCSG P30CA006973 and U01CA140204), and National Research Foundation of Korea (NRF-2019R1G1A009158). Authors’ contributions JHO, SJL, HW, JPL, HGS and RLW participated in the study design, collecting data and data analysis. JHO, SJL, HW, JPL prepared the manuscript and contributed to data analysis and interpretation. HGS and RLW supervised the project and reviewed manuscript. All authors read and approved the final manuscript. Acknowledgements The QIN Readers were Seong Young Kwon from Chonnam National University; Hui-Kuo Shu from Emory University; Masatoyo Nakajo from Kagoshima University; Evelyn de Jong from Maastricht University; Tadashi Watabe from Osaka University; Jin Chul Paeng and Seo Young Kang from Seoul National University; Woo Hee Choi, Eun Ji Han and Hyelim Park from The Catholic University of Korea; Ella Jones and Youngho Seo from University of California, San Francisco; John Buatti from University of Iowa; James Mounts and Matthew Oborski from University Pittsburgh Medical Center; and Joyce Mhlanga from Washington University in St. Louis. References Barrington SF, Mikhaeel NG, Kostakoglu L, et al. Role of imaging in the staging and response assessment of lymphoma: consensus of the International Conference on Malignant Lymphomas Imaging Working Group. J Clin Oncol. 2014;32:3048-3058. Gallamini A, Barrington SF, Biggi A, et al. The predictive role of interim positron emission tomography for Hodgkin lymphoma treatment outcome is confirmed using the interpretation criteria of the Deauville five-point scale. Haematologica. 2014;99:1107-1113. Wahl RL, Zasadny K, Helvie M, Hutchins GD, Weber B, Cody R. Metabolic monitoring of breast cancer chemohormonotherapy using positron emission tomography: initial evaluation. J Clin Oncol. 1993;11:2101-2111. Connolly RM, Leal JP, Solnes L, et al. TBCRC026: Phase II Trial Correlating Standardized Uptake Value With Pathologic Complete Response to Pertuzumab and Trastuzumab in Breast Cancer. J Clin Oncol. 2019;37:714-722. Wahl RL, Jacene H, Kasamon Y, Lodge MA. From RECIST to PERCIST: Evolving Considerations for PET response criteria in solid tumors. J Nucl Med. 2009;50 Suppl 1:122S-150S. O JH, Lodge MA, Wahl RL. Practical PERCIST: A Simplified Guide to PET Response Criteria in Solid Tumors 1.0. Radiology. 2016;280:576-584. Lodge MA, Chaudhry MA, Wahl RL. Noise considerations for PET quantification using maximum and peak standardized uptake value. J Nucl Med. 2012;53:1041-1047. Quak E, Le Roux PY, Lasnon C, et al. Does PET SUV Harmonization Affect PERCIST Response Classification? J Nucl Med. 2016;57:1699-1706. O JH, Jacene H, Luber B, et al. Quantitation of Cancer Treatment Response by (18)F-FDG PET/CT: Multicenter Assessment of Measurement Variability. J Nucl Med. 2017;58:1429-1434. Krippendorff K. Measuring the reliability of qualitative text analysis data. Quality & Quantity. 2004;38:787-800. Ding Q, Cheng X, Yang L, et al. PET/CT evaluation of response to chemotherapy in non-small cell lung cancer: PET response criteria in solid tumors (PERCIST) versus response evaluation criteria in solid tumors (RECIST). J Thorac Dis. 2014;6:677-683. Lodge MA. Repeatability of SUV in Oncologic (18)F-FDG PET. J Nucl Med. 2017;58:523-532. Mhlanga JC, Chirindel A, Lodge MA, Wahl RL, Subramaniam RM. Quantitative PET/CT in clinical practice: assessing the agreement of PET tumor indices using different clinical reading platforms. Nucl Med Commun. 2018;39:154-160. Fox JJ, Autran-Blanc E, Morris MJ, et al. Practical approach for comparative analysis of multilesion molecular imaging using a semiautomated program for PET/CT. J Nucl Med. 2011;52:1727-1732. Fraum TJ, Fowler KJ, Crandall JP, et al. Measurement Repeatability of (18)F-FDG PET/CT Versus (18)F-FDG PET/MRI in Solid Tumors of the Pelvis. J Nucl Med. 2019;60:1080-1086. Kumar V, Nath K, Berman CG, et al. Variance of SUVs for FDG-PET/CT is greater in clinical practice than under ideal study settings. Clin Nucl Med. 2013;38:175-182. Burggraaff CN, Cornelisse AC, Hoekstra OS, et al. Interobserver Agreement of Interim and End-of-Treatment (18)F-FDG PET/CT in Diffuse Large B-Cell Lymphoma: Impact on Clinical Practice and Trials. J Nucl Med. 2018;59:1831-1836. Han EJ, O JH, Yoon H, et al. FDG PET/CT response in diffuse large B-cell lymphoma: Reader variability and association with clinical outcome. Medicine (Baltimore). 2016;95:e4983. Pinker K, Riedl C, Weber WA. Evaluating tumor response with FDG PET: updates on PERCIST, comparison with EORTC criteria and clues to future developments. Eur J Nucl Med Mol Imaging. 2017;44:55-66. Tixier F, Hatt M, Valla C, et al. Visual versus quantitative assessment of intratumor 18F-FDG PET uptake heterogeneity: prognostic value in non-small cell lung cancer. J Nucl Med. 2014;55:1235-1241. Huang W, Fan M, Liu B, et al. Value of metabolic tumor volume on repeated 18F-FDG PET/CT for early prediction of survival in locally advanced non-small cell lung cancer treated with concurrent chemoradiotherapy. J Nucl Med. 2014;55:1584-1590. Lee JW, Lee SM. Radiomics in Oncological PET/CT: Clinical Applications. Nuclear Medicine and Molecular Imaging. 2018;52:170-189. Cite Share Download PDF Status: Published Journal Publication published 12 Feb, 2021 Read the published version in EJNMMI Research → Version 2 posted Review # 3 received at journal 26 Jan, 2021 Editorial decision: Accept 26 Jan, 2021 Review # 1 received at journal 20 Jan, 2021 Reviewer # 3 agreed at journal 19 Jan, 2021 Review # 2 received at journal 18 Jan, 2021 Reviewer # 2 agreed at journal 17 Jan, 2021 Reviewer # 1 agreed at journal 16 Jan, 2021 Editor assigned by journal 14 Jan, 2021 Reviewers invited by journal 14 Jan, 2021 Submission checks completed at journal 14 Jan, 2021 Editor invited by journal 14 Jan, 2021 You are reading this latest preprint version Show more versions Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-93282","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Original research","associatedPublications":[],"authors":[{"id":8545835,"identity":"22def93e-a855-4dde-8038-9207a062cad5","order_by":0,"name":"JooHyun O","email":"","orcid":"","institution":"Catholic University of Korea","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"JooHyun","middleName":"","lastName":"O","suffix":""},{"id":8545836,"identity":"4048ca9e-c76e-4b7f-b785-a0f37d5c972a","order_by":1,"name":"Su Jin Lim","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Su","middleName":"Jin","lastName":"Lim","suffix":""},{"id":8545837,"identity":"6575c61f-c2aa-4f0e-a026-dbaf6b7d6195","order_by":2,"name":"Hao Wang","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Wang","suffix":""},{"id":8545838,"identity":"364a94af-319d-4241-a4ca-618e09f78050","order_by":3,"name":"Jeffrey P. Leal","email":"","orcid":"","institution":"Johns Hopkins University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jeffrey","middleName":"P.","lastName":"Leal","suffix":""},{"id":8545839,"identity":"7e43b1fb-575a-4feb-a09a-205592b3d73a","order_by":4,"name":"Hui-Kuo G. Shu","email":"","orcid":"","institution":"Emory University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Hui-Kuo","middleName":"G.","lastName":"Shu","suffix":""},{"id":8545840,"identity":"fb9e1ee7-14bb-4946-aae4-cee25f58902b","order_by":5,"name":"Richard L. Wahl","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAs0lEQVRIiWNgGAWjYHACNoYHFSCauQHEJlJLwhkQxUiKlsQ2UrSYz0h+9iBx3uHE+fMbGxg+lB0mrEXmRpq5QeK2w4kbjjE2MM44R4QWCekEM4nEbbcTNwAdxszbRpSW9G8SiXNuJ85vA2r5S5yWHKAtDbcTG4AOY2YkSov8mzKJhGP/jTccS2w42HMunQgtPMe3SXyoSZOd33z44IMfZdaEtaCAAySqHwWjYBSMglGACwAAa1087BAyaM4AAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-7306-2590","institution":"Washington University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Richard","middleName":"L.","lastName":"Wahl","suffix":""}],"badges":[],"createdAt":"2020-10-15 16:20:34","currentVersionCode":2,"declarations":"","doi":"10.21203/rs.3.rs-93282/v2","doiUrl":"https://doi.org/10.21203/rs.3.rs-93282/v2","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s13550-021-00754-1","type":"published","date":"2021-02-12T15:00:32+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":5276635,"identity":"fea46639-e428-4fa0-a819-19fb5b8bad16","added_by":"auto","created_at":"2021-01-26 16:43:53","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":119250,"visible":true,"origin":"","legend":"Screen captures from Auto-PERCISTTM. (A) The software detected all sites with SULpeak higher than the computed or manually set threshold. (B) The reader than selected the true tumor lesions (shaded in green), excluding physiologic [18F]FDG activity. ","description":"","filename":"Fig1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-93282/v2/107018de580a41a4e762c489.jpg"},{"id":5276543,"identity":"491f0391-11c4-4d0d-86ee-9f9ebd86cf3b","added_by":"auto","created_at":"2021-01-26 16:37:53","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":36864,"visible":true,"origin":"","legend":"Bland-Altman plot of the percentage change of tumor [18F]FDG uptake from baseline to follow-up. The plot is for the percentage changes of SULpeak for all reads. Each dot represents a case (30 cases in total). The x-axis represents the average mean percentage change measurement by all readers. The y-axis represents the average difference between the 12 readers and the reference reader (readreference). The solid line represents the average bias, and the dashed lines represent the corresponding bias +/- 2 standard deviations (SD). ","description":"","filename":"Fig2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-93282/v2/503ac17ad21c6980e5bb6cbb.jpg"},{"id":5276544,"identity":"ecb5ac3c-c5ab-4d6e-af6a-1e01c6f90c3b","added_by":"auto","created_at":"2021-01-26 16:37:53","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":34478,"visible":true,"origin":"","legend":"Bland-Altman plot of the percentage change of tumor [18F]FDG uptake from baseline to follow-up. The plot is for the percentage changes of SULpeak (%ΔSULpeak) for only the reads with same lesion selected as the readreference. Each dot represents a case (30 cases in total). The x-axis represents the average mean %ΔSULpeak measurement by all readers. The y-axis represents the average difference between the 12 readers and the reference reader (readreference) and the y-axis unit is one tenth of one percent. The solid line represents the average bias, and the dashed lines represent the corresponding bias +/- 2 standard deviations (SD). ","description":"","filename":"Fig3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-93282/v2/1d218953537382308eef5ed2.jpg"},{"id":5276540,"identity":"b909c980-c8e3-4b2f-9b9f-4deb0aead8fa","added_by":"auto","created_at":"2021-01-26 16:37:53","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":30321,"visible":true,"origin":"","legend":"(A) PET maximum intensity projection (MIP) image of patient with right axillary node metastasis at baseline with SULpeak of 1.62 and tumor volume of less than 1.00 cc. Though visually perceptible, Auto-PERCISTTM failed to detect the lesion due to small size. (B) On the follow-up MIP image, the number of metastatic nodes and the [18F]FDG uptake intensity are increased to SULpeak of 2.84, allowing detection by Auto-PERCISTTM.","description":"","filename":"Fig4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-93282/v2/49abc818c2f2168cf4fa221c.jpg"},{"id":15670784,"identity":"02642273-72c3-48c9-b8bf-cf3a4a97367b","added_by":"auto","created_at":"2021-11-18 14:01:51","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":494814,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-93282/v2/ff27b25a-6dda-4cc3-8434-bbf3ef4b4295.pdf"}],"financialInterests":"","formattedTitle":"\u003cp\u003eQuantitation of cancer treatment response by 2-[\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT: multi-center assessment of measurement variability using AUTO-PERCIST™\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003e2-deoxy-2-[\u003csup\u003e18\u003c/sup\u003eF]fluoro-D-glucose positron emission tomography/computed tomography ([\u003csup\u003e18\u003c/sup\u003eF]FDG\u0026nbsp;PET/CT)\u0026nbsp;is increasingly applied in monitoring treatment response in patients with cancer. While PET is intrinsically a quantitative imaging technique, many PET assessments of cancer response are qualitative, as for example in lymphoma where quantitative PET data are converted into a 5 point qualitative scale which is practical and highly useful (\u003cem\u003e1,2\u003c/em\u003e). Quantitative PET assessments of response have been deployed in many research imaging studies, especially in examining early treatment response related changes in metabolism including breast cancer where these changes can predict much later pathological outcomes (\u003cem\u003e3,4\u003c/em\u003e). The PET Response Criteria in Solid Tumors 1.0 (PERCIST 1.0) were proposed in 2009 as a method to standardize the assessment of tumor response on [\u003csup\u003e18\u003c/sup\u003eF]FDG PET and emphasized use of the peak standard uptake value corrected for lean body mass (SUL\u003csub\u003epeak\u003c/sub\u003e) in contrast to the maximum standardized uptake value (SUV\u003csub\u003emax\u003c/sub\u003e) (\u003cem\u003e5,6\u003c/em\u003e). While the SUV\u003csub\u003emax\u0026nbsp;\u003c/sub\u003eis reasonably easy to determine with many forms of software though the SUL\u003csub\u003epeak\u003c/sub\u003e is more challenging to measure (\u003cem\u003e7\u003c/em\u003e).\u003c/p\u003e\n\u003cp\u003eThus despite its attractiveness, quantitative PET utilizing PERCIST is not routinely performed for\u0026nbsp;assessing response to therapy in\u0026nbsp;patients with cancer\u0026nbsp;in the clinic or many clinical trial settings, contrary to the routinely utilized Response Evaluation Criteria in Solid Tumors for assessment of anatomical imaging.\u0026nbsp;One way to expand the use of quantitative [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT in clinical trials and clinical practice is to reduce reader variability of SUV measurements and make the measurements rapid and automated. In a previous multi-center, multi-reader study we conducted, multiple sites assessed the same paired pre- and post-treatment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT images in cancer patients. The intra-class correlation coefficient (ICC) of percent change in SUV\u003csub\u003emax\u003c/sub\u003e was 0.89 (95% confidence interval (CI): [0.81, 0.94]) across multiple performance sites using a variety of analytical software tools. The ICC for the SUL\u003csub\u003epeak\u003c/sub\u003e was lower at 0.70 (95% CI: [0.54, 0.80]). SUL\u003csub\u003epeak\u003c/sub\u003e is, in principle, the more statistically sound of the PET parameters and it is the suggested metric in PERCIST (\u003cem\u003e7\u003c/em\u003e). However, if there is considerable variability among sites in how SUL\u003csub\u003epeak\u003c/sub\u003e is generated and measured, then the PERCIST metric potentially may introduce variability into assessments of treatment response, as opposed to reducing variability (\u003cem\u003e8\u003c/em\u003e).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe aim of the present study was to determine\u0026nbsp;if the utilization of Auto-PERCIST\u0026trade;, a semi-automated software system for the quantitative assessment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET images, could lower the reader variability in quantitatively assessing pre- and post-treatment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT studies for response in a multi-center, multi-reader, multi-national study assessing identical images. \u0026nbsp;\u003c/p\u003e"},{"header":"Materials And Methods","content":"\u003cp\u003ePre- and post-treatment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT images of 30 oncologic patients\u0026nbsp;selected from a group of tumor types having\u0026nbsp;representative patterns of FDG-avidity contained a mix of single and multiple tumors on the pretreatment scan (1 tumor, n=6; \u0026gt;1 but \u0026lt; 10 tumors, n=19; \u0026ge; 10 tumors, n=5), and a mix of the four major response categories using PERCIST (complete metabolic response, n=6; partial metabolic response, n=11; stable metabolic disease, n=4; and progressive metabolic disease, n=9).\u003c/p\u003e\n\u003cp\u003eSites both with\u0026nbsp;National Cancer Institute\u0026nbsp;Quantitative Imaging Network\u0026nbsp;affiliation and without which did not participate in the previous study with the same data set were recruited by email and conference calls. The dataset was the based on a previous study of reader variability (\u003cem\u003e9\u003c/em\u003e).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThirty\u0026nbsp;anonymized\u0026nbsp;cases of\u0026nbsp;pre- and post-treatment\u0026nbsp;[\u003csup\u003e18\u003c/sup\u003eF]FDG\u0026nbsp;PET/CT studies (total 60 studies) were distributed along\u0026nbsp;with directions for installing and utilizing the Auto-PERCIST\u0026trade; software.\u0026nbsp;Approval from the Johns Hopkins Institutional Review Board\u0026nbsp;was obtained, and the need for patient informed consent was waived for this study of anonymized\u0026nbsp;image data.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026nbsp;Measurement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIndividual measurements from coupled pre- and post-treatment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT images from one patient were counted as a \u003cem\u003eread\u003c/em\u003e. The coupled pre- and post-treatment measurements for all 30 cases from a single reader were counted as a \u003cem\u003eset of reads\u003c/em\u003e. One reader from the central site (reader 1) had full knowledge of the primary tumors, treatment histories and subsequent follow-up results, but all other readers had no knowledge of the patients\u0026rsquo; medical histories as the reader is often intentionally blinded in the setting of multicenter trials. For statistical purpose, the measurements by reader 1 were considered as the reference standard for comparison (read\u003csub\u003ereference\u003c/sub\u003e).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEach reader determined which tumor to measure. The Auto-PERCIST\u0026trade; loads the PET images and automatically obtains liver measurements from a 3 cm diameter sphere in the right side of the liver to compute the threshold for lesion detection. The default setting is 1.5 x liver mean + 2 standard deviations (SD) at baseline to ensure the decline in [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake is less likely due to chance and to minimize overestimation of response or progression. For follow up images, the default setting is lower at 1.0 x liver mean + 2SD, to allow detection of lesions with lower SULpeak. If a lesion was perceptible visually but not detected using the default threshold settings, the reader had the choice to manually lower the threshold for detection. The Auto-PERCIST\u0026trade; would detect all sites with SUL\u003csub\u003epeak\u003c/sub\u003e higher than the threshold (Figure 1). It was up to the readers to determine whether the detected sites were true tumor lesions or not. The reader could also separate a detected focus of [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake into separate smaller lesions when needed \u0026ndash; to exclude adjacent physiologic [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake or break down a large conglomeration of tumors into smaller separate lesions. The reader could also add smaller [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake lesions to make them a single lesion if the reader decided the separate [\u003csup\u003e18\u003c/sup\u003eF]FDG uptakes were parts of a larger single lesion. The readers were instructed to select up to 5 of the hottest tumors for cases with multiple lesions. The readers could view the PET/CT images on any reading software they preferred, but the measurements came only from the Auto-PERCIST\u0026trade;. The measurements from Auto-PERCIST\u0026trade; included SUL\u003csub\u003epeak\u003c/sub\u003e, maximum and mean SUL, number of counts, geometric mean, exposure, kurtosis, skewness, and metabolic volume. After the readers selected and quantified the lesions, the measurements were saved as text files and sent for central compilation and analysis to the Image Response Assessment Core at Johns Hopkins University.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStatistical analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe primary study metric was the percentage change in SUL\u003csub\u003epeak\u003c/sub\u003e (%\u0026Delta;SUL\u003csub\u003epeak\u003c/sub\u003e) from baseline to follow-up. Percentage change was defined as [(follow-up measurement \u0026ndash; baseline measurement) / (baseline measurement)] x 100. For assessment of up to 5 lesions, the percentage change was computed from the sum of the lesions. Treating both case and site as random-effects, a linear random-effects model was fit via the restricted maximum likelihood estimation method, which estimated variance components of the random-effects in the model. As a measure of inter-rater agreement, the intra-class correlation coefficient (ICC) was computed using the variance components of the random-effects. The ICC was computed as [inter-subject variance / (inter-subject variance + intra-subject variance + residual variance)]. The bias-corrected and accelerated bootstrap method was implemented with 1,000 bootstrap replicates to construct the 95% confidence interval of the computed ICC. The sampling unit was a \u003cem\u003eread\u003c/em\u003e.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo assess agreement between the reference reader (read\u003csub\u003ereference\u003c/sub\u003e) and another reader, the ICC was computed for each pair of the reference reader and 12 other readers. The mean of these ICCs and its range (minimum, maximum) were reported.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eKrippendorff alpha reliability coefficient was computed as a measure of agreement between multiple readers for response outcome, which was classified into four ordered major response categories using PERCIST 1.0 as: complete metabolic response (CMR), partial metabolic response (PMR), stable metabolic disease (SMD), and progressive metabolic disease (PMD). The measurements were classified: PMD for SUL\u003csub\u003epeak\u003c/sub\u003e increase\u0026nbsp;\u0026sup3;\u0026nbsp;30% (and 0.8 units) or new lesions; SMD for SUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003eincrease or decrease \u0026lt; 30% (or 0.8 units); PMR for SUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003edecrease\u0026nbsp;\u0026sup3;\u0026nbsp;30% (and 0.8 units); and CMR for no perceptible tumor lesion. Additionally, Krippendorff coefficient was computed with the response categories being dichotomized into two levels: clinical benefit (CMR/PMR/SMD) and no benefit (PMD); or response (CMR/PMR) and no-response (SMD/PMD). Krippendorff suggests 0.8 as a threshold for satisfactory reliability, but if tentative conclusions are acceptable, 0.667 is the lowest conceivable threshold (\u003cem\u003e10\u003c/em\u003e).\u0026nbsp;\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003e\u003cstrong\u003eAll \u003cem\u003ereads\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eR\u003c/em\u003e\u003cem\u003eeads\u003c/em\u003e were received from 13 different\u0026nbsp;sites\u0026nbsp;from January to September of 2018.\u0026nbsp;A single\u0026nbsp;reader (nuclear medicine physician/radiologist/radiological scientist) at each site\u0026nbsp;measured all\u0026nbsp;30\u0026nbsp;cases. Measurements were treated as missing when a reader did not submit data.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAmong a total of 390 possible \u003cem\u003ereads\u003c/em\u003e by 13 readers, 347 baseline \u003cem\u003ereads\u0026nbsp;\u003c/em\u003eand 329 follow-up \u003cem\u003ereads\u003c/em\u003e were reported, of which 297 \u003cem\u003ereads\u003c/em\u003e were complete baseline and follow-up pairs. Such \u003cem\u003ereads\u0026nbsp;\u003c/em\u003ewere used to compute the ICC with all readers and agreement with read\u003csub\u003ereference\u003c/sub\u003e for the baseline, follow-up and percentage change in SUL\u003csub\u003epeak\u003c/sub\u003e, respectively. The ICC for %\u0026Delta;SUL\u003csub\u003epeak\u003c/sub\u003e was 0.87 (95% CI: [0.78, 0.92]), and agreement with read\u003csub\u003ereference\u003c/sub\u003e was 0.88 (Range: [0.61, 1.00]). The ICC and agreement with read\u003csub\u003ereference\u003c/sub\u003e of other metrics are in Table 1. The overall within-subject coefficient of variance (COV; overall SD / average of the case means) for %\u0026Delta;SULpeak change was computed as 2.293. The Bland-Altman plot of the %\u0026Delta;SUL\u003csub\u003epeak\u003c/sub\u003e is shown in Figure 2.\u0026nbsp;\u003c/p\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:200%;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:16px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003eTABLE 1. ICC and agreement for single or up-to-5 SUL\u003csub\u003epeak\u003c/sub\u003e selections\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"width: 4.7e+2pt;border-collapse:collapse;border:none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"2\" style=\"width: 69.2pt;border: 1pt solid windowtext;padding: 0in 5.4pt;height: 34.2pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" style=\"width:191.35pt;border:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;height:34.2pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eICC\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(95% CI)\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"3\" style=\"width:212.65pt;border:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;height:34.2pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eAgreement with\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family: \"Times New Roman\",serif;'\u003eread\u003csub\u003ereference\u003c/sub\u003e\u003c/span\u003e\u003c/strong\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cstrong\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;ICC (Min, Max)\u003c/span\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eBaseline\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eFollow-up\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003ePercentage Change\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eBaseline\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eFollow-up\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:28.9pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003ePercentage Change\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 69.2pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 37.35pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(All reads)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e90\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e82\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e, 0.9\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e4\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e75\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e64\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e, 0\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e.82\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e87\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e78\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e, 0.\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e92\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.95\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.72, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.78\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.37, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:37.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.88\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.61, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 69.2pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 56.35pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSUL\u003csub\u003epeak\u003c/sub\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(Reads with same tumor selected)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.49, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(1.00, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(1.00, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.95\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.43, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(1.00, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:56.35pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(1.00, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 69.2pt;border-top: none;border-left: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-right: none;padding: 0in 5.4pt;height: 10.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding: 0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 69.2pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 10.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSum of up to 5\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(All reads)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.93\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.85, 0.96)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.85\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.67, 0.92)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.77\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.61, 0.85)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.95\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.84, 0.99)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.89\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.46, 0.99)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.80\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.39, 0.95)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 69.2pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 10.6pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSum of up to 5\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003eSUL\u003csub\u003epeak\u003c/sub\u003e\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(Reads with same tumors selected)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.98\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.94, 0.98)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.98\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.94, 0.98)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.75pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.96\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.92, 0.98)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:85.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.97\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.95, 0.99)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.98\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.97, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:63.8pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:10.6pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e0.93\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:12px;font-family:\"Times New Roman\",serif;'\u003e(0.87, 1.00)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eICC -\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eIntra-class correlation coefficient\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eCI - Confidence interval\u003c/span\u003e\u003c/p\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eSUL\u003csub\u003epeak\u003c/sub\u003e \u0026ndash; Peak standardized uptake value corrected for lean body mass\u003c/span\u003e\u003c/p\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eAgreement with read\u003csub\u003ereference\u003c/sub\u003e - ICC between reference reader and each of the other 12 readers\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eReads\u0026nbsp;\u003c/em\u003e\u003c/strong\u003e\u003cstrong\u003ewith same target tumor\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAmong the 360 possible \u003cem\u003ereads\u0026nbsp;\u003c/em\u003efrom the 12 readers,\u0026nbsp;the readers selected a different lesion compared to the read\u003csub\u003ereference\u003c/sub\u003e in 46 \u003cem\u003ereads\u003c/em\u003e at baseline, 43 \u003cem\u003ereads\u003c/em\u003e at follow-up, and 29 \u003cem\u003ereads\u0026nbsp;\u003c/em\u003eat both baseline and follow-up. The 241 \u003cem\u003ereads\u003c/em\u003e agreeing on target selection with the read\u003csub\u003ereference\u003c/sub\u003e were used to compute the ICCs with all readers and agreement with read\u003csub\u003ereference\u003c/sub\u003e. The ICC for %\u0026Delta;SUL\u003csub\u003epeak\u003c/sub\u003e among all readers was 1.00 (95% CI: [1.00, 1.00]), and agreement with read\u003csub\u003ereference\u003c/sub\u003e was 1.00 (Range: [1.00, 1.00]). The ICC and agreement with read\u003csub\u003ereference\u003c/sub\u003e of other metrics are in Table 1. The overall within-subject COV for %\u0026Delta;SULpeak change was computed as 0.007. The Bland-Altman plot is shown in Figure 3.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSum of up to 5 SUL\u003csub\u003epeak\u003c/sub\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn addition to the SUL\u003csub\u003epeak\u003c/sub\u003e measurement of a single lesion, the sum of SUL\u003csub\u003epeak\u003c/sub\u003e measurements of up to 5 of the selected lesions was used to compute the ICC and agreement with read\u003csub\u003ereference\u003c/sub\u003e for all reads and reads with the same target lesion (Table 1). Even when the same lesions were selected, the ICCs and agreement with read\u003csub\u003ereference\u0026nbsp;\u003c/sub\u003ewere not a perfect 1.00 due to (a) differences in the manual thresholds used for lesion detection and (b) utilization of the \u0026lsquo;erosion option\u0026rsquo; for breaking up [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake volumes by the individual readers.\u0026nbsp;\u003c/p\u003e\n\u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:200%;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:16px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003eTABLE 2. Inter-rater reliability of readers on response assessment\u003c/span\u003e\u003c/p\u003e\n\u003ctable style=\"border-collapse:collapse;border:none;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 182.6pt;border-top: 1pt solid windowtext;border-right: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-bottom: none;padding: 0in 5.4pt;height: 27.1pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width:248.05pt;border:solid windowtext 1.0pt;border-left:none;padding:0in 5.4pt 0in 5.4pt;height:27.1pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eKrippendorff alpha coefficient\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eof 13 readers on response\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 182.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;height: 50.7pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:113.4pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:50.7pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eAll reads\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e(n=380*)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:134.65pt;border:solid windowtext 1.0pt;border-left: none;padding:0in 5.4pt 0in 5.4pt;height:50.7pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eReads with same target (n=270*)\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 182.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eResponse vs. no response\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e(CMR/PMR vs. SMD/PMD)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:113.4pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e0.91\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:134.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 182.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eClinical benefit vs. no benefit\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e(CMR/PMR/SMD vs. PMD)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:113.4pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e0.81\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:134.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 182.6pt;border-right: 1pt solid windowtext;border-bottom: 1pt solid windowtext;border-left: 1pt solid windowtext;border-image: initial;border-top: none;padding: 0in 5.4pt;vertical-align: top;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eResponse categories (ordinal scale)\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003eCMR vs. PMR vs. SMD vs. PMD\u003c/span\u003e\u003c/p\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:113.4pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e0.86\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:134.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;line-height:normal;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;text-align:center;'\u003e\u003cspan style='font-size:13px;font-family:\"Times New Roman\",serif;'\u003e1.00\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp style='margin-top:12.0pt;margin-right:0in;margin-bottom:10.0pt;margin-left:0in;line-height:200%;font-size:15px;font-family:\"Malgun Gothic\",sans-serif;'\u003e\u003cspan style='font-family:\"Times New Roman\",serif;'\u003e*\u003c/span\u003e\u003cem\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003eR\u003c/span\u003e\u003c/em\u003e\u003cem\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003eeads\u003c/span\u003e\u003c/em\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;with missing response were excluded\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003e(10\u0026nbsp;\u003c/span\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003efor all reads,\u003c/span\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;and\u003c/span\u003e\u003cspan style='font-size:13px;line-height:200%;font-family:\"Times New Roman\",serif;'\u003e\u0026nbsp;1 for reads with same target)\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInter-rater reliability of readers on responses\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAmong the 390 \u003cem\u003ereads\u0026nbsp;\u003c/em\u003efor all reads, 380 \u003cem\u003ereads\u003c/em\u003e reported response categories. Among the 271 \u003cem\u003ereads\u003c/em\u003e agreeing on target selection with the read\u003csub\u003ereference\u003c/sub\u003e, 270 reported response categories. The Krippendorff alpha coefficient of 13 readers for binary response measure (response (CMR/PMR) versus no-response (SMD/PMD)) was 0.91 for all \u003cem\u003ereads\u003c/em\u003e, and 1.00 for only the \u003cem\u003ereads\u003c/em\u003e with the same target lesion selection. When assessing clinical benefit (SMD/PMR/CMR representing clinical benefit versus PMD representing no benefit), the Krippendorff alpha coefficient was 0.81 for all \u003cem\u003ereads\u003c/em\u003e and 1.00 for only the \u003cem\u003ereads\u003c/em\u003e with the same target selection. With the four response categories treated in an ordinal scale, the Krippendorff alpha coefficient was 0.86 for all \u003cem\u003ereads\u003c/em\u003e and 1.00 for only the \u003cem\u003ereads\u003c/em\u003e with the same target selection (Table 2).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eVariability in measurements across readers and sites is an often cited hurdle to broader utilization of quantitative [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT for response assessment of cancer treatment (\u003cem\u003e11\u003c/em\u003e). Test-retest studies have demonstrated high repeatability of [\u003csup\u003e18\u003c/sup\u003eF]FDG and other radiopharmaceutical PET parameters (\u003cem\u003e12-15\u003c/em\u003e). The variance of SUVs could be greater in clinical practice compared to ideal study setting (\u003cem\u003e16\u003c/em\u003e). In the clinical setting, measurement of SUV\u003csub\u003emax\u003c/sub\u003e was demonstrated to have high agreement in our previous paper, while the statistically more robust SUL\u003csub\u003epeak\u003c/sub\u003e showed suboptimal agreement (\u003cem\u003e9\u003c/em\u003e). We wanted to know if using uniform software could eliminate the variability associated with the computation differences for SUL\u003csub\u003epeak\u003c/sub\u003e across multiple vendors and software.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe localization of the liver, SUL measurements from the liver, computation of a threshold for lesion detection, and identification of candidate lesions were all performed automatically on Auto-PERCIST\u0026trade;. Following detection of all sites with SUL\u003csub\u003epeak\u003c/sub\u003e higher than the set threshold, various [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake intensity or pattern measurements and textural features for each of the detected sites were also performed automatically. When the readers chose the same single target tumor, the measurements were identical, as could be expected. For up to 5 hottest lesions measurements, the agreement was near perfect. However, agreement was not a perfect 1.00 even when the readers chose the same tumors because the readers had the option to break down a single volume of [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake to separate parts, or add up two or more [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake sites to a single volume as they determined appropriate. Some readers chose to break down a lesion detected on Auto-PERCIST\u0026trade; to avoid including physiologic [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake, or to separate a conglomeration of multiple tumors lesions. And some readers intentionally chose a detection threshold lower than the default software setting to include lesions with relatively low [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake for assessment on the follow-up PET images. The agreement was lower for follow-up images for the all-reads assessment. The readers disagreed more often on what was tumor and what was physiologic or inflammatory response on the follow-up images.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eA previous paper that showed excellent correlation between two different vendor software for SUL\u003csub\u003epeak\u003c/sub\u003e had the tumor sites predefined by the readers to exclude interpretive error (\u003cem\u003e13\u003c/em\u003e). Determining which [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake site is true tumor remained a challenge even for experienced readers. In the outlier case in Figure 2 showing an average difference greater than 100%, some readers considered an intense [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake in the colon on the follow-up image to be new tumor lesion, while the read\u003csub\u003ereference\u0026nbsp;\u003c/sub\u003econsidered it physiologic in nature. Of the 360 non-reference baseline reads (including missing measurements) in this study, only 241 reads (67%) chose the same lesion and went on to make the same measurements as the read\u003csub\u003ereference\u003c/sub\u003e at both baseline and follow-up. Among the 30 cases, the target lesion (hottest tumor) on the post-therapy scan was different from the target lesion noted on the pre-therapy scan in 11 cases. For example in one case, the target lesion was in a mediastinal node on pre-therapy scan, and then a lung lesion became the hottest tumor in the post-therapy scan. In 3 cases, nodes in different stations were the target lesions at different time points. Among patients with multiple bone or lung metastases, different lesions in the same organ could be observed becoming the target tumors at different time points. As seen in inter-observer agreement studies of [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT performed in patients with lymphoma after therapy, even experienced readers do not always agree on what is tumor [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake and what is physiologic [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake (\u003cem\u003e17,18\u003c/em\u003e). Rather than relying solely on the reading experience of the local site, discussions and consensus meetings and better training methods are necessary to implement [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT to its full potential. It almost certainly is the case that the availability of more relevant patient history would result in better accuracy and consistency in tumor detection.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWhile PERCIST 1.0 is quantitative, the category of CMR is dependent on the reader\u0026rsquo;s judgement, and software quantification alone could not determine the response to be CMR. There were 6 cases considered to have reached CMR by the read\u003csub\u003ereference\u003c/sub\u003e. The twelve other readers categorized the case correctly as CMR in 44 \u003cem\u003ereads\u003c/em\u003e out of 72 (12 readers x 6 cases), PMR was designated in 21 \u003cem\u003ereads\u003c/em\u003e, SMD in 5 \u003cem\u003ereads\u003c/em\u003e and PMD in 1 \u003cem\u003eread\u003c/em\u003e, with 1 missing \u003cem\u003eread.\u003c/em\u003e Thus, in addition to selection of different target tumor from the read\u003csub\u003ereference\u003c/sub\u003e, the reader\u0026rsquo;s decision between CMR and PMR leaves room for variability in response categorization, even if quantitation produces identical results. Detailed definition or consensus on findings compatible with the CMR category, or addition of quantitative threshold to clarify the CMR category is necessary for use in trials and in the clinical setting. A lesion could be considered present and thus not CMR even with very low SUL\u003csub\u003epeak\u003c/sub\u003e for example in the lungs, or a lesion could be considered resolved and thus CMR even with relatively high SULpeak for example in tonsils. The threshold computed from liver measurements (liver SUL\u003csub\u003emean\u003c/sub\u003e + 2SD) was viewed by the readers as too high a cutoff for CMR in this study as could be inferred by how the readers manually lowered the threshold on the follow-up images.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eRevealing a potential limitation in the software, and of the PERCIST criteria, there was a small tumor with clearly perceptible [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake visually, which was not detectable by Auto-PERCIST\u0026trade; due to the volume below the PERCIST definition of SUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003esphere of 1 cubic centimeter (Figure 4). More mundane limitation of applying PERCIST includes the need to measure the patient\u0026rsquo;s height. That many of the referring physicians and radiologist are not familiar with the SUL\u003csub\u003epeak\u0026nbsp;\u003c/sub\u003eparameters is another limitation to overcome. When there are multiple lesions showing intense [\u003csup\u003e18\u003c/sup\u003eF]FDG uptake, the lesion with the worst response may not be the target lesion, and PERCIST needs to specify how to address such poorly behaving lesions for categorizing the overall response.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAuto-PERCIST\u003csup\u003eTM\u003c/sup\u003e has the ability to automatically detect potentially new lesions for co-registered studies based on the location of the classified lesions. Auto-PERCIST\u003csup\u003eTM\u003c/sup\u003e also computed additional PET parameters representing tumor features, such as metabolic tumor volume, geometric mean, exposure, kurtosis and skewness, which have been reported as prognostic markers and diagnostic tools (\u003cem\u003e19-22\u003c/em\u003e). Discordance among readers was minimal for the additional PET parameters, and the cause for any variance arose when the reader manually changed the tumor boundary. Even with the addition of several PET parameters, the measurement took seconds to at the longest and a few minutes for cases with many lesions. In addition to reducing variability in measurement, the software reduced the measurement time radically. Auto-PERCIST\u003csup\u003eTM\u0026nbsp;\u003c/sup\u003emay become adjunct reading software the way myocardial perfusion and metabolism studies utilize cardiac image analysis software. Auto-PERCIST is available to academic researchers who register their interest with the Johns Hopkins Technology Transfer office.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eHarmonization of methods to single software Auto-PERCIST\u003csup\u003eTM\u003c/sup\u003e resulted in virtually identical extraction of quantitative data including the SUL\u003csub\u003epeak\u003c/sub\u003e when the readers selected the same target tumor, and should promote greater use of [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT for response assessment in cancer treatment. Nonetheless, the findings show caution remains in order as lesion selection still results on qualitative assessments of whether a lesion is tumor or physiological uptake.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e[\u003csup\u003e18\u003c/sup\u003eF]FDG: 2-deoxy-2-[\u003csup\u003e18\u003c/sup\u003eF]fluoro-D-glucose; PET/CT: positron-emission tomography/computed tomography; SULpeak : peak standard uptake value corrected for lean body mass; %\u0026Delta;SULpeak: percent change in the SULpeak; PERCIST 1.0: PET Response Criteria in Solid Tumors 1.0; SUVmax: maximum standardized uptake value; CI: confidence interval; SD: standard deviation; ICC: intraclass correlation coefficient; CMR: complete metabolic response; PMR: partial metabolic response; SMD: stable metabolic disease; PMD: progressive metabolic disease; COV: coefficient of variance\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis retrospective study was approved by our Institutional Review Board. Informed consent was waived.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and material\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets used in this study are available from the corresponding author on reasonable request.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo relevant conflicts of interest were identified except two of the authors, JL and RW, are co-inventors on a patent underlying the Auto-PERCIST\u0026auml; software.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSupported in part by grants awarded by the Radiological Society of North America, the Quantitative Imaging Biomarkers Alliance, National Cancer Institute (5U01CA140204-04), National Institutes of Health (NCI CCSG P30CA006973 and U01CA140204), and National Research Foundation of Korea (NRF-2019R1G1A009158).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026rsquo; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eJHO, SJL, HW, JPL, HGS and RLW participated in the study design, collecting data and data analysis. JHO, SJL, HW, JPL prepared the manuscript and contributed to data analysis and interpretation. HGS and RLW supervised the project and reviewed manuscript. All authors read and approved the final manuscript.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe QIN Readers were Seong Young Kwon from Chonnam National University; Hui-Kuo Shu from Emory University; Masatoyo Nakajo from Kagoshima University; Evelyn de Jong from Maastricht University; Tadashi Watabe from Osaka University; Jin Chul Paeng and Seo Young Kang from Seoul National University; Woo Hee Choi, Eun Ji Han and Hyelim Park from The Catholic University of Korea; Ella Jones and Youngho Seo from University of California, San Francisco; John Buatti from University of Iowa; James Mounts and Matthew Oborski from University Pittsburgh Medical Center; and Joyce Mhlanga from Washington University in St. Louis.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBarrington SF, Mikhaeel NG, Kostakoglu L, et al. Role of imaging in the staging and response assessment of lymphoma: consensus of the International Conference on Malignant Lymphomas Imaging Working Group. \u003cem\u003eJ Clin Oncol. \u003c/em\u003e2014;32:3048-3058.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"2\"\u003e\n\u003cli\u003eGallamini A, Barrington SF, Biggi A, et al. The predictive role of interim positron emission tomography for Hodgkin lymphoma treatment outcome is confirmed using the interpretation criteria of the Deauville five-point scale. \u003cem\u003eHaematologica. \u003c/em\u003e2014;99:1107-1113.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"3\"\u003e\n\u003cli\u003eWahl RL, Zasadny K, Helvie M, Hutchins GD, Weber B, Cody R. Metabolic monitoring of breast cancer chemohormonotherapy using positron emission tomography: initial evaluation. \u003cem\u003eJ Clin Oncol. \u003c/em\u003e1993;11:2101-2111.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"4\"\u003e\n\u003cli\u003eConnolly RM, Leal JP, Solnes L, et al. TBCRC026: Phase II Trial Correlating Standardized Uptake Value With Pathologic Complete Response to Pertuzumab and Trastuzumab in Breast Cancer. \u003cem\u003eJ Clin Oncol. \u003c/em\u003e2019;37:714-722.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"5\"\u003e\n\u003cli\u003eWahl RL, Jacene H, Kasamon Y, Lodge MA. From RECIST to PERCIST: Evolving Considerations for PET response criteria in solid tumors. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2009;50 Suppl 1:122S-150S.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"6\"\u003e\n\u003cli\u003eO JH, Lodge MA, Wahl RL. Practical PERCIST: A Simplified Guide to PET Response Criteria in Solid Tumors 1.0. \u003cem\u003eRadiology. \u003c/em\u003e2016;280:576-584.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"7\"\u003e\n\u003cli\u003eLodge MA, Chaudhry MA, Wahl RL. Noise considerations for PET quantification using maximum and peak standardized uptake value. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2012;53:1041-1047.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"8\"\u003e\n\u003cli\u003eQuak E, Le Roux PY, Lasnon C, et al. Does PET SUV Harmonization Affect PERCIST Response Classification? \u003cem\u003eJ Nucl Med. \u003c/em\u003e2016;57:1699-1706.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"9\"\u003e\n\u003cli\u003eO JH, Jacene H, Luber B, et al. Quantitation of Cancer Treatment Response by (18)F-FDG PET/CT: Multicenter Assessment of Measurement Variability. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2017;58:1429-1434.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"10\"\u003e\n\u003cli\u003eKrippendorff K. Measuring the reliability of qualitative text analysis data. \u003cem\u003eQuality \u0026amp; Quantity. \u003c/em\u003e2004;38:787-800.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"11\"\u003e\n\u003cli\u003eDing Q, Cheng X, Yang L, et al. PET/CT evaluation of response to chemotherapy in non-small cell lung cancer: PET response criteria in solid tumors (PERCIST) versus response evaluation criteria in solid tumors (RECIST). \u003cem\u003eJ Thorac Dis. \u003c/em\u003e2014;6:677-683.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"12\"\u003e\n\u003cli\u003eLodge MA. Repeatability of SUV in Oncologic (18)F-FDG PET. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2017;58:523-532.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"13\"\u003e\n\u003cli\u003eMhlanga JC, Chirindel A, Lodge MA, Wahl RL, Subramaniam RM. Quantitative PET/CT in clinical practice: assessing the agreement of PET tumor indices using different clinical reading platforms. \u003cem\u003eNucl Med Commun. \u003c/em\u003e2018;39:154-160.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"14\"\u003e\n\u003cli\u003eFox JJ, Autran-Blanc E, Morris MJ, et al. Practical approach for comparative analysis of multilesion molecular imaging using a semiautomated program for PET/CT. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2011;52:1727-1732.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"15\"\u003e\n\u003cli\u003eFraum TJ, Fowler KJ, Crandall JP, et al. Measurement Repeatability of (18)F-FDG PET/CT Versus (18)F-FDG PET/MRI in Solid Tumors of the Pelvis. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2019;60:1080-1086.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"16\"\u003e\n\u003cli\u003eKumar V, Nath K, Berman CG, et al. Variance of SUVs for FDG-PET/CT is greater in clinical practice than under ideal study settings. \u003cem\u003eClin Nucl Med. \u003c/em\u003e2013;38:175-182.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"17\"\u003e\n\u003cli\u003eBurggraaff CN, Cornelisse AC, Hoekstra OS, et al. Interobserver Agreement of Interim and End-of-Treatment (18)F-FDG PET/CT in Diffuse Large B-Cell Lymphoma: Impact on Clinical Practice and Trials. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2018;59:1831-1836.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"18\"\u003e\n\u003cli\u003eHan EJ, O JH, Yoon H, et al. FDG PET/CT response in diffuse large B-cell lymphoma: Reader variability and association with clinical outcome. \u003cem\u003eMedicine (Baltimore). \u003c/em\u003e2016;95:e4983.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"19\"\u003e\n\u003cli\u003ePinker K, Riedl C, Weber WA. Evaluating tumor response with FDG PET: updates on PERCIST, comparison with EORTC criteria and clues to future developments. \u003cem\u003eEur J Nucl Med Mol Imaging. \u003c/em\u003e2017;44:55-66.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"20\"\u003e\n\u003cli\u003eTixier F, Hatt M, Valla C, et al. Visual versus quantitative assessment of intratumor 18F-FDG PET uptake heterogeneity: prognostic value in non-small cell lung cancer. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2014;55:1235-1241.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"21\"\u003e\n\u003cli\u003eHuang W, Fan M, Liu B, et al. Value of metabolic tumor volume on repeated 18F-FDG PET/CT for early prediction of survival in locally advanced non-small cell lung cancer treated with concurrent chemoradiotherapy. \u003cem\u003eJ Nucl Med. \u003c/em\u003e2014;55:1584-1590.\u003c/li\u003e\n\u003c/ol\u003e\n\u003col start=\"22\"\u003e\n\u003cli\u003eLee JW, Lee SM. Radiomics in Oncological PET/CT: Clinical Applications. \u003cem\u003eNuclear Medicine and Molecular Imaging. \u003c/em\u003e2018;52:170-189.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"ejnmmi-research","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ejre","sideBox":"Learn more about [EJNMMI Research](http://ejnmmires.springeropen.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ejre/default.aspx","title":"EJNMMI Research","twitterHandle":"@officialEANM","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"[18F]FDG PET/CT, response assessment, quantification","lastPublishedDoi":"10.21203/rs.3.rs-93282/v2","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-93282/v2","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: The aim of this study was to assess the reader variability in quantitatively assessing pre- and post-treatment 2-deoxy-2-[\u003csup\u003e18\u003c/sup\u003eF]fluoro-D-glucose positron emission tomography/computed tomography ([\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT) scans in a defined set of images of cancer patients using the same semi-automated analytical software (Auto-PERCIST™), which identifies tumor peak standard uptake value corrected for lean body mass (SUL\u003csub\u003epeak\u003c/sub\u003e) to determine [\u003csup\u003e18\u003c/sup\u003eF]FDG PET quantitative parameters.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eMethods:\u003c/strong\u003e Paired pre- and post-treatment [\u003csup\u003e18\u003c/sup\u003eF]FDG PET/CT images from 30 oncologic patients and Auto-PERCIST™ semi-automated software were distributed to 13 readers across US and international sites. One reader was aware of the relevant medical history of the patients (read\u003csub\u003ereference\u003c/sub\u003e), whereas the 12 other readers were blinded to history but had access to the correlative images. Auto-PERCIST™ was set up to first automatically identify the liver and compute the threshold for tumor measurability (1.5 x liver mean) + (2 x liver standard deviation [SD]), and then detect all sites with SUL\u003csub\u003epeak\u003c/sub\u003e greater than the threshold. Next, the readers selected sites they believed to represent tumor lesions. The main performance metric assessed was the percent change in the SUL\u003csub\u003epeak\u003c/sub\u003e (%ΔSUL\u003csub\u003epeak\u003c/sub\u003e) of the hottest tumor identified on the baseline and follow up images. \u003c/p\u003e\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eThe intra-class correlation coefficient (ICC) for the %ΔSUL\u003csub\u003epeak\u003c/sub\u003e of the hottest tumor was 0.87 (95%CI: [0.78, 0.92]) when all reads were included (n=297). Including only the measurements that selected the same target tumor as the read\u003csub\u003ereference\u003c/sub\u003e (n=224), the ICC for %ΔSUL\u003csub\u003epeak\u003c/sub\u003e was 1.00 (95%CI: [1.00, 1.00]). The Krippendorff alpha coefficient for response (complete or partial metabolic response, versus stable or progressive metabolic disease on PET Response Criteria in Solid Tumors 1.0) was 0.91 for all reads (n=380), and 1.00 including for reads with the same target tumor selection (n=270).\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eConclusion: \u003c/strong\u003eQuantitative tumor [\u003csup\u003e18\u003c/sup\u003eF]FDG SUL\u003csub\u003epeak \u003c/sub\u003echanges measured across multiple global sites and readers utilizing Auto-PERCIST™ show very high correlation. Harmonization of methods to single software, Auto-PERCIST™, resulted in virtually identical extraction of quantitative tumor response data from [\u003csup\u003e18\u003c/sup\u003eF]FDG PET images when the readers select the same target tumor.\u0026nbsp;\u0026nbsp;\u003c/p\u003e","manuscriptTitle":"Quantitation of cancer treatment response by 2-[18F]FDG PET/CT: multi-center assessment of measurement variability using AUTO-PERCIST™","msid":"","msnumber":"","nonDraftVersions":[{"code":2,"date":"2021-01-26 16:37:51","doi":"10.21203/rs.3.rs-93282/v2","editorialEvents":[{"type":"communityComments","content":0},{"type":"editorInvitedReview","content":"","date":"2021-01-27T00:00:00+00:00","index":3,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"decision","content":"Accept","date":"2021-01-27T00:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-01-21T00:00:00+00:00","index":1,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"reviewerAgreed","content":"","date":"2021-01-20T00:00:00+00:00","index":3,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2021-01-19T00:00:00+00:00","index":2,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"reviewerAgreed","content":"","date":"2021-01-18T00:00:00+00:00","index":2,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2021-01-17T00:00:00+00:00","index":1,"fulltext":""},{"type":"editorAssigned","content":"","date":"2021-01-15T00:00:00+00:00","index":"","fulltext":""},{"type":"reviewersInvited","content":"","date":"2021-01-15T00:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2021-01-14T23:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2021-01-14T23:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"ejnmmi-research","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ejre","sideBox":"Learn more about [EJNMMI Research](http://ejnmmires.springeropen.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ejre/default.aspx","title":"EJNMMI Research","twitterHandle":"@officialEANM","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}},{"code":1,"date":"2020-10-21 21:32:16","doi":"10.21203/rs.3.rs-93282/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Major Revision","date":"2020-12-16T00:00:00+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-12-09T00:00:00+00:00","index":3,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"editorInvitedReview","content":"","date":"2020-12-09T00:00:00+00:00","index":2,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"reviewerAgreed","content":"","date":"2020-12-04T01:00:00+00:00","index":4,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-12-04T00:00:00+00:00","index":3,"fulltext":""},{"type":"editorInvitedReview","content":"","date":"2020-11-02T00:00:00+00:00","index":1,"fulltext":"Recommendation: Reviewer's comments unavailable due to the journal's policy.\n"},{"type":"reviewerAgreed","content":"","date":"2020-10-23T12:00:00+00:00","index":2,"fulltext":""},{"type":"reviewerAgreed","content":"","date":"2020-10-15T12:00:00+00:00","index":1,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2020-10-14T12:00:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2020-10-13T12:00:00+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2020-10-12T12:00:00+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2020-10-12T12:00:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"","date":"2020-10-08T12:00:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"ejnmmi-research","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ejre","sideBox":"Learn more about [EJNMMI Research](http://ejnmmires.springeropen.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ejre/default.aspx","title":"EJNMMI Research","twitterHandle":"@officialEANM","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0a378b01-bc76-4d24-b618-d4bd9e415a76","owner":[],"postedDate":"January 26th, 2021","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":2087123,"name":"Bioinformatics"}],"tags":[],"updatedAt":"2021-02-14T15:00:58+00:00","versionOfRecord":{"articleIdentity":"rs-93282","link":"https://doi.org/10.1186/s13550-021-00754-1","journal":{"identity":"ejnmmi-research","isVorOnly":false,"title":"EJNMMI Research"},"publishedOn":"2021-02-12 15:00:32","publishedOnDateReadable":"February 12th, 2021"},"versionCreatedAt":"2021-01-26 16:37:51","video":"","vorDoi":"10.1186/s13550-021-00754-1","vorDoiUrl":"https://doi.org/10.1186/s13550-021-00754-1","workflowStages":[]},"version":"v2","identity":"rs-93282","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-93282","identity":"rs-93282","version":["v2"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}