Validation of new, circulating biomarkers for glioma

Validation of new, circulating biomarkers for glioma | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Validation of new, circulating biomarkers for glioma Miyo K. Chatanaka, Lisa M. Avery, Eleftherios P. Diamandis This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4397157/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Biomarkers are useful clinical tools but only a handful of them are used routinely for patient care. Despite intense efforts to discover new, clinically useful biomarkers, very few new circulating biomarkers were implemented in clinical practice in the last 40 years. This is mainly due to poor clinical performance. Here, our goal was to validate a group of newly discovered circulating biomarkers for glioma by comparing our data with data from a paper recently published in Science Advances. We analyzed our own sets of clinical samples and a different analytical assay to compare the results of Shen and colleagues. Despite the sophistication of the utilized discovery method, we found that the newly proposed biomarkers for glioma (such as SERPINA6) did not perform as expected. Scientific irreproducibility has been extensively discussed in the literature. A large proportion of newly discovered candidate biomarkers likely represent “false discovery” and contribute to irreproducible results. The best way to assess the value of any new biomarker is by independent and extensive validation. Based on our previous classification, we believe that this work represents another example of a false discovery. Biological sciences/Cancer Health sciences/Biomarkers Health sciences/Medical research biomarkers glioma false discovery biomarker failures SERPINA6 biomarker validation Figures Figure 1 Figure 2 Introduction Recently, Shen et al. completed a large study aiming to reveal mechanistic insights into glioma formation and identify circulating (plasma-based) biomarkers for disease diagnosis [ 1 ]. They performed multidimensional proteomics of many tissues and fluids, with some collected at the vicinity of the tumor, and compared within and between patients to spot mechanistically and diagnostically interesting molecules for gliomas. As expected, they generated long lists of differentially expressed candidate proteins. A number of these proteins were variably present in the peripheral blood of glioma and non-glioma patients, presumably opening the window for biomarker-based clinical applications, such as diagnostics. The author’s confidence about this possibility is summarized verbatim from their paper: ‘ The results further demonstrate the promise of SERPINA6 as a blood biomarker for rapid initial screening for gliomas.’ Among the plethora of candidate biomarkers originating from this study were proteins associated with ROS metabolic processes, mismatch repair-related, platelet-related, and glycolysis-related proteins, tumor suppressors, immunoglobulins, and protease inhibitors. Among all candidates, members of the SERPIN family of serine protease inhibitors, namely SERPINA2, 4, 6 and 7, were given special attention, with SERPINA6 appearing to be lower in the blood of glioma vs non-glioma patients (some data are shown in their Fig. 6, where the sensitivity and specificity of SERPINA6 to detect glioma vs facial paralysis was approximately 88% [ 1 ]). Despite the claim, such sensitivity/specificity values are not suited for “rapid initial screening for gliomas” since, given the low prevalence of gliomas in the general population (about 2–5 cases per 100,000), the expected positive predictive value (PPV) of this test would be < 2% [ 2 ]. The PPV represents the chance of somebody having the disease if the test is positive and is the ratio of the true positives over all positive results (for more details and additional explanations please see our previous contributions [ 2 – 4 ]. Furthermore, the overlap of SERPINA6 plasma concentration between glioma and non-glioma patients was substantial. This is a significant weakness of any diagnostic biomarker (see their Figs. 5 and 6 [ 1 ]). Here, we validate some diagnostic findings of the Shen et.al. paper by using independent patient groups from gliomas and meningiomas and a different (orthogonal) assay (PEA) to analyze the plasma samples. Materials and Methods We analyzed a cohort of the following plasma samples, provided by the Northwestern University Brain Tumor Biobank that were collected at diagnosis but before therapy, from patients with gliomas (N = 30), and meningiomas (as benign controls) (N = 20). Analysis of these samples was performed at OLINK Proteomics facilities using the PEA technology, as described in detail elsewhere [ 5 – 7 ]. The list of the 3,000 proteins that are included in the OLINK panel can be found on the OLINK Website ( [email protected] ). As per our previous validation [ 7 ], PEA is reproducible, with coefficients of variation (CVs) of 99% of the proteins. Our own efforts to discover novel glioma biomarkers have been published elsewhere [ 8 ]. To successfully compare our results with those of Shen et al., an initial analysis of our data for the 15 proteins that were also discovered by Shen et al. was performed. SERPINA6 was selected for an in-depth analysis, due to the proposed diagnostic strength of this marker by the group [ 1 ]. In particular, re-analysis of the SERPINA6 data from Shen et al. was conducted, followed by student-t tests and Wilcoxon signed-rank tests for both our analysis and the re-analysis. Results Shen et al. used 4 cohorts of patients for their discovery and validation experiments [ 1 ]. These cohorts included serum and tissues and can be seen in Fig. 1 of their paper. Below, we will show the re-analyzed Shen et al. data from all four cohorts, and our data, for several biomarker candidates. Figure 1 illustrates the differences in relative protein intensity between patients with glioma vs meningioma (our patient cohorts) for all the proteins included in the OLINK panel and that were promising biomarkers in the Shen et.al paper. As mentioned above, we compared the protein values through t-tests. Our obtained data (Fig. 1 ) do not confirm the Shen et al findings. None of the examined proteins, including the most promising candidate biomarker, SERPINA6, was different between gliomas and meningiomas. SERPINA6 was slightly lower in the glioma group, but the difference was not statistically significant (p = 0.47). Re-analysis of the data for SERPINA6 We separately re-analyzed the data for SERPINA6, which was the most promising glioma biomarker identified in the study of Shen et al. (raw data were available by the authors through their publication: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10871530/bin/sciadv.adk1721_data_s1_to_s4.zip ). Blood samples from patients with glioma were collected by the authors at three different locations (Cohort 1). Although it is stated in the paper that SERPINA6 is decreased in glioma-derived peripheral serum (pV), in comparison to glioma artery (gA) and glioma vein (gV) serum, the plotted concentrations of SERPINA6 are slightly increased in the peripheral serum (pV) of some patients but the differences were not statistically significant (paired t-tests) (Fig. 2 A). For the second cohort, tissue samples were collected from normal brain, perineural and glioma patients. There were no measured SERPINA6 values from the normal tissues. A Wilcoxon test looking for differences in medians between the groups was not significant. The distribution of SERPINA6 in peritumoral tissue and glioma tissue is shown in Fig. 2 B (Cohort 2). We then compared the concentration of SERPINA6 in peripheral blood of controls (facial paralysis) (red dots) vs glioma patients (blue dots) (Cohort 3 data). The glioma group had lower values than the controls (Wilcoxon p < 0.001) (Fig. 2 C). This observation agrees with the reported data by Shen et al. [ 1 ]. We finally compared SERPINA6 concentration in peripheral blood of controls (facial paralysis) vs glioma patients, from cohort 4 data (Fig. 2 D). Indeed, the glioma patients had slightly lower values, in agreement with the authors’ claim, but the differences were barely significant by Wilcoxon test (p = 0.046). In ROC analysis for predicting glioma from controls by using serum SERPINA6, we found, in agreement with the authors, some discriminatory potential, which, however, was not sufficient for reliably diagnosing glioma with their blood test. ROC curves to predict glioma vs non-glioma with SERPINA6 as biomarker (cohort 4) were constructed. The data were similar to the author’s reported results for cohort 4. Discussion There is a considerable interest in developing disease-related biomarkers and applying them to improve patient care. Despite major investments in this area by granting agencies and commercial organizations, the yield has been disappointingly poor. The handful of cancer biomarkers that are used in the clinic today were discovered more than 40 years ago [ 9 ]. Previously, we and others commented on several reasons that contribute to newly discovered biomarker failures and identified pre-analytical, analytical, and post-analytical shortcomings which may affect a biomarker’s performance [ 10 – 17 ]. In short, most of the failures are due to unrecognized biases/differences between the diseased and controlled clinical samples, the groups and numbers of patients and their clinical information, the analytical method used, and more recently, the way the data are interpreted (black box approach) [ 17 , 18 ]. Recently, considerable efforts were made, to find ways to better reproduce published and seemingly promising biomarkers. It has been realized that a large number of manuscripts, published in even top-rated journals, describe false discovery [ 14 ]. The number of retractions of manuscripts published in highly reputable journals is at the all-time high [ 19 , 20 ]. One of the ways to decrease false discovery is to reproduce the findings, preferably independently, from the original investigators. This is not an easy task, since specific reagents and techniques may not be available to the validators. Also, this process is time-consuming and expensive. We proposed a simpler way to tackle the irreproducibility problem that we coined “the 5-year reflection” [ 21 ]. Reproducing at least a fraction of high impact studies may reveal weaknesses which can lead to improved outcomes in the future. In the paper under discussion, Shen et al. tried to identify new biomarkers for glioma by collecting blood samples from the peripheral circulation and from glioma arteries and veins in the vicinity of the tumor. Tissue samples were also collected. To validate their discovery data, they collected samples that were not used in the discovery phase. In our previous work, we described the identification of glioma biomarkers by a different proteomic technique, the Proximity Expression Assay (PEA) [ 6 , 8 ]. Taking these results, we then examined which of the proteins identified by Shen et al. and showed biomarker promise, were in the PEA panel. Surprisingly, we found no overlap between the candidates in the two studies. Even the best putative biomarker selected by Shen et al. (SERPINA6) showed no differences between gliomas and meningiomas in our patients. Additional analyses have shown that SERPINA6 concentration was no different between the tumoral and peritumoral tissue. To explain some of these discrepancies, we suggest that stricter quantitative technologies are likely needed to identify differences between glioma and non-glioma patients in the discovery phase, with emphasis to be given to clinical utility. Regarding the diagnostic power of SERPINA6, the sensitivity and specificity of the test was about 88%. This led the authors to conclude that it may be suitable for clinical use. However, even these seemingly high sensitivities and specificities are likely not enough for clinical use, as we exemplified elsewhere [ 2 – 4 ]. Moreover, for a biomarker to be promising it needs to have clinical utility that complements statistical significance. A clinically useful biomarker needs to have either very high specificity + good sensitivity or very high sensitivity + good specificity. Simply put, having a significant difference in median/mean value between groups, does not confer clinical utility. We conclude that the biomarkers identified by Shen et al. [ 1 ] were not successfully validated with our own sets of data from glioma and meningioma patients and the characteristics of the test, as published, is not sufficient to warrant any clinical applications at present. Declarations Author contributions statement MKC drafted and edited the manuscript, and prepared figure 2. LMA performed the statistical analysis, prepared figures 1 and 2 and edited the manuscript. EPD conceptualized, drafted and edited the manuscript. Data availability statement The excel file with the concentrations of the biomarkers and their concentrations are available by request from the corresponding author. Competing Interests Statement The authors report no conflict of interests. References Shen, L. et al. Mechanistic insight into glioma through spatially multidimensional proteomics. Sci. Adv. 10, eadk1721 (2024). Fiala, C. & Diamandis, E. P. A multi-cancer detection test: focus on the positive predictive value. Ann. Oncol. 31, 1267–1268 (2020). Diamandis, E. P. & Li, M. The side effects of translational omics: overtesting, overdiagnosis, overtreatment. Clin. Chem. Lab. Med. 54, 389–396 (2016). Fiala, C., Taher, J. & Diamandis, E. P. P4 Medicine or O4 Medicine? Hippocrates Provides the Answer. J. Appl. Lab. Med. 4, 108–119 (2019). Chen, M. et al. Plasma Protein Profiling by Proximity Extension Assay Technology Reveals Novel Biomarkers of Traumatic Brain Injury-A Pilot Study. J. Appl. Lab. Med. 6, 1165–1178 (2021). Ren, A. H., Diamandis, E. P. & Kulasingam, V. Uncovering the Depths of the Human Proteome: Antibody-based Technologies for Ultrasensitive Multiplexed Protein Detection and Quantification. Mol. Cell. Proteomics MCP 20, 100155 (2021). Ren, A. et al. Comparison of two multiplexed technologies for profiling > 1,000 serum proteins that may associate with tumor burden. F1000Research 10, 509 (2021). Ghorbani, A. et al. Discovery of novel glioma serum biomarkers by proximity extension assay. Clin. Proteomics 20, 12 (2023). Sturgeon, C. M. et al. National Academy of Clinical Biochemistry laboratory medicine practice guidelines for use of tumor markers in testicular, prostate, colorectal, breast, and ovarian cancers. Clin. Chem. 54, e11-79 (2008). Diamandis, E. P. The failure of protein cancer biomarkers to reach the clinic: why, and what can be done to address the problem? BMC Med. 10, 87 (2012). Diamandis, E. P. Cancer Biomarkers: Can We Turn Recent Failures into Success? JNCI J. Natl. Cancer Inst. 102, 1462–1467 (2010). Diamandis, E. P. Analysis of serum proteomic patterns for early cancer diagnosis: drawing attention to potential problems. J. Natl. Cancer Inst. 96, 353–356 (2004). Ransohoff, D. F. Bias as a threat to the validity of cancer molecular-marker research. Nat. Rev. Cancer 5, 142–149 (2005). Ioannidis, J. P. A. Discussion: Why ‘An estimate of the science-wise false discovery rate and application to the top medical literature’ is false. Biostat. Oxf. Engl. 15, 28–36; discussion 39–45 (2014). Ioannidis, J. P. A. Molecular bias. Eur. J. Epidemiol. 20, 739–745 (2005). Prassas, I. et al. False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. Clin. Chem. 60, 381–388 (2014). Petricoin, E. F. et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet Lond. Engl. 359, 572–577 (2002). Schully, S. D. et al. Leveraging biospecimen resources for discovery or validation of markers for early cancer detection. J. Natl. Cancer Inst. 107, djv012 (2015). Levett, J. J. et al. Publication retraction in spine surgery: a systematic review. Eur. Spine J. Off. Publ. Eur. Spine Soc. Eur. Spinal Deform. Soc. Eur. Sect. Cerv. Spine Res. Soc. 32, 3704–3712 (2023). Ball, P. Is AI leading to a reproducibility crisis in science? Nature 624, 22–25 (2023). Fiala, C. & Diamandis, E. P. How to reduce scientific irreproducibility: the 5-year reflection. Clin. Chem. Lab. Med. 55, 1845–1848 (2017). Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4397157","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":306730623,"identity":"b4f194ca-2f7a-47da-976a-6d2fe20f0b7b","order_by":0,"name":"Miyo K. Chatanaka","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Miyo","middleName":"K.","lastName":"Chatanaka","suffix":""},{"id":306730624,"identity":"a241ca59-1283-4f21-98a1-3a6e6f54e13c","order_by":1,"name":"Lisa M. Avery","email":"","orcid":"","institution":"University of Toronto","correspondingAuthor":false,"prefix":"","firstName":"Lisa","middleName":"M.","lastName":"Avery","suffix":""},{"id":306730625,"identity":"c96ffa00-efa2-4e4d-a2ba-247e468316f4","order_by":2,"name":"Eleftherios P. Diamandis","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9ElEQVRIie3PMWuDQBjG8VcOdLnG9cQWv8JNLg75LE5dnUIHKYJwWWrm5FuYJWSMvJAsQlfFDsnQbl2DHRJ6MVMgGrNluP9wPMuP4wVQqR4xbF5CwYij0zLBAN6T0KwhVkRukdWZADC/JxlsyHf6t/SendlO2MHbFzMJSaEO24mFulsm+SvllS/saf7DrFgPtI91O+EIbvEkkHJbEjneudwE9C5i7MuDJM4sk+SIbNiQYxehbnX6BQpNkggZJ5JoousWOqpehLwl92OPrpEx1IMsmbSTwedmUf4Kb+iMMatoiMwcx/NtvW8n11vdC1QqlUp12T/JiVBy/BvKwQAAAABJRU5ErkJggg==","orcid":"","institution":"Lunenfeld-Tanenbaum Research Institute, Mount Sinai Hospital","correspondingAuthor":true,"prefix":"","firstName":"Eleftherios","middleName":"P.","lastName":"Diamandis","suffix":""}],"badges":[],"createdAt":"2024-05-09 21:23:57","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4397157/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4397157/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":57790218,"identity":"d29f01bb-a9f7-4eee-934a-9f6b79f0f340","added_by":"auto","created_at":"2024-06-05 17:22:15","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":672931,"visible":true,"origin":"","legend":"\u003cp\u003eBoxplots of log2 intensity for glioma vs meningioma patients (our cohorts) across 15 shown proteins from our OLINK data.\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4397157/v1/6236eec9546daafe83c08068.png"},{"id":57790219,"identity":"50a50eb4-1399-4f57-9a38-6370dda810e1","added_by":"auto","created_at":"2024-06-05 17:22:15","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":72644,"visible":true,"origin":"","legend":"\u003cp\u003eConcentration of SERPINA6 in glioma patient blood serum, collected at three different locations (cohort 1). Abbreviations: glioma-arterial (gA), glioma-venous (gV), and peripheral circulation (pV).\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-4397157/v1/c074ceb14291e678054f9926.png"},{"id":67188434,"identity":"19193acf-b6b8-4b0b-a579-a9a7612fc512","added_by":"auto","created_at":"2024-10-22 07:47:26","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":887041,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4397157/v1/46a0816e-37af-4102-9038-759346144c10.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Validation of new, circulating biomarkers for glioma","fulltext":[{"header":"Introduction","content":"\u003cp\u003eRecently, Shen et al. completed a large study aiming to reveal mechanistic insights into glioma formation and identify circulating (plasma-based) biomarkers for disease diagnosis [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. They performed multidimensional proteomics of many tissues and fluids, with some collected at the vicinity of the tumor, and compared within and between patients to spot mechanistically and diagnostically interesting molecules for gliomas. As expected, they generated long lists of differentially expressed candidate proteins. A number of these proteins were variably present in the peripheral blood of glioma and non-glioma patients, presumably opening the window for biomarker-based clinical applications, such as diagnostics. The author\u0026rsquo;s confidence about this possibility is summarized verbatim from their paper: \u0026lsquo;\u003cem\u003eThe results further demonstrate the promise of SERPINA6 as a blood biomarker for rapid initial screening for gliomas.\u0026rsquo;\u003c/em\u003e\u003c/p\u003e \u003cp\u003eAmong the plethora of candidate biomarkers originating from this study were proteins associated with ROS metabolic processes, mismatch repair-related, platelet-related, and glycolysis-related proteins, tumor suppressors, immunoglobulins, and protease inhibitors. Among all candidates, members of the SERPIN family of serine protease inhibitors, namely SERPINA2, 4, 6 and 7, were given special attention, with SERPINA6 appearing to be lower in the blood of glioma vs non-glioma patients (some data are shown in their Fig.\u0026nbsp;6, where the sensitivity and specificity of SERPINA6 to detect glioma vs facial paralysis was approximately 88% [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]).\u003c/p\u003e \u003cp\u003eDespite the claim, such sensitivity/specificity values are not suited for \u0026ldquo;rapid initial screening for gliomas\u0026rdquo; since, given the low prevalence of gliomas in the general population (about 2\u0026ndash;5 cases per 100,000), the expected positive predictive value (PPV) of this test would be \u0026lt;\u0026thinsp;2% [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The PPV represents the chance of somebody having the disease if the test is positive and is the ratio of the true positives over all positive results (for more details and additional explanations please see our previous contributions [\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Furthermore, the overlap of SERPINA6 plasma concentration between glioma and non-glioma patients was substantial. This is a significant weakness of any diagnostic biomarker (see their Figs.\u0026nbsp;5 and 6 [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]).\u003c/p\u003e \u003cp\u003eHere, we validate some diagnostic findings of the Shen et.al. paper by using independent patient groups from gliomas and meningiomas and a different (orthogonal) assay (PEA) to analyze the plasma samples.\u003c/p\u003e"},{"header":"Materials and Methods","content":"\u003cp\u003eWe analyzed a cohort of the following plasma samples, provided by the Northwestern University Brain Tumor Biobank that were collected at diagnosis but before therapy, from patients with gliomas (N\u0026thinsp;=\u0026thinsp;30), and meningiomas (as benign controls) (N\u0026thinsp;=\u0026thinsp;20). Analysis of these samples was performed at OLINK Proteomics facilities using the PEA technology, as described in detail elsewhere [\u003cspan additionalcitationids=\"CR6\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. The list of the 3,000 proteins that are included in the OLINK panel can be found on the OLINK Website ([email protected]). As per our previous validation [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], PEA is reproducible, with coefficients of variation (CVs) of \u0026lt;\u0026thinsp;20% for \u0026gt;\u0026thinsp;99% of the proteins. Our own efforts to discover novel glioma biomarkers have been published elsewhere [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTo successfully compare our results with those of Shen et al., an initial analysis of our data for the 15 proteins that were also discovered by Shen et al. was performed. SERPINA6 was selected for an in-depth analysis, due to the proposed diagnostic strength of this marker by the group [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. In particular, re-analysis of the SERPINA6 data from Shen et al. was conducted, followed by student-t tests and Wilcoxon signed-rank tests for both our analysis and the re-analysis.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eShen et al. used 4 cohorts of patients for their discovery and validation experiments [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. These cohorts included serum and tissues and can be seen in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e of their paper. Below, we will show the re-analyzed Shen et al. data from all four cohorts, and our data, for several biomarker candidates.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the differences in relative protein intensity between patients with glioma vs meningioma (our patient cohorts) for all the proteins included in the OLINK panel and that were promising biomarkers in the Shen et.al paper. As mentioned above, we compared the protein values through t-tests. Our obtained data (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) do not confirm the Shen et al findings. None of the examined proteins, including the most promising candidate biomarker, SERPINA6, was different between gliomas and meningiomas. SERPINA6 was slightly lower in the glioma group, but the difference was not statistically significant (p\u0026thinsp;=\u0026thinsp;0.47).\u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eRe-analysis of the data for SERPINA6\u003c/h2\u003e \u003cp\u003eWe separately re-analyzed the data for SERPINA6, which was the most promising glioma biomarker identified in the study of Shen et al. (raw data were available by the authors through their publication: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC10871530/bin/sciadv.adk1721_data_s1_to_s4.zip\u003c/span\u003e\u003cspan address=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10871530/bin/sciadv.adk1721_data_s1_to_s4.zip\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eBlood samples from patients with glioma were collected by the authors at three different locations (Cohort 1). Although it is stated in the paper that SERPINA6 is decreased in glioma-derived peripheral serum (pV), in comparison to glioma artery (gA) and glioma vein (gV) serum, the plotted concentrations of SERPINA6 are slightly increased in the peripheral serum (pV) of some patients but the differences were not statistically significant (paired t-tests) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eA).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFor the second cohort, tissue samples were collected from normal brain, perineural and glioma patients. There were no measured SERPINA6 values from the normal tissues. A Wilcoxon test looking for differences in medians between the groups was not significant. The distribution of SERPINA6 in peritumoral tissue and glioma tissue is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eB (Cohort 2).\u003c/p\u003e \u003cp\u003eWe then compared the concentration of SERPINA6 in peripheral blood of controls (facial paralysis) (red dots) vs glioma patients (blue dots) (Cohort 3 data). The glioma group had lower values than the controls (Wilcoxon p\u0026thinsp;\u0026lt;\u0026thinsp;0.001) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eC). This observation agrees with the reported data by Shen et al. [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe finally compared SERPINA6 concentration in peripheral blood of controls (facial paralysis) vs glioma patients, from cohort 4 data (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003eD). Indeed, the glioma patients had slightly lower values, in agreement with the authors\u0026rsquo; claim, but the differences were barely significant by Wilcoxon test (p\u0026thinsp;=\u0026thinsp;0.046).\u003c/p\u003e \u003cp\u003eIn ROC analysis for predicting glioma from controls by using serum SERPINA6, we found, in agreement with the authors, some discriminatory potential, which, however, was not sufficient for reliably diagnosing glioma with their blood test.\u003c/p\u003e \u003cp\u003eROC curves to predict glioma vs non-glioma with SERPINA6 as biomarker (cohort 4) were constructed. The data were similar to the author\u0026rsquo;s reported results for cohort 4.\u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eThere is a considerable interest in developing disease-related biomarkers and applying them to improve patient care. Despite major investments in this area by granting agencies and commercial organizations, the yield has been disappointingly poor. The handful of cancer biomarkers that are used in the clinic today were discovered more than 40 years ago [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e \u003cp\u003ePreviously, we and others commented on several reasons that contribute to newly discovered biomarker failures and identified pre-analytical, analytical, and post-analytical shortcomings which may affect a biomarker\u0026rsquo;s performance [\u003cspan additionalcitationids=\"CR11 CR12 CR13 CR14 CR15 CR16\" citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. In short, most of the failures are due to unrecognized biases/differences between the diseased and controlled clinical samples, the groups and numbers of patients and their clinical information, the analytical method used, and more recently, the way the data are interpreted (black box approach) [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eRecently, considerable efforts were made, to find ways to better reproduce published and seemingly promising biomarkers. It has been realized that a large number of manuscripts, published in even top-rated journals, describe false discovery [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. The number of retractions of manuscripts published in highly reputable journals is at the all-time high [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eOne of the ways to decrease false discovery is to reproduce the findings, preferably independently, from the original investigators. This is not an easy task, since specific reagents and techniques may not be available to the validators. Also, this process is time-consuming and expensive. We proposed a simpler way to tackle the irreproducibility problem that we coined \u0026ldquo;the 5-year reflection\u0026rdquo; [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Reproducing at least a fraction of high impact studies may reveal weaknesses which can lead to improved outcomes in the future.\u003c/p\u003e \u003cp\u003eIn the paper under discussion, Shen et al. tried to identify new biomarkers for glioma by collecting blood samples from the peripheral circulation and from glioma arteries and veins in the vicinity of the tumor. Tissue samples were also collected. To validate their discovery data, they collected samples that were not used in the discovery phase.\u003c/p\u003e \u003cp\u003eIn our previous work, we described the identification of glioma biomarkers by a different proteomic technique, the Proximity Expression Assay (PEA) [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Taking these results, we then examined which of the proteins identified by Shen et al. and showed biomarker promise, were in the PEA panel. Surprisingly, we found no overlap between the candidates in the two studies. Even the best putative biomarker selected by Shen et al. (SERPINA6) showed no differences between gliomas and meningiomas in our patients. Additional analyses have shown that SERPINA6 concentration was no different between the tumoral and peritumoral tissue. To explain some of these discrepancies, we suggest that stricter quantitative technologies are likely needed to identify differences between glioma and non-glioma patients in the discovery phase, with emphasis to be given to clinical utility.\u003c/p\u003e \u003cp\u003eRegarding the diagnostic power of SERPINA6, the sensitivity and specificity of the test was about 88%. This led the authors to conclude that it may be suitable for clinical use. However, even these seemingly high sensitivities and specificities are likely not enough for clinical use, as we exemplified elsewhere [\u003cspan additionalcitationids=\"CR3\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Moreover, for a biomarker to be promising it needs to have clinical utility that complements statistical significance. A clinically useful biomarker needs to have either very high specificity\u0026thinsp;+\u0026thinsp;good sensitivity or very high sensitivity\u0026thinsp;+\u0026thinsp;good specificity. Simply put, having a significant difference in median/mean value between groups, does not confer clinical utility.\u003c/p\u003e \u003cp\u003eWe conclude that the biomarkers identified by Shen et al. [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] were not successfully validated with our own sets of data from glioma and meningioma patients and the characteristics of the test, as published, is not sufficient to warrant any clinical applications at present.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAuthor contributions statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMKC drafted and edited the manuscript, and prepared figure 2. LMA performed the statistical analysis, prepared figures 1 and 2 and edited the manuscript. EPD conceptualized, drafted and edited the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe excel file with the concentrations of the biomarkers and their concentrations are available by request from the corresponding author.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting Interests Statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors report no conflict of interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eShen, L. \u003cem\u003eet al.\u003c/em\u003e Mechanistic insight into glioma through spatially multidimensional proteomics. Sci. Adv. 10, eadk1721 (2024).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFiala, C. \u0026amp; Diamandis, E. P. A multi-cancer detection test: focus on the positive predictive value. Ann. Oncol. 31, 1267\u0026ndash;1268 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiamandis, E. P. \u0026amp; Li, M. The side effects of translational omics: overtesting, overdiagnosis, overtreatment. Clin. Chem. Lab. Med. 54, 389\u0026ndash;396 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFiala, C., Taher, J. \u0026amp; Diamandis, E. P. P4 Medicine or O4 Medicine? Hippocrates Provides the Answer. J. Appl. Lab. Med. 4, 108\u0026ndash;119 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, M. \u003cem\u003eet al.\u003c/em\u003e Plasma Protein Profiling by Proximity Extension Assay Technology Reveals Novel Biomarkers of Traumatic Brain Injury-A Pilot Study. J. Appl. Lab. Med. 6, 1165\u0026ndash;1178 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRen, A. H., Diamandis, E. P. \u0026amp; Kulasingam, V. Uncovering the Depths of the Human Proteome: Antibody-based Technologies for Ultrasensitive Multiplexed Protein Detection and Quantification. Mol. Cell. Proteomics MCP 20, 100155 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRen, A. \u003cem\u003eet al.\u003c/em\u003e Comparison of two multiplexed technologies for profiling\u0026thinsp;\u0026gt;\u0026thinsp;1,000 serum proteins that may associate with tumor burden. F1000Research 10, 509 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhorbani, A. \u003cem\u003eet al.\u003c/em\u003e Discovery of novel glioma serum biomarkers by proximity extension assay. Clin. Proteomics 20, 12 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSturgeon, C. M. \u003cem\u003eet al.\u003c/em\u003e National Academy of Clinical Biochemistry laboratory medicine practice guidelines for use of tumor markers in testicular, prostate, colorectal, breast, and ovarian cancers. Clin. Chem. 54, e11-79 (2008).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiamandis, E. P. The failure of protein cancer biomarkers to reach the clinic: why, and what can be done to address the problem? BMC Med. 10, 87 (2012).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiamandis, E. P. Cancer Biomarkers: Can We Turn Recent Failures into Success? JNCI J. Natl. Cancer Inst. 102, 1462\u0026ndash;1467 (2010).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDiamandis, E. P. Analysis of serum proteomic patterns for early cancer diagnosis: drawing attention to potential problems. J. Natl. Cancer Inst. 96, 353\u0026ndash;356 (2004).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRansohoff, D. F. Bias as a threat to the validity of cancer molecular-marker research. Nat. Rev. Cancer 5, 142\u0026ndash;149 (2005).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIoannidis, J. P. A. Discussion: Why \u0026lsquo;An estimate of the science-wise false discovery rate and application to the top medical literature\u0026rsquo; is false. \u003cem\u003eBiostat. Oxf. Engl.\u003c/em\u003e 15, 28\u0026ndash;36; discussion 39\u0026ndash;45 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIoannidis, J. P. A. Molecular bias. Eur. J. Epidemiol. 20, 739\u0026ndash;745 (2005).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePrassas, I. \u003cem\u003eet al.\u003c/em\u003e False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. Clin. Chem. 60, 381\u0026ndash;388 (2014).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePetricoin, E. F. \u003cem\u003eet al.\u003c/em\u003e Use of proteomic patterns in serum to identify ovarian cancer. Lancet Lond. Engl. 359, 572\u0026ndash;577 (2002).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchully, S. D. \u003cem\u003eet al.\u003c/em\u003e Leveraging biospecimen resources for discovery or validation of markers for early cancer detection. J. Natl. Cancer Inst. 107, djv012 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLevett, J. J. \u003cem\u003eet al.\u003c/em\u003e Publication retraction in spine surgery: a systematic review. Eur. Spine J. Off. Publ. Eur. Spine Soc. Eur. Spinal Deform. Soc. Eur. Sect. Cerv. Spine Res. Soc. 32, 3704\u0026ndash;3712 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBall, P. Is AI leading to a reproducibility crisis in science? Nature 624, 22\u0026ndash;25 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFiala, C. \u0026amp; Diamandis, E. P. How to reduce scientific irreproducibility: the 5-year reflection. Clin. Chem. Lab. Med. 55, 1845\u0026ndash;1848 (2017).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"biomarkers, glioma, false discovery, biomarker failures, SERPINA6, biomarker validation","lastPublishedDoi":"10.21203/rs.3.rs-4397157/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4397157/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eBiomarkers are useful clinical tools but only a handful of them are used routinely for patient care. Despite intense efforts to discover new, clinically useful biomarkers, very few new circulating biomarkers were implemented in clinical practice in the last 40 years. This is mainly due to poor clinical performance. Here, our goal was to validate a group of newly discovered circulating biomarkers for glioma by comparing our data with data from a paper recently published in Science Advances. We analyzed our own sets of clinical samples and a different analytical assay to compare the results of Shen and colleagues. Despite the sophistication of the utilized discovery method, we found that the newly proposed biomarkers for glioma (such as SERPINA6) did not perform as expected. Scientific irreproducibility has been extensively discussed in the literature. A large proportion of newly discovered candidate biomarkers likely represent \u0026ldquo;false discovery\u0026rdquo; and contribute to irreproducible results. The best way to assess the value of any new biomarker is by independent and extensive validation. Based on our previous classification, we believe that this work represents another example of a false discovery.\u003c/p\u003e","manuscriptTitle":"Validation of new, circulating biomarkers for glioma","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-05 17:22:10","doi":"10.21203/rs.3.rs-4397157/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2184154e-5f73-4c3d-b081-1fb53c91c81a","owner":[],"postedDate":"June 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":32399994,"name":"Biological sciences/Cancer"},{"id":32399995,"name":"Health sciences/Biomarkers"},{"id":32399996,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2024-10-22T07:39:15+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-05 17:22:10","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4397157","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4397157","identity":"rs-4397157","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}