Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies.

Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies. | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies. Charles D. G. Burns, Alessio Fracasso, Guillaume A. Rousselet This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4457116/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 19 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract Recent studies have leveraged consortium neuroimaging data to answer an important question: how many subjects are required for reproducible brain-wide association studies? These data-driven approaches could be considered a framework for testing the reproducibility of several neuroimaging models and measures. Here we test part of this framework, namely estimates of statistical errors of univariate brain-behaviour associations obtained from resampling large datasets with replacement. We demonstrate that reported estimates of statistical errors are largely a consequence of bias introduced by random effects when sampling with replacement close to the full sample size. We show that future meta-analyses can largely avoid these biases by only resampling up to 10% of the full sample size. We discuss implications that reproducing mass-univariate association studies requires tens-of-thousands of participants, urging researchers to adopt other methodological approaches. Biological sciences/Neuroscience Biological sciences/Computational biology and bioinformatics/Communication and replication Biological sciences/Computational biology and bioinformatics Biological sciences/Computational biology and bioinformatics/Statistical methods Physical sciences/Mathematics and computing/Statistics Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction The question of scientific reliability of brain-wide association studies (BWAS) was brought to the attention of many 1 , 2 by Marek, Tervo-Clemmens et al. 3 , reigniting discussions 4 – 7 about the ongoing reproducibility crisis in neuroscience and psychology 8 – 12 . Independent researchers are failing to reproduce (same results using the same methods and data) and replicate (similar results using the same methods and new data) many published findings. In the field of neuroimaging, reproducibility issues are further exacerbated by variability in the methods used by researchers, which can lead to conflicting results 13 . Our trust in the scientific field therefore relies on how well we can estimate its reproducibility. For any given study design, a reliable way of increasing the likelihood of replication is to recruit more subjects, which will reduce the sampling variability and in turn increase statistical power 8 . For a brain-wide association study (BWAS) which aims to characterise associations between brain measures and behaviours, collecting data is expensive. So how many subjects are required? How do we know? Thousands are required 3 , 14 , according to data-driven approaches which quantify the issue of reproducibility for BWAS using large neuroimaging datasets from the Human Connectome Project 15 (HCP with n = 1,200), the Adolescent Brain Cognitive Development study 16 (ABCD with n = 11,874), and the UK Biobank 17 (UKB with n = 35,735). Among numerous analyses in their study, Marek, Tervo-Clemmens et al . 3 estimated statistical errors of univariate BWAS as a function of sample size. Such univariate BWAS often involve tens of thousands of correlations between a brain measure and a behavioural measure, most of which fail to replicate even with thousands of participants. These replication failures can be explained by statistical errors of a study design such as false positive rates 11 and low statistical power 8 , 18 – 22 . To estimate statistical errors in univariate BWAS, Marek, Tervo-Clemmens et al . 3 treated a large discovery dataset as a population and then drew replication samples by resampling with replacement (henceforth resampling) from that population. Here, we report the first test of validity of this data-driven approach by using simulated data as ground truth. Results Resampling methods strongly bias statistical error estimates when there are no true effects First, we simulated a discovery null sample with n = 1,000 subjects each with 1,225 brain connectivity measures (random Pearson correlations) and a single behavioural measure (normally distributed across participants). We correlated each brain connectivity measure with the behaviour across all subjects to obtain 1,225 brain-behaviour correlations. Since brain connectivity estimates and behavioural factors were simulated independently from each other, any resulting brain-behaviour correlations were entirely random. Data dimensions were chosen to be computationally feasible for reproducibility, however we invite readers to adjust these and re-run analyses using the openly available code (analyses recoded in R with supporting packages 23 – 25 for open-source accessibility https://github.com/charlesdgburns/rwr/ ). We then resampled our null-sample for 100 iterations across logarithmically spaced sample size bins ( n = 25, to 1,000) and estimated statistical errors, following the methods described in Marek, Tervo-Clemmens et al .1. Surprisingly, we saw the same trends of statistical errors and reproducibility as those reported by Marek, Tervo-Clemmens et al .1 but with random data (see Fig. 1 ), with strongly biased statistical power estimates. These trends in statistical errors thus do not depend on absolute sample size, but the resample size relative to the full sample size. By repeatedly generating new null-samples, rather than resampling from a single null-sample, we verified that these statistical error estimates are indeed biased under the null as the resample size approaches the full sample size (Fig. 2 ). For example, uncorrected (α = .05), statistical power was estimated to be 63% when resampling at the full sample size (n = 1,000, Fig. 1 . d), rather than the expected 5% obtained when generating new null- samples (n = 1,000, Fig. 2 . d). One concern is that power is the most inflated while also being the most relevant for failed replications 8 , 18 , 19 , which could potentially result in misleading meta-science. Compounding sampling variability underlies biased statistical errors under the null To explain why biases arise under the null, we investigated the underlying brain-behaviour correlations used in the calculation of statistical errors. Here we focused on resampling at the full sample size ( n = 1,000) where these biases are most dramatic. As indicated by the false positive rate (Fig. 1 f.), the null distribution of brain-behaviour correlations is not preserved when resampling at the full sample size (Fig. 3 ). Instead, resampling subjects and computing correlations again results in a distribution wider than expected (comparing Fig. 3 a. and c .). This is because resampling involves two sources of sampling variability, first at the level of the discovery sample and again for the resampled replication sample. For instance, if a correlation in the discovery sample is randomly observed to be r = 0.11, then resampling participants and computing the same correlation again results in a correlation which varies around r = 0.11 (Fig. 3 e.). We can formalise this mathematically as nested distributions 26 , or a convolution 27 of two probability distributions, here approximating Pearson null distributions with normal distributions for analytical simplicity. It then follows that given a discovery sample X ~ N(µ, σ 1 2 ), for each observation in our original sample, x i ∈ X , resampling participants and recomputing correlations corresponds to sampling from several distributions X i ~ N ( x i , σ 2 2 ), resulting in a final set of correlations distributed according to X* ~ N(µ, σ 1 2 + σ 2 2 ). Note that σ 1 2 depends on the size of the discovery sample, while σ 2 2 is determined by the resample size. The influence on statistical error estimates such as statistical power is two-fold. First, random correlations in the tail of a discovery sample are more likely to be in the tail of correlations in a resampled replication sample. This inflates power when estimated as the proportion of significant effects in the discovery sample which are significant again in the resampled replication sample (1 – false negative rates). Second, increased sampling variability alone leads to a wider-than-expected distribution of correlations with more extreme tails. These more extreme tails lead to an inflation of P values close to 0 in our resample (compare Fig. 3 b. and d .) when calculated using a standard correlation function (e.g., ‘corr’ in MATLAB). We note that simply correcting for this widened null distribution will over-correct for bias in statistical error estimates when true effects are present (see Supplementary Information). Bias in ground truth simulations depends on statistical power of the full sample size While we have shown clear biases when there are no true effects, this does not directly imply biases when true effects are present. We note that Marek, Tervo-Clemmens et al. 3 have already shown that the largest univariate effect is highly replicable even for moderate sample sizes, so there are at least some true BWAS effects in the real world. However, as the average true effect size remains unknown, we systematically simulated a range of discovery samples, each representing a study where the size of the underlying true effects corresponded with different levels of statistical power. We note that since the bias under the null is driven by the false rejection of null hypotheses, here we adopt a fixed significance threshold after Bonferroni correction which controls for at least one false positive (family-wise error rate). Focusing on statistical power estimates, we show that the bias near the full sample size depends on the true statistical power of the discovery sample (Fig. 4 ). Power estimates are inflated if the discovery sample is underpowered, but on the other hand a highly powered discovery sample may give conservative power estimates. Note that regardless of power at the full sample size, bias in statistical power is largely avoided when subsampling up to around 10% of the full sample size (see also Supplementary Information for subject-level simulations). Discussion Accurately estimating reproducibility of scientific methods is critical for guiding researcher’s methodological decisions. Our results demonstrate that estimating statistical errors by resampling with replacement from random data results in large biases when resampling near the full sample size. We explain this fully by compounding sampling variability of test statistics when resampling and its knock-on effects on estimated statistical errors. We further simulate ground truth data with true effects to show that statistical power is inflated when the true power of the discovery sample is low and slightly deflated when true power is high. This could lead to circular reasoning in cases where we must assume we have high statistical power before we can rely on the estimation that we have high statistical power. Lastly, we show that this bias is largely avoided when subsampling only up to 10% of the full sample size after Bonferroni correction. This 10% rule of thumb is consistent with the use of resampling techniques in a recent evaluation of statistical power and false discovery rates for genome-wide association studies with hundreds-of-thousands of participants 28 , as well as recommendations for 10-fold cross-validation to reduce prediction error in machine-learning 29 . What are the implications for the results presented by Marek, Tervo-Clemmens et al .1? For the strictly denoised Adolescent Brain Cognitive Development (ABCD) sample ( n = 3,928), they report around 68% power at n = 3,928 after Bonferroni correction when resampling at the full sample size (Marek, Tervo-Clemmens et al .1 Fig. 3 d.). Our true effect simulation results indicate that this estimate could be inflated from a true average power anywhere between 1% and 40%. Furthermore, when subsampling from the UK Biobank with a full sample size of n = 32,572 Marek, Tervo-Clemmens et al .1 report around 1% power for n = 4,000 and α = 10 − 7. We therefore argue that the 68% power reported for the full ABCD sample ( n = 3,928, α = 10 − 7) more likely reflects methodological bias, rather than a result of increased signal after strict denoising of brain data. While the largest BWAS effects may be highly reproducible with 4,000 participants, the average univariate BWAS effect is most likely not reproducible. On the other hand, our true effect simulations (Fig. 4 .) also indicate that the UK Biobank estimates at the full sample size are more reliable, with an underlying power likely between 70% and 90% at n = 32,572 after Bonferroni correction. Ultimately, our results suggest that replicating the univariate BWAS tested in Marek, Tervo-Clemmens et al . requires tens-of-thousands of individuals. Our results only have direct implications for mass univariate association studies, however it is worth noting how methodological decisions could influence reproducibility in neuroimaging. For example, it should be noted that inter-individual correlation studies offer “as little as 5%-10% of the power” of within-subject t-test studies with the same number of participants4. Other methodological choices, such as data modelling, should also be carefully considered. The lack of power in univariate BWAS considered by Marek, Tervo-Clemmens et al . could also be influenced by the choice of a group-averaged brain parcellation 30 , which fails to account for individual level variations in resting state functional connectivity 31 , 32 . Brain models 33 which do account for such individual variability generalise better, as demonstrated by stronger out-of-sample prediction 31 , 34 , and could also lead to higher replication rates in null-hypothesis significance tests. Note also that how we model null distributions 35 of brain-wide statistics has a large influence on resulting P values. With this in mind, one could consider a predictive framework rather than an explanatory one 36 , which could be replicable with only hundreds of participants 37 , 38 It is clear that investigations of reproducibility of wider BWAS methods are required. We urge such meta-analyses to evaluate their meta-analytic methods, for example with null data, so they may reliably evaluate the reproducibility of scientific methods used in research. Methods Simulating null data at subject level We simulated random phenotype associations with simulated functional connectivity measures. We generated a null-sample with n = 1,000 subjects each with 1,225 edges (random Pearson correlations between 50 random time series) and a single behavioural factor (normally distributed across participants). We correlated each edge with the behaviour across all subjects to obtain 1,225 brain-behaviour correlations. By generating edge connectivity estimates and behavioural factors independently from each other, we ensured that any resulting brain-behaviour correlations are entirely random (ρ = 0), hence obtaining a sample where the null hypothesis is true (i.e., a null-sample). Estimating statistical errors We closely followed the methods of Marek, Tervo-Clemmens et al. 3 , first running analyses on MATLAB using their code 'abcd_edgewise_correlation_iterative_reliability_single_factor.m' and 'abcd_statisticalerrors.m' ( https://gitlab.com/DosenbachGreene/bwas ). These analyses were then independently recoded in R with supporting packages 23 – 25 for open-source accessibility https://github.com/charlesdgburns/rwr/ . Notably, statistical error estimations involve two-tailed P values derived from parametric null distributions on a given resample size. Simulating ground truth data with known statistical power At this stage we take a computationally more efficient approach and simulate summary statistics rather than subject-level data, which allows us to simulate many more true effect scenarios so we can compare estimates with true statistical errors. This approach also lets us increase the number of effects, so we now simulate samples with 55,278 (333 choose 2) effects, the number of resting-state functional connectivity measures which feature in Marek, Tervo-Clemmens et al . Figure 2 3 . The size of true effects was determined by an inverse power analysis 39 with a fixed sample size (n = 1,000) and Bonferroni corrected significance threshold (α = 0.05/55278), which involved a Fisher z-transformation for calculating the critical Pearson r for a given power level (power = 1%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 99%). The Fisher z-transformation 40 , F (r) = atanh(r) = z was also used to sample Pearson correlations using the approximation that the z-statistic is asymptotically normally distributed with mean F (r) and standard deviation \(1/\sqrt{\left(n-3\right)}\) . The z-statistic was then transformed into Pearson correlations to simulate brain-behaviour correlations. For each power level, we simulated a discovery sample by first drawing 55,278 random effects (ρ = 0) and afterwards replacing 500 of those with true effects, where ρ corresponds to the critical r for a given power level computed earlier. The choice of 500 true effects was somewhat arbitrary, being moderate enough to seem probable but also sufficiently many true effects to reduce noise in bias estimates. Note that while real world effect sizes of a single BWAS may vary, we instead simulate several effects with the same underlying effect size. This should not be an issue, since the statistical error summary statistics are given as an average across effects, so in this simulation we can think of the underlying effects having an average effect size according to a given power level. Simulating statistical power estimations from resampling with replacement Given only summary statistics rather than individual subjects, we cannot resample participants and recompute P values, but instead also simulate obtaining estimates by resampling with replacement as in subject-level analyses 3 . First, we generate ground truth data with known statistical power, which we treat as our discovery sample. Then, we follow the assumption that the observed effects in a discovery sample are the population effects: for a given resample size n , a resampled effect size was drawn from a normal distribution \(N\left(\text{a}\text{t}\text{a}\text{n}\text{h}\left({r}^{*}\right),1/\sqrt{n-3}\right)\) , where r* is a given Pearson correlation from the discovery sample, and then Fisher z-transformed. We then derived P values from an uncorrected null distribution of Pearson correlations with degrees of freedom computed relative to the resample size. We resampled across the same range of sample sizes as in previous analyses (n = 25, … 1,000). We continued to estimate statistical power across 1,000 iterations of resampled brain-behaviour correlations as in Marek, Tervo-Clemmens et al. 1 , specifically as the proportion of significant effects in a discovery sample which were significant again in a resample (1 – false negative rates, with α = 0.05/55278). These were then compared to analytical power curves 39 computed using Fisher z-transformations for varying sample sizes and effect sizes corresponding to critical r for power levels at the full sample size (n = 1,000) computed earlier. Declarations Competing interests None. Author Contribution C.D.G.B.: Conceptualisation, design, implementation, analysis, interpretation, writing - original draft. A.F.: Interpretation of results, writing - review & editing. G.A.R.: Conceptualisation, design, interpretation, writing - review & editing, supervision. Acknowledgements A.F. is supported by a grant from the Biotechnology and Biology research council (BBSRC, grant number: BB/S006605/1) and the Fundação Bial, Fundação Bial Grants Programme 2020/21, A- 29315, number 203/2020, grant edition: G-15516. Data Availability R and MATLAB code used for data simulation, statistical analyses, and plotting is available on GitHub: https://github.com/charlesdgburns/rwr/. References Callaway, E. Can brain scans reveal behaviour? Bombshell study says not yet. Nature 603, 777–778 (2022). Richtel, M. Brain-Imaging Studies Hampered by Small Data Sets, Study Finds. The New York Times (2022). Marek, S. et al. Reproducible brain-wide association studies require thousands of individuals. Nature 603, 654–660 (2022). Gratton, C., Nelson, S. M. & Gordon, E. M. Brain-behavior correlations: Two paths toward reliability. Neuron 110, 1446–1449 (2022). Rosenberg, M. D. & Finn, E. S. How to establish robust brain–behavior relationships without thousands of individuals. Nat. Neurosci. 25, 835–837 (2022). Botvinik-Nezer, R. & Wager, T. D. Reproducibility in Neuroimaging Analysis: Challenges and Solutions. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 8, 780–788 (2023). Helwegen, K., Libedinsky, I. & van den Heuvel, M. P. Statistical power in network neuroscience. Trends Cogn. Sci. 27, 282–301 (2023). Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013). Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 1–9 (2017). Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). Ioannidis, J. P. A. Why Most Published Research Findings Are False. PLOS Med. 2, e124 (2005). Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F. & Baker, C. I. Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535–540 (2009). Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020). Liu, S., Abdellaoui, A., Verweij, K. J. H. & van Wingen, G. A. Replicable brain–phenotype associations require large-scale neuroimaging data. Nat. Hum. Behav. 7, 1344–1356 (2023). Van Essen, D. C. et al. The WU-Minn Human Connectome Project: An overview. NeuroImage 80, 62–79 (2013). Casey, B. J. et al. The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018). Sudlow, C. et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 12, e1001779 (2015). Ingre, M. Why small low-powered studies are worse than large high-powered studies and how to protect against “trivial” findings in research: Comment on Friston (2012). NeuroImage 81, 496–498 (2013). Yarkoni, T. Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary on Vul et al. (2009). Perspect. Psychol. Sci. 4, 294–298 (2009). Cremers, H. R., Wager, T. D. & Yarkoni, T. The relation between statistical power and inference in fMRI. PLOS ONE 12, e0184923 (2017). Szucs, D. & Ioannidis, J. PA. Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals. NeuroImage 221, 117164 (2020). Poldrack, R. A. et al. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat. Rev. Neurosci. 18, 115–126 (2017). Ripley, B. et al. MASS: Support Functions and Datasets for Venables and Ripley’s MASS. (2023). Wickham, H. et al. Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019). Kassambara, A. ggpubr: ‘ggplot2’ Based Publication Ready Plots. (2022). El Otmani, S. & Maul, A. Probability distributions arising from nested Gaussians. Comptes Rendus Math. 347, 201–204 (2009). Convolution of Gaussians is Gaussian. https://jeremy9959.net/Math-5800-Spring-2020/notebooks/convolution_of_gaussians.html . Chen, Z., Boehnke, M., Wen, X. & Mukherjee, B. Revisiting the genome-wide significance threshold for common variant GWAS. G3 GenesGenomesGenetics 11, jkaa056 (2021). Witten, I. H., Frank, E., Hall, M. A., Pal, C. J. & DATA, M. Practical machine learning tools and techniques. Data Min. Fourth Ed. Elsevier Publ. (2017). Gordon, E. M. et al. Generation and Evaluation of a Cortical Area Parcellation from Resting-State Correlations. Cereb. Cortex 26, 288–303 (2016). Kong, R. et al. Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior. Cereb. Cortex 31, 4477–4500 (2021). Gordon, E. M. et al. Precision Functional Mapping of Individual Human Brains. Neuron 95, 791–807.e7 (2017). Bijsterbosch, J. D., Valk, S. L., Wang, D. & Glasser, M. F. Recent developments in representations of the connectome. NeuroImage 243, 118533 (2021). Farahibozorg, S.-R. et al. Hierarchical modelling of functional brain networks in population and individuals from big fMRI data. NeuroImage 243, 118513 (2021). Markello, R. D. & Misic, B. Comparing spatial null models for brain maps. NeuroImage 236, 118052 (2021). Yarkoni, T. & Westfall, J. Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspect. Psychol. Sci. 12, 1100–1122 (2017). Spisak, T., Bingel, U. & Wager, T. D. Multivariate BWAS can be replicable with moderate sample sizes. Nature 615, E4–E7 (2023). Chen, J. et al. Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study. NeuroImage 274, 120115 (2023). Designing Clinical Research . (Wolters Kluwer/Lippincott Williams & Wilkins, Philadelphia, 2013). Fisher, R. A. Frequency Distribution of the Values of the Correlation Coefficient in Samples from an Indefinitely Large Population. Biometrika 10, 507–521 (1915). Additional Declarations No competing interests reported. Supplementary Files Supplementarydraft.docx Cite Share Download PDF Status: Published Journal Publication published 19 Feb, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 22 Jul, 2024 Reviews received at journal 19 Jul, 2024 Reviews received at journal 10 Jul, 2024 Reviewers agreed at journal 14 Jun, 2024 Reviewers agreed at journal 13 Jun, 2024 Reviewers invited by journal 08 Jun, 2024 Editor assigned by journal 04 Jun, 2024 Editor invited by journal 29 May, 2024 Submission checks completed at journal 27 May, 2024 First submitted to journal 21 May, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4457116","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":310996695,"identity":"737924f0-dd0e-4035-ad7f-5baf0c9b0f0e","order_by":0,"name":"Charles D. G. Burns","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/klEQVRIiWNgGAWjYDACCSB+wMDA2CAB5toAMWPjAYJaEhBa0kBaGkjSchhM4tXCP7v52YeEijrZ+dE9Zp8L/py3W9t+GGhLjU00TkvuHDOekXDmsPHGO2eMZ89su5287UwiUMuxtNwGHFoMJBKMGRLbDiRunJFjzMzbcDvZ7ABQC2PDYTxa0j8zJP6rg2jh+XMu2ez8Q0JacoC2NDAnzpcAaWE7YGd2g4AtEjdyihkSjh023iBzrJiZty05wewG0JYEPH7hn5G+meFDDTDEZjdvBjrMzt7sfPrDBx9qbHBqQbjwAIROBKtMIKQcBOShhtoTo3gUjIJRMApGFgAAMlFlXqhcpHIAAAAASUVORK5CYII=","orcid":"","institution":"University of Glasgow","correspondingAuthor":true,"prefix":"","firstName":"Charles","middleName":"D. G.","lastName":"Burns","suffix":""},{"id":310996696,"identity":"2e9bf944-1cc7-40aa-ba71-7d1b1049fac6","order_by":1,"name":"Alessio Fracasso","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"prefix":"","firstName":"Alessio","middleName":"","lastName":"Fracasso","suffix":""},{"id":310996699,"identity":"af1ce5a2-5587-4d21-9440-1834e9a39660","order_by":2,"name":"Guillaume A. Rousselet","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"prefix":"","firstName":"Guillaume","middleName":"A.","lastName":"Rousselet","suffix":""}],"badges":[],"createdAt":"2024-05-21 21:38:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4457116/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4457116/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-89257-w","type":"published","date":"2025-02-19T15:57:55+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":58146345,"identity":"ec325130-6e75-4581-9fee-c9627f47c9f4","added_by":"auto","created_at":"2024-06-11 18:35:31","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":470020,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eEstimated statistical errors and reproducibility of random noise (\u003c/strong\u003e\u003cem\u003eρ \u003c/em\u003e\u003cstrong\u003e= 0). \u003c/strong\u003eWe reproduce statistical error estimates after resampling from a simulated null-sample where observed significant effects are random (ρ = 0). These results are notably comparable to Fig. 3 in Marek et al.\u003csup\u003e3\u003c/sup\u003e \u003cstrong\u003ea\u003c/strong\u003e, False negative rates computed relative to full sample size decrease as the resample size increases, across a range of significance thresholds which were passed after sampling 1,225 random correlations. \u003cstrong\u003eb\u003c/strong\u003e, For a given magnitude of inflation, when inflation rates are computed relative to effect sizes at the full sample size they fall to 0% as the resample size approaches the full sample size. \u003cstrong\u003ec\u003c/strong\u003e, Sign errors computed relative to signs of effects at the full sample size show a downwards trend as resample sizes increase from around chance level of 50% to 30%. \u003cstrong\u003ed\u003c/strong\u003e, Statistical power when computed relative to full sample size shows a strong upwards trend as resample sizes increase, reaching around 60% for all significance thresholds which were crossed when simulating 1,225 random effects (α=0.05 to 0.0001). \u003cstrong\u003ee\u003c/strong\u003e, Probability of replication, computed by resampling both equally sized ‘in-sample’ and ‘out-of-sample’ subsamples from the full dataset, stays low for small significance thresholds, but reaches above 10% for α=0.05. \u003cstrong\u003ef,\u003c/strong\u003e False positive rates, computed by counting correlations which are significant only in the resample but not in the full sample size, also reach about 10% for α=0.05 in the full sample size. Note that the darkest lines are drawn after random effects pass thresholds as low as α=10\u003csup\u003e-7\u003c/sup\u003e after resampling from the full sample, which is lower than Bonferroni correction 0.05/1225 = 4x10\u003csup\u003e-5\u003c/sup\u003e.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/dc696eb63acc7e20b7a7353a.jpeg"},{"id":58146349,"identity":"8b3c1472-5f8c-4161-a3ac-15dec72244e6","added_by":"auto","created_at":"2024-06-11 18:35:31","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":412524,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eExpected statistical error and reproducibility of random noise (\u003c/strong\u003e\u003cem\u003eρ \u003c/em\u003e\u003cstrong\u003e= 0)\u003c/strong\u003e. We obtain a ground truth of statistical errors under the null by iteratively generating null-samples at increasing sample sizes (\u003cem\u003en\u003c/em\u003e=25,...,1,000) instead of resampling from a single null-sample, averaging estimates for each sample size over 100 simulations. This corresponds to sampling from an infinite-size population. \u003cstrong\u003ea, \u003c/strong\u003etrue false negative rates under the null are constant across sample sizes and equivalent to 1 - \u003cem\u003eα \u003c/em\u003efor a given significance threshold (\u003cem\u003eα \u003c/em\u003e= .05, .01, .001 plotted). \u003cstrong\u003eb, \u003c/strong\u003esince inflation rates are given as a proportion of replicated (same sign, and significant across discovery and replication sample) correlations, we can expect these to be high for small sample sizes as the critical r for significance is higher, and so the likelihood of being inflated decreases as sample sizes increase. Since we are averaging across correlations, few of these will be very inflated while many will be less inflated so that on average this cancels out to 50% across inflation thresholds. \u003cstrong\u003ec, \u003c/strong\u003eWe expect 50% sign errors regardless of sample size as the sign of a given correlation in a replication null- sample will be random. \u003cstrong\u003ed, e, f, \u003c/strong\u003eestimates of statistical power, probability of replication, and false positives are based on proportions of significant correlations in replication null-samples, so in each case the probability of a correlation being significant in a newly generated null-sample is exactly determined by the significance threshold (\u003cem\u003eα \u003c/em\u003e= .05, ..., 10\u003csup\u003e-7\u003c/sup\u003e).\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/8a8435720a8776c36b8f2753.jpeg"},{"id":58146344,"identity":"e15a1409-0e95-4e5d-9145-c76bf6b22247","added_by":"auto","created_at":"2024-06-11 18:35:31","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":116197,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eNull distributions under resampling with replacement (\u003c/strong\u003e\u003cem\u003eρ \u003c/em\u003e\u003cstrong\u003e= 0)\u003c/strong\u003e. \u003cstrong\u003ea\u003c/strong\u003e, Distribution of simulated random brain-behaviour correlations (1,225 total) treated as a discovery sample. There are n = 1,000 subjects for each random Pearson correlation, here compared to a Gaussian curve with mean µ = 0 and variance σ\u003csub\u003e1\u003c/sub\u003e\u003csup\u003e2\u003c/sup\u003e = .001 drawn in black. \u003cstrong\u003eb\u003c/strong\u003e, We verify that the two-tailed \u003cem\u003eP \u003c/em\u003evalues of our 1,225 random brain-behaviour correlations are uniformly distributed. \u003cstrong\u003ec\u003c/strong\u003e, Distribution of all 1,225 brain-behaviour correlations computed after resampling subjects at full sample size (n = 1,000). This distribution is clearly wider than our discovery sample null distribution. The solid black line shows the distribution expected from an interaction of sampling variability (see \u003cstrong\u003ee\u003c/strong\u003e.), namely a Gaussian distribution with mean µ = 0 and variance σ\u003csub\u003e1\u003c/sub\u003e\u003csup\u003e2\u003c/sup\u003e +σ\u003csub\u003e2\u003c/sub\u003e\u003csup\u003e2\u003c/sup\u003e = .002. \u003cstrong\u003ed\u003c/strong\u003e, Distribution of two-tailed \u003cem\u003eP\u003c/em\u003e values of all 1,225 brain-behaviour correlations computed after resampling at full sample size (n = 1,000). The distribution is inflated around 0 due to the wider tails in our null distribution. \u003cstrong\u003ee\u003c/strong\u003e, To help explain the widened distribution, we track the largest correlation observed in our original null-sample (r = 0.11), plotting the distribution of corresponding brain-behaviour correlations across the 100 iterations of resampling at the full sample size (n = 1,000). The solid black line represents a Gaussian with mean µ = 0.11 and variance σ\u003csub\u003e2\u003c/sub\u003e\u003csup\u003e2\u003c/sup\u003e = .001. The interaction of variability across iterations (\u003cstrong\u003ee\u003c/strong\u003e) and variability in the discovery sample (\u003cstrong\u003ea\u003c/strong\u003e) results in the widened distribution (\u003cstrong\u003ec\u003c/strong\u003e) by additive variance.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/94c00d70e84821ebf830ce5b.png"},{"id":58146348,"identity":"4ba169b1-d3d2-416c-b6db-3260c53a9b11","added_by":"auto","created_at":"2024-06-11 18:35:31","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":237850,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eBias in estimated statistical power depends on true statistical power\u003c/strong\u003e. \u003cstrong\u003ea\u003c/strong\u003e, Simulated discovery samples are represented for each of the underlying power scenarios. For each scenario, grey violin plots show the distribution of 54,778 random effects and red violin plots represent the distribution of 500 true effects. The dashed line represents the critical Pearson \u003cem\u003er\u003c/em\u003e for a Bonferroni corrected significance level (α = 0.05/55278). \u003cstrong\u003eb\u003c/strong\u003e, We estimated power across sample size by simulating the resampling methods in Marek, Tervo-Clemmens \u003cem\u003eet al.\u003c/em\u003e\u003csup\u003e1\u003c/sup\u003e using a Bonferroni corrected significance threshold. Line colour represents the ground truth statistical power of the study at full sample size (n = 1,000, α = 0.05/55278). \u003cstrong\u003ec\u003c/strong\u003e, To demonstrate bias across different sample sizes, we subtracted analytical power curves from the estimated power (panel b), with lines coloured as in panel b. Note that underpowered discovery samples inflate statistical power estimates near the full sample size.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/94855fa17b2046e0cc6aa72a.png"},{"id":77052671,"identity":"7631015c-4db8-4616-8907-daca7b4ef039","added_by":"auto","created_at":"2025-02-24 16:22:48","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2031551,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/5c10dc7b-155b-431e-bbcc-1b38e6ac8d89.pdf"},{"id":58146347,"identity":"8dddb796-1270-416a-8859-6f83c707aa08","added_by":"auto","created_at":"2024-06-11 18:35:31","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":521487,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementarydraft.docx","url":"https://assets-eu.researchsquare.com/files/rs-4457116/v1/69def97c11f6f03f8ae4a3ac.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies.","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe question of scientific reliability of brain-wide association studies (BWAS) was brought to the attention of many\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e,\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e by Marek, Tervo-Clemmens \u003cem\u003eet al.\u003c/em\u003e\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, reigniting discussions\u003csup\u003e\u003cspan additionalcitationids=\"CR5 CR6\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e about the ongoing reproducibility crisis in neuroscience and psychology\u003csup\u003e\u003cspan additionalcitationids=\"CR9 CR10 CR11\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u003c/sup\u003e. Independent researchers are failing to reproduce (same results using the same methods and data) and replicate (similar results using the same methods and new data) many published findings. In the field of neuroimaging, reproducibility issues are further exacerbated by variability in the methods used by researchers, which can lead to conflicting results\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. Our trust in the scientific field therefore relies on how well we can estimate its reproducibility.\u003c/p\u003e \u003cp\u003eFor any given study design, a reliable way of increasing the likelihood of replication is to recruit more subjects, which will reduce the sampling variability and in turn increase statistical power\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. For a brain-wide association study (BWAS) which aims to characterise associations between brain measures and behaviours, collecting data is expensive. So how many subjects are required? How do we know? Thousands are required\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e,\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e, according to data-driven approaches which quantify the issue of reproducibility for BWAS using large neuroimaging datasets from the Human Connectome Project\u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e (HCP with n\u0026thinsp;=\u0026thinsp;1,200), the Adolescent Brain Cognitive Development study\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e (ABCD with n\u0026thinsp;=\u0026thinsp;11,874), and the UK Biobank\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e (UKB with n\u0026thinsp;=\u0026thinsp;35,735). Among numerous analyses in their study, Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e estimated statistical errors of univariate BWAS as a function of sample size. Such univariate BWAS often involve tens of thousands of correlations between a brain measure and a behavioural measure, most of which fail to replicate even with thousands of participants. These replication failures can be explained by statistical errors of a study design such as false positive rates\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e and low statistical power\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e,\u003cspan additionalcitationids=\"CR19 CR20 CR21\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u003c/sup\u003e. To estimate statistical errors in univariate BWAS, Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e treated a large discovery dataset as a population and then drew replication samples by resampling with replacement (henceforth resampling) from that population. Here, we report the first test of validity of this data-driven approach by using simulated data as ground truth.\u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eResampling methods strongly bias statistical error estimates when there are no true effects\u003c/h2\u003e \u003cp\u003eFirst, we simulated a discovery null sample with \u003cem\u003en\u0026thinsp;=\u003c/em\u003e\u0026thinsp;1,000 subjects each with 1,225 brain connectivity measures (random Pearson correlations) and a single behavioural measure (normally distributed across participants). We correlated each brain connectivity measure with the behaviour across all subjects to obtain 1,225 brain-behaviour correlations. Since brain connectivity estimates and behavioural factors were simulated independently from each other, any resulting brain-behaviour correlations were entirely random. Data dimensions were chosen to be computationally feasible for reproducibility, however we invite readers to adjust these and re-run analyses using the openly available code (analyses recoded in R with supporting packages\u003csup\u003e\u003cspan additionalcitationids=\"CR24\" citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e for open-source accessibility \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/charlesdgburns/rwr/\u003c/span\u003e\u003cspan address=\"https://github.com/charlesdgburns/rwr/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). We then resampled our null-sample for 100 iterations across logarithmically spaced sample size bins (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;25, to 1,000) and estimated statistical errors, following the methods described in Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.1. Surprisingly, we saw the same trends of statistical errors and reproducibility as those reported by Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.1 but with random data (see Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), with strongly biased statistical power estimates.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThese trends in statistical errors thus do not depend on absolute sample size, but the resample size relative to the full sample size. By repeatedly generating new null-samples, rather than resampling from a single null-sample, we verified that these statistical error estimates are indeed biased under the null as the resample size approaches the full sample size (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). For example, uncorrected (α\u0026thinsp;=\u0026thinsp;.05), statistical power was estimated to be 63% when resampling at the full sample size (n\u0026thinsp;=\u0026thinsp;1,000, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. d), rather than the expected 5% obtained when generating new null- samples (n\u0026thinsp;=\u0026thinsp;1,000, Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. d). One concern is that power is the most inflated while also being the most relevant for failed replications\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e,\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e,\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e, which could potentially result in misleading meta-science.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eCompounding sampling variability underlies biased statistical errors under the null\u003c/h2\u003e \u003cp\u003eTo explain why biases arise under the null, we investigated the underlying brain-behaviour correlations used in the calculation of statistical errors. Here we focused on resampling at the full sample size (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1,000) where these biases are most dramatic. As indicated by the false positive rate (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003ef.), the null distribution of brain-behaviour correlations is not preserved when resampling at the full sample size (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Instead, resampling subjects and computing correlations again results in a distribution wider than expected (comparing Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ea. and \u003cb\u003ec\u003c/b\u003e.). This is because resampling involves two sources of sampling variability, first at the level of the discovery sample and again for the resampled replication sample. For instance, if a correlation in the discovery sample is randomly observed to be r\u0026thinsp;=\u0026thinsp;0.11, then resampling participants and computing the same correlation again results in a correlation which varies around r\u0026thinsp;=\u0026thinsp;0.11 (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ee.).\u003c/p\u003e \u003cp\u003eWe can formalise this mathematically as nested distributions\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e, or a convolution\u003csup\u003e\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e of two probability distributions, here approximating Pearson null distributions with normal distributions for analytical simplicity. It then follows that given a discovery sample \u003cem\u003eX\u0026thinsp;~\u0026thinsp;N(\u0026micro;, σ\u003c/em\u003e\u003csub\u003e\u003cem\u003e1\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e), for each observation in our original sample, \u003cem\u003ex\u003c/em\u003e\u003csub\u003e\u003cem\u003ei\u003c/em\u003e\u003c/sub\u003e \u0026isin; \u003cem\u003eX\u003c/em\u003e, resampling participants and recomputing correlations corresponds to sampling from several distributions \u003cem\u003eX\u003c/em\u003e\u003csub\u003e\u003cem\u003ei\u003c/em\u003e\u003c/sub\u003e ~ \u003cem\u003eN\u003c/em\u003e(\u003cem\u003ex\u003c/em\u003e\u003csub\u003e\u003cem\u003ei\u003c/em\u003e\u003c/sub\u003e, \u003cem\u003eσ\u003c/em\u003e\u003csub\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e), resulting in a final set of correlations distributed according to \u003cem\u003eX* ~ N(\u0026micro;, σ\u003c/em\u003e\u003csub\u003e\u003cem\u003e1\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e\u0026thinsp;\u003cem\u003e+\u0026thinsp;σ\u003c/em\u003e\u003csub\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e\u003cem\u003e).\u003c/em\u003e Note that \u003cem\u003eσ\u003c/em\u003e\u003csub\u003e\u003cem\u003e1\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e depends on the size of the discovery sample, while \u003cem\u003eσ\u003c/em\u003e\u003csub\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sub\u003e\u003csup\u003e\u003cem\u003e2\u003c/em\u003e\u003c/sup\u003e is determined by the resample size.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe influence on statistical error estimates such as statistical power is two-fold. First, random correlations in the tail of a discovery sample are more likely to be in the tail of correlations in a resampled replication sample. This inflates power when estimated as the proportion of significant effects in the discovery sample which are significant again in the resampled replication sample (1 \u0026ndash; false negative rates). Second, increased sampling variability alone leads to a wider-than-expected distribution of correlations with more extreme tails. These more extreme tails lead to an inflation of \u003cem\u003eP\u003c/em\u003e values close to 0 in our resample (compare Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003eb. and \u003cb\u003ed\u003c/b\u003e.) when calculated using a standard correlation function (e.g., \u0026lsquo;corr\u0026rsquo; in MATLAB). We note that simply correcting for this widened null distribution will over-correct for bias in statistical error estimates when true effects are present (see Supplementary Information).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eBias in ground truth simulations depends on statistical power of the full sample size\u003c/h2\u003e \u003cp\u003eWhile we have shown clear biases when there are no true effects, this does not directly imply biases when true effects are present. We note that Marek, Tervo-Clemmens \u003cem\u003eet al.\u003c/em\u003e\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e have already shown that the largest univariate effect is highly replicable even for moderate sample sizes, so there are at least some true BWAS effects in the real world. However, as the average true effect size remains unknown, we systematically simulated a range of discovery samples, each representing a study where the size of the underlying true effects corresponded with different levels of statistical power. We note that since the bias under the null is driven by the false rejection of null hypotheses, here we adopt a fixed significance threshold after Bonferroni correction which controls for at least one false positive (family-wise error rate). Focusing on statistical power estimates, we show that the bias near the full sample size depends on the true statistical power of the discovery sample (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e). Power estimates are inflated if the discovery sample is underpowered, but on the other hand a highly powered discovery sample may give conservative power estimates. Note that regardless of power at the full sample size, bias in statistical power is largely avoided when subsampling up to around 10% of the full sample size (see also Supplementary Information for subject-level simulations).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cp\u003eAccurately estimating reproducibility of scientific methods is critical for guiding researcher\u0026rsquo;s methodological decisions. Our results demonstrate that estimating statistical errors by resampling with replacement from random data results in large biases when resampling near the full sample size. We explain this fully by compounding sampling variability of test statistics when resampling and its knock-on effects on estimated statistical errors. We further simulate ground truth data with true effects to show that statistical power is inflated when the true power of the discovery sample is low and slightly deflated when true power is high. This could lead to circular reasoning in cases where we must assume we have high statistical power before we can rely on the estimation that we have high statistical power. Lastly, we show that this bias is largely avoided when subsampling only up to 10% of the full sample size after Bonferroni correction. This 10% rule of thumb is consistent with the use of resampling techniques in a recent evaluation of statistical power and false discovery rates for genome-wide association studies with hundreds-of-thousands of participants\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e, as well as recommendations for 10-fold cross-validation to reduce prediction error in machine-learning\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eWhat are the implications for the results presented by Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.1? For the strictly denoised Adolescent Brain Cognitive Development (ABCD) sample (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;3,928), they report around 68% power at \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;3,928 after Bonferroni correction when resampling at the full sample size (Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.1 Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003ed.). Our true effect simulation results indicate that this estimate could be inflated from a true average power anywhere between 1% and 40%. Furthermore, when subsampling from the UK Biobank with a full sample size of \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;32,572 Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e.1 report around 1% power for \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;4,000 and \u003cem\u003eα\u003c/em\u003e\u0026thinsp;=\u0026thinsp;10\u0026thinsp;\u003cem\u003e\u0026minus;\u003c/em\u003e\u0026thinsp;7. We therefore argue that the 68% power reported for the full ABCD sample (\u003cem\u003en\u0026thinsp;=\u003c/em\u003e\u0026thinsp;3,928, \u003cem\u003eα\u003c/em\u003e\u0026thinsp;=\u0026thinsp;10\u0026thinsp;\u003cem\u003e\u0026minus;\u003c/em\u003e\u0026thinsp;7) more likely reflects methodological bias, rather than a result of increased signal after strict denoising of brain data. While the largest BWAS effects may be highly reproducible with 4,000 participants, the average univariate BWAS effect is most likely not reproducible. On the other hand, our true effect simulations (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e.) also indicate that the UK Biobank estimates at the full sample size are more reliable, with an underlying power likely between 70% and 90% at \u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;32,572 after Bonferroni correction. Ultimately, our results suggest that replicating the univariate BWAS tested in Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e. requires tens-of-thousands of individuals.\u003c/p\u003e \u003cp\u003eOur results only have direct implications for mass univariate association studies, however it is worth noting how methodological decisions could influence reproducibility in neuroimaging. For example, it should be noted that inter-individual correlation studies offer \u0026ldquo;as little as 5%-10% of the power\u0026rdquo; of within-subject t-test studies with the same number of participants4. Other methodological choices, such as data modelling, should also be carefully considered. The lack of power in univariate BWAS considered by Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e. could also be influenced by the choice of a group-averaged brain parcellation\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e, which fails to account for individual level variations in resting state functional connectivity\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e,\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e. Brain models\u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e which do account for such individual variability generalise better, as demonstrated by stronger out-of-sample prediction\u003csup\u003e\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e,\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e, and could also lead to higher replication rates in null-hypothesis significance tests. Note also that how we model null distributions\u003csup\u003e\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e\u003c/sup\u003e of brain-wide statistics has a large influence on resulting \u003cem\u003eP\u003c/em\u003e values. With this in mind, one could consider a predictive framework rather than an explanatory one\u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e, which could be replicable with only hundreds of participants\u003csup\u003e\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e,\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eIt is clear that investigations of reproducibility of wider BWAS methods are required. We urge such meta-analyses to evaluate their meta-analytic methods, for example with null data, so they may reliably evaluate the reproducibility of scientific methods used in research.\u003c/p\u003e"},{"header":"Methods","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eSimulating null data at subject level\u003c/h2\u003e \u003cp\u003eWe simulated random phenotype associations with simulated functional connectivity measures. We generated a null-sample with n\u0026thinsp;=\u0026thinsp;1,000 subjects each with 1,225 edges (random Pearson correlations between 50 random time series) and a single behavioural factor (normally distributed across participants). We correlated each edge with the behaviour across all subjects to obtain 1,225 brain-behaviour correlations. By generating edge connectivity estimates and behavioural factors independently from each other, we ensured that any resulting brain-behaviour correlations are entirely random (ρ\u0026thinsp;=\u0026thinsp;0), hence obtaining a sample where the null hypothesis is true (i.e., a null-sample).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eEstimating statistical errors\u003c/h2\u003e \u003cp\u003eWe closely followed the methods of Marek, Tervo-Clemmens \u003cem\u003eet al.\u003c/em\u003e\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, first running analyses on MATLAB using their code 'abcd_edgewise_correlation_iterative_reliability_single_factor.m' and 'abcd_statisticalerrors.m' (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://gitlab.com/DosenbachGreene/bwas\u003c/span\u003e\u003cspan address=\"https://gitlab.com/DosenbachGreene/bwas\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e). These analyses were then independently recoded in R with supporting packages\u003csup\u003e\u003cspan additionalcitationids=\"CR24\" citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e for open-source accessibility \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/charlesdgburns/rwr/\u003c/span\u003e\u003cspan address=\"https://github.com/charlesdgburns/rwr/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Notably, statistical error estimations involve two-tailed \u003cem\u003eP\u003c/em\u003e values derived from parametric null distributions on a given resample size.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eSimulating ground truth data with known statistical power\u003c/h2\u003e \u003cp\u003eAt this stage we take a computationally more efficient approach and simulate summary statistics rather than subject-level data, which allows us to simulate many more true effect scenarios so we can compare estimates with true statistical errors. This approach also lets us increase the number of effects, so we now simulate samples with 55,278 (333 choose 2) effects, the number of resting-state functional connectivity measures which feature in Marek, Tervo-Clemmens \u003cem\u003eet al\u003c/em\u003e. Figure\u0026nbsp;2\u003csup\u003e3\u003c/sup\u003e. The size of true effects was determined by an inverse power analysis\u003csup\u003e\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e with a fixed sample size (n\u0026thinsp;=\u0026thinsp;1,000) and Bonferroni corrected significance threshold (α\u0026thinsp;=\u0026thinsp;0.05/55278), which involved a Fisher z-transformation for calculating the critical Pearson r for a given power level (power\u0026thinsp;=\u0026thinsp;1%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 99%). The Fisher z-transformation\u003csup\u003e\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e\u003c/sup\u003e, \u003cem\u003eF\u003c/em\u003e(r)\u0026thinsp;=\u0026thinsp;atanh(r)\u0026thinsp;=\u0026thinsp;z was also used to sample Pearson correlations using the approximation that the z-statistic is asymptotically normally distributed with mean \u003cem\u003eF\u003c/em\u003e(r) and standard deviation \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(1/\\sqrt{\\left(n-3\\right)}\\)\u003c/span\u003e\u003c/span\u003e. The z-statistic was then transformed into Pearson correlations to simulate brain-behaviour correlations. For each power level, we simulated a discovery sample by first drawing 55,278 random effects (ρ\u0026thinsp;=\u0026thinsp;0) and afterwards replacing 500 of those with true effects, where ρ corresponds to the critical r for a given power level computed earlier. The choice of 500 true effects was somewhat arbitrary, being moderate enough to seem probable but also sufficiently many true effects to reduce noise in bias estimates. Note that while real world effect sizes of a single BWAS may vary, we instead simulate several effects with the same underlying effect size. This should not be an issue, since the statistical error summary statistics are given as an average across effects, so in this simulation we can think of the underlying effects having an average effect size according to a given power level.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eSimulating statistical power estimations from resampling with replacement\u003c/h2\u003e \u003cp\u003eGiven only summary statistics rather than individual subjects, we cannot resample participants and recompute \u003cem\u003eP\u003c/em\u003e values, but instead also simulate obtaining estimates by resampling with replacement as in subject-level analyses\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. First, we generate ground truth data with known statistical power, which we treat as our discovery sample. Then, we follow the assumption that the observed effects in a discovery sample are the population effects: for a given resample size \u003cem\u003en\u003c/em\u003e, a resampled effect size was drawn from a normal distribution \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(N\\left(\\text{a}\\text{t}\\text{a}\\text{n}\\text{h}\\left({r}^{*}\\right),1/\\sqrt{n-3}\\right)\\)\u003c/span\u003e\u003c/span\u003e, where r* is a given Pearson correlation from the discovery sample, and then Fisher z-transformed. We then derived \u003cem\u003eP\u003c/em\u003e values from an uncorrected null distribution of Pearson correlations with degrees of freedom computed relative to the resample size. We resampled across the same range of sample sizes as in previous analyses (n\u0026thinsp;=\u0026thinsp;25, \u0026hellip; 1,000). We continued to estimate statistical power across 1,000 iterations of resampled brain-behaviour correlations as in Marek, Tervo-Clemmens et al.\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e, specifically as the proportion of significant effects in a discovery sample which were significant again in a resample (1 \u0026ndash; false negative rates, with α\u0026thinsp;=\u0026thinsp;0.05/55278). These were then compared to analytical power curves\u003csup\u003e\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e\u003c/sup\u003e computed using Fisher z-transformations for varying sample sizes and effect sizes corresponding to critical r for power levels at the full sample size (n\u0026thinsp;=\u0026thinsp;1,000) computed earlier.\u003c/p\u003e \u003c/div\u003e "},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eCompeting interests\u003c/h2\u003e \u003cp\u003eNone.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eC.D.G.B.: Conceptualisation, design, implementation, analysis, interpretation, writing - original draft. A.F.: Interpretation of results, writing - review \u0026amp; editing. G.A.R.: Conceptualisation, design, interpretation, writing - review \u0026amp; editing, supervision.\u003c/p\u003e\u003ch2\u003eAcknowledgements\u003c/h2\u003e \u003cp\u003eA.F. is supported by a grant from the Biotechnology and Biology research council (BBSRC, grant number: BB/S006605/1) and the Funda\u0026ccedil;\u0026atilde;o Bial, Funda\u0026ccedil;\u0026atilde;o Bial Grants Programme 2020/21, A- 29315, number 203/2020, grant edition: G-15516.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eR and MATLAB code used for data simulation, statistical analyses, and plotting is available on GitHub: https://github.com/charlesdgburns/rwr/.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eCallaway, E. Can brain scans reveal behaviour? Bombshell study says not yet. Nature 603, 777\u0026ndash;778 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRichtel, M. Brain-Imaging Studies Hampered by Small Data Sets, Study Finds. The New York Times (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarek, S. \u003cem\u003eet al.\u003c/em\u003e Reproducible brain-wide association studies require thousands of individuals. Nature 603, 654\u0026ndash;660 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGratton, C., Nelson, S. M. \u0026amp; Gordon, E. M. Brain-behavior correlations: Two paths toward reliability. Neuron 110, 1446\u0026ndash;1449 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRosenberg, M. D. \u0026amp; Finn, E. S. How to establish robust brain\u0026ndash;behavior relationships without thousands of individuals. Nat. Neurosci. 25, 835\u0026ndash;837 (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBotvinik-Nezer, R. \u0026amp; Wager, T. D. Reproducibility in Neuroimaging Analysis: Challenges and Solutions. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 8, 780\u0026ndash;788 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHelwegen, K., Libedinsky, I. \u0026amp; van den Heuvel, M. P. Statistical power in network neuroscience. Trends Cogn. Sci. 27, 282\u0026ndash;301 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eButton, K. S. \u003cem\u003eet al.\u003c/em\u003e Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365\u0026ndash;376 (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMunaf\u0026ograve;, M. R. \u003cem\u003eet al.\u003c/em\u003e A manifesto for reproducible science. Nat. Hum. Behav. 1, 1\u0026ndash;9 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOpen Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIoannidis, J. P. A. Why Most Published Research Findings Are False. PLOS Med. 2, e124 (2005).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F. \u0026amp; Baker, C. I. Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535\u0026ndash;540 (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBotvinik-Nezer, R. \u003cem\u003eet al.\u003c/em\u003e Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84\u0026ndash;88 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, S., Abdellaoui, A., Verweij, K. J. H. \u0026amp; van Wingen, G. A. Replicable brain\u0026ndash;phenotype associations require large-scale neuroimaging data. Nat. Hum. Behav. 7, 1344\u0026ndash;1356 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVan Essen, D. C. \u003cem\u003eet al.\u003c/em\u003e The WU-Minn Human Connectome Project: An overview. NeuroImage 80, 62\u0026ndash;79 (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCasey, B. J. \u003cem\u003eet al.\u003c/em\u003e The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43\u0026ndash;54 (2018).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSudlow, C. \u003cem\u003eet al.\u003c/em\u003e UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 12, e1001779 (2015).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIngre, M. Why small low-powered studies are worse than large high-powered studies and how to protect against \u0026ldquo;trivial\u0026rdquo; findings in research: Comment on Friston (2012). \u003cem\u003eNeuroImage\u003c/em\u003e 81, 496\u0026ndash;498 (2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYarkoni, T. Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power\u0026mdash;Commentary on Vul et al. (2009). \u003cem\u003ePerspect. Psychol. Sci.\u003c/em\u003e 4, 294\u0026ndash;298 (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCremers, H. R., Wager, T. D. \u0026amp; Yarkoni, T. The relation between statistical power and inference in fMRI. PLOS ONE 12, e0184923 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSzucs, D. \u0026amp; Ioannidis, J. PA. Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990\u0026ndash;2012) and of latest practices (2017\u0026ndash;2018) in high-impact journals. NeuroImage 221, 117164 (2020).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePoldrack, R. A. \u003cem\u003eet al.\u003c/em\u003e Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat. Rev. Neurosci. 18, 115\u0026ndash;126 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRipley, B. \u003cem\u003eet al.\u003c/em\u003e MASS: Support Functions and Datasets for Venables and Ripley\u0026rsquo;s MASS. (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWickham, H. \u003cem\u003eet al.\u003c/em\u003e Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKassambara, A. ggpubr: \u0026lsquo;ggplot2\u0026rsquo; Based Publication Ready Plots. (2022).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEl Otmani, S. \u0026amp; Maul, A. Probability distributions arising from nested Gaussians. Comptes Rendus Math. 347, 201\u0026ndash;204 (2009).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eConvolution of Gaussians is Gaussian. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://jeremy9959.net/Math-5800-Spring-2020/notebooks/convolution_of_gaussians.html\u003c/span\u003e\u003cspan address=\"https://jeremy9959.net/Math-5800-Spring-2020/notebooks/convolution_of_gaussians.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, Z., Boehnke, M., Wen, X. \u0026amp; Mukherjee, B. Revisiting the genome-wide significance threshold for common variant GWAS. \u003cem\u003eG3 GenesGenomesGenetics\u003c/em\u003e 11, jkaa056 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWitten, I. H., Frank, E., Hall, M. A., Pal, C. J. \u0026amp; DATA, M. Practical machine learning tools and techniques. Data Min. Fourth Ed. Elsevier Publ. (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGordon, E. M. \u003cem\u003eet al.\u003c/em\u003e Generation and Evaluation of a Cortical Area Parcellation from Resting-State Correlations. Cereb. Cortex 26, 288\u0026ndash;303 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKong, R. \u003cem\u003eet al.\u003c/em\u003e Individual-Specific Areal-Level Parcellations Improve Functional Connectivity Prediction of Behavior. Cereb. Cortex 31, 4477\u0026ndash;4500 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGordon, E. M. \u003cem\u003eet al.\u003c/em\u003e Precision Functional Mapping of Individual Human Brains. Neuron 95, 791\u0026ndash;807.e7 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBijsterbosch, J. D., Valk, S. L., Wang, D. \u0026amp; Glasser, M. F. Recent developments in representations of the connectome. NeuroImage 243, 118533 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarahibozorg, S.-R. \u003cem\u003eet al.\u003c/em\u003e Hierarchical modelling of functional brain networks in population and individuals from big fMRI data. NeuroImage 243, 118513 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarkello, R. D. \u0026amp; Misic, B. Comparing spatial null models for brain maps. NeuroImage 236, 118052 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYarkoni, T. \u0026amp; Westfall, J. Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspect. Psychol. Sci. 12, 1100\u0026ndash;1122 (2017).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpisak, T., Bingel, U. \u0026amp; Wager, T. D. Multivariate BWAS can be replicable with moderate sample sizes. Nature 615, E4\u0026ndash;E7 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen, J. \u003cem\u003eet al.\u003c/em\u003e Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study. NeuroImage 274, 120115 (2023).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u003cem\u003eDesigning Clinical Research\u003c/em\u003e. (Wolters Kluwer/Lippincott Williams \u0026amp; Wilkins, Philadelphia, 2013).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFisher, R. A. Frequency Distribution of the Values of the Correlation Coefficient in Samples from an Indefinitely Large Population. Biometrika 10, 507\u0026ndash;521 (1915).\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4457116/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4457116/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recent studies have leveraged consortium neuroimaging data to answer an important question: how many subjects are required for reproducible brain-wide association studies? These data-driven approaches could be considered a framework for testing the reproducibility of several neuroimaging models and measures. Here we test part of this framework, namely estimates of statistical errors of univariate brain-behaviour associations obtained from resampling large datasets with replacement. We demonstrate that reported estimates of statistical errors are largely a consequence of bias introduced by random effects when sampling with replacement close to the full sample size. We show that future meta-analyses can largely avoid these biases by only resampling up to 10% of the full sample size. We discuss implications that reproducing mass-univariate association studies requires tens-of-thousands of participants, urging researchers to adopt other methodological approaches.","manuscriptTitle":"Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies.","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-11 18:35:26","doi":"10.21203/rs.3.rs-4457116/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-07-22T04:29:47+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-07-19T16:31:19+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-07-10T23:13:39+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"332308589551983945790649099324155405711","date":"2024-06-14T14:10:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"3758464779003275205307396338313969870","date":"2024-06-13T19:45:37+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-08T20:21:58+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-06-04T14:59:54+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-05-29T17:57:12+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-05-27T04:32:11+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-05-21T21:31:50+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"97e8c857-9141-4781-afde-c1822129eaa3","owner":[],"postedDate":"June 11th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":32868814,"name":"Biological sciences/Neuroscience"},{"id":32868815,"name":"Biological sciences/Computational biology and bioinformatics/Communication and replication"},{"id":32868816,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":32868817,"name":"Biological sciences/Computational biology and bioinformatics/Statistical methods"},{"id":32868818,"name":"Physical sciences/Mathematics and computing/Statistics"}],"tags":[],"updatedAt":"2025-02-24T16:03:56+00:00","versionOfRecord":{"articleIdentity":"rs-4457116","link":"https://doi.org/10.1038/s41598-025-89257-w","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-02-19 15:57:55","publishedOnDateReadable":"February 19th, 2025"},"versionCreatedAt":"2024-06-11 18:35:26","video":"","vorDoi":"10.1038/s41598-025-89257-w","vorDoiUrl":"https://doi.org/10.1038/s41598-025-89257-w","workflowStages":[]},"version":"v1","identity":"rs-4457116","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4457116","identity":"rs-4457116","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}