Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a table of summary statistics that compares randomised groups at baseline. Researchers who fraudulently generate trials often create randomised groups that are implausibly similar (under-dispersed) or accidentally create large differences between groups (over-dispersed) because they do not understand how to create realistic summary statistics from truly random data. We aimed to automatically screen for under- and over-dispersion in the baseline tables of published randomised trials. Methods: We examined randomised controlled trials published in open access health and medical journals on PubMed Central . We estimated probability that a trial’s summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences from the baseline table, and compared this with an expected distribution without dispersion. We used a simulation study to test the ability of the model to find over- or under-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, as it had a higher probability of detecting over-dispersed data and lower false positive percentage for skewed data that was not under- or over-dispersed. For the trials published on PubMed Central , some tables were flagged because: they were not from trials, had a non-standard presentation, or had likely reporting errors. The algorithm also flagged some trials as under-dispersed where there was a striking similarity between groups. Conclusion: Due to the variance in reporting across journals, the current method may not be suitable for automated screening for fraud of all submitted trials, but could be useful in targeted checks of suspected trials. If used in automated screening of papers at the submission stage, it would likely flag many instances of non-standard table presentation and poor reporting.