Review of samples sizes used in agreement studies published in the PubMed database

Introduction: A sample size justification is required for all studies and should give the minimum number of subjects to be recruited for the study to achieve its primary objective. This aim of this review is to describe sample sizes from agreement studies with continuous or categorical endpoints published within the PubMed database, and determine whether sample size justification was provided. Methods: : Data were gathered from the PubMed database with a time interval of 28 th September 2018 to 28 th September 2020. The search returned 5257 studies of which 82 studies were eligible for final assessment after duplicates and ineligible studies were excluded. Results: : We observed a wide range of sample sizes. Of the 82 studies assessed, only 27 (32.9%) gave justification for their sample size. Forty-six studies (56.1%) used a continuous outcome measure, 28 (34.1%) used categorical and eight (9.8%) used both. Median sample sizes were 50 for continuous endpoints and 119 for categorical endpoints. Bland-Altman limits of agreement were the most common method of statistical analysis for continuous variables and Kappa coefficients for categorical variables. Conclusions: : All studies need a sample size justification. Not all studies need a sample size calculation. Despite the importance of a sample size justification we found that two-thirds of agreement studies did not provide one. We recommend that all agreement studies provide rationale for their sample size even if they do not include a formal sample size calculation.