Randomized clinical trials with run-in periods: frequency, characteristics and reporting.

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

We screened 748 PubMed items and identified 470 randomized clinical trials in order to obtain 25 publications (5%) reporting a run-in period ( Figure 1 ). 9 – 33 The trials with run-in periods were larger than trials without (median number of randomized patients 217 and 90, respectively, P =0.01). All the run-in trials were published in English. Six of the eight trials with non-pharmacological interventions studied dietary or lifestyle interventions ( Tables 1 and 2 ). Also, 16 of 25 trials with a run-in period (64%) were industry trials. This proportion was much higher than among trials without run-in periods (23 out of 82, 28%, P <0.01, the denominator 82 came from disregarding 18 out of 100 trials with unclear industry status; Table 1 ). Extrapolating the latter proportion of 28% from the 82 trials in the sample to all 445 trials without a run-in period, these would includê125 industry trials. Thus, among all 470 randomized clinical trials with and without run-in periods, there would be a total of 141 industry trials (the sum of 16 and 125) and conversely 329 non-industry trials. It follows that an estimated proportion of publications reporting a run-in period was 11% among industry trials (16 out of 141) and 3% among non-industry trials (9 out of 329). In the 13 of 25 run-in period trials reporting relevant data, a median of 16% of enrolled patients were excluded during the run-in period (IQR: 5%-24%). In the 24 of 25 trials reporting relevant data, the median duration of the run-in period was 14 days (IQR: 11–28); for 20 trials, the duration of the run-in period was stated as a fixed number, whereas 4 trials reported varying duration (eg, “7±3 days”). The intervention during the run-in period was, in most cases, no intervention (36%, 9 out of 25) or placebo (28%, 7 out of 25). No trial used the experimental treatment of the randomized phase as the run-in intervention ( Table 3 ). Common purposes of the run-in period were to ensure symptom stability (7 out of 25, 28%), for example, headache frequency over 1 month, and “baseline data collection” (5 out of 25, 2%). Most of the trials (20 out of 25, 80%) used the term “run-in period” (in some variation), whereas 3 trials (12%) used the term “baseline period” and 3 other trials (12%) used “lead-in period”. Two trials (8%) had complete reporting of run-in periods, because there were no exclusions during the run-in period in both cases. In 23 of the trials (92%), the reporting of run-in periods was incomplete according to our definition. The main reason for incomplete reporting was that trials did not report the characteristics of excluded patients (22 out of 25 trials, 88%). Reporting for the two other aspects was also incomplete in many trials: 48% and 72% of trials did not report number of excluded patients and exclusion reasons, respectively ( Table 4 ). Our main results were robust to variations in how we operationally defined “run-in period” and “industry trial” (see Supplementary materials).

Randomized clinical trials are generally considered the most reliable method for evaluating the effects of health care interventions. However, randomized trials vary in design, and different design characteristics may impact on internal validity (risk of bias), external validity (generalizability) and costs (both economical and logistical). One such design characteristic is a run-in period, which is a planned time period from formal patient enrollment to randomization that enables exclusion of certain patients, for example, if they experience harms (see Box 1 ). 1 In trials with run-in periods, randomization may take place days or weeks after formal enrollment. During this post-inclusion pre-randomization period, all patients receive the same treatment, for example, a placebo (“placebo run-in”), the experimental drug (“active run-in”) or observation only (“no-treatment” or “washout” run-in). 1 , 2 The main rationale for a run-in period in a trial is to adjust the selection of patients for the post-randomization phase of the trial. The principal difference between standard screening by eligibility criteria in a trial and the procedures in a run-in phase is that the latter permits exclusions based on observations of patients’ compliance or responses to trial interventions. Thus, for example, an active run-in period enables exclusion of patients who respond poorly to the experimental intervention. A placebo run-in period enables exclusion of patients who respond well to the placebo intervention. Both active and placebo run-in periods enable exclusion of patients who do not comply with trial procedures. 1 , 3 The number of exclusions in run-in periods may be considerable. For example, in a trial of the effect of extended-release opioid for lower back pain, 191 out of 459 enrolled patients (42%) were excluded during the active run-in period. 4 In another trial of the effect of aspirin and β-carotene on cardiovascular disease and cancer, 11,152 out of 33,223 patients (34%) were excluded during the run-in period. 5 Run-in periods can affect the validity of a study’s results. When patients are excluded during a run-in period, for example, due to harms or lack of response, the trial population may increasingly differ from the typical clinical patient population. The balance between benefits and harms of the trial intervention may appear more beneficial than if exclusions had not taken place. Thus, interpretation of results from a trial with a run-in period is challenging as it involves careful consideration of how pre-randomization exclusions of patients could have affected trial validity. Such considerations presuppose access to relevant information on patient exclusions, typically through adequate reporting of the run-in period in the trial publication. We, therefore, thought it relevant to study run-in periods in randomized clinical trials. Our objectives were to 1) assess the proportion of trial publications that report a run-in period; 2) characterize such trials and the types of run-in periods and 3) evaluate the completeness of reporting.

The frequency of randomized clinical trials with run-in periods was, on average, ~5%, but three times as frequent in industry trials as compared to non-industry trials. The run-in procedures differed in design and purpose, but a median of 16% of the included patients were excluded during the run-in periods of a median of 14 days. In approximately nine out of ten trial publications, the reporting on the run-in period was too incomplete for a meaningful assessment of its potential impact on the trial results. We suggest that updates of reporting guidelines for randomized trials address run-in periods. We propose the minimum information needed for complete reporting, and we recommend that results from trials with run-in periods are interpreted cautiously with respect to both internal and external validity.

In a representative sample of randomized clinical trial publications, ~1 in 20 reported a run-in period, though in industry trials, the proportion was higher (1 in 10) than in non-industry trials (1 in 30). Trials with run-in periods were typically large industry trials with a placebo control group. A median of 16% of included patients were excluded during run-in periods of a median of 14 days. The run-in procedures differed in design and purpose, but in approximately nine of ten trial publications, the reporting on the run-in period was too incomplete for a meaningful assessment of its potential impact on the trial result. To our knowledge, this is the first study of the characteristics and reporting of run-in periods in a random sample of randomized trials. It was based on contemporary trial publications indexed in PubMed, publications that a clinician may typically access. Our sample size of 25 trials was chosen to provide an overview of the typical trials with a run-in period. A considerably larger sample size would have been required for a comprehensive overview of possible subgroup characteristics. Some clinical fields were not covered by our sample. For example, we screened ~30 psychiatry trials, but none of these reported a run-in period, even though run-in periods are often believed to occur frequently in psychiatry trials. A previous review has studied the impact of the placebo run-in period on the efficacy of antidepressants. Sixty-seven of the 141 trials included (48%) used a placebo run-in period. 34 Our sample size would also be too small for the detection of small or modest differences between trials with and without run-in periods. We did, however, detect a clear difference between trials with and without a run-in period with respect to industry status. Our study addressed reporting in trial publications, and not the frequency of conducted but unreported run-in periods. Twelve of the 25 trials from our study were reported in multiple publications, and in 2 cases, we found examples of unreported run-in periods. We have not investigated whether run-in periods are reported in more detail in other formats than trial publications, for example, protocols, trial registers, study reports or regulatory agency documents. We chose to focus on trial publications as this is by far the most accessible and accessed format for communicating trial findings. Two previous reviews have addressed run-in periods in trials of patients with chronic pain. 35 , 36 The reviews analyzed randomized clinical trials using “enriched enrollment”, a variant of the active run-in-period where patients were randomized if they tolerated and responded to active treatment. The first review described characteristics of the trials and enrichment, including discontinuation rate. In the eight included trials, the average discontinuation rate during the enrichment phase was 35% (compared to our median of 16%). 35 The second review identified many of the same trials as the first one. 36 Our study sample did not include active run-in periods that may cause even more frequent patient exclusions due to harms and lack of response. In our study, we were not able to obtain the characteristics of the excluded patients during the run-in period for any of the 25 trials. We are not aware of reviews of run-in trials that have compared excluded and randomized patients, but publications on single trials have been published. These indicate that the characteristics of excluded patients and randomized patients may be similar or quite different, depending on the study. 37 – 41 It would be relevant for the clinician to inspect the characteristics closely in order to relate the trial population to their own patients. We did not investigate the impact of run-in periods on post-randomization attrition rates, partly because attrition was reported poorly. However, in a previous review for depression, trials with run-in periods did not seem to lower the attrition rates. 42 Reviews of interventions for depression, weight loss and chronic pain trials reported that run-in periods also did not seem not to alter the effect sizes. 34 , 42 – 45 These empirical results are somewhat at odds with the theoretical reasoning behind using run-in periods. Possible explanations for the unexpected results could be unusual clinical settings, low run-in exclusion rates and low statistical power in the trials or in the reviews in question. Further adequately powered empirical review studies would be interesting. Approximately 20,000 new randomized clinical trial publications are listed each year in PubMed, so we estimate that about 1,000 trial publications yearly report a run-in period (see Supplementary materials). The impact of such trials is larger than reflected solely by their number because they tend to be comparatively larger industry trials which inform decisions made by physicians and regulatory authorities more often than smaller non-industry trials. From the perspective of trial logistics, a run-in period is used to make a trial more statistically efficient, that is, better at detecting a presumed effect of an intervention. Assuming a moderately effective intervention, a trial with a run-in phase will need fewer patients to reach a statistically significant result, if 1) patient attrition is reduced, 2) non-adherence to experimental intervention is reduced, 3) missed appointments and resulting lack of data is reduced and 4) fewer patients participate who respond well to placebo or poorly to the experimental interventions. Similarly, a run-in phase will fit a trial with the restricted objective to evaluate the effect of an intervention under ideal conditions, that is, an “explanatory” or a “proof-of-concept” trial assessing “efficacy”, and not a “pragmatic” trial assessing clinical “effectiveness” under conditions close to the expected standard clinical situation. 46 – 48 Thus, a run-in period will tend to improve the sensitivity of the instrument used to detect a treatment effect in compliant patients under a nonstandard clinical situation. However, from the perspective of users of information derived from clinical trials – patients, physicians, authors of systematic reviews and clinical guidelines, and policymakers – the most relevant information are the estimates of the treatment effect sizes applicable to the clinically relevant patient population. A run-in period resulting in pre-randomization exclusion of patients may, if the exclusions are not clearly reported, generally not facilitate such information. Some proponents argue that active run-in periods can actually imitate the clinical practice of closely monitoring patients when they start a new therapy, 49 and that the relevant effect estimate is the one deriving from compliant patients experiencing minimal harms. However, this is a problematic comparison for three reasons. First, the typical clinical monitoring of patients starting a new therapy is often fairly informal and will often differ considerably from the stricter monitoring in a clinical trial. Second, most clinicians cannot reliably predict or detect noncompliant patients. 1 Third, the risk of harmful effects or anticipated intention-to-treat effect is relevant for those patients who start on the drug. In other words, if patients are excluded in a trial with a run-in period due to noncompliance, a clinician will have considerable difficulty in identifying and treating the patient group for whom a treatment effect has been shown. The problem with applicability also applies with regards to harms of the intervention. In active or placebo run-in periods, the occurrence of harms may be difficult to interpret when no comparison arm is present, and the exclusion of patients who experience harms may underestimate their clinically relevant occurrence. 45 , 50 , 51 The more efficient a run-in phase is for excluding a specific category of patients, the less directly clinically applicable the trial result may tend to be. A run-in period may also impact directly on the internal validity of a trial. Half of the patients in trials with an active or placebo run-in period change study intervention at randomization, either from placebo to experimental or vice versa. This enables them to directly compare experimental and control interventions and increases the risk of bias due to unblinding. 52 In our sample, this problem was relevant in 6 out of 25 trials (24%). Furthermore, selection of highly compliant patients who tolerate treatment well may result in a lower post-randomization attrition rate and lower loss of outcome data. This may inflate the estimated effect of assignment of treatment, the intention-to-treat effect. 53 In 7 of 25 trials (12%) in our sample, patient compliance was one of the purposes of the run-in design. A run-in period may thus impact on both the external and internal validity of a trial. Furthermore, the run-in period may be considered as an example of “bias by design”. Bias is usually understood as synonymous with internal validity, but “bias by design” is a broader concept incorporating aspects of external validity, and refers to design features which increase the chance of detecting an effect at the cost of clinical applicability. Other potential examples of bias by design are selection of an inadequate comparator, 54 – 57 short trial duration, 54 , 57 , 58 selection of clinically irrelevant outcome measures 54 , 59 and narrow inclusion criteria. 54 , 60 , 61 Bias by design has been suggested as one possible mechanism for why industry trials tend to have more favorable conclusions and outcomes than non-industry trials. 62 Disagreement may exist as to if and when a run-in period affects external and internal validity. However, the proper assessment of the impact of a run-in period on both internal and external validity of a trial relies substantially on adequate reporting. We document that reporting of run-in periods in trial publications is generally inadequate, in line with similar findings for randomization, blinding and attrition. 63 – 66 Ideally, a reader of a trial publication wants to be able to apply a reliable trial result to an identifiable group of patients. Thus, it is important that the exclusion process in a run-in period is transparent and that any excluded patients are described in sufficient detail. At present, this is far from the case in the vast majority of trials. One suggestion for improvement is to include reporting of run-in periods in the next revision of the CONSORT (Consolidated Standards of Reporting Trials) statement. The present version CONSORT 2010 on the reporting of trial publications does not offer advice on how to report the use of run-in periods. 67 , 68 We suggest that trials with a run-in period could report this in an adjusted CONSORT flow diagram (eg, interposed between screening and randomization in the current CONSORT flow diagram) and include information on the number of excluded patients and reasons for exclusions. We also suggest that trial publications report the baseline characteristics of excluded patients (eg, in a table). The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2013 statement on reporting trial protocols briefly states that a protocol should report run-in periods when describing intervention dosing schedules and in the participant timeline. 69 , 70 One option is to expand that section for the revised version of the statement. While awaiting reporting guideline updates and improved reporting of run-in periods in trial publications, we suggest that results from trials with run-in periods are always interpreted cautiously with respect to external validity and, in many cases, also with respect to internal validity.

This was an observational study of a random sample of PubMed-indexed trial publications. We identified publications indexed in PubMed as “randomized controlled trial” and published in 2014, and listed them in a random order using random.org . 6 One reference at a time, one author (DL) screened titles and abstracts (and full text if needed) to check if the publication reported a randomized clinical trial. If so, the same author read the full text of the trial publication and determined whether the trial used a run-in period according to our definition below. We continued the process until 25 trials with a run-in period were identified. Randomized clinical trials were included if they had a parallel, crossover or split-body design (but excluded if they used cluster randomization). We considered trials to be “clinical” if they assessed the benefits or harms of a health care intervention. We planned to include trial publications written in all languages, using Google Translate as an aid. 7 We operationally defined a “run-in period” as fulfilling either a main or a secondary criterion. The main criterion was that a trial publication had to 1) use one of the following terms: “run-in”, “lead-in”, “enrichment”, “single-blind placebo” or “baseline”, indicating a time period after (explicitly reported) registration of trial patients, but before their randomization and 2) explicitly report possible exclusion of patients during this period due to non-response to experimental intervention, harms, response to control intervention, noncompliance to experimental or control intervention or noncompliance to data collection. The secondary criterion was that a trial publication had to 1) explicitly use the following terms: “run-in”, “lead-in” or “enrichment”, indicating a time period after (explicitly or implicitly reported) registration of trial patients, but before their randomization and 2) report no indications that investigator-driven patient exclusions due to response to, or compliance with, treatments were disallowed. The distinction between trials qualified as run-in trials according to the main and secondary criteria was used in sensitivity analyses (see Supplementary materials). If the trial publication referred to previous trial publications (eg, published protocols), we included information from these to determine whether a run-in period was reported or not. Thus, “trial publication” in this study means the index publication identified in our sample and previous journal publications on the same trial cited in the index publication. From the sample of 25 randomized clinical trials with a run-in period as well as a random selection of 100 trials without a run-in period, one author (DL) extracted descriptive data on publication and trial: language, clinical specialty (medical, surgery, others), trial design (eg, parallel, crossover), number of intervention arms, types of control interventions (eg, active or placebo), treatment class (pharmacological or non-pharmacological), number of patients randomized and industry involvement. We operationally defined an “industry trial” as a trial in which a commercial company (eg, a drug or a device company) had participated in the trial design, and thus had potentially influenced the decision to use a run-in period. A trial was categorized as an industry trial if a drug or device company was listed as “responsible party” or “sponsor” in ClinicalTrials.gov (or in other trial registries if the trial publication mentioned these), if industry employees had participated in the design of the trial (according to, eg, the trial report), if industry authors were mentioned as co-authors or if the trial was mentioned as industry funded, but did not describe who had designed the study. From the trial publications reporting a run-in period, the same author (DL) extracted the following additional data: intervention type during run-in period (eg, active, placebo, no intervention), duration of the run-in period, number of patients enrolled and excluded in the run-in period, reasons for exclusion and characteristics at inclusion (the “baseline characteristics”). Furthermore, we noted the term used for run-in period (eg, “run-in” or “lead-in period”) and purposes of the run-in period (eg, selecting patients compliant to treatment). For the publications describing a trial with a run-in period, one author (DL) evaluated the completeness of reporting. We defined complete reporting of a run-in period as the unambiguous description of 1) the number of patients enrolled to the run-in period and the number of patients excluded during the run-in period (and, consequently, the number of patients randomized after the run-in period); 2) the reasons for exclusion of all patients during the run-in period and 3) the characteristics of excluded patients. For each paper, we also noted specifically which of our three criteria were met or not. A second author (AP) independently repeated the data extraction, assessment of industry involvement and evaluation of completeness of reporting of run-in period. This was done for all 25 run-in trial publications and for 20 randomly selected trial publications not reporting a run-in period. Disagreements were settled by discussion. We tabulated descriptive data as numbers (and percentages) or medians (and interquartile ranges [IQRs]). For trials with run-in periods, we summarized trial data, run-in period data and completeness of reporting. We also planned to compare patient characteristics for excluded and randomized patients, and we summarized reasons for using a run-in period and the terminology involved. We used Fisher’s exact test or Mann– Whitney U test to compare characteristics of trials with and without run-in phases. The software used for data analysis was Microsoft Excel and the Real Statistics Resource Pack. 8 We performed sensitivity analyses to study the robustness of our results to our definition of a “run-in period” and to our definition of “industry trials” (see Supplementary materials).

To study how sensitive our results were to our definition of a run-in period, we evaluated the consequences of a narrower and a broader definition. In comparison with the original definition, the narrower definition only consisted of the main criterion and not the secondary criterion. The broader definition added the terms “washout” to the main and the secondary criterion and “screening” to the main criterion. It also included trials with unclear reporting of the formal time of enrollment, for example, a trial where patients completed a “2-week run-in period to confirm they met the criteria before enrollment”. To study how sensitive our results were to our definition of “industry trial”, we defined a broader definition (but not a narrower definition as we did not consider that meaningful). This definition included trials that reported any support, financial or otherwise, from a commercial company (eg, a company provided the study drug free of charge). To study the impact of trials with unclearly reported industry status, we also performed a sensitivity analysis where half of the unclear trials were categorized as industry trials and the other half as not industry trials. Finally, we also presented industry status for the subgroup of pharmacological trials separately. We also studied how robust our calculation of the frequency of run-in trials among industry trials and non-industry trials was to variation. We calculated the 95% CI of the proportion of industry trials among our sample of trials without run-in periods. Then, we performed the calculations with these upper and lower bounds. Our finding of incomplete reporting of run-in periods, and our estimation of proportion of trials reporting run-in periods, was robust to variations in how we operationally defined “run-in period”. Nine out of the original 25 trials complied with a narrower definition of the term and 11 additional trials would have been included had we adopted a broader definition. So, our estimation of proportion of trials with run-in periods varied from 9 out of 470 (2%) to 36 out of 470 (8%). The two trials without participant exclusions (ie, with complete reporting) during the run-in period were not among the nine publications complying with the narrower definition, nor did any of the additional eleven trials complying with the broader definition present any of the three aspects required for complete reporting. Therefore, in all relevant scenarios, reporting of run-in periods remained clearly incomplete. Our assessment of industry involvement of the design of the trial was also robust to unclear categorization in 18 of 100 trials with no run-in period and to our operational definition of industry trial. In the subgroup of trials with pharmacological interventions, industry trials were even more drastically overrepresented (94% vs 39% in trials with and without run-in periods, respectively; see Table S1 ). The proportion of industry trials among trials without a run-in period was 28% (23 out of 82, 95% CI =19%–39%). Performing our calculations with the upper bound would reveal that 8.4% of the industry trials and 3.2% of the non-industry trials used a run-in period (using the lower bound: 16.0% and 2.4%, respectively). Therefore, at the least, run-in trials seemed to occur 2.6 times as often in industry trials as in non-industry trials. On November 23, 2018, we performed a search in PubMed for randomized clinical trials in the years 2013–2017 using the search string: “(Randomized Controlled Trial[ptyp]) AND (“2013/01/01”[Date - Publication] : “2017/12/31”[Date - Publication])”. This yielded 122,408 hits, or an average of 24,482 trial publications per year. A conservative number that excluded, for example, misclassifications and duplicates could then be 20,000. In our study sample, the proportion of publications reporting on trials with a run-in period was ~5%, yielding around 1,000 publications per year. Industry status in trials with and without run-in periods a Notes: Absolute numbers are shown, as well as percentages in parentheses and P -values from Fisher’s two-sided exact test. The P -value is followed by an asterisk when P <0.05. For the trials without a run-in period, we extracted data from a random sample of 100 trials out of the total 445 trials. Trials were classified as unclear in the main analysis if no information was available on study designers, funding, sponsorship, support or similar. In a sensitivity analysis, we reclassified half of the trials with unclear industry status as industry trials and the other half as not industry trials. In that analysis, the difference between the two groups remained significant ( P <0.01). With the broader definition, trials were classified as industry trials if they received any support, financial or otherwise, from a commercial company.