Another Caution for Difference-in-Differences: Expected Gains

Another Caution for Difference-in-Differences: Expected Gains | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Method Article Another Caution for Difference-in-Differences: Expected Gains Bryan Dowd, Melissa Garrido This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5000288/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Many interventions are based on voluntary participation in the treatment group and difference-in-differences (DID) models frequently are used to estimate the effect of the treatment on treatment group versus the untreated control group. Expected gains in the form of resolve or capacity to adhere to the intervention are likely to be unobserved by the analyst and affect outcomes only after the subject learns the actual content of the intervention effect. When an omitted variable is both time-varying and subject-varying, it will not be undetectable by all the usual DID specification tests, including tests of the parallel trends assumption, and will not be corrected by the standard two-way fixed effect model. Both the internal and external validity of estimated treatment effect can be threatened, whether the estimates are biased from a policy standpoint depends on how the intervention will be expanded if it proves to be successful. When the analyst suspects that unobserved expected gains are a source of bias in a DID model, there are a number of appropriate econometric methods available that double as specification tests. We provide a simulation example to show how the problem arises, and how it can be addressed. Difference-in-differences Research Design Expected Gains Analytic Methods Figures Figure 1 Figure 2 Figure 3 Figure 4 1. Introduction Econometricians have had a long-standing interest in correcting for the effects of unobserved confounders (spurious correlation, omitted variable bias) in the analysis of causal effects. Ronald Fisher introduced randomized, controlled trials in 1926 (Fisher 1926 ); Philip Wright introduced the concept of instrumental variables in 1928 (Wright 1928; Stock and Trebbi 2003 ); and Heckman ( 1974 ) and Lee ( 1976 ) developed the sample selection model in the mid-1970s. Natural experiments were the approach du jour in the 1990s and remain popular today. Each of these approaches can be implemented with cross-sectional data in which the subject is observed only once. As the availability of panel data increased, methods that measure causal effects expanded from comparisons across subjects to include differences in the same subject’s responses over time. Foremost among those methods was difference-in-differences (DID). In evaluations of treatment effects, the DID approach compares outcomes ( \(\:{Y}_{it}\) ) for subjects in the treatment and comparison groups, before and after the application of the treatment. The simplest version of the DID estimator is: $$\:{Y}_{it}={\beta\:}_{0}+\:{Treat}_{i}{\beta\:}_{1}+{Post}_{t}{\beta\:}_{2}+({Treat\times\:Post)}_{it}{\beta\:}_{3}{+u}_{it}$$ 1 Where: Treat = 1 in both the pre-treatment and post-treatment time periods if the subject receives the treatment. Post = 1 if the observation is from the post-treatment time period \(\:\beta\:s\) are estimated coefficients \(\:{u}_{it}\) = random error The variable Treat removes the effect of average differences in time-invariant characteristics of subjects in the treatment versus comparison group, and the variable Post removes the effect of average differences in subject-invariant time-related characteristics between the pre- and post-treatment time periods. In a more flexible form of Eq. 1 , Treat is replaced by a vector of fixed effects for each subject ( Subject in Eq. 2 ), and Post in Eq. 1 is replaced by a vector of fixed effects for each time period ( Year in Eq. 2 ) (Bertrand, Duflo, and Mullainathan 2004). (For simplicity, we exclude the intercept and any explanatory variables other than TreatEffect it that vary across both subjects and time.) $$\:{Y}_{it}={Subject}_{i}{\beta\:}_{1}+{Year}_{t}{\beta\:}_{2}+{TreatEffect}_{it}{\beta\:}_{3}\:{+u}_{it}$$ 2 The interaction of Treat and Post is replaced by the variable TreatEffect which is equal to one if the subject is a member of the treatment group and the observation is from a post-treatment time period. In this estimator, known as the two-way fixed effect (TWFE) estimator, the subject and time fixed effects are designed to isolate the effect of the treatment from the effect of time-invariant subject effects and subject-invariant time effects. The TWFE estimator is the most popular DID approach, although many variations have been developed in recent years to improve estimation of treatments in a broader range of settings, including those with a staggered rollout, growing or decaying effects, and treatments that turn on and off. Callaway and Sant’ Anna 2024, Borusyak and Jaravel 2017 ; Sun and Abraham 2020; de Chaisemartin and D’Haultfoeuille 2020 ). Our discussion of unobserved expected gains applies to the simplest DID model: an intervention that is applied continuously, starting at the same time for all members of the treatment group in the post-treatment period, without consideration of growing or decaying effects. 2. Methods DID estimators are identified by consistency, that is, a treated unit’s observed outcome equals the counterfactual treated outcome (Hernan and Robbins 2020), irreversibility of the treatment (the treatment does not turn on and off), positivity (for all treated units with a given set of characteristics, there was a non-zero chance of being in the comparison group), and counterfactual assumptions, e.g., parallel trends in untreated potential outcomes across treatment and comparison groups, and no unique shocks (defined below). The parallel trends assumption states that, absent treatment, the treatment group’s outcome would evolve in a manner similar to the control group’s outcome in the post-treatment period. Although the assumption cannot be directly tested, investigators commonly test the likelihood of the assumption being true by examining whether the outcome variable evolves in a similar manner in the two groups in the pre-treatment period. When the parallel trends test is satisfied, the comparison group’s time trend in the post-treatment period is assumed to be a valid counterfactual for the treatment group’s post-treatment trend in the absence of the treatment. Daw and Hatfield 2018 ) and Zeldow and Hatfield ( 2021 ) discuss the ways in which confounding can arise in a DID analysis. They focus on observed confounders and define a confounder as “a variable with a time-varying effect on the outcome or a time-varying difference between groups. In the presence of confounding, the ATT estimates both the effect of the treatment as well as the magnitude of selection bias (Rambachan 2024). However, there is an additional concern regarding unobserved variables that impact both treatment probability and outcomes. Equations 1 and 2 assume that u it is uncorrelated with the treatment effect variable. This is the familiar common shocks assumption that rules out the occurrence of some unobserved outcome-shifting event that occurs at the same time the treatment is applied and affects the treatment and comparison groups differently. Expected gains are one source of unobserved confounding. Heckman, Urzua, and Vytlacil ( 2006 ) and Heckman, Schmierer, and Urzua ( 2010 ) discussed the problem of subjects’ unobserved anticipated gains from an intervention in cross-sectional data. In the context of a job training program, the concern was that subjects with higher unobserved expected gains from the training would be more likely to apply for participation in the training program, and also more likely to have higher post-training earnings than non-applicants. The authors referred to the general problem as “essential heterogeneity.” The degree to which expected gains impacts the validity of inferences from a DID model depends on whether the effect of expected gains varies with time, as well as on the population to whom one wishes to generalize the results. In the following sections, we define expected gains in the context of DID models, illustrate the difficulties in detecting expected gains, discuss when expected gains threaten the validity of inferences, and provide potential solutions. Because much of the DID literature uses potential outcomes notation, and much of the expected gains literature uses two-stage regression model notation, we use both to illustrate concepts of interest. 3. Results 3a. Potential outcomes framework Following Ghanem, Sant’Anna, and Wüthrich ( 2024 ), we define G as membership in the treatment group. The probability that G = 1 is a function of unobserved time-invariant expected gains (U). For simplicity, let G = 1 if U > 0 and G = 0 if U ≤ 0. Let ε be some positive value of U. Our DID estimate can be written as: {E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]} the ATT of G. Let the variable D represent receipt of the treatment. D = 1 for members of the treatment group in the post-treatment period, and D = 0 otherwise. The observed outcomes in the treated group are a function of both receipt of treatment and expected gains. That is, E[Y(post) | G = 1] = E[Y(post) | D = 1] + E[Y(post) | U = ε] and E[Y(pre) | G = 1] = E[Y(pre) | D = 1] + E[Y(pre) | U = ε] When there are no expected gains, E[Y(post) | G = 1] = E[Y(post) | D = 1] and the ATT of G equals the ATT of D. If membership in the treatment group is effectively randomly determined, the ATT of D can be generalized to the ATE of D. Similarly, Gupta, Martinez, and Navathe ( 2023 ) conceptualize the ATT as being composed of the ATE and the effect of expected gains. If the effect of expected gains on the outcome is time-invariant (E[Y(post) | U = ε] = E[Y(pre) | U = ε], the ATT of G will also equal the ATT of D, and no further adjustment is needed: {E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]} = {( E[Y(post) | D = 1] + E[Y(post) | U = ε]) – (E[Y(pre) | D = 1] + E[Y(pre) | U = ε]) } - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]} = {E[Y(post) | D = 1] – E[Y(pre) | D = 1]} - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]} = the ATT of D. However, if the effect of expected gains on the outcome varies with time (E[Y(post) | U = ε] > E[Y(pre) | U = ε]), Fig. 1 ), the ATT of G will no longer equal the ATT of D: {E[Y(post) | G = 1] – E[Y(pre) | G = 1]} - {E[Y(post) | G = 0] – E[Y(pre) | G = 0]} = {( E[Y(post) | D = 1] + E[Y(post) | U = ε]) – (E[Y(pre) | D = 1] )} - {E[Y(post) | D = 0] – E[Y(pre) | D = 0]}. What sort of unobserved expected gains might affect membership in the treatment group and have time-varying effects on outcomes? In a job training or smoking cessation program, or a difficult treatment regimen that entails active participation by the patient, a subject’s resolve to faithfully attend all the sessions, absorb the training material, and follow the instructor’s advice, might not manifest itself at all in the pre-treatment period. Only when the subject’s resolve is combined with the information and instructions that are part of the intervention does higher resolve affect the outcome. Similarly, some subjects may know that they have an inherent capacity such as personal or human capital that allow them to respond better to the prescribed treatment. Those resources also may be unobserved to the analyst and manifest themselves only in the post-treatment time period and only for subjects in the treatment group. Essential heterogeneity is possible for units of analysis besides individuals. Alternative payment model interventions conducted by the Center for Medicare and Medicaid Innovation, often rely on volunteer enrollment and where unobserved organizational-level expected gains may be associated with both participation and changes in outcomes. If unobserved expected gains affect outcomes only in the post-treatment period, the effect of expected gains will not be detected by any comparison of pre-treatment trends in the treatment and comparison groups (Fig. 2). The analyst will proceed, assuming the post-treatment experience of the comparison group is a valid counterfactual for the treatment group in the absence of the treatment. 3b. Two stage estimation methods Any model of a subject-specific endogenous explanatory variable, e.g., an unobserved confounder, consists of at least two equations. The first equation, often referred to as the sample selection equation, describes how the subject is assigned to the treatment or comparison group. The second equation is the outcome equation or “equation of interest.” Either equation can be estimated using any appropriate functional form (Lee 1983 ). Equations 2a and 2b show a different way of writing the DID model. Again, we assume that the treatment is administered at the same point in time for all subjects in the treatment group. The variable \(\:{w}_{i}\) represents the subject’s expected gain and appears in both equations and \(\:{v}_{i}\) is a “clean” error term, uncorrelated with \(\:{w}_{i}\) . \(\:{Treat}_{i}=f(\gamma\:{w}_{i}+{v}_{i}\:)\) (treatment selection equation) (2a) $$\:{Y}_{it}={Subject}_{i}{\beta\:}_{1}+{Year}_{t}{\beta\:}_{2}+{TreatEffect}_{it}{\beta\:}_{3}{+(\delta\:}_{TreatEffect}{w}_{it}+\:{u}_{it})$$ (outcome “equation of interest”) (2b) The coefficient on w in Eq. 2a, (γ), represents the causal effect of expected gains on selection into the treatment group. The coefficient on w in Eq. 2b, ( \(\:\:{\delta\:}_{TreatEffect})\) represents the causal effect of expected gains on the outcome variable. Notice that in Eq. 2b, w has acquired a t subscript, indicating that expected gains affects Y only in the post-treatment period. If w retained only an i subscript in the outcome equation, it’s effect would be cancelled out by the pre-post comparison of the same individual. But because w has both an i and a t subscript, its effect on outcomes cannot be addressed by subject or time fixed effects, because the effect of expected gains exists only for subjects in the treatment group, and only in the post-treatment time period. Thus, the effect of unobserved expected gains must be addressed through econometric techniques including instrumental variables, sample selection models, two-stage residual inclusion (Terza 2018 ), and regression discontinuity (Cook 2008 ). 3c. Illustration with Simulated Data A simulated dataset can illustrate the impact of time-varying effects of expected gains on estimates. Using Stata, we generate a dataset with 10,000 individual observations, each followed for 20 time periods (see appendix for details of the data generating process and results). Years 10–20 are considered the post-treatment period. Receipt of treatment is modeled as a function of expected gains, an instrumental variable Z that affects assignment to the treatment group but has no direct effect on the outcome variable, and a random error term. We then generate untreated and treated potential outcomes, where the treated potential outcomes among recipients of the treatment are a function of a time-varying effect of expected gains. We set the true treatment effect to zero so that the only effect on the outcome is due to expected gains. 3d. When Do Expected Gains Threaten the Validity of Inferences? Does the time-varying effect of unobserved expected gains on outcomes in in DID models necessarily represent bias? That depends on the research question. The research question will determine the population to which one wants to generalize the results from the evaluation intervention (the target population), and the target population, in turn, depends on what policymakers intend to do if the initial evaluation suggests that the intervention is successful (Imai, King, and Stuart 2008 ; Lesko, et al., 2017 ). The potential effect of expected gains on program effects can arise through non-random assignment to treatment, thus threatening internal validity (Abadie, et al., 2020 ), as well as through non-random sampling, thus threatening external validity). However, the threats to internal validity only have a practical impact if one wishes to generalize the treatment effect estimate to a population with a different treatment assignment mechanism. If the intent is to continue offering the program on a voluntary basis to a new, perhaps larger, but similar population, then the research question essentially concerns the effect of offering the treatment (intention to treat). In that case, the ATT for the voluntary participants in the evaluation sample need exhibit only internal validity. In the expanded application, subjects with relatively high expected gains will continue to enroll in the treatment group, thus replicating the results from the original evaluation. The estimated ATT will capture the effect of the program on a population of volunteers with higher than average expected gains relative to the comparison group with lower expected gains. No correction for the effect of expected gains is needed in this case. However, if policymakers intend to expand the program to a population with different characteristics, perhaps by mandating participation in the treatment, the most policy relevant parameter would be the ATE for the entire population. The ATE will capture the effect of the treatment for the entire target population with average expected gains, not just volunteers. That ATE could be obtained from a randomized, controlled trial (RCT) conducted on the expanded target population, but an RCT may be politically difficult, expensive, or unethical (Black 1996 ). Another option would be to collect additional data, presumably survey data, on subjects’ expected gains, and control for that variable in the DID analyses. However, that approach also may be infeasible. In some cases, the analyst must turn to econometric corrections for unobserved confounders, remembering that methods such as instrument variables estimate only a local average treatment effect for the subpopulation whose membership in the treatment versus control group is influenced by the value of the instrument (Imbens and Angrist 1994 ; Imbens and Rubin 1007). Although our focus is on the simplest DID model, expected gains also influence analyses that use more flexible DID estimators. Some estimators allow for parallel trends after conditioning on covariates – both pre and post, in the case of the two-way Mundlak estimator, and only in the post period, in the case of the Callaway & Sant’Anna estimator. However, these methods do not adjust for unobserved confounding. Although coefficients on treatment indicators in the two-way Mundlak estimator allow an investigator to “study the nature of selection bias into exposure”, they do not allow one to isolate the effect of a treatment from the influence of selection (Wooldridge 2021 ). Similarly, robustness tests meant to assess the degree to which results of DID models that assume parallel trends differ from results of models that allow time trends to differ for treatment and comparison groups will not allow the analyst to isolate the impact of unobserved, time varying, treatment group-specific confounders like expected gains (Bilinski and Hatfield 2020). 4. Conclusion and Discussion The literature on unobserved confounders in cross-sectional observational data studies is well-developed, but less so in panel data, including DID estimators. DID estimators are a useful way to deal with unobserved confounders in the form of time-invariant characteristics of subjects and subject-invariant time characteristics, but they are inadequate to correct for unobserved confounders that vary across the combination of time and subjects. The subject’s unobserved expected gain from an intervention is a time-varying, and subject-varying source of essential heterogeneity that can occur exactly when the treatment is applied and has different effects for members of the treatment and comparison groups. As in all models of endogenous assignment to the treatment group, the effect of essential heterogeneity depends on what will be done next if the initial evaluation suggests that treatment is successful. If expansion of the intervention will involve offering the intervention with voluntary assignment to a similar population, then the results that incorporate expected gains will provide a policy-relevant estimate of the expected outcome. But if the expansion involves a population with different characteristics, e.g., mandated versus voluntary participants, then treatment effect estimates will need to correct for unobserved expected gains through randomization, collection of additional data, or econometric techniques. Declarations Author Contribution Both authors wrote sections of the manuscript and accompanying Stata code. Data Availability Our submission include Stata code to run a simulation. (I'm not sure if that's what you're looking for.) References Abadie, A., Athey, S., Imbens, G.W., Wooldridge, J.M.: Sampling-based versus design-based uncertainty in regression analysis. Econometrica. 88 (1), 265–296 (2020) Baiocchi, M., Cheng, J., Small, D.S.. Instrumental variable methods for causal inference. Stat Med. ;33(13):2297–2340.Bertrand, Marianne, Duflo, Ester, and, Mullainathan, S.: How Much Should We Trust Difference-in-Differences Estimates? Quarterly Journal of Economics 119:1 (February 2004) 249–275. (2014) Bilinski, A., Hatfield, L.A.: Nothing to see here: Non-inferiority approaches to parallel trends and other model assumptions. Published online 2020. https://arxiv.org/abs/1805.03273 Black, N.: Why we need observational studies to evaluate the effectiveness of health care. BMJ. 312 , 1215 (1996) Borusyak, K., Jaravel, X.: Revisiting Event Study Designs. SSRN Work Pap. Published online May 8 , (2017). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2826228 Callaway, B., Sant’Anna, P.H.C.: Difference-in-Differences with multiple time periods. J. Econom. 225 (2), 200–230 (2021). 10.1016/j.jeconom.2020.12.001 Cook, T.D.: Waiting for Life to Arrive: A history of the regression-discontinuity design in Psychology, Statistics and Economics. J. Econ. 142 (2), 636–654 (2008) de Chaisemartin, C., D’Haultfoeuille, X.: Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. Am. Econ. Rev. 110 (9), 2964–2996 (2020). 10.1257/aer.20181169 Daw, J.R., Hatfield, L.A.: Matching in Difference in Differences: between a Rock and a Hard. Place Health Serv. Res. 53 (6), 4111–4117 (2018) Fisher, R.: ‘The Arrangement of Field Experiments’. J. Ministry Agric. 33 , 500–513 (1926) Ghanem, D., Sant’Anna, P.H.C., Wüthrich, K.: Selection and parallel trends. arXiv:2203.09001v9. (2024) Gupta, A., Martinez, J.R., Navathe, A.S.: Selection and causal effects in voluntary programs: Bundled payments in Medicare. NBER working paper series. Working paper 31256. ; (2023). http://www.nber.org/papers/w31256 Heckman, J.: Shadow Prices, Market Wages, and Labor Supply. Econometrica. 42 (4), 679–694 (1974) Heckman, J.J., Urzua, S., Vytlacil, E.: Understanding instrumental variables in models with essential heterogeneity. Rev. Econ. Stat. 88 (3), 389–342 (2006) Heckman, J.J., Schmierer, D., Urzua, S.: Testing the correlated random coefficient model. J. Econ. 158 , 177–203 (2010) Hernán, M.A., Robins, J.M.: Causal Inference: What If. Chapman & Hall/CRC, Boca Raton (2020) Imai, K., King, G., Stuart, G.: Misunderstandings between experimentalists and observationalists about causal inference. J. Royal Stat. Soc. Ser. A. 171 (2), 481–502 (2008) Imbens, G.W., Angrist, J.D.: Identification and Estimation of Local Average Treatment Effects. Econometrica. 62 (2), 467–475 (1994) Imbens, G.W., Donald, B., Rubin: Estimating Outcome Distributions for Compliers in Instrumental Variables Models. Rev. Econ. Stud. 64 , 555–574 (1997) Lee, L.F.: Estimation of Limited Dependent Variables by Two Stage Method, unpublished Ph.D. thesis, Department of Economics, University of Rochester. (1976) Lee, L.: Generalized Econometric Models Selectivity Econometrica. 51 (2), 507–512 (1983) Lesko, C.R., Buchanan, A.L., Westreich, D., Edwards, J.K., Hudgens, M.G., Cole, S.R.: Generalizing study results: A potential outcomes perspective. Epidemiology. 28 (4), 553–561 (2017) Rambachan, A., Roth, J.: A more credible approach to parallel trends. Rev. Econ. Stud. (2023) Stock, J.H., Trebbi, F.: Retrospectives: Who Invented Instrumental Variable Regression? J. Economic Perspect. 17 (3), 177–194 (2003) Sun, L., Abraham, S.: Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects. ArXiv180405785 Econ. Published online September 22, 2020. http://arxiv.org/abs/1804.05785 Terza, J.V.: Two-stage residual inclusion estimation. Health Serv. Res. Health Econ. 53 (3), 1890–1899 (2018) Wooldridge, J.M.: Two-Way Fixed Effects, the Two-Way Mundlak Regression, and Difference-in-Differences Estimators. SSRN Electron. J. Published online. (2021). 10.2139/ssrn.3906345 Wright, P.: The Tariff on Animal and Vegetable Oils: Appendix B. 1928. Reproduced in Stock and Trebbi (2003) Zeldow, B., Hatfield, L.A.: Confounding and egression adjustment in difference-in-differences studies. Health Serv. Res. 1–10 (2021). https://doi.org/10.1111/1475-6773.13666 Additional Declarations No competing interests reported. Supplementary Files StatadofileinWord.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5000288","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Method Article","associatedPublications":[],"authors":[{"id":360298252,"identity":"a0684211-08c4-4149-803b-f283f6a207a5","order_by":0,"name":"Bryan Dowd","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAsUlEQVRIiWNgGAWjYDCCAwwMEgwGEjz8EC4zsVoqbGQkG0jTcibNxuAAsVr4jvcY3uZtO8xjfCM77QFDhXViAyEtkmfOGFvOBGoxu5G73YDhTDphLQY3crdJfIRo2SbB2HaYSC2JIIfNAGn5R6yWD2fSeAwkQFoaiNAieeb8Z8sZFTY8EmfebpNIOJZuTFAL3/G2xNtAK+z520HW1VjLEtSCChJIUz4KRsEoGAWjABcAAC7YQcpRjWraAAAAAElFTkSuQmCC","orcid":"","institution":"University of Minnesota","correspondingAuthor":true,"prefix":"","firstName":"Bryan","middleName":"","lastName":"Dowd","suffix":""},{"id":360298254,"identity":"54642dce-3461-4b10-b27b-a9d4026c13f9","order_by":1,"name":"Melissa Garrido","email":"","orcid":"","institution":"Boston University","correspondingAuthor":false,"prefix":"","firstName":"Melissa","middleName":"","lastName":"Garrido","suffix":""}],"badges":[],"createdAt":"2024-08-30 00:39:23","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5000288/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5000288/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":66267747,"identity":"0f231504-fb85-4e9c-9476-17d1ca2b5e66","added_by":"auto","created_at":"2024-10-09 12:10:51","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":37226,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/14b5ea4b8db4be4000156806.png"},{"id":66267751,"identity":"858c9e39-2a4d-459f-8497-c5d78e5c59d6","added_by":"auto","created_at":"2024-10-09 12:10:51","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":43174,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/4b4d8e4225ba5d168af042e7.png"},{"id":66268069,"identity":"4d301598-9f1b-49c0-bd9a-ca4366593a22","added_by":"auto","created_at":"2024-10-09 12:18:51","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":45491,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/45a536b024de4da5643210df.png"},{"id":66267748,"identity":"ae8ffebe-31cc-441c-b83a-97a2ed280714","added_by":"auto","created_at":"2024-10-09 12:10:51","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":34210,"visible":true,"origin":"","legend":"\u003cp\u003eSee image above for figure legend.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/5e2d3680e29fe9a1a8d6fe76.png"},{"id":80685005,"identity":"b443d5dd-ccdd-4021-9699-88e4ded28f73","added_by":"auto","created_at":"2025-04-16 03:16:23","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":589934,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/5f4665e0-2e56-4974-a997-5e91a86ef76a.pdf"},{"id":66267749,"identity":"035f1615-ee65-47a6-a559-207c3c361e80","added_by":"auto","created_at":"2024-10-09 12:10:51","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":15706,"visible":true,"origin":"","legend":"","description":"","filename":"StatadofileinWord.docx","url":"https://assets-eu.researchsquare.com/files/rs-5000288/v1/f24650fd00dfcdf231f6d7e4.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Another Caution for Difference-in-Differences: Expected Gains","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eEconometricians have had a long-standing interest in correcting for the effects of unobserved confounders (spurious correlation, omitted variable bias) in the analysis of causal effects. Ronald Fisher introduced randomized, controlled trials in 1926 (Fisher \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e1926\u003c/span\u003e); Philip Wright introduced the concept of instrumental variables in 1928 (Wright 1928; Stock and Trebbi \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2003\u003c/span\u003e); and Heckman (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e1974\u003c/span\u003e) and Lee (\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e1976\u003c/span\u003e) developed the sample selection model in the mid-1970s. Natural experiments were the approach du jour in the 1990s and remain popular today. Each of these approaches can be implemented with cross-sectional data in which the subject is observed only once. As the availability of panel data increased, methods that measure causal effects expanded from comparisons across subjects to include differences in the same subject\u0026rsquo;s responses over time. Foremost among those methods was difference-in-differences (DID). In evaluations of treatment effects, the DID approach compares outcomes (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{Y}_{it}\\)\u003c/span\u003e\u003c/span\u003e ) for subjects in the treatment and comparison groups, before and after the application of the treatment.\u003c/p\u003e \u003cp\u003eThe simplest version of the DID estimator is:\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\:{Y}_{it}={\\beta\\:}_{0}+\\:{Treat}_{i}{\\beta\\:}_{1}+{Post}_{t}{\\beta\\:}_{2}+({Treat\\times\\:Post)}_{it}{\\beta\\:}_{3}{+u}_{it}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere:\u003c/p\u003e \u003cp\u003e \u003cem\u003eTreat\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1 in both the pre-treatment and post-treatment time periods if the subject receives the treatment.\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003e \u003cem\u003ePost\u003c/em\u003e\u0026thinsp;=\u0026thinsp;1 if the observation is from the post-treatment time period\u003c/p\u003e\u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(\\:\\beta\\:s\\)\u003c/span\u003e \u003c/span\u003e are estimated coefficients\u003c/p\u003e\u003cp\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(\\:{u}_{it}\\)\u003c/span\u003e \u003c/span\u003e = random error\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe variable \u003cem\u003eTreat\u003c/em\u003e removes the effect of average differences in time-invariant characteristics of subjects in the treatment versus comparison group, and the variable \u003cem\u003ePost\u003c/em\u003e removes the effect of average differences in subject-invariant time-related characteristics between the pre- and post-treatment time periods.\u003c/p\u003e \u003cp\u003eIn a more flexible form of Eq.\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, \u003cem\u003eTreat\u003c/em\u003e is replaced by a vector of fixed effects for each subject (\u003cem\u003eSubject\u003c/em\u003e in Eq.\u0026nbsp;\u003cspan refid=\"Equ2\" class=\"InternalRef\"\u003e2\u003c/span\u003e), and \u003cem\u003ePost\u003c/em\u003e in Eq.\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e is replaced by a vector of fixed effects for each time period (\u003cem\u003eYear\u003c/em\u003e in Eq.\u0026nbsp;\u003cspan refid=\"Equ2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) (Bertrand, Duflo, and Mullainathan 2004). (For simplicity, we exclude the intercept and any explanatory variables other than \u003cem\u003eTreatEffect\u003c/em\u003e\u003csub\u003e\u003cem\u003eit\u003c/em\u003e\u003c/sub\u003e that vary across both subjects and time.)\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$\\:{Y}_{it}={Subject}_{i}{\\beta\\:}_{1}+{Year}_{t}{\\beta\\:}_{2}+{TreatEffect}_{it}{\\beta\\:}_{3}\\:{+u}_{it}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eThe interaction of \u003cem\u003eTreat\u003c/em\u003e and \u003cem\u003ePost\u003c/em\u003e is replaced by the variable \u003cem\u003eTreatEffect\u003c/em\u003e which is equal to one if the subject is a member of the treatment group \u003cem\u003eand\u003c/em\u003e the observation is from a post-treatment time period. In this estimator, known as the two-way fixed effect (TWFE) estimator, the subject and time fixed effects are designed to isolate the effect of the treatment from the effect of time-invariant subject effects and subject-invariant time effects. The TWFE estimator is the most popular DID approach, although many variations have been developed in recent years to improve estimation of treatments in a broader range of settings, including those with a staggered rollout, growing or decaying effects, and treatments that turn on and off. Callaway and Sant\u0026rsquo; Anna 2024, Borusyak and Jaravel \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2017\u003c/span\u003e; Sun and Abraham 2020; de Chaisemartin and D\u0026rsquo;Haultfoeuille \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). Our discussion of unobserved expected gains applies to the simplest DID model: an intervention that is applied continuously, starting at the same time for all members of the treatment group in the post-treatment period, without consideration of growing or decaying effects.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cp\u003eDID estimators are identified by consistency, that is, a treated unit\u0026rsquo;s observed outcome equals the counterfactual treated outcome (Hernan and Robbins 2020), irreversibility of the treatment (the treatment does not turn on and off), positivity (for all treated units with a given set of characteristics, there was a non-zero chance of being in the comparison group), and counterfactual assumptions, e.g., parallel trends in untreated potential outcomes across treatment and comparison groups, and no unique shocks (defined below).\u003c/p\u003e \u003cp\u003eThe parallel trends assumption states that, absent treatment, the treatment group\u0026rsquo;s outcome would evolve in a manner similar to the control group\u0026rsquo;s outcome in the post-treatment period. Although the assumption cannot be directly tested, investigators commonly test the likelihood of the assumption being true by examining whether the outcome variable evolves in a similar manner in the two groups in the pre-treatment period. When the parallel trends test is satisfied, the comparison group\u0026rsquo;s time trend in the post-treatment period is assumed to be a valid counterfactual for the treatment group\u0026rsquo;s post-treatment trend in the absence of the treatment.\u003c/p\u003e \u003cp\u003eDaw and Hatfield \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) and Zeldow and Hatfield (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) discuss the ways in which confounding can arise in a DID analysis. They focus on observed confounders and define a confounder as \u0026ldquo;a variable with a time-varying effect on the outcome or a time-varying difference between groups. In the presence of confounding, the ATT estimates both the effect of the treatment as well as the magnitude of selection bias (Rambachan 2024).\u003c/p\u003e \u003cp\u003eHowever, there is an additional concern regarding \u003cem\u003eunobserved\u003c/em\u003e variables that impact both treatment probability and outcomes. Equations\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e and \u003cspan refid=\"Equ2\" class=\"InternalRef\"\u003e2\u003c/span\u003e assume that \u003cem\u003eu\u003c/em\u003e\u003csub\u003e\u003cem\u003eit\u003c/em\u003e\u003c/sub\u003e is uncorrelated with the treatment effect variable. This is the familiar common shocks assumption that rules out the occurrence of some unobserved outcome-shifting event that occurs at the same time the treatment is applied and affects the treatment and comparison groups differently. Expected gains are one source of unobserved confounding.\u003c/p\u003e \u003cp\u003eHeckman, Urzua, and Vytlacil (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2006\u003c/span\u003e) and Heckman, Schmierer, and Urzua (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2010\u003c/span\u003e) discussed the problem of subjects\u0026rsquo; unobserved anticipated gains from an intervention in cross-sectional data. In the context of a job training program, the concern was that subjects with higher unobserved expected gains from the training would be more likely to apply for participation in the training program, and also more likely to have higher post-training earnings than non-applicants. The authors referred to the general problem as \u0026ldquo;essential heterogeneity.\u0026rdquo; The degree to which expected gains impacts the validity of inferences from a DID model depends on whether the effect of expected gains varies with time, as well as on the population to whom one wishes to generalize the results.\u003c/p\u003e \u003cp\u003eIn the following sections, we define expected gains in the context of DID models, illustrate the difficulties in detecting expected gains, discuss when expected gains threaten the validity of inferences, and provide potential solutions. Because much of the DID literature uses potential outcomes notation, and much of the expected gains literature uses two-stage regression model notation, we use both to illustrate concepts of interest.\u003c/p\u003e"},{"header":"3. Results","content":"\u003ch3\u003e3a. Potential outcomes framework\u003c/h3\u003e\n\u003cp\u003eFollowing Ghanem, Sant\u0026rsquo;Anna, and W\u0026uuml;thrich (\u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e), we define G as membership in the treatment group. The probability that G\u0026thinsp;=\u0026thinsp;1 is a function of unobserved time-invariant expected gains (U). For simplicity, let G\u0026thinsp;=\u0026thinsp;1 if U\u0026thinsp;\u0026gt;\u0026thinsp;0 and G\u0026thinsp;=\u0026thinsp;0 if U\u0026thinsp;\u0026le;\u0026thinsp;0. Let \u0026epsilon; be some positive value of U. Our DID estimate can be written as:\u003c/p\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e{E[Y(post) | G\u0026thinsp;=\u0026thinsp;1] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;1]} - {E[Y(post) | G\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;0]}\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003ethe ATT of G. Let the variable D represent receipt of the treatment. D\u0026thinsp;=\u0026thinsp;1 for members of the treatment group in the post-treatment period, and D\u0026thinsp;=\u0026thinsp;0 otherwise. The observed outcomes in the treated group are a function of both receipt of treatment and expected gains. That is,\u003c/p\u003e\n\u003cp\u003eE[Y(post) | G\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;=\u0026thinsp;E[Y(post) | D\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;+\u0026thinsp;E[Y(post) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]\u003c/p\u003e\n\u003cp\u003eand E[Y(pre) | G\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;=\u0026thinsp;E[Y(pre) | D\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;+\u0026thinsp;E[Y(pre) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]\u003c/p\u003e\n\u003cp\u003eWhen there are no expected gains, E[Y(post) | G\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;=\u0026thinsp;E[Y(post) | D\u0026thinsp;=\u0026thinsp;1] and the ATT of G equals the ATT of D. If membership in the treatment group is effectively randomly determined, the ATT of D can be generalized to the ATE of D. Similarly, Gupta, Martinez, and Navathe (\u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e) conceptualize the ATT as being composed of the ATE and the effect of expected gains.\u003c/p\u003e\n\u003cp\u003eIf the effect of expected gains on the outcome is time-invariant (E[Y(post) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]\u003c/p\u003e\n\u003cp\u003e= E[Y(pre) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;], the ATT of G will also equal the ATT of D, and no further adjustment is needed:\u003c/p\u003e\n\u003cp\u003e{E[Y(post) | G\u0026thinsp;=\u0026thinsp;1] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;1]} - {E[Y(post) | G\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;0]} =\u003c/p\u003e\n\u003cp\u003e{( E[Y(post) | D\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;+\u0026thinsp;E[Y(post) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]) \u0026ndash; (E[Y(pre) | D\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;+\u0026thinsp;E[Y(pre) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]) } - {E[Y(post) | D\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | D\u0026thinsp;=\u0026thinsp;0]} =\u003c/p\u003e\n\u003cp\u003e{E[Y(post) | D\u0026thinsp;=\u0026thinsp;1] \u0026ndash; E[Y(pre) | D\u0026thinsp;=\u0026thinsp;1]} - {E[Y(post) | D\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | D\u0026thinsp;=\u0026thinsp;0]} =\u003c/p\u003e\n\u003cp\u003ethe ATT of D.\u003c/p\u003e\n\u003cp\u003eHowever, if the effect of expected gains on the outcome varies with time (E[Y(post) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]\u0026thinsp;\u0026gt;\u0026thinsp;E[Y(pre) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]), Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e), the ATT of G will no longer equal the ATT of D:\u003c/p\u003e\n\u003cp\u003e{E[Y(post) | G\u0026thinsp;=\u0026thinsp;1] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;1]} - {E[Y(post) | G\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | G\u0026thinsp;=\u0026thinsp;0]} = {( E[Y(post) | D\u0026thinsp;=\u0026thinsp;1]\u0026thinsp;+\u0026thinsp;E[Y(post) | U\u0026thinsp;=\u0026thinsp;\u0026epsilon;]) \u0026ndash; (E[Y(pre) | D\u0026thinsp;=\u0026thinsp;1] )} - {E[Y(post) | D\u0026thinsp;=\u0026thinsp;0] \u0026ndash; E[Y(pre) | D\u0026thinsp;=\u0026thinsp;0]}.\u003c/p\u003e\n\u003cp\u003eWhat sort of unobserved expected gains might affect membership in the treatment group and have time-varying effects on outcomes? In a job training or smoking cessation program, or a difficult treatment regimen that entails active participation by the patient, a subject\u0026rsquo;s \u003cem\u003eresolve\u003c/em\u003e to faithfully attend all the sessions, absorb the training material, and follow the instructor\u0026rsquo;s advice, might not manifest itself at all in the pre-treatment period. Only when the subject\u0026rsquo;s resolve is combined with the information and instructions that are part of the intervention does higher resolve affect the outcome. Similarly, some subjects may know that they have an inherent capacity such as personal or human capital that allow them to respond better to the prescribed treatment. Those resources also may be unobserved to the analyst and manifest themselves only in the post-treatment time period and only for subjects in the treatment group.\u003c/p\u003e\n\u003cp\u003eEssential heterogeneity is possible for units of analysis besides individuals. Alternative payment model interventions conducted by the Center for Medicare and Medicaid Innovation, often rely on volunteer enrollment and where unobserved organizational-level expected gains may be associated with both participation and changes in outcomes.\u003c/p\u003e\n\u003cp\u003eIf unobserved expected gains affect outcomes only in the post-treatment period, the effect of expected gains will not be detected by any comparison of pre-treatment trends in the treatment and comparison groups (Fig.\u0026nbsp;2). The analyst will proceed, assuming the post-treatment experience of the comparison group is a valid counterfactual for the treatment group in the absence of the treatment.\u003c/p\u003e\n\u003ch3\u003e3b. Two stage estimation methods\u003c/h3\u003e\n\u003cp\u003eAny model of a subject-specific endogenous explanatory variable, e.g., an unobserved confounder, consists of at least two equations. The first equation, often referred to as the sample selection equation, describes how the subject is assigned to the treatment or comparison group. The second equation is the outcome equation or \u0026ldquo;equation of interest.\u0026rdquo; Either equation can be estimated using any appropriate functional form (Lee \u003cspan class=\"CitationRef\"\u003e1983\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eEquations\u0026nbsp;2a and 2b show a different way of writing the DID model. Again, we assume that the treatment is administered at the same point in time for all subjects in the treatment group. The variable \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{w}_{i}\\)\u003c/span\u003e\u003c/span\u003e represents the subject\u0026rsquo;s expected gain and appears in both equations and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{v}_{i}\\)\u003c/span\u003e\u003c/span\u003e is a \u0026ldquo;clean\u0026rdquo; error term, uncorrelated with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:{w}_{i}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u0026nbsp;\u003cspan class=\"mathinline\"\u003e\\(\\:{Treat}_{i}=f(\\gamma\\:{w}_{i}+{v}_{i}\\:)\\)\u003c/span\u003e\u0026nbsp;\u003c/span\u003e (treatment selection equation) (2a)\u003c/p\u003e\n\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e$$\\:{Y}_{it}={Subject}_{i}{\\beta\\:}_{1}+{Year}_{t}{\\beta\\:}_{2}+{TreatEffect}_{it}{\\beta\\:}_{3}{+(\\delta\\:}_{TreatEffect}{w}_{it}+\\:{u}_{it})$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e(outcome \u0026ldquo;equation of interest\u0026rdquo;) (2b)\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eThe coefficient on w in Eq.\u0026nbsp;2a, (\u0026gamma;), represents the causal effect of expected gains on selection into the treatment group. The coefficient on w in Eq.\u0026nbsp;2b, (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\:\\:{\\delta\\:}_{TreatEffect})\\)\u003c/span\u003e\u003c/span\u003e represents the causal effect of expected gains on the outcome variable. Notice that in Eq.\u0026nbsp;2b, \u003cem\u003ew\u003c/em\u003e has acquired a \u003cem\u003et\u003c/em\u003e subscript, indicating that expected gains affects \u003cem\u003eY\u003c/em\u003e only in the post-treatment period.\u003c/p\u003e\n\u003cp\u003eIf \u003cem\u003ew\u003c/em\u003e retained only an \u003cem\u003ei\u003c/em\u003e subscript in the outcome equation, it\u0026rsquo;s effect would be cancelled out by the pre-post comparison of the same individual. But because \u003cem\u003ew\u003c/em\u003e has both an \u003cem\u003ei\u003c/em\u003e and a \u003cem\u003et\u003c/em\u003e subscript, its effect on outcomes cannot be addressed by subject or time fixed effects, because the effect of expected gains exists only for subjects in the treatment group, and only in the post-treatment time period. Thus, the effect of unobserved expected gains must be addressed through econometric techniques including instrumental variables, sample selection models, two-stage residual inclusion (Terza \u003cspan class=\"CitationRef\"\u003e2018\u003c/span\u003e), and regression discontinuity (Cook \u003cspan class=\"CitationRef\"\u003e2008\u003c/span\u003e).\u003c/p\u003e\n\u003ch3\u003e3c. Illustration with Simulated Data\u003c/h3\u003e\n\u003cp\u003eA simulated dataset can illustrate the impact of time-varying effects of expected gains on estimates. Using Stata, we generate a dataset with 10,000 individual observations, each followed for 20 time periods (see appendix for details of the data generating process and results). Years 10\u0026ndash;20 are considered the post-treatment period. Receipt of treatment is modeled as a function of expected gains, an instrumental variable Z that affects assignment to the treatment group but has no direct effect on the outcome variable, and a random error term. We then generate untreated and treated potential outcomes, where the treated potential outcomes among recipients of the treatment are a function of a time-varying effect of expected gains. We set the true treatment effect to zero so that the only effect on the outcome is due to expected gains.\u003c/p\u003e\n\u003ch3\u003e3d. When Do Expected Gains Threaten the Validity of Inferences?\u003c/h3\u003e\n\u003cp\u003eDoes the time-varying effect of unobserved expected gains on outcomes in in DID models \u003cem\u003enecessarily\u003c/em\u003e represent bias? That depends on the research question. The research question will determine the population to which one wants to generalize the results from the evaluation intervention (the target population), and the target population, in turn, depends on what policymakers intend to do if the initial evaluation suggests that the intervention is successful (Imai, King, and Stuart \u003cspan class=\"CitationRef\"\u003e2008\u003c/span\u003e; Lesko, et al., \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e).\u003c/p\u003e\n\u003cp\u003eThe potential effect of expected gains on program effects can arise through non-random assignment to treatment, thus threatening internal validity (Abadie, et al., \u003cspan class=\"CitationRef\"\u003e2020\u003c/span\u003e), as well as through non-random sampling, thus threatening external validity). However, the threats to internal validity only have a practical impact if one wishes to generalize the treatment effect estimate to a population with a different treatment assignment mechanism. If the intent is to continue offering the program on a voluntary basis to a new, perhaps larger, but similar population, then the research question essentially concerns the effect of \u003cem\u003eoffering\u003c/em\u003e the treatment (intention to treat). In that case, the ATT for the voluntary participants in the evaluation sample need exhibit only internal validity. In the expanded application, subjects with relatively high expected gains will continue to enroll in the treatment group, thus replicating the results from the original evaluation. The estimated ATT will capture the effect of the program on a population of volunteers with higher than average expected gains relative to the comparison group with lower expected gains. No correction for the effect of expected gains is needed in this case.\u003c/p\u003e\n\u003cp\u003eHowever, if policymakers intend to expand the program to a population with different characteristics, perhaps by \u003cem\u003emandating\u003c/em\u003e participation in the treatment, the most policy relevant parameter would be the ATE for the entire population. The ATE will capture the effect of the treatment for the entire target population with average expected gains, not just volunteers. That ATE could be obtained from a randomized, controlled trial (RCT) conducted on the expanded target population, but an RCT may be politically difficult, expensive, or unethical (Black \u003cspan class=\"CitationRef\"\u003e1996\u003c/span\u003e). Another option would be to collect additional data, presumably survey data, on subjects\u0026rsquo; expected gains, and control for that variable in the DID analyses. However, that approach also may be infeasible. In some cases, the analyst must turn to econometric corrections for unobserved confounders, remembering that methods such as instrument variables estimate only a local average treatment effect for the subpopulation whose membership in the treatment versus control group is influenced by the value of the instrument (Imbens and Angrist \u003cspan class=\"CitationRef\"\u003e1994\u003c/span\u003e; Imbens and Rubin 1007).\u003c/p\u003e\n\u003cp\u003eAlthough our focus is on the simplest DID model, expected gains also influence analyses that use more flexible DID estimators. Some estimators allow for parallel trends after conditioning on covariates \u0026ndash; both pre and post, in the case of the two-way Mundlak estimator, and only in the post period, in the case of the Callaway \u0026amp; Sant\u0026rsquo;Anna estimator. However, these methods do not adjust for unobserved confounding. Although coefficients on treatment indicators in the two-way Mundlak estimator allow an investigator to \u0026ldquo;study the nature of selection bias into exposure\u0026rdquo;, they do not allow one to isolate the effect of a treatment from the influence of selection (Wooldridge \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e). Similarly, robustness tests meant to assess the degree to which results of DID models that assume parallel trends differ from results of models that allow time trends to differ for treatment and comparison groups will not allow the analyst to isolate the impact of unobserved, time varying, treatment group-specific confounders like expected gains (Bilinski and Hatfield 2020).\u003c/p\u003e"},{"header":"4. Conclusion and Discussion","content":"\u003cp\u003eThe literature on unobserved confounders in cross-sectional observational data studies is well-developed, but less so in panel data, including DID estimators. DID estimators are a useful way to deal with unobserved confounders in the form of time-invariant characteristics of subjects and subject-invariant time characteristics, but they are inadequate to correct for unobserved confounders that vary across the combination of time and subjects.\u003c/p\u003e \u003cp\u003eThe subject\u0026rsquo;s unobserved expected gain from an intervention is a time-varying, and subject-varying source of essential heterogeneity that can occur exactly when the treatment is applied and has different effects for members of the treatment and comparison groups. As in all models of endogenous assignment to the treatment group, the effect of essential heterogeneity depends on what will be done next if the initial evaluation suggests that treatment is successful. If expansion of the intervention will involve offering the intervention with voluntary assignment to a similar population, then the results that incorporate expected gains will provide a policy-relevant estimate of the expected outcome. But if the expansion involves a population with different characteristics, e.g., mandated versus voluntary participants, then treatment effect estimates will need to correct for unobserved expected gains through randomization, collection of additional data, or econometric techniques.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eBoth authors wrote sections of the manuscript and accompanying Stata code.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eOur submission include Stata code to run a simulation. (I'm not sure if that's what you're looking for.)\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eAbadie, A., Athey, S., Imbens, G.W., Wooldridge, J.M.: Sampling-based versus design-based uncertainty in regression analysis. Econometrica. \u003cb\u003e88\u003c/b\u003e(1), 265\u0026ndash;296 (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaiocchi, M., Cheng, J., Small, D.S.. Instrumental variable methods for causal inference. Stat Med. ;33(13):2297\u0026ndash;2340.Bertrand, Marianne, Duflo, Ester, and, Mullainathan, S.: How Much Should We Trust Difference-in-Differences Estimates? Quarterly Journal of Economics 119:1 (February 2004) 249\u0026ndash;275. (2014)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBilinski, A., Hatfield, L.A.: Nothing to see here: Non-inferiority approaches to parallel trends and other model assumptions. Published online 2020. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/1805.03273\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/1805.03273\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlack, N.: Why we need observational studies to evaluate the effectiveness of health care. BMJ. \u003cb\u003e312\u003c/b\u003e, 1215 (1996)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBorusyak, K., Jaravel, X.: Revisiting Event Study Designs. SSRN Work Pap. Published online May \u003cb\u003e8\u003c/b\u003e, (2017). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=2826228\u003c/span\u003e\u003cspan address=\"https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2826228\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCallaway, B., Sant\u0026rsquo;Anna, P.H.C.: Difference-in-Differences with multiple time periods. J. Econom. \u003cb\u003e225\u003c/b\u003e(2), 200\u0026ndash;230 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jeconom.2020.12.001\u003c/span\u003e\u003cspan address=\"10.1016/j.jeconom.2020.12.001\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCook, T.D.: Waiting for Life to Arrive: A history of the regression-discontinuity design in Psychology, Statistics and Economics. J. Econ. \u003cb\u003e142\u003c/b\u003e(2), 636\u0026ndash;654 (2008)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ede Chaisemartin, C., D\u0026rsquo;Haultfoeuille, X.: Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. Am. Econ. Rev. \u003cb\u003e110\u003c/b\u003e(9), 2964\u0026ndash;2996 (2020). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1257/aer.20181169\u003c/span\u003e\u003cspan address=\"10.1257/aer.20181169\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDaw, J.R., Hatfield, L.A.: Matching in Difference in Differences: between a Rock and a Hard. Place Health Serv. Res. \u003cb\u003e53\u003c/b\u003e(6), 4111\u0026ndash;4117 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFisher, R.: \u0026lsquo;The Arrangement of Field Experiments\u0026rsquo;. J. Ministry Agric. \u003cb\u003e33\u003c/b\u003e, 500\u0026ndash;513 (1926)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGhanem, D., Sant\u0026rsquo;Anna, P.H.C., W\u0026uuml;thrich, K.: Selection and parallel trends. arXiv:2203.09001v9. (2024)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGupta, A., Martinez, J.R., Navathe, A.S.: Selection and causal effects in voluntary programs: Bundled payments in Medicare. NBER working paper series. Working paper 31256. ; (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.nber.org/papers/w31256\u003c/span\u003e\u003cspan address=\"http://www.nber.org/papers/w31256\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeckman, J.: Shadow Prices, Market Wages, and Labor Supply. Econometrica. \u003cb\u003e42\u003c/b\u003e(4), 679\u0026ndash;694 (1974)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeckman, J.J., Urzua, S., Vytlacil, E.: Understanding instrumental variables in models with essential heterogeneity. Rev. Econ. Stat. \u003cb\u003e88\u003c/b\u003e(3), 389\u0026ndash;342 (2006)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeckman, J.J., Schmierer, D., Urzua, S.: Testing the correlated random coefficient model. J. Econ. \u003cb\u003e158\u003c/b\u003e, 177\u0026ndash;203 (2010)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHern\u0026aacute;n, M.A., Robins, J.M.: Causal Inference: What If. Chapman \u0026amp; Hall/CRC, Boca Raton (2020)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eImai, K., King, G., Stuart, G.: Misunderstandings between experimentalists and observationalists about causal inference. J. Royal Stat. Soc. Ser. A. \u003cb\u003e171\u003c/b\u003e(2), 481\u0026ndash;502 (2008)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eImbens, G.W., Angrist, J.D.: Identification and Estimation of Local Average Treatment Effects. Econometrica. \u003cb\u003e62\u003c/b\u003e(2), 467\u0026ndash;475 (1994)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eImbens, G.W., Donald, B., Rubin: Estimating Outcome Distributions for Compliers in Instrumental Variables Models. Rev. Econ. Stud. \u003cb\u003e64\u003c/b\u003e, 555\u0026ndash;574 (1997)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee, L.F.: Estimation of Limited Dependent Variables by Two Stage Method, unpublished Ph.D. thesis, Department of Economics, University of Rochester. (1976)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee, L.: Generalized Econometric Models Selectivity Econometrica. \u003cb\u003e51\u003c/b\u003e(2), 507\u0026ndash;512 (1983)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLesko, C.R., Buchanan, A.L., Westreich, D., Edwards, J.K., Hudgens, M.G., Cole, S.R.: Generalizing study results: A potential outcomes perspective. Epidemiology. \u003cb\u003e28\u003c/b\u003e(4), 553\u0026ndash;561 (2017)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRambachan, A., Roth, J.: A more credible approach to parallel trends. Rev. Econ. Stud. (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eStock, J.H., Trebbi, F.: Retrospectives: Who Invented Instrumental Variable Regression? J. Economic Perspect. \u003cb\u003e17\u003c/b\u003e(3), 177\u0026ndash;194 (2003)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun, L., Abraham, S.: Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects. ArXiv180405785 Econ. Published online September 22, 2020. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://arxiv.org/abs/1804.05785\u003c/span\u003e\u003cspan address=\"http://arxiv.org/abs/1804.05785\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTerza, J.V.: Two-stage residual inclusion estimation. Health Serv. Res. Health Econ. \u003cb\u003e53\u003c/b\u003e(3), 1890\u0026ndash;1899 (2018)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWooldridge, J.M.: Two-Way Fixed Effects, the Two-Way Mundlak Regression, and Difference-in-Differences Estimators. SSRN Electron. J. Published online. (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2139/ssrn.3906345\u003c/span\u003e\u003cspan address=\"10.2139/ssrn.3906345\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWright, P.: The Tariff on Animal and Vegetable Oils: Appendix B. 1928. Reproduced in Stock and Trebbi (2003)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZeldow, B., Hatfield, L.A.: Confounding and egression adjustment in difference-in-differences studies. Health Serv. Res. 1\u0026ndash;10 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1111/1475-6773.13666\u003c/span\u003e\u003cspan address=\"10.1111/1475-6773.13666\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Difference-in-differences, Research Design, Expected Gains, Analytic Methods","lastPublishedDoi":"10.21203/rs.3.rs-5000288/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5000288/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Many interventions are based on voluntary participation in the treatment group and difference-in-differences (DID) models frequently are used to estimate the effect of the treatment on treatment group versus the untreated control group. Expected gains in the form of resolve or capacity to adhere to the intervention are likely to be unobserved by the analyst and affect outcomes only after the subject learns the actual content of the intervention effect. When an omitted variable is both time-varying and subject-varying, it will not be undetectable by all the usual DID specification tests, including tests of the parallel trends assumption, and will not be corrected by the standard two-way fixed effect model. Both the internal and external validity of estimated treatment effect can be threatened, whether the estimates are biased from a policy standpoint depends on how the intervention will be expanded if it proves to be successful. When the analyst suspects that unobserved expected gains are a source of bias in a DID model, there are a number of appropriate econometric methods available that double as specification tests. We provide a simulation example to show how the problem arises, and how it can be addressed.\n ","manuscriptTitle":"Another Caution for Difference-in-Differences: Expected Gains","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-09 12:10:46","doi":"10.21203/rs.3.rs-5000288/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fbccfcf7-eb12-4053-938e-3ce55227c5f4","owner":[],"postedDate":"October 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-04-16T03:08:13+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-09 12:10:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5000288","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5000288","identity":"rs-5000288","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}