Partial Identification of Population Average and Quantile Treatment Effects in Observational Data under Sample Selection

In this paper, we provide efficient estimators and honest confidence bands for a variety of treatment effects including local average (LATE) and local quantile treatment effects (LQTE) in data-rich environments. We can handle very many control variables, endogenous receipt of treatment, heterogeneous treatment effects, and function-valued outcomes. Our framework covers the special case of exogenous receipt of treatment, either conditional on controls or unconditionally as in randomized control trials. In the latter case, our approach produces efficient estimators and honest bands for (functional) average treatment effects (ATE) and quantile treatment effects (QTE). To make informative inference possible, we assume that key reduced form predictive relationships are approximately sparse. This assumption allows the use of regularization and selection methods to estimate those relations, and we provide methods for post-regularization and post-selection inference that are uniformly valid (honest) across a wide-range of models. We show that a key ingredient enabling honest inference is the use of orthogonal or doubly robust moment conditions in estimating certain reduced form functional parameters. We illustrate the use of the proposed methods with an application to estimating the effect of 401(k) eligibility and participation on accumulated assets. The results on program evaluation are obtained as a consequence of more general results on honest inference in a general moment condition framework, which arises from structural equation models in econometrics. Here too the crucial ingredient is the use of orthogonal moment conditions, which can be constructed from the initial moment conditions. We provide results on honest inference for (function-valued) parameters within this general framework where any high-quality, modern machine learning methods can be used to learn the nonparametric/high-dimensional components of the model. These include a number of supporting auxilliary results that are of major independent interest: namely, we (1) prove uniform validity of a multiplier bootstrap, (2) offer a uniformly valid functional delta method, and (3) provide results for sparsity-based estimation of regression functions for function-valued outcomes.

ABSTRACTBackgroundSurvey data are essential in paediatric epidemiology, providing valuable insights into child health outcomes. The potential outcomes framework has advanced causal inference using observational data. However, traditional design‐based adjustments, especially sample weights, are often overlooked. This omission limits the ability to generalise findings to the broader population.ObjectiveThis study demonstrates three approaches for estimating the population average treatment effect (PATE) in a practical example, examining the impact of household second‐hand smoke (SHS) exposure on blood pressure in school‐aged children.MethodsUsing data from the National Health and Nutrition Examination Survey (NHANES) 2017–2020, we assessed the effect of household SHS exposure, a non‐randomised treatment, on blood pressure in school‐aged children. We applied estimators based on Inverse Probability of Treatment Weighting (IPTW), G‐computation, Targeted Maximum Likelihood Estimation (TMLE), and regression adjustment. Models without adjustments were run for comparison. We examined point estimates and the efficiency of the estimates obtained from these methods.ResultsThe largest differences were observed between the unadjusted regression models and the fully adjusted methods (IPTW, G‐computation, and TMLE), which account for both confounding and survey weights. While the inclusion of the sample weights leads to wider confidence intervals for all methods, G‐computation and TMLE showed comparatively narrower confidence intervals. Confidence intervals for the models not adjusted for sample weights were likely underestimated.ConclusionsThis study highlights the important role of sample weights in causal inference. Generalisability of the average treatment effect as estimated on data sampled using common survey designs to a defined population requires the use of sample weights. The estimators described provide a framework for incorporating sample weights, and their use in health research is recommended.

Combining observational and experimental data for causal inference can improve treatment effect estimation. However, many observational datasets cannot be released due to data privacy considerations, so one researcher may not have access to both experimental and observational data. Nonetheless, a small amount of risk of disclosing sensitive information might be tolerable to organizations that house confidential data. In these cases, organizations can employ data privacy techniques, which decrease disclosure risk, potentially at the expense of data utility. In this study, we explore disclosure limiting transformations of observational data, which can be combined with experimental data to estimate the sample and population average treatment effects. We consider leveraging observational data to improve generalizability of treatment effect estimates, when a randomized controlled trial (RCT) is not representative of the population of interest, and to increase precision of treatment effect estimates. Through simulation studies, we illustrate the trade-off between privacy and utility when employing different disclosure limiting transformations. We find that leveraging transformed observational data in treatment effect estimation can still improve estimation over only using data from an RCT.

This paper provides a review of methodological advancements in the evaluation of heterogeneous treatment effect models based on instrumental variable (IV) methods. We focus on models that achieve identification by assuming monotonicity of the treatment in the IV and analyze local average and quantile treatment effects for the subpopulation of compliers. We start with a comprehensive discussion of the binary treatment and binary IV case as for instance relevant in randomized experiments with imperfect compliance. We then review extensions to identification and estimation with covariates, multi-valued and multiple treatments and instruments, outcome attrition and measurement error, and the identification of direct and indirect treatment effects, among others. We also discuss testable implications and possible relaxations of the IV assumptions, approaches to extrapolate from local to global treatment effects, and the relationship to other IV approaches.

Sample selection and attrition are inherent in a range of treatment evaluation problems such as the estimation of the returns to schooling or training. Conventional estimators tackling selection bias typically rely on restrictive functional form assumptions that are unlikely to hold in reality. This paper shows identification of average and quantile treatment effects in the presence of the double selection problem into (i) a selective subpopulation (e.g., working—selection on unobservables) and (ii) a binary treatment (e.g., training—selection on observables) based on weighting observations by the inverse of a nested propensity score that characterizes either selection probability. Weighting estimators based on parametric propensity score models are applied to female labor market data to estimate the returns to education.

Who does Collaborative Learning Help? The Pedagogical Efficacy of Student Learning Through Collaborative Learning Sessions

Propensity score methods have been widely adopted in observational studies, however research on propensity score-based weighting (PSW) methods in complex survey data settings is lacking, particularly for binary outcomes. We conducted a simulation study to compare eight propensity score weighting approaches for estimating treatment effects using survey weighted data. Each of the eight methods is applied to estimation of two measures of the population-level treatment effect: the population average treatment effect (PATE), and the population average treatment effect on the treated (PATT). The methods are compared in terms of mean relative bias and coverage probability under different scenarios by varying the treatment effect, degrees of model misspecification, and levels of overlap in the propensity score. The results demonstrate that the two-stage methods with predicted outcomes weighted by survey weights consistently outperform the other methods for estimating the PATT; for estimating the PATE, the best performing PSW method depends on the degree of model misspecification and propensity score overlap. When the outcome model is correctly specified, four two-stage methods produce better estimates depending on the propensity score overlap. The methods are applied to the 2015 National Health Interview Survey data to estimate the effect of provider-patient discussion about smoking on smoking cessation.

A fundamental challenge in observational causal inference is that assumptions about unconfoundedness are not testable from data. Assessing sensitivity to such assumptions is therefore important in practice. Unfortunately, some existing sensitivity analysis approaches inadvertently impose restrictions that are at odds with modern causal inference methods, which emphasize flexible models for observed data. To address this issue, we propose a framework that allows (1) flexible models for the observed data and (2) clean separation of the identified and unidentified parts of the sensitivity model. Our framework extends an approach from the missing data literature, known as Tukey’s factorization, to the causal inference setting. Under this factorization, we can represent the distributions of unobserved potential outcomes in terms of unidentified selection functions that posit a relationship between treatment assignment and unobserved potential outcomes. The sensitivity parameters in this framework are easily interpreted, and we provide heuristics for calibrating these parameters against observable quantities. We demonstrate the flexibility of this approach in two examples, where we estimate both average treatment effects and quantile treatment effects using Bayesian nonparametric models for the observed data.

O estudo busca mensurar o efeito das condições de saúde sobre o rendimento dos trabalhadores brasileiros, usando dados da PNAD de 2008. Para tanto, foram estimados o efeito de tratamento médio e quantílico com correção para o viés de seleção; o método de Lewbel (2012) que gera instrumentos internos a partir da heterocedasticidade dos resíduos e bounds para o efeito do tratamento seguindo Oster (2015). Os resultados mostram um impacto negativo de uma má condição de saúde na renda do trabalho para todas as especificações utilizadas, sendo o maior impacto sofrido por indivíduos localizados nos quantis mais baixos de renda.

While sample sizes in randomized clinical trials are large enough to estimate the average treatment effect well, they are often insufficient for estimation of treatment-covariate interactions critical to studying data-driven precision medicine. Observational data from real world practice may play an important role in alleviating this problem. One common approach in trials is to predict the outcome of interest with separate regression models in each treatment arm, and estimate the treatment effect based on the contrast of the predictions. Unfortunately, this simple approach may induce spurious treatment-covariate interaction in observational studies when the regression model is misspecified. Motivated by the need of modeling the number of relapses in multiple sclerosis (MS) patients, where the ratio of relapse rates is a natural choice of the treatment effect, we propose to estimate the conditional average treatment effect (CATE) as the ratio of expected potential outcomes, and derive a doubly robust estimator of this CATE in a semiparametric model of treatment-covariate interactions. We also provide a validation procedure to check the quality of the estimator on an independent sample. We conduct simulations to demonstrate the finite sample performance of the proposed methods, and illustrate their advantages on real data by examining the treatment effect of dimethyl fumarate compared to teriflunomide in MS patients. Supplementary materials for this article are available online.

This chapter explains a new approach that survey samplers can use when designing probability samples for survey experiments where there is a possibility of treatment heterogeneity. It begins by explaining why probability samples are preferred to nonprobability samples for estimating two quantities (or estimands): population average treatment effects and treatment effects within subgroups. The chapter furthermore explains why typical probability sampling methods that optimize statistical power for the average effect in a population do not necessarily optimize statistical power for the subgroup effects of interest – especially when one-s interest is in estimating effects within a rare subgroup. Next, it explains why even large, well-constructed, highly representative probability samples with randomized treatments can produce confounded analyses of differences across subgroups. The chapter illustrates the proposed approach using an empirical case study of a survey-administered behavioral science intervention: The US National Study of Learning Mindsets.

Poor mental health is a pressing global health problem, with high prevalence among poor populations from low-income countries. Existing studies of conditional cash transfer (CCT) effects on mental health have found positive effects. However, there is a gap in the literature on population-wide effects of cash transfers on mental health and if and how these vary by the severity of mental illness. We use the Malawian Longitudinal Study of Family and Health containing 790 adult participants in the Malawi Incentive Programme, a year-long randomized controlled trial. We estimate average and distributional quantile treatment effects and we examine how these effects vary by gender, HIV status and usage of the cash transfer. We find that the cash transfer improves mental health on average by 0.1 of a standard deviation. The effect varies strongly along the mental health distribution, with a positive effect for individuals with worst mental health of about four times the size of the average effect. These improvements in mental health are associated with increases in consumption expenditures and expenditures related to economic productivity. Our results show that CCTs can improve adult mental health for the poor living in low-income countries, particularly those with the worst mental health.

It is indispensable for young farmers to remain in agriculture and/or to sustain farm holding operations. In this study, the impact of the Young Farmer Support Program (YFSP) implemented in Turkey is tried to be define by the counterfactual impact assessment methodology in holdings with livestock such as cattle, sheep/goat. The study was conducted throughout Turkey, the questionnaire with the farmers was completed at the end of 2018. The data obtained intended for the YFSP, which was used in 2016, was taken as basis for the impact assessment. Considering the distribution and number of project subjects from each region, the provinces that would represent the region in terms of its differences were purposefully selected and sample selection was made. The focus is on two important outputs, machine equipment purchased value per livestock unit (LSU) and average income value of cattle and sheep/goat sold per LSU in the last three years which are important in the study. As explanatory variables, age, gender, marriage, educational status, number of family members, non-agricultural income, etc. are employed in the models. According to the two important indicators, Average Treatment Effect (ATE) and Average Treatment Effect on the Treated (ATET), it is revealed that YFSP has positive effects on the dependent variables examined in cattle and sheep / goat farms. This knowledge and accumulation of experience can be transferred to other main components of rural development programs.

ABSTRACTRandomized controlled trials (RCTs) provide reliable evidence for approval of new treatments, informing clinical practice , and coverage decisions. The participants in RCTs are often not a representative sample of the larger at-risk population. Hence, it is argued that the average treatment effect from the trial is not generalizable to the larger at-risk population. An essential premise of this argument is that there is significant heterogeneity in the treatment effect (HTE). We present a new method to extrapolate the treatment effect from a trial to a target group that is inadequately represented in the trial, when HTE is present. Our method integrates trial and observational data (cross-design synthesis). The target group is assumed to be well-represented in the observational database such that the observational study essentially acts as an “evidentiary bridge” that allows evidence to be transported from the trial sample to the target population. An essential component of the methodology is the estimation of calibration adjustments for unmeasured confounding in the observational sample. The estimate of treatment effect, adjusted for unmeasured confounding, is projected onto the target sample using a weighted G-computation approach. We present simulation studies to demonstrate the methodology for estimating the marginal treatment effect in a target sample that differs from the trial sample to varying degrees. In a companion article, we demonstrate and validate the methodology in a clinical application.

Unequal cluster sizes are common in cluster randomized trials (CRTs). While there are a number of previous investigations studying the impact of unequal cluster sizes on the power for testing the average treatment effect in CRTs, little is known about the impact of unequal cluster sizes on the power for testing the heterogeneous treatment effect (HTE) in CRTs. In this work, we expand the sample size procedures for studying HTE in CRTs to accommodate cluster size variation under the linear mixed model framework. Through analytical derivation and graphical exploration, we show that the sample size for the HTE with an individual-level effect modifier is less affected by unequal cluster sizes than with a cluster-level effect modifier. The impact of cluster size variability jointly depends on the mean and coefficient of variation of cluster sizes, covariate intraclass correlation coefficient (ICC) and the conditional outcome ICC. In addition, we demonstrate that the HTE-motivated analysis of covariance framework can be used for analyzing the average treatment effect, and offer a more efficient sample size procedure for studying the average treatment effect adjusting for the effect modifier. We use simulations to confirm the accuracy of the proposed sample size procedures for both the average treatment effect and HTE in CRTs. Extensions to multivariate effect modifiers are provided and our procedure is illustrated in the context of the Strategies to Reduce Injuries and Develop Confidence in Elders trial.