Program evaluation and causal inference with high-dimensional data

This paper provides a review of methodological advancements in the evaluation of heterogeneous treatment effect models based on instrumental variable (IV) methods. We focus on models that achieve identification by assuming monotonicity of the treatment in the IV and analyze local average and quantile treatment effects for the subpopulation of compliers. We start with a comprehensive discussion of the binary treatment and binary IV case as for instance relevant in randomized experiments with imperfect compliance. We then review extensions to identification and estimation with covariates, multi-valued and multiple treatments and instruments, outcome attrition and measurement error, and the identification of direct and indirect treatment effects, among others. We also discuss testable implications and possible relaxations of the IV assumptions, approaches to extrapolate from local to global treatment effects, and the relationship to other IV approaches.

Political scientists frequently use instrumental variables (IV) estimation to estimate the causal effect of an endogenous treatment variable. However, when the treatment effect is heterogeneous, this estimation strategy only recovers the local average treatment effect (LATE). The LATE is an average treatment effect (ATE) for a subset of the population: units that receive treatment if and only if they are induced by an exogenous IV. However, researchers may instead be interested in the ATE for the entire population of interest. In this article, we develop a simple reweighting method for estimating the ATE, shedding light on the identification challenge posed in moving from the LATE to the ATE. We apply our method to two published experiments in political science in which we demonstrate that the LATE has the potential to substantively differ from the ATE.

The local average treatment effect (LATE) may differ from the average treatment effect (ATE) when those influenced by the instrument are not representative of the overall population. Heterogeneity in treatment effects may imply that parameter estimates from 2SLS are uninformative regarding the average treatment effect, motivating a search for instruments that affect a larger share of the population. In this paper, I present and estimate a model of home production with heterogeneous costs and benefits to fertility. The results indicate that a sex-preference instrument in Taiwan produces IV estimates closer to the estimated ATE than in the United States, where sex preference is weaker.

Under treatment effect heterogeneity, an instrument identifies the instrument-specific local average treatment effect (LATE). With multiple instruments, two-stage least squares (2SLS) estimand is a weighted average of different LATEs. What is often overlooked in the literature is that the postulated moment condition evaluated at the 2SLS estimand does not hold unless those LATEs are the same. If so, the conventional heteroscedasticity-robust variance estimator would be inconsistent, and 2SLS standard errors based on such estimators would be incorrect. I derive the correct asymptotic distribution, and propose a consistent asymptotic variance estimator by using the result of Hall and Inoue (2003, Journal of Econometrics) on misspecified moment condition models. This can be used to correctly calculate the standard errors regardless of whether there is more than one LATE or not.

Unequal cluster sizes are common in cluster randomized trials (CRTs). While there are a number of previous investigations studying the impact of unequal cluster sizes on the power for testing the average treatment effect in CRTs, little is known about the impact of unequal cluster sizes on the power for testing the heterogeneous treatment effect (HTE) in CRTs. In this work, we expand the sample size procedures for studying HTE in CRTs to accommodate cluster size variation under the linear mixed model framework. Through analytical derivation and graphical exploration, we show that the sample size for the HTE with an individual-level effect modifier is less affected by unequal cluster sizes than with a cluster-level effect modifier. The impact of cluster size variability jointly depends on the mean and coefficient of variation of cluster sizes, covariate intraclass correlation coefficient (ICC) and the conditional outcome ICC. In addition, we demonstrate that the HTE-motivated analysis of covariance framework can be used for analyzing the average treatment effect, and offer a more efficient sample size procedure for studying the average treatment effect adjusting for the effect modifier. We use simulations to confirm the accuracy of the proposed sample size procedures for both the average treatment effect and HTE in CRTs. Extensions to multivariate effect modifiers are provided and our procedure is illustrated in the context of the Strategies to Reduce Injuries and Develop Confidence in Elders trial.

A fundamental challenge in observational causal inference is that assumptions about unconfoundedness are not testable from data. Assessing sensitivity to such assumptions is therefore important in practice. Unfortunately, some existing sensitivity analysis approaches inadvertently impose restrictions that are at odds with modern causal inference methods, which emphasize flexible models for observed data. To address this issue, we propose a framework that allows (1) flexible models for the observed data and (2) clean separation of the identified and unidentified parts of the sensitivity model. Our framework extends an approach from the missing data literature, known as Tukey’s factorization, to the causal inference setting. Under this factorization, we can represent the distributions of unobserved potential outcomes in terms of unidentified selection functions that posit a relationship between treatment assignment and unobserved potential outcomes. The sensitivity parameters in this framework are easily interpreted, and we provide heuristics for calibrating these parameters against observable quantities. We demonstrate the flexibility of this approach in two examples, where we estimate both average treatment effects and quantile treatment effects using Bayesian nonparametric models for the observed data.

In this paper, we consider a switching regression model with count data outcomes, where the possible outcome differs across two alternate states and individuals endogenously select one of the states. We assume lognormal latent heterogeneity. Building on the switching regression model, we derive estimators of various treatment effects: the average treatment effect, the average treatment effect on the treated, the local average treatment effect, and the marginal treatment effect. We illustrate an application that examines the effects of public insurance on the number of doctor visits using the data employed by previous studies.

Abstract: Real-world evidence (RWE) studies face bias from confounding, missingness, and informative censoring, limiting credible effect estimation for policy and practice. This study proposes an AI-augmented targeted learning workflow in Stata for average and heterogeneous treatment effects. The approach integrates programmatic cohort construction, principled handling of missing data and censoring, and automated model selection via a Super Learner ensemble that screens candidate learners using cross-validation and meta-learning heuristics. Methods include estimation of nuisance functions for propensity and outcome, targeted maximum likelihood estimation (TMLE) for average treatment effects, and doubly robust DR-learners for conditional effects; robustness checks cover overlap diagnostics, sensitivity to unmeasured confounding, and transportability assessments across sites. In synthetic yet realistic RWE benchmarks, the pipeline attains lower absolute bias and shorter confidence intervals while retaining nominal coverage, with stable runtimes suitable for routine analyses. Implications include an auditable, reproducible pathway to deploy targeted learning within standard Stata workflows, enabling decision-grade RWE under common data irregularities; limitations involve reliance on positivity, data quality, and the external validity of simulated scenarios. This matters because health and policy stakeholders require defensible causal estimates from imperfect observational data. What is provided is an end-to-end template, with diagnostics and reporting artifacts aligned to regulatory expectations. Keywords Real-world evidence, Stata, Targeted maximum likelihood estimation, Super Learner, Doubly robust estimation, Propensity score, Causal inference, Treatment effect heterogeneity, DR-learner, Sensitivity analysis, Positivity and overlap, Reproducible analytics

We revisit the classic semi‐parametric problem of inference on a low‐dimensional parameter θ0 in the presence of high‐dimensional nuisance parameters η0. We depart from the classical setting by allowing for η0 to be so high‐dimensional that the traditional assumptions (e.g. Donsker properties) that limit complexity of the parameter space for this object break down. To estimate η0, we consider the use of statistical or machine learning (ML) methods, which are particularly well suited to estimation in modern, very high‐dimensional cases. ML methods perform well by employing regularization to reduce variance and trading off regularization bias with overfitting in practice. However, both regularization bias and overfitting in estimating η0 cause a heavy bias in estimators of θ0 that are obtained by naively plugging ML estimators of η0 into estimating equations for θ0. This bias results in the naive estimator failing to be N−1/2 consistent, where N is the sample size. We show that the impact of regularization bias and overfitting on estimation of the parameter of interest θ0 can be removed by using two simple, yet critical, ingredients: (1) using Neyman‐orthogonal moments/scores that have reduced sensitivity with respect to nuisance parameters to estimate θ0; (2) making use of cross‐fitting, which provides an efficient form of data‐splitting. We call the resulting set of methods double or debiased ML (DML). We verify that DML delivers point estimators that concentrate in an N−1/2‐neighbourhood of the true parameter values and are approximately unbiased and normally distributed, which allows construction of valid confidence statements. The generic statistical theory of DML is elementary and simultaneously relies on only weak theoretical requirements, which will admit the use of a broad array of modern ML methods for estimating the nuisance parameters, such as random forests, lasso, ridge, deep neural nets, boosted trees, and various hybrids and ensembles of these methods. We illustrate the general theory by applying it to provide theoretical properties of the following: DML applied to learn the main regression parameter in a partially linear regression model; DML applied to learn the coefficient on an endogenous variable in a partially linear instrumental variables model; DML applied to learn the average treatment effect and the average treatment effect on the treated under unconfoundedness; DML applied to learn the local average treatment effect in an instrumental variables setting. In addition to these theoretical applications, we also illustrate the use of DML in three empirical examples.

This paper presents deep learning models—specifically, Long Short-Term Memory (LSTM) networks and hybrid Convolutional Neural Network–LSTM (CNN-LSTM) with a Copula-Based Random Forest (CBRF) model to estimate Heterogeneous Treatment Effects (HTEs) in survival analysis. The proposed method is designed to capture non-linear relationships and temporal dependencies in clinical and genomic data, with a particular focus on exploring how treatment effects vary by race as a moderating factor. Using breast cancer data from the TCGA-BRCA dataset, which includes both clinical variables and gene expression profiles, we filter the data to focus on two racial groups: Black or African American and White. Dimensionality reduction is performed using Principal Component Analysis (PCA). We compare the CNN-LSTM, LSTM, and CBRF models under three weighting strategies—no weights, Horvitz–Thompson (HT) weights, and Inverse Probability of Treatment Weighting (IPTW)—for predicting treatment effects. Model performance is evaluated using Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Concordance statistic (C-statistic), Average Treatment Effect (ATE), and Conditional Average Treatment Effect (CATE) by race. The CNN-LSTM model consistently outperforms the others, achieving the lowest prediction errors and highest discrimination, particularly under IPTW. Among the weighting strategies, IPTW yields the most substantial improvements in model performance and bias reduction. Importantly, race-specific treatment effects exhibit notable variation: CNN-LSTM estimates a slightly higher CATE for Black individuals under IPTW. Overall, CNN-LSTM with IPTW is recommended for robust and equitable causal inference, especially in racially stratified settings.

Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically measure how every code or feature change impacts millions of users with great heterogeneity (e.g. countries, ages, devices). The most commonly used A/B testing framework in many companies is based on Average Treatment Effect (ATE), which cannot detect the heterogeneity of treatment effect on users with different characteristics. In this paper, we propose statistical methods that can systematically and accurately identify Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g. mobile device type, country), and determine which factors (e.g. age, gender) of users contribute to the heterogeneity of the treatment effect in an A/B test. By applying these methods on both simulation data and real-world experimentation data, we show how they work robustly with controlled low False Discover Rate (FDR), and at the same time, provides us with useful insights about the heterogeneity of identified user groups. We have deployed a toolkit based on these methods, and have used it to measure the Heterogeneous Treatment Effect of many A/B tests at Snap.

A large part of the recent literature on program evaluation has focused on estimation of the average effect of the treatment under assumptions of unconfoundedness or ignorability following the seminal work by In many cases however, researchers are interested in the effects of programs beyond estimates of the overall average or the average for the subpopulation of treated individuals. It may be of substantive interest to investigate whether there is any subpopulation for which a program or treatment has a nonzero average effect, or whether there is heterogeneity in the effect of the treatment. The hypothesis that the average effect of the treatment is zero for all subpopulations is also important for researchers interested in assessing assumptions concerning the selection mechanism. In this paper we develop two nonparametric tests. The first test is for the null hypothesis that the treatment has a zero average effect for any subpopulation defined by covariates. The second test is for the null hypothesis that the average effect conditional on the covariates is identical for all subpopulations, in other words, that there is no heterogeneity in average treatment effects by covariates. Sacrificing some generality by focusing on these two specific null hypotheses we derive tests that are straightforward to implement.

Treatment effect estimation from observational data relies on auxiliary prediction exercises. This chapter presents recent developments in the econometrics literature showing that machine learning methods can be fruitfully applied for this purpose. The double machine learning (DML) approach is concerned primarily with selecting the relevant control variables and functional forms necessary for the consistent estimation of an average treatment effect. We explain why the use of orthogonal moment conditions is crucial in this setting. Another, somewhat distinct, strand of the literature focuses on treatment effect heterogeneity through the discovery of the conditional average treatment effect (CATE) function. Here we distinguish between methods aimed at estimating the entire function and those that project it on a pre-specified coordinate. We also present an empirical application that illustrates some of the methods.

Heckman and Vytlacil (1) synthesize and extend a recent body of research in economics and statistics on the identification and estimation of treatment effects when the subjects have heterogeneous responses. This research has demonstrated the importance of distinguishing between several different types of treatment effects and the need to establish firmly the relationship between the different types. Heckman and Vytlacil (H&V) distinguish between four different types: average treatment effects (ATE), effects of the treatment on the treated (TT), local average treatment effects (LATE), and local instrumental variable (LIV) treatment effects. Each is conceptually different, and each has a different set of conditions for identification. The research discussed by H&V represents a genuine advance and clarification of concepts in models of treatment effects.

Loneliness is a distressing feeling that influences well-being. Immigrants' experience of acculturation to a new dominant culture places them at risk for maladaptive behaviors and daily rhythms leading to loneliness. Identifying daily-living features that causally influence loneliness is essential for developing effective preventive mental health screening. To identify the important daily living-features related to loneliness for the development of robust screening solutions using causal machine learning for health providers working with first-generation immigrants. We monitored 39 immigrants in Finland for 28 days using mobile devices and wearables under free-living conditions. Data included ecological momentary assessments of loneliness, social interactions, physical activity, sleep, and cardiac features. We estimated the average treatment effect (ATE) of each daily-living feature (treatment variable) on loneliness scores (outcome) and validated the robustness of causal estimates using three refutation techniques. Our results reveal the ATE of various daily-living features on loneliness. Features such as longer outgoing call durations (ATE = 0.197, p < 0.001), higher LF/HF ratio (ATE = 0.129, p < 0.0001), higher respiratory rate (ATE = 0.144, p < 0.001), and increased inactivity (ATE = 0.130, p < 0.001) causally increased loneliness. Conversely, certain features exhibit negative ATEs, such as higher activity calories (ATE = -0.174, p < 0.001), sleep RMSSD (ATE = -0.128, p < 0.001), longer home duration (ATE = -0.107, p < 0.001), and more sleep time (ATE = -0.103, p < 0.001) mitigated loneliness. Daily-living features, including social interactions, activity, sleep, and cardiac features, causally influence loneliness. Our findings provide a basis for loneliness screening targeting immigrant populations. Future work should refine the measurement and incorporate contextual information to establish more reliable causal links in real life.