Nonparametric Tests for Treatment Effect Heterogeneity

In this paper, we provide efficient estimators and honest confidence bands for a variety of treatment effects including local average (LATE) and local quantile treatment effects (LQTE) in data-rich environments. We can handle very many control variables, endogenous receipt of treatment, heterogeneous treatment effects, and function-valued outcomes. Our framework covers the special case of exogenous receipt of treatment, either conditional on controls or unconditionally as in randomized control trials. In the latter case, our approach produces efficient estimators and honest bands for (functional) average treatment effects (ATE) and quantile treatment effects (QTE). To make informative inference possible, we assume that key reduced form predictive relationships are approximately sparse. This assumption allows the use of regularization and selection methods to estimate those relations, and we provide methods for post-regularization and post-selection inference that are uniformly valid (honest) across a wide-range of models. We show that a key ingredient enabling honest inference is the use of orthogonal or doubly robust moment conditions in estimating certain reduced form functional parameters. We illustrate the use of the proposed methods with an application to estimating the effect of 401(k) eligibility and participation on accumulated assets. The results on program evaluation are obtained as a consequence of more general results on honest inference in a general moment condition framework, which arises from structural equation models in econometrics. Here too the crucial ingredient is the use of orthogonal moment conditions, which can be constructed from the initial moment conditions. We provide results on honest inference for (function-valued) parameters within this general framework where any high-quality, modern machine learning methods can be used to learn the nonparametric/high-dimensional components of the model. These include a number of supporting auxilliary results that are of major independent interest: namely, we (1) prove uniform validity of a multiplier bootstrap, (2) offer a uniformly valid functional delta method, and (3) provide results for sparsity-based estimation of regression functions for function-valued outcomes.

Unequal cluster sizes are common in cluster randomized trials (CRTs). While there are a number of previous investigations studying the impact of unequal cluster sizes on the power for testing the average treatment effect in CRTs, little is known about the impact of unequal cluster sizes on the power for testing the heterogeneous treatment effect (HTE) in CRTs. In this work, we expand the sample size procedures for studying HTE in CRTs to accommodate cluster size variation under the linear mixed model framework. Through analytical derivation and graphical exploration, we show that the sample size for the HTE with an individual-level effect modifier is less affected by unequal cluster sizes than with a cluster-level effect modifier. The impact of cluster size variability jointly depends on the mean and coefficient of variation of cluster sizes, covariate intraclass correlation coefficient (ICC) and the conditional outcome ICC. In addition, we demonstrate that the HTE-motivated analysis of covariance framework can be used for analyzing the average treatment effect, and offer a more efficient sample size procedure for studying the average treatment effect adjusting for the effect modifier. We use simulations to confirm the accuracy of the proposed sample size procedures for both the average treatment effect and HTE in CRTs. Extensions to multivariate effect modifiers are provided and our procedure is illustrated in the context of the Strategies to Reduce Injuries and Develop Confidence in Elders trial.

Fixed effects estimation, with linear controls for stratum membership, is often used to estimate treatment effects when assignment propensities differ across strata. In the presence of heterogeneity in treatment effects across strata, this estimator does not target the average treatment effect, however. Indeed, the implied estimand can range anywhere from the lowest to the highest stratum-level average effect. To facilitate the interpretation of results using this approach, I establish that if stratum-level average effects are monotonic in the shares assigned to treatment, then the fixed effects estimand lies between the average treatment effect for the treated and the average treatment effect for the controls.

The local average treatment effect (LATE) may differ from the average treatment effect (ATE) when those influenced by the instrument are not representative of the overall population. Heterogeneity in treatment effects may imply that parameter estimates from 2SLS are uninformative regarding the average treatment effect, motivating a search for instruments that affect a larger share of the population. In this paper, I present and estimate a model of home production with heterogeneous costs and benefits to fertility. The results indicate that a sex-preference instrument in Taiwan produces IV estimates closer to the estimated ATE than in the United States, where sex preference is weaker.

Abstract: Real-world evidence (RWE) studies face bias from confounding, missingness, and informative censoring, limiting credible effect estimation for policy and practice. This study proposes an AI-augmented targeted learning workflow in Stata for average and heterogeneous treatment effects. The approach integrates programmatic cohort construction, principled handling of missing data and censoring, and automated model selection via a Super Learner ensemble that screens candidate learners using cross-validation and meta-learning heuristics. Methods include estimation of nuisance functions for propensity and outcome, targeted maximum likelihood estimation (TMLE) for average treatment effects, and doubly robust DR-learners for conditional effects; robustness checks cover overlap diagnostics, sensitivity to unmeasured confounding, and transportability assessments across sites. In synthetic yet realistic RWE benchmarks, the pipeline attains lower absolute bias and shorter confidence intervals while retaining nominal coverage, with stable runtimes suitable for routine analyses. Implications include an auditable, reproducible pathway to deploy targeted learning within standard Stata workflows, enabling decision-grade RWE under common data irregularities; limitations involve reliance on positivity, data quality, and the external validity of simulated scenarios. This matters because health and policy stakeholders require defensible causal estimates from imperfect observational data. What is provided is an end-to-end template, with diagnostics and reporting artifacts aligned to regulatory expectations. Keywords Real-world evidence, Stata, Targeted maximum likelihood estimation, Super Learner, Doubly robust estimation, Propensity score, Causal inference, Treatment effect heterogeneity, DR-learner, Sensitivity analysis, Positivity and overlap, Reproducible analytics

Treatment effect estimation from observational data relies on auxiliary prediction exercises. This chapter presents recent developments in the econometrics literature showing that machine learning methods can be fruitfully applied for this purpose. The double machine learning (DML) approach is concerned primarily with selecting the relevant control variables and functional forms necessary for the consistent estimation of an average treatment effect. We explain why the use of orthogonal moment conditions is crucial in this setting. Another, somewhat distinct, strand of the literature focuses on treatment effect heterogeneity through the discovery of the conditional average treatment effect (CATE) function. Here we distinguish between methods aimed at estimating the entire function and those that project it on a pre-specified coordinate. We also present an empirical application that illustrates some of the methods.

Political scientists frequently use instrumental variables (IV) estimation to estimate the causal effect of an endogenous treatment variable. However, when the treatment effect is heterogeneous, this estimation strategy only recovers the local average treatment effect (LATE). The LATE is an average treatment effect (ATE) for a subset of the population: units that receive treatment if and only if they are induced by an exogenous IV. However, researchers may instead be interested in the ATE for the entire population of interest. In this article, we develop a simple reweighting method for estimating the ATE, shedding light on the identification challenge posed in moving from the LATE to the ATE. We apply our method to two published experiments in political science in which we demonstrate that the LATE has the potential to substantively differ from the ATE.

Researchers often use regressions with a logarithmic link function to evaluate the effects of a treatment on a count variable. In order to judge the average effectiveness of the treatment on the original count scale, they compute average treatment effects, which are defined as the average difference between the expected outcomes under treatment and under control. Current practice is to evaluate the expected differences at every observation and use the sample mean of these differences as a point estimate of the average effect. The standard error for this average effect estimate is based on the implicit assumption that covariate values are fixed, i.e., do not vary across different samples. In this paper, we present a new way of analytically computing average effects based on regressions with log link using stochastic covariates and develop new formulas to obtain standard errors for the average effect. In a simulation study, we evaluate the statistical performance of our new estimator and compare it with the traditional approach. Our findings suggest that the new approach gives unbiased effect estimates and standard errors and outperforms the traditional approach when strong interaction and/or a skewed covariate is present.

This paper presents deep learning models—specifically, Long Short-Term Memory (LSTM) networks and hybrid Convolutional Neural Network–LSTM (CNN-LSTM) with a Copula-Based Random Forest (CBRF) model to estimate Heterogeneous Treatment Effects (HTEs) in survival analysis. The proposed method is designed to capture non-linear relationships and temporal dependencies in clinical and genomic data, with a particular focus on exploring how treatment effects vary by race as a moderating factor. Using breast cancer data from the TCGA-BRCA dataset, which includes both clinical variables and gene expression profiles, we filter the data to focus on two racial groups: Black or African American and White. Dimensionality reduction is performed using Principal Component Analysis (PCA). We compare the CNN-LSTM, LSTM, and CBRF models under three weighting strategies—no weights, Horvitz–Thompson (HT) weights, and Inverse Probability of Treatment Weighting (IPTW)—for predicting treatment effects. Model performance is evaluated using Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Concordance statistic (C-statistic), Average Treatment Effect (ATE), and Conditional Average Treatment Effect (CATE) by race. The CNN-LSTM model consistently outperforms the others, achieving the lowest prediction errors and highest discrimination, particularly under IPTW. Among the weighting strategies, IPTW yields the most substantial improvements in model performance and bias reduction. Importantly, race-specific treatment effects exhibit notable variation: CNN-LSTM estimates a slightly higher CATE for Black individuals under IPTW. Overall, CNN-LSTM with IPTW is recommended for robust and equitable causal inference, especially in racially stratified settings.

This paper presents an empirical study of endogenous treatment effects in the presence of heterogeneous responses. We estimate the incentive and selection effects of having prescription drug coverage on total drug expenditures using a sample from the Medicare Current Beneficiary Survey (MCBS). A Dirichlet process mixture (DPM) model is used to model the heterogeneity in treatment effects. Rather than estimating a finite mixture model with a fixed number of components, we specify a DP prior on the parameters, thus allowing the data and prior knowledge to determine the number of components. Endogeneity is modeled by the covariance of the error terms of the selection and the outcome equations in a two-equation selection model. We find that there are strong incentive and advantageous selection effects, with the average treatment effect and the average treatment effect for the treated estimated at $1132 and $858, respectively. Substantial heterogeneity is found to exist in the selection effects, particularly for those having lower drug expenditures. Copyright © 2014 John Wiley & Sons, Ltd.

Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically measure how every code or feature change impacts millions of users with great heterogeneity (e.g. countries, ages, devices). The most commonly used A/B testing framework in many companies is based on Average Treatment Effect (ATE), which cannot detect the heterogeneity of treatment effect on users with different characteristics. In this paper, we propose statistical methods that can systematically and accurately identify Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g. mobile device type, country), and determine which factors (e.g. age, gender) of users contribute to the heterogeneity of the treatment effect in an A/B test. By applying these methods on both simulation data and real-world experimentation data, we show how they work robustly with controlled low False Discover Rate (FDR), and at the same time, provides us with useful insights about the heterogeneity of identified user groups. We have deployed a toolkit based on these methods, and have used it to measure the Heterogeneous Treatment Effect of many A/B tests at Snap.

Background Symptomatic acute occlusions of the internal carotid artery (ICA) below the circle of Willis can cause a variety of stroke symptoms, even if the major intracranial cerebral arteries remain patent; however, outcome and safety data are limited. Purpose To compare treatment effects and procedural safety of endovascular treatment (EVT) and best medical treatment (BMT) in patients with symptomatic acute occlusions of the ICA below the circle of Willis. Materials and Methods This retrospective, multicenter cohort study from 22 comprehensive stroke centers in Europe and Asia includes patients treated between January 1, 2008, and December 31, 2022. Functional (modified Rankin Scale [mRS]) and clinical (National Institutes of Health Stroke Scale [NIHSS]) outcomes, safety measures (symptomatic intracerebral hemorrhage), mortality, and procedural complications were assessed. Results A total 354 patients met the inclusion criteria (median age, 72 years [IQR, 60-81 years]; median NIHSS, 13 [IQR, 7-19]). Most frequent occlusions were in the C1 segment (243 of 354; 68.6%). Of 354 patients, 82.2% (291 patients) were administered EVT. In the overall population, favorable outcomes (mRS 0-2), mortality, and symptomatic intracerebral hemorrhage occurred in 40.6% (108 of 266 patients), 25.2% (67 of 266 patients), and 7.1% (25 of 350 patients), respectively. After adjustment, no statistically significant difference in functional outcome was observed (adjusted odds ratio [AOR], 0.82 [95% CI: 0.31, 2.12]; average treatment effect, -12.7%; P = .19) in the EVT compared with BMT group. Symptomatic intracerebral hemorrhage (average treatment effect, -0.28%; P = .95) and mortality did not differ between both groups (average treatment effect, -17.1%; P = .07). EVT resulted in complete recanalization of the occlusion in 80.9% (229 of 283) of cases. Periprocedural distal embolization occurred in 27.8% (81 of 291 patients) and was associated with poor outcomes (AOR, 0.41; 95% CI: 0.18, 0.93; P = .03). Conclusion EVT did not reveal a favorable treatment effect over BMT, and both therapies were safe. EVT had a risk for periprocedural distal embolization associated with poor outcomes. © RSNA, 2025 Supplemental material is available for this article. See also the editorial by Daou and Chaudhary in this issue.

BackgroundTwo propensity score (PS) based balancing covariate methods, the overlap weighting method (OW) and the fine stratification method (FS), produce superb covariate balance. OW has been compared with various weighting methods while FS has been compared with the traditional stratification method and various matching methods. However, no study has yet compared OW and FS. In addition, OW has not yet been evaluated in large claims data with low prevalence exposure and with low frequency outcomes, a context in which optimal use of balancing methods is critical. In the study, we aimed to compare OW and FS using real-world data and simulations with low prevalence exposure and with low frequency outcomes.MethodsWe used the Texas State Medicaid claims data on adult beneficiaries with diabetes in 2012 as an empirical example (N = 42,628). Based on its real-world research question, we estimated an average treatment effect of health center vs. non-health center attendance in the total population. We also performed simulations to evaluate their relative performance. To preserve associations between covariates, we used the plasmode approach to simulate outcomes and/or exposures with N = 4,000. We simulated both homogeneous and heterogeneous treatment effects with various outcome risks (1-30% or observed: 27.75%) and/or exposure prevalence (2.5-30% or observed:10.55%). We used a weighted generalized linear model to estimate the exposure effect and the cluster-robust standard error (SE) method to estimate its SE.ResultsIn the empirical example, we found that OW had smaller standardized mean differences in all covariates (range: OW: 0.0–0.02 vs. FS: 0.22–3.26) and Mahalanobis balance distance (MB) (< 0.001 vs. > 0.049) than FS. In simulations, OW also achieved smaller MB (homogeneity: <0.04 vs. > 0.04; heterogeneity: 0.0-0.11 vs. 0.07–0.29), relative bias (homogeneity: 4.04–56.20 vs. 20–61.63; heterogeneity: 7.85–57.6 vs. 15.0-60.4), square root of mean squared error (homogeneity: 0.332–1.308 vs. 0.385–1.365; heterogeneity: 0.263-0.526 vs 0.313-0.620), and coverage probability (homogeneity: 0.0–80.4% vs. 0.0-69.8%; heterogeneity: 0.0-97.6% vs. 0.0-92.8%), than FS, in most cases.ConclusionsThese findings suggest that OW can yield nearly perfect covariate balance and therefore enhance the accuracy of average treatment effect estimation in the total population.

Summary We construct tests for the null hypothesis that the conditional average treatment effect is non-negative, conditional on every possible value of a subset of covariates. Testing such a null hypothesis can provide more information than the sign of the average treatment effects parameter. The null hypothesis can be characterized as infinitely many of unconditional moment inequalities. A Kolmogorov–Smirnov test is constructed based on these unconditional moment inequalities, and a simulated critical value is proposed. It is shown that our test can control the size uniformly over a broad set of data-generating processes asymptotically, that it is consistent against fixed alternatives and that it is unbiased against some N−1/2 local alternatives. Several extensions of our test are also considered and we apply our tests to examine the effect of a job-training programme on real earnings.

In this paper, we consider a switching regression model with count data outcomes, where the possible outcome differs across two alternate states and individuals endogenously select one of the states. We assume lognormal latent heterogeneity. Building on the switching regression model, we derive estimators of various treatment effects: the average treatment effect, the average treatment effect on the treated, the local average treatment effect, and the marginal treatment effect. We illustrate an application that examines the effects of public insurance on the number of doctor visits using the data employed by previous studies.