NM2 Using Randomized and Observational DATA to Predict Heterogeneous Treatment Effects

Objective: To estimate the long-term comparative effectiveness research (CER) of first-line treatment with glatiramer acetate (GA), fingolimod, or natalizumab, for relapse-remitting multiple sclerosis (RRMS). Background No randomized controlled trial (RCT) compares all disease modifying therapies (DMT) for RRMS. Further, RCTs lack evidence on long-term (> 2 years) clinical outcomes. Clinical equipoise suggests a critical need to estimate CER evidence among older and newer DMTs. Design/Methods: We developed a simulation model to estimate the average 20-year comparative effectiveness of GA, fingolimod, and natalizumab for a cohort of 30 year olds with RRMS. Key inputs to the model included published natural history progressions within the Expanded Disability Status Scale (EDSS), treatment effects from RCTs on slowing disease progression and reducing relapse rates, risk of progressive multifocal leukoencephalopathy (PML), and utility preference scores. Outputs were DMT-specific long-term harm (PML risk) and benefits (average relapse rate and time to disability (EDSS ≥ 7)) and quality-adjusted life years (QALYs). Results: For treatment duration of 20 years, average per-patient relapses were: 8.8, 5.9, and 4.1, average time (years) to disability was: 18.4, 18.9, 19.0, for GA, fingolimod, and natalizumab respectively. Average QALYs were: 11.1, 11.8, and 12.1 for GA, fingolimod, and natalizumab respectively. Compared to GA, fingolimod resulted in 2.9 fewer relapses, 0.5 more years of disability free time, and an incremental 0.7 QALYs gained. Compared to GA, natalizumab resulted in 4.6 fewer relapses, 0.6 more years of disability free time, 0.017 more cases of PML per treated patient, and an incremental 1.0 QALYs gained. Conclusions: Long-term clinical benefit outcomes were favored by natalizumab treatment, but must be weighed against an increased risk of PML. The QALY outcome, a composite measure that incorporates both benefits and harms, was favored by natalizumab treatment. CER models like this can be used to compare all DMTs to provide projections about long-term harms and benefits. Supported by: Funded in part by The Agency for Healthcare Research and Quality as part of the Colorado Comparative Effectiveness Research K12 Training Program: K12 HS019464. Disclosure: Dr. Campbell has received personal compensation for activities with Veritech Inc and Amgen Inc for consulting services. Dr. McQueen has nothing to disclose. Dr. Corboy has received personal compensation for activities with ProCE, Celgene Therapeutics, and Teva Neuroscience. Dr. Corboy has received personal compensation in an editorial capacity for Neurology in Clinical Practice. Dr. Corboy has received research from BioMS, Genentech, Inc., Orasi Medical, Eli Lilly & Company, Celgene Therapeutics, and Immune Tolerance Network. Dr. Vollmer has received personal compensation for activities with Teva Neuroscience, Daiichi Pharmaceutical Corporation, Eli Lilly & Company, Sanofi-Aventis Pharmaceuticals, Inc., PRIME Education, Inc., Projects in Knowledge, Guidepoint Global, Eisai Inc., Biogen Idec, Xenoport, Novartis, Schering-Plough Biopharma, Ono Pharmaceutical, Elan Corporation, Acorda, IMPAX, GlaxoSmithKline, Inc., and EMD Serono. Dr. Vollmer has recieved research support from EMD Serono, Teva Neuroscience, Novartis, Daiichi Pharmaceutical Corporation, Genzyme Corporation, Ono Pharmaceutical, Eli Lilly & Company, Sanofi-Aventis Pharmaceuticals, Inc., IMPAX, and Biogen Idec. Dr. Nair received has received personal compensation for activities with Centercor as a consultant.

Background and importancePharmacists are well positioned to improve clinical outcomes for patients by assisting with individualised patient in the outpatient setting. Given the difficulty of measuring the health outcome of...

Since the first approved parenteral drug for the treatment of multiple sclerosis (MS) in 1993 (interferon [IFN] beta, and later glatiramer acetate [GA]), today there are both parenteral and oral treatment options for MS. After IFN beta preparations, glatiramer acetate was developed; and, until the approval of natalizumab in 2006, those dominated the treatment of MS. Later on, among oral drug options, cladribine made a promising entry; however, due to safety concerns, it was withdrawn soon. Afterwards, with the understanding of the role of sphingosine-1 phosphate (S1P) receptors in the pathogenesis of MS, fingolimod was approved in 2010, which was followed by other oral agents such as teriflunomide and dimethyl fumarate. Recently newer IV treatment options such as alemtuzumab, rituximab and ocrelizumab have widened the treatment arena. Recently, after submitting new efficacy and safety data, cladribine was approved for MS by EMA, in 2017. Moreover, seven years after its rejection due to safety reasons, in August 2018 FDA accepted to re-evaluate the data of cladribine as a treatment option for relapsing remitting MS (RRMS). Another oral treatment option, Laquinimod, was not approved because it could not be shown to slow disability progression despite favourable effect in relapsing MS. Newer generation S1P receptor modulators are being investigated currently, and they are expected to come into the treatment arena soon. In this article, mechanisms of actions, clinical trial results, and side effects of the newer drugs used for MS, are reviewed. IFN beta and glatiramer acetate were not included since they have clinical experience nearing 30 years.

Randomized controlled trials are the gold standard for estimating the average effect of a treatment in a target population, but the same treatment may benefit some patients while having no effect on or even harming others. This phenomenon, termed heterogeneous treatment effects, can be quantified by estimating treatment effects within subgroups of patients, defined by various combinations of baseline covariates. One approach for quantifying heterogeneous treatment effects is to develop "effect models" that directly model complex interactions between baseline covariates and treatment assignment. "Effect scores," derived from effect models, can then be used to rank patients based on their predicted treatment benefit, enabling targeted treatment regimens. In this article, we provide a rigorous general framework for developing and evaluating effect models to characterize heterogeneous treatment effects from a single randomized control trial. We address challenges in valid model development, such as overfitting, and illustrate our approach in a real-world dataset with time-to-event outcomes subject to right-censoring.

Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically measure how every code or feature change impacts millions of users with great heterogeneity (e.g. countries, ages, devices). The most commonly used A/B testing framework in many companies is based on Average Treatment Effect (ATE), which cannot detect the heterogeneity of treatment effect on users with different characteristics. In this paper, we propose statistical methods that can systematically and accurately identify Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g. mobile device type, country), and determine which factors (e.g. age, gender) of users contribute to the heterogeneity of the treatment effect in an A/B test. By applying these methods on both simulation data and real-world experimentation data, we show how they work robustly with controlled low False Discover Rate (FDR), and at the same time, provides us with useful insights about the heterogeneity of identified user groups. We have deployed a toolkit based on these methods, and have used it to measure the Heterogeneous Treatment Effect of many A/B tests at Snap.

Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference research. However, when outcomes are latent variables assessed via psychometric instruments such as educational tests, standard methods ignore the potential HTE that may exist among the individual items of the outcome measure. Failing to account for “item‐level” HTE (IL‐HTE) can lead to both underestimated standard errors and identification challenges in the estimation of treatment‐by‐covariate interaction effects. We demonstrate how Item Response Theory (IRT) models that estimate a treatment effect for each assessment item can both address these challenges and provide new insights into HTE generally. This study articulates the theoretical rationale for the IL‐HTE model and demonstrates its practical value using 75 datasets from 48 randomized controlled trials containing 5.8 million item responses in economics, education, and health research. Our results show that the IL‐HTE model reveals item‐level variation masked by single‐number scores, provides more meaningful standard errors in many settings, allows for estimates of the generalizability of causal effects to untested items, resolves identification problems in the estimation of interaction effects, and provides estimates of standardized treatment effect sizes corrected for attenuation due to measurement error.

Although a growing number of political scientists are conducting randomized experiments, many of them only report the average treatment effects and do not systematically explore the variation in treatment effects across subpopulations. This is unfortunate from a scientific point of view because heterogeneous treatment effects can provide additional substantive insights. This current state of affairs is also problematic from a policy makers' perspective since such studies do not identify subgroups for which treatments are effective. In this paper, we propose a formal two-step framework that first identifies heterogeneous treatment effects from a randomized experiment and then uses this information to derive an optimal policy about which treatment should be given to whom. Our proposed method avoids the risk of false discoveries that are likely in post hoc subgroup analysis routinely conducted in the discipline. We discuss our methodology in the context of get-out-the-vote randomized field experiments and show how the proposed two-step framework can be applied in real-world settings.

Lifting the veil off treatment effect heterogeneity

Unequal cluster sizes are common in cluster randomized trials (CRTs). While there are a number of previous investigations studying the impact of unequal cluster sizes on the power for testing the average treatment effect in CRTs, little is known about the impact of unequal cluster sizes on the power for testing the heterogeneous treatment effect (HTE) in CRTs. In this work, we expand the sample size procedures for studying HTE in CRTs to accommodate cluster size variation under the linear mixed model framework. Through analytical derivation and graphical exploration, we show that the sample size for the HTE with an individual-level effect modifier is less affected by unequal cluster sizes than with a cluster-level effect modifier. The impact of cluster size variability jointly depends on the mean and coefficient of variation of cluster sizes, covariate intraclass correlation coefficient (ICC) and the conditional outcome ICC. In addition, we demonstrate that the HTE-motivated analysis of covariance framework can be used for analyzing the average treatment effect, and offer a more efficient sample size procedure for studying the average treatment effect adjusting for the effect modifier. We use simulations to confirm the accuracy of the proposed sample size procedures for both the average treatment effect and HTE in CRTs. Extensions to multivariate effect modifiers are provided and our procedure is illustrated in the context of the Strategies to Reduce Injuries and Develop Confidence in Elders trial.

250 μg or 500 μg interferon beta-1b versus 20 mg glatiramer acetate in relapsing-remitting multiple sclerosis: a prospective, randomised, multicentre study

ObjectiveThe Escala Study evidenced that the administration of glatiramer acetate for relapsing-remitting multiple sclerosis improved the spasticity of patients previously treated with interferon-β. However, whether such an improvement was translated into cost savings remained unclear. We therefore conducted a cost analysis of glatiramer acetate versus interferon-β in these patients with multiple sclerosis and spasticity.MethodsThis cost analysis encompassed data from the observational Escala Study, which included patients with relapsing-remitting multiple sclerosis and spasticity whose treatment had been switched from interferon-β to glatiramer acetate. Costs prior to starting glatiramer acetate (interferon-β period) were compared to the subsequent six months on glatiramer acetate (glatiramer acetate period). The analysis was carried out following the recommendations for conducting pharmacoeconomic studies and from the Spanish National Health System perspective. Costs associated with multiple sclerosis treatment, spasticity treatment and relapse management were expressed in 2014 euros (€); a 7.5 % discount was applied—when needed—as stipulated in Spanish law.ResultsThe management of relapsing-remitting multiple sclerosis, spasticity and relapses accounted for a 6-month cost per patient of 7,078.02€ when using interferon-β and 4,671.31€ when using glatiramer acetate. Switching from interferon-β to glatiramer acetate therefore represented a cost saving of 2,406.72€ per patient in favour of glatiramer acetate, which resulted from savings in treatment costs, relapse management and spasticity treatment of 1,890.02€, 430.48€ and 86.21€, respectively. The ratio of the costs during interferon-β was 1.5 times the costs during glatiramer acetate; thus, a fixed budget of 5,000,000€ would enable 1,070 patients to be treated with glatiramer acetate and only 706 patients with interferon-β.ConclusionsThe treatment of relapsing-remitting multiple sclerosis with glatiramer acetate entailed cost savings when compared to interferon-β in patients with spasticity, which not only resulted from its lower costs of therapy and relapse management but also from its favourable effect on reducing spasticity. Thus, glatiramer acetate may be regarded as a more efficient alternative than interferon-β from the perspective of the Spanish National Health System.

BackgroundBecause of methodological challenges comparisons of multiple treatments in multiple sclerosis cohorts have been limited to pairwise and triple comparisons.ObjectivesExtend marginal structural models (MSM) to allow simultaneous comparisons of multiple...

IntroductionIn multiple sclerosis (MS), treatment start or switch is prompted disease activity, often represented by relapses. Immunomodulatory therapies have potent effects on relapse rates but the time required to attain maximal effect is unclear. We aim to develop a method that allows identification of the time to full clinically manifest effect of treatment on relapses.MethodsData from MSBase, a multinational MS registry, were used. Inclusion criteria consisted of patients with remitting relapsing MS or clinically isolated syndrome (CIS), minimum 3-year pre-treatment follow up, 1-year treatment persistence, yearly review and availability of the minimum dataset. Stratified by therapy, density curves representing relapses occurrence were created. The first local minimum of the first derivative after treatment start was identified, representing stabilisation of treatment effect. Similar method was utilised to calculate the last pre-treatment point of stabilisation. Annualised relapse rates (ARR) were compared in the pre-treatment pre stabilisation and post-treatment post stabilisation periods.Results4979 eligible patients with 6218 treatment epochs were identified for analysis. Time, in years, to treatment effect was shortest for interferon beta-1a sc (0.22, 0.19–0.22), interferon beta-1b (0.24, 0.21–0.24) and fingolimod (0.26, 0.23–0.26) and longest for dimethyl fumarate (0.54, 0.51–0.54) and glatiramer acetate (0.62, 0.60–0.62). Significant differences in pre vs post treatment ARR were present for patients on natalizumab, fingolimod and dimethyl fumarate. A sequential analysis confirmed outcome stability after approximately 1000 recorded number of events.ConclusionsWe have developed a method to objectively quantify time from commencing therapy to its full effect. Time to full effect varies among therapies.

BackgroundNew frontline oral therapies for relapsing remitting multiple sclerosis (RRMS) are available. The change to the oral route of administration could increase adherence.1PurposeThe aim was to assess patient adherence to...

In cluster-randomized trials, sometimes the effect of the intervention being studied differs between clusters, commonly referred to as treatment effect heterogeneity. In the analysis of stepped wedge and cluster-randomized crossover trials, it is possible to include terms in outcome regression models to allow for such treatment effect heterogeneity yet this is not frequently considered. Outside of some simulation studies of specific cases where the outcome is binary, the impact of failing to include terms for treatment effect heterogeneity on the variance of the treatment effect estimator is unknown. We analytically examine the impact of failing to include terms for treatment effect heterogeneity on the variance of the treatment effect estimator, when outcomes are continuous. Using analysis of variance and feasible generalized least squares we provide expressions for this variance. For both the cluster-randomized crossover design and the stepped wedge design, our analytic derivations indicate that failing to include treatment effect heterogeneity results in the estimates for variance of the treatment effect that are too small, leading to inflation of type I error rates. We therefore recommend assessing the sensitivity of sample size calculations and conclusions drawn from the analysis of cluster randomized trials to the inclusion of treatment effect heterogeneity.