Comparing methods to estimate treatment effects on a continuous outcome in multicentre randomized controlled trials: a simulation study.

BackgroundThe classical linear model is widely used in the analysis of clinical trials with continuous outcomes. However, required model assumptions are frequently not met, resulting in estimates of treatment effect that can be inefficient and biased. In addition, traditional models assess treatment effect only on the mean response, and not on other aspects of the response, such as the variance. Distributional regression modelling overcomes these limitations. The purpose of this paper is to demonstrate its usefulness for the analysis of clinical trials, and superior performance to that of traditional models.MethodsDistributional regression models are demonstrated, and contrasted with normal linear models, on data from the LIPID randomized controlled trial, which compared the effects of pravastatin with placebo in patients with coronary heart disease. Systolic blood pressure (SBP) and the biomarker midregional pro-adrenomedullin (MR-proADM) were analysed. Treatment effect was estimated in models that used response distributions more appropriate than the normal (Box-Cox-t and Johnson’s Su for MR-proADM and SBP, respectively), applied censoring below the detection limit of MR-proADM, estimated treatment effect on distributional parameters other than the mean, and included random effects for longitudinal observations. A simulation study was conducted to compare the performance of distributional regression models with normal linear regression, under conditions mimicking the LIPID study. The R package gamlss (Generalized Additive Models for Location, Scale and Shape), which implements maximum likelihood estimation for distributional regression modelling, was used throughout.ResultsIn all cases the distributional regression models fit the data well, in contrast to poor fits obtained for traditional models; for MR-proADM a small but significant treatment effect on the mean was detected by the distributional regression model and not the normal model; and for SBP a beneficial treatment effect on the variance was demonstrated. In the simulation study distributional models strongly outperformed normal models when the response variable was non-normal and heterogeneous; and there was no disadvantage introduced by the use of distributional regression modelling when the response satisfied the normal linear model assumptions.ConclusionsDistributional regression models are a rich framework, largely untapped in the clinical trials world. We have demonstrated a sample of the capabilities of these models for the analysis of trials. If interest lies in accurate estimation of treatment effect on the mean, or other distributional features such as variance, the use of distributional regression modelling will yield superior estimates to traditional normal models, and is strongly recommended.Trial registrationThe LIPID trial was retrospectively registered on ANZCTR on 27/04/2016, registration number ACTRN12616000535471.

Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias), provided that a sufficient number of people are randomised. To assess the effects of randomisation and concealment of allocation on the results of healthcare trials. We searched the Cochrane Methodology Register, MEDLINE, SciSearch, reference lists up to August 2000 and used personal communication. Cohorts of trials, systematic reviews or meta-analyses of healthcare interventions that compared outcomes or prognostic factors for one of the following comparisons: randomised versus non-randomised trials, randomised trials with adequately versus inadequately concealed allocation, or high versus low quality trials where selection bias could not be separated from other sources of bias. One of us went through all of the citations in the Cochrane Methodology Register and accumulated reference lists. Studies that appeared to meet the inclusion criteria were retrieved and assessed independently by two of the reviewers. The methodological quality of included studies was appraised and information extracted by one of us and checked by a second. Tabular summaries of the results were prepared for each comparison and the results across studies were assessed qualitatively to identify common trends or discrepancies. We identified 32 studies including over 3000 trials. Twenty-two studies compared randomised versus non-randomised trials, three compared adequately versus inadequately concealed allocation, and nine compared high versus low quality trials (some studies included more than one comparison). Five studies were of high methodological quality. In 15 of the 22 studies that compared randomised and non-randomised trials of the same intervention, important differences were found in the estimates of effect. Some of these differences were due to a poorer prognosis in the control groups in the non-randomised trials. The results of the other seven studies that compared randomised and non-randomised trials across different interventions are less clear. Comparisons of adequately and inadequately concealed allocation in randomised trials of the same intervention provided high quality evidence that concealment can be crucial in achieving similar treatment groups and, therefore, unbiased estimates of treatment effects. Studies with inadequate concealment tended to overestimate treatment effects. Comparisons of high and low quality trials of the same intervention have found important differences in estimates of effect, but it is not possible to determine the extent to which these differences can be attributed to randomisation or concealment of allocation. Omitting comparisons between randomised trials and non-randomised trials using historical controls did not substantially alter the results or conclusions of our review. On average, non-randomised trials and randomised trials with inadequate concealment of allocation tend to result in larger estimates of effect than randomised trials with adequately concealed allocation. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects.

Randomized controlled trials provide unbiased estimates of treatment effects through reducing the attribution of observed differences in outcome to the play of chance or the treatment of interest. Issues in the design of randomized trials may lead to a dilution or magnification of treatment effect when compared with clinical practice. The intention-to-treat principle will lead to dilution in the treatment effect, while the selective recruitment of adherent and motivated subjects may lead to enhanced treatment effects estimates. Observational data, on the other hand, may reflect clinical practice but cannot provide unbiased estimates of treatment effect. Health policy may be informed by the combination of randomized and observational data in decision-analytic models.

BackgroundPropensity scores (PS) are typically evaluated using balance metrics that focus on covariate balance, often without considering their predictive power for the outcome. This approach may not always result in optimal bias reduction in the treatment effect estimate. To address this issue, evaluating covariate balance through prognostic scores, which account for the relationship between covariates and the outcome, has been proposed. Similarly, using a typical model averaging approach for PS estimation that minimizes prediction error for treatment status and covariate imbalance does not necessarily optimize PS-based confounding adjustment. As an alternative approach, using the averaged PS model that minimizes inter-group differences in the prognostic score may further reduce bias in the treatment effect estimate. Moreover, since the prognostic score is also an estimated quantity, model averaging in the prognostic scores can help identify a better prognostic score model. Utilizing the model-averaged prognostic scores as the balance metric for constructing the averaged PS model can contribute to further decreasing bias in treatment effect estimates. This paper demonstrates the effectiveness of the PS model averaging approach based on prognostic score balance and proposes a method that uses the model-averaged prognostic score as a balance metric, evaluating its performance through simulations and empirical analysis.MethodsWe conduct a series of simulations alongside an analysis of empirical observational data to compare the performances of weighted treatment effect estimates using the proposed and existing approaches. In our examination, we separately provid four candidate estimates for the PS and prognostic score models using traditional regression and machine learning methods. The model averaging of PS based on these candidate estimators is performed to either maximize the prediction accuracy of the treatment or to minimize intergroup differences in covariate distributions or prognostic scores. We also utilize not only the prognostic scores from each candidate model but also an averaged score that best predicted the outcome, for the balance assessment.ResultsThe simulation and empirical data analysis reveal that our proposed model-averaging approaches for PS estimation consistently yield lower bias and less variability in treatment effect estimates across various scenarios compared to existing methods. Specifically, using the optimally averaged prognostic scores as a balance metric significantly improves the robustness of the weighted treatment effect estimates.DiscussionThe prognostic score-based model averaging approach for estimating PS can outperform existing model averaging methods. In particular, the estimator using the model averaging prognostic score as a balance metric can produce more robust estimates. Since our results are obtained under relatively simple conditions, applying them to real data analysis requires adjustments to obtain accurate estimates according to the complexity and dimensionality of the data.ConclusionsUsing the prognostic score as the balance metric for the PS model averaging enhances the performance of the treatment effect estimator, which can be recommended for a wide variety of situations. When applying the proposed method to real-world data, it is important to use it in conjunction with techniques that mitigate issues arising from the complexity and high dimensionality of the data.

Trial Quality in Nephrology: How Are We Measuring Up?

Recent FDA guidance on adaptive clinical trial designs defines bias as “a systematic tendency for the estimate of treatment effect to deviate from its true value”, and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realised trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This paper is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this paper can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

Theory and simulation suggest that randomized controlled trials (RCTs) stopped early for benefit (truncated RCTs) systematically overestimate treatment effects for the outcome that precipitated early stopping. To compare the treatment effect from truncated RCTs with that from meta-analyses of RCTs addressing the same question but not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect. Search of MEDLINE, EMBASE, Current Contents, and full-text journal content databases to identify truncated RCTs up to January 2007; search of MEDLINE, Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effects to identify systematic reviews from which individual RCTs were extracted up to January 2008. Selected studies were RCTs reported as having stopped early for benefit and matching nontruncated RCTs from systematic reviews. Independent reviewers with medical content expertise, working blinded to trial results, judged the eligibility of the nontruncated RCTs based on their similarity to the truncated RCTs. Reviewers with methodological expertise conducted data extraction independently. The analysis included 91 truncated RCTs asking 63 different questions and 424 matching nontruncated RCTs. The pooled ratio of relative risks in truncated RCTs vs matching nontruncated RCTs was 0.71 (95% confidence interval, 0.65-0.77). This difference was independent of the presence of a statistical stopping rule and the methodological quality of the studies as assessed by allocation concealment and blinding. Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of relative risks <0.75) occurred with truncated RCTs having fewer than 500 events. In 39 of the 63 questions (62%), the pooled effects of the nontruncated RCTs failed to demonstrate significant benefit. Truncated RCTs were associated with greater effect sizes than RCTs not stopped early. This difference was independent of the presence of statistical stopping rules and was greatest in smaller studies.

5 - THE BALANCED TWO-WAY FACTORIAL DESIGN WITHOUT INTERACTION

Resections of the pancreatic body and tail reaching to the left of the superior mesenteric vein are defined as distal pancreatectomy. Most distal pancreatectomies are elective treatments for chronic pancreatitis, benign or malignant diseases, and they have high morbidity rates of up to 40%. Pancreatic fistula formation is the main source of postoperative morbidity, associated with numerous further complications. Researchers have proposed several surgical resection and closure techniques of the pancreatic remnant in an attempt to reduce these complications. The two most common techniques are scalpel resection followed by hand-sewn closure of the pancreatic remnant and stapler resection and closure. To compare the rates of pancreatic fistula in people undergoing distal pancreatectomy using scalpel resection followed by hand-sewn closure of the pancreatic remnant versus stapler resection and closure. We searched Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE, Biosis and Science Citation Index from database inception to October 2015. We included randomised controlled trials (RCTs) comparing stapler versus scalpel resection followed by hand-sewn closure of the pancreatic remnant for distal pancreatectomy (irrespective of language or publication status). Two authors independently assessed trials for inclusion and extracted the data. Taking into consideration the clinical heterogeneity between the trials (e.g. different endpoint definitions), we analysed data using a random-effects model with Review Manager (RevMan), calculating risk ratio (RR) or mean difference (MD) with 95% confidence intervals (CI). In two eligible trials, a total of 381 participants underwent distal pancreatic resection and were randomised to closure of the pancreatic remnant either with stapler (n = 191) or scalpel resection followed by hand-sewn closure (n = 190). One was a single centre pilot RCT and the other was a multicentre blinded RCT. The single centre pilot RCT evaluated 69 participants in five intervention arms (stapler, hand-sewn, fibrin glue, mesh and pancreaticojejunostomy), although we only assessed the stapler and hand-sewn closure groups (14 and 15 participants, respectively). The multicentre RCT had two interventional arms: stapler (n = 177) and hand-sewn closure (n = 175). The rate of postoperative pancreatic fistula was the main outcome, and it occurred in 79 of 190 participants in the hand-sewn group compared to 65 of 191 participants in the stapler group. Neither the individual trials nor the meta-analysis showed a significant difference between resection techniques (RR 0.90; 95% CI 0.55 to 1.45; P = 0.66). In the same way, postoperative mortality and operation time did not differ significantly. The single centre RCT had an unclear risk of bias in the randomisation, allocation and both blinding domains. However, the much larger multicentre RCT had a low risk of bias in all domains. Due to the small number of events and the wide confidence intervals that cannot exclude clinically important benefit or harm with stapler versus hand-sewn closure, there is a serious possibility of imprecision, making the overall quality of evidence moderate. The quality of evidence is moderate and mainly based on the high weight of the results of one multicentre RCT. Unfortunately, there are no other completed RCTs on this topic except for one relevant ongoing trial. Neither stapler nor scalpel resection followed by hand-sewn closure of the pancreatic remnant for distal pancreatectomy showed any benefit compared to the other method in terms of postoperative pancreatic fistula, overall postoperative mortality or operation time. Currently, the choice of closure is left up to the preference of the individual surgeon and the anatomical characteristics of the patient. Another (non-European) multicentre trial (e.g. with an equality or non-inferiority design) would help to corroborate the findings of this meta-analysis. Future trials assessing novel methods of stump closure should compare them either with stapler or hand-sewn closure as a control group to ensure comparability of results.

Random assignment to treatment is not always possible in evaluative research. The semi-experimental design discussed here has aspects of both full experimental and quasi-experimental designs. Monte Carlo studies are used to explore and exemplify the strengths and weaknesses of the design. The analysis suggested is found to give unbiased estimates of treatment effects and error mean squares but biased estimates of assignment effects. Some further aspects of the design and its use are discussed.

Statistical heterogeneity and small-study effects are 2 major issues affecting the validity of meta-analysis. In this article, we introduce the concept of a limit meta-analysis, which leads to shrunken, empirical Bayes estimates of study effects after allowing for small-study effects. This in turn leads to 3 model-based adjusted pooled treatment-effect estimators and associated confidence intervals. We show how visualizing our estimators using the radial plot indicates how they can be calculated using existing software. The concept of limit meta-analysis also gives rise to a new measure of heterogeneity, termed G(2), for heterogeneity that remains after small-study effects are accounted for. In a simulation study with binary data and small-study effects, we compared our proposed estimators with those currently used together with a recent proposal by Moreno and others. Our criteria were bias, mean squared error (MSE), variance, and coverage of 95% confidence intervals. Only the estimators arising from the limit meta-analysis produced approximately unbiased treatment-effect estimates in the presence of small-study effects, while the MSE was acceptably small, provided that the number of studies in the meta-analysis was not less than 10. These limit meta-analysis estimators were also relatively robust against heterogeneity and one of them had a relatively small coverage error.

When are Field Experiments with Individual Assignment Too Risky? Lessons from a Center-Based Child Care Study in Guatemala

In clinical trials with time-to-event outcomes, it is common to estimate the marginal hazard ratio from the proportional hazards model, even when the proportional hazards assumption is not valid. This is unavoidable from the perspective that the estimator must be specified a priori if probability statements about treatment effect estimates are desired. Marginal hazard ratio estimates under non-proportional hazards are still useful, as they can be considered to be average treatment effect estimates over the support of the data. However, as many have shown, under non-proportional hazard, the 'usual' unweighted marginal hazard ratio estimate is a function of the censoring distribution, which is not normally considered to be scientifically relevant when describing the treatment effect. In addition, in many practical settings, the censoring distribution is only conditionally independent (e.g., differing across treatment arms), which further complicates the interpretation. In this paper, we investigate an estimator of the hazard ratio that removes the influence of censoring and propose a consistent robust variance estimator. We compare the coverage probability of the estimator to both the usual Cox model estimator and an estimator proposed by Xu and O'Quigley (2000) when censoring is independent of the covariate. The new estimator should be used for inference that does not depend on the censoring distribution. It is particularly relevant to adaptive clinical trials where, by design, censoring distributions differ across treatment arms.

Regulatory agencies typically evaluate the efficacy and safety of new interventions and grant commercial approval based on randomized controlled trials (RCTs). Other major healthcare stakeholders, such as insurance companies and health technology assessment agencies, while basing initial access and reimbursement decisions on RCT results, are also keenly interested in whether results observed in idealized trial settings will translate into comparable outcomes in real world settings—that is, into so‐called “real world” effectiveness. Unfortunately, evidence of real world effectiveness for new interventions is not available at the time of initial approval. To bridge this gap, statistical methods are available to extend the estimated treatment effect observed in a RCT to a target population. The generalization is done by weighting the subjects who participated in a RCT so that the weighted trial population resembles a target population. We evaluate a variety of alternative estimation and weight construction procedures using both simulations and a real world data example using two clinical trials of an investigational intervention for Alzheimer's disease. Our results suggest an optimal approach to estimation depends on the characteristics of source and target populations, including degree of selection bias and treatment effect heterogeneity.

This paper considers the analysis of a repeat event outcome in clinical trials of chronic diseases in the context of dependent censoring (e.g. mortality). It has particular application in the context of recurrent heart failure hospitalisations in trials of heart failure. Semi‐parametric joint frailty models (JFMs) simultaneously analyse recurrent heart failure hospitalisations and time to cardiovascular death, estimating distinct hazard ratios whilst individual‐specific latent variables induce associations between the two processes. A simulation study was carried out to assess the suitability of the JFM versus marginal analyses of recurrent events and cardiovascular death using standard methods. Hazard ratios were consistently overestimated when marginal models were used, whilst the JFM produced good, well‐estimated results. An application to the Candesartan in Heart failure: Assessment of Reduction in Mortality and morbidity programme was considered. The JFM gave unbiased estimates of treatment effects in the presence of dependent censoring. We advocate the use of the JFM for future trials that consider recurrent events as the primary outcome. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.