Randomization-based covariance adjustment of win ratios and win odds for randomized multi-visit studies with ordinal outcomes

Generalized pairwise comparison (GPC) methods are extensions of the Mann-Whitney approach that allow comparisons of outcomes through prioritized ranking, and they have been widely applied in randomized controlled trials (RCTs). Importantly, GPC methods can be adapted to handle censored time-to-event data. GPC methods are based on assigning scores to pairs of subjects where all pairs of treatment and control subjects are evaluated: the outcome of each treatment subject is compared with each control subject. The GPC test statistic can be expressed as a treatment effect for the therapeutic intervention by measures such as the net benefit, win odds, win ratio (WR), or probability index. As the focus for this study, the WR has an alternative interpretation as the inverse of the hazard ratio under proportional hazards. However, its estimate could be biased in the presence of substantial censoring. Censoring increases the number of indeterminate treatment and control pairs, where the win or loss is undetermined due to the censored observation(s) and a definitive score cannot be assigned. We propose a novel method leveraging pseudo-observations to address the issue of uninformative pairs resulting from censoring for a time-to-event outcome. We compare the performance of our method with existing GPC methods in simulations under various censoring scenarios. For equal drop-out and administrative censoring, our method provides results that are comparable to existing GPC methods. However, for unequal drop-out, which is common in clinical trials, the performance of our approach relative to existing methods depends on the censoring proportion and distribution. The proposed approach reduced bias and root mean squared error relative to Gehan and Latta under several censoring conditions, but these improvements did not extend to gains in statistical power. Lastly, we illustrate this new GPC approach using two reconstructed RCT datasets.

The rapid finding of effective therapeutics requires efficient use of available resources in clinical trials. Covariate adjustment can yield statistical estimates with improved precision, resulting in a reduction in the number of participants required to draw futility or efficacy conclusions. We focus on time‐to‐event and ordinal outcomes. When more than a few baseline covariates are available, a key question for covariate adjustment in randomised studies is how to fit a model relating the outcome and the baseline covariates to maximise precision. We present a novel theoretical result establishing conditions for asymptotic normality of a variety of covariate‐adjusted estimators that rely on machine learning (e.g., ℓ1‐regularisation, Random Forests, XGBoost, and Multivariate Adaptive Regression Splines [MARS]), under the assumption that outcome data are missing completely at random. We further present a consistent estimator of the asymptotic variance. Importantly, the conditions do not require the machine learning methods to converge to the true outcome distribution conditional on baseline variables, as long as they converge to some (possibly incorrect) limit. We conducted a simulation study to evaluate the performance of the aforementioned prediction methods in COVID‐19 trials. Our simulation is based on resampling longitudinal data from over 1500 patients hospitalised with COVID‐19 at Weill Cornell Medicine New York Presbyterian Hospital. We found that using ℓ1‐regularisation led to estimators and corresponding hypothesis tests that control type 1 error and are more precise than an unadjusted estimator across all sample sizes tested. We also show that when covariates are not prognostic of the outcome, ℓ1‐regularisation remains as precise as the unadjusted estimator, even at small sample sizes (n=100). We give an R package adjrct that performs model‐robust covariate adjustment for ordinal and time‐to‐event outcomes.

The propensity score is defined to be a subject's probability of treatment selection, conditional on observed baseline covariates. Conditional on the propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. In the medical literature, there are three commonly employed propensity-score methods: stratification (subclassification) on the propensity score, matching on the propensity score, and covariate adjustment using the propensity score. Methods have been developed to assess the adequacy of the propensity score model in the context of stratification on the propensity score and propensity-score matching. However, no comparable methods have been developed for covariate adjustment using the propensity score. Inferences about treatment effect made using propensity-score methods are only valid if, conditional on the propensity score, treated and untreated subjects have similar distributions of baseline covariates. We develop both quantitative and qualitative methods to assess the balance in baseline covariates between treated and untreated subjects. The quantitative method employs the weighted conditional standardized difference. This is the conditional difference in the mean of a covariate between treated and untreated subjects, in units of the pooled standard deviation, integrated over the distribution of the propensity score. The qualitative method employs quantile regression models to determine whether, conditional on the propensity score, treated and untreated subjects have similar distributions of continuous covariates. We illustrate our methods using a large dataset of patients discharged from hospital with a diagnosis of a heart attack (acute myocardial infarction). The exposure was receipt of a prescription for a beta-blocker at hospital discharge.

Benkeser et al. present a very informative paper evaluating the efficiency gains of covariate adjustment in settings with binary, ordinal, and time‐to‐event outcomes. The adjustment method focuses on estimating the marginal treatment effect averaged over the covariate distribution in both arms combined. The authors show that covariate adjustment can achieve power gains that could find answers more quickly. The suggested approach is an important weapon in the armamentarium against epidemics like COVID‐19. I recommend evaluating the procedure against more traditional approaches for conditional analyses (e.g., logistic regression) and against blinded methods of building prediction models followed by randomization‐based inference.

Ordinal variables, such as those measured on a five-point Likert scale, are ubiquitous in the behavioral sciences. However, machine learning methods for modeling ordinal outcome variables (i.e., ordinal classification) are not as well-developed or widely utilized, compared to classification and regression methods for modeling nominal and continuous outcomes, respectively. Consequently, ordinal outcomes are often treated “naively” as nominal or continuous outcomes in practice. This study builds upon previous literature that has examined the predictive performance of such naïve approaches of treating ordinal outcome variables compared to ordinal classification methods in machine learning. We conducted a Monte Carlo simulation study to systematically assess the relative predictive performance of an ordinal classification approach proposed by Frank and Hall (2001) against naïve approaches according to two key factors that have received limited attention in previous literature: (1) the machine learning algorithm being used to implement the approaches and (2) the class distribution of the ordinal outcome variable. The consideration of these important, practical factors expands our knowledge on the consequences of naïve treatments of ordinal outcomes, which are shown in this study to vary substantially according to these factors. Given the ubiquity of ordinal measures coupled with the growing presence of machine learning applications in the behavioral sciences, these are important considerations for building high-performing predictive models in the field.

Introduction: Standard analysis of clinical trials with composite endpoints compare time-to-first event. Subsequent events are not counted and therefore not taken into account in treatment comparisons. We compared standard analyses to analyses that take into account all the events a patient may have experienced. Methods: The Trial to Assess Chelation Therapy (TACT) enrolled 1708 post-MI pts in a factorial trial comparing edetate disodium infusions versus placebo infusions and oral vitamins and minerals versus oral placebo. TACT had a composite primary endpoint: death, MI, stroke, coronary revascularization, and hospitalization for angina. Pre-specified primary analysis for TACT was based on the standard approach of considering time until the occurrence of the first event. Two different approaches for multiple events analyses were compared to the standard approach: 1] a generalization of the Cox model to handle recurrent events, developed by Andersen & Gill (AG), and 2] a marginal modeling approach developed by Wie, Lin, and Weissfeld (WLW). Each considers the correlation among multiple events occurring within a pt. Results: There were 743 events in TACT. 483/1708 (28%) of the patients had at least 1 event. Time-to-first-event analysis did not use information on 260 (35%) events. The standard analysis for edetate disodium vs placebo produced a hazard ratio (HR) 0.82, 95% confidence intervals 0.69,0.99, p=0.035. Both multiple events analyses demonstrated narrower confidence intervals and lower P values. AG: HR 0.80 (0.66,0.96), p=0.017; and WLW: HR 0.79 (0.66, 0.95), p=0.010. In the comparison of the active edetate + active MVM vs placebo edetate + placebo MVM, there were 247/858 (29%) pts with at least 1 event, but there were 371 events. Therefore, 124 (33%) were not used. Results of the standard analysis for edetate + MVM showed HR 0.74 (0.57,0.95), p=0.016; for AG HR 0.66 (0.51,0.85),p=0.002, and for WLW HR 0.38 (0.30,0.49), p<0.001. Conclusion: In this post-MI trial, a standard time to first event analysis did not make use of over 1/3 of events. Multiple events analysis can improve the precision of the estimate of benefit by producing narrower confidence intervals. These methods could be considered to reduce sample size and budgets in future clinical trials.

In ROC analysis, covariate adjustment is advocated when the covariates impact the magnitude or accuracy of the test under study. Meanwhile, for many large scale screening tests, the true condition status may be subject to missingness because it is expensive and/or invasive to ascertain the disease status. The complete-case analysis may end up with a biased inference, also known as "verification bias." To address the issue of covariate adjustment with verification bias in ROC analysis, we propose several estimators for the area under the covariate-specific and covariate-adjusted ROC curves (AUCx and AAUC). The AUCx is directly modeled in the form of binary regression, and the estimating equations are based on the U statistics. The AAUC is estimated from the weighted average of AUCx over the covariate distribution of the diseased subjects. We employ reweighting and imputation techniques to overcome the verification bias problem. Our proposed estimators are initially derived assuming that the true disease status is missing at random (MAR), and then with some modification, the estimators can be extended to the not missing at random (NMAR) situation. The asymptotic distributions are derived for the proposed estimators. The finite sample performance is evaluated by a series of simulation studies. Our method is applied to a data set in Alzheimer's disease research.

The value of environmental goods to individuals often depends on spatial features such as distance. The most common approach of accounting for distance decay is to model utility as some function of distance. It has been suggested to instead model the value as a function of the quantity of an environmental good within a certain distance. We develop three novel quantity-within-distance models that may be more suited for evaluating quantity changes in an environmental good. We argue that these models could capture spatial patterns better than distance-based models when i) secondary benefits are a relevant source of welfare, ii) the environmental change is spatially scattered, iii) the distribution of the endowment, i.e. the present availability of the environmental good, matters. Using data from choice experiments on the extension of green space and trees in two urban areas, we compare required assumptions, model fit, and size and precision of aggregated welfare estimates. Our results indicate limited differences in model fit. However, the quantity-within-distance models consistently produce aggregate welfare estimates roughly half of common distance decay models and have narrower confidence intervals. While it is not possible to infer which is more accurate, the large differences can have considerable policy implications.

Application of Refined Exposure Surrogates in Ambient PM2.5 Epidemiology Studies: Accounting for Temporal/Spatial Variations in Particulate Matter Infiltration

Summary In this study, we show significant improvements over conventional pressure-transient analysis on the basis of nonlinear regression by using total least squares (TLS), which minimizes errors in both pressure and time simultaneously. To our knowledge, TLS has not been applied to pressure-transient analysis before in this sense. TLS regression is not an easy problem to solve mathematically, especially for nonlinear pressure-transient-model functions. In this work, we compare four different versions of TLS. We formulate a robust approximation of the TLS solution, which can handle a variety of tests and reservoir models yet does not compromise the performance of the analysis. We show that our technique reduces ambiguity in the estimation of parameters to a large extent, especially in the presence of noise in time. Using our TLS algorithm, we obtain much narrower confidence intervals on the parameter estimates of a variety of real data sets, compared to the conventional least squares (LS) approach. For synthetic data sets, we observe that the TLS estimates are often closer to the true values than estimates made with LS, especially for poorly determined problems. When the deviation is in pressure only, TLS and LS results are comparable. However, in the presence of deviations in time in addition to pressure, the performance of TLS algorithms is substantially better. We, therefore, expect that our technique will provide more accurate estimation of reservoir parameters, allowing for better forecasting of reservoir performance.

Because original high-quality non-market valuation studies can be expensive, perhaps prohibitively so, benefits transfer (BT) approaches are often used for valuing, e.g., the outputs of multifunctional agriculture. Here we focus on the use of BT functions, a preferred method, and address an under-appreciated problem – variable selection uncertainty – and demonstrate a conceptually superior method of resolving it. We show that the standard method of value-function BT, using the full estimated model, may generate BT values that are too sensitive to insignificant variables, whereas models reduced by backward elimination of insignificant variables pay no attention to insignificant variables that may in fact have some influence on values. Rather than searching for the best single model for BT, Bayesian model averaging (BMA) is attentive to all of the variables that are a priori relevant, but uses posterior model probabilities to give systematically lower weight to less significant variables. We estimate a full value model for wetlands in the US, and then calculate BT values from the full model, a reduced model, and by BMA. Variable selection uncertainty is exemplified by regional variables for wetland location. Predicted values from the full model are quite sensitive to region; reduced models pay no attention to regional variables; and the BMA predictions are attentive to region but give it relatively low weight. However, the suite of insignificant RHS variables, taken together, have non-trivial influence on BT values. BMA predicted values, like values from reduced models, have much narrower confidence intervals than values calculated from the full model.

The aim of this study was to investigate the use of nonlinear mixed effects (NLME) models in a real bioequivalence study and compare it to noncompartmental analysis (NCA) which is proposed by regulatory agencies. NCA requires few hypotheses but a large number of samples per subject. On the other hand, NLME approach is more complex than NCA but it has some advantages such as it requires few samples per subject. A real data application was provided for the study, which was get from Ege University Drug Development and Pharmacokinetics Research Center. For NLME, we used the stochastic approach to expectation-maximization (SAEM) algorithm, whereas linear trapezoidal rule was used for NCA. We estimate pharmacokinetic parameters, area under the curve (AUC0-∞) and maximum concentration (Cmax), and perform a bioequivalence tests using NCA and NLME. According to real data analysis, NLME approach has smaller within subject error, narrower confidence intervals than non-compartmental analysis. However, NLME models have some limitations because of increasing type I error. Therefore, caution is needed for small sample size and data with high variability.

We are interested in the risk of large losses of certain common financial portfolios (e.g., credit portfolios with default risk). In these cases, we would like to estimate a risk statistic called Value-at-Risk (VaRα) at an extremely high risk level α (typically 99.99%). This high risk level corresponds to rare events for which it is difficult or impossible to obtain data. We first use Monte Carlo simulation to compute a probability distribution for the portfolio loss. We then use Extreme Value Theory (EVT) to study the tail of this loss distribution. Finally we compute the VaR and associated confidence intervals using bootstrap techniques. For the portfolios under consideration, we have observed that the EVT-based approach results in narrower confidence intervals and hence less sampling uncertainty in computing the VaR. We have also observed that the bootstrap replicate's distribution for the EVT-based method demonstrates a better shape than the empirical method (which is typically very noisy).

Urinary incontinence is common after both radical prostatectomy and transurethral resection of the prostate (TURP). Conservative management includes pelvic floor muscle training with or without biofeedback, electrical stimulation, extra-corporeal magnetic innervation (ExMI), compression devices (penile clamps), lifestyle changes, or a combination of methods. To assess the effects of conservative management for urinary incontinence after prostatectomy. We searched the Cochrane Incontinence Group Specialised Register (searched 24 August 2011), EMBASE (January 1980 to Week 48 2009), CINAHL (January 1982 to 20 November 2009), the reference lists of relevant articles, handsearched conference proceedings and contacted investigators to locate studies. Randomised or quasi-randomised controlled trials evaluating conservative interventions for urinary continence in men after prostatectomy. Two or more review authors assessed the methodological quality of trials and abstracted data. We tried to contact several authors of included studies to obtain extra information. Thirty-seven trials met the inclusion criteria, 33 amongst men after radical prostatectomy, three trials after transurethral resection of the prostate (TURP) and one trial after either operation. The trials included 3399 men, of whom 1937 had an active conservative intervention. There was considerable variation in the interventions, populations and outcome measures. Data were not available for many of the pre-stated outcomes. Men's symptoms improved over time irrespective of management. Adverse effects did not occur or were not reported.There was no evidence from eight trials that pelvic floor muscle training with or without biofeedback was better than control for men who had urinary incontinence after radical prostatectomy (e.g. 57% with urinary incontinence versus 62% in the control group, risk ratio (RR) for incontinence after 12 months 0.85, 95% confidence interval (CI) 0.60 to 1.22) as the confidence intervals were wide, reflecting uncertainty.However, one large multicentre trial of one-to-one therapy showed no difference in any urinary or quality of life outcome measures and had narrower confidence intervals. There was also no evidence of benefit for erectile dysfunction (56% with no erection in the pelvic floor muscle training group versus 55% in the control group after one year, RR 1.01, 95% CI 0.84 to 1.20). Individual small trials provided data to suggest that electrical stimulation, external magnetic innervation or combinations of treatments might be beneficial but the evidence was limited.One large trial demonstrated that there was no benefit for incontinence or erectile dysfunction from a one-to-one pelvic floor muscle training based intervention to men who were incontinent after transurethral resection of the prostate (TURP) (e.g. 65% with urinary incontinence versus 62% in the control group, RR after 12 months 1.05, 95% CI 0.91 to 1.23).In eight trials of conservative treatment of all men after radical prostatectomy aimed at both treatment and prevention, there was an overall benefit from pelvic floor muscle training versus control management in terms of reduction of UI (e.g. 10% with urinary incontinence after one year versus 32% in the control groups, RR for urinary incontinence 0.32, 95% CI 0.20 to 0.51). However, this finding was not supported by other data from pad tests. The findings should be treated with caution, as most trials were of poor to moderate quality and confidence intervals were wide.Men in one trial were more satisfied with one type of external compression device, which had the lowest urine loss, compared to two others or no treatment. The effect of other conservative interventions such as lifestyle changes remains undetermined as no trials involving these interventions were identified. The value of the various approaches to conservative management of postprostatectomy incontinence after radical prostatectomy remains uncertain. It seems unlikely that men benefit from one-to-one pelvic floor muscle training therapy after transurethral resection of the prostate (TURP). Long-term incontinence may be managed by external penile clamp, but there are safety problems.

As more and more data are available, training a machine learning model can be extremely intractable, especially for complex models like Support Vector Regression (SVR) training of which requires solving a large quadratic programming optimization problem. Selecting a small data subset that can effectively represent the characteristic features of training data and preserve their distribution is an efficient way to solve this problem. This paper proposes a systematic approach to select the best representative data for SVR training. The distributions of both predictor and response variables are preserved in the selected subset via a 2-layer data clustering strategy. A 2-layer step-wise greedy algorithm is introduced to select best data points for constructing a reduced training set. The proposed method has been applied for predicting deck’s win rates in the Clash Royale Challenge, in which 10 subsets containing hundreds of data examples were selected from 100k for training 10 SVR models to maximize their prediction performance evaluated using R-squared metric. Our final submission having a R2 score of 0.225682 won the 3rd place among over 1200 solutions submitted by 115 teams.