The use of minimization to form comparison groups in educational research

An Argument for Conducting Methodologically Strong, Randomized, Controlled Trials in Educational Research

The pursuit of balance: An overview of covariate-adaptive randomization techniques in clinical trials

The statistical power of cross-over trials can be increased by taking 'baseline' measurements of the outcome variable at the start of each treatment period. Analysis of covariance (ANCOVA), rather than analysis of change scores, takes best advantage of this. However, ANCOVA can give biased treatment effect estimates in observational studies with true baseline imbalance. While in truth balanced, chance baseline imbalance is possible in individual randomized cross-over studies due to their typically small sample size. Although such chance imbalance does not cause biased estimation on average over repeated trials, this simulation study will aim to confirm the appropriateness of ANCOVA when faced with the analysis of data from an individual trial in which chance baseline imbalance is clearly apparent. Randomized cross-over trials were simulated, varying in sample size and the pattern and strength of correlation between repeated measures. Estimates from ANCOVA, change scores, and post-treatment difference were unbiased on average across each set of simulated data sets. ANCOVA and change scores could use baseline information to improve precision, but change scores could also reduce precision if baseline measures were uninformative. Change scores only were correlated with chance within-subject baseline imbalance. All three estimators could be correlated with chance between-subjects imbalance in the first period baseline measurements, the strongest associations being with the post-treatment difference. Consistent results were obtained from a real data example. In conclusion, ANCOVA took best advantage of baseline measures to improve precision, and avoided bias in the widest set of circumstances with chance imbalance in those baseline measures.

Concerns about the replication crisis and unreliable findings have spread through several fields, including education and psychological research. In some areas of education, researchers have begun to adopt reforms that have proven useful in other fields. These include preregistration, open materials and data, and registered reports. These reforms offer education research a path toward increased credibility and social impact. In this article, we discuss models of large-scale collaborative research practices and how they can be applied to education research. We discuss five types of large-scale collaboration: participating teams run different studies, multiteam collaboration projects, collaborative analysis, preregistered adversarial collaboration, and persistent collaboration. The combination of large-scale collaboration with open and transparent research practices offers education researchers opportunity to test theories, verify what is known about a topic, resolve disagreements, and explore new questions.

Cronbach’s alpha remains one of the most widely used reliability coefficients in educational and psychological research. However, its application is often accompanied by misinterpretations and methodological limitations. This systematic review was conducted in accordance with PRISMA 2020 guidelines to examine the use, assumptions, cut-off criteria, and alternatives to Cronbach’s alpha in educational research. A comprehensive search of major academic databases identified 44 eligible studies published between 2000 and 2026, which were critically analyzed based on their relevance, methodological rigor, and contributions to reliability assessment. The review provides key conceptual clarifications, summarizes empirical applications, highlights methodological advancements, and identifies emerging alternatives to Cronbach’s alpha. The findings indicate that although Cronbach’s alpha remains a foundational index of internal consistency, its interpretation must be done with careful consideration of scale dimensionality, sample characteristics, and item properties. The review further emphasizes the importance of reporting Cronbach’s alpha alongside complementary reliability indices, adopting context-specific cut-off criteria, and ensuring transparent and robust estimation procedures. This synthesis offers evidence-based recommendations to enhance reliability assessment practices in educational research.

This report is the fourth of four that stem from a study of the Impact of Approved Induction

There is a longstanding debate on whether the analysis of covariance (ANCOVA) or the change score approach is more appropriate when analyzing non-experimental longitudinal data. In this article, we use a structural modeling perspective to clarify that the ANCOVA approach is based on the assumption that all relevant covariates are measured (i.e., covariates are sufficient to remove confounding) and provides biased estimates of the treatment effect in the presence of unmeasured confounding variables. By contrast, the change score approach offers the option of controlling for unobserved confounders but relies on strong assumptions about the effects of these unobserved confounders and does not allow for dynamic causal relationships (i.e., pretest affects treatment variable). Furthermore, we discuss four issues that received less attention in previous research: (1) the inclusion of additional covariates in the ANCOVA and change score approaches, (2) the correction for measurement error in the predictor and treatment variables, (3) multilevel designs when the treatment is a cluster-level variable, and (4) the estimation of treatment effects in the presence of heterogeneity and nonlinearity. Implications for the analysis of non-experimental longitudinal data in educational and psychological research are discussed.

Introduction Platform trials have gained prominence, particularly during the COVID-19 pandemic, due to their flexibility and efficiency in evaluating multiple interventions. However, their complex design introduces methodological and regulatory challenges, especially in randomisation when adding new treatment arms. Method This study uses simulations based on a real platform trial to assess the performance of different randomisation methods: simple randomisation, stratified block randomisation, stratified block urn design, and minimisation. We evaluate these methods under various allocation ratios, focusing on covariate balance, allocation predictability, and allocation accuracy, particularly for newly added arms. We also examine the trade-off between covariate balance and allocation predictability. Results This study highlights the inherent trade-off between achieving good covariate balance and maintaining adequate allocation unpredictability. SBUD slightly outperforms SBR for certain stage 2 allocation ratios, though not consistently across all scenarios. Minimisation achieves the best covariate balance among all methods but at the cost of reduced randomness. Allocating more patients exclusively to the newly added arm does not effectively reduce covariate imbalance; however, allocating more patients to both the control and new arms yields better balance. Additionally, incorporating non-concurrent data in the minimisation process improves covariate balance compared to using only concurrent data. Conclusion The findings suggest that when the primary goal is to achieve covariate balance, especially for new arms, trialists should consider using minimisation with non-concurrent data and allocate patients proportionately to control and new arms. Balancing covariate distribution while maintaining sufficient randomness is crucial to ensuring the methodological robustness and reliability of platform trials.

Clinical trialists have long been searching for approaches to increase statistical power without increasing sample size. Conventional wait-list controlled (WLC) trials are limited to two trial arms and two or three repeated measurements per person. These features limit statistical power. Furthermore, their analysis is usually based on analysis of covariance or mixed effects modelling, with a focus on estimating treatment effect at one time-period after initiation of therapy. We propose two 3-arm WLC trial designs together with a mixed-effects analysis framework. The designs require three or four repeated measurements per person. The analytic framework defines up to three treatment effect estimands, representing the effects at one to three time-periods after initiation of therapy. The precision (inverse of variance) of the treatment effect estimators in the new and conventional trial designs are analytically derived and evaluated in simulations. The results are interpreted in the context of a cognitive training trial in older people. The proposed designs and analysis methods increase the precision level of treatment effect estimators as compared to conventional designs and analyses. Given a target level of statistical power, the proposed methods require a smaller number of participants per trial than the conventional methods, without necessarily increasing the number of measurements per trial. Furthermore, the proposed analytic framework sheds light on the treatment effects at different times after initiation of therapy, which is not usually considered in conventional WLC trial analysis. In situations that a WLC trial is appropriate, the 3-arm designs are useful alternatives to existing 2-arm designs.

Predictive models using microbiome data often suffer from covariate imbalance and class imbalance, biasing results. Propensity Score Matching (PSM) balances covariates but reduces sample size, while borderline synthetic minority oversampling technique (borderline-SMOTE) oversamples minority classes but can generate uninformative examples. To develop and evaluate PSM-SMOTE, a novel hybrid sampling method that integrates PSM and borderline-SMOTE to handle both covariate and class imbalance in microbiome data. We developed PSM-SMOTE, a three-step hybrid sampling algorithm for microbiome data: (1) PSM at four caliper levels to balance covariates, (2) selection of at least ten robust differential markers via seven statistical tests with false discovery rate correction, and (3) application of borderline-SMOTE on the marker-based distance matrix to oversample minority classes. We evaluated PSM-SMOTE on three publicly available microbiome case-control datasets: pancreatic ductal adenocarcinoma (PDAC), colorectal cancer (CRC), and obesity, using logistic regression (LR), random forest (RF), and support vector machine (SVM) classifiers. Performance was assessed via area under the ROC curve (AUC). PSM-SMOTE improved test AUCs in multiple model-dataset combinations compared with using PSM alone. Notably, for the RF model, PSM-SMOTE consistently enhanced AUC across nearly all oversampling settings in the PDAC and obesity cohorts. For the SVM model, PSM-SMOTE also achieved a significant AUC increase in the CRC cohort. For the LR model, PSM-SMOTE showed modest improvement under strict matching. PSM-SMOTE effectively addresses dual imbalance in microbiome data and consistently enhances performance, providing a practical solution for imbalanced data analyses.

Statistical properties of randomization in clinical trials

The traditional research backgrounds of engineering academics do not adequately prepare them to conduct quality education research. Nevertheless, there is institutional pressure on these academics to take a scholarly approach to teaching and conduct education research. What is required is an effective strategy to systematically help engineering academics to innovate, evaluate and publish education research and scholarship. This paper explores one model for structuring activities to help engineering and science academics develop their practices of education research. It is argued that the proposed Model for Action and Learning in Engineering Education Research (MALEER) presented is an effective strategy to prepare engineering academics to conduct quality education research. An initial model was developed, situated in various bodies of literature including communities of practice, professional identity development and becoming a professional, adult learning and reflection in and on action. It was then trialled with an engineering and science education research group set-up to help academics develop their education research practice. Evidence of the success of MALEER was drawn from the reflections of the coordinators of the group and the groups’ developed practices and outcomes.

The traditional research backgrounds of engineering academics do not adequately prepare them to conduct quality education research. Nevertheless, there is institutional pressure on these academics to take a scholarly approach to teaching and conduct education research. What is required is an effective strategy to systematically help engineering academics to innovate, evaluate and publish education research and scholarship. This paper explores one model for structuring activities to help engineering and science academics develop their practices of education research. It is argued that the proposed Model for Action and Learning in Engineering Education Research (MALEER) presented is an effective strategy to prepare engineering academics to conduct quality education research. An initial model was developed, situated in various bodies of literature including communities of practice, professional identity development and becoming a professional, adult learning and reflection in and on action. It was then trialled with an engineering and science education research group set-up to help academics develop their education research practice. Evidence of the success of MALEER was drawn from the reflections of the coordinators of the group and the groups' developed practices and outcomes.

Measurement is fundamental to scientific inference, yet educational and psychological research often relies on instruments that are shortened, partially administered, or otherwise modified without evidence that the resulting scores remain valid. This article examines measurement fragmentation, defined as the use of selected parts, abbreviated forms, or derivative versions of an instrument as though they were interchangeable with the validated form. The Torrance Tests of Creative Thinking (TTCT) are used as a case example because their strong full-version psychometric foundation and frequent fragmented use in published research make the consequences of non-equivalent measurement unusually visible. The full TTCT is supported by stronger reliability and validity evidence than fragmented TTCT uses, whereas derivative forms such as the Abbreviated Torrance Test for Adults have weaker support. When these non-equivalent forms are treated under a single instrument label, apparent theoretical inconsistency and attenuated meta-analytic estimates may reflect measurement artifacts rather than properties of creativity itself. The article concludes that stronger educational and psychological research requires explicit reporting of test versions, version-specific reliability and validity evidence, and closer editorial scrutiny of altered measurement procedures.

Asynchronous e-learning allows for targeted teaching, particularly advantageous when bedside and didactic education is insufficient. An asynchronous e-learning curriculum has not been studied across multiple centers in the context of a clinical rotation. We hypothesize that an asynchronous e-learning curriculum during the pediatric emergency medicine (EM) rotation improves medical knowledge among residents and students across multiple participating centers. Trainees on pediatric EM rotations at four large pediatric centers from 2012 to 2013 were randomized in a Solomon four-group design. The experimental arms received an asynchronous e-learning curriculum consisting of nine Web-based, interactive, peer-reviewed Flash/HTML5 modules. Postrotation testing and in-training examination (ITE) scores quantified improvements in knowledge. A 2 × 2 analysis of covariance (ANCOVA) tested interaction and main effects, and Pearson's correlation tested associations between module usage, scores, and ITE scores. A total of 256 of 458 participants completed all study elements; 104 had access to asynchronous e-learning modules, and 152 were controls who used the current education standards. No pretest sensitization was found (p = 0.75). Use of asynchronous e-learning modules was associated with an improvement in posttest scores (p < 0.001), from a mean score of 18.45 (95% confidence interval [CI] = 17.92 to 18.98) to 21.30 (95% CI = 20.69 to 21.91), a large effect (partial η(2) = 0.19). Posttest scores correlated with ITE scores (r(2) = 0.14, p < 0.001) among pediatric residents. Asynchronous e-learning is an effective educational tool to improve knowledge in a clinical rotation. Web-based asynchronous e-learning is a promising modality to standardize education among multiple institutions with common curricula, particularly in clinical rotations where scheduling difficulties, seasonality, and variable experiences limit in-hospital learning.