Computing Statistical Power for the Difference in Differences Design.

The title of this part of the book—“Discovering Natural Experiments”—suggests a first foundational issue for discussion. The random or as-if random assignment that characterizes natural experiments occurs as a feature of social and political processes—not in connection with a manipulation planned and carried out by an experimental researcher. This is what makes natural experiments observational studies, not true experiments. For this reason, however, researchers face a major challenge in identifying situations in which natural experiments occur. Scholars often speak not of “creating” a natural experiment, but of “exploiting” or “leveraging” an opportunity for this kind of approach in the analysis of observational data. In an important sense, natural experiments are not so much designed as discovered. How, then, does one uncover a natural experiment? As the survey in Part I of the book will suggest, new ideas for sources of natural experiments—such as close elections or weather shocks—seem to arise in unpredictable ways. Moreover, their successful use in one context does not guarantee their applicability to other substantive problems. The discovery of natural experiments is thus as much art as science: there appears to be no algorithm for the generation of convincing natural experiments, and analysts are challenged to think carefully about whether sources of natural experiments discovered in one context are applicable to other settings. Yet, the best way to recognize the potential for using a natural experiment productively is often through exposure to examples. This can generate ideas for new research, as existing approaches are modified to suit novel contexts and questions, and it can also lead researchers to recognize new sources of natural experiments. Part I of the book therefore surveys and discusses in detail existing research, as a way to broach the central topic of how to discover natural experiments.

Cluster randomized trials are frequently used in educational research for methodological reasons. This study aims to improve the efficiency of cluster randomized trials on computer/information literacy and computational thinking. The study employs a two-level hierarchical linear model to estimate (i) intraclass correlation coefficients, (ii) the amount of explained variances given selected predictors, and (iii) minimum detectable effect sizes given the set of plausible scenarios. Two data cycles from the International Computer and Information Study were used. The covariates at the student level are gender, interest in ICT, parents’ highest education level, ICT self-efficacy, and experience with computers. The covariates at school/teacher level are teacher’s ICT use, ratio of school size to the number of computers for student use, availability of ICT resources at school, approximate teacher age, and ICT self-efficacy. Findings showed that the most precise effect could be measured when student and teacher/school covariates are both adopted. Lastly, it was revealed that increasing the number of schools is effective to get the most precise effect.

Although there is much literature on testing the treatment effect in experiments with correlated binary outcomes, the existing methodologies only work well when the cluster size is small. In this paper, I propose a simple method to test the treatment effect in experiments with large groups of correlated binary outcomes. I use the weighted estimating equations approach to estimate the treatment effect. The proposal is appropriate under any correlation structure. Power comparisons show the advantage of the new procedure.

In this commentary, invited for the 100th anniversary of the Journal, we discuss the addition of randomized experiments, along with natural experiments that emulate randomized trials using observational data, as designs in the social epidemiologist's toolbox. These approaches transform the way we define and ask questions about social exposures. They compel us to ask questions about how well-defined interventions change a social exposure that might lead to changes in health. As such, experiments are of unique public health and policy significance. We argue that they are a powerful approach to advance our understanding of how well-defined changes in social exposures impact health, and how credible social policy reforms may be instrumental to address health inequalities. We focus on two research designs. The first is a "pure" randomized controlled trial (RCT) in which the investigator defines and randomly assigns the intervention. The second is a natural experiment, which exploits the fact that policies or interventions in the real world often involve an element of random assignment, emulating an RCT. To give the reader our bottom line: While acknowledging their limits, we continue to be very excited about the promise of RCTs and natural experiments to advance social epidemiology.

Trial effect: a sum of many effects A trial effect is considered a benefit that trial participants experience merely by the act of trial participation. The experimental treatment effect, protocol effect, care effect, Hawthorne effect and the placebo effect, are all potential components of an overall trial effect and the terms ‘participation benefit’ or ‘participation effect’ have been used to cover four out of the five effects (excludes treatment effect). Similarly, older studies referred to a trial effect as an ‘inclusion benefit’ [1,2]. The experimental treatment effect is thought to occur when treatment offered in a study is better than the current standard of care and would be expected to accrue only to subjects randomized to the experimental therapy. The protocol effect is a possible benefit arising from closely adhering to the treatment regimens and procedures that are clearly outlined in the clinical trial’s manual. The care effect arises from differences in care between trial and non-trial participants that may not be specifically codified in the protocol. The Hawthorne effect is due to changes in patient or clinician behavior as a result of being under close observation in a trial. The placebo effect arises from psychologically mediated benefits associated with the administration of a sham or simulated intervention. The components of a participation effect are hypothesized to accrue to trial participants regardless of study arm and result in improved outcomes when comparing trial with non-trial participants.

Preference, Expectation, and Satisfaction in a Clinical Trial of Behavioral Interventions for Acute and Sub-Acute Low Back Pain

Mammography is the only technique currently used for detecting microcalcification (MC) clusters, an early indicator of breast cancer. However, mammographic images superimpose a three-dimensional compressed breast image onto two-dimensional projection views, resulting in overlapped anatomical breast structures that may obscure the detection and visualization of MCs. One possible solution to this problem is the use of cone beam computed tomography (CBCT) with a flat-panel (FP) digital detector. Although feasibility studies of CBCT techniques for breast imaging have yielded promising results, they have not shown how radiation dose and x-ray tube voltage affect the accuracy with which MCs are detected by CBCT experimentally. We therefore conducted a phantom study using a FP-based CBCT system with various mean glandular doses and kVp values. An experimental CBCT scanner was constructed with a data acquisition rate of 7.5 frames/s. 10.5 and 14.5 cm diameter breast phantoms made of gelatin were used to simulate uncompressed breasts consisting of 100% glandular tissue. Eight different MC sizes of calcium carbonate grains, ranging from 180-200 microm to 355-425 microm, were used to simulate MCs. MCs of the same size were arranged to form a 5 x 5 MC cluster and embedded in the breast phantoms. These MC clusters were positioned at 2.8 cm away from the center of the breast phantoms. The phantoms were imaged at 60, 80, and 100 kVp. With a single scan (360 degrees), 300 projection images were acquired with 0.5 x, 1x, and 2x mean glandular dose limit for 10.5 cm phantom and with 1x, 2x, and 4x for 14.5 cm phantom. A Feldkamp algorithm with a pure ramp filter was used for image reconstruction. The normalized noise level was calculated for each x-ray tube voltage and dose level. The image quality of the CBCT images was evaluated by counting the number of visible MCs for each MC cluster for various conditions. The average percentage of the visible MCs was computed and plotted as a function of the MGD, the kVp, and the average MC size. The results showed that the MC visibility increased with the MGD significantly but decreased with the breast size. The results also showed that the x-ray tube voltage affects the detection of MCs under different circumstances. With a 50% threshold, the minimum detectable MC sizes for the 10.5 cm phantom were 348(+/-2), 288(+/-7), 257(+/-2) microm at 3, 6, and 12 mGy, respectively. Those for the 14.5 cm phantom were 355 (+/-1), 307 (+/-7), 275 (+/-5) microm at 6, 12, and 24 mGy, respectively. With a 75% threshold, the minimum detectable MC sizes for the 10.5 cm phantom were 367 (+/-1), 316 (+/-7), 265 (+/-3) microm at 3, 6, and 12 mGy, respectively. Those for the 14.5 cm phantom were 377 (+/-3), 334 (+/-5), 300 (+/-2) microm at 6, 12, and 24 mGy, respectively.

Education is related to a wide variety of beneficial health, behavioral, and societal outcomes. However, whether education causes long-term structural changes in the brain remains unclear. A pressing challenge is that individuals self-select into continued education, thereby introducing a wide variety of environmental and genetic confounders. Fortunately, natural experiments allow us to isolate the causal impact of increased education from individual (and societal) characteristics. Here, we exploit a policy change in the UK (the 1972 ROSLA act) that increased the amount of mandatory schooling from 15 to 16 years of age to study the impact of education on long-term structural brain outcomes in the UK Biobank. Using regression discontinuity (RD) – a causal inference method – we find no evidence of an effect from an additional year of education on any structural neuroimaging outcomes. This null result is robust across modalities, regions, and analysis strategies. An additional year of education is a substantial cognitive intervention, yet we find no evidence for sustained experience-dependent plasticity. Our results provide a challenge for prominent accounts of cognitive or ‘brain reserve’ theories which identify education as a major protective factor to lessen adverse aging effects. Our preregistered findings are one of the first implementations of RD on neural data – opening the door for causal inference in population-based neuroimaging.

Most agricultural experiments involve evaluation of multiple variables and at times it can be difficult to identify the biologically relevant effects of the experimental treatments after performing the traditional ANOVA, Tukey and t-tests. The coefficient of variation formula could be an important tool to focus ‘Result and Discussion’ sections only on the most important changes produced by the experimental treatments. This short report is intended to exemplify the use of the coefficient of variation in three plant physiology experiments. The first one dealt with the effects of common bean plantlet exposure to high temperature under controlled conditions (levels: 28 and 40 °C). The second experiment was related to common bean seed exposure to liquid nitrogen during five different periods of time (levels: 0, 7, 14, 21 and 28 days). The third experiment was bi-factorial: factor 1 was the ‘type of plant material’ (levels: pineapple plants genetically transformed and the untransformed control); and factor 2 was the ‘time of in vitro-plantlet hardening’ (levels: 0, 15 and 30 days). Contents of phenolics, aldehydes, chlorophylls and proteins were determined. Percentage of seed germination, electrolyte leakage, peroxidase activity, plant height and weight were also measured. Experiments were monofactorial with two levels, monofactorial with five levels and bifactorial, respectively, with randomized design. The coefficient of variation showed that the most remarkable effects of high temperature were recorded in free phenolics and chlorophylls (a, b, total). Electrolyte leakage and chlorophyll b concentration were the most modified indicators as a result of seed exposure to liquid nitrogen. In the third experiment, modification in the levels of malondialdehyde and other aldehydes were the most relevant changes resulting from factors interactions. A similar procedure has not been published, except for our previous publications, not focused on the use of the coefficient of variation, just on the biological results.

It has been suggested that the random assignment of participants does not cause harm to patients in clinical trials. However, in a recent study, patient preference affected the health outcome of the patients. It can therefore be assumed that patients who are allocated at random may be affected by placebo effects that are associated with the treatment of their choice. The aim of this study was to determine whether random participant assignment in acupuncture trials causes fewer benefits to the participants. The following computerized databases were searched from their inception to December 2008: MEDLINE, SCOPUS, CINAHL, Cochrane Registered Trial, British Library Direct, and Google Scholar. This was a systematic review of partially randomized acupuncture trials with adequate randomization generation and concealment of participant group allocation as well as large enough sample sizes to satisfy a power calculation. Six (6) trials that met the selection criteria were included. The randomized acupuncture group comprised patients with different demographics and health outcomes than the observational group. Substantial numbers of eligible patients refused to participate in the randomization. The baseline patient characteristics were different between the randomized acupuncture group and the observational group. The proportion of patients who dropped out of or withdrew from the study was also different between the groups, with significantly more dropping out from the observational group than the randomized group. Patients in the observational group had significantly better health outcomes than patients in the randomized group, with a standardized mean difference of 0.06 (95% confidence interval 0.03-0.09). We found that patients who were randomly assigned to treatment groups in acupuncture trials had fewer health benefits than those who were nonrandomly assigned.

Why do peers influence people's choices? Is it for information (social learning) or for socializing (social utility)? Exploiting unique data and natural experiments from the United States Naval Academy (USNA), we analyze data on major selections of USNA students. We find peers influence students into selecting different academic paths than they would have chosen independently. Through random reassignments, “shot‐guns”, of students into new peer groups along with random assignments into courses, we explore the reasons why herding occurs. Evidence suggests that social learning, as opposed to social utility, is an important driver for herding behavior.

SAS Code for Calculating Intraclass Correlation Coefficients and Effect Size Benchmarks for Site-Randomized Education Experiments

Social scientists search for interventions in the real world that approximate the conditions of an experiment. One form of such natural experiments that is increasingly used in social science research is regression discontinuity (RD). RD designs are possible when there are thresholds that cause large changes in the assignment of treatments on the basis of small differences in a variable. For example, a high school junior in the state of Pennsylvania who scored 214 out of 240 on the 2012 PSAT test received the treatment of being a National Merit Semi‐Finalist, whereas a comparable student who scored 213 did not. The intuition behind a RD design is that we often can learn something about the effects of a treatment by comparing observations that barely receive a treatment (e.g., individuals with scores of 214 and just above on the PSAT) to observations that barely miss receiving a treatment (e.g., individuals who score 213 and just below on the PSAT). We discuss the assumptions under which the effects of treatment that are assigned based on a discontinuous threshold can be estimated using a RD design. We then illustrate how graphical analysis can be used to illustrate whether these assumptions are likely to hold. We conclude by discussing two examples of cutting‐edge research that employs RD designs and discussing areas of future research.

Protocol non-adherence is common and poses unique challenges in the interpretation of trial outcomes, especially in non-inferiority trials. We performed simulations of a non-inferiority trial with a time-fixed treatment and a binary endpoint in order to: i) explore the impact of various patterns of non-adherence and analysis methods on treatment effect estimates; ii) quantify the probability of claiming non-inferiority when the experimental treatment effect is actually inferior; and iii) evaluate alternative methods such as inverse probability weighting and instrumental variable estimation. We found that the probability of concluding non-inferiority when the experimental treatment is actually inferior depends on whether non-adherence is due to confounding or non-confounding factors, and the actual treatments received by the non-adherent participants. With non-adherence, intention-to-treat analysis has a higher tendency to conclude non-inferiority when the experimental treatment is actually inferior under most patterns of non-adherence. This probability of concluding non-inferiority can be increased to as high as 0.1 from 0.025 when the adherence is relatively high at 90%. The direction of bias for the per-protocol analysis depends on the directions of influence the confounders have on adherence and probability of outcome. The inverse probability weighting approach can reduce bias but will only eliminate it if all confounders can be measured without error and are appropriately adjusted for. Instrumental variable estimation overcomes this limitation and gives unbiased estimates even when confounders are not known, but typically requires large sample sizes to achieve acceptable power. Investigators need to consider patterns of non-adherence and potential confounders in trial designs. Adjusted analysis of the per-protocol population with sensitivity analyses on confounders and other approaches, such as instrumental variable estimation, should be considered when non-compliance is anticipated. We provide an online power calculator allowing for various patterns of non-adherence using the above methods.

Who does Collaborative Learning Help? The Pedagogical Efficacy of Student Learning Through Collaborative Learning Sessions