Social Experimentation, Program Evaluation, and Public Policy

Journal of Policy Analysis and Management, Vol. 24, No. 4, 687–690 (2005) © 2005 by the Association for Public Policy Analysis and Management Published by Wiley Periodicals, Inc. Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/pam.20132 Many previous studies of WIC conclude that “WIC works,” but in their article in this issue, Ted Joyce, Diane Gibson, and Silvie Colman come to the opposite conclusion (Joyce, Gibson, & Colman, 2005). They arrive at this position even though their actual estimates are consistent with those of previous studies. For example, like our largescale study of women on Medicaid in 19 states (Bitler & Currie, 2005), they find that prenatal participation in the WIC program reduces the incidence of low birth weight and preterm birth. We also find that WIC participation is associated with decreases in birth weight adjusted for gestation, the nights an infant or woman was in the hospital at delivery, and positively associated with weight gain during pregnancy, gestation, birthweight, and use of prenatal care during the first trimester. Why then are Joyce, Gibson, and Colman’s conclusions so different than those of previous authors? The crux of the matter is that Joyce et al. reject the idea that WIC could have any effect on preterm birth, while neglecting discussion of other possible positive impacts of WIC. Hence, they reason that any estimated effects of WIC on preterm birth reflect selection effects in the data and ought to be discounted out of hand. Further, they argue that WIC should only affect birth weight through effects on fetal growth, and they do not find any effects on fetal growth (except in their subsample of Black twins). We think that this line of reasoning is flawed on several counts, and perhaps too narrowly focused. First, Joyce et al. overstate the degree of medical consensus that exists over the question of whether or not prenatal intervention can affect the probability of preterm birth. They cite several studies describing specific clinical interventions that were not effective in preventing preterm births. But the clinical literature also suggests, for example, that smoking causes preterm labor, so that interventions that were effective in reducing smoking might be expected to have an effect on prematurity (compare Werler, 1997).1 Similarly, gestational diabetes is a common complication of pregnancy that can lead to preterm birth, but it is easily controlled if it is detected early (Xiong, Saunders, Wang, & Demianczuk, 2001). Maternal infections are thought to account for up to a third of preterm births, and some trials have shown that treatment of high-risk women for specific infections can increase the length of pregnancy (Gibbs & Eschenbach, 1997; Goldenberg, Hauth, & Andrews, 2000; Locksmith & Duff, 2001), while others have shown a possible positive impact of treatment for wider groups of women (Lamont, Duncan, Mandal, & Bassett, 2003; Kiss, Petricevic, & Husslein, 2004; Ugwumadu, Manyonda, Reid, & Hay,

Evaluation influence is a reconceptualization of evaluation use that reflects the broad and diffuse impacts an evaluation can have on social programs and policies. This way of thinking about impact provides an opportunity to investigate how and why evaluations influence social programs and policy. Twenty participants (practitioners and managers) from two child protection programs evaluated in the previous 24months were interviewed about the influence of these evaluations, which was complemented with the collection of internal documents about changes to the programs. A qualitative case study analysis of evaluation influence was conducted using the interviews and documents to investigate the influence of two evaluations at different stages in the dissemination process. The participants identified that the evaluations appeared to have significant high-level policy level influence; however, limited examples of influence on practices in the programs were identified. There was some suggestion that the evaluations had increased practitioner interest in working with and participating in program evaluations. The findings suggest the importance of developmental evaluation approaches and practitioner engagement in evaluation to improve the influence and adoption of new knowledge from the evaluation of social programs.

Prenatal WIC Is Associated with Increased Birth Weight of Infants Born in the United States with Immigrant Mothers

Not available

The Brazilian Ministry of Social Development and Fight against Hunger (MDS) regularly promotes the evaluation of its social programs, such as those developed in the Reference Centers for Social Assistance (CRAS). Such evaluations make use of a web system that supports the collection and processing of information as well as the dissemination of its results to local, regional and central government officials through the so-called CRAS Census. A meta-evaluation of the CRAS 2008 Census was carried out based on criteria specified by the Joint Committee (1994), from which we elicited requirements that enabled improvements of the web system. The article reports new requirements elicited from the meta-evaluation of the CRAS 2008 Census, held in the period 2009-2010. The approach of meta-evaluation as an alternative source of requirements elicitation took into consideration results from evaluations of social programs in order to identify system problems without the usual need of intense interaction with users. This approach revealed opportunities for improvements in the evaluation process that led to the elicitation of requirements for the computerized system. Some of the elicited features were incorporated into the Census 2010 and others may be incorporated in future censuses.

Social marketing of emergency contraception: Are we missing a valuable opportunity?

PurposeThe purpose of this paper is to present a case study on the methodology developed for establishing a Mexican User Satisfaction Index for a particular kind of Social Program (MUSI‐SP), taking into consideration the example of a Concrete Floor Social Program.Design/methodology/approachThe MUSI‐SP methodology has the aim to evaluate the satisfaction of the beneficiaries of social programs, as well as its causes (program practices and perceived quality) and results (measured as trust). The MUSI‐SP methodology for this case study covers the design of a specific structural equation model based on the model for government services and nonprofit organizations of the American Customer Satisfaction Index (ACSI) and a qualitative study, with the help of the design and pilot testing of the questionnaire for measuring satisfaction in a population with a high grade of marginalization, followed by data collected through a national face‐to‐face survey, then the analysis of validity and dependability of the specific structural equation model and the identification of opportunity areas.FindingsThe evaluation shows that the variable “Results” derived by having a concrete floor has a larger impact on “Perceived Quality” and it has the best users’ assessment (86 in a scale from 0 to 100). Furthermore, a high and positive relationship from “Perceived Quality” with “Satisfaction,” and “Satisfaction” with “Trust” were confirmed.Practical implicationsThe MUSI‐SP methodology would be an appropriate indicator as a complement to current evaluations of government social programs, because it shows a path to citizen satisfaction and citizen trust.Originality/valueThe Mexican Concrete Floor Program, illustrated in this paper, aims to increase the quality of life conditions of the citizens through concrete floor installation for housing improvement.

Precision in prescription: multiple micronutrient supplements in pregnancy

BackgroundCannabis social equity programs intend to redress inequities experienced by low income and Black, Indigenous, and People of Color (BIPOC) during cannabis prohibition in the United States. In Los Angeles County (LA), the approach is to increase cannabis outlet licensure and employment for low income and BIPOC communities. Monitoring locations of both licensed and unlicensed outlets over time is critical to informing how local social equity programs may affect communities.MethodsWe identified locations of licensed and unlicensed cannabis outlets in LA, from February to April 2019 and again from March to April 2020, and calculated the number and type of outlets by socio-demographic characteristics of census tracts (race/ethnicity, poverty, education, unemployment) using the 2013–2017 American Community Survey 5-year estimates.ResultsLicensed outlets increased in LA from 162 in 2019 to 195 in 2020; unlicensed outlets decreased from 286 to 137 over the same time period. In 2020, more licensed outlets were in tracts with majority white residents and adults with at least a bachelor’s degree; fewer licensed outlets were in tracts with larger Latinx or Black populations, whereas 71% of unlicensed outlets in 2020 were in low-income tracts, and more unlicensed outlets were in predominately Latinx tracts, high poverty and high unemployment tracts, and tracts with more single female-headed households.ConclusionsNeighborhood-level analyses are an important first step, but more data are needed for comprehensive evaluations of social equity programs—from individual businesses to the communities living nearby—to understand the impacts on low income and BIPOC populations.

This paper explores issues that arise in the evaluation of social programs using experimental data in the frequently encountered case where some of the experimental treatment group members drop out of the program prior to receiving treatment. We begin with the standard estimator for this case and the identifying assumption upon which it rests. We then examine the behavior of the estimator when the dropouts receive a partial “dose” of the program treatment prior to dropping out of the program. In the case of partial treatment, the identifying assumption is typically violated, thereby making the estimator inconsistent for the conventional parameter of interest: the impact of full treatment on the fully treated. We develop a test of the identifying assumption underlying the standard estimator and consider whether exclusion restrictions produce identification of the mean impact of the program when this assumption fails to hold. Finally, we discuss alternative parameters of interest in the presence of partial treatment among the dropouts and argue that the conventional parameter is not always the economically interesting one. We apply our methods to data from a recent experimental evaluation of the Job Training Partnership Act (JTPA) program.

Book Reviews: Designing Evaluations of Educational and Social Programs. By Lee J. Cronbach. Jossey-Bass Publishers, San Francisco (374 pages; $18.95), 1982

Designing Evaluations of Educational and Social Programs.

Regression discontinuity design (RDD) is a quasi-experimental approach to study the causal effect of an exposure on later outcomes by exploiting the discontinuity in the exposure probability at an assignment variable cut-off. With the intent of facilitating the use of RDD in the Developmental Origins of Health and Disease (DOHaD) research, we describe the main aspects of the study design and review the studies, assignment variables and exposures that have been investigated to identify short- and long-term health effects of early life exposures. We also provide a brief overview of some of the methodological considerations for the RDD identification using an example of a DOHaD study. An increasing number of studies investigating the effects of early life environmental stressors on health outcomes use RDD, mostly in the context of education, social and welfare policies, healthcare organization and insurance, and clinical management. Age and calendar time are the mostly used assignment variables to study the effects of various early life policies and programs, shock events and guidelines. Maternal and newborn characteristics, such as age, birth weight and gestational age are frequently used assignment variables to study the effects of the type of neonatal care, health insurance, and newborn benefits, while socioeconomic measures have been used to study the effects of social and welfare programs. RDD has advantages, including intuitive interpretation, and transparent and simple graphical representation. It provides valid causal estimates if the assumptions, relatively weak compared to other non-experimental study designs, are met. Its use to study health effects of exposures acting early in life has been limited to studies based on registries and administrative databases, while birth cohort data has not been exploited so far using this design. Local causal effect around the cut-off, difficulty in reaching high statistical power compared to other study designs, and the rarity of settings outside of policy and program evaluations hamper the widespread use of RDD in the DOHaD research. Still, the assignment variables' cut-offs for exposures applied in previous studies can be used, if appropriate, in other settings and with additional outcomes to address different research questions.

The physical positions that are adopted by women during childbirth significantly impact their childbirth outcomes and experiences. Literature studies have associated using a squatting position with reduced childbirth pain and increased comfort and pushing efficiency. However, the major disadvantage of the squatting position is that women may lack the muscular fitness and stamina necessary to sustain this position for a long period. The aim of this study was to compare the pushing experiences and birth outcomes of three different pushing positions during the second stage of labor. A randomized controlled trial was conducted at a regional teaching hospital in northern Taiwan. Data were collected from 168 primiparous women during the 38th to 42nd gestational weeks. None of the participants received epidural analgesia during labor, and all were free of pregnancy and labor-related complications. During labor, after full cervical dilation and when the fetal head had descended to at least the +1 station and had turned to the occiput anterior position, the experimental group was asked to push in the squatting position while using the ergonomically designed ankle support. For purposes of comparison, Comparison Group A was asked to push in the squatting position without the use of the support, and Comparison Group B was asked to push in a standard semirecumbent position. All of the participants completed a demographic and obstetrics data sheet, the short-form McGill Pain Questionnaire, and the Labor Pushing Experience scale within 4 hours postpartum. In terms of delivery time, the duration between the start of pushing to crowning for the experimental group (squatting with ankle supports) averaged 25.79 minutes less (F = 6.02, p < .05) than the time for Comparison Group B (semirecumbent). The duration between the start of pushing to infant birth averaged 25.21 minutes less for the experimental group than for Comparison Group B (F = 6.14, p < .05). Moreover, the experimental group had a lower average visual analog scale pain score (5.05 ± 3.22) than Comparison Group B (F = 42.67, p < .001), and the average McGill pain score for the experimental group was lower than both comparison groups (F = 18.12, p < .001). The participants in the group that delivered from a squatting position with ankle support had better labor pushing experiences than the comparison groups (F = 14.69, p < .001). In comparison with both unsupported squatting and semirecumbent pushing, squatting with the aid of ergonomically designed ankle supports reduced pushing times, ameliorated labor pain, and improved the pushing experience. Thus, this intervention may reduce the caring needs of women during the second stage of labor. This intervention may be introduced in midwifery education programs and in clinical practice as a method to improve the care of women during the second stage of labor.

The impact of aging out of WIC on food security in households with children