Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education). Intervention: Not applicable Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.

In this mural we demonstrate liberation of evaluation from normalized publication structures. The article is a challenge to publication structures, more specifically journals, to open space for diversity of expression of thought. In this mural, practitioners, academics, policymakers, development partners, VOPEs (Voluntary Organizations for Professional Evaluations), speak to the strategies to decolonize evaluation. The immediate impact of this mural is a demonstrated commitment to decolonize evaluation by more than 50 individuals and entities. Taking up the commitment will be a demonstrable change that came with the mural. The mural also provides an open space for the exchange of ideas, and strategies to decolonize evaluation. Moreover, the contributing voices are preserved in their own voices and their contribution to global knowledge production is acknowledged. This is indeed a departure from the ethics of confidentiality in research where the participants in an evaluation inquiry largely remain anonymous even in the co-creation of knowledge. We believe that when participants in an evaluation inquiry come across their commitments in evaluation journals, they are encouraged to further form coalitions with like minds and hold themselves accountable collectively to liberate evaluation from the remnants of colonialism.

The theme of the special issue decolonizing evaluation: towards a fifth paradigm, an initiative of the International Evaluation academy (IEA), was inspired by the concerns that while evaluation reports largely tell stories of success, on the ground there is minimal change, communities remain impoverished, interventions cause harm to the environment and evaluation allow that to happen.

The concept of Indigenization of research has been increasingly explored in recent studies, with emphasis placed on the ontological, epistemological, and axiological perspectives of Indigenous peoples to find effective solutions to their challenges. This also applies to the evaluation of guidance and counseling approaches in Africa and other nations, where Indigenous therapies are developed based on different philosophical foundations, such as Ubuntu (Africa). Relational ontologies and epistemologies appear to be common across various Indigenous nations in Africa, Australia, Canada, and North America. This article analyzes studies from these regions on evaluations of Indigenous guidance and counseling therapies. The majority of the evaluations use conventional paradigmatic assumptions in their approach, rather than relational models that are participatory and respectful of participants’ worldviews, including the living, non-living, metaphysical, and spiritual aspects of Indigenous people. However, the Indigenous therapeutic programs analyzed in this study incorporate culturally appropriate activities and curricula that align with relational axioms. This article proposes the use of relational models of evaluation to assess Indigenous counseling programs, where researchers can draw conclusions that align with the cultural contexts of the Indigenous people being researched.

Background: The Program Evaluation Standards that were developed and approved by the Joint Committee on Standards for Educational Evaluation have served as a resource to the broader evaluation field for over four decades. However, little evidence has been collected regarding the extent to which the standards have influenced the field through scholarship or professional practice. Purpose: This study seeks to estimate the prevalence of the Program Evaluation Standards in evaluation scholarship and professional practice. Setting: Not applicable. Intervention: Not applicable. Research Design: The study combines a systematic review of evaluation literature with a survey of American Evaluation Association (AEA) and Canadian Evaluation Society (CES) members. Data Collection and Analysis: A systematic review of articles published in 14 evaluation-specific journals from 2010 to 2020 was conducted to identify and typify articles citing the standards. Additionally, AEA and CES members were surveyed, with a focus on knowledge and use of the standards. Descriptive analyses are presented to quantify the prevalence of the standards in evaluation scholarship and practice, respectively. Findings: The systematic review revealed that 4.48% of the 4,460 articles published in 14 evaluation-specific journals from 2010 to 2020 contained some use of the standards. Survey results show that 53.14% of AEA members and 67.12% of CES members are familiar with the standards and that, among those with knowledge of the standards, most AEA (67.67%) and CES (71.74%) members use them at least “occasionally” in their professional work, education, and scholarship activities.

This one-page paper introduces the special issue devoted to the Program Evaluation Standards with a brief summary of the purpose of the special issue and an overview of the individual papers is provided.

Background: Public health practitioners, including injury and violence prevention (IVP) professionals, are responsible for implementing evaluations, but often lack formal evaluation training. Impacts of many practitioner-focused evaluation trainings—particularly their ability to help participants successfully start and complete evaluations—are unknown. Objectives: We assessed the impact of the Injury and Violence Prevention (IVP) Program & Policy Evaluation Institute (“Evaluation Institute”), a team-based, multidisciplinary, and practitioner-focused evaluation training designed to teach state IVP practitioners and their cross-sector partners how to evaluate program and policy interventions. Design: Semi-structured interviews were conducted with members of 13 evaluation teams across eight states at least one year after training participation (24 participants in total). Document reviews were conducted to triangulate, supplement, and contextualize reported improvements to policies, programs, and practices. Intervention: Teams of practitioners applied for and participated in the Evaluation Institute, a five-month evaluation training initiative that included a set of online training modules, an in-person workshop, and technical support from evaluation consultants. Main Outcome Measure(s): The successful start and/or completion of a program or policy evaluation focused on an IVP intervention. Results: Of the 13 teams studied, a total of 12 teams (92%) reported starting or completing an evaluation. Four teams (31%) reported fully completing their evaluations; eight teams (61%) reported partially completing their evaluations. Teams identified common facilitators and barriers that impacted their ability to start and complete their evaluations. Nearly half of the 13 teams (46%) – whether or not they completed their evaluation – reported at least one common improvement made to a program or policy as a result of engaging in an evaluative process. Conclusion: Practitioner-focused evaluation trainings are essential to build critical evaluation skills among public health professionals and their multidisciplinary partners. The process of evaluating an intervention—even if the evaluation is not completed—has substantial value and can drive improvements to public health interventions. The Evaluation Institute can serve as a model for training public health practitioners and their partners to successfully plan, start, complete, and utilize evaluations to improve programs and policies.

Background: Research on the role and effects of evaluation policy is limited. Some research on the policy’s role in enhancing organizational evaluation capacity (EC) is beginning to accrue but to date it has been limited largely to global Western evaluation contexts. Purpose: We employed an ecological conceptual framework arising from our own empirical research to explore the interface between evaluation policy and EC in non-western contexts. We asked—To what extent does this framework resonate across these contexts? In the selected non-Western context, what are the salient variables moderating the relationship between policy and EC in the selected contexts? Are there differences across countries? Setting: The present research is focused on perceptions about evaluation culture and experiences in two countries situated in the Middle East and North Africa (MENA) region, namely Turkey and Jordan. Intervention: Not applicable. Research design: We conducted focus groups within the respective countries with a combined total of 18 participants associated with country-level voluntary organizations for professional evaluation (VOPE). Participants worked in government, non-governmental aid agencies, universities and private sector organizations. Data collection and analysis: We introduced the focus group participants to our ecological framework and then guided the conversation using semi-structured questions. Data were audio-recorded, transcribed and subsequently thematically analyzed using NVivo. Findings: The ecological framework was found to resonate well but the findings were weighted heavily toward macro-level contextual variables. Even though important contextual and cultural differences between Turkey and Jordan were evident, leadership emerged as a significant meso-level moderating variable in both settings. The discussion of the results included implications for ongoing research.


 
 
 
 Background: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without reference to predetermined goals or objectives. 
 
 Purpose: The purpose of this study is to examine GFE in actual practice focusing on what GFE is, how it is conducted, and why the evaluators use it.
 
 Setting: Not applicable.
 
 Intervention: Not applicable.
 
 Research Design: Document analysis.
 
 
 Data Collection and Analysis: The researcher collected data from a non-random sample of 12 cases of GFE found in published and non-published papers, reports, and guidebooks. The researcher analyzed the documents using quantitative and qualitative content analysis.
 
 Findings: The findings suggest that goal-free evaluators consider GFE an outcome evaluation that supplements GBE. These goal-free evaluators typically used an ex post facto evaluation design, non-random sampling of stakeholders, and semi-structured interviewing to collect data. The evaluators described using GFE to improve the evaluand, to find side-effects, and to evaluate highly complex evaluands.