Rubrics in program evaluation

Not available

As we began to review articles for this special issue, we realized that we knew very little about the interaction between program implementation and program evaluation on the Federal Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services grants. Although it seemed obvious to us that the relationship between program directors and evaluators is important for the collaborative success of the Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services grant-funded projects, we felt that further information was needed to help identify the strengths and limitations of the various structural arrangements that guide these relationships. This article presents the results of a survey of Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services program directors and evaluators that explored the working relationship between the directors and evaluators in the implementation of Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services grant-funded projects.

Let’s get technical: Enhancing program evaluation through the use and integration of internet and mobile technologies

The evaluation of Competence by Design (CBD) residency programs is crucial for enhancing program effectiveness. However, literature on evaluating CBD programs is limited. We conducted a 2-phase mixed-methods study to (a) assess the extent of program evaluation activities in CBD residency programs in Canada, (b) explore reasons for engaging or not engaging in these activities, (c) examine how CBD programs are conducting program evaluations, and (d) identify ways to build capacity for program evaluation. Phase 1 involved surveying 149 program directors from specialty/subspecialty programs that transitioned to CBD between 2017 and 2020. We calculated descriptive statistics for 22 closed-ended survey items. Phase 2 comprised interviews with a subset of program directors from Phase 1. Data analysis followed a 3-step iterative process: data condensation, data display, and drawing and verifying conclusions. In Phase 1, we received 149 responses, with a 33.5% response rate. Of these, 127 (85.2%) indicated their programs engage in evaluation, while 22 (14.8%) do not. Among the 127 programs that engage in evaluation, 29 (22.8%) frequently or always develop evaluation questions, and 23 (18.1%) design evaluation proposals/plans. Reasons for engaging in evaluation included decision-making and stimulating changes in educational practices. Conversely, reasons for not engaging included lack of knowledge, personnel, and funding. In Phase 2, 15 program directors were interviewed. They reported that CBD programs face challenges such as limited resources and buy-in, rely on ad hoc evaluation methods, and use a team-based evaluation format. To enhance evaluation capacities, interviewees suggested (a) developing expertise in program evaluation, (b) acquiring evaluation resources, and (c) advocating for clear evaluation expectations. Most CBD residency programs are engaged in program evaluations, but the quality is often questionable. To fully realize the potential of program evaluation, CBD programs need additional resources and support to improve evaluation practices and outcomes.

Effective program evaluation is a systematic way to improve and account for public health actions by involving procedures that are useful, feasible, ethical, and accurate. The framework guides public health professionals in their use of program evaluation. It is a practical, nonprescriptive tool, designed to summarize and organize essential elements of program evaluation. The framework comprises steps in program evaluation practice and standards for effective program evaluation. Adhering to the steps and standards of this framework will allow an understanding of each program's context and will improve how program evaluations are conceived and conducted. Furthermore, the framework encourages an approach to evaluation that is integrated with routine program operations. The emphasis is on practical, ongoing evaluation strategies that involve all program stakeholders, not just evaluation experts. Understanding and applying the elements of this framework can be a driving force for planning effective public health strategies, improving existing programs, and demonstrating the results of resource investments.

Many financial education providers still do not have a basic level of evaluation capacity and are unable to identify program outcomes and design effective evaluation instruments. It is difficult to propose a national evaluation strategy without a basic understanding of current evaluation capacity and of the critical gaps in program evaluation. In addition, there has been little discussion about the challenges facing financial professionals and educators who are on the “front lines” delivering and evaluating programs. The purpose of this survey article is to address these critical gaps in the literature and to provide an overview of the current state of financial education and program evaluation. Using qualitative and quantitative data collected from financial professionals and educators nationwide, this study provides insight into what can be done to build national evaluation capacity and conduct more effective program evaluations.

Many authors who write about instructional supervision link research and evaluation skills when discussing program evaluation. A wide variety of approaches can be employed in any program evaluation, including objectives-oriented, management-oriented, consumer-oriented, expertise-oriented, and participant-oriented evaluation. The first standard for program evaluation asserts the necessity for involving key stakeholders in identifying the purpose and goals for the program evaluation as well as the questions and standards that will drive the evaluation process. The supervisor must play a leading role in determining which of these data sources are most germane to the local school curricular and instructional program evaluations. In making decisions concerning program evaluation design, it is wise to consider the contribution that each of these data types could make to answering the evaluation questions developed through interaction with stakeholders. It is possible to collect a wide variety of data concerning an educational program but fail to capture data that meet accepted standards for validity and reliability.

The purpose of the current study was to investigate the relationship between the perceived characteristics of television presenters, program evaluations, viewer satisfaction, and intention to re-watch in the context of golf television programs. Specifically, this study focused on the importance of television presenters’ characteristics for program viewers in golf television programs. Data were collected from 248 television viewers who have watched at least one golf programs other than sport telecasts of official golf games. Data were analyzed using descriptive analysis, factor analysis, correlational analysis, and structural equation modeling (SEM) technique. The results of SEM indicated that the television presenters’ characteristics had a significant influence on program evaluation and viewer satisfaction but did not have a significant direct on intention to re-watch. Meanwhile, viewers’ program evaluation and viewer satisfaction had significant influences on intention to re-watch.

It is generally recognized that for any evaluation to be successful there needs to be a significant amount of involvement from program staff and other stakeholders who are invested in the program. Ideally, these individuals would be highly engaged in the evaluation process; however, in practice this ideal is rarely met. Two research studies were undertaken to explore different stakeholder motives for involvement in program evaluations and their relative importance in involvement decisions. In Study 1, stakeholder views of their own involvement (or lack thereof) in recent program evaluations were analyzed qualitatively using a thematic analysis approach. The findings revealed diverse stakeholder motives for evaluation involvement that include personal/human factors (e.g., opportunity for personal advancement), evaluation factors (e.g., clarity of evaluation goals) and organizational factors (e.g., funding implications). A second study looked more closely at these motives to determine their relative importance to evaluation involvement decision-making. Using the Q-sort method, three involvement profiles were identified which revealed different pathways to involvement based on individual concerns for program funding, learning opportunities, and evaluation quality. A preliminary model of stakeholder involvement motives is proposed in which the value and relevance of the evaluation are determined by expected outcomes of the evaluation.

In 1980, a Joint Committee appointed by 12 organizations1 concerned with educational evaluation issued one of the most significant documents to date in the field of educational evaluation. It consisted of a set of 30 standards to be used both to guide the conduct of evaluation of educational programs, projects, and materials and also to judge the soundness of such evaluations. The document, entitled Standards for Evaluations of Educational Programs, Projects, and Materials, 2 was published in 1981 by the McGraw-Hill Company. The standards were the result of an extensive developmental process, which involved the work of about 200 people and required more than four years to complete. The committee’s work did not stop with the publication of the standards, and it presently works to promote sound use and conducts a process of ongoing review and development. This chapter briefly describes the development of the standards and the nature of the standards, and summarizes them for the reader.

Applicability of standards for evaluations of educational programs, projects and materials in an international setting

Anyone who trains others in the practice of program evaluation should be aware of how necessary it is for the practitioner to maneuver the interpersonal dynamics within a program evaluation setting. This paper discusses why interpersonal skills are an invaluable tool for any program evaluator, which interpersonal skills are most important, and how instructors can integrate these skills in classroom activities and assess student learning. In fact, how instructors choose to use interpersonal skills as an objective in their program evaluation classrooms influences the level of skill development acquired by the student. This work merges a combination of Dreyfus and Dreyfus’s (1980) five-stage skill development model and Glaser’s (1983) behavioral approach with Bloom et al.’s (1956) taxonomy of cognitive learning objectives to construct a taxonomy of interpersonal skill competence for program evaluators that bridges the gap between theory and practice.

Performance measurement has often been defined in contrast with program evaluation. However, many now recognize that the methods and experiences of performance measurement and program evaluation can support and strengthen each other, thereby making relevant and timely contributions to evidence‐based decision making. The argument in this chapter is that recent developments, emphasizing citizen engagement and incorporating community indicators, broaden the opportunities for collaboration between program evaluators and performance measurement professionals. Such collaboration, with an appreciation of the complementary strengths of the two approaches, can not only improve the practice of evaluation, but also enhance governance. © Wiley Periodicals, Inc., and the American Evaluation Association.

Graduate medical education programs are expected to conduct an annual program evaluation. While general guidelines exist, innovative and feasible approaches to program evaluations may help efforts at program improvement. Appreciative Inquiry is an approach that focuses on successful moments, effective processes, and programs' strengths. We implemented a novel application of Appreciative Inquiry and its 4 phases (Inquire, Imagine, Innovate, and Implement) and demonstrate how it led to meaningful improvements in a pediatric pulmonology fellowship program. As part of the Inquire and Imagine phases, the authors developed an interview guide that aligned with Appreciative Inquiry concepts. Two faculty members conducted semistructured interviews with a convenience sample of 11 of 14 fellowship alumni. Interviews were audiotaped, transcribed, and reviewed. A summary of the findings was presented to the Program Evaluation Committee, which then directed the Innovate and Implement phases. Appreciative Inquiry was acceptable to the alumni and feasible with the authors' self-directed learning approach and minimal administrative and financial support. In the Inquire phase, alumni identified program strengths and successes. In the Imagine phase, alumni identified program changes that could aid transition to independent practice for future fellows (an identified program goal). Based on the results of the Appreciative Inquiry, program leadership and the Program Evaluation Committee selected improvements for implementation. For small programs, Appreciative Inquiry is an innovative and feasible approach to program evaluation that facilitates actionable program improvement recommendations.

To increase knowledge and skills in program evaluation, a peer-coaching intervention provided one-on-one professional development to gifted program administrators. This randomized field study examined the effects of peer coaching on evaluation knowledge and skills and on administrators’ concerns about implementing more rigorous program evaluations. In addition, the peer-coaching intervention focused on increased access to program services for culturally diverse and low-income gifted learners. Results revealed a statistically significant increase in gifted administrators’ knowledge of and skills in program evaluation attributed to the peer-coaching intervention. Peer-coached program administrators were less distracted and more focused on implementing program evaluation and reported decreased concerns about implementing them. Peer coaching did not increase placements for culturally diverse and low-income learners in program services; however, peer coaching positively affected referral rate for traditionally underrepresented groups.