Evaluation of health programs, services and technologies

BackgroundOral health-related quality of life (OHRQoL) measures are being increasingly used to introduce dimensions excluded by normative measures. Consequently, there is a need for an index which evaluates children's OHRQoL validated for Brazilian population, useful for oral health needs assessments and for the evaluation of oral health programs, services and technologies. The aim of this study was to do a cross-cultural adaptation of the Child Oral Impacts on Daily Performances (Child-OIDP) index, and assess its reliability and validity for application among Brazilian children between the ages of eleven and fourteen.MethodsFor cross-cultural adaptation, a translation/back-translation method integrated with expert panel reviews was applied. A total of 342 students from four public schools took part of the study.ResultsOverall, 80.7% of the sample reported at least one oral impact in the last three months. Cronbach's alpha was 0.63, the weighted kappa 0.76, and the intraclass correlation coefficient (ICC) 0.79. The index had a significant association with self-reported health measurements (self-rated oral health, satisfaction with oral health, perceived dental treatment needs, self-rated general health; all p < 0.01).ConclusionIt was concluded that the Child-OIDP index is a measure of oral health-related quality of life that can be applied to Brazilian children.

This social and historical study drew on Bourdieu's genetic sociology to analyze the establishment and development of a specialized space for the production of knowledge and practices in health evaluation in Brazil. The study analyzed the trajectories of 28 researchers and policymakers and the historical conditions that allowed establishing this space, using in-depth interviews, document analysis, and a literature review. The resulting material was analyzed according to Bourdieu's concepts of field, habitus, and capital. The results point to the constitution of a sub-space for evaluation within Public Health, resulting from interaction between actors from the administrative and scientific fields, respectively, represented by management institutions in the Brazilian Unified National Health System at its various levels and research groups affiliated with public universities. No common habitus was found between the interviewees and the inherent issues and disputes in this sub-space.

Public health intelligence is the ongoing systematic collection, analysis, interpretation, and dissemination of health and social care data. Public health professionals use data to describe and monitor health-related events in their areas (local, regional, and national). Analysis of this information allows for effective setting of public health priorities, planning, implementation, and ultimately evaluation of public health interventions, programmes, and services. This chapter aims to describe the key data sources accessed and the different statistical methods and techniques, used to determine the burden of disease in populations and where these populations may be geographically or demographically placed.

Independent foundations' primary focus is grant making.In general, their assets come from the gift of an individual or family.Community foundations' assets usually come from many donors rather than a single source, and grant making is limited to a particular region.Community foundations are usually classified as private charities under the tax law and are subject to different rules and regulations than are other private foundations. Access To CareAccording to 1988 Census Bureau estimates, over thirty-one million Americans are without health insurance.The demographics of these people are diverse, and their ranks continue to grow as the service industry burgeons and as competition in health care increases.Regional disparities in access also exist.Foundation funding in this area ranges from economic analyses of the problem to direct provision of care.

In Australia, mental health services are delivered by a complex web of public- and private-sector providers. There is a growing recognition that linkages between these groups are not optimal, and a concern that this may lead to poor outcomes. This paper illustrates a conceptual framework for developing, implementing and evaluating programmes concerned with linkages. Drawing on theoretical and practical literature, this paper identifies different levels of integration, issues in evaluating programmes to address poor linkages, and features of useful evaluations. Within this context, it describes the method by which the Public and Private Partnerships in Mental Health Project (Partnership Project) is being evaluated. Conducted by St Vincent's Mental Health Service and The Melbourne Clinic, this is one of several Demonstration Projects in Integrated Mental Health Care funded under the National Mental Health Strategy. Collaboration is hard to conceptualize and collaborative programmes usually have many players and components, and tend to operate within already-complex systems. This creates difficulties for evaluation, in terms of what to measure, how to measure it, and how to interpret findings. In spite of these difficulties, the illustrative example demonstrates a model for evaluating collaborative programmes that is currently working well because it is strongly conceptualized, descriptive, comparative, constructively sceptical, positioned from the bottom up, and collaborative. This model, or aspects of it, could be extended to the evaluation of other mental health programmes and services that have collaborative elements.

THE MISMATCH BETWEEN US HEALTH EXPENDITURES AND the resources devoted to learning which health interventions are most effective is both striking and unwise. Each year US individuals spend more than $2 trillion on health care. More than $100 billion is spent for research and development and for regulatory approval of new technologies. Yet total spending on technology assessment almost certainly falls short of $1 billion per year— 0.05% of all US health care spending. Some of the $2 trillion in health care expenditures buys services of little or no value. This waste has been attributed to misleading advertisements, media hype, misguided state and federal mandates, fear of malpractice litigation, misaligned reimbursement incentives, and generous insurance that encourages patients to ignore the cost of services. Efforts to curb the inappropriate use of medical technologies, however, can have only limited success unless they address the paucity of reliable information about their benefits, cost, and value. For decades, calls for more systematic assessment of medical technologies and outcomes have gone unheeded. Recently, however, federal legislators and officials have recognized that better information is imperative. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 mandated research on “outcomes, comparative clinical effectiveness, and appropriateness of health care.” The former Medicare administrator, Gail Wilensky, has described alternative structures for a technology assessment organization. Representatives, senators, and presidential candidates have supported legislation to fund comparative effectiveness initiatives. Renewed interest in technology and outcomes assessment efforts can be traced to several factors: disillusionment with traditional cost-containment approaches, deepening anxiety about the safety and effectiveness of drugs and medical care, recognition that little is known about the optimal use of existing diagnostic procedures and treatments, and the explosion in health care expenditures anticipated as baby boomers age. By 2015, the number of US individuals in their 60s—a decade of heavy use of medical care— will increase by nearly 50%. Simultaneously, because of scientific advances, many new technologies will enter clinical practice. The combination of new technologies and greater use of older medical interventions are the fundamental drivers of increasing health care costs. Increasing health care costs have induced employers and insurance companies to shift more financial responsibility onto individuals through “consumer-directed” health plans and health savings accounts. In addition, private health plans, Medicare, and Medicaid are likely to urge hospitals and clinicians to become agents of cost control. Essential to these efforts to enhance quality and lower costs is comprehensive, objective information about the absolute and relative costs and benefits of medical interventions. Technology assessment in the United States has been hampered by pressure and limited resources. In the early 1990s, key federal agencies dedicated to technology assessment, such as the Congressional Office of Technology Assessment, were eliminated. Efforts by other federal agencies are fragmented and underfunded. The Department of Veterans Affairs, the National Institutes of Health, and the Centers for Medicare & Medicaid Services have little money for technology assessment. The mission of the Agency for Healthcare Policy and Research (AHCPR) was technology and outcomes evaluation. But, in 1994, when the AHCPR sponsored research showing that there was inadequate evidence to support commonly performed back operations, its funding was almost eliminated at the behest of disgruntled orthopedic and neurosurgeons and congressional critics of the Clinton health plan. While AHCPR survived, it was chastened. Its name was changed to the Agency for Healthcare Research and Quality and it has generally avoided controversial issues. Most importantly, little of its small budget—$320 million—is dedicated to evaluative research. State and private technology evaluation activities supplement federal efforts. In 1985, the Blue Cross and Blue Shield Association established the Technology Evaluation Center “for assessing medical technologies through comprehen-

In this issue of JME, Watson et al call for research evaluation of government health programmes and identify ethical guidance, including the Ottawa Statement on the ethical design and conduct...

Not available

Integration of leprosy control into primary health care is the most comprehensive and permanent system of delivering care to leprosy patients. But so far only a few countries have adopted this approach, largely on account of a fear of failure. Over the past decade Ghana has developed a model approach towards the transition from a vertical to an integrated programme. The highlights of our approach included the development of the leprosy service as part of the overall development of the health service, increasing capacity building for leprosy control at the district and subdistrict levels as well as the establishment of a regular and effective monitoring to identify and correct operational problems early. This paper describes the principles behind the integration, the strategies adopted and how they were implemented. It also includes the achievements made as well as the problems that were encountered and how they were solved.

Chapter 6 - Arab American youth: considerations for mental health and community engagement

Les interventions sanitaires sélectives: Un piège pour les politiques de santé du tiers monde

definitive overview of health technology assessment in the U.S. and Europe, the biotech industry, adoption and use of medtech in health care.---The World Future Society, Best Books and ReportsThis excellent book provides a broad overview of the development, impact, and evaluation of health technology in the United States. . . .The authors take a well-organized and thorough approach to address these topics, combining reviews of each with case examples of particular technologies. . . .Given the broad scope of the book, it could serve as a text for students, an introduction to the field for healthcare professionals, or a tool for academics and policymakers wishing to fill knowledge gaps outside their disciplines. . . .The book makes a compelling case for the logic and potential benefits of medical technology evaluation as a tool for improving health care.---Journal of the American Medical Association By being comprehensive in their review, the authors chart a clear path to understanding the future of health care technology in America. They clarify the technical methods for evaluation and provide insight into the sociopolitical aspects of development and diffusion. Case studies are informative. Excellent reading for students and health professionals either as a textbook or as an off-the-shelf guide to methods for deciding among alternative technologies.---Norman W. Weissman, University of Alabama Technology today dominates every aspect of health care. This useful book offers a diverse range of perspectives for students, professors, and medical practitioners who wish to understand how to evaluate medical technology.---Joel Howell, University of Michigan Medical School Technology in American Health Care is a comprehensive, multidisciplinary guide to understanding how medical advances-new drugs, biological devices, and surgical procedures-are developed, brought to market, evaluated, and adopted into health care. Cost-effective delivery of evidence-based health care is the sine qua non of American medicine in the twenty-first century. Health care decision makers, providers, payers, policymakers, and consumers all need vital information about the risks, benefits, and costs of new technologies in order to make informed decisions about which ones to adopt and how to use them. Alan B. Cohen and Ruth S. Hanft explore the evolving field of medical technology evaluation (MTE), as well as the current controversies surrounding the evaluation and diffusion of medical technologies, including the methods employed in their assessment and the policies that govern their adoption and use. The book opens with an introduction that provides basic definitions and the history of technological change in American medicine, and a second chapter that explores critical questions regarding medical technology in health care. Part I discusses biomedical innovation, the development and diffusion of medical technology, and the adoption and use of technology by hospitals, physicians, and other health care organizations and professions under changing health care market conditions. Part II examines the methods of MTE-including randomized controlled trials, meta-analyses, economic evaluation methods (such as cost-benefit, cost-effectiveness, and cost-utility analyses), and clinical decision analysis. Part III focuses on key public policy issues and concerns that affect the organization, financing, and delivery of health care and that relate importantly to medical technology, including safety, efficacy, quality, cost, access, equity, social, ethical, legal, and evaluation concerns. All three parts of the book provide a historical perspective on the relevant issues, methods, and policy concerns and contain examples of technologies whose development, adoption, evaluation, and use have contributed to our understanding of the field. This book will be invaluable in making MTE more accessible to individuals who are directly involved in the evaluation process and those who are touched by it in their professional lives-policymakers, clinicians, managers, and researchers.

Sophisticated methodological approaches and measures for scaling up are key elements of effective evaluation Digital health technologies and services are significant contributors to the transformation of health care delivery. It has been estimated that 80% of technology projects fail1 due to uncertainty, abandonment and lack of organisational willingness to adopt.2 In response to the high failure rate, the discipline of benefits management has emerged, with the aim of measuring and optimising the value of digital health initiatives. The development and application of benefits management has received some attention,3 but owing to the infancy of the discipline there has been limited assessment of methodological frameworks and their application. In this article, we describe the framework that is being used to measure and quantify the benefits of the My Health Record system in Australia. We consider the strengths and limitations of this framework in the context of existing frameworks, and its ability to demonstrate digital health system benefits. We also identify priority areas for further development of digital health benefit evaluation frameworks. Further, we provide an overview of the approach to digital health system benefits evaluation in Australia, in the context of the My Health Record system. A role of the Australian Digital Health Agency has been to lead the development of the National Digital Health Strategy, to lay the foundations for a safe, seamless and secure health system.4 This strategy comprises seven priority areas to be achieved by 2022. A key strategic priority is to provide health information whenever and wherever it is needed, and this is underpinned by the My Health Record system. My Health Record is a secure online summary of health information which can be accessed by people and their health care providers, and is patient-controlled. It is a personal health record, enabling people to access, manage and share their health information with their clinicians using a range of privacy controls. This functionality includes the ability to decline access to specific documents, set a control to restrict access to the entire record, see an audit trail of any organisations that have accessed the record, and block organisations from viewing the record. The record can contain summary information from general practice, hospital, pharmacy and other health care settings. It may also contain results of investigations, plus documents that patients create themselves (such as advance care plans and personal notes). In addition, it can provide access to Medicare documents such as Medicare Benefits Schedule and Pharmaceutical Benefits Scheme information, the Australian Immunisation Register and the Australian Organ Donor Register. A multimethod evaluation framework has been developed to comprehensively evaluate the benefits of the My Health Record system. It draws on approaches that have been used overseas and assesses the range of clinical contexts in which the system is used. Connecting people to their own health information has been shown to produce a range of benefits relating to patient engagement and a 60–80% improvement in their adherence to treatment regimens.5 In addition, enabling self-management has been associated with significant financial savings in terms of health care costs and avoided hospital admissions.6 It is also theorised that enabling clinicians across different health care settings to share information will result in improved patient safety (eg, fewer medication errors), improved care coordination, a reduction in unnecessarily duplicated investigations, and efficiency gains for clinicians in terms of time savings. In Australia, 2–3% of hospital admissions each year relate to medication misadventure, costing an estimated $1.2 billion annually.7 Improved access to medication information from a range of settings through the medicines information view in My Health Record should provide clinicians with more comprehensive information, and it is hoped that this will result in a reduction of medication misadventure events. Moreover, the use of digital health records to enable test results to be shared has been shown to reduce duplicated pathology tests by 18% per week,8 and in primary care settings it has been found that 13.6% of visits were missing important clinical information.9 Several lessons relating to the application of digital health research and evaluation frameworks have been learned. Many different "key measures" have been described, and there is an overall lack of consensus as to the "who, why, how, when and what" that should apply to an evaluation.10 The impact of this has been a failure to capture the complete range of players involved in the successful delivery of a system — players who do not necessarily share identical perspectives. A recent systematic review recommends that future frameworks present better methods for stakeholder identification and have a greater focus on understanding the context in which the system is delivered. From a systems perspective, this includes usability and organisational impact.10 The foundations of Australia's digital health system benefits evaluation framework are based on applying a range of measurement methods to capture a broad variety of outcome measures, reflecting a multistakeholder National Digital Health Strategy which offers different types of benefits to its different stakeholders. Five benefit workstreams have been introduced to evaluate the My Health Record system using qualitative, quantitative and mixed-method designs, as well as behavioural economic and health economic evaluation methods (Box). These workstreams are: Perceived improvement in access to patients' information and reduction in the need to order pathology tests and diagnostic imaging Self-reported experience of being able to view information that was previously unknown, and saving time requesting information Reduction in adverse medicine-related events through having access to a patient's medicines information Reduction in unnecessary duplication of pathology tests and diagnostic imaging The workstreams have been designed with data sources in mind to assist with planning and prioritising evaluation measures. These have been ranked by feasibility11 and impact, taking into account the relative importance of each measure to different stakeholders.10 To support two of these workstreams — customer and market insights, and impact evaluations — we have taken a similar approach to Canada's Clinical Adoption Framework.3 The impact evaluations workstream in particular focuses on outcomes and how these are being realised. For example, it focuses on measuring usability and adoption of digital tools from clinicians and consumers from their inception and throughout their development. The behavioural economics workstream refers to the evaluation methods which aim to measure change in the behaviour of system users (eg, clinicians and patients), organisational issues which can affect adoption, and indicators that would inform change management requirements to improve usability and adoption of digital health services.12 The data analytics workstream enables monitoring of trends in adoption and usage. For example, this workstream is investigating, through data modelling, whether My Health Record use is associated with fewer medication errors and reduced unnecessary duplication of pathology tests, to provide comprehensive results from a range of settings. The health economics evaluations workstream evaluates and forecasts indirect population health outcomes and downstream economic benefits using health economics modelling. For example, it is developing a health economics model to inform the ongoing business case for the My Health Record system, and conducting modelling that will support measurement of current and future benefits. To support an iterative process in product and service design, the Australian Digital Health Agency also undertakes user experience research and agile project management methods.13 The advantage of this approach is that it can deliver user insights which inform continual design and development of new features. Further, conducting field research to observe a range of digital system users in their environment is a way of identifying attributes that can influence the methods and measures used more broadly in our benefits evaluation framework. Current approaches and frameworks that have been developed overseas to support evaluation and benefits measurement of digital health services are at various stages of maturity. An increasing range of methods to evaluate digital health technologies is being supported by publications, which recognise that they are being deployed in complex health systems that require a contextual understanding of users, clinical settings and the policy environment in which they operate. We must not underestimate the challenges of evaluating benefits of digital health system delivery, and new methods to support evaluations continue to be developed and validated.2 Benefits evaluation frameworks offer a platform that can guide researchers and policy makers in generating and translating evidence to support future directions and ongoing investment in digital health services. Recent debates highlight the importance of fostering evaluation designs which combine different research methods, using qualitative, quantitative and co-design principles, as well as process measures1 that we have embedded into our benefits framework. To strengthen our current benefits evaluation framework and overall approach to digital health service evaluation, we are actively introducing methods that ascertain how services can be scaled up, to identify enablers and barriers to implementation across a range of settings.1 In this context, we define scalability as "the ability … to be expanded under real-world conditions to reach a greater proportion of the eligible population, while retaining effectiveness".14 Measuring scalability is not a commonly undertaken process and has been described as poorly understood.15 But the Australian Digital Health Agency is working towards applying these methods to build an understanding of impact. These findings form a picture of how, where and for whom the intervention could have the greatest impact and, conversely, what adaptations are needed for interventions to work across different population groups. A practical example of how this is being undertaken through the Agency is the introduction of and investment in a range of "test bed" studies. Test beds are projects assessing new digital-enabled models of care that are instigated and delivered cooperatively through sustainable and viable partnerships between industry, government and other organisations. Their purpose is to promote innovation to address Australia's highest priority health challenges and generate evidence on how the new approaches improve health outcomes. Currently, there are 15 test beds across Australia which are testing digital infrastructure and integration of digital health into clinical workflows.16 Despite ongoing interest in digital health benefits evaluation frameworks, few examples of their use in evaluation of digital health services have been published. The Australian Digital Health Agency's benefits evaluation framework will be used to justify future funding of digital health and to inform community and clinical education material. Moreover, the findings will be used to inform enhancements of the My Health Record system, ensuring that its progress is relevant and appropriate for clinicians and consumers. We are grateful to Darian Eckersley, from the Australian Digital Health Agency, for advice on product design methods. Many thanks to the broader Research Programs team at the Agency for reviewing parts of this manuscript and commenting on an earlier version. We are all employed by the Australian Digital Health Agency. Commissioned; externally peer reviewed.

Health Technology Assessment for Cardiovascular Digital Health Technologies and Artificial Intelligence: Why Is It Different?

This study explored the use of three participatory tools within a Philippines-based case study with community health workers (CHWs) by comparing and contrasting the process and data generated across the tools, and critically reflecting on adaptations and facilitation considerations that affected the tools’ use. Facilitator notes and audio-recordings of discussions were integrated and analyzed thematically. Tools differed by the type of data generated: program-specific data related to CHWs’ roles and responsibilities or data on broader structural factors. A stepwise approach within each tool facilitated focused, in-depth sharing, as did initial paired discussions that allowed exchange of knowledge and experiences among CHWs. Facilitators required topic- and context-specific knowledge to guide discussion effectively. CHWs discussed challenges and successes in their roles; program recommendations; and broader challenges related to healthcare delivery in their communities. This study contributes critical insights on the use of participatory tools to promote the inclusion of implementer perspectives in health program co-design, implementation, and evaluation.