Digital health benefits evaluation frameworks: building the evidence to support Australia's National Digital Health Strategy.
Sophisticated methodological approaches and measures for scaling up are key elements of effective evaluation Digital health technologies and services are significant contributors to the transformation of health care delivery. It has been estimated that 80% of technology projects fail1 due to uncertainty, abandonment and lack of organisational willingness to adopt.2 In response to the high failure rate, the discipline of benefits management has emerged, with the aim of measuring and optimising the value of digital health initiatives. The development and application of benefits management has received some attention,3 but owing to the infancy of the discipline there has been limited assessment of methodological frameworks and their application. In this article, we describe the framework that is being used to measure and quantify the benefits of the My Health Record system in Australia. We consider the strengths and limitations of this framework in the context of existing frameworks, and its ability to demonstrate digital health system benefits. We also identify priority areas for further development of digital health benefit evaluation frameworks. Further, we provide an overview of the approach to digital health system benefits evaluation in Australia, in the context of the My Health Record system. A role of the Australian Digital Health Agency has been to lead the development of the National Digital Health Strategy, to lay the foundations for a safe, seamless and secure health system.4 This strategy comprises seven priority areas to be achieved by 2022. A key strategic priority is to provide health information whenever and wherever it is needed, and this is underpinned by the My Health Record system. My Health Record is a secure online summary of health information which can be accessed by people and their health care providers, and is patient-controlled. It is a personal health record, enabling people to access, manage and share their health information with their clinicians using a range of privacy controls. This functionality includes the ability to decline access to specific documents, set a control to restrict access to the entire record, see an audit trail of any organisations that have accessed the record, and block organisations from viewing the record. The record can contain summary information from general practice, hospital, pharmacy and other health care settings. It may also contain results of investigations, plus documents that patients create themselves (such as advance care plans and personal notes). In addition, it can provide access to Medicare documents such as Medicare Benefits Schedule and Pharmaceutical Benefits Scheme information, the Australian Immunisation Register and the Australian Organ Donor Register. A multimethod evaluation framework has been developed to comprehensively evaluate the benefits of the My Health Record system. It draws on approaches that have been used overseas and assesses the range of clinical contexts in which the system is used. Connecting people to their own health information has been shown to produce a range of benefits relating to patient engagement and a 60–80% improvement in their adherence to treatment regimens.5 In addition, enabling self-management has been associated with significant financial savings in terms of health care costs and avoided hospital admissions.6 It is also theorised that enabling clinicians across different health care settings to share information will result in improved patient safety (eg, fewer medication errors), improved care coordination, a reduction in unnecessarily duplicated investigations, and efficiency gains for clinicians in terms of time savings. In Australia, 2–3% of hospital admissions each year relate to medication misadventure, costing an estimated $1.2 billion annually.7 Improved access to medication information from a range of settings through the medicines information view in My Health Record should provide clinicians with more comprehensive information, and it is hoped that this will result in a reduction of medication misadventure events. Moreover, the use of digital health records to enable test results to be shared has been shown to reduce duplicated pathology tests by 18% per week,8 and in primary care settings it has been found that 13.6% of visits were missing important clinical information.9 Several lessons relating to the application of digital health research and evaluation frameworks have been learned. Many different "key measures" have been described, and there is an overall lack of consensus as to the "who, why, how, when and what" that should apply to an evaluation.10 The impact of this has been a failure to capture the complete range of players involved in the successful delivery of a system — players who do not necessarily share identical perspectives. A recent systematic review recommends that future frameworks present better methods for stakeholder identification and have a greater focus on understanding the context in which the system is delivered. From a systems perspective, this includes usability and organisational impact.10 The foundations of Australia's digital health system benefits evaluation framework are based on applying a range of measurement methods to capture a broad variety of outcome measures, reflecting a multistakeholder National Digital Health Strategy which offers different types of benefits to its different stakeholders. Five benefit workstreams have been introduced to evaluate the My Health Record system using qualitative, quantitative and mixed-method designs, as well as behavioural economic and health economic evaluation methods (Box). These workstreams are: Perceived improvement in access to patients' information and reduction in the need to order pathology tests and diagnostic imaging Self-reported experience of being able to view information that was previously unknown, and saving time requesting information Reduction in adverse medicine-related events through having access to a patient's medicines information Reduction in unnecessary duplication of pathology tests and diagnostic imaging The workstreams have been designed with data sources in mind to assist with planning and prioritising evaluation measures. These have been ranked by feasibility11 and impact, taking into account the relative importance of each measure to different stakeholders.10 To support two of these workstreams — customer and market insights, and impact evaluations — we have taken a similar approach to Canada's Clinical Adoption Framework.3 The impact evaluations workstream in particular focuses on outcomes and how these are being realised. For example, it focuses on measuring usability and adoption of digital tools from clinicians and consumers from their inception and throughout their development. The behavioural economics workstream refers to the evaluation methods which aim to measure change in the behaviour of system users (eg, clinicians and patients), organisational issues which can affect adoption, and indicators that would inform change management requirements to improve usability and adoption of digital health services.12 The data analytics workstream enables monitoring of trends in adoption and usage. For example, this workstream is investigating, through data modelling, whether My Health Record use is associated with fewer medication errors and reduced unnecessary duplication of pathology tests, to provide comprehensive results from a range of settings. The health economics evaluations workstream evaluates and forecasts indirect population health outcomes and downstream economic benefits using health economics modelling. For example, it is developing a health economics model to inform the ongoing business case for the My Health Record system, and conducting modelling that will support measurement of current and future benefits. To support an iterative process in product and service design, the Australian Digital Health Agency also undertakes user experience research and agile project management methods.13 The advantage of this approach is that it can deliver user insights which inform continual design and development of new features. Further, conducting field research to observe a range of digital system users in their environment is a way of identifying attributes that can influence the methods and measures used more broadly in our benefits evaluation framework. Current approaches and frameworks that have been developed overseas to support evaluation and benefits measurement of digital health services are at various stages of maturity. An increasing range of methods to evaluate digital health technologies is being supported by publications, which recognise that they are being deployed in complex health systems that require a contextual understanding of users, clinical settings and the policy environment in which they operate. We must not underestimate the challenges of evaluating benefits of digital health system delivery, and new methods to support evaluations continue to be developed and validated.2 Benefits evaluation frameworks offer a platform that can guide researchers and policy makers in generating and translating evidence to support future directions and ongoing investment in digital health services. Recent debates highlight the importance of fostering evaluation designs which combine different research methods, using qualitative, quantitative and co-design principles, as well as process measures1 that we have embedded into our benefits framework. To strengthen our current benefits evaluation framework and overall approach to digital health service evaluation, we are actively introducing methods that ascertain how services can be scaled up, to identify enablers and barriers to implementation across a range of settings.1 In this context, we define scalability as "the ability … to be expanded under real-world conditions to reach a greater proportion of the eligible population, while retaining effectiveness".14 Measuring scalability is not a commonly undertaken process and has been described as poorly understood.15 But the Australian Digital Health Agency is working towards applying these methods to build an understanding of impact. These findings form a picture of how, where and for whom the intervention could have the greatest impact and, conversely, what adaptations are needed for interventions to work across different population groups. A practical example of how this is being undertaken through the Agency is the introduction of and investment in a range of "test bed" studies. Test beds are projects assessing new digital-enabled models of care that are instigated and delivered cooperatively through sustainable and viable partnerships between industry, government and other organisations. Their purpose is to promote innovation to address Australia's highest priority health challenges and generate evidence on how the new approaches improve health outcomes. Currently, there are 15 test beds across Australia which are testing digital infrastructure and integration of digital health into clinical workflows.16 Despite ongoing interest in digital health benefits evaluation frameworks, few examples of their use in evaluation of digital health services have been published. The Australian Digital Health Agency's benefits evaluation framework will be used to justify future funding of digital health and to inform community and clinical education material. Moreover, the findings will be used to inform enhancements of the My Health Record system, ensuring that its progress is relevant and appropriate for clinicians and consumers. We are grateful to Darian Eckersley, from the Australian Digital Health Agency, for advice on product design methods. Many thanks to the broader Research Programs team at the Agency for reviewing parts of this manuscript and commenting on an earlier version. We are all employed by the Australian Digital Health Agency. Commissioned; externally peer reviewed.
Read more
10