Mixed methods analysis strategies in program evaluation beyond “a little quant here, a little qual there”

A genuine mixed methods study culminates in a meta-inference, a conclusion that connects or integrates various claims, some resulting from the analysis of qualitative data, others from the analysis of quantitative data. Paradoxically, however, the integration of claims has received very little attention in the mixed methods literature. When mixed methods scholars discuss integration, they are almost always referring to mixing methods, often called integration of methods. This chapter seeks to fill this gap. Using three real-life examples of mixed methods research, it shows how the meta-inference and its internal structure are developed and emerge in successive steps of separate instances of claim integration. In addition to (a) mixed methods or (b) multi-method, an instance of integration can be (c) mono-method (involving only one method) or involve claims obtained (d) in previous research or (e) through reasoning (without data collection). Furthermore, this chapter shows that claim integration is not unique to mixed methods research but occurs in mono-method research as well. In line with the emergent nature of the meta-inference, furthering claims through contradiction and explanation, rather than robustness of claims, determines the value of a mixed methods or mono-method study.

A mixed methods research design, which is a complex approach, combines both quantitative and qualitative data in a single study or succession of studies. This design can be particularly functional for exploring complex research questions that cannot be fully answered by using a single research design. Moreover, a mixed methods design is necessary to examine the relationships between different variables because examining the relationships between diverse variables is not viable just through a single research design. This design is required to complement and cater to the increasingly multifarious requirements of contemporary researchers. This article, which explores and discusses types, purposes, strengths, challenges and criticisms of the mixed methods research design as its objectives, stems from an analysis of some notable works in the field. It is grounded on the secondary qualitative data accumulated in the forms of words from journal articles and books related to the research designs. It assists the novices in the field of research in particular and other researchers in general by providing them with an overview of mixed methods design along with its types, such as convergent parallel, explanatory sequential, exploratory sequential, embedded, transformative and multi-phage designs.

Chapter 28 - Best practices in mixed methods for pharmacy and health services research

Mixed methods research is defined as the collection, analysis, and integration of both quantitative data (e.g., outcome of a randomized trial, such as decrease in depression score) and qualitative data (e.g., observations or interviews about the experience of persons with depression) to provide a more comprehensive understanding of a research problem than might be obtained through quantitative or qualitative approaches alone. Because mixed methods research designs place high value on the stories behind the numbers– both in exploratory designs where the experiences and insights of the community under study inform the quantitative investigation, and in explanatory designs where they illuminate the quantitative data– mixed methods are especially appropriate in studying the problems of older adults and caregivers. The goal of this chapter is to introduce the concept of mixed methods as an approach for research in Geriatrics and Gerontology. We will discuss (1) why an investigator might consider using mixed methods; (2) basic and complex mixed methods designs; (3) examples of mixed methods projects, drawing upon our own research; and, (4) challenges in carrying out mixed methods research.

ABSTRACTWhile social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.

Mixed methods, cross-cultural designs are a special case of the more general, mixed methods between-groups designs. The defining feature of mixed methods is the principled integration of qualitative and quantitative approaches, and cross-cultural comparison is premised on the systematic collection and analysis of data from two or more cultural settings for the explicit purpose of between-groups comparisons. This article explores the interaction of mixed methods integration and cross-cultural between-groups comparison to develop cross-cultural versions of five prototypical mixed methods designs: sequential (exploratory and explanatory), parallel, conversion, fully integrated, and multilevel. Each is exemplified by published studies. Final reflections explore possible additional contributions that mixed methods might make to issues of cross-cultural equivalence in data collection and analysis.

Triangulation is a measurement technique often used by surveyors to locate an object in space by relying on two known points in order to ‘‘triangulate’’ on an unknown fixed point in that same space. Early on, social scientists borrowed the concept of triangulation to argue for its use in the validation process in assessing the veracity of social science research results. There are alternative perspectives on the use of triangulation that argue for its usefulness as a ‘‘dialectical’’ process whose goals seek a more in-depth nuanced understanding of research findings and clarifying disparate results by placing them in dialogue with one another. This special issue of the Journal of Mixed Methods Research (JMMR) analyzes and explores the variety of ways triangulation is used in mixed methods research and the range of issues and controversies surrounding triangulation praxis. To date, there are few scholarly in-depth discussions of its deployment in mixed methods research. The choice of triangulation as the topic for this first special issue of JMMR is based on the claims made by many scholars in the field that triangulation provides a justification for the use of mixed methods. The contributors to this volume raise many questions about the meaning of triangulation, its philosophical positioning in the mixed methods community, and strategies for using triangulation in the design of mixed methods studies, analysis and interpretation of data, and making visible subjugated voices. They take provocative positions, suggesting that qualitative, constructivist, and interpretive pathways provide greater potential for research to address the social good than has been possible using mixed methods approaches that are more closely aligned with the postpositivist paradigm. They revisit the ‘‘paradigm wars’’ and ask this question: Are we still stuck with the incompatibility thesis that paralyzed advances in mixed methods in past decades? They explore and critique the potential of alternative methodologies for harnessing the synergy that is said to lie in the application of mixed methods research designs by asking another set of questions: Have members of the mixed methods community done an injustice to pragmatism as a philosophical frame for mixed methods? Is qualitatively framed mixed methods the way forward? Is it possible that qualitatively framed mixed methods are better suited to the ability of mixed methods researchers to demonstrate a causal relationship between variables? How and when should triangulation be brought into mixed methods research to obtain a more nuanced

Deaf people, members of a complex global language minority, have been excluded from positions of power in the field of humanitarian aid and development assistance, as well as from playing a significant role in the conduct of research that affects their lives. Deaf people rarely lead or participate as researchers in studies about their own communities or about the assistance they receive. Transformative mixed methods research strategies focus on the development of culturally respectful relationships to enhance collaboration between members of dominant and marginalized communities in order to improve the accuracy and usefulness of the findings. In this article, the application of transformative mixed methods in the deaf community provides examples of how research reveals complex realities and how researchers employ mixed methods strategies to appropriately engage members of marginalized communities in the context of development assistance to support human rights.

Mixed methods research has emerged alongside qualitative and quantitative approaches as an important tool for health services researchers. Despite growing interest, among health services researchers, in using mixed methods designs, little has been done to identify the procedural aspects of doing so. To describe how mixed methods researchers mix the qualitative and quantitative aspects of their studies in health services research. We searched the PubMed for articles, using mixed methods in health services research, published between January 1, 2006 and December 30, 2010. We identified and reviewed 30 published health services research articles on studies in which mixed methods had been used. We selected 3 articles as illustrations to help health services researcher conceptualize the type of mixing procedures that they were using. Three main "mixing" procedures have been applied within these studies: (1) the researchers analyzed the 2 types of data at the same time but separately and integrated the results during interpretation; (2) the researchers connected the qualitative and quantitative portions in phases in such a way that 1 approach was built upon the findings of the other approach; and (3) the researchers mixed the 2 data types by embedding the analysis of 1 data type within the other. "Mixing" in mixed methods is more than just the combination of 2 independent components of the quantitative and qualitative data. The use of "mixing" procedure in health services research involves the integration, connection, and embedding of these 2 data components.

A well-recognized gap exists between findings from public health research and their use in public health practice to improve outcomes. The World Health Organization (WHO) has identified implementation research as vital to improving the adaptation and dissemination of public health interventions into real-world settings. Implementation research encompasses multimethod evaluation approaches; 'mixed methods', the planned integration of qualitative and quantitative methods, is a key tool. We argue that mixed methods designs are crucial for design and evaluation of public health interventions, provide illustrative case studies, discuss key analytic approaches in mixed methods design, identify resources for mixed methods research and advocate for more training. On behalf of the International Network for Epidemiology in Policy (INEP), an international non-profit organization of 24 member societies that promotes the development of equitable, evidence-based health policies, we call for increased collaboration between qualitative and quantitative research teams to improve the design and evaluation of public health interventions.

Rising quality assurance standards and levels of accountability in higher education have placed stress on my organization, a two-year community college located in Micronesia, a sub-region of Oceania, in the western Pacific Ocean, which has historically maintained a culture of reactivity to accreditation sanctions. As quality standards increase in rigor and accountability, the demands on employee engagement become ever greater. Today, accreditation requires every employee be engaged in cycles of sustainable continuous quality improvement, assessment of student learning, purposeful dialogue, and institutional effectiveness. US regional accreditation is vital to maintain fiscal integrity of the organization. If terminated, students lose access to Pell Grant, a US Department of Education Title IV funding initiative for low-income students in undergraduate programs of study. Pell Grant is the organization’s single most important revenue source, captured from students through tuition and fees. Because revenue sources are not diversified, loss of US regional accreditation, and subsequent loss of Pell Grant, would thus negatively impact organizational stability. This study examined transformational leadership as a potential means for enhancing employee engagement, thereby increasing organizational potential for responding to evolving accreditation standards. This mixed methods study explored the relationship between perceived transformational leadership and follower work engagement within my organization, described how leaders enhanced follower engagement, and investigated to what extent and in what ways the background, training, development, and experiences of organization leaders contributed to leadership skills and their ability to enhance follower engagement. A sequential mixed methods design was employed for which first quantitative data, and then qualitative data, were collected and analyzed. Quantitative data were used on a Micronesian higher education context to test transformational leadership theory that predicts transformational leadership is positively correlated with employee engagement, using the Multifactor Leadership Questionnaire (MLQ). Quantitative follower engagement data were collected using the Utrecht Work Engagement Scale (UWES). Qualitative data from semi-structured interviews were then analyzed to further explore leadership and its ability to enhance employee engagement for followers at this Micronesian higher education institution. This study does not substantiate the importance of transformational leadership for enhancing employee engagement, but instead shows transactional contingent reward is more important in this cultural and institutional context. The college leadership does not generally meet the expectations of the Multifactor Leadership Questionnaire full range leadership model, yet college employees are significantly highly engaged. Additionally, analysis of the quantitative data obtained contributes to academic discussions on potential problems with the MLQ and UWES and show these constructs may not be ideal for measuring transformational leadership or engagement. Institutional recommendations for training current and future college leaders and for developing cross-sector partnerships are given. In addition, the wider implications for future research and practice are provided.

The study has been taken in order to explore the concept of entrepreneurship ecosystem and how the development of ecosystem model has an impact on the development of entrepreneurship in Pakistan. Hence, the top-down and bottom up approach both were used simultaneously in order to comprehend the entrepreneurship ecosystem phenomenon under investigation. The entrepreneurship ecosystem has been analyzed through SMEDA SME policies, cultural tendency and private educational institutions’ capacities in Pakistan in the quantitative part of the study. This study is however, guided by Isenberg (2011) and Mason and Brown (2014) through their scholarly work on entrepreneurship and entrepreneurship ecosystem. The mixed methods research strategy and parallel convergent design were used to understand and explore the phenomenon and hence quantitative and qualitative data were collected simultaneously. Although, the quantitative data were collected through convenience sampling from six universities, interviews were conducted at the same time from academic professionals, incubation center representatives and entrepreneurs as well. The quantitative data were then analyzed through SPSS and qualitative data through trial version of NVivo 12. The quantitative results show the significant relationship between SMEDA SME policies and development of entrepreneurship; however, the strength of relationship was considered to be weak. Similarly, the significant relationship was also found between cultural tendency and development of entrepreneurship in Pakistan; however strength of relationship was also weak. Moreover, the significant relationship was found between private educational institutions’ capacities and development of entrepreneurship but, the strength of relationship was weak in nature. However, while testing the final hypothesis, the relationship between cumulative SMEDA policies, cultural tendency, private educational intuitions’ capacities and entrepreneurship development in Pakistan was also found with moderate strength of association. While converging the data from quantitative and qualitative part, it was observed that, both data do converge at some point, because factors emerged in the qualitative data (family system, educational system, entrepreneurial attributes, and legal framework and incubation centers) and the results found in the quantitative data do find the relationship between with the entrepreneurship ecosystem and the development of entrepreneurship in Pakistan.

To examine how physical layouts and space in ambulatory oncology practices influence patient safety and clinician communication. Ambulatory oncology practices face unique challenges in delivering safe care. With increasing patient volumes, these settings require additional attention to support patient safety and efficient clinical work processes. This study used a mixed methods design with sequential data collection. Eight ambulatory oncology practices (of 29 participating practices) participated in both the quantitative and qualitative phases. In surveys, clinicians (n = 56) reported on safety organizing and communication satisfaction measures. Qualitative data included observations and semistructured interviews (n = 46) with insight into how physical layout influenced care delivery. Quantitative analysis of survey data included descriptive and correlational statistics. Qualitative analysis used inductive and thematic content analysis. Quantitative and qualitative data were integrated using side-by-side comparison tables for thematic analysis. Safety organizing performance was positively correlated with clinician communication satisfaction, r(54 df) = .414, p = .002. Qualitative analyses affirmed that the physical layout affected communication around chemotherapy infusion and ultimately patient safety. After data integration, safety organizing and clinician communication were represented by two themes: visibility of patients during infusion and the proximity of clinicians in the infusion center to clinicians in the clinic where providers see patients. Physical layouts of ambulatory oncology practices are an important factor to promote patient safety. Our findings inform efforts to construct new and modify existing infusion centers to enhance patient safety and clinician communication.

Mixed method designs are often used in scale development to generate potential items and enhance face validity. Less frequently, mixed method designs are used to examine other aspects of validity. As part of a larger evaluation in the Lao People’s Democratic Republic, we developed a scale to assess household self-reported changes in access to livelihood assets following a mine action program. In this article, we detail the process of assessing measurement validity using mixed methods. The qualitative method of evaluation was based on an analysis of in-depth interviews with key informants. The quantitative evaluation relied on data collected from the scale administration ( N = 271) evaluated using Rasch measurement. The article offers an example to researchers interested in using mixed methods to assess validity in measurement development. It contributes to the growing literature on using mixed method designs to evaluate the properties of scales and to the methodological work on the measurement of access to livelihood assets.

<h3>Background</h3> Ensuring effective access to vaccinations for people experiencing homelessness is crucial to protecting the health of a vulnerable, yet often overlooked population. Reaching this goal with COVID-19 vaccines takes more than a one size fits all approach. This study evaluates how a dedicated health team in Cork City collaborated with multiple agencies to register and deliver the COVID-19 vaccine to one of the most difficult to reach populations. <h3>Methods</h3> This mixed methods study includes the collection and analysis of quantitative and qualitative data and was co-designed with the Adult Homeless Integrated Health Team. Quantitative data on age, gender, nationality, language and accommodation status collected by the health team at the point of vaccine registration will be available for service users who participated in the vaccination programme. Semi-structured interviews will be conducted with staff and service users to explore their experiences with the pandemic and the delivery (staff) or receipt (service users) of the COVID-19 vaccine. The eleven agencies involved will be invited to participate in the interviews. A manager in each of the homeless agencies will ask service users within their agencies if they would like to participate in the study. <h3>Results</h3> The health team in partnership with all the agencies administered 728 vaccine doses to people experiencing homelessness in Cork City since 10th May 2021, 401 first doses were administered with a 95.2% uptake for the second dose. Study participants are mainly male (77%), middle aged (mean aged 41 years) and predominantly Irish (87%). Ten staff members who were involved in the vaccine rollout have agreed to participate in an interview and these interviews are ongoing. Initially, four interviews with staff have identified barriers for accessing mainstream vaccination services including addiction, mental health disorders and external stresses. Facilitators included organised transport, the help of key workers and trusting relationships between staff and clients. Face to face interviews with service users will commence in March. <h3>Conclusion</h3> This mixed method evaluation of an innovative approach to the delivery of vaccinations to very vulnerable and high-risk groups will provide insights into the design of services for the homeless population and will help inform future interventions for practitioners and policy makers, therefore improving health services for the homeless population.