Abstract University research has a vital role to play in addressing complex societal challenges. The research impact (RI) agenda should enable this but is critiqued for creating an audit culture focused narrowly on economic returns on investment and university rankings. There is a need for alternative approaches that better support research for societal benefit. A current hiatus in research assessment processes in Australia provides an opportunity to explore alternatives. In this study, we elicited responses from 53 university staff in academic and professional roles to explore what constitutes research impact in practice, and what helps to achieve it. The responses highlight a disconnect between the current institutional framing of research impact and both the practices and values of those seeking to create societal benefit through research. We identify four tensions between the motivations and practice of research staff on one hand and the research impact agenda on the other. Tensions related to (1) narrow definitions of impact inadequately encompassing valuable work; (2) the premise of linear impact pathways inaccurately portraying the complexity of impact; (3) assessment rewarding individual endeavour over collaboration; and (4) assessment focusing on auditing rather than learning through evaluation. We take these findings and apply current theories of public and cultural value to offer ‘research value’ as an alternative approach to address the four tensions and nurture research for societal benefit.

Abstract Through an umbrella review, this article identified and surveyed 24 societal impact of research (SIR) models. Most of these models were developed within health domains and in Anglo-Saxon countries. The authors mapped the SIR models against constituent components of a robust theory of change. The study found that logic models were predominantly used to conceive SIR models. Yet, only nine models had explicit causal links, and only two made explicit assumptions about why research contributes to societal change. The old proverb among evaluators—when using theories of change to describe change—“and then a miracle occurs…,” rings uncomfortably true to the current state of SIR theorizing. Further theorizing and conceptual clarity are needed to advance the science of research impact.

Abstract In the current higher education landscape, universities are facing expanding requirements beyond teaching and research. Evaluation methods must evolve accordingly to prevent universities from facing development dilemmas. Current mainstream evaluation methods primarily emphasize the research domain, often failing to holistically capture a university’s overall performance and multifaceted contributions. From the functional perspective, this study views the university as a parallel input-output model consisting of separate subsystems. The proposed model demonstrates that a unique solution can be identified when assessing different functions, and it offers a cross-efficiency algorithm based on evidential-reasoning that can integrate peer evaluation. Utilizing data from 94 high-level universities in China, the proposed model exhibits improvement over alternatives. The analysis encompasses short- and long-term perspectives, subsystem and overall system viewpoints, and cross-year comparisons, while exploring dimensions like regional location, school type, and policy support. Lastly, this study analyzes China’s current policy situation and provides relevant recommendations based on the research findings.

Abstract Peer review by external experts is widely recognized as a legitimate and trustworthy academic practice, essential for ensuring the quality and rigor of research, providing more objective and less impartial assessments, and promoting transparent decision-making in science and academia. Research Funding Organisations (RFOs) usually rely on some form of peer review to evaluate the scientific quality of research proposals to allocate their limited resources. The peer review system is, however, also associated with several weaknesses, such as risks for bias and conflict of interest. This article explores the implications of replacing National Review Panels (NRPs) with International Review Panels (IRPs) in a national RFO, examining how this shift may impact the peer review process. Drawing on semi-structured interviews with staff from a national RFO in a European country and members of its IRPs, the article provides a nuanced analysis of both the potential benefits and challenges with substituting NRPs with IRPs. The results highlight how IRPs increase the distance between applicants and reviewers, which benefits the impartiality of the process. Nevertheless, this distance needs to be balanced by domestic panel members, chairs or research officers possessing appropriate knowledge of the local academic context, culture and structure. IRPs also introduce a greater diversity of perspectives into the assessments of applicants, which may promote objective and balanced assessments. The diversity may however also lower inter-reviewer reliability, and, in turn, complicate calibration practices and hinder the development of informal deliberative norms during the process of reaching decisions and consensus.

Abstract This paper presents a comparative study between the Brazilian and Dutch national evaluation systems of research and graduate education. Science systems can be as distinct as the social-economic circumstances, established governance, and cultural realities of each country, and both evaluation systems under analysis were developed from very different conceptions of assessment, university autonomy, and governance of higher education, science, and technology. From the study of policy and guiding documents, connected legislation, and related literature, we compare the design of the science system in each country, examining their impact on the adopted evaluation models. For that, we adapt established comparison frameworks and focus the analyses on influential aspects for each system, such as the links between evaluation and funding, or the consequential effect the results can have on researchers’ behaviour. While Brazil has lately sought inspiration from the long-standing and stable Dutch evaluation—which is recognized as a critical factor in the country’s quality assurance—the Latin American country has already developed one of the most sophisticated performance-based evaluation systems worldwide. Thus, we conclude by highlighting the risk of replicating international evaluation experiences without considering the local contexts. However, we consider that the differences and similarities unearthed in our study can inspire both nations, potentially inspiring methods and approaches from each evaluation system so that those lessons could lead to positive change for both countries.

Abstract Since 2022, China has achieved the global top position in the number of papers published in Nature Index journals, which has sparked assertions that China is challenging the scientific dominance of the United States. This study critically examines 27 bibliometric indicators used for national research assessment by comparing China with 19 other most productive countries in Nature Index journals. The findings indicate that different metrics yield diverse results, with China’s rankings ranging from first to 20th across the 27 indicators. On one hand, this highlights the need for cautious use of rankings and the importance of looking beyond mere rankings to understand in which aspects China has surpassed the USA. On the other hand, it underscores the necessity for a pluralistic approach to research evaluation, incorporating a variety of indicators to minimize potential biases and achieve a more comprehensive understanding of China’s research performance. Notably, metrics designed to capture quality, such as Novelty and Disruption, show limited discriminative power at the national level, as articles with high values for these indicators appear to be randomly distributed across countries, showing no systematic pattern that aligns with other performance indicators.

Abstract Canada does not have a national system wide assessment of the socioeconomic impacts of academic research. We do not have a Research Excellence Framework such as in the United Kingdom. Yet Canadian researchers, funders and institutions are interested in research impact, particularly the methods and processes for generating impacts to complement methods for assessing impact. At its heart, Canada is moving to combine its expertise in ‘how do we get to impact?’ with international expertise in ‘did we get to impact?’.

Abstract The Journal Impact Factor is not only one of the most widely accepted metrics for journal evaluation, but also serves as a guiding tool in shaping scientific content. Different direction innovation exhibits significant variations in terms of its conceptual definitions, diffusion processes, and societal impact. Therefore, exploring how JIF shapes the direction of innovation becomes a critical prerequisite for understanding innovation development. The paper focuses on a common negative signal—the declining JIF quartiles—to investigate how that affects the innovation direction of subsequent publications. To address this, the study employs the CD index to quantify journal’s innovation direction and utilizes staggered DID for empirical analysis. The results reveal that journal papers tend to be more conservative when the declining JIF quartiles, with disruptive innovations dropping by 23.81% from a quantitative perspective. Furthermore, this effect intensifies over time and exhibits significant variation across different quartiles. We further explored the potential origins of effect, suggesting that it may stem from the characteristics of disruptive innovations, which are misaligned with the interests inherent in the citation-based evaluation system.

Abstract This paper analyses the effects of new public management governance on academics’ job choices. Based on a choice experiment carried out with faculty from a sample of Germany’s leading technical universities, we find that working environments characterized by varying levels of administrative burdens and high expectations concerning third-party funding acquisition are detrimental to self-actualization and hence tend to repel potential candidates. More specifically, we find this effect to be most pronounced for those candidates that universities would be strategically most interested in: researchers with a strong track record and those with an interdisciplinary profile. Without denying potential benefits of external incentives for existing faculty, we therefore suggest acknowledging intrinsic motivation as the key driving factor of academics’ choices and to design future governance structures accordingly.

Abstract This study analyzed 33 articles related to 11 countries with a performance-based research funding (PBRF) system to explore whether the adoption of a PBRF contributed to research performance after these countries adopted this system. In our analysis, we found that PBRF resulted in positive outcomes in general. In addition, these positive outcomes differ by the type of PBRF design where a metric-based peer review system resulted in more positive outcomes than the systems with peer review system and bibliometric system. This is because a metric-based peer review system combines the strengths from peer review and a bibliometric system. However, this result does not imply that a metric-based peer review system is more effective than the other two systems in any context because each country developed their own systems within their own contexts. This systematic literature review contributes to understanding the overall effects of PBRF across countries and between different policy designs. The details of policy design and their outcomes in a specific research system should be further studied and discussed through case studies.