Peer code review in research software development: The research software engineer perspective

Background: Research software is software developed by and/or used by researchers, across a wide variety of domains, to perform their research. Because of the complexity of research software, developers cannot conduct exhaustive testing. As a result, researchers have lower confidence in the correctness of the output of the software. Peer code review, a standard software engineering practice, has helped address this problem in other types of software. Aims: Peer code review is less prevalent in research software than it is in other types of software. In addition, the literature does not contain any studies about the use of peer code review in research software. Therefore, through analyzing developers perceptions, the goal of this work is to understand the current practice of peer code review in the development of research software, identify challenges and barriers associated with peer code review in research software, and present approaches to improve the peer code review in research software. Method: We conducted interviews and a community survey of research software developers to collect information about their current peer code review practices, difficulties they face, and how they address those difficulties. Results: We received 84 unique responses from the interviews and surveys. The results show that while research software teams review a large amount of their code, they lack formal process, proper organization, and adequate people to perform the reviews. Conclusions: Use of peer code review is promising for improving the quality of research software and thereby improving the trustworthiness of the underlying research results. In addition, by using peer code review, research software developers produce more readable and understandable code, which will be easier to maintain.

Peer review is the standard process within academia for maintaining publication quality, but it is also widely employed in other settings, such as education and industry, for improving work quality and for generating actionable feedback to content authors. For example, in the software industry peer review of program source code—or peer code review—is a key technique for detecting bugs and maintaining coding standards. In a programming education context, although peer code review offers potential benefits to both code reviewers and code authors, individuals are typically less experienced, which presents a number of challenges. Some of these challenges are similar to those reported in the educational literature on peer review in other academic disciplines, but reviewing code presents unique difficulties. Better understanding these challenges and the conditions under which code review can be taught and implemented successfully in computer science courses is of value to the computing education community. In this work, we conduct a systematic review of the literature on peer code review in higher education to examine instructor motivations for conducting peer code review activities, how such activities have been implemented in practice, and the primary benefits and difficulties that have been reported. We initially identified 187 potential studies and analyzed 51 empirical studies pertinent to our goals. We report the most commonly cited benefits (e.g., the development of programming-related skills) and barriers (e.g., low student engagement), and we identify a wide variety of tools that have been used to facilitate the peer code review process. While we argue that more empirical work is needed to validate currently reported results related to learning outcomes, there is also a clear need to address the challenges around student motivation, which we believe could be an important avenue for future research.

This full research category paper reports the result of an experiment that designed and implemented peer code review (PCR) to help introductory computer science (CS1) students understand programming concepts and improve their programming skills. Instructors at North Dakota State University have observed that students enrolled in CS1 programming course experience difficulty understanding programming concepts. We performed an empirical study that evaluated if PCR can help address the problem. We analyzed data collected from the PCR session, code development activity and end-of-study questionnaire. Our result provides insights into the most and least reported error types. Students were able to systematically improve the review output. More importantly, PCR had a significant positive impact on students' performance when developing their own code. PCR can be an effective teaching tool that when used in CS1 programming course can improve students' understanding of programming concepts and improve their programming skills.

Peer code review is an effective method to reduce the number of defects and maintain source code integrity. Peer re-views in most of the Open Source Software (OSS) communities are conducted via mailing list, which are difficult to manage at times. Code review tools aim to ease the review process and keep track of the review requests. In this paper, we describe preliminary results of our study to evaluate code review process using a popular open source code review tool (ReviewBoard) in OSS communities. Some of our study findings are similar to the findings of previous studies on code reviews. In the projects under our study, we found that, most of the revisions are not submitted for peer review. More than 80% of the review requests are responded by two or less number of reviewers. Top committers of the projects are also top contributors of code reviews. Most of the review requests get prompt feedback within a day; however, some requests might wait for feed-back for a long time. Most importantly, we have identified some interesting directions for future research.

Peer Code Review (PCR) is a proven pedagogical method for improving programming skills and collaboration. However, most gamified PCR systems emphasise reviewers while giving limited recognition to authors. This paper presents CODUAL, a conceptual dual-role gamified PCR framework that applies points, leaderboards, and badges for both roles. Using Likert-scale ratings and alignment-based rewards, CODUAL promotes balanced engagement and sustained effort. Grounded in Self-Determination Theory (SDT), it supports autonomy, competence, and relatedness to strengthen motivation. Future empirical evaluation, planned with undergraduate programming students at University Technical Malaysia Melaka (UTeM), will compare CODUAL with a non-gamified PCR platform using the Technology Acceptance Model (TAM) and BPNSFS.

Across essentially all fields of research, many aspects of the respective research processes – whether experimental, theoretical, empirical or outright computational – are closely related to software. Yet the process of finding software that is directly suitable or at least a good starting point for a given research task is cumbersome. This project aims to develop a community-driven system that provides potential users of research software with a diversity of pathways towards actually finding software that closely matches their research needs if such software exists. Conversely, it will provide software developers with mechanisms to make their software findable for research-related tasks and it will highlight mismatches between software supply and demand for specific tasks. To this end, we will document how various stakeholders of the research landscape have been searching for – or stumbling upon – research software so far, identify variables associated with successful search outcomes and build workflows that assist in describing software and associated concepts in a standardised fashion. These descriptions will then be aligned across various sources of relevant information and integrated into Wikidata, the knowledge graph that anyone can edit and that already contains considerable breadth and depth of information related to research, software and their interactions. While keeping an eye on similar approaches to software discovery that might work in parts of the research ecosystem, existing Wikidata content and workflows will be reviewed and built upon. Additional documentation, tooling and workflows will be developed to enrich, expand, curate, query and explore this content, both for specific use cases and with ongoing engagement of the communities involved in research software, open data or collaborative curation. Within its three years, the project seeks to establish a dedicated community overseeing a well-documented and smoothly running infrastructure for software discovery and to devise a plan for how this can be sustained for the longer term.

Process mining consists of mining event logs generated from business process execution supported by Information Systems (IS). Process mining of software repositories has diverse applications because vast data is generated during Software Development Life Cycle (SDLC) and archived in IS such as Version Control System (VCS), Peer Code Review (PCR) System, Issue Tracking System (ITS), and mail archives. There is need to explore its applications on different repositories to aid managers in process management. We conduct two phase surveys and interviews with managers in a large, global, IT company. The first survey and in-person interviews identify the process challenges encountered by them that can be addressed by novel applications of process mining. We filter, group and abstract responses formulating 30 generic problem statements. On the basis of process mining type, we classify identified problems to eight categories such as control analysis, organizational analysis, conformance analysis, and preventive analysis. The second survey asks distinct participants the importance of solving identified problems. We calculate proposed Net Importance Metric (NIM) using 1262 ratings from 43 participants. Combined analysis of NIM and first survey responses reveals that the problems mentioned by few practitioners in first survey are considered important by majority in the second survey. We elaborate on possible solutions and challenges for most frequent and important problems. We believe solving these validated problems will help managers in improving project quality and productivity.

Background and Context Peer code review activities provide well-documented benefits to students in programming courses. Students develop relevant skills through exposure to alternative coding solutions, producing and receiving feedback, and collaboration with peers. Despite these benefits, low student motivation has been identified as one of the challenges leading to poor engagement and substandard review quality. Objective This research investigates gamification as a technique for motivating students to generate high-quality reviews. Method We conduct a randomised controlled study, explore the nature and length of the feedback produced by students, and measure how students perceive the value of the feedback they receive and produce. We manually categorise students’ feedback into several categories adapted from a published taxonomy for student peer review. This categorisation indicates whether the feedback contains actionable advice and identifies strengths and weaknesses in the code. Findings We found that the quality of the feedback differed significantly between experimental and control conditions. Students in the experimental condition wrote longer comments and tended to produce more specific advice for their peers. Implications The findings can assist educators in understanding how the chosen game mechanics can be a potential strategy to motivate students to produce high-quality written feedback in peer code review activities.

A FAIRification roadmap for ELIXIR Software Management Plans

Research software is often built as prototypes that never get widespread usage and are left unmaintained after a few papers get published. To counteract this trend, we propose a method for building research software with scale and sustainability in mind so that it can organically grow a large userbase and enable longer-term research. To illustrate this method, we present the design and implementation of Python Tutor (pythontutor.com), a code visualization tool that is, to our knowledge, one of the most widely-used pieces of research software developed within a university lab. Over the past decade, it has been used by over ten million people in over 180 countries. It has also contributed to 55 publications from 35 research groups in 13 countries. We distilled lessons from working on Python Tutor into three sets of design guidelines: 1) user experience design for scale and sustainability, 2) software architecture design for long-term sustainability, and 3) designing a sustainable software development workflow within academia. These guidelines can enable a student to create long-lasting software that reaches many users and facilitates research from many independent groups.

Outside of performance-focused domains, research software is typically designed with output in mind rather than runtime efficiency. So, the resulting software consumes more resources (time, hardware) and is less scalable, hindering larger or longitudinal studies without adaptations. In this paper, we report our experiences of iteratively identifying and optimizing performance bottlenecks to enable such analyses in an established research software. Specifically, we applied a top-down strategy to Arcan, an architecture-smell detection tool, to develop a tool (AsTdEA) for tracing architecture smells through software evolution. To identify performance bottlenecks and benchmark our improvements, we used the Qualitas Corpus and a custom dataset. We achieved a reduction in processing time of approx. 98 % and reduced the runtime complexity from almost quadratic to close-to-linear. By sharing our process and insights, we hope to guide researchers in optimizing their research software in the future.

Scientific research relies on computer software, yet software is not always developed following practices that ensure its quality and sustainability. This manuscript does not aim to propose new software development best practices, but rather to provide simple recommendations that encourage the adoption of existing best practices. Software development best practices promote better quality software, and better quality software improves the reproducibility and reusability of research. These recommendations are designed around Open Source values, and provide practical suggestions that contribute to making research software and its source code more discoverable, reusable and transparent. This manuscript is aimed at developers, but also at organisations, projects, journals and funders that can increase the quality and sustainability of research software by encouraging the adoption of these recommendations.

This is the slide deck for the talk given by Bhavesh Patel at the Bioinformatics Open Source Conference (BOSC) 2024 conference, part of the International Conference on Intelligent Systems for Molecular Biology ( ISMB ) held in Montreal, Canada. Details about the conference are available at https://www.open-bio.org/events/bosc-2024. The talk instroduced Codefair, an open source and free GitHub app for making biomedical research software FAIR. Abstract: Abstract: We present Codefair, an innovative solution that helps researchers make their biomedical research software Findable, Accessible, Interoperable, and Reusable (FAIR), i.e. optimally reusable by humans and machines. The FAIR Biomedical Research Software (FAIR-BioRS) guidelines provide actionable instructions for making biomedical research software FAIR. While designed to be convenient to follow, we learned that their implementation can still be time consuming for researchers. To address this challenge, we are developing Codefair, a free and open source GitHub app that acts as a personal assistant for making research software FAIR. Researchers simply need to install Codefair from the GitHub marketplace and proceed with their software development as usual. By leveraging GitHub’s tools such as Probot, Codefair monitors activities on the software repository, communicates via GitHub issues, and submits pull requests to help researchers make their software FAIR. Currently, Codefair helps with including essential metadata elements such as license file, CITATION.cff metadata file, and codemeta.json metadata file. Additional features are being added to provide support for complying with best coding practices, archiving on Zenodo, registering on bio.tools, and much more to cover all the steps for making software FAIR. By alleviating their burden in the process, we believe Codefair will empower and encourage biomedical researchers into adopting FAIR and open practices for their research software. We present here our approach to developing Codefair, highlight the current and planned features, and explain how the community can benefit from and contribute to Codefair.

ELIXIR is an intergovernmental organization that brings together life science resources across Europe. These resources include databases, software tools, training materials, cloud storage, and supercomputers. One of the goals of ELIXIR is to coordinate these resources so that they form a single infrastructure. This infrastructure makes it easier for scientists to find and share data, exchange expertise, and agree on best practices. ELIXIR's activities are divided into the following five areas Data, Tools, Interoperability, Compute and Training known as “platforms”. The ELIXIR Tools Platform works to improve the discovery, quality and sustainability of software resources. Software Best Practices task of the Tools Platform aims to raise the quality and sustainability of research software by producing, adopting, promoting and measuring information standards and best practices applied to the software development life cycle. We have published four (4OSS) simple recommendations to encourage best practices in research software and the Top 10 metrics for life science software good practices. The next step is to adopt, promote, and recognise these information standards and best practices, by developing comprehensive guidelines for software curation, and through workshops for training researchers and developers towards the adoption of software best practices.

Semantic Web research relies on a number of key methodologies such as knowledge representation languages or reasoning algorithms. As a research community, however, we could not progress based on these methodologies exclusively, but require tools and systems that realize our research results as key technologies for the Semantic Web. There would be no Linked Data without repositories to store them, no Semantic Web applications without application programming interfaces to load and edit ontologies, and no semanticsbased user interfaces without reasoners. Consequently, some of these technologies are research enablers in their own rights. Unfortunately, the developers of such software— often researchers themselves—rarely get appropriate credit for their work. Attempts to publish papers describing their software systems frequently provoke reviews and decision letters questioning their ’conceptual contribution’. This criticism may be correct, be it because the conceptual underpinnings have been published previously, or because there may in fact be little conceptual underpinnings apart from excellent software engineering. As a result, developers of key research enabling software often have considerable difficulties publishing their work in high-quality journals. In turn, paradoxically, researchers using these systems within their own work lack a primary citation source for the used software, and may have to settle for a URL, a short demo paper, or for a workshop contribution as citation. These outlets, however, can not provide the rigid quality criteria and visibility compared to journal publications. There is a need for publishing Tools and Systems papers in primary journals. The criteria applied to evaluate such papers have to differ from those applied to classical research articles. Unfortunately, these alternative evaluation criteria are often not spelled out in detail and therefore not consistently taken into account by reviewers and editors. As a consequence, the Semantic Web journal has adopted the Tools and Systems article as one of its regular paper types and provides crisp criteria for their evaluation: Reports on tools and systems—short papers describing mature Semantic Web related tools and systems. These reports should be brief and pointed, indicating clearly the capabilities of the described tool or system. It is strongly encouraged, that the described tools or systems are free, open, and accessible on the Web. If this is not possible, then they have to be made available to the reviewers. For commercial tools and systems, exceptions can be arranged through the editors. These submissions will be reviewed along the following dimensions: (1) Quality, importance, and impact of the described tool or system (convincing evidence must be provided). (2) Clarity, illustration, and readability of the describing paper, which shall convey to the reader both the capabilities and the limitations of the tool. The impact of the described tool or system is indeed decisive. Authors are asked to provide evidence for a substantial number of researchers using their software for research and applications. In exceptional cases, authors may also argue about the potential future impact of their software given that they can provide clear evidence for this. Clearly, software prototypes which were only implemented as proof-of-concepts, or software which is closely tied to a particular (externally funded) project, are out of scope. Originality or novelty—which are decisive dimensions for high-quality research articles—are not mentioned among the evalu-