Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review

Artificial intelligence and recent advances in deep learning architectures, including transformer networks and large language models, change the way people think and act to solve problems. Software engineering, as an increasingly complex process to design, develop, test, deploy, and maintain large-scale software systems for solving real-world challenges, is profoundly affected by many revolutionary artificial intelligence tools in general and machine learning in particular. In this roadmap for artificial intelligence in software engineering, we highlight the recent deep impact of artificial intelligence on software engineering by discussing successful stories of applications of artificial intelligence to classic and new software development challenges. We identify the new challenges that the software engineering community has to address in the coming years to successfully apply artificial intelligence in software engineering, and we share our research roadmap toward the effective use of artificial intelligence in the software engineering profession, while still protecting fundamental human values. We spotlight three main areas that challenge the research in software engineering: the use of generative artificial intelligence and large language models for engineering large software systems, the need of large and unbiased datasets and benchmarks for training and evaluating deep learning and large language models for software engineering, and the need of a new code of digital ethics to apply artificial intelligence in software engineering.

We welcome you to the Twenty-Eighth International Conference on Software Engineering and Shanghai, China. This is the first instance of the ICSE conference series to be held in China, and it is an exciting time and place for ICSE 2006. The conference brings researchers, educators, practitioners and students together to present, exchange ideas and engage in discussions about the recent and exciting innovations and research in software engineering. Exciting sessions on practical experience, educational issues, and doctoral and new faculty symposia have been organized. The conference also includes a wide variety of tutorial, workshops and demonstration offerings. Finally, since research and markets in our host country China are continuously growing, ICSE includes a separate Far East Track for discussions on the contributions and challenges of our Chinese colleagues. We trust that this mix will provide interesting and stimulating offerings to all members of our growing community. The technical program committee is pleased to present this conference proceeding that contains the research papers of the technical program for ICSE 2006. The technical program for this conference focuses on new and innovative software engineering contributions in theory, formal methods, system building and empirical studies. The papers represent an alluring array of topics, including software design, architecture, reuse, testing, analysis, processes and formal methods. Because of the growing number of paper submission and the resulting large program committees in the past, we instituted a new two-phase reviewing process this year. Our program committee had 36 members. In the first phase, each submitted paper was reviewed by 3 reviewers: 2 from the program committee and an external reviewer from the software engineering community. We had 265 external reviewers who contributed 370 reviews for this phase. Those papers receiving the best reviews were considered in the second phase. A total of 99 papers entered the second phase. Each of these papers was reviewed by another program committee member. Thus, each accepted paper had a total of 4 reviews (3 reviews from pc members and 1 review from an external reviewer). Each rejected paper had at least 3 reviews. The pc members reviewed 24-25 papers in the two phases, a reasonable number. We feel that this system worked very well. The program committee met for two days in December, 2005 in Frankfurt, Germany. By the end of the meeting, a consensus emerged for the 36 papers in this proceedings (9% of the total submissions). There were many high quality submissions that could not be accommodated in the program; hopefully a revised version will be published elsewhere. In addition to the papers, we organized four invited sessions. One invited session focuses on empirical studies, an increasingly important component of software engineering research and practice. We also have an invited panel that addresses the value and practice of formal methods. Since software engineering techniques are growing in importance in challenging safety critical domains, we included two invited sessions: one on software engineering for automotive engineering and the other on software engineering for assisted living. We are very pleased that a notably diverse and outstanding group of three leaders of the software engineering community have agreed to present keynote speeches. The opening day keynote will be presented by Prof. Yang Fuqing, from Peking University, a woman who has been at the very forefront of the software engineering communities of research and practice in China for decades. Her perspectives on the past and future of software engineering in China will be complemented by a keynote on similar themes presented on the second day. This second keynote will be delivered by Prof. Barry Boehm, from the University of Southern California, USA. His talk will review the history of software engineering research and practice in the West over the past decades, and project it forward into the 21st century. Finally, the keynote speaker on the third day, Mr. Reinhold Achatz, from Siemens Corp., will present a complementary view of software engineering research and practice, as viewed from the perspective of one of the great multinational corporations, based in Europe. We are truly excited at the prospect that these three keynotes will provide a searching multidimensional view of our community at this time of historic convergence of communities. The continued success and high quality of the ICSE conference series would not be possible without the help and support of the software engineering community. First, we would like to gratefully acknowledge and thank all of the authors who submitted papers. We especially thank the program committee members for their efforts and commitments in reviewing the papers and securing reviews from external reviewers. They put in an extraordinary amount of their time to provide thoughtful and knowledgeable reviews. The exceptionally strong technical program represented by the papers in this proceeding could not be possible without their efforts.

Introduction and Vision I am honored to be taking over as Editor in Chief of the IEEE Transactions of Software Engineering (TSE). Jeff Kramer will be a hard act to follow, but I am immensely grateful to him for handing over the journal in such a healthy state, and for helping me learn the new job to ensure a smooth transition. It goes without saying that my primary goal as Editor is to maintain the journal’s reputation and standing as the leading forum for publishing the highest quality of research in software engineering. However, these are exciting and changing times in the world of publishing and in the discipline of software engineering, and I believe that TSE must lead the way by finding innovative ways to disseminate software engineering research and by contributing to setting and reflecting the research agenda in the field. To this end I have three medium term goals for my tenure as Editor: 1. To reemphasize the broad scope of the journal, encouraging the publication of multidisciplinary research as well as specialization in software engineering. 2. To engage with the software engineering community at large, including other journals, about the software engineering discipline, using not only the journal itself as a forum but through other professional outlets such as conferences and the web. 3. To support a debate on the nature of scholarly discourse in software engineering, such as the kind and length of papers published in the journal, or the different forms of peer review and discussion around what is published. Tactics The length of papers published in TSE has on occasion been a topic for some discussion. I do not propose to change the policy in this area but to clarify it, as I think it hides a number of issues within it. TSE publishes papers of any length, as long as their length is appropriate for the content presented. The discussion has often centered around what to do if accepted papers are over the IEEE Computer Society’s recommended length of 15 pages. However, there is an equally interesting discussion to be had for short papers. My view is this: There is nothing sacred about the length of a software engineering research contribution. Substantial pieces of work may need lengthy and detailed accounts of techniques, or of empirical studies or other kinds of evidence; but equally valuable results may be published in short papers. Indeed, other disciplines manage to publish seminal results in four pages. I will continue to be guided by my associate editors as to what constitutes an appropriate length for any particular contribution. I believe that we, as a community, have largely won the battle of persuading our deans and managers that publishing in conferences is a respectable form of publication. However, in a world of economic difficulties, carbon footprint consciousness, and fast internet-time publication, there is a strong case to be made that we need to redress the balance and publish more of our research in journals. Indeed, I would go as far as saying that many research students now go through their entire doctoral studies publishing in workshops and conferences only, and think that all research contributions can fit into the 2-column, 10-page format of IEEE conference publications. As a result, other than their PhD dissertations, these students do not get an opportunity to publish their work in an archival journal format. Of course, the frequency of conference reviewing has also had an impact on the nature of reviews that referees are asked to write. With little scope for improving conference papers before publication and heavy referee review loads, reviews tend to be short critiques of papers, rather than substantive engagements with the content that authors are trying to present. I therefore believe that journal reviews and revision allow for better quality work to be produced incrementally and then published. Of course, the journal publication process is not without cost to authors, referees, and editors, and we need to acknowledge this and tackle it head on. I was very taken by the discussion by Crowcroft et al. [1] on the varying incentives of the different stakeholders in the publication process: authors, referees, and editors. For TSE, the kind of behavior I would like to incentivize is for authors to submit innovative, rigorous, and well-presented accounts of their research and not overload the system with half-baked ideas or very small increments on previous work. I would like to incentivize referees to agree to review papers and to produce rigorous, thoughtful reviews. And, I would like to incentivize editors to seek and encourage submission of a wider range of contributions that will serve our research and practitioner community better. I have no magic to generate these incentives. However, in a year’s time TSE will make a number of awards to authors, referees, and (associate) editors to acknowledge the quality of research, feedback, and editing. I welcome feedback from the community on the specification of such awards.

There has long been a close association between research in software engineering and the design of programming languages. Part of the IMPACT project involves an exploration of the interrelations of these two fields and documentation in a report of how fundamental research in software engineering has been a valuable resource for programming language features commonly used today. The resulting report investigates the relationship by considering features in currently used languages, including exceptions, control and data abstractions, types, inheritance, concurrency and visualization mechanisms.This paper, exerpted from the report, focuses on the influence of software engineering research on the development of exceptions. The paper demonstrates that there is a symbiotic relationship between software engineering research and the design of exception handing in programming languages. Publication of these partial results is aimed at soliciting feedback and comments from both the programming languages and software engineering communities.

In this paper,I first review the seminal work by Thomas Kuhn — The Structure of Scientific Revolutions — and elaborate my view on paradigm shifts in software engineering research and practice as it turns 50 years old in 2018. I then examine major undertakings of the computing profession since early days of modern computing, especially those done by the software engineering community as a whole. I also enumerate anomalies and crises that occurred at various stages, and the attempts to provide solutions by the software engineering professionals in the past five decades. After providing such a background, I direct readers attention toward emerging anomalies in software engineering, at a severity level that is causing another software engineering crisis, and suggest a set of criteria for feasible solutions. The main theme of this paper is to advocate that situation analytics, equipped with necessary definitions of essential concepts including situation and intention as parts of a new computational framework, can serve as the foundation for a new software engineering paradigm named the Situation-Centric Paradigm. In this framework, situation is considered a new abstraction for computing and is clearly differentiated from the widely accepted existing abstractions, namely function and object. I argue that the software engineering professionals will inevitably move into this new paradigm, willingly or unwillingly, to empower Human-Embedded Computing (HEC) and End-User Embedded Computing (EUEC), much more than what they have done with traditional human-centered or user-centric computing altogether. In the end, I speculate that an ultimate agile method may be on the rise, and challenge readers to contemplate “what if" hundreds of thousands “end-user developers" emerge into the scene where the boundaries between end users and developers become much more blurred.

Since their wide adoption by the software engineering community in the mid-90's, design patterns have become an important educational component in the training of novice software engineers due to the common belief that, as distilled experience of seasoned software designers, they can transform novice software engineers into skilled professionals in a relatively short time frame. This paper questions the validity of this commonly-held belief about the educational value of conventional patterns, arguing that although there is tremendous value in capturing and disseminating software engineering experience in the form of patterns, conventional design patterns cannot deliver on this expectation to significantly shorten the experience growth curve. We report initial results from an ongoing research project that aims to not only identify the difficulties in using conventional patterns as educational means for novices, but also to use such insights to develop new forms of patterns to help educators and experts to more effectively communicate and transfer their software engineering knowledge, experience, and understanding of best practices to novices.

Theory-oriented software engineering

Over the last decade, the software engineering community has been discussing how to better support the planning, execution and analysis of controlled experiments. There is a growing interesting in this topic because it is a mean to meet empirical evidence facilitating the work of researchers. In this paper, we present a process-oriented environment proposed to support the conduction of controlled experiments in software engineering. We describe the desired requirements for such kind of experimental supporting environment and analyze how our experimental environment addresses these requirements.

Software Engineering (SE) communities such as Stack Overflow have become unwelcoming, particularly through members' use of offensive language. Research has shown that offensive language drives users away from active engagement within these platforms. This work aims to explore this issue more broadly by investigating the nature of offensive language in comments posted by users in four prominent SE platforms - GitHub, Gitter, Slack and Stack Overflow (SO). It proposes an approach to detect and classify offensive language in SE communities by adopting natural language processing and deep learning techniques. Further, a Conflict Reduction System (CRS), which identifies offence and then suggests what changes could be made to minimize offence has been proposed. Beyond showing the prevalence of offensive language in over 1 million comments from four different communities which ranges from 0.07% to 0.43%, our results show promise in successful detection and classification of such language. The CRS system has the potential to drastically reduce manual moderation efforts to detect and reduce offence in SE communities.

Software engineers have always been experimenting with various models in an endeavor to improve upon software processes. Recently, a new area of software process improvement through experience management has got attention of the software engineering community. Software developers are experimenting with experience management to improve software process. This chapter explores the role of experience management in mitigating the effects of software crisis and investigates the current state of EM in Indian software engineering environment. The chapter discusses the need and integration of knowledge and experience management practices in software engineering environments. The chapter begins with the discussion of the ills of software engineering, classically referred to as “software crisis,” like late delivery of the systems, ill-functionality, and cost overruns, and justifies how managing the knowledge and experience of software engineers and practitioners can ward off these evils. The chapter then discusses the models and methodologies for knowledge and experience management in software engineering. Finally, the chapter sheds light on the major works carried out in this area and suggests possible future research avenues in this domain.

Many software engineering problems are multi-objective in nature, which has been largely recognized by the Search-based Software Engineering (SBSE) community. In this regard, Pareto- based search algorithms, e.g., Non-dominated Sorting Genetic Algorithm II, have already shown good performance for solving multi-objective optimization problems. These algorithms produce Pareto fronts, where each Pareto front consists of a set of non- dominated solutions. Eventually, a user selects one or more of the solutions from a Pareto front for their specific problems. A key challenge of applying Pareto-based search algorithms is to select appropriate quality indicators, e.g., hypervolume, to assess the quality of Pareto fronts. Based on the results of an extended literature review, we found that the current literature and practice in SBSE lacks a practical guide for selecting quality indicators despite a large number of published SBSE works. In this direction, the paper presents a practical guide for the SBSE community to select quality indicators for assessing Pareto-based search algorithms in different software engineering contexts. The practical guide is derived from the following complementary theoretical and empirical methods: 1) key theoretical foundations of quality indicators; 2) evidence from an extended literature review; and 3) evidence collected from an extensive experiment that was conducted to evaluate eight quality indicators from four different categories with six Pareto-based search algorithms using three real industrial problems from two diverse domains.

In recent years, the use and investigations about Grey Literature (GL) increased, in particular, in Software Engineering (SE) research. However, its understanding is still scarce and sometimes controversial, such as interpreting GL types and assessing their credibility. This study aimed to understand the credibility aspects that SE researchers consider in assessing GL and its types. To achieve this goal, we surveyed 53 SE researchers (who answered that they have used GL in our previous investigation), receiving a total of 34 valid responses. Our main findings show that: 1) GL source produced or cited by a renowned source is the main credibility criteria used to assess GL, 2) most of the GL types tend to have a Low to Moderate level of Control and Expertise, 3) there is a positive statistical correlation between the level of Control and Expertise for most GL types, and 4) the different respondent profiles shared similar opinions about the credibility criteria. Our investigation contributes to helping future SE researchers that intend to use GL with more credibility. Additionally, shows the need for future studies to better understand the GL types in SE research.

[Background:] Experimentation in Software Engineering plays a central role on sharing and verifying scientific findings. As experiments have increased significantly in Software Engineering area, we observe that most of them fail to provide a way to be repeated, replicated or reproduced, thus jeopardizing or delaying the evolution of the Software Engineering area. [Aims:] In this vision paper, we present and discuss techniques and infrastructure to continuously improve experiments towards repeatability, replicability, and reproducibility. [Method:] We define these techniques and infrastructure based on experiences of our research groups and existing literature. Furthermore, we follow Open Science principles. [Results:] We provide incipient results and foresee a central infrastructure composed of two repositories and two recommendation systems to support techniques for: reporting experiments; developing ontologies for experiments and open educational resources; mining and recommending experiments; specifying data management plans, identifying families of experiments; and teaching and learning experimentation. [Conclusions:] Our techniques and infrastructure will prospectively motivate and benefit Software Engineering evolution by improving the conduction and further reproducibility of experiments.

The evolution of Science has been supported by complex computerized infrastructures with growing interest in simulation based experiments. This trend can also be observed in Software Engineering. Our capacity of acquiring evidences to describe phenomena of interest in the field allowed the building of in silico models that can virtually replicate feasible software behaviors and improve our capacity of observation. In silico experiments demand additional concerns for its planning. One of them is regarding the scientific workflow conception. This task is not easy and to apply ad-hoc approaches can risky the experiment execution, validity and future replications. Regarding this topic, this paper presents some initial results from our research towards an approach to support the conception of scientific workflows for in silico experiments in Software Engineering are presented.

(2004). Common Errors in Statistics (and How to Avoid Them) The American Statistician: Vol. 58, No. 4, pp. 359-359.