Paradigms for experimentation and empirical studies in software engineering

The popularity of empirical methods in software engineering research is on the rise. Surveys, experiments, metrics, case studies, and field studies are examples of empirical methods used to investigate both software engineering processes and products. The increased application of empirical methods has also brought about an increase in discussions about adapting these methods to the peculiarities of software engineering. In contrast, the ethical issues raised by empirical methods have received little, if any, attention in the software engineering literature. This article is intended to introduce the ethical issues raised by empirical research to the software engineering research community and to stimulate discussion of how best to deal with these ethical issues. Through a review of the ethical codes of several fields that commonly employ humans and artifacts as research subjects, we have identified major ethical issues relevant to empirical studies of software engineering. These issues are illustrated with real empirical studies of software engineering.

Since at least 15 years, software testing has been one of the areas of software engineering that is increasing its importance in many organizations around the world. Due to this, guides, standards, and maturity models, such as the Brazilian Test Process Improvement model (MPT.Br), are gaining visibility to ensure that a company or organization, even a small one, has a quality level in their testing processes. Such models are important to ensure customer satisfaction and quality analysis of business processes in the production of a system. Given this context, this research aims to present the implementation of the MPT.br (in the first two maturity levels) in a telecommunications company in Brazil to present the benefits of defining the software test process according to a maturity model. To achieve the research goal, we are using a research method called action-research, which is based on empirical studies in software engineering. As results, we expect that this deployment in the company will contribute to a significant improvement in its testing processes, thus contributing to the improvement of the entire organization.

Empirical studies are one of the main inputs of Evidence-based Software Engineering to generate a body of knowledge. For this reason, it is important to know the reliability of the results applying empirical methods. This article presents a method that evaluates the reliability of empirical studies related to software engineering. For this, the authors considered aspects deal with the mitigation of threats to validity from scientific literature and generate an evaluation mechanism. The method is validated by applying it to a scientific article. Although the results seem to give a degree of reliability, it is necessary replicated in other studies to perform a calibration of the method.

Context: Software projects are common inputs in Empirical Software Engineering (ESE) studies, although they are often selected with ad-hoc strategies that reduce the generalizability of the results. An alternative is the usage of available datasets of software projects, which should be current and follow explicit rules for ensuring their validity over time. Goal: In this context, it is important to assess the general state of software datasets in terms of purpose, last update, project characterization, source code metrics, and tools to extract source-code-related artifacts. Method: We conducted a systematic mapping study retrieving software datasets used in ESE studies published from January 2013 to December 2021. Results: We selected 74 datasets created mainly for software defects, software estimation, and software maintainability studies. The majority of these datasets (64%) explicitly stated the characteristics to select the projects, and the most common programming languages were Java and C. Conclusions: Our study identified scarce efforts to keep datasets updated over time and also provides recommendations to support their construction and consumption for ESE studies.

Software systems form the foundation of modern information society and many of those systems are among the most complex things ever created by man. The quotation that follows is from the 1999 President’s Information Technology Advisory Committee Report, but is as valid today as it was then:

Vegas et al. IEEE Trans Softw Eng 42(2):120:135 (2016) raised concerns about the use of AB/BA crossover designs in empirical software engineering studies. This paper addresses issues related to calculating standardized effect sizes and their variances that were not addressed by the Vegas et al.’s paper. In a repeated measures design such as an AB/BA crossover design each participant uses each method. There are two major implication of this that have not been discussed in the software engineering literature. Firstly, there are potentially two different standardized mean difference effect sizes that can be calculated, depending on whether the mean difference is standardized by the pooled within groups variance or the within-participants variance. Secondly, as for any estimated parameters and also for the purposes of undertaking meta-analysis, it is necessary to calculate the variance of the standardized mean difference effect sizes (which is not the same as the variance of the study). We present the model underlying the AB/BA crossover design and provide two examples to demonstrate how to construct the two standardized mean difference effect sizes and their variances, both from standard descriptive statistics and from the outputs of statistical software. Finally, we discuss the implication of these issues for reporting and planning software engineering experiments. In particular we consider how researchers should choose between a crossover design or a between groups design.

Although most scientific and engineering disciplines view empiricism as a basic aspect of their discipline, that view has not been the tradition in software engineering. There should be a symbiotic relationship between the development of theories and empirical studies that test and evolve that theory. This talk discusses of the role of empirical study in the understanding and improvement of the software product and process. It offers a personal, historical perspective of the use of empiricism through a series of example applications that demonstrate the various roles that empiricism can play. The examples are taken from the author?s own experience and include the use of empirical study to improve an organization's product quality and productivity in the Software Engineering Laboratory at NASA/Goddard and to evaluate and evolve the effectiveness of various software technologies for use in improving the dependability of software. It also suggests research directions for empirical study in software engineering.

Context: Capture-recapture method has long been adopted in software engineering as a relatively objective way for defect estimation. While many relevant studies have been carried out to evaluate various capture-recapture models and estimators, there still lacks common understanding on the adoption status of the method in software engineering. It is necessary to systematically collect empirical evidence of Capture-recapture adoption hence form necessary understanding on the method.Objective: This study aims to synthesize relevant primary studies on the adoption of capture-recapture method in software engineering, and try to identify possible gaps between the state-of-practice and the state-of-art so as to provide clues for future research.Method: By following the guidelines of Kitchenham, we conducted a Systematic Literature Review(SLR) on studies of the adoption of capture-recapture method in software engineering.Results: From 5 common digital libraries, we retrieved 506 published articles, among them 44 were identified as relevant primary studies. We identified 18 capture-recapture estimators under 4 basic models. Types of the currently existing studies as well as the relevant influencing factors to adoption of the capture-recapture method are also discussed.Conclusion: Results show that there are no conclusive decisions on the best capture-recapture models and estimators. Besides, the number of inspectors and their capability to detect defects as well as the difficulty to detect defects are most critical influencing factors. In addition, lacking of industrial application may be the major issue of current adoption status of capture-recapture method in software engineering.

Software engineering is a design discipline. As such, its engineering methods are based on cognitive instead of physical laws, and their effectiveness depends highly on context. Empirical methods can be used to observe the effects of software engineering methods in vivo and in vitro, to identify improvement potentials, and to validate new research results. This paper summarizes both the current body of knowledge and further challenges wrt. empirical methods in software engineering as well as empirically derived evidence regarding software typical engineering methods. Finally, future challenges wrt. education, research, and technology transfer will be outlined.

Without careful methodological guidance, case studies in software engineering are difficult to plan, design and execute. While there are a number of broad guidelines for case study research, there are none that specifically address the needs of a software engineer undertaking multiple case studies in an industrial setting. Through a synthesis of existing best practices in case study research, we provide a set of comprehensive guidelines for conducting multiple case studies in software engineering research. Our guidelines can assist software engineering researchers with all stages of multiple case study research, although in this paper we concentrate on the early phases, such as focusing the case study and detailed plan design. To date, three exploratory research projects found our guidelines very useful. We illustrate our guidelines with examples from one of these projects.

The evolution of Science has been supported by complex computerized infrastructures with growing interest in simulation based experiments. This trend can also be observed in Software Engineering. Our capacity of acquiring evidences to describe phenomena of interest in the field allowed the building of in silico models that can virtually replicate feasible software behaviors and improve our capacity of observation. In silico experiments demand additional concerns for its planning. One of them is regarding the scientific workflow conception. This task is not easy and to apply ad-hoc approaches can risky the experiment execution, validity and future replications. Regarding this topic, this paper presents some initial results from our research towards an approach to support the conception of scientific workflows for in silico experiments in Software Engineering are presented.

The classical scientific method has been settled through the last centuries as a cyclic, iterative process of observation, hypothesis formulation, and confirmation/refutation of hypothesis through experimentation. This “experimental scientific method” was mainly developed in the context of natural sciences dealing with the physical world, such as Mechanics, Thermodynamics, Electromagnetism, Chemistry, and so on. But when trying to apply this classical view of the scientific method to the various branches of Computer Science and Computer Engineering, among which Software Engineering, there are two kinds of obstacles. First, Computer Science is rooted both in formal sciences such as Mathematics and experimental sciences such as Physics, and therefore, an excessive emphasis on the experimental side is not appropriate to give a full account of this kind of scientific activity. Second, the production of software systems has to deal not only with the behavior of complex physical systems such as computers, but also with the behavior of complex human systems (developers interacting with stakeholders, for instance, or users interacting with machines) where educational, cultural, sociological, and economical factors are essential. Therefore, empirical methods in their narrow sense, even though valuable in some respects, are rather limited to understand a reality that exceeds the mere physical world. Moreover, neither formal nor empirical methods can provide a full account of scientific activity, which relies on something that is beyond any established method. Qualitative (i.e. meta-methodical) reasoning plays the directive role in scientific activity. In this chapter, the authors claim that acknowledging a plurality of research methods in software engineering will benefit the advancement of this branch of science.

[Background:] Experimentation in Software Engineering plays a central role on sharing and verifying scientific findings. As experiments have increased significantly in Software Engineering area, we observe that most of them fail to provide a way to be repeated, replicated or reproduced, thus jeopardizing or delaying the evolution of the Software Engineering area. [Aims:] In this vision paper, we present and discuss techniques and infrastructure to continuously improve experiments towards repeatability, replicability, and reproducibility. [Method:] We define these techniques and infrastructure based on experiences of our research groups and existing literature. Furthermore, we follow Open Science principles. [Results:] We provide incipient results and foresee a central infrastructure composed of two repositories and two recommendation systems to support techniques for: reporting experiments; developing ontologies for experiments and open educational resources; mining and recommending experiments; specifying data management plans, identifying families of experiments; and teaching and learning experimentation. [Conclusions:] Our techniques and infrastructure will prospectively motivate and benefit Software Engineering evolution by improving the conduction and further reproducibility of experiments.

As a basis subject of computer, Computer software engineering has been increasingly used in various fields, Games management system has been pre-designed on the basis of application of software engineering theory and method. The actual games management system software has been developed from using software engineering methods, in the actual design and development, it has been proved that a practical system has been perfect created with efficiency design, fully functional form, rational and affordable design.

Empirical research is playing a significant role in software engineering (SE), and it has been applied to evaluate software artifacts and technologies. There have been a great number of empirical research articles published recently. There is also a large research community in empirical software engineering (ESE). In this paper, we identify both the overall landscape and detailed implementations of ESE, and investigate frequently applied empirical methods, targeted research purposes, used data sources, and applied data processing approaches and tools in ESE. The aim is to identify new trends and obtain interesting observations of empirical software engineering across different sub-fields of software engineering. We conduct a mapping study on 538 selected articles from January 2013 to November 2017, with four research questions. We observe that the trend of applying empirical methods in software engineering is continuously increasing and the most commonly applied methods are experiment, case study and survey. Moreover, open source projects are the most frequently used data sources. We also observe that most of researchers have paid attention to the validity and the possibility to replicate their studies. These observations are carefully analyzed and presented as carefully designed diagrams. We also reveal shortcomings and demanded knowledge/strategies in ESE and propose recommendations for researchers.