Bayesian Data Analysis in Empirical Software Engineering Research

Empirical research is playing a significant role in software engineering (SE), and it has been applied to evaluate software artifacts and technologies. There have been a great number of empirical research articles published recently. There is also a large research community in empirical software engineering (ESE). In this paper, we identify both the overall landscape and detailed implementations of ESE, and investigate frequently applied empirical methods, targeted research purposes, used data sources, and applied data processing approaches and tools in ESE. The aim is to identify new trends and obtain interesting observations of empirical software engineering across different sub-fields of software engineering. We conduct a mapping study on 538 selected articles from January 2013 to November 2017, with four research questions. We observe that the trend of applying empirical methods in software engineering is continuously increasing and the most commonly applied methods are experiment, case study and survey. Moreover, open source projects are the most frequently used data sources. We also observe that most of researchers have paid attention to the validity and the possibility to replicate their studies. These observations are carefully analyzed and presented as carefully designed diagrams. We also reveal shortcomings and demanded knowledge/strategies in ESE and propose recommendations for researchers.

Bayesian data analysis (BDA) is today used by a multitude of research disciplines. These disciplines use BDA as a way to embrace uncertainty by using multilevel models and making use of all available information at hand. In this chapter, we first introduce the reader to BDA and then provide an example from empirical software engineering, where we also deal with a common issue in our field, i.e., missing data. The example we make use of presents the steps done when conducting state-of-the-art statistical analysis. First, we need to understand the problem we want to solve. Second, we conduct causal analysis. Third, we analyze non-identifiability. Fourth, we conduct missing data analysis. Finally, we do a sensitivity analysis of priors. All this before we design our statistical model. Once we have a model, we present several diagnostics one can use to conduct sanity checks. We hope that through these examples, the reader will see the advantages of using BDA. This way, we hope Bayesian statistics will become more prevalent in our field, thus partly avoiding the reproducibility crisis we have seen in other disciplines.

The International Symposium on Empirical Software Engineering and Measurement (ESEM) is one of the most important forums at which to present and discuss empirical research on software engineering and measurement. Spain was the first country to host the ESEM in 2007. What, then, could be more natural than to return to Spain, this time in Ciudad Real, for the 10th ESEM conference? The papers cover a variety of research topics and research methods in empirical software engineering and measurement, including agile development, verification and validation, the innovative behavior of software engineers, repository mining for defect predictions, productivity measurement and improvement, bug-fixing, system safety and security, shared code, eye tracking, continuous delivery, code analysis, improved methods for systematic reviews and the sampling of subjects, evidence briefing to transfer knowledge to practitioners, blind analysis of data to avoid research bias and software maintenance. In addition to the research papers, we are delighted to include two very interesting keynote speeches. Prof. Claes Wohlin, from the Blekinge Institute of Technology (Karlskrona, Sweden), who presents a keynote about the need for more and better empirical software engineering research. He addresses the question of how to support an increased use of empirical evidence so as to guide important decisions in the software industry. Prof. Joe Peppard, from the European School of Management and Technology (Berlin, Germany), who tackles the importance of benefit management in software engineering, and how to ensure the usefulness and benefits of the software we build.

Several factors make empirical research in software engineering particularly challenging as it requires studying not only technology but its stakeholders' activities while drawing concepts and theories from social science. Researchers, in general, agree that selecting a research design in empirical software engineering research is challenging, because the implications of using individual research methods are not well recorded. The main objective of this article is to make researchers aware and support them in their research design, by providing a foundation of knowledge about empirical software engineering research decisions, in order to ensure that researchers make well-founded and informed decisions about their research designs. This article provides a decision-making structure containing a number of decision points, each one of them representing a specific aspect on empirical software engineering research. The article provides an introduction to each decision point and its constituents, as well as to the relationships between the different parts in the decision-making structure. The intention is the structure should act as a starting point for the research design before going into the details of the research design chosen. The article provides an in-depth discussion of decision points in relation to the research design when conducting empirical research.

CERSE - Catalog for empirical research in software engineering: A Systematic mapping study

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

Complex software development projects rely on the contribution of teams of developers, who are required to collaborate and coordinate their efforts. The productivity of such development teams, i.e., how their size is related to the produced output, is an important consideration for project and schedule management as well as for cost estimation. The majority of studies in empirical software engineering suggest that - due to coordination overhead - teams of collaborating developers become less productive as they grow in size. This phenomenon is commonly paraphrased as Brooks' law of software project management, which states that "adding manpower to a software project makes it later". Outside software engineering, the non-additive scaling of productivity in teams is often referred to as the Ringelmann effect, which is studied extensively in social psychology and organizational theory. Conversely, a recent study suggested that in Open Source Software (OSS) projects, the productivity of developers increases as the team grows in size. Attributing it to collective synergetic effects, this surprising finding was linked to the Aristotelian quote that "the whole is more than the sum of its parts". Using a data set of 58 OSS projects with more than 580,000 commits contributed by more than 30,000 developers, in this article we provide a large-scale analysis of the relation between size and productivity of software development teams. Our findings confirm the negative relation between team size and productivity previously suggested by empirical software engineering research, thus providing quantitative evidence for the presence of a strong Ringelmann effect. Using fine-grained data on the association between developers and source code files, we investigate possible explanations for the observed relations between team size and productivity. In particular, we take a network perspective on developer-code associations in software development teams and show that the magnitude of the decrease in productivity is likely to be related to the growth dynamics of co-editing networks which can be interpreted as a first-order approximation of coordination requirements.

The issue of validity threats in empirical software engineering research is important. However, some authors overlook this, focusing on validating their work through application of fundamental testing techniques, instead. However, testing is different to empirical validation, with the latter being more concerned about how experimental conclusions are justified. An important factor that can render an experimental conclusion incorrect is researcher’s bias, which can be especially relevant when setting the experimental parameters. Therefore, consideration of validity threats is essential to enable confidence in research results and assure the research quality. This paper provides a practical approach for mitigating threats to validity in empirical software engineering using a sequence of software activities. The paper is based on a real-world traceability case study for illustration purposes.

One of the first things one learns in a basic psychology or statistics course is that you cannot prove the null hypothesis that there is no difference between two conditions such as a patient group and a normal control group. This remains true. However now, thanks to ongoing progress by a special group of devoted methodologists, even when the result of an inferential test is p > .05, it is now possible to rigorously and quantitatively conclude that (a) the null hypothesis is actually unlikely, and (b) that the alternative hypothesis of an actual difference between treatment and control is more probable than the null. Alternatively, it is also possible to conclude quantitatively that the null hypothesis is much more likely than the alternative. Without Bayesian statistics, we couldn't say anything if a simple inferential analysis like a t-test yielded p > .05. The present, mostly non-quantitative article describes free resources and illustrative procedures for doing Bayesian analysis, with t-test and ANOVA examples.

This paper reconciles the asymptotic disagreement between Bayesian and frequentist inference in set-identified models by adopting a multiple-prior (robust) Bayesian approach. We propose new tools for Bayesian inference in set-identified models. We show that these tools have a well-defined posterior interpretation in finite samples and are asymptotically valid from the frequentist perspective. The main idea is to construct a prior class that removes the source of the disagreement: the need to specify an unrevisable prior. The corresponding class of posteriors can be summarized by reporting the "posterior lower and upper probabilities" of a given event and/or the "set of posterior means" and the associated "robust credible region". We show that the set of posterior means is a consistent estimator of the true identified set and the robust credible region has the correct frequentist asymptotic coverage for the true identified set if it is convex. Otherwise, the method can be interpreted as providing posterior inference about the convex hull of the identified set. For impulse-response analysis in set-identified Structural Vector Autoregressions, the new tools can be used to overcome or quantify the sensitivity of standard Bayesian inference to the choice of an unrevisable prior.

This paper reconciles the asymptotic disagreement between Bayesian and frequentist inference in set-identified models by adopting a multiple-prior (robust) Bayesian approach. We propose new tools for Bayesian inference in set-identified models and show that they have a well-defined posterior interpretation infinite samples and are asymptotically valid from the frequentist perspective. The main idea is to construct a prior class that removes the source of the disagreement: the need to specify an unrevisable prior for the structural parameter given the reduced-form parameter. The corresponding class of posteriors can be summarized by reporting the 'posterior lower and upper probabilities' of a given event and/or the 'set of posterior means' and the associated 'robust credible region'. We show that the set of posterior means is a consistent estimator of the true identi?ed set and the robust credible region has the correct frequentist asymptotic coverage for the true identified set if it is convex. Otherwise, the method provides posterior inference about the convex hull of the identified set. For impulse-response analysis in set-identified Structural Vector Autoregressions, the new tools can be used to overcome or quantify the sensitivity of standard Bayesian inference to the choice of an unrevisable prior.

Development and comparison in uncertainty assessment based Bayesian modularization method in hydrological modeling

Event Abstract Back to Event 2nd level modelling in fMRI analysis with a clinically depressed sample - Comparisons between classical and Bayesian methods Peter Goodin1, 2*, Joseph Ciorciari1, Susan Rossell1, 2, Matt Hughes1 and Richard Nibbs1 1 Swinburne University, BPsyC, Australia 2 Monash Alfred Psychiatric Research Centre, Australia The use of the Frequentist statistics in fMRI analysis has been the mainstay of the field for the last 25 years. Several other methods have been developed as alternatives, including those based on Bayesian statistics. Bayesian analysis methods have the benefit of allowing interrelation of data through effect size and confidence thresholds, can be more robust to outlier data and may negate the need for stringent multiple comparison correction. When examining functional differences in those with a mental illness , this can lead to detection of more subtle activation that may be of importance. Depression is a disorder that despite a common theme of symptoms is also characterised by functional heterogeneities between individuals, making inferences using Frequentist methods sometimes difficult. Utilising both Frequentist and Bayesian 2nd level analysis methods offered by SPM8 , we examined differences in activation between a healthy control sample and those with major depressive disorder during a novel emotional processing n-back task. Activation using classical methods showed small differences in more posterior visual regions however this was only detectable with liberal thresholding (p = .001 uncorrected, k = 0). In contrast Bayesian analysis (y = 0, 95% confidence) uncovered widespread differences (including overlap with those areas found with the Frequentist method) in multiple regions implicated with the disorder. This suggests that using Bayesian methods in fMRI analysis with a clinical population could be extremely useful in detecting responses that may otherwise go unnoticed or under-represented in more traditional methods, however determination of effect size importance is paramount. Keywords: Depression, fMRI, working memory, Bayesian, Frequentist Conference: XII International Conference on Cognitive Neuroscience (ICON-XII), Brisbane, Queensland, Australia, 27 Jul - 31 Jul, 2014. Presentation Type: Poster Topic: Methods Development Citation: Goodin P, Ciorciari J, Rossell S, Hughes M and Nibbs R (2015). 2nd level modelling in fMRI analysis with a clinically depressed sample - Comparisons between classical and Bayesian methods. Conference Abstract: XII International Conference on Cognitive Neuroscience (ICON-XII). doi: 10.3389/conf.fnhum.2015.217.00203 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 19 Feb 2015; Published Online: 24 Apr 2015. * Correspondence: Mr. Peter Goodin, Swinburne University, BPsyC, Hawthorn, Australia, peter@hitiq.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Peter Goodin Joseph Ciorciari Susan Rossell Matt Hughes Richard Nibbs Google Peter Goodin Joseph Ciorciari Susan Rossell Matt Hughes Richard Nibbs Google Scholar Peter Goodin Joseph Ciorciari Susan Rossell Matt Hughes Richard Nibbs PubMed Peter Goodin Joseph Ciorciari Susan Rossell Matt Hughes Richard Nibbs Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

BackgroundTau aggregates in Alzheimer’s disease (AD) induce loss of synapses and neurons, leading to cognitive impairment. Predicting tau and neurodegeneration temporal evolution could be used for prognostication and for assessing results of therapeutic trials. Tau PET and MRI volumetry are reliable markers of disease stage, but cost and radiation protection considerations limit research measurement frequency, lowering the accuracy of disease progression modeling. Here, we evaluate, using Bayesian analysis, whether models based on limited numbers of observations can be refined to better predict the temporal trajectory of pathology.MethodImaging data comes from subjects (113; 68 females; 18 AD dementia, 23 MCI and 72 cognitively normal) of the TRIAD cohort (McGill University) who have been evaluated at least twice ( 1 year interval) with both tau PET ([18F]MK‐6240) and structural MRI. Four probability models were evaluated: 1‐ a basic one, assuming that all data points come from 1 data distribution; 2‐ one where subjects' observations are clustered within anatomical ROIs, where an independent distribution is hypothesized; 3‐ data is clustered within known physiological networks, each networks’ distribution parameters having their own specific values ; 4‐ a model assuming that subjects’ observations are described by a distribution of voxel parameters dictated by both the ROI and network(s) in which they lay. Bayesian data analysis was used to compare the predictive accuracy of those models for progression at 1 year from baseline of tau PET and MRI data.ResultModel 4 was the most accurate model for both tau and cortical thickness prediction. We therefore used it to perform posterior predictions across hemispheres, showing that the prediction curves of the left and right hemispheres for the pericalcarine cortex differ. We also noticed a decreasing trend in the CN tau curve for the left hemisphere as the rate of cortical thinning increases. In contrast, there is an increasing trend in the AD tau curve as the rate of cortical thinning increases.ConclusionThe model that incorporated both ROI‐level and network‐level information was the best predictor of progression, and such an approach can reveal underappreciated properties of the disease (i.e., laterality).

We tested whether it is beneficial for the accuracy of phylogenetic inference to sample characters that are evolving under different sets of parameters, using both Bayesian MCMC (Markov chain Monte Carlo) and parsimony approaches. We examined differential rates of evolution among characters, differential character-state frequencies and character-state space, and differential relative branch lengths among characters. We also compared the relative performance of parsimony and Bayesian analyses by progressively incorporating more of these heterogeneous parameters and progressively increasing the severity of this heterogeneity. Bayesian analyses performed better than parsimony when heterogeneous simulation parameters were incorporated into the substitution model. However, parsimony outperformed Bayesian MCMC when heterogeneous simulation parameters were not incorporated into the Bayesian substitution model. The higher the rate of evolution simulated, the better parsimony performed relative to Bayesian analyses. Bayesian and parsimony analyses converged in their performance as the number of simulated heterogeneous model parameters increased. Up to a point, rate heterogeneity among sites was generally advantageous for phylogenetic inference using both approaches. In contrast, branch-length heterogeneity was generally disadvantageous for phylogenetic inference using both parsimony and Bayesian approaches. Parsimony was found to be more conservative than Bayesian analyses, in that it resolved fewer incorrect clades.