DiffGen: Automated Regression Unit-Test Generation

Different techniques to automatically generate unit tests for object oriented classes have been proposed, but how to integrate these tools into the daily activities of software development is a little investigated question. In this paper, we report on our experience in supporting industrial partners in introducing the EvoSuite automated JUnit test generation tool in their software development processes. The first step consisted of providing a plugin to the Apache Maven build infrastructure. The move from a research-oriented point-and-click tool to an automated step of the build process has implications on how developers interact with the tool and generated tests, and therefore, we produced a plugin for the popular IntelliJ Integrated Development Environment (IDE). As build automation is a core component of Continuous Integration (CI), we provide a further plugin to the Jenkins CI system, which allows developers to monitor the results of EvoSuite and integrate generated tests in their source tree. In this paper, we discuss the resulting architecture of the plugins, and the challenges arising when building such plugins. Although the plugins described are targeted for the EvoSuite tool, they can be adapted and their architecture can be reused for other test generation tools as well.

MARVEL is a kernel for software development environments that enacts a processing modeling notation, called the MARVEL Strategy Language (MSL). The software process is defined in MSL as a collection of modular strategies, where different (sets of these) strategies are appropriate for different tool suites, user roles, phases of the lifecycle, management policies, etc. Each strategy specifies a partial object-oriented data model and a partial rule-based process model. The strategies are supplemented by a collection of envelopes, each of which describes how to apply one or more of the external tools used during software development activities; this includes the mapping from the inputs/outputs of these tools and the MARVEL objectbase and the command lines to invoke these tools. Multiple MSL strategies are merged to obtain the full data and process models for enaction by the MARVEL kernel.Each software development activity is described in MSL by a rule specifying the precondition and multiple postconditions of the activity. Both preconditions and postconditions are written in a subset of the first order predicate logic, stylized to incorporate directives for efficient queries and updates. A precondition is a logical statement that must be true before the activity is initiated; a postcondition is a logical statement asserted to be true when the activity completes. Multiple mutually exclusive postconditions reflect the multiple possible results of many activities. An activity is typically a particular combination of software development tool, options specific to that tool, and typed arguments. The notion of activity could be extended to include loading a new set of strategies to take over at this point in the process, to support both nesting of hierarchical activities and successive activity phases.Enaction of an MSL process model is implemented by what we call opportunistic processing, consisting primarily of forward and backward chaining among the rules. When a user initiates an activity, MARVEL applies backward chaining to attempt to satisfy its precondition. If this is not possible, MARVEL could use the rules in the style of an expert system to explain the problem. Once an activity has completed, MARVEL asserts one of its postconditions and uses forward chaining to automatically carry out other activities whose results are presumed to be needed in the near future.We have designed two facilities for controlling this processing, to prevent runaway automation. Hints are additional MSL preconditions that affect automatic but not user-initiated invocation of tools. Implicit queries represent a reflective capacity to keep tabs on estimated costs of automatic behavior and request user confirmation when certain user-selected thresholds are reached.MARVEL version 2.10 was recently completed, an upgrade over the version 2.01 that was demonstrated at the 3rd PSDE conference in November 1988. This latest version of MARVEL adds an X11 windows graphical user interface and new subsystems called Marvelizer and Organ. Marvelizer is a tool for immigrating existing software systems from the Unix file system or (a restricted class of) existing software development environment into a MARVEL objectbase. Organ is a set of commands for reorganizing MARVEL objectbases as the project and/or system structure evolves. Marvelizer has been used to immigrate Marvel itself, including about 35,000 lines of C code, into a C/Marvel objectbase. We would like to use the C/Marvel set of MSL strategies, which support activities using the user's favorite editor, lint, cc, ar, ranlib, less (and soon rcs, dbx and mail), for our own further development of MARVEL. Both MARVEL and C/Marvel were in use during the Spring 1989 semester by students in my E6123y Programming Environments and Software Tools course (admittedly a captive audience).We have discovered several difficulties with the MARVEL paradigm that seem likely to apply at least partially to enaction of other notations for software process modeling. First, constructing and debugging a process model is hard. We have run into many unanticipated interactions among rules, particularly among those defined in different strategies that happen to be loaded into the MARVEL kernel at the same time. We are working on debugging facilities that will uncover all direct and transitive dependencies by constructing a graphical dataflow graph of the currently loaded rules. It is feasible for a user to enable and disable individual rules, both during debugging to see how that affects the dataflow among rules and also during operation because certain behavior is not currently desired. This would be dangerous, however, if done by a naive user: if rules were turned off and on at arbitrary points, it would be possible for MARVEL to get into what is an inconsistent state with respect to the full set of rules in such a way that automatic recovery is impossible.Similar difficulties seem likely to arise for any enactable process modeling paradigm. The following questions need to be answered: How does an implementor go about writing, testing and debugging a process model? For non-trivial processes, the model is likely to be very large and hence should be modularized. What is the appropriate style of interface among process model modules and how is it enforced? Is the process model visible to the users and, if so, how is it presented? If the users can modify the process model during enaction, how does this affect continuing the process? Should (can) consistency be enforced or should (can) inconsistencies be detected and repaired?Second, constructing envelopes is even harder. We currently use the Shell languages (e.g., CShell) as notations for defining envelopes. Writing the Shell scripts requires an intimate knowledge of both the tool and the mapping of the MARVEL objectbase onto the file system. Although there is no way to avoid the former, we hope to minimize the latter by developing a higher level envelope language that understands the MARVEL objectbase implementation so the envelope writer doesn't have to. However, this is currently a relatively low priority due to lack of manpower.Again, similar difficulties seem inevitable for other enaction mechanisms intended to work with external tools. The questions: Is it possible (or desirable) to support existing tools, particularly tools where retrofitting is impractical? If only new tools are supported, can a relatively simple standard interface be developed and published so these tools can be developed separately by vendors? Would such an interface pre-empt too many design decisions, or would it be feasible for vendors to upgrade their current products?Third, supporting multiple users is hardest of all. The current MARVEL kernel does not provide any facilities for concurrency control. It would be easy to add locking of the entire objectbase, but a more flexible transaction model is needed. We are working on a number of alternative models that we think are promising for incorporation into the MARVEL kernel. Commit-serializability introduces two new transaction operations, split-transaction and join-transaction, that manipulate in-progress transactions to support coordination among multiple users. Participant transactions (AKA participation domains) distinguishes participants in group software development tasks from observers, and relaxes serializability as viewed by participants to permit cooperation.It is essential for enactable process models to support coordination and cooperation among the members of software development teams. There are two sets of questions: First, what are appropriate processes for programming-in-the-many? This is itself an active area of research, and there is certainly no consensus. One opinion, voiced at the previous workshop, is that the community should not be enacting process models until we understand the processes we are modeling. An opposing opinion is that enaction gives the community the capability to experiment with different processes without prematurely making a full scale commitment to a particular process. Assuming agreement with the second view, what process modeling notations and enaction mechanisms can support a reasonable range of cooperative work facilities?The database and operating system communities have agreed on atomic and serializable transactions, which can be implemented using a wide variety of technologies, as the basis for data consistency, failure recovery and concurrency control across their traditional application areas. Can the software engineering community develop an equally all-encompassing “transaction” model? Or is there some characteristic of the software development process such that coordination and cooperation policies will always require special support on a case-by-case basis? If so, what does this say about enaction of process models in general?In summary, MARVEL is implemented and in limited use, it works in some cases, it doesn't work in others, it raises a lot of questions, and both we the MARVEL group and we the software engineering community have a lot more work to do.

Author index to volume 28

Leveraging the abundant community resources, e.g. campus students, programming fans, software amateurs, is promising to be a flexible and effective approach for relieving the enterprise's IT resource shortage. However, the concern about confidential knowledge exposure makes most enterprises hesitate to outsource software development to community resources. Unlike the traditional legal protection mechanism that prevents community resources from exposing the already known confidential knowledge, this paper presents a proactive confidential knowledge protection technology, which prevents the community resources from learning the confidential knowledge in software development activities. First, the client is facilitated on identifying confidential knowledge involved in the software development activity. Then, the risky software design elements that might expose confidential knowledge are identified in both qualitative and quantitative way. Based on the identification result, a holistic proactive protection solution is presented by orchestrating task reservation, task pretending and task partition activities. Finally, the processed software development tasks are dispatched to community developers without worrying about the confidential knowledge exposure. The technology has been preliminarily validated in two community-based software development cases.

Bots are used to support different software development activities, from automating repetitive tasks to bridging knowledge and communication gaps in software teams. We anticipate the use of Bots will increase and lead to improvements in software quality and developer and team productivity, but what if the disruptive effect is not what we expect?

Software project risk analysis using Bayesian networks with causality constraints

As software permeates all areas of society, it becomes essential to deliver quality products that meet the demands of stakeholders and thus, the software industry is always challenged to improve its development activities. Artificial intelligence has been successfully used in software development activities; identifying its potential to improve productivity in software development teams in requirements engineering defines the object of study. The research used systematic mapping to identify the contribution of artificial intelligence in requirements engineering practices between 2010 and 2021. The execution of the methodology allowed answering the research questions from twenty-one studies that were selected based on the definition of inclusion and exclusion criteria. The results revealed that the relationship between Artificial Intelligence and Software Engineering is complementary to allow its application in the automation of development activities. This trend regarding the use of artificial intelligence techniques suggests ways to improve delivery times and quality of software products.

User Interface (UI) testing is a popular approach to ensure the quality of mobile apps. Numerous test generation tools have been developed to support UI testing on mobile apps, especially for Android apps. Previous work evaluates and compares different test generation tools using only relatively simple open-source apps, while real-world industrial apps tend to have more complex functionalities and implementations. There is no direct comparison among test generation tools with regard to effectiveness and ease-of-use on these industrial apps. To address such limitation, we study existing state-of-the-art or state-of-the-practice test generation tools on 68 widely-used industrial apps. We directly compare the tools with regard to code coverage and fault-detection ability. According to our results, Monkey, a state-of-the-practice tool from Google, achieves the highest method coverage on 22 of 41 apps whose method coverage data can be obtained. Of all 68 apps under study, Monkey also achieves the highest activity coverage on 35 apps, while Stoat, a state-of-the-art tool, is able to trigger the highest number of unique crashes on 23 apps. By analyzing the experimental results, we provide suggestions for combining different test generation tools to achieve better performance. We also report our experience in applying these tools to industrial apps under study. Our study results give insights on how Android UI test generation tools could be improved to better handle complex industrial apps.

In large projects, software developers struggle with two sources of complexity - the complexity of the code itself, and the complexity of the process of producing it. Both of these concerns have been subjected to considerable research investigation, and tools and techniques have been developed to help manage them. However, these solutions have generally been developed independently, making it difficult to deal with problems that inherently span both dimensions. We describe Augur, a visualization tool that supports distributed software development processes. Augur creates visual representations of both software artifacts and software development activities, and, crucially, allows developers to explore the relationship between them. Augur is designed not for managers, but for the developers participating in the software development process. We discuss some of the early results of informal evaluation with open source software developers. Our experiences to date suggest that combining views of artifacts and activities is both meaningful and valuable to software developers.

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A developer would first need to know when and how to apply such a tool, and would then need to understand the resulting tests in order to provide test oracles and to diagnose and fix any faults that the tests reveal. Considering all this, does automatically generating unit tests provide any benefit over simply writing unit tests manually? We empirically investigated the effects of using an automated unit test generation tool (EvoSuite) during development. A controlled experiment with 41 students shows that using EvoSuite leads to an average branch coverage increase of +13%, and 36% less time is spent on testing compared to writing unit tests manually. However, there is no clear effect on the quality of the implementations, as it depends on how the test generation tool and the generated tests are used. In-depth analysis, using five think-aloud observations with professional programmers, confirms the necessity to increase the usability of automated unit test generation tools, to integrate them better during software development, and to educate software developers on how to best use those tools.

Measurement is needed for project control, quality assurance, and process improvement activities in software development. Measurement planning is a difficult and time-consuming activity. Therefore, a great interest exists in reusing measurement plans. However, most measurement plans are structured intuitively and are written informally. Therefore, they are difficult to understand and to reuse. The contents of measurement plans strongly depend on the object being measured and the quality focus of measurement. Process models (describing the object) and quality models (describing the quality focus) change from project to project and over time. Therefore, in order to reuse measurement plans from one project to the next, relationships between measurement plans and process and quality models are needed. These relationships can be used to identify and adapt existing measurement plans to varying process and quality models. This paper presents a formal conceptual model of measurement plans including relationships to process and quality models and a development process for such measurement plans. This model and process build a sound basis for reuse of measurement plans. To further support reuse, a conceptual model for components of measurement plan elements is sketched. Components can be reused to build new measurement plans. A rudimentary reuse process is proposed that starts from a similarity analysis of the (reused and new) underlying process model and quality models.

The emergence of DevOps and Site Reliability Engineering (SRE) is a continuing merging of technical,operational, and organizational activities in software development and IT infrastructure operations. Itcreates a bridge between development and operations with rapid iteration and maintenance of system stability and reliability.

Getting the requirements right is one of the most important activities in software development. Making a crucial misstep at this phase can easily lead to large amounts of rework when the customer simply can't accept a system the way it was developed. When used correctly, approaches such as incremental development or agile methods can mitigate the risks of getting the requirements wrong by making sure that systems are developed in smaller chunks and that each chunk can be shown to the customer for approval. However, the best way to develop a high-quality system with minimal effort is still to get the requirements right the first time.

Since its establishment, the major objective of the Tropos methodology has been to develop an approach for the systematic engineering of agentoriented information systems. In this chapter we illustrate a number of approaches to deal with complexity, which address different activities in software development and are deemed to be used in combination. We begin with handling complexity at requirements levels. In particular we examine how modularization can be improved using some of Aspect Oriented Software Development Principles. We then examine how model-based testing applied in parallel to requirements analysis and design can support incremental validation and testing of software components, as well as help to clarify ambiguities in requirements. We also look at how Tropos can help to address complexity in social context when making design decisions. Last but not least, we show how to tackle complexity at the process modelling level. We explore iterative development extension to Tropos as well as perspectives taken from software project management. This allows us to deal with the complexity of large real world projects.

Software integration and reuse is becoming one of the most relevant activities in software development. This chapter will discuss how F# programs can interoperate with the outside world by accessing code available from both .NET and other languages.