Assessing the complexity of upgrading software modules

In software evolution, keeping the test code co‐change with the production code is important, because the outdated test code may not work and is ineffective in revealing faults in the production code. However, due to the tight development time, the production and test code may not be co‐changed immediately by developers. For example, we analysed the top 1003 popular Java projects on GitHub and found that nearly 9.3% of cases (i.e., 464,417) did not update their production and test code at the same time, that is, the production code is updated first, and then the test code is updated at intervals. The result indicates that much test code will not be updated in time. In this paper, we propose a novel approach, Jtup, to remind developers to co‐change the production code and test code in time. Specifically, we first define the co‐changed production and test code as a positive instance, while unchanged test code (i.e., production code changed and test code unchanged) as a negative instance. Then, we extract multidimensional features from the production code to characterize the possibility of their co‐change, including code change features, code complexity features, and code semantic features. Finally, several machine learning‐based methods are employed to identify the co‐changed production and test code. We conduct comprehensive experiments on 20 datasets, and the results show that the Accuracy, Precision, and Recall achieved by Jtup are 76.7%, 78.1%, and 77.4%, which outperforms the state‐of‐the‐art method.

Many software processes advocate that the test code should co-evolve with the production code. Prior work usually studies such co-evolution based on production-test co-evolution samples mined from software repositories. A production-test co-evolution sample refers to a pair of a test code change and a production code change where the test code change triggers or is triggered by the production code change. The quality of the mined samples is critical to the reliability of research conclusions. Existing studies mined production-test co-evolution samples based on the following assumption: if a test class and its associated production class change together in one commit, or a test class changes immediately after the changes of the associated production class within a short time interval, this change pair should be a production-test co-evolution sample . However, the validity of this assumption has never been investigated. To fill this gap, we present an empirical study, investigating the reasons for test code updates occurring after the associated production code changes, and revealing the pervasive existence of noise in the production-test co-evolution samples identified based on the aforementioned assumption by existing works. We define a taxonomy of such noise, including six categories (i.e., adaptive maintenance, perfective maintenance, corrective maintenance, indirectly related production code update, indirectly related test code update, and other reasons). Guided by the empirical findings, we propose CHOSEN (an identifi C ation met H od O f production-te S t co- E volutio N ) based on a two-stage strategy. CHOSEN takes a test code change and its associated production code change as input, aiming to determine whether the production-test change pair is a production-test co-evolution sample. Such identified samples are the basis of or are useful for various downstream tasks. We conduct a series of experiments to evaluate our method. Results show that (1) CHOSEN achieves an AUC of 0.931 and an F1-score of 0.928, significantly outperforming existing identification methods, and (2) CHOSEN can help researchers and practitioners draw more accurate conclusions on studies related to the co-evolution of production and test code. For the task of Just-In-Time (JIT) obsolete test code detection, which can help detect whether a piece of test code should be updated when developers modify the production code, the test set constructed by CHOSEN can help measure the detection method’s performance more accurately, only leading to 0.76% of average error compared with ground truth. In addition, the dataset constructed by CHOSEN can be used to train a better obsolete test code detection model, of which the average improvements on accuracy, precision, recall, and F1-score are 12.00%, 17.35%, 8.75%, and 13.50% respectively.

Numerous software development practices suggest updating the test code whenever the production code is changed. However, previous studies have shown that co-evolving test and production code is generally a difficult task that needs to be thoroughly investigated. In this paper we perform a study that, following a mixed methods approach, investigates fine-grained co-evolution patterns of production and test code. First, we mine fine-grained changes from the evolution of 5 open-source systems. Then, we use an association rule mining algorithm to generate the co-evolution patterns. Finally, we interpret the obtained patterns by performing a qualitative analysis. The results show 6 co-evolution patterns and provide insights into their appearance along the history of the analyzed software systems. Besides providing a better understanding of how test code evolves, these findings also help identify gaps in the test code thereby assisting both researchers and developers.

Code completion is an important feature in Integrated Development Environments (IDEs). These years, researchers have been making efforts for intelligent code completion. However, existing work on intelligent code completion either only considered production code, or did not distinguish between production code and test code. It is unclear how effective existing completion models are for test code completion, nor whether we can further improve it. In this work, we focus on the completion of test code. We first find through experiments that completion models for production code are suboptimal for test code completion. Then we analyze the specific characteristics of test code, and observe that test code has inter- and intra-project similarities, and a strong relationship with its focal class and other production classes depending on the focal class (i.e., focal-related code). By incorporating test code from other projects to fine-tune existing models, we leverage the inter-project similarity of test code to improve the completion of tokens specific to test code. By introducing a local component and constructing existing test code as well as the focal-related code in the project as references, we enhance existing code completion models with the intra-project similarity and the focal-related code of test code. Experiments show that each characteristic of test code we exploit can bring substantial improvement to test code completion and our integrated framework outperforms other baseline frameworks. Compared to the base completion model, on token-level completion, our optimal model for test code completion relatively improves all-token and identifier completion accuracy by 7.68% and 19.96%, respectively; on line-level completion, it relatively improves edit-distance similarity and exact-match metrics by 8.89% and 22.82%, respectively. Moreover, we perform error analysis and point out potential directions for future work.

Recent research has shown that clones occur far more often in test code than in production code. This is explained by the typical template of unit test code (the setup-stimulate-verify-teardown cycle), a template which lends itself well to parameterisation of proven solutions. However, little is known on how these clones evolve over time. In this paper, we compare the evolution of clones in test code with those in production code. Based on a quantitative and qualitative study of eight representative open-source systems, we conclude that clones in test code are inherently different than clones in production code throughout development. As a consequence, consistent maintenance and tracking of test clones becomes more important, hence the need for special purpose clone tracking and management tools.

Automated testing (hereafter referred to as just `testing') has become an essential process for improving the quality of software systems. In fact, testing can help to point out defects and to ensure that production code is robust under many usage conditions. However, writing and maintaining high-quality test code is challenging and frequently considered of secondary importance. Managers, as well as developers, do not treat test code as equally important as production code, and this behaviour could lead to poor test code quality, and in the future to defect-prone production code. The goal of my research is to bring awareness to developers on the effect of poor testing, as well as helping them in writing better test code. To this aim, I am working on 2 different perspectives: (1) studying best practices on software testing, identifying problems and challenges of current approaches, and (2) building new tools that better support the writing of test code, that tackle the issues we discovered with previous studies. Pre-print: https://doi.org/10.5281/zenodo.1411241

Log statements play a critical role in modern software development, capturing essential runtime information necessary for software maintenance. Recently, new techniques have been developed to automate logging activities, allowing log statements to be injected into code by identifying specific code locations, selecting the appropriate log level, and generating meaningful log messages that describe the behavior being logged. Although automated logging in production code has attracted significant attention, little focus has been given to the injection of logs in test code. To fill this gap, we conduct an empirical study on 5,206,759 Java test methods collected from 6,405 GitHub projects to explore and disclose the effectiveness and limitations of Pre-trained Language Models (PLMs) and Large Language Models (LLMs) for generating and injecting test log statements. Our findings demonstrate that general-purpose LLMs like GPT-3.5-Turbo, when properly instructed to inject logging statements in test methods, performs comparably to the best-performing PLMs on predicting log level. Additionally, GPT-3.5-Turbo substantially outperforms the best in PLMs on predicting log position with a 33.97% improvement while also achieving superior performance in predicting log messages in terms of BLEU and ROUGE . This work takes the first step toward evaluating the capability of PLMs and LLMs to generate test log statement. This work takes the first step toward evaluating the capability of PLMs and LLMs to generate test log statements. To facilitate future research, we have open-sourced all data and source code used in this work.

Refactoring is an activity that improves the internal structure of the code without altering its external behavior. When performed on the production code, the tests can be used to verify that the external behavior of the production code is preserved. However, when the refactoring is performed on test code, there is no safety net that assures that the external behavior of the test code is preserved. In this paper, we propose to adopt mutation testing as a means to verify if the behavior of the test code is preserved after refactoring. Moreover, we also show how this approach can be used to identify the part of the test code which is improperly refactored.

Production and test code co-evolution is known to result in high quality, maintainable, more sustainable software artifacts. This report discusses the challenges and experiences obtained in the transformation from a traditional development process, where most of the testing has been conducted manually and in a subsequent development step, to an agile development process that enforces a certain number of test code coverage by automated tests. Within an industrial project we analyze the deviations from the aimed co-evolution path by means of customized visualizations and list and discuss the observed challenges.

Unit testing represents a key activity in software development and maintenance. Test suites with high internal quality facilitate maintenance activities, such as code comprehension and regression testing. Several guidelines have been proposed to help developers write good test suites. Unfortunately, such rules are not always followed resulting in the presence of bad test code smells (or simply test smells). Test smells have been defined as poorly designed tests and their presence may negatively affect the maintainability of test suites and production code. Despite the many studies that address code smells in general, until now there has been no empirical evidence regarding test smells (i) distribution in software systems nor (ii) their impact on the maintainability of software systems. This paper fills this gap by presenting two empirical studies. The first study is an exploratory analysis of 18 software systems (two industrial and 16 open source) aimed at analyzing the distribution of test smells in source code. The second study, a controlled experiment involving twenty master students, is aimed at analyzing whether the presence of test smells affects the comprehension of source code during software maintenance. The results show that (i) test smells are widely spread throughout the software systems studied and (ii) most of the test smells have a strong negative impact on the comprehensibility of test suites and production code.

Software quality depends not only on production code but also on the quality of test code. Among the factors that compromise this quality, test smells stand out, poor implementation patterns in tests that negatively affect the readability, maintainability, and reliability of the test suite. Although tools such as tsDetect and JNose support test smell detection, they operate post-hoc, requiring the code to be complete and analyzed outside the development environment. Other tools, like RAIDE, offer detection within the IDE via plugins; however, the analysis is not instantaneous and still occurs after the code is written. These reactive approaches result in delayed feedback and significantly increase the refactoring effort. In this context, we propose AriesLinter, a static analysis tool that can be integrated into major development environments and is capable of detecting test smells in real time, as the test code is being written. Built on top of the Checkstyle infrastructure, AriesLinter automatically detects 10 of the most recurrent test smells reported in the literature and commonly found in Java projects. The tool is cross-platform and compatible with multiple IDEs. By anticipating the detection of poor patterns and embedding it into the natural workflow of developers, AriesLinter acts proactively, reducing the likelihood of test smells being introduced into the codebase and fostering a culture of continuous attention to test quality. Demo video: https://youtu.be/Cj6mQGGHlvk

Bug reports are submitted by the software stakeholders to foster the location and elimination of bugs. However, in large-scale software systems, it may be impossible to track and solve every bug, and thus developers should pay more attention to High-Impact Bugs (HIBs). Previous studies analyzed textual descriptions to automatically identify HIBs, but they ignored the quality of code, which may also indicate the cause of HIBs. To address this issue, we integrate the features reflecting the quality of production (i.e. CK metrics) and test code (i.e. test smells) into our textual similarity based model to identify HIBs. Our model outperforms the compared baseline by up to 39% in terms of AUC-ROC and 64% in terms of F-Measure. Then, we explain the behavior of our model by using SHAP to calculate the importance of each feature, and we apply case studies to empirically demonstrate the relationship between the most important features and HIB. The results show that several test smells (e.g. Assertion Roulette, Conditional Test Logic, Duplicate Assert, Sleepy Test) and product metrics (e.g. NOC, LCC, PF, and ProF) have important contributions to HIB identification.

Legacy code lacking unit tests is difficult to refactor. Without unit tests all changes to the code are dangerous. On the other hand in its unrefactored state the code lacks the modularity necessary for adding unit tests. Therefore, use Probe Tests. Probe tests are assertions placed in code which instead of crashing the program on failure simply log their success or failure to a file. The logged success or failure can then be presented through any xUnit framework. Probe tests balance these forces: time to recover from test failure, ease of parsing log files, difficulty of testing poorly refactored code, separation of production and test code.

Automation of acceptance test suites becomes necessary in the context of agile software development practices, which require rapid feedback on the quality of code changes. To this end, companies try to automate their acceptance tests as much as possible. Unfortunately, the growth of the automated test suites, by several automation testers, gives rise to potential test smells, i.e., poorly designed test code, being introduced in the test code base, which in turn may increase the cost of maintaining the code and creating new one. In this paper, we investigate this problem in the context of our industrial partner, BGL BNP Paribas, and introduce Ukwikora, an automated tool that statically analyzes acceptance test suites, enabling the continuous inspection of the test code base. Ukwikora targets code written in the Robot Framework syntax, a popular framework for writing Keyword-Driven tests. Ukwikora has been successfully deployed at BGL BNP Paribas, detecting issues otherwise unknown to the automation testers, such as the presence of duplicated test code, dead test code and dependency issues among the tests. The success of our case study reinforces the need for additional research and tooling for acceptance test suites.

From 2010 to 2016, as part of the priority programme SPP1476 “Small Machine Tools for Small Workpieces” funded by the German Research Foundation (DFG) a new modular machine tools concept was scientifically established. The SPP1476 consisted of 13 projects from 18 leading institutes of Production Technology allocated to 13 German universities and research organisations. The objective of the priority programme was to solve the anomaly of an inappropriate machine tool size to workpiece size ratio. This was done by interdisciplinary and multidisciplinary development of new approaches for machine tools, which are used for manufacturing small parts, based on a methodical concept. By the consistent use of enablers which arise from miniaturisation, new allocation of machine elements to functions was possible. The individual project partners developed different modules for modular small machine tools. This includes various tools, feed units, a measurement system, machine frame, interface and other additional modules. Eventually, the generated methods have been validated by the realisation of prototypes and the configuration and assembling to functional machines tools.