Practices and tools for better software testing

In software evolution, keeping the test code co‐change with the production code is important, because the outdated test code may not work and is ineffective in revealing faults in the production code. However, due to the tight development time, the production and test code may not be co‐changed immediately by developers. For example, we analysed the top 1003 popular Java projects on GitHub and found that nearly 9.3% of cases (i.e., 464,417) did not update their production and test code at the same time, that is, the production code is updated first, and then the test code is updated at intervals. The result indicates that much test code will not be updated in time. In this paper, we propose a novel approach, Jtup, to remind developers to co‐change the production code and test code in time. Specifically, we first define the co‐changed production and test code as a positive instance, while unchanged test code (i.e., production code changed and test code unchanged) as a negative instance. Then, we extract multidimensional features from the production code to characterize the possibility of their co‐change, including code change features, code complexity features, and code semantic features. Finally, several machine learning‐based methods are employed to identify the co‐changed production and test code. We conduct comprehensive experiments on 20 datasets, and the results show that the Accuracy, Precision, and Recall achieved by Jtup are 76.7%, 78.1%, and 77.4%, which outperforms the state‐of‐the‐art method.

Numerous software development practices suggest updating the test code whenever the production code is changed. However, previous studies have shown that co-evolving test and production code is generally a difficult task that needs to be thoroughly investigated. In this paper we perform a study that, following a mixed methods approach, investigates fine-grained co-evolution patterns of production and test code. First, we mine fine-grained changes from the evolution of 5 open-source systems. Then, we use an association rule mining algorithm to generate the co-evolution patterns. Finally, we interpret the obtained patterns by performing a qualitative analysis. The results show 6 co-evolution patterns and provide insights into their appearance along the history of the analyzed software systems. Besides providing a better understanding of how test code evolves, these findings also help identify gaps in the test code thereby assisting both researchers and developers.

Many software processes advocate that the test code should co-evolve with the production code. Prior work usually studies such co-evolution based on production-test co-evolution samples mined from software repositories. A production-test co-evolution sample refers to a pair of a test code change and a production code change where the test code change triggers or is triggered by the production code change. The quality of the mined samples is critical to the reliability of research conclusions. Existing studies mined production-test co-evolution samples based on the following assumption: if a test class and its associated production class change together in one commit, or a test class changes immediately after the changes of the associated production class within a short time interval, this change pair should be a production-test co-evolution sample . However, the validity of this assumption has never been investigated. To fill this gap, we present an empirical study, investigating the reasons for test code updates occurring after the associated production code changes, and revealing the pervasive existence of noise in the production-test co-evolution samples identified based on the aforementioned assumption by existing works. We define a taxonomy of such noise, including six categories (i.e., adaptive maintenance, perfective maintenance, corrective maintenance, indirectly related production code update, indirectly related test code update, and other reasons). Guided by the empirical findings, we propose CHOSEN (an identifi C ation met H od O f production-te S t co- E volutio N ) based on a two-stage strategy. CHOSEN takes a test code change and its associated production code change as input, aiming to determine whether the production-test change pair is a production-test co-evolution sample. Such identified samples are the basis of or are useful for various downstream tasks. We conduct a series of experiments to evaluate our method. Results show that (1) CHOSEN achieves an AUC of 0.931 and an F1-score of 0.928, significantly outperforming existing identification methods, and (2) CHOSEN can help researchers and practitioners draw more accurate conclusions on studies related to the co-evolution of production and test code. For the task of Just-In-Time (JIT) obsolete test code detection, which can help detect whether a piece of test code should be updated when developers modify the production code, the test set constructed by CHOSEN can help measure the detection method’s performance more accurately, only leading to 0.76% of average error compared with ground truth. In addition, the dataset constructed by CHOSEN can be used to train a better obsolete test code detection model, of which the average improvements on accuracy, precision, recall, and F1-score are 12.00%, 17.35%, 8.75%, and 13.50% respectively.

Code completion is an important feature in Integrated Development Environments (IDEs). These years, researchers have been making efforts for intelligent code completion. However, existing work on intelligent code completion either only considered production code, or did not distinguish between production code and test code. It is unclear how effective existing completion models are for test code completion, nor whether we can further improve it. In this work, we focus on the completion of test code. We first find through experiments that completion models for production code are suboptimal for test code completion. Then we analyze the specific characteristics of test code, and observe that test code has inter- and intra-project similarities, and a strong relationship with its focal class and other production classes depending on the focal class (i.e., focal-related code). By incorporating test code from other projects to fine-tune existing models, we leverage the inter-project similarity of test code to improve the completion of tokens specific to test code. By introducing a local component and constructing existing test code as well as the focal-related code in the project as references, we enhance existing code completion models with the intra-project similarity and the focal-related code of test code. Experiments show that each characteristic of test code we exploit can bring substantial improvement to test code completion and our integrated framework outperforms other baseline frameworks. Compared to the base completion model, on token-level completion, our optimal model for test code completion relatively improves all-token and identifier completion accuracy by 7.68% and 19.96%, respectively; on line-level completion, it relatively improves edit-distance similarity and exact-match metrics by 8.89% and 22.82%, respectively. Moreover, we perform error analysis and point out potential directions for future work.

Recent research has shown that clones occur far more often in test code than in production code. This is explained by the typical template of unit test code (the setup-stimulate-verify-teardown cycle), a template which lends itself well to parameterisation of proven solutions. However, little is known on how these clones evolve over time. In this paper, we compare the evolution of clones in test code with those in production code. Based on a quantitative and qualitative study of eight representative open-source systems, we conclude that clones in test code are inherently different than clones in production code throughout development. As a consequence, consistent maintenance and tracking of test clones becomes more important, hence the need for special purpose clone tracking and management tools.

Effective and efficient testing with reduced costs and a high fault detection capability is the desirable goal in industry which can be achieved only through automation of all parts of the testing process. In the past decades, a great amount of research effort has been spent on automating all various parts of the testing process such as test case derivation, test selection, test oracle construction, test execution, and others. In addition, there has been a rapid growth in automated software testing tools which is stimulated in part through the shift towards agile development practices in industry that demands a high level of automation. Work on this topic has long been published as an important part of software engineering. In recent years, testing has been consistently among the top-most popular topics in submissions to software engineering conferences. The practice of software test automation (TA) has also moved forward significantly in the past few years. However, progress in TA is still required. Software systems have become more and more complicated through the integration of components developed by different vendors and using different techniques in different programming languages running on different platforms. The advent of cloud computing, mobile computing and the Internet of Things has imposed grave new challenges to TA. Those systems become increasingly reactive to changes in their environment, requiring equally adaptive TA approaches. Few software testing tools can currently handle the needed requirements to test such systems.

Test smells are bad practices to either design or implement a test code. Their presence may reduce the test code quality, harming the software testing activities, primarily from a maintenance perspective. Therefore, defining strategies and tools to handle test smells and improve the test code quality is necessary. State-of-the-art strategies encompass automated support mainly based on hard thresholds of rules, static and dynamic metrics to identify the test smells. Such thresholds are subjective to interpretation and may not consider the complexity of the software projects. Moreover, they are limited as they do not automate test refactoring but only count on developers’ expertise and intuition. In this context, a technique that uses historical implicit or tacit data to generate knowledge could assist the identification and refactoring of test smells. This study aims to establish a novel approach based on machine learning techniques to suggest developers refactoring strategies for test smells. As an expected result, we could understand the applicability of the machine learning techniques to handle test smells and a framework proposal that helps developers in decision-making regarding the refactoring of test smells.

Bug reports are submitted by the software stakeholders to foster the location and elimination of bugs. However, in large-scale software systems, it may be impossible to track and solve every bug, and thus developers should pay more attention to High-Impact Bugs (HIBs). Previous studies analyzed textual descriptions to automatically identify HIBs, but they ignored the quality of code, which may also indicate the cause of HIBs. To address this issue, we integrate the features reflecting the quality of production (i.e. CK metrics) and test code (i.e. test smells) into our textual similarity based model to identify HIBs. Our model outperforms the compared baseline by up to 39% in terms of AUC-ROC and 64% in terms of F-Measure. Then, we explain the behavior of our model by using SHAP to calculate the importance of each feature, and we apply case studies to empirically demonstrate the relationship between the most important features and HIB. The results show that several test smells (e.g. Assertion Roulette, Conditional Test Logic, Duplicate Assert, Sleepy Test) and product metrics (e.g. NOC, LCC, PF, and ProF) have important contributions to HIB identification.

Software quality depends not only on production code but also on the quality of test code. Among the factors that compromise this quality, test smells stand out, poor implementation patterns in tests that negatively affect the readability, maintainability, and reliability of the test suite. Although tools such as tsDetect and JNose support test smell detection, they operate post-hoc, requiring the code to be complete and analyzed outside the development environment. Other tools, like RAIDE, offer detection within the IDE via plugins; however, the analysis is not instantaneous and still occurs after the code is written. These reactive approaches result in delayed feedback and significantly increase the refactoring effort. In this context, we propose AriesLinter, a static analysis tool that can be integrated into major development environments and is capable of detecting test smells in real time, as the test code is being written. Built on top of the Checkstyle infrastructure, AriesLinter automatically detects 10 of the most recurrent test smells reported in the literature and commonly found in Java projects. The tool is cross-platform and compatible with multiple IDEs. By anticipating the detection of poor patterns and embedding it into the natural workflow of developers, AriesLinter acts proactively, reducing the likelihood of test smells being introduced into the codebase and fostering a culture of continuous attention to test quality. Demo video: https://youtu.be/Cj6mQGGHlvk

Background & Problem Statement - Software testing is a critical phase in the software development lifecycle (SDLC), ensuring that applications function correctly, meet user requirements, and maintain high- quality standards. Traditional software testing approaches, including manual testing and rule-based automation, often face challenges in scalability, efficiency, and adaptability to dynamic software environments. Traditional testing methods are overwhelmed by complex software systems which slows down defect detection and extends both testing costs and release schedules. Machine Learning (ML) has emerged as a transformative solution, introducing predictive and adaptive capabilities that optimize test case selection, automate defect detection, and enhance overall software quality assurance (QA). This study explores the integration of ML in software testing, addressing the challenges of traditional QA methodologies and demonstrating how AI-driven frameworks improve testing efficiency. Methodology - To investigate the impact of ML in software testing, this research adopts a systematic approach by analyzing ML-driven test automation techniques, including predictive testing, adaptive test execution, and automated test case generation. Research reviews how Google Microsoft Facebook IBM and Deep Code put ML-based quality assurance frameworks into operation. The study leverages supervised learning, reinforcement learning, deep learning, and NLP-based techniques to demonstrate how ML models predict software defects, dynamically adapt test cases, and optimize testing resources. The research tests how ML-based testing models operate within CI/CD pipelines to improve ongoing testing and deployment flow. Analysis & Results - The analysis of ML-driven software testing reveals that predictive analytics improves early defect detection rates. It helps developers spend 37% less time debugging their work. Adaptive testing models, including self-healing test scripts, minimize maintenance costs by 50% and enhance test reliability in agile environments. The integration of NLP-based test case generation increases test coverage. NLP technology enables automatic connection between requirements and test cases at 89% success rate. Additionally, reinforcement learning techniques improve test case selection, reducing redundant test executions by 43%. Our research shows different ML methods work well to lessen incorrect error alerts. ML integration for QA surely increasing defect prediction accuracy and optimizing test execution time. Findings & Contributions - This research contributes to the field of AI-driven software testing by providing a comprehensive framework for ML-based QA methodologies. Our study shows that machine learning helps find more software problems better adapts test cases and lowers testing expenses to solve present software development needs. The study also identifies critical challenges, including data availability, model interpretability, and computational overhead, suggesting future research directions in Explainable AI (XAI), hybrid AI-ML testing models, and AI-driven security testing. As the industry moves toward AI-first software testing, this research paves the way for fully autonomous QA frameworks, enabling intelligent, scalable, and cost- effective software validation techniques. Keywords - Machine Learning, Software Testing, Quality Assurance, Predictive Testing, Adaptive Testing, Test Automation, Defect Prediction, Self- Healing Test Scripts, AI-Driven QA, Reinforcement

In the process of software development, software testing is a huge project, and most of the software functionality testing is complex and repetitive, should to spend a lot of cost and manpower. Therefore, you can save a lot of manpower and costs, avoid human error and improve software quality if we can achieve automated testing. In the software development process, the application of model-driven developments has grown wider recently. The main concept is to fetch all of the significant properties referring to system design, and use formal or semi-formal model to describe system in different levels of abstraction. In this paper, we apply a tool based on high level Petri Net model of automated testing -- MISTA [1](Model based Integration and System Testing Automation) to automated generate test code for system. In this paper, through capture the information of the state machine diagram describes the behavior of the system in the Unified Modeling Language. Through the eXtensible Markup Language automation converted to High Level Petri Net test model, and then using automated test code generating software MISTA generation test cases and test code of system to achieve automated testing, and save the software required for testing a large number of human costs, and the software development process from design architecture to the software testing for a bridging.

Refactoring is an activity that improves the internal structure of the code without altering its external behavior. When performed on the production code, the tests can be used to verify that the external behavior of the production code is preserved. However, when the refactoring is performed on test code, there is no safety net that assures that the external behavior of the test code is preserved. In this paper, we propose to adopt mutation testing as a means to verify if the behavior of the test code is preserved after refactoring. Moreover, we also show how this approach can be used to identify the part of the test code which is improperly refactored.

Software testing provides a means to reduce errors, cut maintenance and overall software costs. Numerous software development and testing methodologies, tools, and techniques have emerged over the last few decades promising to enhance software quality. While it can be argued that there has been some improvement it is apparent that many of the techniques and tools are isolated to a specific lifecycle phase or functional area. This paper presents a set of best practice models and techniques integrated in optimized and quantitatively managed software testing process (OptimalSQM), expanding testing throughout the SDLC. Further, we explained how can Quantitative Defect Management Model be enhanced to be practically useful for determining which activities need to be addressed to improve the degree of early and cost-effective software fault detection with assured confidence is proposed. To enable software designers to achieve a higher quality for their design, a better insight into quality predictions for their design choices, test plans improvement using Simulated Defect Removal Cost Savings model is offered in this paper.

Automated and systematic testing of automation systems (AS) and production systems (PS) require an integrated testing tool chain for test case development, execution and reporting. In practice, the test automation tool chain cannot be fully automated because of missing links between different tools used in the test automation process. Closing these gaps typically require (high) human effort. Furthermore, domain and software testing expertise is often bundled by one (expensive) engineer who is responsible for the application domain (reflected in use cases and test cases) and software tests (software test code). This paper presents a flexible Testing Automation Framework (TAF) that enables the configuration of test processes involving different tools and various layers for test automation and enables separated roles for the application domain and software tests. We build on best-practice test automation from Software Engineering and design a test automation process for the automation systems domain. We demonstrate the feasibility with a use case, derived from production systems automation, with selected tools covering all test automation layers. First results showed the feasibility of the framework in the evaluation use case making test processes more flexible and automated. Although the successful implementation of the TAF can support the efficient configuration and execution of test processes, there is additional effort for preparing the flexible and automated tool chain.

Software Testing is known as an important part in software development as the product needs to meet the user's expectations and requirements. The testing tool is developed along with the development of automated software testing to act as a medium to test the software, and the existence of these tools eventually cut down the human's effort, energy, and time. Software testing tools are the key assets of a project that can improve productivity and software quality. As there are many tools in the market nowadays, the process of selecting the tool should be explained in detail and specific towards its functions and efficiency. This paper aims to help the authors in making the tool selection from the various versions of testing tools. This study finally chose the most suitable tool according to the research needs.