Unit testing tool competition: round three

This paper describes the third round of the Java Unit Testing Tool Competition. This edition of the contest evaluates no less than seven automated testing tools! And, like during the second round, test suites written by human testers are also used for comparison. This paper contains the full results of the evaluation.

Use of mobile applications are trending these days due to adoption of handheld mobile devices with operating systems such as Android, iOS and Windows. Delivering quality mobile apps is as important as in any other web or desktop application. Simplification and ease of quality assurance or evaluation in mobile devices is achieved by using automated testing tools. These tools have been evaluated for their features, platforms, code coverage, and efficiency. However, they have not been evaluated and compared to each other for different quality attributes they can enhance in the apps under test. This research study aims to evaluate different testing tools focusing on identifying quality factors they aid to achieve in the apps under test. Furthermore, it aims to measure overall trends of essential quality factors achieved using automated testing tools. The findings of this study are beneficial to the practitioners and researchers. The practitioners need to look up for specific tools which aid them to assure the desired quality factors in the apps under test. The researchers may base their studies on the findings of this study to propose solutions or revise existing tools in order to achieve maximum number of critical quality attributes in the app under test. This study revealed that the trend of automated testing is high on usability, correctness and robustness. Moreover, the trend is average on testability and performance. However, for assurance of extensibility, maintainability, scalability, and platform compatibility, only a few tools are available.

This paper describes the methodology and results of the 4th edition of the Java Unit Testing Tool Competition. This year's competition features a number of infrastructure improvements, new test effectiveness metrics, and the evaluation of the test generation tools for multiple time budgets. Overall, the competition evaluated four automated test generation tools. This paper details the methodology and contains the full results of the competition.

This paper studies the software test with automated test tool under cloud environment. The automated software test technology could speed up the test efficiency and less down the work of test engineers. On the cloud technology, the automated test tool should make some modification to adapt for cloud. The paper proposes the test integrated technology with automated test tool on cloud-based environment. This technology connects cloud-based environment and the automated test tool. The fully test procedure also are proposed to support test design, script develop and script execute. The cloud technology is used to test some applications with the automated test tool and archive the expected outcome.

Software testing is an integral part of software development lifecycle. Lack of testing can often lead to disastrous consequences including lost of data, fortunes, and even lives. Despite its importance, current software testing practice lacks automation, and is still primarily based on highly manual processes from the generation of test cases up to the actual execution of the test. Although the emergence of helpful automated testing tools in the market is blooming, their adoptions are lacking as they do not adequately provide the right level abstraction and automation required by test engineers. JTst is a Java based automated unit testing tool that addresses some of the aforementioned issues. The main novel features are the fact that JTst automates the test preparation activities, facilitates the test data generation through recombination, and allows concurrent execution of test data, in order to encourage higher product quality at lower testing costs.

In software testing process, test data generation represents an important step for high quality software, even for mobile devices. As proposed in previous works, a potential source for random data generation is represented by the UI layout files that are used for almost all mobile platforms (Android, iOS, Windows Phone/Mobile). This paper continues the previous work and presents a test data generation system based on Android layout files. The test data generator uses DSL files as input and generates test data that conform to several testing principles. The generated test data could be stored in XML files or any format required by the testing frameworks.

Dynamic analysis is a technique that is used to fully understand the internals of a system at runtime. On Android, dynamic security analysis involves real-time assessment and active adaptation of an app’s behaviour, and is used for various tasks, including network monitoring, system-call tracing, and taint analysis. The research on dynamic analysis has made significant progress in the past years. However, to the best of our knowledge, there is a lack in secondary studies that analyse the novel ideas and common limitations of current security research. The main aim of this work is to understand dynamic security analysis research on Android to present the current state of knowledge, highlight research gaps, and provide insights into the existing body of work in a structured and systematic manner. We conduct a systematic literature review (SLR) on dynamic security analysis for Android. The systematic review establishes a taxonomy, defines a classification scheme, and explores the impact of advanced Android app testing tools on security solutions in software engineering and security research. The study’s key findings centre on tool usage, research objectives, constraints, and trends. Instrumentation and network monitoring tools play a crucial role, with research goals focused on app security, privacy, malware detection, and software testing automation. Identified limitations include code coverage constraints, security-related analysis obstacles, app selection adequacy, and non-deterministic behaviour. Our study results deepen the understanding of dynamic analysis in Android security research by an in-depth review of 43 publications. The study highlights recurring limitations with automated testing tools and concerns about detecting or obstructing dynamic analysis.

Due to incremental upgrade approach, growing number of enterprise systems present heterogeneous UIs, in which certain sections/applications are 32 bit whereas others are 64 bit. Typical business process use-case interactions in these enterprise systems span both these sections, simultaneously. However conventional test automation tools have been designed to test either 32 bit applications or 64 bit applications only, but not both simultaneously. The test tools cannot directly detect custom controls or third party components and hence rely on "hooking into these UI controls" to get access. These hooking components need to be managed for interoperability. Hooking allows test code to monitor messages received by the application under test. However Windows OS puts the following constraints: 1. If an application requires the use of hooks in other processes, it is required that a 32-bit application can inject a 32-bit DLL into 32-bit processes, and a 64-bit application can inject a 64-bit DLL into 64-bit processes. 2. The 32-bit and 64-bit DLLs must have different names. The first constraint impedes seamless interoperation of the 32 bit and 64 bit accessing sections of the test script. The second constraint implies separate management of the same injection code increasing maintenance effort. In this paper we report our experience of co-ordination between the hooking components as a mechanism to interact with these heterogeneous user interfaces. Our bit sensitive manager component, co-ordinates the 64-bit test environment, allowing 32-bit controls to be accessed seamlessly without maintaining separately named injection components. This facilitates efficient interaction even if the communication between the 32 and 64 bit sections are intense. Importantly, the mechanism does not cause instability in the application under test nor does it increase the application's memory footprint significantly. This mechanism is envisioned to drive the "simultaneous 32bit - 64 bit handling" capability in the commercial test automation tools and will be used widely as enterprise systems take an incremental upgrade approach to migrate towards fully 64 bit systems.

This paper introduces a testing tool for regular expressions (regexes) named AREXT. AREXT can automatically extract regexes from C++ source files and visually represent them as DFA graphs. Given a regex, AREXT can generate a set of positive and negative strings with 100% coverage of nodes, edges, and edge pairs. We leverage prior works of synthesizing regexes from natural language to create benchmarks for evaluating AREXT. Some current tools, i.e., EGRET and MUTREX, are also being evaluated and compared. Experiments show that AREXT can outperform EGRET as AREXT can detect more unexpected synthesized regexes in almost all benchmarks. The evaluation results indicate that strings with 100% coverage metrics (generated by AREXT) or strings with maximum mutation score (generated by MUTREX) are not enough to ensure the correctness of regexes under test. Experiments also show that combining AREXT, EGRET, and MUTREX can detect a majority of unwanted synthesized regexes (87–91%).

Performance analysis of software reliability growth models with testing-effort and change-point

We discuss the potential for automatically prioritizing accessibility issues found by automated test tools in an application, based on 'accessibility impact' to the end user. We examine three influential factors in assessing the severity of an issue: criticality of the affected component, impact of the type of issue, and mitigating effect of the application context. We describe how these factors could be quantified and combined to yield severity measures consistent with those of a human expert, illustrating with a set of accessibility issues found in a real online web task.

The authors propose a software testing method for the detection of modification errors caused by functional specification changes. In the method, if inappropriate transformations are selected, the following problems may occur: (1) test cases cannot be obtained, (2) efficiency becomes worse, and (3) effectiveness declines.This paper identifies the transformations that the method can most effectively apply, employing Hatley's model as a software functional specification description model. It introduces the concept “change level” based on the analysis of actual software defects and shows that: (1) the transformations used in the change level are optimal for testing, and (2) when there exist two transformations in the change level whose restrictions are complementary, only one must be tested. The effectiveness of the proposed method is demonstrated by applying the method to an actual software product. The proposed testing method detected 90 percent of the defects due to functional specification changes in a software product. It can readily produce test cases even without automated testing tools when the restrictions consist of simple inequalities or equalities of the input parameters.

The use of an automated testing tool, the Circuit Switch Automated Test System (CSATS), is described. Use of this tool which has proven effective as a step in improving the quality of circuit switching systems. Benefits of automated testing, the CSATS concept and implementation, and example CSATS test scripts and their applications are summarized. The use of CSATS software in switching system analysis is illustrated, with emphasis on call processing testing. Associated capabilities and techniques are discussed. It is concluded that CSATS has proven effective in identifying requirements violations, minimizing analysis team size, shortening analysis intervals, maintaining test results for auditing, and consistently repeating tests to confirm fault removal. >

"Within the automotive product development cycle virtual and heterogeneous testing is becoming increasingly established through component, module and vehicle-level simulation. Though a number of standards in this field have been established, models are still mostly created in a fragmented manner: using domain-specific tools to create, manage and execute simulations without standardization of the content of the functional interfaces (FMI does only standardize the format) and limited scalability. This fragmentation leads to a lot of redundant effort as models of the same component or system are re-created several times. HIFI-ELEMENTS project addressed this fragmentation through two main mechanisms: Firstly, developing, validating and publishing a recommendation for standardization of model interfaces for common e-drive components (e-machine, inverter, battery, DC/DC converter, thermal management) and implementation of compliant versions of existing models. Secondly, implementing a seamless workflow linking extended versions of existing tools with effort-saving automated methods for model parameterization and test case generation. This seamless integration will substantially increase the number of integrations and test cases that can be early validated through simulation, leading to optimized efficiency designs and development effort reduction. The standardization also guarantees scalability among fidelity levels, from concept design to XiL through detailed modelling. In this paper we present the results of the Use Case C: Component co-optimization. The purpose of use cases is the demonstration of the advantages of the standardized models and workflow industry relevant scenarios. The work content performed in the use case is very extensive and multidisciplinary. In the first step, the high fidelity models from the expert components developers were validated independently with automated testing tools and later integrated to create a complete vehicle architecture integration. The standardization permitted to seamlessly test several component variants developed within the project for the same architecture, including tens of motor models with different technologies, inverters and high voltage converters with different IGBT technology and various battery packs. This possibility was exploited through co-optimization with multi objective Genetic Algorithm, permitting to select the optimal component combination, powertrain architecture (with and without high voltage DCDC converter) and components parametrization considering the trade-off of consumption and performances. The optimized and baseline variants were used to demonstrate the scalability of the models to different simulation objectives. The model was co-simulated with a traffic simulation environment in order to evaluate the impact of eco-driving recommendation algorithms in a realistic driving situation. The optimized solution was also validated against a wide database of driving conditions including real driving cycles, performance and vehicle dynamics. Finally, the integrated models were seamlessly transferred to real time simulation platform for Model-in-the-Loop testing with a simulated 3D environment aimed at ADAS testing. Real-time capability demonstrates that next steps such as Driver-in-the-Loop and Hardware-in-the loop can be achieved smoothly. The extensive simulation activities performed in this use case demonstrate the benefits of the standard in models exchangeability and effort reduction in model based development. This project received funding from the European Union’s (EU) Horizon 2020 Research and innovation program under grant agreement N 769935."

The accessibility of State universities and colleges (SUCs) websites is crucial for ensuring inclusivity and equal access to educational resources for all individuals, regardless of their abilities. This study evaluated the accessibility of SUCs in the Philippines websites using automated testing tools like WAVE and TAW, which analyzed web pages based on predefined accessibility criteria such as the Web Content Accessibility Guidelines (WCAG) 2.0. The evaluation process involved scanning and analyzing the websites to identify potential accessibility barriers or issues. The results of the accessibility evaluation reveal significant challenges in meeting WCAG guidelines across various aspects, such as Perceivable and Operable guidelines. Common issues include missing alternative text for images, unclear link descriptions, color contrast combinations, and inadequate form labeling. These findings emphasize the need for SUCs to prioritize web accessibility to ensure equal access to educational resources. Furthermore, it includes ranking SUCs' websites based on their accessibility scores, highlighting institutions that have demonstrated a strong commitment to accessibility, and identifying areas for improvement. MPSPC leads with a minimal mean error rate of 0.33, setting a benchmark for other institutions. PRMSU and GSU follow closely, demonstrating a commitment to enhancing web accessibility. The findings highlight the importance of adhering to accessibility guidelines to create an inclusive online learning environment for all users.