An empirical study of Android test generation tools in industrial cases

Since the emergence of web 2.0, the architecture of web applications has been transformed significantly and its complexity has grown enormously. In such web applications, the user interface (UI) is an important ingredient and with the increased complexity, its testing is getting increasingly complex and cost/time-consuming process. Recently introduced, interaction flow modeling language (IFML) is an object management group (OMG) standard. IFML is gaining popularity for developing web applications, primarily, because of its excellent features for modeling UI elements, their content, and their interaction capturing capabilities. However, despite its superior UI modeling features, its UI testing is accomplished through traditional time-consuming techniques, which are employed after implementing the UI code. Hence, to overcome these limitations, this paper introduces a novel model-based testing approach for IFML UI elements. The proposed approach provides complete navigation testing using formal models. Moreover, the approach transforms the IFML models to all necessary UI testing artifacts by generating state transition matrix plus detailed UI test case document. As a part of a research, model-based user interface test case (MBUITC) generator tool is implemented to automatically generate navigation model for formal verification, test case document, and transition matrices from IFML models. The applicability of the proposed approach is validated through two benchmark case studies. The results have shown that the proposed approach provides test cases at the early stages of development, i.e., specification and analysis, which eventually helps in building a right product at the right time at a comparatively lower cost.

In Combinatorial Interaction Testing, models specify a set of parameters with associated domains, and some constraints over the parameters. Test generation tools produce, starting from these models, test suites for achieving some given coverage criteria. The validation of both the models and the produced test suites is a worthwhile activity. Validating the models permits to early discover possible defects in them, to feed the test generations tools with good inputs and, possibly, to improve the quality of the testing process. Validating the produced test suites, instead, permits to check if the test generation tools are correct and to judge their quality. This paper proposes to validate the models by checking that the constraints are consistent, that there is no constraint implied by the other constraints, and that the parameters and their values are really necessary. The proposed test suite validation, instead, consists in checking that the tests respect the type definitions and the constraints, that all the test requirements are covered, and that all the tests in the test suite are valid and necessary. For every error we propose a possible technique able to identify the potential causes and to suggest fixes for those problems. Experiments show that the targeted defects are widespread both in benchmark and real-life models.

Does the interface quality of mobile shopping apps affect purchase intention? An empirical study

Adapting automated test generation to GUI testing of industry applications

Activity theory is commonly used in designing Human-device interaction. Donald Norman in his paper Human-Centered Design Considered Harmful criticized traditional approach, which he called Human-Centered Design, and introduced a new approach, called Activity-Centered Design. In this paper his new approach is directly used and widened in order to be used with a tool for test generation. It is used as a point of view, a philosophy, on which graphical user interfaces are designed. Workflow paradigm is used for sequential flow of elements, called activities. Activity theory is directly implemented on Workflow elements by splitting the work on: Activities, Actions and Operations. Combination of Activity theory and Workflow is used as start point for broad range of Human computer interaction, and in this paper it is used as upgrade of test generation framework for TV and STB testing called BBT (Black Box Testing). Goal of this approach is to simplify test script creation process for user with modest or none programming experience.

Nonspecific low back pain (NSLBP) carries significant socioeconomic relevance and leads to substantial difficulties for those who are affected by it. The effectiveness of app-based treatments has been confirmed, and clinicians are recommended to use such interventions. As 88.8% of the German population uses smartphones, apps could support therapy. The available apps in mobile app stores are poorly regulated, and their quality can vary. Overviews of the availability and quality of mobile apps for Australia, Great Britain, and Spain have been compiled, but this has not yet been done for Germany. We aimed to provide an overview of the availability and content-related quality of apps for the treatment of NSLBP in the German language. A systematic search for apps on iOS and Android was conducted on July 6, 2022, in the Apple App Store and Google Play Store. The inclusion and exclusion criteria were defined before the search. Apps in the German language that were available in both stores were eligible. To check for evidence, the apps found were assessed using checklists based on the German national guideline for NSLBP and the British equivalent of the National Institute for Health and Care Excellence. The quality of the apps was measured using the Mobile Application Rating Scale. To control potential inaccuracies, a second reviewer resurveyed the outcomes for 30% (3/8) of the apps and checked the inclusion and exclusion criteria for these apps. The outcomes, measured using the assessment tools, are presented in tables with descriptive statistics. Furthermore, the characteristics of the included apps were summarized. In total, 8 apps were included for assessment. Features provided with different frequencies were exercise tracking of prefabricated or adaptable workout programs, educational aspects, artificial intelligence-based therapy or workout programs, and motion detection. All apps met some recommendations by the German national guideline and used forms of exercises as recommended by the National Institute for Health and Care Excellence guideline. The mean value of items rated as "Yes" was 5.75 (SD 2.71) out of 16. The best-rated app received an answer of "Yes" for 11 items. The mean Mobile Application Rating Scale quality score was 3.61 (SD 0.55). The highest mean score was obtained in "Section B-Functionality" (mean 3.81, SD 0.54). Available apps in the German language meet guideline recommendations and are mostly of acceptable or good quality. Their use as a therapy supplement could help promote the implementation of home-based exercise protocols. A new assessment tool to obtain ratings on apps for the treatment of NSLBP, combining aspects of quality and evidence-based best practices, could be useful. Open Science Framework Registries sq435; https://osf.io/sq435.

User Interface (UI) testing has become a common practice for quality assurance of industrial mobile applications (in short as apps). While many automated tools have been developed, they often do not satisfy two major industrial requirements that make a tool desirable in industrial settings: high applicability across platforms (e.g., Android, iOS, AliOS, and Harmony OS) and high capability to handle apps with non-standard UI elements (whose internal structures cannot be acquired using platform APIs). Toward addressing these industrial requirements, automated visual testing emerges to take only device screenshots as input in order to support automated test generation. In this paper, we report our experiences of developing and deploying VTest, our industrial visual testing framework to assure high quality of Taobao, a highly popular industrial app with about one billion monthly active users. VTest includes carefully designed techniques and infrastructure support, outperforming Monkey (which has been popularly deployed in industry and shown to perform superiorly or similarly compared to state-of-the-art tools) with 87.6% more activity coverage. VTEST has been deployed both internally in Alibaba and externally in the Software Green Alliance to provide testing services for top smart-phone vendors and app vendors in China. We summarize five major lessons learned from developing and deploying VTEST.

Purpose Explore smartphone apps that may be recommended by clinicians for the self-management of low back pain. Methods Prospectively registered systematic assessment of self-contained apps for self-management of low back pain on the Google Play and Apple App stores (Oceania), including ≥1 NICE low back pain and sciatica clinical guideline-recommended component and functioning without health professional input. Outcomes were quality (Mobile App Rating Scale; MARS), and self-management (Self-Management Support Checklist; SMS-14) and behaviour change potential (App Behaviour Change Scale; ABACUS). Results 25 apps were included. The average quality of included apps was acceptable (Mean MARS score of 3.9 out of a maximum possible 5). The self-management support and behaviour change potential of included apps appeared low (mean SMS-14 score was 3.4/14; mean ABACUS score was 5.4/21). The apps showed no significant correlation between app consumer ratings and MARS scores. App quality was significantly correlated with app price (p = 0.049) but not consumer ratings, however, these findings were based on a small number of studies and the overall model was not significant. Conclusions Smartphone apps for the self-management of low back pain are of average to good quality, with questionable potential for self-management and behaviour change. Clinicians should consider that few apps were designed to specifically incorporate self-management support and behaviour change potential when recommending apps to clients. Further development in these areas of app design would be of benefit. Implications for Rehabilitation Smartphone apps have the potential to improve LBP self-management outcomes, however, apps are not well regulated and the quality of information and advice provided is often low quality. The findings from this systematic assessment indicate that LBP self-management apps have limited self-management support and behaviour change potential features. Recommendation of current smartphone apps for LBP should take into consideration that although apps are of acceptable quality they are not specifically designed with self-management support and behaviour change principles. App quality does not appear to be associated with consumer ratings, however, there may be a positive association between app price and quality which requires further investigation.

Different techniques to automatically generate unit tests for object oriented classes have been proposed, but how to integrate these tools into the daily activities of software development is a little investigated question. In this paper, we report on our experience in supporting industrial partners in introducing the EvoSuite automated JUnit test generation tool in their software development processes. The first step consisted of providing a plugin to the Apache Maven build infrastructure. The move from a research-oriented point-and-click tool to an automated step of the build process has implications on how developers interact with the tool and generated tests, and therefore, we produced a plugin for the popular IntelliJ Integrated Development Environment (IDE). As build automation is a core component of Continuous Integration (CI), we provide a further plugin to the Jenkins CI system, which allows developers to monitor the results of EvoSuite and integrate generated tests in their source tree. In this paper, we discuss the resulting architecture of the plugins, and the challenges arising when building such plugins. Although the plugins described are targeted for the EvoSuite tool, they can be adapted and their architecture can be reused for other test generation tools as well.

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A developer would first need to know when and how to apply such a tool, and would then need to understand the resulting tests in order to provide test oracles and to diagnose and fix any faults that the tests reveal. Considering all this, does automatically generating unit tests provide any benefit over simply writing unit tests manually? We empirically investigated the effects of using an automated unit test generation tool (EvoSuite) during development. A controlled experiment with 41 students shows that using EvoSuite leads to an average branch coverage increase of +13%, and 36% less time is spent on testing compared to writing unit tests manually. However, there is no clear effect on the quality of the implementations, as it depends on how the test generation tool and the generated tests are used. In-depth analysis, using five think-aloud observations with professional programmers, confirms the necessity to increase the usability of automated unit test generation tools, to integrate them better during software development, and to educate software developers on how to best use those tools.

With the rapidly growing scale of modern software, the reliability of software systems has become essential. To ease the developers' pressure of writing unit tests manually, test generation tools such as EvoSuite and Randoop were proposed. Although these approaches have been shown to be able to automatically generate tests for achieving high coverage, the generated tests may be ineffective in detecting real faults. Particularly, these automatically generated tests may suffer from several problems (we call them problematic tests): (1) incorrect oracle. (2) unexpected exception/error. (3) flaky test. We present a comprehensive study of EvoSuite in Defects4j, and performed a detailed analysis of the reasons behind these automatically generated problematic tests. Our analysis identifies 528 problematic tests: 208 (39.4%) of them are caused by incorrect oracle, 319 (60.4%) are caused by unexpected exception/error, and one flaky test.

BackgroundThe use of mobile apps for health and well being promotion has grown exponentially in recent years. Yet, there is currently no app-quality assessment tool beyond “star”-ratings.ObjectiveThe objective of this study was to develop a reliable, multidimensional measure for trialling, classifying, and rating the quality of mobile health apps.MethodsA literature search was conducted to identify articles containing explicit Web or app quality rating criteria published between January 2000 and January 2013. Existing criteria for the assessment of app quality were categorized by an expert panel to develop the new Mobile App Rating Scale (MARS) subscales, items, descriptors, and anchors. There were sixty well being apps that were randomly selected using an iTunes search for MARS rating. There were ten that were used to pilot the rating procedure, and the remaining 50 provided data on interrater reliability.ResultsThere were 372 explicit criteria for assessing Web or app quality that were extracted from 25 published papers, conference proceedings, and Internet resources. There were five broad categories of criteria that were identified including four objective quality scales: engagement, functionality, aesthetics, and information quality; and one subjective quality scale; which were refined into the 23-item MARS. The MARS demonstrated excellent internal consistency (alpha = .90) and interrater reliability intraclass correlation coefficient (ICC = .79).ConclusionsThe MARS is a simple, objective, and reliable tool for classifying and assessing the quality of mobile health apps. It can also be used to provide a checklist for the design and development of new high quality health apps.

The user interface is an influential element in software applications. A well-designed user interface will potentially increase the usability of software applications. Therefore, user interface designers should deliberate when considering which approach and method to implement for designing user interfaces. Design thinking is currently a widely followed approach in user interface design practices. Hence, this study aimed to explore research trends and current practices of design thinking approach for user interface design. This study employed a systematic literature review following the Kitchenham method. This study found 39 articles deemed relevant to the design thinking approach. In general, our study found five common stages broadly mentioned in the articles, i.e., empathize, define, ideate, prototype, and test. The most widely practiced method during those five stages is interview, user persona, brainstorming, user interface, and usability testing. However, there is no consensus on what kind of stage(s) and which method(s) should be employed when following the design thinking approach for user interface design. Therefore, it will depend on the designer's decision in choosing which stage(s) and their related method(s) will be employed.

Boundary scan provides test engineers with a powerful tool for detecting both board-level and chip-level faults. The benefits of boundary scan can be diminished, however, when commonly used design and test generation tools are tied to a specific implementation (standard) of boundary scan. By requiring a specific implementation, test tools may limit a designer's ability to design an optimal product. A higher level standard is required which abstracts the concept of accessibility to device I/O from specific implementation details. Abstracting information required for tool developers from implementation details should allow tools to become more flexible, enabling designers to optimize their specific boundary scan implementation without losing the benefit of the tools themselves.

Software programs continue to evolve throughout their lifetime. Maintenance of such evolving programs, including regression testing, is one of the most expensive activities in software development. We present an approach and its implementation called DiffGen for automated regression unit-test generation and checking for Java programs. Given two versions of a Java class, our approach instruments the code by adding new branches such that if these branches can be covered by a test generation tool, behavioral differences between the two class versions are exposed. DiffGen then uses a coverage-based test generation tool to generate test inputs for covering the added branches to expose behavioral differences. We have evaluated DiffGen on finding behavioral differences between 21 classes and their versions. Experimental results show that our approach can effectively expose many behavioral differences that cannot be exposed by state-of-the-art techniques.