- Research Article
- 10.1002/stvr.443
Editorial: Agility must be good for testing
- Sep 01, 2010
- Software Testing, Verification and Reliability
- Jeff Offutt
Editorial: Agility must be good for testing
In spite of cultural difference between the corresponding scientific communities, recognition is growing that test-based and specification- based approaches to software development actually complement each other. The revival of interest in testing tools and techniques follows in particular from the popularity of Development; rigorous specification and proofs have, for their part, also made considerable progress. There remains, however, a fundamental superiority of specifications over test: you can derive tests from a specification, but not the other way around. Contract-Driven Development is a new approach to systematic software construction combining ideas from Design by Contract, from Test-Driven Development, from work on formal methods, and from advances in automatic testing as illustrated for example in our AutoTest tool. Like TDD it gives tests a central role in the development process, but these tests are deduced from possibly partial specifications (contracts) and directly supported by the development environment. This talk will explain the concepts and demonstrate their application.
Editorial: Agility must be good for testing
Editorial: Agility must be good for testing
Taming complexity while gaining efficiency: Requirements for the next generation of test automation tools
The current product development environment for mechatronic systems is characterized by tight budgets, reduced development times and immense complexity across the industries - aerospace, automotive, commercial vehicles, etc. A huge emphasis is being given to increase the efficiency in product testing and to make the process more productive in increasing quality, while being cost effective. Companies are seeking software testing tools that offer a comprehensive solution that helps in achieving this goal. Automated software testing for both hardware and the software components is one of the ways the industries are trying to gain efficiency in testing. Given the fact that current mechatronic systems are quite complex, have large numbers and types of IO channels, and are implementing distributed real-time control with multiple communication protocols, a test automation tool that incorporates and tests all of the system functionality is highly desirable. In particular, due to the focus on software quality, as more and more functions are added to the network of Electronic Control Units (ECU), a greater emphasis is laid on the requirements for a comprehensive automated testing tool. The efficiency in development and planned increase in reuse of test assets is an important consideration for efficiency gain. Bringing clarity to the test description would improve readability and maintainability of these test assets. This paper shows the lessons learned from a similar evolution in Model-Based Design, moving from handwritten code to model-based graphical executable representation, can be applied to the testing arena to develop better and more efficient tests. To make the testing process efficient, the test tool should be able to not just execute tests, but organize the test results and present them in a manner easily understood by the end user. The user should be easily able to select tests to be executed and visualize execution results of those tests without need of external data analysis and reporting tools. A user should also be able to gather data related to test execution for reporting on overall software quality metrics. In the case of real-time embedded systems, the traditional approach for software testing of test execution, data capture, analysis and reporting creates a lot of inefficiency and repeated testing. A new approach to run test evaluation in parallel to test execution, using the so-called real-time testing solution, eliminates these inefficiencies and results in significant gains in testing time and improving utilization of the test infrastructure. Globalized, distributed development environments may lead to organizations adopting a variety of tool chains and test platforms. Therefore, to gain efficiency, it is desired that the test automation solution should offer easy integration with other third-party test platforms and be independent of the hardware to protect the investment in the long run. Basing such integration interface on established standards gives the user freedom to make choices. Some standards, such as HIL API from ASAM, are being proposed for test systems in the automotive industry. The paper reviews applicability and benefits of such standards. Various industry standards, such as automotive functional safety standard ISO 26262, IEC 61508, IEC 62304 and DO-178C, have addressed requirements for software testing and test tools. It is therefore critical that the selected test automation tool provide support for the prescribed methods and support the standard compliant development process. Tool qualification is a critical topic addressed by these standards and is discussed in this paper. Further, increasingly important requirements such as integration with the overall software development process, management of tests and test data, test reusability, hardware platform independence, test development and execution efficiency, etc. are discussed.
Read moreQuality
High quality assurance is a fundamental element of every engineering process and is considered to be one of the more difficult things to achieve and sustain. Since high quality is also a basic concern of software engineering, there are values and practices that support the assurance of high quality software products and processes. However, these are not sufficient, and in many cases, software products lack the required quality. In this chapter, among other issues, we analyze why this happens. We also describe how quality is perceived using the agile approach, starting withvalues values and practicespractices that support and control the process quality, such as customer collaboration and the planning of a typical agile software development process. We continue with values and practices that support the product quality, for example, refactoring and the feedback gained by exhaustive testing and test automation. Finally, we focus on the test driven development Test Driven Development (TDD)TDD practice, analyze its acceptance by software developers and present a way to measure and control TDD processes.KeywordsSoftware DevelopmentSoftware ProjectAcceptance TestSoftware Development ProcessQuality QualityThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Read moreDoes the Performance of TDD Hold Across Software Companies and Premises? A Group of Industrial Experiments on TDD
Test-Driven Development (TDD) has been claimed to increase external software quality. However, the extent to which TDD increases external quality has been seldom studied in industrial experiments. We conduct four industrial experiments in two different companies to evaluate the performance of TDD on external quality. We study whether the performance of TDD holds across the premises of the same company and across companies. We identify participant-level characteristics impacting results. Iterative-Test Last (ITL), the reverse approach of TDD, outperforms TDD in three out of four premises. ITL outperforms TDD in both companies. The larger the experience with unit testing and testing tools, the larger the difference in performance between ITL and TDD (in favour of ITL). Technological environment (i.e., programming language and testing tool) seems not to impact results. Evaluating participant-level characteristics impacting results in industrial experiments may ease the understanding of TDD’s performance in realistic settings.
Read moreFormal Specification and Verification of Complex Systems
Formal Specification and Verification of Complex Systems
Decision Support for Selecting Tools for Software Test Automation
Context: Test automation is an investment having a high initial economic impact on software development. Utilization of test automation may positively affect the costs (e.g. by speeding up development iterations by providing repeatable tests and regression testing) and the quality of software or system, in large scale. Approaches to test automation may not always be appropriate or successful. The trade-off between manual and automated testing and the tools to be used have to be identified and justified. The task to decide which tools to use, to maximize the benefits is not a trivial one. There are numerous software testing or software test automation tools available, both commercial and open source and unique, multifaceted goals in every development environment (context). The exact number of tools is unknown and chances or resources to try out different choices are very limited. Objective: Contextual factors are acknowledged as an issue and well known and common to both practitioners in the field and consultation service providers. Selecting and utilizing the most effective and efficient tool(s) for specific purpose(s) in a specific context is essential for the success of business. The goal of the research is to define a systematic, empirically validated decision support system (DSS) for selecting a tool for software test automation
Read moreAnalyzing the effects of test driven development in GitHub
Testing is an integral part of the software development lifecycle, approached with varying degrees of rigor by different process models. Agile process models recommend Test Driven Development (TDD) as a key practice for reducing costs and improving code quality. The objective of this work is to perform a cost-benefit analysis of this practice. Previous work by Fucci et al. [2, 3] engaged in laboratory studies of developers actively engaged in test-driven development practices. Fucci et al. found little difference between test-first behaviour of TDD and test-later behaviour. To that end, we opted to conduct a study about TDD behaviours in the "wild" rather than in the laboratory. Thus we have conducted a comparative analysis of GitHub repositories that adopts TDD to a lesser or greater extent, in order to determine how TDD affects software development productivity and software quality. We classified GitHub repositories archived in 2015 in terms of how rigorously they practiced TDD, thus creating a TDD spectrum. We then matched and compared various subsets of these repositories on this TDD spectrum with control sets of equal size. The control sets were samples from all GitHub repositories that matched certain characteristics, and that contained at least one test file. We compared how the TDD sets differed from the control sets on the following characteristics: number of test files, average commit velocity, number of bug-referencing commits, number of issues recorded, usage of continuous integration, number of pull requests, and distribution of commits per author. We found that Java TDD projects were relatively rare. In addition, there were very few significant differences in any of the metrics we used to compare TDD-like and non-TDD projects; therefore, our results do not reveal any observable benefits from using TDD.
Read moreMoFQA: A TDD Process and Tool for Automatic Test Case Generation from MDD Models
Techniques for quality assurance have to deal with the complexity of software systems and the high probabilities of new errors appearing in any stage of the software life cycle. Software testing is a widely used approach but, due to the costs involved in this process, development teams often debate its applicability in their projects. In the endeavor to reduce the complexity of this process, this study presents an approach for software development based in Test-Driven Development (TDD) supported by Model-Based Testing (MBT) tools to allow automatic test-case generation. The approach, called MoFQA (Model-First Quality Assurance), consists of two main aspects: i) a method to drive software development based on testing techniques which defines steps and recom-mended practices; and, ii) a tool-set to allow clients and stakeholders to model system requirements, testers to create models that represent unit and abstract tests, and transformation tools used to generate executable tests. The tools that MoFQA provides are focusing on web applications. In order to validate the MoFQA tools, two validation experiences are presented.
Read moreUsing spring Roo for the test-driven development of Web applications
Software should be developed in a way that achieves high external and internal quality. Most programs have to be maintained or are extended. They should not only adhere to their requirements but allow to be efficiently changed. Model-driven software development (MDSD) and test-driven development (TDD) are techniques that help to satisfy this demand. We present a novel approach combining MDSD and TDD while using the Spring Roo Web development framework. Besides discussing existing work and the background, we sketch a test-driven Web development process. We then evaluate its feasibility in an exemplary scenario. Thereby, we contribute to the body of scientific knowledge on Web development frameworks in the field of MDSD and TDD.
Read moreUnit Testing in ASP Revisited: Language and Test-Driven Development Environment
Unit testing frameworks are nowadays considered a best practice, included in almost all modern software development processes, to achieve rapid development of correct specifications. Knowledge representation and reasoning paradigms such as Answer Set Programming (ASP), that have been used in industry-level applications, are not an exception. Indeed, the first unit testing specification language for ASP was proposed in 2011 as a feature of the ASPIDE development environment. Later, a more portable unit testing language was included in the LANA annotation language. In this paper we revisit both languages and tools for unit testing in ASP. We propose a new unit test specification language that allows one to inline tests within ASP programs, and we identify the computational complexity of the tasks associated with checking the various program-correctness assertions. Test-case specifications are transparent to the traditional evaluation, but can be interpreted by a specific testing tool. Thus, we present a novel environment supporting test-driven development of ASP programs.
Read moreA conceptual model supporting decision-making for test automation in Agile-based Software Development
A conceptual model supporting decision-making for test automation in Agile-based Software Development
A Comprehensive Review of Automated Software Testing Tools and Techniques
Automation testing is often needed in the software testing process to improve test results and save time and money. Automation testing is very useful for evaluating web applications based on load, stress, security, vulnerability, and performance. The many studies on automated software testing methods and tools are reviewed in this article. This comprehensive review highlights that no single automated testing tool excels in all aspects; the selection depends on project-specific criteria such as cost, usability, support, and test type. Tools like Selenium and WATIR offer flexibility and cost-efficiency for web applications, while UFT/QTP supports both web and desktop testing but at a higher cost. The rise of AI-powered, self-repairing frameworks signals a shift toward more adaptive testing solutions. Additionally, parallel execution methods significantly enhance regression testing efficiency. Ultimately, selecting the right tool and execution strategy is crucial to minimizing time and cost in software development.
Read moreTowards empirical evaluation of test-driven development in a university environment
Test driven development (TDD) is an agile software development technique and it is one of the core development practices of Extreme programming (XP). In TDD, developers write automatically executable tests prior to writing the code they test. We ran a set of experiments to empirically assess different parameters of the TDD. We compared TDD to a more "traditionally" oriented iterative test-last development process (ITL). Our preliminary results show that TDD is not substantially different from ITL and our qualitative findings about a development process are different from results obtained from other researches.
Read moreThe Marine Derrick Test Tool Design and Optimization
In this paper, aim the circumstances of derrick and substructure for a specific offshore platform, two derrick test tools be devised, The substructure and the test tool model established and assembled by Workbench software, and through the ANSYS finite element analysis software calculated the substructure and the test tool, then based on the analysis in contact with the forces condition of the test tool that were optimized. By comparing the test tools can be found, the test tooling option II that the weight, the forces condition and operation is better than another one, so when the conditions to meet the circumstances, option II is recommended to use. The design ideas and methods for test tool design and manufacturing provides a useful reference.
Read moreAnalyzing the effects of test driven development in GitHub
Testing is an integral part of the software development lifecycle, approached with varying degrees of rigor by different process models. Agile process models recommend Test Driven Development (TDD) as a key practice for reducing costs and improving code quality. The objective of this work is to perform a cost-benefit analysis of this practice. To that end, we have conducted a comparative analysis of GitHub repositories that adopts TDD to a lesser or greater extent, in order to determine how TDD affects software development productivity and software quality. We classified GitHub repositories archived in 2015 in terms of how rigorously they practiced TDD, thus creating a TDD spectrum. We then matched and compared various subsets of these repositories on this TDD spectrum with control sets of equal size. The control sets were samples from all GitHub repositories that matched certain characteristics, and that contained at least one test file. We compared how the TDD sets differed from the control sets on the following characteristics: number of test files, average commit velocity, number of bug-referencing commits, number of issues recorded, usage of continuous integration, number of pull requests, and distribution of commits per author. We found that Java TDD projects were relatively rare. In addition, there were very few significant differences in any of the metrics we used to compare TDD-like and non-TDD projects; therefore, our results do not reveal any observable benefits from using TDD.
Read more