Developing multithreaded techniques and improved constraint handling for the tool CAgen

Many industrial and academic optimization problems have successfully been tackled with ant colony optimization (ACO), but the ability of ACO to deal with hard constraints has not yet been investigated widely. Most real-world applications, such as fleet routing, rostering and timetabling, however, are subject to hard constraints. Extensions of ACO with hard constraint handling are thus a worthwhile field of study. Constraint handling in meta-heuristics has mostly been studied in the context of evolutionary algorithms. Many of the techniques proposed are penalty-based. A difficulty with such methods is the correct adjustment of penalty factors. Recently a different approach, termed stochastic ranking (SR), has been suggested which does not use a penalty formulation. Benchmarks indicate that SR is currently among the best performing methods for evolutionary constraint optimization. We discuss how the ideas underlying SR can be used in ACO and evaluate two different ways to achieve this integration: one derived from ACS (ACSsr) and one based on ranked ant system (ASRsr). We compare the performance of these to penalty-based ACO as well as to CPACS, an ACO algorithm hybridized with explicit constraint propagation. Initial benchmark results are encouraging.

PHP frameworks are widely used in web application development because they speed up the coding process and simplify application management. Among the popular frameworks, CodeIgniter and Laravel are often the top choices. This study aims to compare the performance of the two frameworks through literature studies from various relevant sources. The method used is a literature study of previous references, which is analyzed based on five main aspects: basic performance, ease of development, database management, security, and features and scalability. Data was obtained from official documentation, benchmark results, and scientific publications that discuss both frameworks. The analysis results show that CodeIgniter excels in execution speed and simplicity and is suitable for lightweight applications and novice developers. On the other hand, Laravel stands out for its modern features, complex database management, and more integrated security systems, making it better suited for large-scale and complex applications. The conclusion of this study shows that the selection of a framework should be adjusted to the needs of the project, the complexity of the system, and the development team's ability. This research is expected to guide developers in determining the right framework choice and encouraging further study based on direct experimental tests for more applicable results.

Despite recent progress in automated test generation research, significant challenges remain for applying these techniques on large-scale software systems. These systems under test often require structurally complex test inputs within a large input domain. It is challenging to automatically generate a reasonable number of tests that are both legal and behaviorally-diverse to exercise these systems. Constraint-based test generation is an effective approach for generating structurally complex inputs for systematic testing. While this approach can typically generate large numbers of tests, it has limited scalability – tests generated are usually only up to a small bound on input size. Combinatorial test generation, e.g., pair-wise testing, is a more scalable approach but is challenging to apply on commercial software systems that require complex input structures that cannot be formed by using arbitrary combinations. This paper introduces comKorat, which unifies constraint-based generation of structurally complex tests with combinatorial testing. Specifically, comKorat integrates Korat and ACTS test generators to generate test suites for large scale software systems with structurally complex test inputs. We have successfully applied comKorat on four software applications developed at eBay and Yahoo!. The experimental results show that comKorat outperforms existing solutions in execution time and test coverage. Furthermore, comKorat found a total of 59 previously unknown bugs in the four applications.

<p class="0abstractCxSpFirst">The number of applications incorporating Internet of Things (IoT) concepts increases extraordinarily. This increase cannot continue without high-quality assurance. There are some difficulties in testing IoT applications; the system heterogeneity, test cost and time are taken to test the system, and the precision percentage of test results.</p><p class="0abstractCxSpLast">A well-known and possibly the best solution to overcoming these difficulties is to model the system for evaluation purposes, known as model-based testing (MBT). This paper aims to design a model-based testing approach to assess IoT applications performance quality attributes. The ISO 25000 quality model is used as a standard for software quality assurance applications. IoTMaaS is used as a case study to implement the methodological approach. The possible test cases were generated using the ACTS combinatorial test generation tool. The performance metrics of each test case were monitored until the optimum test case was identified, with the shortest response time and the least amount of resources used. The proposed testing method appears to be successful, according to the results.</p>

Adapting automated test generation to GUI testing of industry applications

New tools for combinatorial test generation are proposed every year. However, different generators may have different performances on different models, in terms of the number of tests produced and generation time, so the choice of which generator has to be used can be challenging. Classical comparison between CIT generators considers only the number of tests composing the test suite. Still, especially when the time dedicated to testing activity is limited, generation time can be determinant. Thus, we propose a benchmarking framework including 1) a set of generic benchmark models, 2) an interface to easily integrate new generators, 3) methods to benchmark each generator against the others and to check validity and completeness. We have tested the proposed environment using five different generators (ACTS, CAgen, CASA, Medici, and PICT), comparing the obtained results in terms of the number of test cases and generation times, errors, completeness, and validity. Finally, we propose a CIT competition, between combinatorial generators, based on our framework.

Numerous combinatorial testing tools are available for generating test cases. However, many of them are never used in practice. One of the reasons is the lack of empirical studies that involve human subjects applying testing techniques. This paper aims to investigate the applicability of a combinatorial testing tool in the company SOFTEAM. A case study is designed and conducted within the development team responsible for a new product. The participants consist of 3 practitioners from the company. The applicability of the tool has been examined in terms of efficiency, effectiveness and learning effort.

In this paper we describe how the test design can be done by using the Combinatorial Testing approach for internet of things operating systems. Contiki operating system is taken as a case study but we discuss what can be the approach for RIOT and Tiny OS operating systems. We discuss how the combinatorial coverage measurement can be gathered in addition to the traditional metrics code coverage. The test design generated by using Advanced Combinatorial Testing for Software is analyzed for Contiki operating system. We elaborate the code coverage gathering technique for Contiki simulator which happens to be in Java. We explain the usage of Combinatorial Coverage Measurement tool. Although we have explained the test design methodology for internet of things operating systems, the approach explained can be followed for other open source software.

Activity theory is commonly used in designing Human-device interaction. Donald Norman in his paper Human-Centered Design Considered Harmful criticized traditional approach, which he called Human-Centered Design, and introduced a new approach, called Activity-Centered Design. In this paper his new approach is directly used and widened in order to be used with a tool for test generation. It is used as a point of view, a philosophy, on which graphical user interfaces are designed. Workflow paradigm is used for sequential flow of elements, called activities. Activity theory is directly implemented on Workflow elements by splitting the work on: Activities, Actions and Operations. Combination of Activity theory and Workflow is used as start point for broad range of Human computer interaction, and in this paper it is used as upgrade of test generation framework for TV and STB testing called BBT (Black Box Testing). Goal of this approach is to simplify test script creation process for user with modest or none programming experience.

User Interface (UI) testing is a popular approach to ensure the quality of mobile apps. Numerous test generation tools have been developed to support UI testing on mobile apps, especially for Android apps. Previous work evaluates and compares different test generation tools using only relatively simple open-source apps, while real-world industrial apps tend to have more complex functionalities and implementations. There is no direct comparison among test generation tools with regard to effectiveness and ease-of-use on these industrial apps. To address such limitation, we study existing state-of-the-art or state-of-the-practice test generation tools on 68 widely-used industrial apps. We directly compare the tools with regard to code coverage and fault-detection ability. According to our results, Monkey, a state-of-the-practice tool from Google, achieves the highest method coverage on 22 of 41 apps whose method coverage data can be obtained. Of all 68 apps under study, Monkey also achieves the highest activity coverage on 35 apps, while Stoat, a state-of-the-art tool, is able to trigger the highest number of unique crashes on 23 apps. By analyzing the experimental results, we provide suggestions for combining different test generation tools to achieve better performance. We also report our experience in applying these tools to industrial apps under study. Our study results give insights on how Android UI test generation tools could be improved to better handle complex industrial apps.

In this paper, we present methods for automatic test generation from formal specifications. These are used to create intelligent test benches that are able to cover corner case behaviors in much less time. We have developed a prototype tool for intelligent test generation within the layered test bench architecture proposed in RVM. We present results on verification IPs of standard bus protocols to show the effectiveness of our approach.

According to a recent synthesis for testability proposal, a test function specified as a finite state machine with the same number of state variables as the given object machine, is incorporated into the state diagram prior to synthesis. Since a complete verification of the test machine is not practical, an often used heuristic sets and observes each state variable. The two machines share logic and a fault can result in partial or total loss of the test function. We show that the tests generated under the assumption that the entire test function is intact can become invalid. We propose a new method of synthesizing PLA-based finite state machines with fault tolerant test machines. Our approach eliminates testing of the test function. A constrained logic minimization phase insures that faults have predictable effect on the state diagram of the composite machine (object machine embedded with the test function). This allows effective use of the test function during test generation even in the presence of faults that effect both object and test machines. Only a combinational test generator is required for test generation. Each combinational vector is augmented by appropriate initialization and propagation sequences. Unlike prior approaches, ourO(log2 n) length test sequence isguaranteed to detect any targeted crosspoint fault. Experimental results on the MCNC Logic Synthesis Workshop finite state machine benchmark set are given as evidence of practicality of the proposed approach.

Verification tool for Vienna Development Method (BWDM) is a test case generation tool for the VDM++ specification. The existing BWDM could cause a combinatorial explosion of the generated test cases. To reduce the number of test cases, there is Pairwise Independent Combinatorial Testing Tool (PICT): a pairwise testing tool. We apply pairwise testing into BWDM. Here, BWDM cannot call PICT library directly. Hence, we have developed PICT-wrapper. It is an interface to connect PICT and BWDM. The extended BWDM eliminate the possibility of the combinatorial explosion.

Full scan designs are widely used for their indisputable benefits of predictably high test coverage, diagnosis and debug. However, for high-performance designs the cost of scan - area and delay - is not acceptable and partial scan is used instead. Unfortunately, partial scan significantly increases test generation complexity. We define a structured partial scan design methodology and specific test generation enhancements, which significantly enhance test coverage and reduce test data and cycles. Selective design areas use special types of nonscan cells which can capture a value in the last few scan load cycles. Combinational test generation is extended to work with this structured partial scan design, resulting in higher coverage and fewer patterns. Experimental results on industrial designs show consistent testability benefits.

We present a technique for extracting functional (control/dataflow) information from register transfer level (RTL) controller/data path circuits and illustrate its use in design for hierarchical testability of these circuits. This testing procedure and design for testability (DFT) technique is general enough to handle RTL control flow intensive circuits like protocol handlers as well as data flow intensive circuits like digital filters. It makes the combined controller-data path highly testable and does not require any external behavioral information. This scheme has the advantages of low area/delay/power overheads (average of 3.2%, 0.9% and 4.1%, respectively, for benchmarks), high fault coverage (over 99% for most cases), very low test generation times (because it is independent of bit-width), and the advantage of at-speed testing. Experiments show a 2-to-4 (1-to-3) orders of magnitude test generation time advantage over an efficient gate-level sequential test generator (combinational test generator that assumes full scan).