Quasi-Random Testing

A timely and complete test is an important factor to assure the functionality and safety of a railway signal system before it is put into service. With the development of rail transportation in China, the traditional semiautomatic test methods cannot satisfy the timely and complete test requirements any longer. This paper proposes an automated model-based test method. First, colored Petri nets are used as a formal language to describe the system specification. Second, the all paths covered optimal algorithm and the sequence priority selected algorithm are proposed to generate the test cases and sequences automatically. Third, taking a typical radio block center (RBC) handover scenario as an example, the generated test cases and sequences are applied into the RBC functionality test platform. The testing result validated the feasibility and efficiency of the proposed automated test method. Compared with the random-walk-based test sequence generation algorithm, the repeatability rate of the generated test sequences is reduced by 46%. The test sequences can cover all the generated test cases, and the cases can cover all the related criteria in the function requirements specification of the Chinese Train Control System Level 3.

Monte Carlo (MC) methods estimate high-dimensional integrals by computing sample averages at independent and identically distributed (IID) random points.Quasi-Monte Carlo (QMC) methods replace IID samples with low-discrepancy (LD) sequences which more uniformly cover the integration domain, leading to faster convergence and reduced computational requirements.Figure 1 visualizes IID and LD sequences.

The importance of testing software has been increasingly acknowledged in recent years. Different approaches have been investigated and reported upon. One method which has aroused considerable discussion is Random Testing (RT), a method of testing software which relies on a random selection of test cases (collections of inputs representing a single use of the software being tested). Among the advantages of RT are its ease of use, the minimal overheads in test case generation, and the statistical support available. Research has indicated that failure patterns (portions of an input domain which, when executed, cause the program to fail or reveal an error) can influence the effectiveness of some testing strategies. For certain types of failure patterns, it has been found that a widespread and even distribution of test cases in the input domain can be significantly more effective at detecting failure than ordinary RT. Testing methods based on RT, but which aim to achieve even and widespread distributions, have been called Adaptive Random Testing (ART) strategies. In this thesis, the background and motivations behind ART are explained. Because there are many ways to achieve a widespread and even distribution of test cases, there are many possible implementations of ART. Several major ART implementations are examined in detail, and various computational cost reduction strategies and enhancements are presented. The methods presented are analysed mathematically and empirically. The methods are applied to simulations and fault-seeded programs, and their performance in terms of the number of test cases required to find a failure (a metric introduced as the F-measure) is examined.

In this paper we describe a third class of low discrepancy sequences. Using a lattice Γ ⊂ ℝs, we construct Kronecker-like and van der Corput-like ergodic transformations T1,Γ and T2,Γ of [0, 1)s. We prove that for admissible lattices Γ, (Tν,Γn(x))n≥0 is a low discrepancy sequence for all x ∈ [0, 1)s and ν ∈ {1, 2}. We also prove that for an arbitrary polyhedron P ⊂ [0, 1)s, for almost all lattices Γ ∈ Ls = SL(s,ℝ)/SL(s, ℤ) (in the sense of the invariant measure on Ls), the following asymptotic formula $$\# \{ 0 \le n < N:T_{v,\Gamma }^n(x) \in P\} = NvolP + O({(\ln N)^{s + \varepsilon }}),N \to \infty$$ holds with arbitrary small ɛ > 0, for all x ∈ [0, 1)s, and ν ∈ {1, 2}.

Adaptive Random Testing (ART) has been proposed as an enhancement to random testing, based on assumptions on how failing test cases are distributed in the input domain. The main assumption is that failing test cases are usually grouped into contiguous regions. Many papers have been published in which ART has been described as an effective alternative to random testing when using the average number of test case executions needed to find a failure (F-measure). But all the work in the literature is based either on simulations or case studies with unreasonably high failure rates. In this paper, we report on the largest empirical analysis of ART in the literature, in which 3727 mutated programs and nearly ten trillion test cases were used. Results show that ART is highly inefficient even on trivial problems when accounting for distance calculations among test cases, to an extent that probably prevents its practical use in most situations. For example, on the infamous Triangle Classification program, random testing finds failures in few milliseconds whereas ART execution time is prohibitive. Even when assuming a small, fixed size test set and looking at the probability of failure (P-measure), ART only fares slightly better than random testing, which is not sufficient to make it applicable in realistic conditions. We provide precise explanations of this phenomenon based on rigorous empirical analyses. For the simpler case of single-dimension input domains, we also perform formal analyses to support our claim that ART is of little use in most situations, unless drastic enhancements are developed. Such analyses help us explain some of the empirical results and identify the components of ART that need to be improved to make it a viable option in practice.

This paper considers the problem of integration testing the components of a timed distributed software system. We assume that communication between the components is specified using timed interface automata and use computational tree logic (CTL) to define communication-based coverage criteria that refer to send- and receive-statements and communication paths. The proposed method enables testers to focus during component integration on such parts of the specification, e.g. behaviour specifications or Markovian usage models, that are involved in the communication between components to be integrated. A more specific application area of this approach is the integration of test-models, e.g. a transmission gear can be tested based on separated models for the driver behaviour, the engine condition, and the mechanical and hydraulical transmission states. Given such a state-based specification of a distributed system and a concrete coverage goal, a model checker is used in order to determine the coverage or generate test sequences that achieve the goal. Given the generated test sequences we derive a partial test-model of the components from which the test sequences were derived. The partial model can be used to drive further testing and can also be used as the basis for producing additional partial models in incremental integration testing. While the process of deriving the test sequences could suffer from a combinatorial explosion, the effort required to generate the partial model is polynomial in the number of test sequences and their length. Thus, where it is not feasible to produce test sequences that achieve a given type of coverage it is still possible to produce a partial model on the basis of test sequences generated to achieve some other criterion. As a result, the process of generating a partial model has the potential to scale to large industrial software systems. While a particular model checker, UPPAAL, was used, it should be relatively straightforward to adapt the approach for use with other CTL based model checkers. A potential additional benefit of the approach is that it provides a visual description of the state-based testing of distributed systems, which may be beneficial in other contexts such as education and comprehension.

Fault Diagnosis of a Batch of Microprocessors

Transforming low-discrepancy sequences from a cube to a simplex

Sparse arrays have grating lobes in the far field pattern due to the large spacing of elements residing in a rectangular or triangular grid. Random element spacing removes the grating lobes but produces large variations in element density across the aperture. In fact, some areas are so dense that the elements overlap. This paper introduces a low discrepancy sequence (LDS) for generating the element locations in sparse planar arrays without grating lobes. This nonrandom alternative finds an element layout that reduces the grating lobes while keeping the elements far enough apart for practical construction. Our studies consider uniform sparse LDS arrays with 86% less elements than a fully populated array, and numerical results are presented that show these sampling techniques are capable of completely removing the grating lobes of sparse arrays. We present the mathematical formulation for implementing an LDS generated element lattice for sparse planar arrays, and present numerical results on their performance. Multiple array configurations are studied, and we show that these LDS techniques are not impacted by the type/shape of the planar array. Moreover, in comparison between the LDS techniques, we show that the Poisson disk sampling technique outperforms all other approaches and is the recommended LDS technique for sparse arrays.

적응적 랜덤 테스팅 (Adaptive Random Testing, ART)은 입력 도메인 내에 테스트 케이스를 넓고 고르게 분산시키는 방법을 통해 입력 도메인 내에 존재하는 오류 패턴을 순수 랜덤 테스팅 (Random Testing, RT)보다 효율적으로 찾아내기 위한 테스트 케이스 선택 기법이다. 테스트 케이스 선택에 많은 연산량을 필요로 하는 초기 ART 기법인 거리 기반 ART (Distance-based ART, D-ART)와 제한 영역 기반 ART (Restricted Random Testing, RRT)의 개선을 위해 입력 도메인을 반복 분할하는 기법들이 제안되었고, 이 기법들은 낮은 연산량 및 성능 향상등의 효과를 가져왔다. 하지만, 입력 도메인 반복 분할 기반 기법에서도 기존 ART 기법에서 나타나는 테스트 케이스 분포 불균일 문제가 존재하고, 이는 기법의 확장성에 장애 요소로 작용한다. 따라서 본 논문에서는 반복 분할 기반 기법에서 나타나는 테스트 케이스 분포의 특성을 파악하고, 이를 적정 수준으로 제어하기 위한 입력 도메인 확장 정책을 제안하였으며, 실험을 통해 2차원 입력 도메인에서 3%, 3차원 입력 도메인에서 10% 수준의 성능 향상을 확인하였다. An Adaptive Random Testing(ART) is one of test case generation algorithms, which was designed to get better performance in terms of fault-detection capability than that of Random Testing(RT) algorithm by locating test cases in evenly spreaded area. Two ART algorithms, such as Distance-based ART(D-ART) and Restricted Random Testing(RRT), had been indicated that they have significant drawbacks in computations, i.e., consuming quadratic order of runtime. To reduce the amount of computations of D-ART and RRT, iterative partitioning of input domain strategy was proposed. They achieved, to some extent, the moderate computation cost with relatively high performance of fault detection. Those algorithms, however, have yet the patterns of non-uniform distribution in test cases, which obstructs the scalability. In this paper we analyze the distribution of test cases in an iterative partitioning strategy, and propose a new method of input domain enlargement which makes the test cases get much evenly distributed. The simulation results show that the proposed one has about 3 percent of improvement in terms of mean relative F-measure for 2-dimension input domain, and shows 10 percent improvement for 3-dimension space.

The authors define the notion of a lingering synchronisation effect. Such an effect occurs when a primary input cube (an incompletely specified primary input vector) determines the state of a circuit for several time units after it is applied. A primary input cube with a lingering synchronisation effect may prevent certain faults from being detected when it appears repeatedly in a test sequence. It should therefore be avoided when the goal is to achieve a high fault coverage. The authors demonstrate that benchmark circuits have primary input cubes with small numbers of specified values (typically one or two), which have lingering synchronisation effects. In some cases, the synchronisation effects linger for large numbers of time units. The authors define a ranking of primary input cubes based on the severity of their lingering synchronisation effects. They describe a random test generation process that avoids primary input cubes with lingering synchronisation effects, and achieves high fault coverage for benchmark circuits. The test generation process uses the severity of the lingering synchronisation effects of the primary input cubes to decide on the ones it should avoid.

Random testing (RT) is a fundamental testing technique to assess software reliability, by simply selecting test cases in a random manner from the whole input domain. As an enhancement of RT, adaptive random testing (ART) has better failure-detection capability and has been widely applied in different scenarios, such as numerical programs, some object-oriented programs, and mobile applications. However, not much work has been done on the effectiveness of ART for the programs with combinatorial input domain (i.e., the set of categorical data). To extend the ideas to the testing for combinatorial input domain, we have adopted different similarity measures that are widely used for categorical data in data mining and have proposed two similarity measures based on interaction coverage. Then, we propose a new version named ART-CID as an extension of ART in combinatorial input domain, which selects an element from categorical data as the next test case such that it has the lowest similarity against already generated test cases. Experimental results show that ART-CID generally performs better than RT, with respect to different evaluation metrics.

A general concept for parallelizing quasi-Monte Carlo methods is introduced. By considering the distribution of computing jobs across a multiprocessor as an additional problem dimension, the straightforward application of quasi-Monte Carlo methods implies parallelization. The approach in fact partitions a single low-discrepancy sequence into multiple low-discrepancy sequences. This allows for adaptive parallel processing without synchronization, i.e. communication is required only once for the final reduction of the partial results. Independent of the number of processors, the resulting algorithms are deterministic, and generalize and improve upon previous approaches.

Quasi-Monte Carlo integration uses deterministic, low discrepancy sequences, such as the Halton or the Sobol sequence, to carry out numerical integration or search. These quasi random sequences are defined over the unit interval, or its multidimensional analogues, i.e. the unit square, cube, or hypercube. However, there are applications where we need quasi random sequences that are defined over a circular, spherical, or ellipsoidal domains. We study the performance of quasi random sequences for integration on a unit disk by generating quasi-random sequences defined over the unit square and transforming them to sequences defined on the unit disk by using coordinate transformations. We evaluate integrals defined over a circular disk and find that quasi-Monte Carlo integration using Sobol sequences outperforms Monte Carlo integration using pseudorandom sequences.

Adaptive random testing (ART) is a family of algorithms to enhance random testing (RT) by generating test cases extensively and evenly. For this purpose, many ART algorithms have been proposed, the most well-known and the first approach is the Fixed-Size-Candidate-Set ART (FSCS-ART). In recent years, researchers have also proposed many ART methods to continuously improve the performance of FSCS-ART, but the focus has been more on reducing the time overhead of FSCSART while retaining its failure detection effectiveness as much as possible due to the boundary effect. To alleviate the boundary effect and improve the effectiveness of FSCS-ART, this paper proposes an algorithm AP-FSCS-ART, an Adaptive Partition-based method on top of FSCS-ART. First, AP-FSCS-ART divides the entire input domain into external and internal sub-domains. Then, two different algorithms are adaptively applied to the two sub-domains to find the next test case from the randomly generated candidate test cases. During the selecting process, APFSCS-ART takes into account not only the most recently executed test case of a candidate test case but also its position relative to the input domain. Experiments using the 12 most common real programs and comparisons with other algorithms in this paper show that the AP-FSCS-ART algorithm has significantly better failure detection capability, with improvements from 8.8% to 11.4% compared to three state-of-the-art ART algorithms, including the FSCS-ART, FSCS-ctsr, and NNDC-ART.