Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing

We present a technique that generates high-quality functional in-field self-tests specifically targeting deep learning (DL) accelerators. These functional tests can be applied in the field during normal operation of a DL accelerator, which is crucial to ensure that the safety and/or reliability requirements are met for any given application, including safety-critical applications such as self-driving cars, robotics, and more.Our technique takes advantage of special architectural characteristics and application properties to achieve high functional test coverage while incurring minimal system-level costs. Moreover, we devise different strategies for the compute units (which support computation operations) and the control units (which control data movement) because these two types of units exhibit different properties. For the compute units of a DL accelerator, we first use combinational ATPG to generate test patterns with high test coverage, which is possible because these units do not contain complex sequential logic. Next, we map the ATPG patterns to one or more equivalent deep neural networks (DNNs) that can be directly executed on the accelerator, which is possible given the well-defined dataflow/reuse algorithm of a DL accelerator. For the control units, we leverage the property that typically only one or a few fixed DNNs are deployed at a time in many application domains (e.g., self-driving cars). Thus, it is sufficient to target only the faults that can directly affect the correctness of the DNNs that are currently deployed. This is done by executing different layers of each target DNN using carefully-crafted input and weight values to maximize test coverage while minimizing test time.We apply our technique using Nvidia’s open-source accelerator as a case study to demonstrate its efficacy. Our results show that our technique achieves high test coverage. For the compute units, 99.9% single stuck-at functional test coverage is achieved. For the control units, we are able to prove that, given any target DNN, 100% coverage can be achieved for a large class of single and multiple fault models. The in-field functional self-test time is also very low, < 17 ms for various representative DNNs. These functional tests can be applied during boot-up, reset, and even concurrently with normal operation by executing DNN test programs directly on the accelerator, without requiring any test support in the hardware.

Integrated circuits used in critical and high reliability applications have often strict test requirements including high test coverage and limited test time. Achieving a high test coverage using built-in self-test (BIST) has proven difficult. Methods such as test point insertion or deterministic BIST can provide high test coverage but introduce significant area overhead and design effort. In this paper, we propose a computational algorithm that uses a linear XOR model of the logic BIST (LBIST) structure and fault partitioning to extract seeds for partial ATPG patterns. Partial ATPG patterns are used to decrease the complexity of the algorithm when solving linear XOR equations to generate deterministic seeds. The extracted seeds are stored in a nonvolatile memory on- or off-chip. Results show that for most designs, patterns generated from the extracted ATPG seeds are significantly more effective in detecting faults and can achieve higher test coverage than LBIST.

Software testing techniques have advanced significantly over several decades; however, most of current techniques still test a target program as it is, and fail to utilize valuable information of diverse test executions on many variants of the original program in test generation. This paper proposes a new direction for software testing - Invasive Software Testing (IST). IST first generates a set of target program variants m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> from an original target program p by applying mutation operations <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, μ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> . Second, given a test suite T, IST executes m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> , ⋯, m <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> with T and records the test runs which increase test coverage compared to p with T. Based on the recorded information, IST generates guideposts for automated test generation on p toward high test coverage. Finally, IST generates test inputs on p with the guideposts and achieves higher test coverage. We developed DEMINER which implements IST for C programs through software mutation and concolic testing. Further, we showed the effectiveness of DEMINER on three real-world target programs Busybox-ls, Busybox-printf, and GNU-find. The experiment results show that the amount of the improved branch coverage by DEMINER is 24.7% relatively larger than those of the conventional concolic testing techniques on average.

Design for Testability (DFT) based on scan and ATPG has been adopted as a reliable and broadly acceptable methodology that provides very high test coverage, but for large circuits, the growing test data volume causes a significant increase in test cost because of much longer test time and elevated tester memory requirements. Test compression or scan compression provides great reduction in test data volume and test time required by adding on-chip decompressor and compactor. In this paper comparative analysis are made for Broadcast, XOR decompressor along with XOR, MISR and Hybrid compactors with respect to test coverage, test cycles required and test data volume by considering Flash Interface as CUT. From the experiments, it is observed that XOR decompressor with MISR compactor architecture provides 17.31% to 49.76% reduction in test data volume compared to other architectures, with 99.76% of fault coverage, 16694 test cycles and 2104µm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> of area overhead.

With the rapid development of integrated circuit production technology, the scale of FPGA circuits has expanded to billions of gates. The complexity of the internal resource structures in the FPGAs (field programmable gate arrays) is continually increasing, and there is an increasing possibility of various faults in these circuits, especially in interconnect resources. These occupy more than 80% of a chip’s area and have the highest fault rate. To ensure the reliability of the FPGAs, it is very important to perform high-coverage testing on the interconnect resources within them. This article uses AMD Xilinx’s Kintex-7 series FPGA as the research object and proposes a deep-priority algorithm based on graph-based models and improved priority algorithms to intelligently wire the FPGA interconnected resources. The routing results were produced using a configuration script written in the XDL language, and the FPGA configuration and testing were conducted accordingly. This approach achieved a high coverage and intelligent testing for the interconnect resources in the FPGAs.

Test issues in application-specific digital filter datapaths are investigated. It is found that such designs can contain hundreds of redundant faults, making it difficult to accurately determine fault coverage. Since these redundant faults tend to appear in the same general location as test-resistant faults, the presence of many redundant faults can hide significant untested faults despite high overall test coverage. Classes of redundant faults that arise in digital filter datapaths are described, and we propose a suite of techniques for identifying and eliminating the most common redundancies based on arithmetic optimization. The approach is suitable as a front-end to more accurate fault simulation, or can be used in the design process to eliminate redundant logic. The approach is validated as a tool for developing very high-coverage built-in self-test circuits, showing that 100% fault coverage can be achieved in 24k-gate filters with as little as 1% area overhead. When used as a datapath optimization technique, the average area reduction over 15 designs was 8.9%, compared with moderately optimized designs. As a front-end to fault simulation, the approach yielded a 97.9% average reduction in the number of undetected faults across the 15 designs.

Benefits of a Testing Framework in Undergraduate C Programming Courses

Supporting e-Science in the EGI e-Infrastructure requires extensive and reliable software, for advanced computing use, deployed across over approximately 300 European and worldwide data centers. The Unified Middleware Distribution (UMD) and Cloud Middleware Distribution (CMD) are the channels to deliver the software for the EGI e-Infrastructure consumption. The software is compiled, validated and distributed following the Software Provisioning Process (SWPP), where the Quality Criteria (QC) definition sets the minimum quality requirements for EGI acceptance. The growing number of software components currently existing within UMD and CMD distributions hinders the application of the traditional, manual-based validation mechanisms, thus driving the adoption of automated solutions. This paper presents umd-verification, an open-source tool that enforces the fulfillment of the QC requirements in an automated way for the continuous validation of the software products for scientific disposal. The umd-verification tool has been successfully integrated within the SWPP pipeline and is progressively supporting the full validation of the products in the UMD and CMD repositories. While the cost of supporting new products is dependant on the availability of Infrastructure as Code solutions to take over the deployment and high test coverage, the results obtained for the already integrated products are promising, as the time invested in the validation of products has been drastically reduced. Furthermore, automation adoption has brought along benefits for the reliability of the process, such as the removal of human-associated errors or the risk of regression of previously tested functionalities.

Static call graph (CG) construction often over-approximates call relations, leading to sound, but imprecise results. Recent research has explored machine learning (ML)-based CG pruning as a means to enhance precision by eliminating false edges. However, current methods suffer from a limited evaluation dataset, imbalanced training data, and reduced recall, which affects practical downstream analyses. Prior results were also not compared with advanced static CG construction techniques yet. This study tackles these issues. We introduce the NYXCorpus, a dataset of real-world Java programs with high test coverage and we collect traces from test executions and build a ground truth of dynamic CGs. We leverage these CGs to explore conservative pruning strategies during the training and inference of ML-based CG pruners. We conduct a comparative analysis of static CGs generated using zero control flow analysis (0-CFA) and those produced by a context-sensitive 1-CFA algorithm, evaluating both with and without pruning. We find that CG pruning is a difficult task for real-world Java projects and substantial improvements in the CG precision (+25%) meet reduced recall (-9%). However, our experiments show promising results: even when we favor recall over precision by using an F2 metric in our experiments, we can show that pruned CGs have comparable quality to a context-sensitive 1-CFA analysis while being computationally less demanding. Resulting CGs are much smaller (69%), and substantially faster (3.5x speed-up), with virtually unchanged results in our downstream analysis.

Automatic testing, in particular test input generation, has become increasingly popular in the research community over the past ten years. In this paper, we conduct a survey on existing concolic testing tools, discussing their strengths and limitations, and environments in which they can be applied. We also conduct a case study to determine the prevalence of the identified limitations in six large software systems (four from open-source and two from ABB), as well as the effectiveness and scalability of the publicly available tools. The results show that pointers and native calls are the most prevalent limitations, preventing tools from generating high branch coverage test cases, and variables of float type are the least prevalent. The scalability of the publically available tools is also a limitation for industrial use, due to the large overhead of creating a test harness. Finally, we propose suggestions on how practitioners can use these tools and how researchers can improve concolic testing.

Faster than at-speed testing provides an effective way to detect small delay defects (SDDs). It requires test patterns to be delicately classified into groups according to the delay of sensitized paths. Each group of patterns is applied at certain frequency. In this paper, we propose to generate tests for faster than at-speed testing using path delay fault (PDF) model and single path sensitization criterion. An effective path selection and grouping method is introduced, which could quickly and accurately identify paths whose delay falls into a given delay span. Several techniques are used to improve the efficiency of the testable path selection procedure. Experimental results on ISCAS'89 benchmark circuits show that the proposed method could achieve high transition fault coverage and high test quality of SDDs with low CPU time.

With the increasing complexity and chip scale of SoC, the test problem is becoming more difficult and important. Adding DFT (design for testability) in SoC design period has become the main method for solving the test problem. Based on analyzing some common DFT structures such as Fscan-Bscan, Fscan-Tbus, and the standard for embedded core test (SECT) IEEE P1500, a system-level mixed DFT-TAM (test access mechanism) structure has been presented in this paper. In this structure, different DFT methods are applied in different characteristic IP (intellectual property) cores to pursue the reasonable testing cost, then, a more flexible TAM is built in the system level for achieving every IP core test and testing the connection faults between two IP cores. By some experiments with the programmable video add data (VAD) SoC introduced in this paper, the higher fault coverage and lower test cost could be gotten

In this paper, an analysis of backtrack behavior in PODEM (the test generation algorithm for combinational circuits presented by P. Goel) is given. It is pointed out that there are still many unnecessary backtracks in PODEM on some occasions. A new test generation algorithm named IPODEM is therefore proposed in this paper. IPODEM is an improvement over PODEM with emphasis on backtrack of decision tree. A new backtrack approach is developed in this algorithm. It is shown that onlyO(j) of backtrack consumption is needed in IPODEM compared withO(2j) in PODEM on certain occasions. Experiments pointed out that these occasions appear in not small proportion. Several other techniques are applied in IPODEM to accelerate test generation process in other aspects. Experimental results demonstrated that IPODEM is faster than PODEM for both hard-testing and easy-testing single stuck fault, and that the former has higher test coverage than the latter.

Today's highly increasing product diversity and decreasing product life cycles, also in the automotive industry lead to fast changing production systems with a high ratio of mechatronic components and (control) software. That again leads to ever increasing use of Virtual Commissioning during the development process of automated manufacturing plants. Paired with the still increasing request towards better control program quality, this leads to the need of improved and more efficient virtual plants with more effortless set ups. The common techniques of simulating the plant within the Virtual Commissioning do no longer fulfil these needs, new approaches have to be developed. At the same time, requests towards higher efficiency and higher test coverage during Virtual Commissioning are rising, which leads to the need of developments of easier testing processes. The presented work is concerned with this further developments, especially the test methodology for virtual commissioning based on appropriate behaviour simulation of automated production systems within the automotive industry. Therefore the state of the art is analysed and existing spheres of activity are identified. Afterwards, new approach and solutions for this fields, namely the appropriate behaviour simulation as well as the automatization of testing, are elaborated. An extensive example implementation evaluates the presented results regarding feasibility and eligible performance. Thus, the elaborated outstanding improvements of Virtual Commissioning are not only presented but the corresponding proof of concept is also adduced.

As today's electronics systems increase in complexity, techniques allowing access to "low level" test resources (built-in self-test, boundary scan) are crucial for developing high fault coverage tests. Two IEEE test bus protocol standards have been defined to provide this access at the module (IEEE 1149.5) and device levels (IEEE 1149.1), respectively. This paper discusses the IEEE 1149.5 standard's proposed method for interfacing to the IEEE 1149.1 boundary scan and the issues associated with this interfacing.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>