Model-Based Real-Time Testing of Embedded Automotive Systems

Human resource management practices are key for the success of software development projects. Practices that promote knowledge sharing and organizational learning are positively related to development---effort curves, and thus software companies are looking for different alternatives oriented to promoting these practices. The model-driven development (MDD) paradigm is positioned as one of the best alternatives for reutilization of development knowledge. In particular, this paradigm considers the specification of conceptual models that can be used as input for automatic code generation to different target platforms. However, testing of applications developed through MDD solutions is still performed by the manual definition and execution of test cases by testers, which negatively impacts in the time reduction obtained from automatic code generation and the reutilization of knowledge generated during the MDD project execution. To address this issue, this paper presents a testing approach that automatically generates executable test cases for software developed by using MDD technologies.

Penetration testing is a widely used method for testing the security of web applications, but it can be inefficient if it is not done systematically. Public databases of web application vulnerabilities can be used to drive penetration testing, but testers need to understand them and interpret them into executable test cases. This requires an in-depth knowledge of security. This paper proposes a model-based testing approach using a data model that describes the relationship between web security knowledge, business domain knowledge, and test case development. The approach consists of a data model that represents the relevance between attack surface, application fingerprint, attack vectors, and fuzz vectors; a test case generator that automatically generates penetration test scenarios for web applications; and a penetration test framework supported by TTCN-3 test environment. The model-based testing approach can be used to provide structured tool support for developing penetration test campaigns. We demonstrate the feasibility and efficiency of the approach at the design level.

In Cyber-Physical Production System (CPPS) engineering for discrete manufacturing, the definition of test cases is vital to ensure correct behavior of production processes and to test risky cases. Unfortunately, the definition of test cases requires know-how both from the CPPS engineering domain and on software test automation, and is time-consuming. In this paper, we investigate how domain experts can efficiently derive test cases for an assembly process step from process preconditions concerning product properties. We introduce the Test Case Derivation for PPR Models (TCD4PPR) method building on the Formalised Process Description and best practices from software testing. We evaluate the TCD4PPR method with an illustrative use case from industry in a feasibility study with domain experts at a large production systems engineering company for discrete manufacturing. The main result was that the domain experts found the TCD4PPR method efficient, usable, and useful. The evaluation results indicate that investing reasonable effort into modeling Product, Process, Resource (PPR) knowledge with preconditions can considerably reduce risks of untested production process behavior.

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

A well-known testing method for the safety evaluation and real-time validation of automotive software systems (ASSs) is Fault Injection (FI). In accordance with the ISO 26262 standard, the faults are introduced artificially for the purpose of analyzing the safety properties and verifying the safety mechanisms during the development phase. However, the current FI method and tools have a significant limitation in that they require manual identification of FI attributes, including fault type, location and time. The more complex the system, the more expensive, time-consuming and labour-intensive the process. To address the aforementioned challenge, a novel Large Language Models (LLMs)-assisted fault test cases (TCs) generation approach for utilization during real-time FI tests is proposed in this paper. To this end, considering the representativeness and coverage criteria, the applicability of various LLMs to create fault TCs from the functional safety requirements (FSRs) has been investigated. Through the validation results of LLMs, the superiority of the proposed approach utilizing gpt-4o in comparison to other state-of-the-art models has been demonstrated. Specifically, the proposed approach exhibits high performance in terms of FSRs classification and fault TCs generation with F1-score of 88% and 97.5 %, respectively. To illustrate the proposed approach, the generated fault TCs were executed in real time on a hardware-in-the-loop system, where a high-fidelity automotive system model served as a case study. This novel approach offers a means of optimizing the real-time testing process, thereby reducing costs while simultaneously enhancing the safety properties of complex safety-critical ASSs.

Test-case prioritization is to schedule the execution order of test cases so as to maximize some objective (e.g., revealing faults early). The existing test-case prioritization approaches separate the process of test-case prioritization and the process of test-case execution by presenting the execution order of all test cases before programmers start running test cases. As the execution information of the modified program is not available for the existing test-case prioritization approaches, these approaches mainly rely on only the execution information of the previous program before modification. To address this problem, we present an adaptive test-case prioritization approach, which determines the execution order of test cases simultaneously during the execution of test cases. In particular, the adaptive approach selects test cases based on their fault-detection capability, which is calculated based on the output of selected test cases. As soon as a test case is selected and runs, the fault-detection capability of each unselected test case is modified according to the output of the latest selected test case. To evaluate the effectiveness of the proposed adaptive approach, we conducted an experimental study on eight C programs and four Java programs. The experimental results show that the adaptive approach is usually significantly better than the total test-case prioritization approach and competitive to the additional test-case prioritization approach. Moreover, the adaptive approach is better than the additional approach on some subjects (e.g, replace and schedule).

In many domains, such as avionics, oil and gas, and maritime, a common practice is to derive and execute test cases manually from requirements, where both requirements and test cases are specified in natural language (NL) by domain experts. The manual execution of test cases is largely dependent on the domain experts who wrote the test cases. The process of manual writing of requirements and test cases introduces ambiguity in their description and, in addition, test cases may not be effective since they may not be derived by systematically applying coverage criteria. In this paper, we report on a systematic approach to support automatic derivation of manually executable test cases from use cases. Both use cases and test cases are specified in restricted NLs along with carefully-defined templates implemented in a tool. We evaluate our approach with four case studies (in total having 30 use cases and 579 steps from flows of events), two of which are industrial case studies from the oil/gas and avionics domains. Results show that our tool was able to correctly process all the case studies and systematically (by following carefully-defined structure coverage criteria) generate 30 TCSs and 389 test cases. Moreover, our approach allows defining different test coverage criteria on requirements other than the one already implemented in our tool.

Software testing is a process where you have the stated requirements, pre-requisite software that is to be installed and the configuration of the machine that need to be done as per the test data for the test case and for a specific platform which when verified works perfectly. This paper describes about Manual test case process overview in the testing life cycle for a large software development project and how you track the effort for the same. A Tester need to know about the fundamentals of testing concepts. Documentation of the testing concepts and the process overview of the testing life cycle for a software product need to be done initially so that a new tester when recruited can understand the basic flow as to how it is carried out in real time in an organization practically. Initially a tester need to execute the manual test case for few platforms and some tougher manual test cases need to be learnt from the tester who is having some experience. The tougher manual test cases need to be identified initially from the experienced tester who will be giving a session for the execution of such test cases. So when the foundation for a new tester is powerful the plan assigned for the execution of test cases is met. The effort tracking for the testing can be captured for a tester and should be matched with planned effort for the execution of test cases. Measurement of the testing effort plays a very important role as to how comfortable a new tester is and further root cause analysis is done if required when the new tester does not meet the planned effort.

Aim. Development of an effective test system for diagnosing feline coronavirus disease based on reverse transcription polymerase chain reaction (RT-PCR). Methods. Polymerase chain reaction (PCR); reverse transcription polymerase chain reaction (RT-PCR); real-time RT-PCR; melting curve analysis; polyacrylamide gel electrophoresis; statistical methods. Results. A PCR test system with electrophoretic detection and a real-time PCR test system were developed for the detection of feline coronavirus (FCoV). The advantages of the real-time PCR test system were demonstrated, and its efficiency was evaluated. Using the real-time test system, diagnostic testing of 160 cats was conducted, revealing the presence of FCoV in 18 % of the examined animals. Conclusions. The high diagnostic efficiency of the real-time PCR test system allows its use for laboratory diagnosis of feline coronavirus disease. Although feline coronavirus disease is not classified as a seasonal illness, a correlation between the detection frequency of FCoV and the season was observed. The detection rate of FCoV was 28 % in spring, decreased to 11 % in summer, and increased again to 24 % in September. One of the possible reasons for these seasonal fluctuations may be the weakening of immunity in animals during autumn and spring, which could influence the risk of infection or virus reactivation.

We present experiences from a case study where a model-based approach to black-box testing is applied to verify that aWireless Application Protocol (WAP) gateway conforms to its specification. TheWAP gateway is developed by Ericsson and used in mobile telephone networks to connect mobile phones with the Internet. We focus on testing the software implementing the session (WSP) and transaction (WTP) layers of the WAP protocol. These layers, and their surrounding environment, are described as a network of timed automata. To model the many sequence numbers (from a large domain) used in the protocol, we introduce an abstraction technique. We believe the suggested abstraction technique will prove useful to model and analyse other similar protocols with sequence numbers, in particular in the context of model-based testing. A complete test bed is presented, which includes generation and execution of test cases. It takes as input a model and a coverage criterion expressed as an observer, and returns a verdict for each test case. The test bed includes existing tools from Ericsson for test-case execution. To generate test suites, we use our own tool CO/ER--a new test-case generation tool based on the real-time model-checker UPPAAL.

Evolving multi-parameter, multi-configuration systems require regression test suite that can be customized. This is in terms of run time. Run time can be customized by generating the combinations using combinatorial techniques. For systems like Contiki operating system, the test cases need to be executed in its simulator Cooja. Executing test cases in a simulator requires functional test cases to be generated from the combinatorial parameter combinations obtained. In this work we present a methodology to generate the functional test cases. We present Functional Test Case Generator for Contiki and Cooja (FTCGCC), which is a tool developed using our methodology. We demonstrate use of our tool by generating customizable regression test suite for Contiki and Cooja using code coverage as criteria. FTCGCC is developed for the test case generation when target System Under Test is IoT operating system Contiki and its simulator Cooja.

Event-driven software systems are hard to thoroughly test because they possess a huge number of possible event sequences to test and also require significant manual effort and/or special hardware support during test case execution. In this paper we propose a novel testing approach, we call model-based replay testing that combines the advantages of model-based testing and an operating system (OS) level replay mechanism to overcome such difficulties of event-driven software testing. Specifically, we use a finite state machine (FSM) model and an n-switch set cover to automatically generate abstract test cases. We then introduce the use of OS-level replay mechanism in transforming abstract test cases into concrete test cases and executing them in an automated manner. To demonstrate the effectiveness of the proposed approach, we also describe a case study on GUI software testing using a prototype implementation of an OS-level replay mechanism in Android software platform.

Fault Localization Prioritization (FLP) aims at reordering existing test cases so that the location of detected faulty components can be identified earlier, using certain fault localization techniques. Although some researchers have proposed adaptive prioritization strategies with white-box code coverage information, such information may not always be available. In this paper, we address the FLP problem using black-box information derived from partitioning the input domain. Based on the well-known technique of Spectra-Based Fault Localization (SBFL), three test case prioritization strategies are designed following some basic SBFL heuristics. The implementation of these proposed strategies relies only on the partition information, and does not require any test case execution history. Experiments show that our strategies, when compared with pure random selection, result in a faster localization of faulty statements, reducing the number of test case executions required. Here, we analyze the characteristics and merits of the three proposed strategies.

Testing has become an integral part of innovation, production and operation of systems. The activity of testing is already a flourishing area with the active participation of a large community of researchers and experts who are highly aware of its importance and impact for the future deployment and use of software and systems. Formal methods have proved to be very promising for developing automated and generic testing methods. Actually,the combination of formal methods and testing is currently well understood and tools to automate testing activities are widely available. (Formal) testing is the assessment, by means of experiments, that a product conforms to its (formal) requirements. Test cases are designed to test particular aspects of the system, called test objectives. In order to formally obtain testing objectives, it is necessary to provide mathematical models for the semantic of the studied system, formal frameworks for testing, languages to describe the expected properties or requirements of the system in a precise and unambiguous way and methods to generate the expected test cases. In addition, active methods require deploying a test environment(test architecture) to execute test cases and to observe implementation reactions. They may also interrupt the system normal functioning arbitrarily, for example by resetting it after each test case execution. However,when a system is deployed in an integrated environment,it becomes quite difficult to access it. Moreover,active methods may disturb the natural operation of the implementation under test. So, these ones may not be suitable in regards to the tested system. Passive testing represents another interesting alternative, which offers several advantages, for example, to not disturb the system while testing. In this paper we present a survey covering the main approaches to formal testing. We have divided the survey in three parts. First, we distinguish between active and passive approaches to formal testing. Next, we review some of the work on testing the cloud and on testing in the cloud.

The trend towards distributed applications and heterogeneous networks increases the complexity of systems and network management. The standardization bodies have defined a framework for the specification and open exchange of management information. Conformance, one of the key requirements for the interworking of open management systems, was hardly discussed. Few groups have realized the importance of defining conformance not only for management protocols but also for management information. This paper presents a methodology for management information testing and explains differences with respect to communication protocol testing. It is shown how conformance requirements can be derived and how tests can be performed. An outlook points out the adaptation of an existing TTCN based test tool for the definition and execution of test cases for management information.KeywordsOSI ManagementConformance TestingManaged Objects