An evaluation of reverse engineering tool capabilities

In this paper we present work in progress towards implementing a benchmark called BEFRIEND (benchmark for reverse engineering tools working on source code), with which the outputs of reverse engineering tools can be evaluated and compared easily and efficiently. Such tools are e.g. design pattern miners, duplicated code detectors and coding rule violation checkers. BEFRIEND supports different kinds of tool families, programming languages and software systems, and it enables the users to define their own evaluation criteria.

Before performing software maintenance and evolution tasks, developers must expend significant effort to understand the design of the subject software system. Reverse engineering and reengineering tools can extract artifacts, such as class diagrams, from source code so that the effort expended on program comprehension can be greatly reduced. Choosing suitable and appropriate tools is itself a difficult process. Many metrics and benchmarks have been proposed to evaluate the quality of reengineering tools, but the process is still complicated and requires much human effort. Furthermore, ignored by most evaluation experiments and approaches is the understanding of differences and similarities of the output artifacts produced by different tools. This paper proposes a novel approach for tool evaluation making a direct comparison of the output artifacts of the chosen tools. We apply our methodology on class diagrams extracted from source code. We utilize a class matching model to automate the process. In this study, we use ten open-source programs as test cases and two open source reverse engineering tools Doxygen and StarUML. We evaluate the output of these two tools based on class matching given the same input. This approach, we believe, provides developers with good guidance in tool selection.

Developers tend to leave some important steps and actions (e.g. properly designing the system's architecture, code review and testing) out of the software development process, and use risky practices (e.g. the copy-paste technique) so that the software can be released as fast as possible. However, these practices may turn out to be critical from the viewpoint of maintainability of the software system. In such cases, a cost-effective solution might be to re-engineer the system. Re-engineering consists of two stages, namely reverse-engineering information from the current system and, based on this information, forward-engineering the system to a new form. In this way, successful re-engineering significantly depends on the reverse engineering phase. Therefore, it is vital to guarantee correctness, and to improve the results of the reverse engineering step. Otherwise, the re-engineering of the software system could fail due to the bad results of reverse engineering. The above issues motivated us to develop a method which extends and improves one of our reverse engineering tools, and to develop benchmarks and to perform experiments on evaluating and comparing reverse engineering tools.

all the unnecessary information of data structures and types is removed at the end of compilation. However, the high level information is extremely valuable and necessary for many fields such as debugging, reverse engineering, security etc. Without data structures information, reverse engineering is extremely hard to perform. Unfortunately, even the most advanced techniques and tools for reverse engineering are very weak in this part. Recently, several new solutions have been proposed and each has made some achievements. In this paper, we summarize challenges in this problem, then analyze three existing outstanding approaches, discuss their limitations and at last propose some possible directions to make further efforts.

In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little effort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their effectiveness, whereas source code changes alone do not influence their effectiveness much.

The increasing complexity associated with many modern engineering applications, including autonomous robot guidance and navigation, process control in sensor-rich environments, and control of biological systems, has far-reaching implications for control system design. As an example, reactive, embedded software systems, interacting among themselves and remote users over communication networks, introduce a whole new set of system-level challenges, and classic control design objectives such as stability, performance, and robustness are being complemented with a number of new questions. These include the cost of hardware implementation, measured for example not only by computational requirements such as speed and memory, but also by communication requirements such as available communication bandwidth. Moreover, the complexity associated with specifying the control procedures and with verifying the behavior of the closed-loop system increasingly plays a fundamental role, especially in safety-critical control systems arising in energy and transportation networks, and in medical applications.

CLAS (C Language Program Analysis System) is a reverse engineering tool intended for use in the verification and validation (V&V) of software programs developed in the ANSI 'C' language. By analysing the source code, CLAS generates data according to two conceptual models of software programs viz, the entity-relationship (ER) model and control flow graph (CFG) model. The tools within CLAS, make use of this data, to provide different graphical views of the programs. We discuss the features provided by CLAS and show how these can aid in automating some aspects of V&V and quality assessment activities.

Droid-MCFG: Android malware detection system using manifest and control flow traces with multi-head temporal convolutional network

The usefulness of design patterns in forward engineering is already well-known and several tools provide support for their application in the development of software systems. While the role of design patterns in reverse engineering is still argued primarily due to their informal definition which leads to various possible implementations of each pattern. One of the most discussed aspects related to design patterns is about the need of their formalization according to the drawbacks this can represent. Formalization leads to the identification of the so-called sub-patterns, which are the recurring fundamental elements design patterns are composed of. In this paper we analyze the role sub-patterns play-in two reverse engineering tools: FUJABA and SPQR. Attention is focused on how sub-patterns are exploited to define and to detect design patterns. To emphasize the similarities and differences between the two approaches, the Composite Design Pattern is considered as example. © 2005 IEEE.

Some of the reasons for difficulties arising in the use of design documents produced by reverse engineering tools are analyzed. With reference to intermodular dataflow analysis for Pascal software systems, an interactive tool is proposed to more effectively help the maintainer. The tool is based on: the production of intermodular dataflow information by static analysis of the code; their representation in a Prolog program dictionary; and a Prolog abstractor that allows specific queries of maintainers to be answered. A logic-based approach to the design and implementation of an interactive tool which analyzes the intermodularal data flow of a program is discussed. >

Code review has been widely acknowledged as a key quality assurance process in both open-source and industrial software development. Due to the asynchronicity of the code review process, the system's codebase tends to incorporate external commits while a source code change is reviewed, which cause the need for rebasing operations. External commits have the potential to modify files currently under review, which causes re-work for developers and fatigue for reviewers. Since source code changes observed during code review may be due to external commits, rebasing operations may pose a severe threat to empirical studies that employ code review data. Yet, to the best of our knowledge, there is no empirical study that characterises and investigates rebasing in real-world software systems. Hence, this paper reports an empirical investigation aimed at understanding the frequency in which rebasing operations occur and their side-effects in the reviewing process. To achieve so, we perform an in-depth large-scale empirical investigation of the code review data of 11 software systems, 28,808 code reviews and 99,121 revisions. Our observations indicate that developers need to perform rebasing operations in an average of 75.35% of code reviews. In addition, our data suggests that an average of 34.21% of rebasing operations tend to tamper with the reviewing process. Finally, we propose a methodology to handle rebasing in empirical studies that employ code review data. We show how an empirical study that does not account for rebasing operations may report skewed, biased and inaccurate observations.

Maintenance and evolution of complex software systems (such as large telecom embedded devices) involve activities such as reverse engineering (RE) and software visualization. Although several RE tools exist, we found their architecture hard to adapt to the domain specific requirements posed by our current practice in Nokia. We present an open architecture which allows easy prototyping of RE data exploration and visualization scenarios for a large range of domain models. We pay special attention to the visual and interactive requirements of the reverse engineering process. The article describes the basic architecture of our toolkit, compares it to the existing RE environments and present several visualizations taken from real cases.

The domain of real-time, concurrent, and embedded software is becoming increasing complex. To effectively develop these systems, greater care must be taken to construct adequate models of the software and to effectively analyze these designs prior to code development. In this tutorial, we will discuss modeling and analysis issues specific to real-time, concurrent, and embedded software systems. Specifically, this tutorial will present guidelines for modeling these systems using the Unified Modeling Language (UML) version 2. A case study will be introduced to solidify the concepts and participant interaction will be encouraged in the construction of the models. Analytical methods will also be discussed to verify that the UML-based designs will produce the desired behavior. Finally, we will discuss how to proceed from the UML models into an Ada (2005) implementation.

Maintenance of embedded software systems is faced with multiple challenges, including the exploration and analysis of the actual system's runtime behavior. As a fundamental technique, tracing can be used to capture data about runtime behavior as a whole, and represents one of the few methods to observe and record data about embedded systems within their production environments. In this paper we present a software-based, function-boundary tracing approach for embedded software systems. It uses static binary instrumentation, which implies only lightweight memory and performance overheads. To further reduce these overheads, instrumentation can be configured per trace, i.e., activated only for a specified group of functions without having to recompile the system. The technique can be characterized by its robust implementation and its versatile usage. It is complemented by a visualization framework that allows for analysis and exploration of a system's runtime behavior, e.g., to examine thread interaction. To show the technique's applicability, we conclude with a case study that has been applied to an industrial embedded software system.

Cognitive science emphasizes the strength of visual formalisms for human learning and problem solving. In software engineering, a clear, visual presentation of a system's architecture can significantly reduce the effort of comprehension. Yet, all too often the documentation of complex software systems lacks clear identification of the architectural constituents and insufficiently relates them to the source code. It is our contention that visualization of the architectural constituents within the source code model is an indispensable aid for the guided evolution of large-scale software systems. We present a prototype tool for visualizing both published, generic design patterns as well as well-thought, ad-hoc design solutions, given the reverse-engineered source code of a system. We discuss the architecture and core functionality of this tool, addressing source code reverse engineering, design repository, design representation, and design clustering. Then, we present our visualization objectives and detail our techniques for pattern visualization. A case study example helps explicate and illustrate our work.