REcover: towards recovering object files from stripped binary executables

In this tutorial, the Ghidra software reverse engineering framework will be presented, its characteristics highlighted and its features to the hitherto industry standard in reverse engineering tools, IDA Pro - the interactive disassembler, compared against. This framework was released on March the 5th 2019, by the National Security Agency under the Apache v2 license and brought with it a powerful decompiler for many different architectures (X86 16/32/64, ARM/AARCH64, Java/DEX bytecode, ...), which will be presented and its underlying intermediate language p-code and the corresponding SLEIGH-format explained. Further, hands-on demonstrations will follow, including the aforementioned SLEIGH-format, the plugin-system and the standalone-mode, showcased on different reverse engineering tasks like binary diffing, code lifting, deobfuscation and patching.

Having about 80 % of the market share, Android is currently the clearly dominating platform for mobile devices. Application theft and repackaging remains a major threat and a cause of significant losses, affecting as much as 97 % of popular paid apps. The ease of decompilation and reverse engineering of high-level bytecode, in contrast to native binary code, is considered one of the main reasons for the high piracy rate. In this paper, we address this problem by proposing four static obfuscation techniques: native opaque predicates, native control flow flattening, native function indirection, and native field access indirection. These techniques provide a simple and yet effective way of reducing the task of bytecode reverse engineering to the much harder task of reverse engineering native code. For this purpose, native function calls are injected into an app’s bytecode, introducing artificial dependencies between the two execution domains. The adversary is forced to analyze the native code in order to be able to comprehend the overall app’s functionality and to successfully launch static and dynamic analyses. Our evaluation results of the proposed protection methods witness an acceptable cost in terms of execution time and application size, while significantly complicating the reverse-engineering process.

Reverse engineering technology starts with a measurement process, inspecting the surface contour of the object, and applies a CAD geometric modelling technique to construct the surface model. The purpose of this study is to develop a freeform surface modelling system involving the functions of data reorganisation and surface construction of reverse engineering. An important task in reverse engineering is the process of reducing the large amount of coordinate data derived by measurement devices. Reduction methods in common use include uniform deviation, division by a factor, spatial sampling, chordal deviation and interpolated sampling. Each of the above modes has its advantages and weaknesses, and suitable application areas. A new data reduction mode, which integrates the space concept and height characteristic, is proposed, called the height decision method. This method can achieve the aim of data simplification while maintaining the curve characteristics of the model. The CAD modelling system is developed by an AutoLISP program with MDT v4.0 software. An egg model is presented in this paper for explanation and verification.

Xref is a cross referencing tool that is being developed at the University of Lowell to aid in the development of software and for use in reverse engineering tasks. The cross referencer will eventually be able to generate information from C, Pascal, FORTRAN, or Modula-2 source that is to be linked into a single run time package. This information generates the cross reference data for an entire system.

Cybersecurity attacks on embedded devices for industrial control systems and cyber-physical systems may cause catastrophic physical damage as well as economic loss. This could be achieved by infecting device binaries with malware that modifies the physical characteristics of the system operation. Mitigating such attacks benefits from reverse engineering tools that recover sufficient semantic knowledge in terms of mathematical equations of the implemented algorithm. Conventional reverse engineering tools can decompile binaries to low-level code, but offer little semantic insight. This article proposes the REMaQE automated framework for reverse engineering of math equations from binary executables. Improving over state-of-the-art, REMaQE handles equation parameters accessed via registers, the stack, global memory, or pointers, and can reverse engineer equations from object-oriented implementations such as C++ classes. Using REMaQE, we discovered a bug in the Linux kernel thermal monitoring tool “tmon.” To evaluate REMaQE, we generate a dataset of 25,096 binaries with math equations implemented in C and Simulink. REMaQE successfully recovers a semantically matching equation for all 25,096 binaries. REMaQE executes in 0.48 seconds on average and in up to 2 seconds for complex equations. Real-time execution enables integration in an interactive math-oriented reverse engineering workflow.

Binary reverse engineering (RE) is critical for many use cases but is cognitively demanding, suffering from compounded uncertainty and lack of full automation. We survey and synthesize interdisciplinary research in cognitive systems engineering, reverse engineering, and immersive analytics to explore how virtual reality (VR) can be applied to better support human cognition for the binary RE task. We identify relevant work using a hybrid literature review process consisting of targeted database searches and thematic synthesis. Our survey includes relevant work in cognitive/mental models of RE, related cognitive theories (such as embodied cognition and cognitive load), and affordances in VR tied to those theories. We synthesize the survey findings to identify several conceptual threads spanning the three surveyed areas and cluster those threads into three overarching themes: enhancing abductive iteration, augmenting working memory, and supporting information organization. Each of these themes yields a recommended set of affordances in VR to prioritize in system design and future research. Our work bridges cognitive theory with immersive technology, providing a foundation for innovative reverse engineering environments and similar analytical use cases.

Identification and fitting is an important task in reverse engineering and virtual/augmented reality. Compared to the traditional approaches, carrying out such tasks with a deep learning-based method have much room to exploit. This paper presents SMA-Net (Spatial Merge Attention Network), a novel deep learning-based end-to-end bottom-up architecture, specifically focused on fast identification and fitting of CAD models from point clouds. The network is composed of three parts whose strengths are clearly highlighted: voxel-based multi-resolution feature extractor, spatial merge attention mechanism and multi-task head. It is trained with both virtually-generated point clouds and as-scanned ones created from multiple instances of CAD models, themselves obtained with randomly generated parameter values. Using this data generation pipeline, the proposed approach is validated on two different data sets that have been made publicly available: robot data set for Industry 4.0 applications, and furniture data set for virtual/augmented reality. Experiments show that this reconstruction strategy achieves compelling and accurate results in a very high speed, and that it is very robust on real data obtained for instance by laser scanner and Kinect.

This paper aims to provide a road map for future works related to reverse engineering field of expertise. Reverse Engineering, in a mechanical context, relates to any process working in a bottom-up fashion, namely that it goes from a lower level concept or product (closer to the final product) to a higher level one (closer to the ideation step). Nowadays, the manufacturing industry is facing unprecedented increase in data exchange and data warehousing. This comes with new issues that our work will not explore, such as “how to store these data in an efficient manner?”, “what should be stored?” and so on. Nonetheless, this trend also creates new opportunities if we manage to integrate these data into the expertise workflows. In this paper we will cover the possibilities offered by machine learning to succeed in this challenge. We will also present a first and major step in our road map in order to achieve our research goals. We plan to design a metric to quantify how well and how precise can we perform some specific reverse engineering tasks such as detection, segmentation and classification of mechanical parts in imagery data. We aspire to open this metric, and make it freely and widely available to researchers and industry in order to compare the effectiveness, robustness and preciseness of the existing and future approaches.

Decompilation plays a critical role in firmware analysis and reverse engineering by enabling the recovery of high-level source code from binary executables. However, existing neural decompilation models often face challenges due to the semantic gap between assembly code and high-level languages, and they typically require large-scale models that impose significant computational demands. In this paper, we propose DECodeT5, a lightweight and efficient neural decompilation method for the C language. DECodeT5 builds upon the CodeT5 code generation model by integrating a pre-trained assembly encoder in place of its original embedding layer. This modification allows for more effective semantic learning from assembly code, improving the model’s ability to capture the intricacies of assembly-to-source code mapping. The architecture of DECodeT5 not only accelerates convergence during end-to-end decompilation tasks but also supports rapid adaptation across different compilers and instruction set architectures (ISAs) by swapping out the encoder module as needed. Our experimental evaluation on the HumanEval and Exebench benchmarks reveals that DECodeT5 significantly improves semantic recovery accuracy, outperforming Ghidra by over 83% and LLM4Decompile by more than 43% in terms of execution recovery. Furthermore, DECodeT5 maintains a compact model size of only 360M parameters, providing inference speeds that are twice as fast as LLM4Decompile. These results underscore DECodeT5’s suitability for deployment in resource-constrained environments and its flexibility in adapting to real-world reverse engineering scenarios, offering a practical solution for modern firmware analysis and security tasks.

Malwares are becoming increasingly stealthy, more and more malwares are using cryptographic algorithms (e.g., packing, encrypting C&C communication) to protect themselves from being analyzed. The use of cryptographic algorithms and truly transient cryptographic secrets inside the malware binary imposes a key obstacle to effective malware analysis and defense. To enable more effective malware analysis, forensics, and reverse engineering, we have developed CipherXRay - a novel binary analysis framework that can automatically identify and recover the cryptographic operations and transient secrets from the execution of potentially obfuscated binary executables. Based on the avalanche effect of cryptographic functions, CipherXRay is able to accurately pinpoint the boundary of cryptographic operation and recover truly transient cryptographic secrets that only exist in memory for one instant in between multiple nested cryptographic operations. CipherXRay can further identify certain operation modes (e.g., ECB, CBC, CFB) of the identified block cipher and tell whether the identified block cipher operation is encryption or decryption in certain cases. We have empirically validated CipherXRay with OpenSSL, popular password safe KeePassX, the ciphers used by malware Stuxnet, Kraken and Agobot, and a number of third party softwares with built-in compression and checksum. CipherXRay is able to identify various cryptographic operations and recover cryptographic secrets that exist in memory for only a few microseconds. Our results demonstrate that current software implementations of cryptographic algorithms hardly achieve any secrecy if their execution can be monitored.

Information from computer programs can be extracted from its source code, external documentation, and compiled code. Although compiled code is an assured information source which is always exists in published computer programs, it is seldom used by the existing search engines since some reverse engineering tasks are needed. In this research, a search engine for Java archives that uses byte code (compiled code for Java Archive) as its information source is developed. It enables user to search within a collection of Java Archives without relying with source code and external documentation. Compared with Penta and FindJar [2] [7], A novel term extraction process beyond the file and class name is proposed, which includes field name, method name, string literal used in program, program flow weighting, and method expansion. Exclusive tokenization, stopping, and stemming are also implemented to improve effectiveness. Based on evaluation, it has a fairly good effectiveness although it may vary based on terms stored on index. Its effectiveness is higher than FindJar main features reimplementation which indicates that detailed compiled code has positive influences in computer programs search engine. Efficiency depends on how many terms stored on index and how many process used at certain step.

Identifying binary code at function level has been applied to a broad range of software security applications and reverse engineering tasks, including patch analysis, vulnerability assessment, code plagiarism detection, malware analysis, etc. However, various anti-reverse engineering techniques (e.g., obfuscation, anti-emulator, etc.) employed by the mobile apps make existing approaches ineffective when performing function identification. In this paper, we propose MobileFindr, an on-device trace-based function similarity identification framework on the mobile platform. MobileFindr runs on real mobile devices and mitigates many prevalent anti-reversing techniques by extracting function execution behaviors via dynamic instrumentation, then characterizing functions with collected behaviors and performing function matching via distance calculation. We have evaluated MobileFindr using real-world top-ranked mobile frameworks and applications. The experimental results showed that MobileFindr outperforms existing state-of-the-art tools in terms of better obfuscation resilience and accuracy.

Identifying library functions in program binaries is important to many security applications, such as threat analysis, digital forensics, software infringement, and malware detection. Today’s program binaries normally contain a significant amount of third-party library functions taken from standard libraries or free open-source software packages. The ability to automatically identify such library functions not only enhances the quality and the efficiency of threat analysis and reverse engineering tasks, but also improves their accuracy by avoiding false correlations between irrelevant code bases. Existing methods are found to either lack efficiency or are not robust enough to identify different versions of the same library function caused by the use of different compilers, different compilation settings, or obfuscation techniques. To address these limitations, we present a scalable and robust system called BinShape to identify standard library functions in binaries. The key idea of BinShape is twofold. First, we derive a robust signature for each library function based on heterogeneous features covering CFGs, instruction-level characteristics, statistical characteristics, and function-call graphs. Second, we design a novel data structure to store such signatures and facilitate efficient matching against a target function. We evaluate BinShape on a diverse set of C/C++ binaries, compiled with GCC and Visual Studio compilers on x86-x64 CPU architectures, at optimization levels \(O0-O3\). Our experiments show that BinShape is able to identify library functions in real binaries both efficiently and accurately, with an average accuracy of \(89\%\) and taking about 0.14 s to identify one function out of three million candidates. We also show that BinShape is robust enough when the code is subjected to different compilers, slight modification, or some obfuscation techniques.

Inhomogeneous layers, such as so-called gradient index layers and rugate filters represent new and prospective thin film designs. Manufacturing such systems in practice requires calculation, deposition, monitoring and characterization of optical coatings with a well-defined continuous refractive index profile along an axis that is perpendicular to the film surface. Those coatings may be manufactured in the Leybold Syrus Pro 1100 deposition system by co-evaporation of SiO2 and Nb2O5 as a sequence of several refractive index gradients. During these experiments our in-situ broadband monitoring system was used to measure the transmittance of the growing film directly at the rotating substrate. This additional information on the intermediate stages of the not yet completed film are extremely helpful in reverse engineering tasks, and clearly superior to the extent of information that may be drawn from the spectra of the completed film only. For characterization of these coatings a new model was developed, which significantly reduces the number of parameters. To generate a feasible parameter set, deposition rates for both materials recorded with quartz crystals monitor during deposition were used. This approach achieves a better accordance between in-situ measured transmittance and modelled transmittance than the intended design. During the optimisation process, a local minimization algorithm was used to vary the refractive index profile of the whole coating and film thickness of the intermediate stages. Finally, a significantly improved accuracy of the modelled transmittance was achieved.

Identifying free open-source software (FOSS) packages on binaries when the source code is unavailable is important for many security applications, such as malware detection, software infringement, and digital forensics. This capability enhances both the accuracy and the efficiency of reverse engineering tasks by avoiding false correlations between irrelevant code bases. Although the FOSS package identification problem belongs to the field of software engineering, conventional approaches rely strongly on practical methods in data mining and database searching. However, various challenges in the use of these methods prevent existing function identification approaches from being effective in the absence of source code. To make matters worse, the introduction of obfuscation techniques, the use of different compilers and compilation settings, and software refactoring techniques has made the automated detection of FOSS packages increasingly difficult. With very few exceptions, the existing systems are not resilient to such techniques, and the exceptions are not sufficiently efficient. To address this issue, we propose FOSSIL , a novel resilient and efficient system that incorporates three components. The first component extracts the syntactical features of functions by considering opcode frequencies and applying a hidden Markov model statistical test. The second component applies a neighborhood hash graph kernel to random walks derived from control-flow graphs, with the goal of extracting the semantics of the functions. The third component applies z-score to the normalized instructions to extract the behavior of instructions in a function. The components are integrated using a Bayesian network model, which synthesizes the results to determine the FOSS function. The novel approach of combining these components using the Bayesian network has produced stronger resilience to code obfuscation. We evaluate our system on three datasets, including real-world projects whose use of FOSS packages is known, malware binaries for which there are security and reverse engineering reports purporting to describe their use of FOSS, and a large repository of malware binaries. We demonstrate that our system is able to identify FOSS packages in real-world projects with a mean precision of 0.95 and with a mean recall of 0.85. Furthermore, FOSSIL is able to discover FOSS packages in malware binaries that match those listed in security and reverse engineering reports. Our results show that modern malware binaries contain 0.10--0.45 of FOSS packages.