Fingerprinting Malware Behavior: A Deep Learning Approach

As malicious programs, Trojan horses have become a huge threat to computer networks security. Trojan horses can easily cause loss, damage or even theft of data because they are usually disguised as something useful or desirable, and are always mistakenly activated by computer users, corporations and other organizations. Thus, it is important to adopt an effective and efficient method to detect the Trojan horses, and the exploration of a new method of detection is of greater significance. Scientists and experts have tried many approaches to detecting Trojan horses since they realized the harms of the programs. Up to now, these methods fall mainly into two categories [2]. The first category is to detect Trojan horses through checking the port of computers since the Trojan horses send out message through computer ports [2]. However, these methods can only detect the Trojan horses that are just working when detected. The second class is to detect Trojan horses by examining the signatures of files [2] [19], in the same way as people deal with computer virus. As new Trojan horses may contain unknown signatures, methods in this category may not be effective enough when new and unknown Trojan horses appear continuously, sending out unknown signatures that escape detection. For the above-mentioned reasons, without exception, there are limitations in the existing methods if the un-awakened and unknown Trojan horses are to be detected. This thesis proposes a new method that can detect un-awakened and unknown Trojan horses- the detection by using of a file's static characteristics. This thesis takes PE file format as the object of the research, because approximately 75% of personal computers worldwide are installed the Microsoft Windows [4], and that Trojan horses usually exist as a Portable Executable (PE) file in the Windows platform. Based on the (PE) file format, my research gets all the static information of each part of PE file which is characteristic of a file. Then, this static information is analyzed by the intelligent information processing techniques. Next, a detection model is established to estimate whether a PE file is a Trojan horse. This model can detect the unknown Trojan horses by analyzing static characteristics of a file. The information that is used to verify detecting model is new and unknown to the detecting model; in other words, the information is not used during the training of the model. The thesis is organized as follows. First, this thesis discusses the limitations of traditional detection techniques, related works of research, and a new method to detect Trojan horse based on file's static information. Second, the thesis focuses on the research of the Trojan horse detecting models, covering the extracting of the static information from PE file, choice of intelligent information processing techniques, and setting up the Trojan horse detecting model. Lastly, the thesis discusses the direction of future research in this field.

Executable files such as .exe, .bat, .msi etc. are used to install the software in Windows-based machines. However, downloading these files from untrusted sources may have a chance of having maliciousness. Moreover, these executables are intelligently modified by the anomalous user to bypass antivirus definitions. In this paper, we propose a method to detect malicious executables by analyzing Portable Executable (PE) files extracted from executable files. We trained a supervised binary classifier using features extracted from the PE files of normal and malicious executables. We experimented our method on a large publicly available dataset and reported more than 95% of classification accuracy.

PE (Portable executable) file is a standard format for executable file and is applied extensively. PE file has diversity, uncertainty of file size, complexity of file structure and singleness of file format, which make PE file easy to be a carrier of information hiding, especially for that of large hiding capacity. A novel software information hiding algorithm is proposed, which makes full use of display characteristics of palette icon of portable executable file. In this algorithm, the information is embedded into the transparent area of the icon by taking advantage of the redundant color items in the palette. The experimental results show that after embedding the information, the size of the icon remains unchanged, that is, the size of the resource section will not change, and the size of the PE file will not change. On the other hand, the icon with embedded data can be correctly analyzed and displayed without any distortion. PE file can run normally, and does not affect the performance of the program, so the algorithm has good concealment. The algorithm selects an index value whose color is black according to rules in the XOR bitmap, these index values are encoded for information hiding, its complexity is low. At the same time, we can further improve the hiding capacity by adding one or more icons to PE file.

A novel deep learning-based approach for malware detection

During the last decade, the cybersecurity literature has conferred a high-level role to machine learning as a powerful security paradigm to recognise malicious software in modern anti-malware systems. However, a non-negligible limitation of machine learning methods used to train decision models is that adversarial attacks can easily fool them. Adversarial attacks are attack samples produced by carefully manipulating the samples at the test time to violate the model integrity by causing detection mistakes. In this paper, we analyse the performance of five realistic target-based adversarial attacks, namely Extend, Full DOS, Shift, FGSM padding + slack and GAMMA, against two machine learning models, namely MalConv and LGBM, learned to recognise Windows Portable Executable (PE) malware files. Specifically, MalConv is a Convolutional Neural Network (CNN) model learned from the raw bytes of Windows PE files. LGBM is a Gradient-Boosted Decision Tree model that is learned from features extracted through the static analysis of Windows PE files. Notably, the attack methods and machine learning models considered in this study are state-of-the-art methods broadly used in the machine learning literature for Windows PE malware detection tasks. In addition, we explore the effect of accounting for adversarial attacks on securing machine learning models through the adversarial training strategy. Therefore, the main contributions of this article are as follows: (1) We extend existing machine learning studies that commonly consider small datasets to explore the evasion ability of state-of-the-art Windows PE attack methods by increasing the size of the evaluation dataset. (2) To the best of our knowledge, we are the first to carry out an exploratory study to explain how the considered adversarial attack methods change Windows PE malware to fool an effective decision model. (3) We explore the performance of the adversarial training strategy as a means to secure effective decision models against adversarial Windows PE malware files generated with the considered attack methods. Hence, the study explains how GAMMA can actually be considered the most effective evasion method for the performed comparative analysis. On the other hand, the study shows that the adversarial training strategy can actually help in recognising adversarial PE malware generated with GAMMA by also explaining how it changes model decisions.

It is a difficult issue in the anti-virus field about how to detect unknown and packed PE (Portable Executable) viruses effectively, and existing schemes for anti-virus detection are dissatisfactory. Based on an analysis of the logical structures of Windows PE file and PE virus, a new method of PE virus detection is presented here, which exploits fragile software watermarking technique for virus detection. It describes how the new scheme embeds a software watermark into the PE files for virus detection. The experiment shows that the new method has a higher rate in detecting unknown PE virus than some existing ones, and can detect directly the packed PE virus.

This study investigates whether convolutional neural networks (CNNs) trained on visual representations of Portable Executable (PE) files can rival traditional machine learning classifiers trained on engineered features. A dataset of over 200,000 PE files [1] was used to derive two feature sets (Basic and Ember-Lite) [2] and to generate 256x256 greyscale and entropy images [3],[4]. Three CNNs (SimpleCNN, ResNet-18 [5], EfficientNet-B0 [6]) were trained and evaluated against five baselines (Random Forest, XGBoost [7], CatBoost [8], LightGBM, Logistic Regression). Tree-based models with enriched features achieved the highest scores, with CatBoost reaching a ROC-AUC of 0.990. The best CNN, EfficientNet-B0 on entropy images, obtained a ROC-AUC of 0.954. Although CNNs did not surpass feature-based models, they showed competitive results when feature engineering was constrained. These findings indicate that visual approaches offer a promising alternative for static malware detection, particularly when combined with entropy-based representations [9].

Malware writers often develop malware with automated measures, so the number of malware has increased dramatically. Automated measures tend to repeatedly use significant modules, which form the basis for identifying malware variants and discriminating malware families. Thus, we propose a novel visualization analysis method for researching malware similarity. This method converts malicious Windows Portable Executable (PE) files into local entropy images for observing internal features of malware, and then normalizes local entropy images into entropy pixel images for malware classification. We take advantage of the Jaccard index to measure similarities between entropy pixel images and the k-Nearest Neighbor (kNN) classification algorithm to assign entropy pixel images to different malware families. Preliminary experimental results show that our visualization method can discriminate malware families effectively.

Traditional malware detection methods cannot keep up with the massive amount of newly created malware quickly and effectively. Machine learning is a promising method for the detection and classification of large-scale newly created malware according to the features of samples. The current research trend is to use machine learning technology, such as the Gradient Boosting Decision Tree (GBDT) and deep neural network technology, to learn newly created malware rapidly and accurately. We propose Control-Flow Graph (CFG)- and Graph Isomorphism Network (GIN)-based malware classification, where we first extract the CFG from portable executable (PE) files and use the large-scale pre-training language model MiniLM to generate the node features of CFG. The extracted CFG is compressed to a feature vector with GIN and classified with Multi-Layer Perceptron. To evaluate our approach, we made a CFG-based malware detection dataset from PE files of the Dike Dataset, which we call the Malware Geometric Dataset (MGD), and collected the results. The evaluation results show that our proposal demonstrated 0.9977 in the Area Under Curve metric and achieved a 97.44 % detection rate when the False Positive Rate was 0.1 %.

Ransomware is malware that encrypts a victim's files and demands payment, typically in cryptocurrencies, to restore access. The global spread of ransomware attacks underscores the need for automatic detection systems. Signature‐based antivirus tools cannot identify novel ransomware strains, while machine learning (ML) and deep learning (DL) models offer promise because they learn data patterns and generalize to new unseen samples. However, the choice of model architecture remains an open question: do classical ML methods suffice or do DL methods bring a considerable performance increase for static malware analysis? In this paper, we benchmark three classic algorithms (Logistic Regression, Random Forest and XGBoost) and two deep architectures (Multi‑Layer Perceptron and TabNet) on the Windows Portable Executable (PE) ransomware detection dataset from Kaggle. We perform comprehensive data preprocessing, 5‑fold stratified cross‑validation, and compute performance metrics like accuracy, precision, recall, F1 score, area under the ROC curve (AUC), and Matthews correlation coefficient (MCC). We further examine the feature importance, plot receiver operating characteristic (ROC) curves, and discuss the computational expense and interpretability of each method. Our experiments show that tree ensemble methods (Random Forest and XGBoost) achieve near‑perfect detection with very low false positives and negatives, outperforming Logistic Regression and MLP. Deep models offer no significant advantages for this tabular data set but with higher computational expense. We discuss the implications of these results for ransomware defense in practice and propose directions for future research.

Present day malware shows stealthy and dynamic capability to avail administrative rights and control the victim computer [10]. Malware writers depend on evasion techniques like code obfuscation, packing, compression, encryption or polymorphism to avoid detection by Anti-Virus (AV) scanners as AV primarily use signature based detection. According to the FireEye Threat report second half of 2011 [15], top 50 malware have generated 80% infections. Malware like Zues, Conficker, Koobface have become more stealthy by use of pay per install toolkits like Blackhole [15]. Pay per install toolkits make the samples dynamic in nature. This has led to exponential increase of unknown, zero-day malware [14].To complement the signatured approach, a good behavioral scheme is imminent due to exponential increase in number of encoded malware samples. Behavioural analysis can detect unknown, encrypted, zero day malware, but these methods result in increased false alarm rate. We propose a behaviour model that represents abstraction of a binary by analyzing the Application Programming Interface (API) strings made by Windows Portable Executable (PE) [25] files. Our focus is based on extracting temporal snapshots of malware and benign executables known as API Call-grams, as API strings are primarily written for software development kits to generate sane code. Malcode writers misues the available functionality to keep the code compact and escape being detected by AV software.

One of the main challenges in security today is defending against unknown malware attacks which have the potential to harm a computer or network. Hence, detecting malware has become one of the most important challenges for the security of computer systems. The known malware detection methods based on the appearance of opcode sequences has to construct a matrix from programs of different architectures to extract high-level features. In order to resolve high dimensional inputs vector and differences assembly instruction, this paper proposes a novel method for detecting static characteristics of 32-bit and 64-bit malicious Portable Executable (PE) Windows files by opcode sequences analysis. By compute the frequency of occurrence of each opcode sequence and distinguishing different types of 32-bit and 64-bit PE files, the proposed method shows promising results with less complexity in comparison with previous studies, which is beneficial to train machine learning model such as k-nearest neighbor (KNN) and back-propagation neural network (BP). Our method is evaluated on more than 20,000 samples, and experimental results show that our system can effectively detect and classify unknown malware.

Classification and detection of zero-day attacks remain a significant challenge within the domain of cybersecurity. Due to the vast types of malware families and the presence of an imbalanced dataset, real-time detection and classification become increasingly complex and inaccurate. Thus, there’s an urgent need to develop an intelligent and adaptive defense mechanism capable of identifying and classifying such attacks with improved precision and robustness. This paper proposed a stacked ensemble federated learning model with an accuracy-aware node weighting scheme to address the challenges posed by inter- and intra-class similarities among different types of malwares. In the initial phase, malware Portable Executable (PE) files are collected from multiple online repositories and validated by three different antivirus programs through VirusTotal to ensure reliability. These validated files are then converted into image form and categorized into 28 families to facilitate feature extraction. In the second phase, deep feature representations are extracted through a transfer learning-based fine-tuned ResNet-50 model, which captures both low-level and high-level patterns that are relevant to malware classification. After feature extraction from multiple distributed nodes, architecture is fed into the novel proposed Ensemble Stacked Federated Model for enhanced generalization and robust classification. The model is tested on both private and publicly available datasets. The experimental results demonstrate that the proposed method outperforms existing baseline approaches in terms of accuracy and computational efficiency. This improvement is achieved because it performs independent training at each federated node separately and then stacks their outputs with a central ensemble model, which enhances the learning rate and reduces overfitting. The code used for the experiments is available here.

Determining potential applications for personalized college English instruction through deep feature extraction and scene feature mining is essential for enabling Deep Learning assisted AI systems to generate effective educational outcomes. Traditional instructional approaches often struggle within dynamic digital environments, as they overlook variations in students’ learning styles, engagement levels, and behavioral patterns. This research proposes a hybrid methodology that integrates Convolutional Neural Networks for deep audio feature extraction with Random Forest algorithms for contextual and scene feature mining. The objective is to construct a personalized learning environment capable of adapting dynamically in real time to individual learner behaviors and preferences. A hybrid deep learning model is introduced to predict teaching effectiveness using speech related features such as clarity, rhythm, and engagement level. Feature fusion is employed by combining audio based deep features with contextual scene features to classify teaching effectiveness into three categories: low, medium, and high. Experimental results demonstrate strong predictive performance, achieving an accuracy of 0.9900, precision of 0.9906, recall of 0.9907, and an F1 measure of 0.9905, with minimal performance fluctuations. This AI assisted instructional framework enables college English teachers to enhance teaching quality by receiving immediate, data driven feedback, ultimately fostering improved student centered learning experiences.

The Portable executable file format is the format of executables, object code and DLL’s (Dynamic Link Library) in Microsoft Windows Operating system. It is the standard of how executable files are organized within file system in Microsoft Windows. The Portable executable file format was designed for Windows NT 3.1 that released in 1993. Many of its features are inherited from COFF (Common object file format) used in Unix Operating systems. It consists of header and sections; headers are rules that tell windows loader how the section should be mapped and loaded into memory. The section are nothing but the data or content. Different sections hold different kind of data for example code section contains executable code while bss (Block Started by Symbol) sections contains uninitialized data. Portable executable file could be used in a way to dent the security of computer, therefore great care should be taken while downloading and running Portable executable files. In our work we develop a static malware analysis tool called ‘Portable Executable File Analysis Framework (PEFAF)’ using data mining techniques. A collection of 8 thousand benign and 7 thousand malicious files were used in this work. We extracted 60 features, analyzed them and found that 34 of them are significant for the detection of malware threats. Based on these 34 indicators our tool classifies input file into malicious or non-malicious.