An Opcode Sequences Analysis Method For Unknown Malware Detection

As malicious programs, Trojan horses have become a huge threat to computer networks security. Trojan horses can easily cause loss, damage or even theft of data because they are usually disguised as something useful or desirable, and are always mistakenly activated by computer users, corporations and other organizations. Thus, it is important to adopt an effective and efficient method to detect the Trojan horses, and the exploration of a new method of detection is of greater significance. Scientists and experts have tried many approaches to detecting Trojan horses since they realized the harms of the programs. Up to now, these methods fall mainly into two categories [2]. The first category is to detect Trojan horses through checking the port of computers since the Trojan horses send out message through computer ports [2]. However, these methods can only detect the Trojan horses that are just working when detected. The second class is to detect Trojan horses by examining the signatures of files [2] [19], in the same way as people deal with computer virus. As new Trojan horses may contain unknown signatures, methods in this category may not be effective enough when new and unknown Trojan horses appear continuously, sending out unknown signatures that escape detection. For the above-mentioned reasons, without exception, there are limitations in the existing methods if the un-awakened and unknown Trojan horses are to be detected. This thesis proposes a new method that can detect un-awakened and unknown Trojan horses- the detection by using of a file's static characteristics. This thesis takes PE file format as the object of the research, because approximately 75% of personal computers worldwide are installed the Microsoft Windows [4], and that Trojan horses usually exist as a Portable Executable (PE) file in the Windows platform. Based on the (PE) file format, my research gets all the static information of each part of PE file which is characteristic of a file. Then, this static information is analyzed by the intelligent information processing techniques. Next, a detection model is established to estimate whether a PE file is a Trojan horse. This model can detect the unknown Trojan horses by analyzing static characteristics of a file. The information that is used to verify detecting model is new and unknown to the detecting model; in other words, the information is not used during the training of the model. The thesis is organized as follows. First, this thesis discusses the limitations of traditional detection techniques, related works of research, and a new method to detect Trojan horse based on file's static information. Second, the thesis focuses on the research of the Trojan horse detecting models, covering the extracting of the static information from PE file, choice of intelligent information processing techniques, and setting up the Trojan horse detecting model. Lastly, the thesis discusses the direction of future research in this field.

PE (Portable executable) file is a standard format for executable file and is applied extensively. PE file has diversity, uncertainty of file size, complexity of file structure and singleness of file format, which make PE file easy to be a carrier of information hiding, especially for that of large hiding capacity. A novel software information hiding algorithm is proposed, which makes full use of display characteristics of palette icon of portable executable file. In this algorithm, the information is embedded into the transparent area of the icon by taking advantage of the redundant color items in the palette. The experimental results show that after embedding the information, the size of the icon remains unchanged, that is, the size of the resource section will not change, and the size of the PE file will not change. On the other hand, the icon with embedded data can be correctly analyzed and displayed without any distortion. PE file can run normally, and does not affect the performance of the program, so the algorithm has good concealment. The algorithm selects an index value whose color is black according to rules in the XOR bitmap, these index values are encoded for information hiding, its complexity is low. At the same time, we can further improve the hiding capacity by adding one or more icons to PE file.

Executable files such as .exe, .bat, .msi etc. are used to install the software in Windows-based machines. However, downloading these files from untrusted sources may have a chance of having maliciousness. Moreover, these executables are intelligently modified by the anomalous user to bypass antivirus definitions. In this paper, we propose a method to detect malicious executables by analyzing Portable Executable (PE) files extracted from executable files. We trained a supervised binary classifier using features extracted from the PE files of normal and malicious executables. We experimented our method on a large publicly available dataset and reported more than 95% of classification accuracy.

This study investigates whether convolutional neural networks (CNNs) trained on visual representations of Portable Executable (PE) files can rival traditional machine learning classifiers trained on engineered features. A dataset of over 200,000 PE files [1] was used to derive two feature sets (Basic and Ember-Lite) [2] and to generate 256x256 greyscale and entropy images [3],[4]. Three CNNs (SimpleCNN, ResNet-18 [5], EfficientNet-B0 [6]) were trained and evaluated against five baselines (Random Forest, XGBoost [7], CatBoost [8], LightGBM, Logistic Regression). Tree-based models with enriched features achieved the highest scores, with CatBoost reaching a ROC-AUC of 0.990. The best CNN, EfficientNet-B0 on entropy images, obtained a ROC-AUC of 0.954. Although CNNs did not surpass feature-based models, they showed competitive results when feature engineering was constrained. These findings indicate that visual approaches offer a promising alternative for static malware detection, particularly when combined with entropy-based representations [9].

It is a difficult issue in the anti-virus field about how to detect unknown and packed PE (Portable Executable) viruses effectively, and existing schemes for anti-virus detection are dissatisfactory. Based on an analysis of the logical structures of Windows PE file and PE virus, a new method of PE virus detection is presented here, which exploits fragile software watermarking technique for virus detection. It describes how the new scheme embeds a software watermark into the PE files for virus detection. The experiment shows that the new method has a higher rate in detecting unknown PE virus than some existing ones, and can detect directly the packed PE virus.

Detecting malware is crucial for ensuring the security of computer systems. Traditional machine learning models face challenges in effectively detecting malware, mainly due to the class imbalance problem, where the number of malware samples is significantly smaller than that of non-malware samples. Additionally, malware’s dynamic and evolving nature, continuously altering its structure and tactics, presents a substantial challenge for conventional artificial intelligence algorithms, further complicating the detection task. In pursuing an optimized malware detection technique, researchers initially explored traditional machine learning algorithms, focusing on the features of Portable Executable (PE) file headers. However, the inherent issues, such as imbalanced datasets and the deceptive nature of malware, have raised concerns about the credibility of the attained results. This can result in misclassifying malware as non-malware, leading to security vulnerabilities. One-Class Classification (OCC) methods have emerged as a promising approach to improve the detection of unknown malware. However, traditional OCC approaches face the challenge of the curse of dimensionality. This research proposes adapting subspace learning-based OCC methods to overcome the curse of dimensionality and effectively handle the class imbalance problem. We propose a pipeline for detecting malware using methods that jointly optimize a subspace and data description for OCC. We evaluate the performance of various one-class classifiers on three different datasets. We observed that the subspace-learning-based OCC is a promising approach. Evaluating various classifiers on three datasets reveals promising results, with a True Positive Rate (TPR) of 100 % for subspace-learning-based OCC. The proposed pipeline can serve as a valuable tool for improving the security of computer systems by accurately detecting malware and protecting against potential attacks.

Distinguishing malicious programs based on visualization and hybrid learning algorithms

Malware writers often develop malware with automated measures, so the number of malware has increased dramatically. Automated measures tend to repeatedly use significant modules, which form the basis for identifying malware variants and discriminating malware families. Thus, we propose a novel visualization analysis method for researching malware similarity. This method converts malicious Windows Portable Executable (PE) files into local entropy images for observing internal features of malware, and then normalizes local entropy images into entropy pixel images for malware classification. We take advantage of the Jaccard index to measure similarities between entropy pixel images and the k-Nearest Neighbor (kNN) classification algorithm to assign entropy pixel images to different malware families. Preliminary experimental results show that our visualization method can discriminate malware families effectively.

Traditional malware detection methods cannot keep up with the massive amount of newly created malware quickly and effectively. Machine learning is a promising method for the detection and classification of large-scale newly created malware according to the features of samples. The current research trend is to use machine learning technology, such as the Gradient Boosting Decision Tree (GBDT) and deep neural network technology, to learn newly created malware rapidly and accurately. We propose Control-Flow Graph (CFG)- and Graph Isomorphism Network (GIN)-based malware classification, where we first extract the CFG from portable executable (PE) files and use the large-scale pre-training language model MiniLM to generate the node features of CFG. The extracted CFG is compressed to a feature vector with GIN and classified with Multi-Layer Perceptron. To evaluate our approach, we made a CFG-based malware detection dataset from PE files of the Dike Dataset, which we call the Malware Geometric Dataset (MGD), and collected the results. The evaluation results show that our proposal demonstrated 0.9977 in the Area Under Curve metric and achieved a 97.44 % detection rate when the False Positive Rate was 0.1 %.

The Portable executable file format is the format of executables, object code and DLL’s (Dynamic Link Library) in Microsoft Windows Operating system. It is the standard of how executable files are organized within file system in Microsoft Windows. The Portable executable file format was designed for Windows NT 3.1 that released in 1993. Many of its features are inherited from COFF (Common object file format) used in Unix Operating systems. It consists of header and sections; headers are rules that tell windows loader how the section should be mapped and loaded into memory. The section are nothing but the data or content. Different sections hold different kind of data for example code section contains executable code while bss (Block Started by Symbol) sections contains uninitialized data. Portable executable file could be used in a way to dent the security of computer, therefore great care should be taken while downloading and running Portable executable files. In our work we develop a static malware analysis tool called ‘Portable Executable File Analysis Framework (PEFAF)’ using data mining techniques. A collection of 8 thousand benign and 7 thousand malicious files were used in this work. We extracted 60 features, analyzed them and found that 34 of them are significant for the detection of malware threats. Based on these 34 indicators our tool classifies input file into malicious or non-malicious.

Ransomware attacks are a growing threat to organizations worldwide, with sensitive data encrypted and held hostage for ransom. Although traditional ransomware detection methods, such as signature-based and heuristic detection, are effective to some extent, they struggle to identify ransomware before encryption begins due to evasion techniques like code obfuscation and polymorphism. This work aims to address this gap by developing a detection method that can identify ransomware early by analyzing static and dynamic features. The system tests ransomware samples using static analysis of Portable Executable (PE) files without execution and dynamic analysis in a sandbox environment to capture pre-encryption behaviors and features. A set of state-of-the-art machine learning algorithms is employed to classify ransomware activity based on these behavioral patterns. The goal is to identify ransomware activity before it executes encryption. By establishing a framework for early ransomware detection, this study provides a pathway for scalable detection systems that can adapt to evolving ransomware threats.

A novel deep learning-based approach for malware detection

The rapid rate of creating malware variants requires advanced detection techniques beyond traditional signature-based techniques. This paper discusses an advanced Behavioral DNA Fingerprinting technique that combines deep learning with cryptographic fingerprinting to detect and re-identify malware. Our technique extracts deep behavioral features from Windows Portable Executable (PE) files, generates unique cryptographic fingerprints, and maintains an up-to-date database for efficient threat detection. Experimental evaluation using the EMBER dataset achieves excellent performance with 91.43% accuracy, 91.66% precision, and 91.16% recall. The fingerprint repository achieves perfect matching accuracy, facilitating efficient re-identification of duplicate malware samples without sacrificing a balanced threat signature coverage.

Classification and detection of zero-day attacks remain a significant challenge within the domain of cybersecurity. Due to the vast types of malware families and the presence of an imbalanced dataset, real-time detection and classification become increasingly complex and inaccurate. Thus, there’s an urgent need to develop an intelligent and adaptive defense mechanism capable of identifying and classifying such attacks with improved precision and robustness. This paper proposed a stacked ensemble federated learning model with an accuracy-aware node weighting scheme to address the challenges posed by inter- and intra-class similarities among different types of malwares. In the initial phase, malware Portable Executable (PE) files are collected from multiple online repositories and validated by three different antivirus programs through VirusTotal to ensure reliability. These validated files are then converted into image form and categorized into 28 families to facilitate feature extraction. In the second phase, deep feature representations are extracted through a transfer learning-based fine-tuned ResNet-50 model, which captures both low-level and high-level patterns that are relevant to malware classification. After feature extraction from multiple distributed nodes, architecture is fed into the novel proposed Ensemble Stacked Federated Model for enhanced generalization and robust classification. The model is tested on both private and publicly available datasets. The experimental results demonstrate that the proposed method outperforms existing baseline approaches in terms of accuracy and computational efficiency. This improvement is achieved because it performs independent training at each federated node separately and then stacks their outputs with a central ensemble model, which enhances the learning rate and reduces overfitting. The code used for the experiments is available here.

The internet’s rapid growth has resulted in an increase in the number of malicious files. Recently, powershell scripts and Windows portable executable (PE) files have been used in malicious behaviors. To solve these problems, artificial intelligence (AI) based malware detection methods have been widely studied. Among AI techniques, the graph convolution network (GCN) was recently introduced. Here, we propose a malicious powershell detection method using a GCN. To use the GCN, we needed an adjacency matrix. Therefore, we proposed an adjacency matrix generation method using the Jaccard similarity. In addition, we show that the malicious powershell detection rate is increased by approximately 8.2% using GCN.