Efficient Dynamic Malware Analysis Based on Network Behavior Using Deep Learning

Malware-infected hosts have typically been detected using network-based Intrusion Detection Systems on the basis of characteristic patterns of HTTP requests collected with dynamic malware analysis. Since attackers continuously modify malicious HTTP requests to evade detection, novel HTTP requests sent from new malware samples need to be exhaustively collected in order to maintain a high detection rate. However, analyzing all new malware samples for a long period is infeasible in a limited amount of time. Therefore, we propose a system for efficiently collecting HTTP requests with dynamic malware analysis. Specifically, our system analyzes a malware sample for a short period and then determines whether the analysis should be continued or suspended. Our system identifies malware samples whose analyses should be continued on the basis of the network behavior in their short-period analyses. To make an accurate determination, we focus on the fact that malware communications resemble natural language from the viewpoint of data structure. We apply the recursive neural network, which has recently exhibited high classification performance in the field of natural language processing, to our proposed system. In the evaluation with 42,856 malware samples, our proposed system collected 94% of novel HTTP requests and reduced analysis time by 82% in comparison with the system that continues all analyses.

Malware classification plays an important role in tracing the attack sources of computer security. However, existing static analysis methods are fast in classification, but they are inefficient in some malware using packing and obfuscation techniques; the dynamic analysis methods have better universality for packing and obfuscation, but they will cause excessive classification cost. To overcome these shortcomings, in this paper, we propose a classification system Malscore based on the probability scoring and machine learning, which sets the probability threshold to concatenate static analysis (called Phase 1) and dynamic analysis (called Phase 2). The convolutional neural networks with spatial pyramid pooling were used to analyze the grayscale images (static features) in Phase 1, and the variable n-grams and machine learning were used to analyze the native API call sequences (dynamic features) in Phase 2. Malscore combined static analysis with dynamic analysis not only accelerated the static analysis process by taking advantage of the CNN in image recognition but also appeared to be more resilient to obfuscation by the dynamic analysis. Different from other static and dynamic analysis techniques, when malware is detected, due to the fact that malware will most likely be labeled only by static analysis, we could reduce the overheads by dynamically analyzing a few malware that has less obvious features or greater confusion in static analysis. We performed experiments on 174607 malware samples from 63 malware families. The result showed that Malscore achieved 98.82% accuracy for malware classification. Furthermore, Malscore was compared with the method of using static and dynamic analysis. The preprocessing and test time represented a reduction of 59.58% and 61.70%, respectively.

In this book, we sought to capture the intrinsic natures in malicious behaviors, in order to build more effective automatic malware analysis systems. We proposed TEMU as a dynamic binary analysis platform. Then on top of TEMU, we proposed and built a series of novel techniques for automatic malware analysis, including Renovo, Panorama, HookFinder, and MineSweeper, for detecting and analyzing various aspects of malware. Since these techniques capture intrinsic characteristics of malware, they are well suited for dealing with new malware samples and attack mechanisms. We also systematically discussed several fundamental limitations in our proposed techniques.More concretely, we pointed out that although our analysis platform is better suited for analyzing malicious code than the other conventional ones (e.g., debugger and disassembler), malware authors may still find ways to detect and evade it. Moreover, an open problem for dynamic analysis lies in its limited test coverage. Finally, as a core analysis technique, dynamic taint analysis has several limitations, including taint explosion and implicit information flow. Such discussions shed light on future directions for automatic malware analysis.KeywordsAutomated Malware AnalysisTaint AnalysisImplicit Information flowCore Analysis TechniquesMalware SamplesThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

To handle the growing flood of malware, security vendors and analysts rely on tools that automatically identify and analyze malicious code. Current systems for automated malware analysis typically follow a dynamic approach, executing an unknown program in a controlled environment (sandbox) and recording its runtime behavior. Since dynamic analysis platforms directly run malicious code, they are resilient to popular malware defense techniques such as packing and code obfuscation. Unfortunately, in many cases, only a small subset of all possible malicious behaviors is observed within the short time frame that a malware sample is executed. To mitigate this issue, previous work introduced techniques such as multipath or forced execution to increase the coverage of dynamic malware analysis. Unfortunately, using these techniques is potentially expensive, as the number of paths that require analysis can grow exponentially. In this paper, we propose REANIMATOR, a novel solution to determine the capabilities (malicious functionality) of malware programs. Our solution is based on the insight that we can leverage behavior observed while dynamically executing a specific malware sample to identify similar functionality in other programs. More precisely, when we observe malicious actions during dynamic analysis, we automatically extract and model the parts of the malware binary that are responsible for this behavior. We then leverage these models to check whether similar code is present in other samples. This allows us to statically identify dormant functionality (functionality that is not observed during dynamic analysis) in malicious programs. We evaluate our approach on thousands of realworld malware samples, and we show that our system is successful in identifying additional, malicious functionality. As a result, our approach can significantly improve the coverage of malware analysis results.

Increased number of malware samples have created many challenges for Antivirus companies. One of these challenges is clustering the large number of malware samples they receive daily. Malware authors use malware generation kits to create different instances of the same malware. So most of these malicious samples are polymorphic instances of previously known malware family only. Clustering these large number of samples rapidly and accurately without spending much time on processing the sample have become a critical requirement. In this paper we proposed, implemented and evaluated a method, called ByteFreq that can cluster large number of samples using byte frequency. Byte frequency is represented as time series and SAX (Symbolic Aggregation approXimation)[1] is used to convert the time series in symbolic representation. We evaluated proposed system on real world malware samples and achieved 0.92 precision and 0.96 recall accuracy.

This paper presents a method to extract important byte sequences in malware samples by application of convolutional neural network (CNN) to images converted from binary data. This method, by combining a technique called the mechanism into CNN, enables calculation of an attention map, which shows regions having higher importance for classification in the image. The extracted region with higher importance can provide useful information for human analysts who investigate the functionalities of unknown malware samples. Results of our evaluation experiment using malware dataset show that the proposed method provides higher classification accuracy than a conventional method. Furthermore, analysis of malware samples based on the calculated map confirmed that the extracted sequences provide useful information for manual analysis.

Distinguishing malicious programs based on visualization and hybrid learning algorithms

Due to their damage to Internet security, malware and phishing website detection has been the Internet security topics that are of great interests. Compared with malware attacks, phishing website fraud is a relatively new Internet crime. However, they share some common properties: 1) both malware samples and phishing websites are created at a rate of thousands per day driven by economic benefits; and 2) phishing websites represented by the term frequencies of the webpage content share similar characteristics with malware samples represented by the instruction frequencies of the program. Over the past few years, many clustering techniques have been employed for automatic malware and phishing website detection. In these techniques, the detection process is generally divided into two steps: 1) feature extraction, where representative features are extracted to capture the characteristics of the file samples or the websites; and 2) categorization, where intelligent techniques are used to automatically group the file samples or websites into different classes based on computational analysis of the feature representations. However, few have been applied in real industry products. In this paper, we develop an automatic categorization system to automatically group phishing websites or malware samples using a cluster ensemble by aggregating the clustering solutions that are generated by different base clustering algorithms. We propose a principled cluster ensemble framework to combine individual clustering solutions that are based on the consensus partition, which can not only be applied for malware categorization, but also for phishing website clustering. In addition, the domain knowledge in the form of sample-level/website-level constraints can be naturally incorporated into the ensemble framework. The case studies on large and real daily phishing websites and malware collection from the Kingsoft Internet Security Laboratory demonstrate the effectiveness and efficiency of our proposed method.

Manual Summarization of large bodies of text involves a lot of human effort and time, especially in the legal domain. Lawyers spend a lot of time preparing legal briefs of their clients' case files. Automatic Text summarization is a constantly evolving field of Natural Language Processing(NLP), which is a subdiscipline of the Artificial Intelligence Field. In this paper a hybrid method for automatic text summarization of legal cases using k-means clustering technique and tf-idf(term frequency-inverse document frequency) word vectorizer is proposed. The summary generated by the proposed method is compared using ROGUE evaluation parameters with the case summary as prepared by the lawyer for appeal in court. Further, suggestions for improving the proposed method are also presented.

Android being the most popular open source mobile operating system, attracts a plethora of app developers. Millions of applications are developed for Android platform with a great extent of behavioral diversities and are available on Play Store as well as on many third party app stores. Due to its open nature, in the past Android Platform has been targeted by many malware writers. The conventional way of signature-based detection methods for detecting malware on a device are no longer promising due to an exponential increase in the number of variants of the same application with different signatures. Moreover, they lack in dynamic analysis too. In this paper, we propose DRACO, which employs a two-phase detection technique that blends the synergy of both static and dynamic analysis. It has two modules, client module that is in the form an Android app and gets installed on mobile devices and a server module that runs on a server. DRACO also explains user about the features contributing to the maliciousness of analyzed app and generates scoring for that maliciousness. It does not require any root or super-user privileges. In an evaluation of 18,000 benign applications and 10,000 malware samples, DRACO performs better than several related existing approaches and detects 98.4% of the malware with few false alerts. On ten popular smartphones, the method requires an average of 6 seconds for on device analysis and 90 seconds on server analysis.

The increased spread of malicious software (malware) through the internet remains a serious threat. Malware authors use obfuscation and deformation techniques to generate new types than can evade traditional detection methods. Hence, it is widely expected that machine learning methods can classify malware and cleanware based on the characteristics of malware samples. This paper investigates malware classification accuracy using static methods for malware detection based on LightGBM by a custom log loss function, which controls learning by installing coefficient α to a loss function of the false-negative side and coefficient β to a loss function of the false-positive side. By installing coefficients, we can create a lopsided classifier. We used two malware datasets, non-public and public, to construct a malware baseline model to verify the effectiveness of the proposed method. We extracted the dataset features from PE-file surface analysis and PE-header dumps and customized a binary log loss function to improve all the classification evaluation metrics to a certain extent. We obtained a better result (AUC = 0.979) at α = 430 and β = 339 than the normal log loss function (AUC = 0.978) on the EMBER dataset. In addition, to maintain malware detection coverage and quick countermeasures to true positive results, we propose a hybrid usage of different custom models to prioritize positive results.

The Internet of Things (IoT) provides the infrastructure for global social connectivity, where everything is connected, and information exchanged over the Internet. Because of this peculiar, the amount of botnet malware infecting IoT devices to carry out attacks such as DDoS, cryptocurrency malicious mining, keylogger, etc., is expanding at a fast pace. Therefore, detecting the IoT botnet is essential. Most IoT botnet detection approaches use either static analysis features or dynamic analysis features, and rarely integrate them as detection features, also no more work is consumed integrating the two types of features. In this paper, we propose a first method that utilizes the integration of both static and dynamic features with machine learning classifiers for distinguishing IoT botnet from benign samples. Experiments show that our proposed method can obtain an accuracy more than 99% on a total of 6520 samples with 4644 IoT botnet samples. Besides, we show that using integrated features for IoT botnet detection outperforms using static feature or dynamic feature alone, and other studies.

Visual question answering (VQA) is receiving increasing attention from researchers in both the computer vision and natural language processing fields. There are two key components in the VQA task: feature extraction and multi-modal fusion. For feature extraction, we introduce a novel co-attention scheme by combining Sentence-guide Word Attention (SWA) and Question-guide Image Attention in a unified framework. To be specific, the textual attention SWA relies on the semantics of the whole question sentence to calculate contributions of different question words for text representation. For the multi-modal fusion, we propose a “Cross-modal Multistep Fusion (CMF)” network to generate multistep features and achieve multiple interactions for two modalities, rather than focusing on modeling complex interactions between two modals like most current feature fusion methods. To avoid the linear increase of the computational cost, we share the parameters for each step in the CMF. Extensive experiments demonstrate that the proposed method can achieve competitive or better performance than the state of the art.

Malware attacks are growing years after years because of increasing android, IOT along with traditional computing devices. To protect all these devices malware analysis is necessary so that interest of the organizations and individuals can be protected. There are different approaches of malware analysis like static, dynamic and heuristic. As the technology is advancing malware authors also use the advanced malware attacking techniques like obfuscation and packing techniques, which cannot be detect by signature based on static approaches. To overcome all these problems behavior of malware must be analyzed using dynamic approaches. Now a days malware author using some more advanced evasion techniques in which malware suspends its malicious behavior after detecting virtual environment. So, evasion techniques give a new challenge to malware analysis because even dynamic approach some time fails to detect and analyze the malwares.

New malware outbreaks cannot provide thousands of training samples which are required to counter malware campaigns. In some cases, there could be just one sample. So, the defense system at the firing line must be able to quickly detect many automatically generated variants using a single malware instance observed from the initial outbreak by statically inspecting the binary executables. As previous research works show, statistical features such as term frequency-inverse document frequency and n-gram are significantly vulnerable to attacks by mutation through reinforcement learning. Recent studies focus on raw binary executable as a base feature which contains instructions describing the core logic of the sample. However, many approaches using image-matching neural networks are insufficient due to the malware mutation technique that generates a large number of samples with high entropy data. Deriving instruction cognitive representation that disambiguates legitimate instructions from the context is necessary for accurate detection over raw binary executables. In this paper, we present a novel method of detecting semantically similar malware variants within a campaign using a single raw binary malware executable. We utilize Discrete Fourier Transform of instruction cognitive representation extracted from self-attention transformer network. The experiments were conducted with in-the-wild malware samples from ransomware and banking Trojan campaigns. The proposed method outperforms several state of the art binary classification models.