ProDigger: Towards Robust Automatic Network Protocol Fingerprint Learning via Byte Embedding

Protocol fingerprinting (PF) focuses on the capability to derive a series of distinguishable features for recognizing which protocol or application generated the network traffic. Unfortunately, deep packet inspection (DPI), a widely adopted method for PF, requires significant expert effort to develop and maintain protocol signatures. Additionally, the new solution paradigm, deep flow inspection (DFI) using machine learning for PF, also relies on hand-designed features. In this paper, we present ProNet, a payload based approach to protocol fingerprinting, which overcomes the limitation of artificial feature engineering. The key novelty of ProNet is two-fold: (i) it takes generic, raw short packet payloads as input, instead of the typical flow-statistical-features (e.g., port, packet size, packet-interval); (ii) it learns to simultaneously extract features via convolutional operations on the byte-level embeddings and ngram-level embeddings. We implement and evaluate ProNet on real-world traces, including DNS, QQLive, PPLive, PPStream, SopCast, DHCP, NBNS, HTTP, SMTP and SMB. Our experiment results show that ProNet achieves over 99% precision and recall with low false-positives (less than 1%) and nearly no false-negatives.

The thirteen articles in this special section explore the technology of deep packet inspection (DPI). DPI examines the content in packet payloads to search for signatures of network applications, signs of malicious activities, and leaks of sensitive information, rather than just examine packet headers for information such as IP addresses and port numbers. The inspection provides network devices with rich information of application protocol messages in packet payloads, and enables them to make intelligent decisions in packet processing based on the information. The papers are organized into the following four sections: (1) Scalable Algorithms and Architectures for DPI, (2) Network Traffic Analysis with DPI, (3) Network Protocol Identification with DPI, and (4) Network Security Analysis with DPI.

There is an increasing demand for network devices to perform deep packet inspection (DPI) in order to enhance network security. In DPI, the packet payload is compared against a set of predefined patterns that can be specified using regular expressions (regexes). It is well-known that mapping regexes to deterministic finite automaton (DFA) may suffer from the state explosion problem. Through observation, we attribute DFA explosion to the necessity of remembering matching history. In this paper, we investigate how to manage matching history efficiently and propose an extended DFA approach for regex matching called fcq-FA, which can make a memory size reduction of about 1,000 times with a fully automated approach. In fcq-FA, we use pipeline queues and counters to help record the matching history. Hence, state explosion caused by Kleene closure and length restriction can be completely avoided. Furthermore, it achieves a fully automated signature compilation with polynomial running time and space. The equivalence between fcq-FA and the traditional DFA is guaranteed by a strict theoretical proof, which means fcq-FA can process all the regexes supported by the traditional DFA.

Many network security applications such as Intrusion Detection System (IDS), Firewall and Data Loss Prevention System (DLPS) are based on deep packet inspection, in this packets header as well as payload of the packets are checked with predefined attack signature to identify whether it contains malicious traffic or not. To perform this checking different pattern matching methods are used by NIDS. The most popular method to implement pattern matching is to use of Finite Automata (FA). Generally, regular expressions are used to represent most of the attack signatures defined by NIDS. They are implemented using finite automata, which takes the payload of packet as input string. However, existing approaches of Finite Automata (FA), both deterministic finite automata (DFA) and non-deterministic finite automata (NFA) for pattern matching are having their own advantages and some drawbacks. The DFA based pattern matching methods are fast enough but require more memory. However, NFA based pattern matching methods are comparatively takes less memory but the speed of matching is very slow, to overcome these drawbacks of finite automata there are many approaches have been proposed. This paper discuses comparative study of some Finite Automata (FA) based techniques for pattern matching in network intrusion detection system (NIDS).

Large scale applications in data centers are composed of computers connected with a network. Traditional network switches do not perform routing based on packet contents. Thus, packets cannot be transmitted to the optimal computer for the request which is written in the packet payload. On the other hand, Deeply Programmable Network (DPN) switches can completely analyze packet payloads and perform routing based on their contents. In this paper, we focus on packet routing based on Deep Packet Inspection (DPI), i.e. analyzing packet payload, using DPN switches, and discuss performance of applications on a network using application aware DPN switches. We introduce application aware network and propose a method for improving application performance with DPN switch support. We evaluate our method with the several applications and show that our method can increase application performance.

Social relationship discovery of IP addresses in the managed IP networks by observing traffic at network boundary

L7-filter is a significant deep packet inspection (DPI) extension to Netfilter in Linux's QoS framework. It classifies network traffic based on information hidden in the packet payload. Although the computationally intensive payload classification can be accelerated with multiple processors, the default OS scheduler is oblivious to both the software characteristics and the underlying multicore architecture. In this paper, we present a parallelized L7-filter algorithm and an efficient scheduler technique for multicore servers. Our multithreaded L7-filter algorithm can process the incoming packets on multiple servers boosting the throughput tremendously. Our scheduling algorithm is based on Highest Random Weight (HRW), which maintains the connection locality for the incoming traffic, but only guarantees load balance at the connection level. We present an Adapted Highest Random Weight (AHRW) algorithm that enhances HRW by applying packet-level load balancing with an additional feedback vector corresponding to the queue length at each processor. We further introduce a Hierarchical AHRW (AHRW-tree) algorithm that considers characteristics of the multicore architecture such as cache and hardware topology by developing a hash tree architecture. The algorithm reduces the scheduling overhead to <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> (log <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> ) instead of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> ) and produces a better balance between locality and load balancing. Results show that the AHRW-tree scheduler can improve the L7-filter throughput by about 50% on a Sun-Niagara-2-based server compared to a connection locality-based scheduler. Although extensively tested for L7-filter traces, our technique is applicable to many other packet processing applications, where connection locality and load balancing are important while executing on multiple processors. With these speedups and inherent software flexibility, our design and implementation provide a cost-effective alternative to the traffic monitoring and filtering ASICs.

With the increasing internet service complexity, providing secured quality service has become a major concern. In the earlier systems, data was believed to be safer on internet without being intercepted. Now these internet vulnerabilities cannot be ignored as these weaknesses are used by many to carry out malicious activities. In order to tackle these problems, internet service providers are trying to find better options. One such technique gaining popularity in the recent decade is Deep Packet Inspection (DPI), which can be provided using software or hardware methods. It is reported that hardware is providing better solution than software. In this review article, we have introduced one such hardware, Ternary Content Addressable Memory (TCAM), which could perform complete packet inspection (packet header and payload inspection). In the first section, we have focused on evolution of packet filtering system. Later the discussion is divided into two parts: (i) packet classification using TCAM, and (ii) payload inspection (pattern matching and regular expression) using TCAM.

Application-level protocol specifications are helpful for network security management, including intrusion detection and intrusion prevention which rely on monitoring technologies such as deep packet inspection. Moreover, detailed knowledge of protocol specifications is also an effective way of detecting malicious code. However, current methods for obtaining unknown and proprietary protocol message formats (i.e., no publicly available protocol specification), especially binary protocols, highly rely on manual operations, such as reverse engineering which is time-consuming and laborious. In this paper, we propose Biprominer, a tool that can automatically extract binary protocol message formats of an application from its real-world network trace. In addition, we present a transition probability model for a better description of the protocol. The chief feature of Biprominer is that it does not need to have any priori knowledge of protocol formats, because Biprominer is based on the statistical nature of the protocol format. We evaluate the efficacy of Biprominer over three binary protocols, with an average precision more than 99% and a recall better than 96.7%.

Identifying applications and classifying network traffic flows according to their source applications are critical for a broad range of network activities. Such classifications can be based on information derived from packet header fields and payload content, or statistical characteristics of flows and communication patterns of hosts. However, most of present methods rely on some forms of priori knowledge. In this paper, an application signature based traffic classification system with a novel approach to fully automate the process of deriving signatures from unknown traffic is proposed. The key idea is to combine traffic clustering based on statistical flow properties in order to generate clusters dominated by a single application on the one hand, and application signature construction solely based on payload content from each cluster on the other hand. Evaluation using real-world traffic traces indicate that the proposed approach is highly effective.

Traditional signature-based intrusion detection systems inspect packet headers and payloads to report any malicious or abnormal traffic behavior that is observed in the network. With the advent and rapid adoption of network encryption mechanisms, typical deep packet inspection systems that focus only on the processing of network packet payload contents are gradually becoming obsolete. Advancing intrusion detection tools to be also effective in encrypted networks is crucial. In this work, we propose a signature language indicating packet sequences. Signatures detect events of possible intrusions and malicious actions in encrypted networks using packet metadata. We demonstrate the effectiveness of this methodology using different tools for penetrating vulnerable web servers and a public dataset with traffic that originates from IoT malware. We implement the signature language and we integrate it into an intrusion detection system. Using our proposed methodology, the generated signatures can effectively and efficiently report intrusion attempts.

FPGAs are often used in high speed networking and telecommunications environments, where they have been shown to be very capable of line rate forwarding and routing. However, complex processes are more easily described in high-level software. In addition, many researchers do not have backgrounds in complex hardware design. NetThreads 10G is a solution to both of these problems - a soft, multithreaded multicore network processor implemented on the NetFPGA-10G[1], and software programmable using C. NefThreads10G is a port and upgrade of the original NetThreads [2] system designed for the NetFPGA: the number of cores has been doubled, packet buffer capacity increased, and a new Ethernet packet based programming system has been implemented. NetThreads 10G has a bus-based architecture connecting four MIPS-like processors to a shared data cache and a shared packet I/O buffer (Figure 1). Each core has a private instruction cache and four independent threads executed in a round robin fashion. Sixteen hardware locks are included for protecting critical code sections. The NetFPGA-10G onboard RLDRAM provides up to 128MB of main memory. During the demonstration, a sample application is developed and compiled using the NetThreads cross compiler tool. NefThreads10G is configured on the NetFPGA10G, and the application is downloaded remotely via Ethernet packets. The application is a deep packet inspection program that can detect suspicious keywords in packet payloads and keeps a record in shared memory. The demo shows how NetThreads affords us complete programmable and stateful control over OSI Layer 2 and above. The demonstration also shows NetThreads in the context of the SAVI (Smart Applications on Virtual Infrastructure) testbed. SAVI [3] is a new approach to network and Internet infrastructure - completely virtualized and extremely flexible, it views infrastructure as converged, where processing, compute, networking and reconfigurable resources are all part of a shared and managed pool. Having reconfigurable hardware in such a virtualized and programmable environment will open up new avenues of research in reconfigurable systems.

Nowadays, massive Internet-of-Thing (IoT) devices are connecting with cyberspace, yet they suffer increased attack risks from known vulnerabilities to low-hanging exploitable manners. A proactive defense can help security professionals to discover potential risks, where IoT device identification is a necessary requisite. However, existing approaches suffer from coarse-grained and manual labor. In this work, we propose an automated semantic extraction approach, called IoTminer, which generates IoT device annotation from the packet payload. Specifically, IoTminer leverages relations between device types, vendors, and products to mine relevant entities for an annotation tuple (type, vendor, product). Further, we have implemented a prototype of IoTminer and conducted a real-world experiment to validate its efficacy. Results show that our IoTminer generates IoT device information at a fine-grained level, achieving 91.33% precision, 93% recall, and 90% F1 score. Moreover, the IoTminer can discover new IoT devices compared with state-of-the-art tools.

Protocol traffic analysis is important for a variety of networking and security infrastructures, such as intrusion detection and prevention systems, network management systems, and protocol specification parsers. In this paper, we propose ProHacker, a nonparametric approach that extracts robust and accurate protocol keywords from network traces and effectively identifies the protocol trace from mixed Internet traffic. ProHacker is based on the key insight that the n-grams of protocol traces have highly predictable statistical nature that can be effectively captured by statistical language models and leveraged for robust and accurate protocol identification. In ProHacker, we first extract protocol keywords using a nonparametric Bayesian statistical model, and then use the corresponding protocol keywords to classify protocol traces by a semi-supervised learning algorithm. We implement and evaluate ProHacker on real-world traces, including SMTP, FTP, PPLive, SopCast, and PPStream, and our experimental results show that ProHacker can accurately identify the protocol trace with an average precision of about 99.42% and an average recall of about 98.64%. We also compare the results of ProHacker to two state-of-the-art approaches ProWord and Securitas using backbone traffic. We show that ProHacker provides significant improvements on precision and recall for online protocol identification.

Over the years, a number of semisupervised deep-learning algorithms have been proposed for time-series classification (TSC). In semisupervised deep learning, from the point of view of representation hierarchy, semantic information extracted from lower levels is the basis of that extracted from higher levels. The authors wonder if high-level semantic information extracted is also helpful for capturing low-level semantic information. This paper studies this problem and proposes a robust semisupervised model with self-distillation (SD) that simplifies existing semisupervised learning (SSL) techniques for TSC, called SelfMatch. SelfMatch hybridizes supervised learning, unsupervised learning, and SD. In unsupervised learning, SelfMatch applies pseudolabeling to feature extraction on labeled data. A weakly augmented sequence is used as a target to guide the prediction of a Timecut-augmented version of the same sequence. SD promotes the knowledge flow from higher to lower levels, guiding the extraction of low-level semantic information. This paper designs a feature extractor for TSC, called ResNet–LSTMaN, responsible for feature and relation extraction. The experimental results show that SelfMatch achieves excellent SSL performance on 35 widely adopted UCR2018 data sets, compared with a number of state-of-the-art semisupervised and supervised algorithms.