Self-Supervised Latent Representations of Network Flows and Application to Darknet Traffic Classification

Graph Convolutional Networks (GCNs) have been widely adopted to learn structured data that denotes structural connections between entities. However, over-smoothing has become a common phenomenon in GCNs, resulting in node representations in different classes are indistinguishable. Inspired by recent works, graph disentanglement, which can learn disentangled latent factors behind the input data and generate latent representations corresponding to latent factors, is adopted to relieve over-smoothing. Although graph disentanglement is promising, it still faces some challenges. (1) How to obtain disentangled latent representations useful for both latent factors and downstream tasks? (2) How to enhance the diversity and independence of latent representations? In this paper, we propose a novel GCN model based on Graph Disentanglement with Latent Bottlenecks (DBGCN) to boost the diversity of node representations and relieve the over-smoothing problem. To learn disentangled node representations that are beneficial to both latent factors and downstream tasks, we propose to identify latent bottlenecks using the information bottleneck, which offers a critical principle for good representation learning. To boost the diversity of latent representations and the independence between them, we design a generative model to search for specific latent bottlenecks that constrain information flow from the input using specific latent factors, and to derive distributions of latent representations. Experimental results on various real-world graphs demonstrate that the proposed model can relieve the over-smoothing problem effectively and achieve significant performance gains. It also suggests that enhancing the diversity of latent representations is helpful to relieve over-smoothing.

A foundation model, a large-scale deep learning model pretrained on vast datasets, has revolutionized data interpretation through self-supervised learning with capacity for various downstream tasks. Concurrently, single-cell genomics is in urgent need of unified frameworks capable of integrating and comprehensively analyzing the rapidly expanding data repositories. Inspired by advances in foundation models, researchers have extended these techniques to single-cell analysis, giving rise to single-cell foundation models (scFMs). Typically, these models use transformer architectures to incorporate diverse omics data and extract latent patterns at both cell and gene/feature levels for the analysis of cellular heterogeneity and complex regulatory networks. Despite their promise, scFMs face challenges including the nonsequential nature of omics data, inconsistency in data quality and the computational intensity required for training and fine-tuning. Furthermore, interpreting the biological relevance of latent embeddings and model representations remains nontrivial. Here we provide an overview of scFMs, highlighting their key concepts and applications across downstream tasks. We critically assess the current limitations and propose future directions aimed at enhancing the robustness, interpretability and scalability of scFMs. Ultimately, addressing these challenges will be crucial for establishing scFMs as pivotal tools in advancing single-cell genomics and unlocking deeper insights into cellular function and disease mechanisms.

Semantic segmentation and activity classification are key components to create intelligent surgical systems able to understand and assist clinical workflow. In the operating room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting relative 3D distance of image patches by exploiting the depth maps. By learning 3D spatial context, it generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multiview data captured from clinical scenarios. We demonstrate a noteworthy improvement in performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest. We propose a novel privacy-preserving self-supervised approach utilizing depth maps. Our proposed method shows performance on par with other self-supervised approaches and could be an interesting way to alleviate the burden of full supervision.

BackgroundThe assessment of cortical thickness through structural magnetic resonance imaging (MRI) reveals various cortical atrophy patterns, delivering crucial insights into the extent of neurodegeneration. Numerous studies have focused on these thickness values to construct models for diverse neurodegeneration‐related tasks, such as classifying disease groups, staging progression, or predicting progression speed. However, these tasks require substantial training datasets tailored to each specific task, necessitating the development of new task‐specific models. We aimed to establish a foundational model using pre‐trained BERT, integrating context‐aware deep latent representations derived from a large cohort of unlabeled cortical atrophy data, to address these challenges.MethodWe collected 10,450 T1‐weighted MRI images within the ADNI dataset. Cortical thickness was derived using FreeSurfer across 68 regions based on the Desikan‐Killiany atlas. Emplying the BERT model, we applied its self‐attention capabilities to encode deep bidrectional representations of the data. The model is pre‐trained with 8,360 unlabeled cortical thickness data through self‐supervised learning, predicting randomly masked thickness values. Subsequently, we fine‐tuned this pre‐trained model with only one additional output layer using a supervised approach, employing independent data subsets for three challenging downstream classification tasks: (1) mild cognitive impairment (MCI) and Alzheimer’s dementia (AD), (2) MCI converters (C‐MCI) and non‐converters (N‐MCI) within 3 years, and (3) Aβ‐positive and negative cases. During fine‐tuning, demographic information encoded by one layer was also incorporated for classification purposes.ResultTable 1 provides demographic details of the subset data for fine‐tuning, showing comparable trends in each set of information, except for MMSE in the MCI and AD comparison. Table 2 outlines our model's performance on three downstream classification tasks. Despite training with a relatively limited amount of data, our model demonstrated competitive performance with state‐of‐the‐art methods, achieving 85% accuracy on the classification of MCI and AD, 85% of C‐MCI and N‐MCI, and 69% of Aβ‐positive and negative.ConclusionThis study introduces the novel cortical atrophy foundation model, demonstrating competitive performance even with a limited dataset across diverse challenging tasks. The model's performance in predictive tasks highlights its potential applicability for various nerodegeneration‐related challenges, paving the way for advanced AI applications within the realm of neurodegenerative diseases.

Foundation vision, audio, and language models enable zero-shot performance on downstream tasks via their latent representations. Recently, unsupervised learning of data group structure with deep learning methods has gained popularity. TURTLE, a state of the art deep clustering algorithm, uncovers data labeling without supervision by alternating label and hyperplane updates, maximizing the hyperplane margin, in a similar fashion to support vector machines (SVMs). However, TURTLE assumes clusters are balanced; when data is imbalanced, it yields non-ideal hyperplanes that cause higher clustering error. We propose PET-TURTLE, which generalizes the cost function to handle imbalanced data distributions by a power law prior. Additionally, by introducing sparse logits in the labeling process, PET-TURTLE optimizes a simpler search space that in turn improves accuracy for balanced datasets. Experiments on synthetic and real data show that PET-TURTLE improves accuracy for imbalanced sources, prevents over-prediction of minority clusters, and enhances overall clustering.

Neuroimaging plays an significant role in diagnosing and pathological study of brain diseases. Considering that both functional and structural abnormalities may lead to brain dis-eases and disorders, single modal neuroimaging approach may not fully characterize brain activities and working modes. Fusion of multimodal neuroimaging data is expected to provide more comprehensive characterization of brain diseases, given that the different modalities contain more complementary information. Recently, Graph Convolutional Networks (GCNs) is shown to have powerful capacity in representation learning for graph-structure data, which is considered to integrate both graph se-mantic structure and node information. Therefore, in this paper, we propose the Weighted Graph AutoEncoder (WGAE), a GCN- driven multimodal fusion model, to learn the combinational latent node representation of fMRI and DTI neuroimaging data, which are used as node features and graph structure respectively in the graph in unsupervised manner. Experimental results on two real-world datasets show the superiority of the proposed model over other existing single-modal or multi-modal methods in learning representations for disease prediction as the downstream task. Furthermore, ablation experiments also show the collaborative contribution of multimodal neuroimaging fusion in the proposed model, and also show the feasibility of assessing the respective importance of the two modalities during the disease prediction.

A new approach to secure economic power dispatch

Attribute graphs are a crucial data structure for graph communities. However, the presence of redundancy and noise in the attribute graph can impair the aggregation effect of integrating two different heterogeneous distributions of attribute and structural features, resulting in inconsistent and distorted data that ultimately compromises the accuracy and reliability of attribute graph learning. For instance, redundant or irrelevant attributes can result in overfitting, while noisy attributes can lead to underfitting. Similarly, redundant or noisy structural features can affect the accuracy of graph representations, making it challenging to distinguish between different nodes or communities. To address these issues, we propose the embedded fusion graph auto-encoder framework for self-supervised learning (SSL), which leverages multitask learning to fuse node features across different tasks to reduce redundancy. The embedding fusion graph auto-encoder (EFGAE) framework comprises two phases: pretraining (PT) and downstream task learning (DTL). During the PT phase, EFGAE uses a graph auto-encoder (GAE) based on adversarial contrastive learning to learn structural and attribute embeddings separately and then fuses these embeddings to obtain a representation of the entire graph. During the DTL phase, we introduce an adaptive graph convolutional network (AGCN), which is applied to graph neural network (GNN) classifiers to enhance recognition for downstream tasks. The experimental results demonstrate that our approach outperforms state-of-the-art (SOTA) techniques in terms of accuracy, generalization ability, and robustness.

In the detection process of server network storm attack, the traditional method takes the signal detection algorithm, because the attack number is large, the detection error is big.According to the problem, an improved server network storm attack detection method is proposed based on improved density clustering algorithm, the server network storm attack detection problem is transformed into a multi class classification problem, wrapper features selection model is taken, IDBC network connection record distance calculation method is combined, based on DBSCAN clustering results, the data clustering and attack detection is obtained.Simulation results show that, the improved density clustering algorithm is applied in detection of server network storm attack, it can reduce the detection error, the performance of the detection system is improved.

We explore the relationship between a yield stress fluid flow in a Hele-Shaw cell with irregular walls, and a non-Darcy 2D porous medium flow with limiting pressure gradient. The continuum (Hele-Shaw) flow is a much-studied simplification of a cementing displacement flow model, solved via a variational formulation that leads to a convex streamfunction minimization problem. Our interest is with the analogy between the discretization used for the numerical solution and network flows that model the associated porous media flow, i.e. via the pore-throat approach. A staggered mesh, with pressure nodes at the cell corners and streamfunction nodes at the cell center is used for the continuum problem, which naturally separates into a network representation comprising primal and dual graphs, linking streamfunction and pressure nodes, respectively. We show explicitly how the continuum model defines a network model and vice versa. We develop the variational form of the network flow, including an appropriate (discrete) streamfunction minimization and a discrete version of the principle of virtual work. Two network models are explored, based on different interpretations of the minimization problem. The network flow results are compared with analogous computed continuum flow results in 3 specific geometries. We find that our network model I, which is the most natural interpretation of the continuum model as a network flow using our discretization, generally under-predicts flow rates. This is problematic from the perspective of considering the network flow as an approximation to the porous media or Hele-Shaw flow. Network model II rectifies this situation, via a pressure interpolation method. In our examples we find that the network II flow converges to the continuum flow as the mesh and network are refined. This is not the usual comparison made, as in many pore-throat models the network is fixed according to the underlying pore-space geometry. Despite the differences, both network models have their own advantages and disadvantages. Network model I offers a more natural way of modelling the flow and is easier to apply, for example to complex meshes, e.g. unstructured triangular. Network model II gives the more accurate physical representation of the Hele-Shaw flow. Lastly, we have developed approximate algebraic relationships for the total flow rate as a function of the total applied pressure gradient, both close to the critical onset pressure and for large pressure gradients. These correlations align well with previous findings for Bingham fluid flows in porous media. • Hele-Shaw cell – porous media analogy derived for Herschel-Bulkley fluids. • Discrete pore-throat network models derived. • Discrete principle of virtual work. • Convergence as network is refined. • Improved network model that gives convergence.

Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks. In this paper, we present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint. The loss function is data driven and automatically adapts to arbitrary multi-label structures. Experiments on several datasets show that our relationship-preserving embedding performs well on a variety of tasks and outperform the base-line supervised and self-supervised approaches. Code is available at https://github.com/salesforce/hierarchicalContrastiveLearning.

The current paradigm of deep learning models for the joint representation of molecules and text primarily relies on 1D or 2D molecular formats, neglecting significant 3D structural information that offers valuable physical insight. This narrow focus inhibits the models' versatility and adaptability across a wide range of modalities. Conversely, the limited research focusing on explicit 3D representation tends to overlook textual data within the biomedical domain. We present a unified pre-trained language model, MolLM, that concurrently captures 2D and 3D molecular information alongside biomedical text. MolLM consists of a text Transformer encoder and a molecular Transformer encoder, designed to encode both 2D and 3D molecular structures. To support MolLM's self-supervised pre-training, we constructed 160K molecule-text pairings. Employing contrastive learning as a supervisory signal for learning, MolLM demonstrates robust molecular representation capabilities across four downstream tasks, including cross-modal molecule and text matching, property prediction, captioning, and text-prompted molecular editing. Through ablation, we demonstrate that the inclusion of explicit 3D representations improves performance in these downstream tasks. Our code, data, pre-trained model weights, and examples of using our model are all available at https://github.com/gersteinlab/MolLM. In particular, we provide Jupyter Notebooks offering step-by-step guidance on how to use MolLM to extract embeddings for both molecules and text.

The complexity of the Internet and the volume of network traffic have dramatically increased in the last few years, making it more challenging to design scalable Network Traffic Monitoring and Analysis (NTMA) systems. Critical NTMA applications such as the detection of network attacks and anomalies require fast mechanisms for on-line analysis of thousands of events per second, as well as efficient techniques for off-line analysis of massive historical data. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks and anomalies, but this higher dimensionality comes with an extra data processing overhead. In this paper we present Big-DAMA, a big data analytics framework (BDAF) for NTMA applications. Big-DAMA is a flexible BDAF, capable to analyze and store big amounts of both structured and unstructured heterogeneous data sources, with both stream and batch processing capabilities. Big-DAMA uses off-the-shelf big data storage and processing engines to offer both stream data processing and batch processing capabilities, decomposing separate engines for stream, batch and query, following a Data Stream Warehouse (DSW) paradigm. Big-DAMA implements several algorithms for anomaly detection and network security using supervised and unsupervised machine learning (ML) models, using off-the-shelf ML libraries. We apply Big-DAMA to the detection of different types of network attacks and anomalies, benchmarking multiple supervised ML models. Evaluations are conducted on top of real network measurements collected at the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling. Big-DAMA can speed up computations by a factor of 10 when compared to a standard Apache Spark cluster, and can be easily deployed in cloud environments, using hardware virtualization technology.

Unsupervised/self-supervised pre-training methods for graph representation learning have recently attracted increasing research interests, and they are shown to be able to generalize to various downstream applications. Yet, the adversarial robustness of such pre-trained graph learning models remains largely unexplored. More importantly, most existing defense techniques designed for end-to-end graph representation learning methods require pre-specified label definitions, and thus cannot be directly applied to the pre-training methods. In this paper, we propose an unsupervised defense technique to robustify pre-trained deep graph models, so that the perturbations on the input graph can be successfully identified and blocked before the model is applied to different downstream tasks. Specifically, we introduce a mutual information-based measure, graph representation vulnerability (GRV), to quantify the robustness of graph encoders on the representation space. We then formulate an optimization problem to learn the graph representation by carefully balancing the trade-off between the expressive power and the robustness (i.e., GRV) of the graph encoder. The discrete nature of graph topology and the joint space of graph data make the optimization problem intractable to solve. To handle the above difficulty and to reduce computational expense, we further relax the problem and thus provide an approximate solution. Additionally, we explore a provable connection between the robustness of the unsupervised graph encoder and that of models on downstream tasks. Extensive experiments demonstrate that even without access to labels and tasks, our model is still able to enhance robustness against adversarial attacks on three downstream tasks (node classification, link prediction, and community detection) by an average of +16.5% compared with existing methods.

We propose to learn latent space representations of radio galaxies, and train a very deep variational autoencoder (VDVAE) on RGZ DR1, an unlabeled dataset, to this end. We show that the encoded features can be leveraged for downstream tasks such as classifying galaxies in labeled datasets, and similarity search. Results show that the model is able to reconstruct its given inputs, capturing the salient features of the latter. We use the latent codes of galaxy images, from MiraBest Confident and FR-DEEP NVSS datasets, to train various non-neural network classifiers. It is found that the latter can differentiate FRI from FRII galaxies achieving accuracy ≥ 76%, roc-auc ≥ 0.86, specificity ≥ 0.73 and recall ≥ 0.78 on MiraBest Confident dataset, comparable to results obtained in previous studies. The performance of simple classifiers trained on FR-DEEP NVSS data representations is on par with that of a deep learning classifier (CNN based) trained on images in previous work, highlighting how powerful the compressed information is. We successfully exploit the learned representations to search for galaxies in a dataset that are semantically similar to a query image belonging to a different dataset. Although generating new galaxy images (e.g. for data augmentation) is not our primary objective, we find that the VDVAE model is a relatively good emulator. Finally, as a step toward detecting anomaly/novelty, a density estimator — Masked Autoregressive Flow (MAF) — is trained on the latent codes, such that the log-likelihood of data can be estimated. The downstream tasks conducted in this work demonstrate the meaningfulness of the latent codes.