Geometric Hawkes Processes with Graph Convolutional Recurrent Neural Networks

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

A point process is a collection of positional variables in a given space, while a marked point process (MPP) is basically a point process that measures some additional feature or value at each point. Existing applications such as financial market models and hospitalization models have demonstrated that MPPS are suitable for modeling random events with meaningful values. A big challenge is to study the relationship between different types of events, or how different types of markers affect each other. While Hawkes procedures are an effective and efficient way to model point processes with internal influences, inferential methods of intensity functions can be expensive, especially for multi-dimensional processes. Graph convolutional Network (GCN) is a powerful neural network designed to directly process graphs. A GCN-based model is presented to study potential patterns in the marking process and to predict the location of future events without estimating the Hawkes intensity function. A real data set of water pipe leakage and rupture records over the last 50 years was used in the experiment, modeled as a marker point process, where leakage and rupture are two marker forms. Experimental results on synthetic data sets and real water pipe data sets show that the proposed model is superior to recent state-of-the-art methods.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Geometric deep learning provides a principled and universal way for the integration of imaging and non-imaging modes in the medical field. Graph Convolutional Networks (GCNs) have been widely explored in a variety of problems, such as disease prediction, segmentation, and matrix completion. Using large, multi-modal data sets, graphs can capture the interaction of individual elements represented as nodes in the graphs. In particular, in medical applications, nodes can represent individuals (patients or healthy controls) in a potentially large population and are accompanied by a set of features, while the edges of the graph contain the associations between subjects in an intuitive way. This representation allows the inclusion of rich imaging and non-imaging information as well as individual subject characteristics in the task of disease classification. This article gives an overview of graph convolutional neural networks, focusing on the application of graph convolutional neural networks in disease prediction, and discusses the challenges and prospects of graph convolutional neural networks in disease prediction.

Exploring the optimization of communication strategies for animation films in the context of cross-cultural communication, this research integrates the Internet of Things (IoT) and convolutional networks. The research constructs a collaborative filtering (CF) movie recommendation model based on a graph convolutional neural network (GCN) and investigates its application in cross-cultural communication. The fusion of IoT and convolutional networks in movie communication is also analyzed, and the effectiveness of the proposed GCN-CF model is validated through comparative experiments. The results indicate that, compared to other models, the GCN-CF model achieves the lowest Root Mean Square Error (RMSE) on the MovieLens 100 K and MovieLens 1 M datasets, with values of 0.8762 and 0.8275, respectively. Compared to traditional models, the GCN-CF model exhibits significantly superior performance in terms of RMSE, with reductions ranging from 0.6 to 5.2%, highlighting its heightened detection accuracy and overall performance. Moreover, the performance of the GCN-CF model is enhanced after introducing attention mechanisms and auxiliary information on both datasets, showing an improvement of 0.4% compared to the scenario without these additions. This data demonstrates the effectiveness of attention mechanisms and auxiliary information. Finally, the research presents an animation film communication strategy based on IoT and convolutional networks, offering novel insights for film production and communication, along with positive implications for cultural exchange and the advancement of the global media industry.

In autonomous driving scenarios, pedestrian trajectory prediction is an important research direction. Based on the spatio-temporal graph convolutional neural network, we propose a new pedestrian trajectory prediction algorithm. The new algorithm constructs a series of new models around pedestrian intention estimation. The construction of the estimation algorithm considers the following aspects: the contextual information of pedestrians and the surrounding environment, the “pedestrian ego-vehicle” interaction combined with the vehicle speed estimation, the pedestrian’s own skeletal structure information and body language estimation, which includes head joints and the relative structural relationship of the torso joints, including whether it is out of the same plane, is rotated, and so on. Skeleton information feature extraction and construction adopts the method of graph convolutional neural network to structure pedestrians into joints in the form of graphs in non-Euclidean space, and further adopts spatial temporal graph convolutional network for feature extraction and learning. The new method is named a “head-torso”-based spatial temporal graph convolutional network (HT-STGCN). On the dataset PID, the novel method achieves substantial improvements over mainstream methods. Experimental results show that combining HT-STGCN with observed action can improve trajectory prediction.

Messenger RNA (mRNA) has emerged as a critical global technology that requires global joint efforts from different entities to develop a COVID-19 vaccine. However, the chemical properties of RNA pose a challenge in utilizing mRNA as a vaccine candidate. For instance, the molecules are prone to degradation, which has a negative impact on the distribution of mRNA among patients. In addition, little is known of the degradation properties of individual RNA bases in a molecule. Therefore, this study aims to investigate whether a hybrid deep learning can predict RNA degradation from RNA sequences. Two deep hybrid neural network models were proposed, namely GCN_GRU and GCN_CNN. The first model is based on graph convolutional neural networks (GCNs) and gated recurrent unit (GRU). The second model is based on GCN and convolutional neural networks (CNNs). Both models were computed over the structural graph of the mRNA molecule. The experimental results showed that GCN_GRU hybrid model outperform GCN_CNN model by a large margin during the test time. Validation of proposed hybrid models is performed by well-known evaluation measures. Among different deep neural networks, GCN_GRU based model achieved best scores on both public and private MCRMSE test scores with 0.22614 and 0.34152, respectively. Finally, GCN_GRU pre-trained model has achieved the highest AuC score of 0.938. Such proven outperformance of GCNs indicates that modeling RNA molecules using graphs is critical in understanding molecule degradation mechanisms, which helps in minimizing the aforementioned issues. To show the importance of the proposed GCN_GRU hybrid model, in silico experiments has been contacted. The in-silico results showed that our model pays local attention when predicting a given position’s reactivity and exhibits interesting behavior on neighboring bases in the sequence.

Automatic building extraction from optical imagery remains a challenge due to, for example, the complexity of building shapes. Semantic segmentation is an efficient approach for this task. The latest development in deep convolutional neural networks (DCNNs) has made accurate pixel-level classification tasks possible. Yet one central issue remains: the precise delineation of boundaries. Deep architectures generally fail to produce fine-grained segmentation with accurate boundaries due to their progressive down-sampling. Hence, we introduce a generic framework to overcome the issue, integrating the graph convolutional network (GCN) and deep structured feature embedding (DSFE) into an end-to-end workflow. Furthermore, instead of using a classic graph convolutional neural network, we propose a gated graph convolutional network, which enables the refinement of weak and coarse semantic predictions to generate sharp borders and fine-grained pixel-level classification. Taking the semantic segmentation of building footprints as a practical example, we compared different feature embedding architectures and graph neural networks. Our proposed framework with the new GCN architecture outperforms state-of-the-art approaches. Although our main task in this work is building footprint extraction, the proposed method can be generally applied to other binary or multi-label segmentation tasks.

Vehicle trajectory prediction is an important research basis for the decision making and path planning of the intelligent and connected vehicle. In the connected vehicle environment, vehicles share information and drive cooperatively, and the intelligent and connected vehicles are able to obtain more accurate and rich perception information, which provides a data basis for accurate prediction of vehicle trajectories. However, attaining accurate and effective vehicle trajectory predictions poses technical challenges due to insufficient extraction of vehicular spatial–temporal interaction features. In this paper, we propose a vehicle trajectory prediction model based on graph convolutional neural network (GCN) in a connected vehicle environment. Specifically, using the driving scene information obtained by the intelligent and connected vehicle, the spatial graph and temporal graph are constructed based on the spatial interaction coefficient (SIC) and self-attention mechanism, respectively. Then, the graph data are entered into the interaction extraction module, and the spatial interaction features and temporal interaction features are extracted separately using the graph convolutional networks, which are fused to obtain the spatial–temporal interaction information. Finally, the interaction features are learned based on the convolutional neural networks to output the future trajectory information of all vehicles in the scene by one forward operation rather than a step-by-step process. The ablation experiment results show that the method proposed in this study to model the spatiotemporal interaction among vehicles based on SIC and self-attention mechanism reduces the prediction error by 5% and 12%, respectively. The results from the model comparison experiment show that the proposed method engenders an 8% improvement in prediction accuracy over the state-of-the-art solution, providing technical and theoretical support for trajectory prediction research of intelligent and connected vehicles.

Contemporary deep learning approaches exhibit state-of-the-art performance in various areas. In healthcare, the application of deep learning remains limited since deep learning methods are often considered as non-interpretable black-box models. However, the Machine Learning (ML) community made recent elaborations to develop the methods of eXplainable Artificial Intelligence (XAI). The explanation methods explain single decisions of an ML model when a single data point is fed into the model’s input. In a clinical setup, a data point can represent a single patient. Data point-specific explanations could possibly assist the need in personalized precision medicine decisions via explaining patient-specific predictions. Convolutional Neural Networks (CNNs) as deep learning methods have been already applied to classify transformed into images gene expression profiles of patients. Gene expression data can be structured by a prior knowledge molecular network (encoded as a graph) representing connections between genes. Each vertex of a molecular network is assigned a gene expression value as an attribute. The set of the attributes creates a graph signal representing a patient. Emerging field of geometric deep learning deals with methods applicable to graph structured data and extends CNNs as Graph Convolutional Neural Networks (GCNNs) classifying graph signals. Layer-wise Relevance Propagation (LRP) is a method to explain decisions of CNNs classifying image data. I extended the LRP method to make it available for GCNNs. Graph Layer-wise Relevance Propagation (GLRP) is presented as a new method to explain single decisions made by a GCNN model. In this thesis, I present a novel methodology generating patient-specific molecular subnetworks as explanations for classification decisions of an ML approach utilizing prior knowledge of molecular networks. GCNN serves as a ML approach, and its decisions are explained by developed GLRP. A sanity check of the developed GLRP method was demonstrated on a hand-written digits dataset. The biological validation was performed by applying the developed methodology to gene expression data from Human Umbilical Vein Endothelial Cells (HUVEC) treated or not treated with tumor necrosis factor alpha. To show the utility of introduced methodology in the scopes of precision medicine, it was applied to a large breast cancer dataset. The generated patient-specific subnetworks largely agree with clinical knowledge and could assist precision medicine approaches by identifying common as well as novel, and potentially druggable, drivers of tumor progression. Apart from generating patient-specific subnetworks, the developed methodology can be used as a general feature selection approach. The outcome of a feature selection approach is a subset of important for classification genes corresponding to a whole dataset. It is essential to sustain stability of selected feature subsets across different datasets with the same clinical endpoint since the selected genes are possible candidates for prognostic biomarkers. I analysed the stability of feature selection performed by GCNN+LRP. I have implemented a graph convolutional layer of GCNN as a Keras layer so that the SHapley Additive exPlanations (SHAP) method could be also applied to a Keras version of a GCNN model. The stability of feature selection performed by GCNN+LRP was compared to the stability of GCNN+SHAP and other ML-based feature selection approaches. The GCNN+LRP approach shows the highest stability. GCNN+LRP subnetworks were compared to GCNN+SHAP subnetworks in terms of connectivity and permutation feature importance. While GCNN+SHAP subnetworks demonstrate higher permutation importance than GCNN+LRP subnetworks, a GCNN+LRP subnetwork of an individual patient is on average substantially more connected and, therefore more interpretable in the context of prior knowledge than a GCNN+SHAP subnetwork which consists mainly of single vertices.

Large volumes of temporal event data, such as online check-ins and electronic records of hospital admissions, are becoming increasingly available in a wide variety of applications including healthcare analytics, smart cities, and social network analysis. Those temporal events are often asynchronous, interdependent, and exhibiting self-exciting properties. For example, in the patient's diagnosis events, the elevated risk exists for a patient that has been recently at risk. Machine learning that leverages event sequence data can improve the prediction accuracy of future events and provide valuable services. For example, in e-commerce and network traffic diagnosis, the analysis of user activities can be used to predict and control dynamic user traffic demand to improves risk response efficiency. In this work, we investigate and design novel point process-based models and learning algorithms to analyze dynamic event sequence data from various aspects. (1) We first propose local low-rank Hawkes processes to capture the mutual influences between sequences of multiple event types. (2) We then develop geometric Hawkes processes to integrate geometric structures to point processes based on graph convolutional recurrent neural networks to improve prediction accuracy. (3) We introduce several novel fairness metrics to penalize the event likelihood function in order to tackle the challenge of data bias and the amplified through self-excitation in point processes. (4) We also propose a novel list-wise fairness criterion for point processes that can efficiently evaluate the ranking fairness in event prediction, and present a strict definition of the unfairness consistency property of a fairness metric.

Graph convolutional neural networks have difficulties in model training takes a lot of time and lack of universality in meta-path design when processing knowledge graphs. To solve these issues, this paper proposes an efficient recommendation algorithm integrating knowledge graphs with graph convolutional networks(EKG-GCN). The algorithm is mainly based on the graph convolution algorithm. Firstly, the relationship between users and entities is standardized and scored, then a new neighborhood aggregation method is proposed. Some nodes in the knowledge graph are set as central nodes. A distance-influence function is set up according to the distance from other nodes to the central node, and the feature extraction of information in the knowledge graph is completed according to the distance-influence function. Finally, the knowledge graph features are used as an auxiliary to complete the recommendation task. In the experiment, three data sets were selected to compare with other advanced models, and it was found that the model is in an advantageous position in terms of training time and recommendation effect, and has a certain degree of universality.

The objective of extracting relations between document components lies in identifying the Within a single document, the focus often lies on understanding the connections between different entities. This type of analysis goes beyond individual sentences, as it demands an understanding of how information from multiple sentences interacts to form these connections. Over recent years, the importance of exploring relationships involving several entities simultaneously has grown significantly. To advance the field of studying such connections across entire documents, a novel collection of data points, known as DocRED, has been introduced. Currently, the standard approach for this task involves using BiLSTM networks to process the entire document as a whole. However, this method struggles to effectively capture the intricate relationships that exist among various entities. To overcome this limitation, a new model designed for document-level relationship identification has been developed, which leverages Graph Convolutional Networks (GCN). GCNs are particularly useful here because they can gather information from surrounding entities, allowing for a more detailed modeling of their interactions. The proposed approach starts by identifying coreferential links to gather features that represent the relationships between pairs of entities. These features are then analyzed using GCN to construct a graph structure that represents the entire document, ultimately revealing the complex interactions between different entities. Testing this model on the large-scale DocRED dataset from Tsinghua University has demonstrated its strong performance in this challenging task.

In biomedical machine learning, data often appear in the form of graphs. Biological systems such as protein interactions and ecological or brain networks are instances of applications that benefit from graph representations. Geometric deep learning is an arising field of techniques that has extended deep neural networks to non-Euclidean domains such as graphs. In particular, graph convolutional neural networks have achieved advanced performance in semi-supervised learning in those domains. Over the last years, these methods have gained traction in neuroscience as they could be the key to a deeper understanding in clinical diagnosis at the systems or network level (for an individual brain but also for across a cohort of subjects). As a proof-of-principle, we study and validate a previous implementation of graph-based semi-supervised classification using a ridge classifier and graph convolutional neural networks. The models are trained on population graphs that integrate imaging and phenotypic information. Our analysis employs neuroimaging data of structural and functional connectivity for prediction of neurodevelopmental and neurodegenerative disorders. Here, we particularly study the effect of different strategies to reduce the dimensionality of the neuroimaging features on the graph nodes on the classification performance.

Reservoir simulation results currently provide the basis for important reservoir engineering decisions; grid complexity and non-linearity of these models demand high computational time and memory. The physics-based simulation process must be repeated to increase model prediction accuracy or to perform history matching; consequently, the simulation process is often time-consuming. This paper describes a new methodology based on a deep neural network (DNN) technique, the graph convolutional neural network (G-CNN). G-CNN increases the modeling prediction speed and efficiency by compressing the computational time and memory usage of the reservoir simulation. A G-CNN model was used to perform the reservoir simulations described. This new methodology combines physics-based and data-driven models in reservoir simulation. The workflow generates training datasets, enabling intelligent sampling of the reservoir production data in the G-CNN training process. Bottomhole pressure constraints were set for all simulations. The production data generated by the reservoir model, with the mesh connectivity information, is used to generate the G-CNN model. This approach can be viewed as hybrid data-driven, retaining the underlying physics of the reservoir simulator. The resulting G-CNN model can perform reservoir simulations for any computational grid and production time horizon. The method uses convolutional neural network and mesh connections in a fully differentiable scheme to compress the simulation state size and learns the reservoir dynamics on this compressed form. G-CNN analysis was performed on an Eagle Ford-type reservoir model. The transmissibility, pore volume, pressure, and saturation at an initial time state were used as input features to predict final pressure and saturations. Prediction accuracy of 95% was obtained by hyperparameter tuning off the G-CNN architecture. By compressing the simulation state size and learning the time-dependent reservoir dynamics on this compressed representation, reservoir simulations can be emulated by a graph neural network which uses significantly less computation and memory. The G-CNN model can be used on any computational grid because it preserves the structure of the physics. The G-CNN model is trained by mapping an initial time state to a future prediction time state; consequently, the model can be used for generalizing predictions in any grid sizes and time steps while maintaining accuracy. The result is a computationally and memory efficient neural network that can be iterated and queried to reproduce a reservoir simulation. This novel methodology combines field-scale physics-based reservoir modeling and DNN for reservoir simulations. The new reservoir simulation workflow, based on the G-CNN model, maps the time state predictions in a resampled grid, reducing computational time and memory. The new methodology presents a general method for compressing reservoir simulations, assisting in fast and accurate production forecasting.