Graph-Based Object Semantic Refinement for Visual Emotion Recognition

Short-term energy prediction plays an important role in green manufacturing in the industrial internet environment and has become the basis of energy wastage identification, energy-saving plans and energy-saving control. However, the short-term energy prediction of multiple nodes in manufacturing systems is still a challenging issue owing to the fuzzy material flow (spatial relationship) and the dynamic production rhythm (temporal relationship). To obtain the complex spatial and temporal relationships, a spatio-temporal deep learning network (STDLN) method is presented for the short-term energy consumption prediction of multiple nodes in manufacturing systems. The method combines a graph convolutional network (GCN) and a gated recurrent unit (GRU) and predicts the future energy consumption of multiple nodes based on prior knowledge of material flow and the historical energy consumption time series. The GCN is aimed at capturing spatial relationships, with the material flow represented by a topology model, and the GRU is aimed at capturing dynamic rhythm from the energy consumption time series. To evaluate the method presented, several experiments were performed on the power consumption dataset of an aluminum profile plant. The results show that the method presented can predict the energy consumption of multiple nodes simultaneously and achieve a higher performance than methods based on the GRU, GCN, support vector regression (SVR), etc.

Summary Oilfield development performance prediction is a significant and complex problem in oilfield development. Reasonable prediction of oilfield development performance can guide the adjustment of the development plan. Moreover, the reservoir will change slowly during reservoir development because of flowing water however, previous networks that forecast production dynamics ignored it, which leads to inaccurate predictions. Routine well-wise injection and production measurements contain important subsurface structure and properties. So, for the dynamic prediction of oil/water two-phase waterflooded reservoirs, we built a deep learning framework named adaptive correction interwell connectivity model based on graph convolutional networks (GCN) and gated recurrent unit (GRU). It includes two parts: The first part is the adaptive correction model based on GCN, which uses dynamic production data to automatically correct the initial interwell connectivity computed by permeability, porosity, interwell distance, and so on. The second part is the adaptive learning model based on GRU, which predicts the production performance of oil wells according to the time characteristics of production performance data. This framework considers the influence that changes in reservoir conditions have on production over time to solve the problem of inaccurate production dynamic prediction. It can also predict interwell connectivity. For oilfields with too many wells, using the embedding idea classifies similar wells into one category, saving time for training and avoiding overfitting problems. Applying the model to five different reservoirs to predict interwell connectivity, well oil production rate, and well water cut compare the results with artificial neural networks (ANN), GRU, and long short-term memory (LSTM) models and compare the interwell connectivity with numerical simulation software ,tNavigator® (Rock Flow Dynamics Llc), too. When the model is applied in Block B of Bohai A reservoir, the mean absolute percentage error of “Adaptive Graph convolutional network and GRU” (AG-GRU) is 2.1150% while the LSTM is 9.8872%. The error reduces by 78.6%. The injected water has a direction from the water injection well to the production well; this paper only considers the interwell connectivity without considering the direction. Further research is needed to consider the water injection direction and form a weighted directed graph.

Aspect Sentiment Triplet Extraction (ASTE) is a complex and challenging task in Natural Language Processing (NLP). It aims to extract the triplet of aspect term, opinion term, and their associated sentiment polarity, which is a more fine-grained study in Aspect Based Sentiment Analysis. Furthermore, there have been a large number of approaches being proposed to handle this relevant task. However, existing methods for ASTE suffer from powerless interactions between different sources of textual features, and they usually exert an equal impact on each type of feature, which is quite unreasonable while building contextual representation. Therefore, in this paper, we propose a novel Multi-Branch GCN (MBGCN)-based ASTE model to solve this problem. Specifically, our model first generates the enhanced semantic features via the structure-biased BERT, which takes the position of tokens into the transformation of self-attention. Then, a biaffine attention module is utilized to further obtain the specific semantic feature maps. In addition, to enhance the dependency among words in the sentence, four types of linguistic relations are defined, namely part-of-speech combination, syntactic dependency type, tree-based distance, and relative position distance of each word pair, which are further embedded as adjacent matrices. Then, the widely used Graph Convolutional Network (GCN) module is utilized to complete the work of integrating the semantic feature and linguistic feature, which is operated on four types of dependency relations repeatedly. Additionally, an effective refining strategy is employed to detect whether word pairs match or not, which is conducted after the operation of each branch GCN. At last, a shallow interaction layer is designed to achieve the final textual representation by fusing the four branch features with different weights. To validate the effectiveness of MBGCNs, extensive experiments have been conducted on four public and available datasets. Furthermore, the results demonstrate the effectiveness and robustness of MBGCNs, which obviously outperform state-of-the-art approaches.

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

A multi-scale semantic attention representation for multi-label image recognition with graph networks

Graph Convolutional Network (GCN) excels at EEG recognition by capturing brain connections, but previous studies neglect the important EEG feature itself. In this study, we propose MSFR-GCN, a multi-scale feature reconstruction GCN for recognizing emotion and cognition tasks. Specifically, MSFR-GCN includes the MSFR and feature-pool characteristically, with the MSFR consisting of two sub-modules, multi-scale Squeeze-and-Excitation (MSSE) and multi-scale sample re-weighting (MSSR). MSSE assigns weights to channels and frequency bands based on their separate statistical information, while MSSR assigns sample weights based on combined channel and frequency information. The feature-pool, which pools across the feature dimension, is applied after GCN to retain EEG channel information. The MSFR-GCN achieves excellent results in emotion recognition when first tested on two public datasets, SEED and SEED-IV. Than the MSFR-GCN is tested on our self-collected Emotion and Cognition EEG dataset (ECED) for both emotion and cognition classification tasks. The results show MSFR-GCN's good performance in emotion and cognition classification tasks and reveal the implicit relationship between the two, which may provide aid in the rehabilitation of people with cognitive impairments from an emotional perspective.

Multi-label classification of remote sensing (RS) images under the deep learning framework has attracted increasing attentions. However, manual annotation of RS images is time-consuming and easily results in missing labels due to the subjectivity of human perception. The deep learning models could suffer from performance drop because of the incomplete labels of the training set. To address the issue, a deep learning model with label enhancement is proposed for incomplete multi-label classification of RS images. The model consists of a semantic feature extraction (SFE) module, a label correlation learning (LCL) module and a label enhancement (LE) module. Firstly, the semantic features of RS images are extracted by an improved ResNet-101 in the SFE module. Secondly, the global label correlations are learned by a graph convolutional network (GCN) in the LCL module. Subsequently, the labels predicted according to the semantic features are enhanced by the label correlations and local semantics in the LE module. Finally, a weighted combination of the label probability generated by the semantic features and the enhanced one through the LE module is applied in the loss function for classification. The multi-label classification experiments on three public RS image datasets prove the effectiveness of our model.

Emotional inference by recognizing facial expressions in an image or video has been actively conducted for several years, for example, visual emotion recognition and emotional state recognition. The limitations of illumination conditions, variations of head poses, and partial occlusion are the most challenging problems in image processing. Furthermore, recognizing and inferring insignificant objects in the background is complex. These factors degrade performance. We collected evidence from human sensing through a psychological approach rather than a numerical approach, and converted it into a language through fuzzy theory. This can not only be intuitively explained, but it is also possible to evaluate and analyze numerically. The predicted inference accuracy increased by approximately 9.7% compared with the simple average method.

Context: Measuring user satisfaction is an essential tool to create value in digital transformation. Measurement allows you to identify future purchase intentions, user loyalty, and retention. The traditional way of measuring satisfaction using self-assessment has problems, such as subjectivity. Therefore, a more objective approach is needed that allows automatic measurement of satisfaction. Objective: This research aims to make a systematic literature mapping (SLM) to identify the state-of-the-art use of Artificial Intelligence tools and techniques related to emotion recognition visually and then use the results to propose a model that can lead to an automated user satisfaction measurement. We present the model with a work process, an exploratory experiment, and an application project to measure the user experience from visual emotion recognition, automatically calculate the satisfaction assessment of a user when using a digital product, and present the positive, neutral, and negative points of using this product. Methods: A systematic literature mapping was used to identify the primary studies related to techniques, benefits, and challenges that associate the measurement of user satisfaction through the recognition of emotions with the use of Artificial Intelligence (AI). In the proposed model, an experiment was planned in two phases to validate research hypotheses related to the objective of the work. The experiment, which will apply the projected application, will be carried out by videoconferencing with the approach of exploratory usability testing on a web product. Results: 10 primary studies were identified in different areas of knowledge: restaurants, television systems, retail stores, artistic shows, and usability testing in a call center system. Two systematic reviews of the literature were also identified. The technique most commonly used in primary studies is the convolutional neural network (CNN). The use of cloud services for emotion recognition was also verified. Benefits related to user feedback, such as user profile mapping, were reported, and challenges for emotion recognition were found, such as user privacy and capture environment inadequacies. Besides, a model to automatically measure user satisfaction in a scalable and replicable way was proposed. Conclusions: The possibilities of applying emotion recognition are countless in terms of contexts, techniques, forms, and components. Despite this, it was possible to identify good practices that could guide the creation of a tool to measure the satisfaction of using a digital product through emotion recognition and Artificial Intelligence. The main contribution of this work is the proposal of a model that guides the creation of an automatic satisfaction measurement tool to identify positive, neutral, and negative points in the use of a digital product. This research also defines a measurement experimentation process that any organization or researcher can reproduce.

The bogie system is a critical system for a high-speed train, which is composed of various mechanical parts. Therefore, the health of the bogie can directly affect the health of high-speed train. Temperature signals, speed signals and pressure signals are collected from the bogie can reflect its health. Hence, the multi-sensor fault diagnosis methods can provide novel solutions for the bogie health monitoring tool. This paper presents a novel bogie fault diagnosis scheme named the AttGGCN model, using graph convolutional network (GCN), gated recurrent unit (GRU) and attention mechanism. In this fault diagnosis scheme, the bogie data network is established firstly. Then, temporal and spatial features are extracted and fused using GCG unit. Finally, the GCN are used for fault identification. Twenty-four kinds of measured signals and seven types of faults from actual High-speed train in operation are utilized for verification. Results show that the AttGGCN model has the highest accuracy compared to conventional models. In addition, experiments on different scales of training sets suggest that the AttGGCN model has strong robustness in small-scale datasets. Besides, ablation experiments certificate that the attention mechanism is able to strengthen the feature extraction ability.

Messenger RNA (mRNA) has emerged as a critical global technology that requires global joint efforts from different entities to develop a COVID-19 vaccine. However, the chemical properties of RNA pose a challenge in utilizing mRNA as a vaccine candidate. For instance, the molecules are prone to degradation, which has a negative impact on the distribution of mRNA among patients. In addition, little is known of the degradation properties of individual RNA bases in a molecule. Therefore, this study aims to investigate whether a hybrid deep learning can predict RNA degradation from RNA sequences. Two deep hybrid neural network models were proposed, namely GCN_GRU and GCN_CNN. The first model is based on graph convolutional neural networks (GCNs) and gated recurrent unit (GRU). The second model is based on GCN and convolutional neural networks (CNNs). Both models were computed over the structural graph of the mRNA molecule. The experimental results showed that GCN_GRU hybrid model outperform GCN_CNN model by a large margin during the test time. Validation of proposed hybrid models is performed by well-known evaluation measures. Among different deep neural networks, GCN_GRU based model achieved best scores on both public and private MCRMSE test scores with 0.22614 and 0.34152, respectively. Finally, GCN_GRU pre-trained model has achieved the highest AuC score of 0.938. Such proven outperformance of GCNs indicates that modeling RNA molecules using graphs is critical in understanding molecule degradation mechanisms, which helps in minimizing the aforementioned issues. To show the importance of the proposed GCN_GRU hybrid model, in silico experiments has been contacted. The in-silico results showed that our model pays local attention when predicting a given position’s reactivity and exhibits interesting behavior on neighboring bases in the sequence.

Locality preserving dense graph convolutional networks with graph context-aware node representations

The recognition of affect in speech has attracted a lot of interest recently; especially in the area of cognitive and computer sciences. Most of the previous studies focused on the recognition of basic emotions (such as happiness, sadness and anger) using categorical approach. Recently, the focus has been shifting towards dimensional affect recognition based on the idea that emotional states are not independent from one another but related in a systematic manner. In this paper, we design a continuous dimensional speech affect recognition model that combines acoustic and semantic features. We design our own corpus that consists of 59 short movie clips with audio and text in subtitle format, rated by human subjects in arousal and valence (A-V) dimensions. For the acoustic part, we combine many features and use correlation based feature selection and apply support vector regression. For the semantic part, we use the affective norms for English words (ANEW), that are rated also in A-V dimensions, as keywords and apply latent semantics analysis (LSA) on those words and words in the clips to estimate A-V values in the clips. Finally, the results of acoustic and semantic parts are combined. We show that combining semantic and acoustic information for dimensional speech recognition improves the results. Moreover, we show that valence is better estimated using semantic features while arousal is better estimated using acoustic features.

BackgroundClassical galactosemia (CG) is due to a severe deficiency of the galactose-1-phosphate uridyl-transferase (GALT), the main enzyme of galactose metabolism. Even early introduction of galactose-restricted diet fails to prevent long-term complications, including cognitive impairment, neurological and psychiatric problems, osteoporosis, premature ovarian failure and infertility. Detailed neuropsychological phenotyping is needed in order to better understand the relevant neurodevelopmental deficiencies and to develop effective treatment strategies.AimTo define specifically and significantly impaired neuropsychological traits in adult CG patients of the Swiss cohort.MethodsProspective cohort study. 22 CG patients, with confirmed genotype and low GALT activity, and 15 controls completed a computer-based neuropsychological test battery (CANTAB). Additionally, broad IQ evaluation was made for the CG patients.ResultsIn most outcome measures of the CANTAB tasks, CG patients performed significantly worse than controls. The deficits in CG patients were most prominent in tasks that involve rapid visual information processing and facial emotion recognition.ConclusionCG patients have specific cognitive problems such as impaired visual information processing and facial emotion recognition. The deficits in facial emotion recognition have not been described before and could help explain difficulties in social interactions often experienced by patients with CG.

In a wide range of social unrest events prediction, the dynamic graph convolutional network (DGCN) have been successfully leveraged to achieve reliable performance. The innovation of dynamic graph convolutional networks mainly focuses on capturing the temporal features of unrest events. Inspired by the dynamic graph convolutional network, we propose a new graph convolutional network model called Contextual Gated Graph Convolutional Network (CGGCN). We apply CGGCN to predict and analyze social unrest events. The CGGCN uses the contextual gated layer, which improves the layer-wise propagation rules of graph convolutional networks. The contextual gated layer can re-learn the keyword representation to capture the contextual semantic features of unrest events by using squeeze & excitation module. The principle of the squeeze & excitation module is to increase the weight of meaningful words for event prediction and suppress weaker ones. In this paper, we obtain historical texts including published news and short tweets related to social unrest events. Based on these historical texts data, the CGGCN can predict the occurrence of social unrest events. In addition, we propose a method for establishing the evolution graph of unrest events. In this way, we can use several core words to summarize the evolution of the event. Finally, we design experiments on the specific unrest events data sets. The experimental results show that the CGGCN leads by about 5% - 7% in the performance of prediction compared with other popular methods.