Structural and positional ensembled encoding for Graph Transformer

The transformer architecture has achieved significant success in computer vision, but challenges remain, particularly in the area of positional encoding within the Vision Transformer (ViT). ViT uses a basic positional encoding method, where positional information is integrated with image tokens through simple addition and random initialization. This approach limits the effective representation of positional information. To address these limitations, we propose Fused Features Adaptive Position Embedding (FFAPE), a novel method that introduces independent linear layers at each level of the model to enhance the flexibility and adaptability of positional encoding. Additionally, FFAPE incorporates both image and edge information into the positional encoding to strengthen the relationship between tokens and their positional encodings. Extensive experiments on multiple datasets demonstrate significant performance improvements in Transformer-based visual models. Specifically, FFAPE-ViT and FFAPE-CVT models show substantial performance gains over their original counterparts, ViT and CVT. This study offers new insights into extending the application of ViT in complex visual scene processing and underscores the considerable potential of fused features adaptive position embedding methods in optimizing the performance of ViT.

Generative models for dialog systems have gained much interest because of the recent success of RNN and Transformer based models in tasks like question answering and summarization. Although the task of dialog response generation is generally seen as a sequence to sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn meaningful utterance and conversation level features, Sordoni et al. (2015b), Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer-based models dominating the seq2seq problems lately, the natural question to ask is the applicability of the notion of hierarchy in transformer-based dialog systems. In this paper, we propose a generalized framework for Hierarchical Transformer Encoders and show how a standard transformer can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog systems through a wide range of experiments.

Graph Neural Networks (GNNs) are deep learning methods which provide the current state of the art performance in node classification tasks. GNNs often assume homophily – neighboring nodes having similar features and labels–, and therefore may not be at their full potential when dealing with non-homophilic graphs. In this work, we focus on addressing this limitation and enable Graph Attention Networks (GAT), a commonly used variant of GNNs, to explore the structural information within each graph locality. Inspired by the positional encoding in the Transformers, we propose a framework, termed Graph Attentional Networks with Positional Embeddings (GAT-POS), to enhance GATs with positional embeddings which capture structural and positional information of the nodes in the graph. In this framework, the positional embeddings are learned by a model predictive of the graph context, plugged into an enhanced GAT architecture, which is able to leverage both the positional and content information of each node. The model is trained jointly to optimize for the task of node classification as well as the task of predicting graph context. Experimental results show that GAT-POS reaches remarkable improvement compared to strong GNN baselines and recent structural embedding enhanced GNNs on non-homophilic graphs.

In recent times, many studies concerning surgical video analysis are being conducted due to its growing importance in many medical applications. In particular, it is very important to be able to recognize the current surgical phase because the phase information can be utilized in various ways both during and after surgery. This paper proposes an efficient phase recognition network, called MomentNet, for cholecystectomy endoscopic videos. Unlike LSTM-based network, MomentNet is based on a multi-stage temporal convolutional network. Besides, to improve the phase prediction accuracy, the proposed method adopts a new loss function to supplement the general cross entropy loss function. The new loss function significantly improves the performance of the phase recognition network by constraining un-desirable phase transition and preventing over-segmentation. In addition, MomnetNet effectively applies positional encoding techniques, which are commonly applied in transformer architectures, to the multi-stage temporal convolution network. By using the positional encoding techniques, MomentNet can provide important temporal context, resulting in higher phase prediction accuracy. Furthermore, the MomentNet applies label smoothing technique to suppress overfitting and replaces the backbone network for feature extraction to further improve the network performance. As a result, the MomentNet achieves 92.31% accuracy in the phase recognition task with the Cholec80 dataset, which is 4.55% higher than that of the baseline architecture.

Graph Transformers have demonstrated outstanding performance across various graph-based applications. Despite their success, applying them to large-scale graphs presents significant scalability challenges, limiting their practical use in industrial environments. Recent studies have attempted to overcome this challenge by focusing on the spatial domain of graphs, leading to the development of various scalable models. However, these approaches neglect the spectral characteristics of graphs, which are crucial for adaptively extracting information from full frequency bands based on the graph's inherent properties. As a result, existing scalable Graph Transformers tend to rely heavily on low-frequency features, overlooking valuable mid- and high-frequency information. This paper proposes the Graph Filter Transformer (GFformer), a framework designed to effectively extract full frequency information from large-scale graphs. Unlike existing Graph Transformers, GFformer integrates graph filters into the Transformer architecture, thereby enhancing its ability to model both structural and frequency-related properties. Utilizing the proposed Spectral Token Converter (ST-Converter), GFformer generates a unique spectral token sequence for each node by incorporating features from diverse frequencies that act as tokens. This design enables the independent learning of node representations in parallel and supports mini-batch training with flexible batch sizes, making GFformer highly scalable. ST-Converter employs spectral graph filters, including low-, mid-, and high-pass filters, to extract features serving as tokens. Consequently, each sequence encompasses features from various frequencies, enabling GFformer to capture comprehensive frequency information effectively. Extensive experiments on datasets of varying scales, including both homophilic and heterophilic graphs, consistently demonstrate that GFformer outperforms existing representative methods.

Deep learning has been instrumental in feature extraction from various data types, such as images and sequences, which inherently possess oriented structures. However, graph data, which delineate relationships between entities, lack such structures, posing challenges for embedding nodes uniquely in high-dimensional spaces. This paper introduces a novel position encoding method, Numbering as a Position Encoding (NAPE), which utilizes precomputed random walks and Hamming distances to approximate node orderings in graphs. NAPE assigns unique integers to nodes based on their local neighbourhoods, facilitating the generation of sinusoidal vectors that uniquely encode node positions. This method mitigates the computational expenses and non-uniqueness issues associated with eigenvector-based approaches, ensuring distinct node embeddings for various graph-based applications, including human action recognition. Our approach, scalable with computational complexity <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(|V|^{2})$ </tex-math></inline-formula>, demonstrates improved efficiency and effectiveness in embedding nodes uniquely. Notably, NAPE improves the accuracy of action recognition models applied to human skeletal graphs. The versatility of NAPE is highlighted by its ability to generalize across different models, making it a robust solution for training action recognition systems with large parameter sizes.

Thangka is a unique art style in Tibetan culture, and the secondary creation of thangka images can help people understand the stylistic characteristics of thangka, and promote the cultural heritage and protection. Aiming at the problems of artifacts, local image blurring and inappropriate processing of detail information in the new image generated by the current thangka image style migration technology, we propose a thangka style migration model FPC-EI, which integrates the feature position encoding and error correction. The two-dimensional relative position encoding is used to obtain the corresponding position information between the content sequences and the style sequences, and to realize the alignment and matching of the image features; the coordinate attention mechanism module reinforces the use of two-dimensional relative position encoding to get the position information between content sequences and style sequences, and realizes the alignment and matching of image features. The coordinate attention mechanism module enhances the ability of the encoder to capture the texture details of the Tangkas and the content images by utilizing 2D relative position coding, and the dense deep inverse projection network uses a module that continuously samples up and down the sampling layer to correct the error and guide the model to improve the quality of the generated images. The experimental results of model training on the self-constructed thangka dataset and arbitrary image dataset show that compared with the style migration models such as AdaIN, ArtFlow, AdaAttN, etc., the SSIM index of FPC-EI is improved by 32% on average, and the LPIPS and MSE indexes are reduced by 7.3% and 2.7% on average, and the structure of the content, the fusion of the features and the local details of the image can be effectively preserved. The FPC-EI can effectively retain the content structure, feature fusion and local details of the image.

The rising demands of more reliable and stable electrical systems attach importance to accurate Remaining Useful Life (RUL) prediction of the lithium-ion batteries. As artificial intelligence and machine learning techniques advance, data-driven methods especially deep learning algorithms have become the rising star in RUL prediction. Recurrent Neural Networks (RNNs) and their variants such as Long Short Term Memory have proven effectiveness in various sequential tasks. However, due to its iterative nature along the time axis, RNNs take much time for information to flow through the network for prediction. Inspired by recent advance brought by Transformer in sequence transduction tasks, we proposed the attention mechanism based Convolutional Neural Network (CNN) with positional encoding to tackle this problem. The attention mechanism enables the network to focus on specific parts of sequences and positional encoding injects position information while utilizing the parallelization merits of CNN on GPUs. Empirical experiments show that the proposed approach is both time effective and accurate in battery RUL prediction.

Spatially-enhanced Spiking neural network for efficient point cloud analysis.

Using graph transformation as a formalism to specify access control has several advantages, from the intuition provided by the visual aspect to the precise semantics and the systematic verification of constraints. Graph Transformations provide a uniform and precise framework for the specification of access control policies. After reviewing the basic notations of graph transformation, this article presents specification formalisms for workflow delegation policies using graph transformation. The authorization states are represented by graphs and state transition by graph transformation. The proposed formalization provides an intuitive description for the manipulation of graph structures as they occur in workflow delegation and a precise specification of consistency conditions on graphs and graph transformations. We specifies a type graph to represents the type information in the graph transformation for workflow delegation, a set of rules to build the system states and sets of positive and negative constraints to specify wanted and unwanted framework.

Purpose In practical engineering applications, the working load of the rotor system keeps changing, and its working environment is severely polluted by noise, leading to a decline in the performance of traditional fault diagnosis methods. To better extract fault features in a noisy environment and improve the diagnostic accuracy, a fault diagnosis model based on the Visibility Graph algorithm and Graph Transformer is proposed. Design/methodology/approach This paper proposes a fault diagnosis model based on Visibility Graph algorithm and Graph Transformer. Firstly, the spectrum of the original vibration signal is constructed into a Visibility Graph, with the spectrum mean value as the node value and the visibility relationship between nodes defined. Subsequently, the graph transformer is used to extract the global node features and topological structure information in the graph, and the long-range dependencies are captured through the self-attention mechanism. Finally, the fully connected layer and the Softmax classifier are combined to achieve end-to-end fault diagnosis. Findings Experimental results on the MAFAULDA dataset demonstrate that the proposed model achieves a diagnostic accuracy of 99.91% under noise-free conditions, surpassing comparative methods by 1.12–14.53%. Under −10 dB Gaussian white noise interference, the model maintains an accuracy of 74.36%, outperforming baseline models by 1.42–48.49%. Further cross-scenario verification on the CWRU and XJTU-SY datasets shows that this method has strong robustness against compound faults and extreme noise. The results show that this method effectively enhances the ability to extract fault features in noisy environments, providing an efficient and reliable solution for intelligent monitoring of rotor systems. Originality/value In order to effectively realize end-to-end intelligent fault diagnosis, enhance the robustness and anti-noise performance of the model, a new fault diagnosis method of the rotor system based on the Visibility Graph algorithm and Graph Transformer is proposed in this paper. By constructing the vibration signal spectrum as a viewable data and then effectively capturing the topological relationship of the graph in the global range through the self-attention mechanism, the fault diagnosis is realized in the form of node classification. Finally, the fault analysis and classification of the vibration signals of the rotor system under the background of no noise and noise are carried out.

A survey and comparison of transformation tools based on the transformation tool contest

This paper proposes an improved Sliding Mode Observer (SMO). Compared with the traditional SMO, it can estimate position and speed information of the rotor more accurately. Its stability is proved by Lyapunov Stability. A control scheme of Permanent Magnet Synchronous Motor (PMSM) under Direct Torque Control (DTC) is designed, using the improved SMO as the feedback of speed information. The results show that the proposed method works well without position encoder. The improved SMO guarantees DTC scheme quick response and strong robustness.

Slimmable transformer with hybrid axial-attention for medical image segmentation

We examine bidirectional brain-machine interfaces that control external devices in a closed loop by decoding motor cortical activity to command the device and by encoding the state of the device by delivering electrical stimuli to sensory areas. Although it is possible to design this artificial sensory-motor interaction while maintaining two independent channels of communication, here we propose a rule that closes the loop between flows of sensory and motor information in a way that approximates a desired dynamical policy expressed as a field of forces acting upon the controlled external device. We previously developed a first implementation of this approach based on linear decoding of neural activity recorded from the motor cortex into a set of forces (a force field) applied to a point mass, and on encoding of position of the point mass into patterns of electrical stimuli delivered to somatosensory areas. However, this previous algorithm had the limitation that it only worked in situations when the position-to-force map to be implemented is invertible. Here we overcome this limitation by developing a new non-linear form of the bidirectional interface that can approximate a virtually unlimited family of continuous fields. The new algorithm bases both the encoding of position information and the decoding of motor cortical activity on an explicit map between spike trains and the state space of the device computed with Multi-Dimensional-Scaling. We present a detailed computational analysis of the performance of the interface and a validation of its robustness by using synthetic neural responses in a simulated sensory-motor loop.