MSST-ResNet: Deep multi-scale spatiotemporal features for robust visual object tracking

Traditional power quality disturbances(PQDs) recognition methods are based on selecting features manually and then use classifiers to classify. For huge amounts of data, this method has the problem of insufficient features or redundant features, which leads to low recognition accuracy. With the development of deep learning technology, this paper proposes a PQDs identification method based on a deep residual network. This method links the two sets of convolutional layers in the traditional convolutional network into a residual block through short-cut. Then, it learns layer by layer through residual blocks, acquiring the deep features of data. Thus it realizes the identity mapping of the redundant layer by short-line connections to solve the network degradation problem of deep networks, and reduces the difficulty of network training. Finally, the softmax layer is used for classification. In order to test the anti-noise ability of the network, the deep residual network model is trained with the data added with Gaussian white noise. Simulation results show that, comparing with ordinary convolutional networks, the method proposed can accurately identify 7 types of single disturbances and 5 types of composite disturbances in different noise conditions. Furthermore, it has higher robustness and better network convergence speed.

Recent studies have demonstrated that a well designed deep convolutional neural network (CNN) model achieves competitive performances on detecting the presence of secret message in digital images, compared with the classical rich model based steganalysis. In this paper, we propose to investigate a category of very deep CNN model–the deep residual network (DRN), for steganalysis. DRN is suitable for steganalysis from two aspects. For the first, the DRN model usually contains a large number of network layers, which proves to be effective to capture the complex statistics of digital images. For the second, DRN's residual learning (ResL) method actively strengthens the signal coming from secret messages, which is extremely beneficial for the discrimination between cover images and stego images. Comprehensive experiments on standard dataset show that the DRN model achieves very low detection error rates for the state of arts steganographic algorithms. It also outperforms the classical rich model method and several recently proposed CNN based methods.

The research aims to explore the application of the deep residual network-spatio-temporal fusion model in the estimation of net primary productivity. The deep residual network-spatiotemporal fusion model is an advanced deep learning model specifically designed to process and fuse remote sensing images from different time points to generate synthetic images with higher spatiotemporal resolution. The study first collected remote sensing data at different time points within the research area, and then processed these data using the deep residual network-spatio-temporal fusion model to reconstruct images with higher resolution. Then, by analyzing the features in these high-resolution images, the model estimates the net primary productivity of the study area. The research results indicate that the deep residual network model combined with the Carnegie-Ames-Stanford model was used to estimate the net primary productivity of vegetation. The mean value of the proposed deep residual network model in estimating net primary productivity of vegetation is 455 gC/m 2 /month, with a standard deviation of 25 gC/m 2 /month, and the coefficient of determination R 2 reaches as high as 0.98, which is significantly better than other algorithms. Compared with the mean value of net primary productivity of vegetation measured on the ground in the study area during the same period of 448 gC/m 2 /month, the deviation is only 7 gC/m 2 /month, with a deviation rate of 1.56%. This further verifies the accuracy of the model’s estimation results. The deep residual network-spatio-temporal fusion model can enhance the accuracy of net primary productivity estimation, providing a new tool for ecological research and environmental monitoring.

Recently, many correlation filter-based tracking methods have received lots of attention and achieved great success in visual object tracking. Among correlation filter-based methods, the most influential one is kernelized correlation filter (KCF) which has excellent performance both in efficiency and accuracy. However, due to the virtual nature of cyclic shifts samples, the training and detecting of KCF are imprecise. To alleviate the influence of virtual samples, we take the following two measures. 1) We extract image patches at positions of samples which have local maximum KCF responses and treat them as candidates. We further evaluate their true responses. 2) Instead of estimate the target position directly according to the KCF results, we use the sparse prototypes (SP) as the target model to evaluate the similarities between candidates and target. The results of KCF and SP are combined by adaptive weight to estimate the target position. In addition, KCF degrades due to its unreasonable update scheme. To do reliable update, we set different update modes and generate an adaptive update rate based on two tracking confidence indices. Experiments on a commonly used tracking benchmark show that the proposed method improves KCF about 8% on the average success rate and 10% on the precision, and achieves better performance than other state-of-the-art trackers.

Seismic interpolation via multi-scale HU-Net

Recent research on single image super-resolution (SISR) using convolutional neural networks (CNNs) with the utilization of residual structures and attention mechanisms to utilize image features has demonstrated excellent performance. However, previous SISR techniques mainly integrated extracted image features within a deep or wide network architecture, ignoring the interaction between multiscale features and the diversity of features. At the same time, SISR is also a typical ill-posed problem in that it allows for several predictions for a given LR image. These problems limit the great learning ability of CNNs. To solve these problems, we propose a closed-loop residual attention network (CLRAN) to extract and interact with all the available diversity of features features efficiently and limit the space of possible function solutions. Specifically, we design an enhanced residual attention block (ERA) to extract features, and it dynamically assigns weight to the internal attention branches. The ERA combines multi-scale block (MSB) and enhanced attention mechanism (EAM) base on the residual module. The MSB adaptively detects multiscale image features of different scales by using different 3 × 3 convolution kernels. The EAM combines multi-spectral channel attention (MSCA) and spatial attention (SA). Therefore, the EAM extracts different frequency component information and spatial information to utilize the diversity features. Furthermore, we apply the progressive network architecture and learn an additional map for model monitoring, which forms a closed-loop with the mapping already learned by the LR to HR function. Extensive experiments demonstrate that our CLRAN outperforms the state-of-the-art SISR methods on public datasets for both ×4 and ×8, proving its accuracy and visual perception.

Person re-identification aims to associate images of the same person over multiple non-overlapping camera views at different times. Depending on the human operator, manual re-identification in large camera networks is highly time consuming and erroneous. Automated person re-identification is required due to the extensive quantity of visual data produced by rapid inflation of large scale distributed multi-camera systems. The state-of-the-art works focus on learning and factorize person appearance features into latent discriminative factors at multiple semantic levels. We propose Deep Parallel Feature Consensus Network (DeepPFCN), a novel network architecture that learns multi-scale person appearance features using convolutional neural networks. This model factorizes the visual appearance of a person into latent discriminative factors at multiple semantic levels. Finally consensus is built. The feature representations learned by DeepPFCN are more robust for the person re-identification task, as we learn discriminative scale-specific features and maximize multi-scale feature fusion selections in multi-scale image inputs. We further exploit average and max pooling in separate scale for person-specific task to discriminate features globally and locally. We demonstrate the re-identification advantages of the proposed DeepPFCN model over the state-of-the-art re-identification methods on three benchmark datasets - Market1501, DukeMTMCreID, and CUHK03. We have achieved mAP results of 75.8%, 64.3%, and 52.6% respectively on these benchmark datasets. KeywordsPerson re-identificationDeep learningArchitecture

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Aiming at solving the problems of limited training data, single input information, and limited diagnostic accuracy under the influence of strong background noise in fault diagnosis of rotating machinery, this paper proposes a fault diagnosis method based on the combination of discriminant correlation analysis (DCA) and convolutional neural network (CNN). Firstly, the original vibration signal is divided into several segments in the time domain, and the training data is directly processed by one CNN branch to extract multi-scale time domain features. Simultaneously, the divided data is subjected to discrete wavelet transform (DWT), and processed by another branch of CNN to extract multi-scale time-frequency features. Then, the DCA feature fusion mechanism is adopted to fuse the two-domain features extracted in the parallel branches to improve the model’ detection ability. Finally, the fused features are input into the deep CNN for training and learning to extract new features and output the classification results. Through the experimental analysis of two different types of data, the results show that the proposed method can be used for fault diagnosis of rotating machinery effectively. Compared with the single CNN network, the proposed method combines the multi-domain multi-scale feature extraction module with the DCA feature fusion module to enrich the feature information extraction ability. At the same time, the network performance is improved to get higher fault classification accuracy higher.

The classification of hyperspectral images (HSIs) is one of the most popular topics in the remote sensing community. Numerous feature extraction methods have been proposed to improve the classification accuracy of HSIs. Recently, deep features extracted by convolution neural network (CNN) have been introduced into the classification process of HSIs. Due to the nonlinear and invariant advantages of the features, CNN methods provide a powerful tool for representing geographic objects and classifying HSIs. However, traditional deep features only extracted at pixel-level and often neglect multiscale characteristics of geographic objects. In this study, a new deep feature extraction method is proposed, which takes advantage of multi-scale object analysis and the CNN model. Firstly, multiscale image objects are obtained by the multiscale segmentation algorithm and multiscale low-level features of objects are extracted. Secondly, the CNN is devoted to obtain deep features from low-level object features at each scale, respectively. Thirdly, the obtained deep features at all scales are stacked and fed to one fully connected layer to extract the multiscale deep learning features for classification. Finally, the logistic regression classifier is applied to hyperspectral image (HSI) classification based on object-oriented multiscale deep features. The proposed method was carried out on three widely used hyperspectral data sets: University of Pavia, Salinas, and Washington DC. The results reveal that the proposed method provides better results than other state-of-the-art methods.

A novel intelligent fault diagnosis method of rotating machinery based on signal-to-image mapping and deep Gabor convolutional adaptive pooling network

Lung field segmentation (LFS) in chest x-rays (CXR) is a key step for the computer-assisted diagnosis of respiratory diseases. However, achieving precise delineation remains challenging in images with poor contrast and superimposed anatomical structures, often resulting in imprecise lung outlines in deep learning (DL) models. Furthermore, existing models often lack generalizability, performing well on specific datasets but failing on unseen domains due to variations in imaging equipment, acquisition protocols, and patient demographics. To overcome these limitations, we propose a two-phase hybrid approach for robust and generalizable LFS in CXR. The first phase employs a heuristic approach to extract and refine multiscale edge features: The Laplacian of Gaussian (LoG) filter detects closed contours that effectively highlight the overall lung shape, whereas the Canny detector captures the finer, high-frequency edge details that preserve the lung outlines often missed by DL models due to overlapping structures. In the second phase, the enriched multiscale feature maps are fused along with the original contrast inputs and then used to train and evaluate various U-Net architectures. The proposed approach is evaluated on three benchmark datasets: Montgomery County (MC), Shenzhen Hospital (SH), and Japanese Society of Radiological Technology (JSRT). Experiments conducted on individual datasets using five-fold cross-validation and testing on a completely unseen separate test set to simulate real-world settings revealed that the proposed edge-assisted filters, especially the Canny filter, significantly improve segmentation performance metrics. Among all U-Net variants, deep attention U-Net achieved the highest performance using a consistent channel combination, attaining a dice coefficient of 0.9815, a Jaccard score of 0.9624 on the JSRT dataset. The proposed approach achieved a dice gain of +0.0011 to +0.0034 and an IoU gain of +0.009 to +0.0034 across the three datasets compared with the baseline configuration using original images. Furthermore, cross-dataset validation also depicted improvement gains in dice, IoU, and other metrics, demonstrating the generalizability of the proposed approach. Further cross-dataset validation confirmed the framework's strong generalization capabilities, demonstrating stable performance even on unseen domains. These findings validate the proposed method as a reliable and effective solution for LFS in diverse and clinically challenging environments, offering promising benefits for respiratory diagnosis and patient care.

In this paper, a scale adaptive target tracking algorithm based on deep residual network and kernel correlation filter is proposed. Although kernel correlation filter is a fast and effective target tracking algorithm, its tracking effect is not ideal in the case of the occlusion, blur and scale change caused by the fast moving target in the real environment. To deal with these problems, corresponding measures are integrated in the proposed new algorithm. Firstly, the structure of the ResNet50 network was adjusted and trained. The deep residual network was used to extract target features and integrate the kernel correlation filter algorithm to carry out adaptive response graph fusion to find the target location. Then, we add the scale estimation module, and use the HOG feature to replace the original deep feature training scale filter. Based on the structure of binary sort tree, binary search is carried out on the scale, and the scale discriminant index is used to judge whether the target has reached the appropriate scale. Experimental results show that our proposed algorithm can achieve scale adaptation of target tracking, and the tracking performance is further improved.

Wind turbine rolling bearings are crucial components for ensuring the reliability and stability of wind power systems. Their failure can lead to significant economic losses and equipment downtime. Therefore, the accurate diagnosis of bearing faults is of great importance. Although existing deep learning fault diagnosis methods have achieved certain results, they still face limitations such as inadequate feature extraction capabilities, insufficient generalization to complex working conditions, and ineffective multi-scale feature capture. To address these issues, this paper proposes an advanced fault diagnosis method named the two-stream feature fusion convolutional neural network (TSFFResNet-Net). Firstly, the proposed method combines the advantages of one-dimensional convolutional neural networks (1D-ResNet) and two-dimensional convolutional neural networks (2D-ResNet). It transforms one-dimensional vibration signals into two-dimensional images through the empirical wavelet transform (EWT) method. Then, parallel convolutional kernels in 1D-ResNet and 2D-ResNet are used to extract multi-scale features, respectively. Next, the Convolutional Block Attention Module (CBAM) is introduced to enhance the network’s ability to capture key features by focusing on important features in specific channels or spatial areas. After feature fusion, CBAM is introduced again to further enhance the effect of feature fusion, ensuring that the features extracted by different network branches can be effectively integrated, ultimately providing more accurate input features for the classification task of the fully connected layer. The experimental results demonstrate that the proposed method outperforms other traditional methods and advanced convolutional neural network models on different datasets. Compared with convolutional neural network models such as LeNet-5, AlexNet, and ResNet, the proposed method achieves a significantly higher accuracy on the test set, with a stable accuracy of over 99%. Compared with other models, it shows better generalization and stability, effectively improving the overall performance of rolling bearing vibration signal fault diagnosis. The method provides an effective solution for the intelligent fault diagnosis of wind turbine rolling bearings.

The sEMG-based gesture recognition has great potential in human-computer interaction. However, the current approaches are far from optimal. In this paper, a novel convolutional neural network which combines the multi-scale kernal and feature fusion (MKFF-CNN) is proposed. This model could extract multi-scale features and make full use of these feature maps. One dataset called “gForce dataset” is recorded in this work, comprising 13 able-bodied participants. To verify the effectiveness of the proposed model, evaluations on both the gForce dataset and the Ninapro DB6 are conducted. The experimental results demonstrate that, MKFF-CNN achieves an accuracy of 97.65% on the gForce dataset, which is 1.39% higher than that of the subnetwork without early fusion, and at least 6.54% higher than that of the subnetwork with single-scale convolution kernel. On the Ninapro DB6, MKFF-CNN achieves 98.52% accuracy, which is 1.3% higher than the state-of-the-art works, showing the superiority of the proposed model.