F2MCANet: joint frequency fusion and multi-channels attention neural network for surface defect recognition

Recent advancements in Internet technology have led to increased multi-modal data posting on social media, online shopping portals, and video repositories recognizing significance of inter-modal utterances before combining multiple modes. In this manuscript, Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis (MDF-DRGAN-SR-SA) is proposed. The input data are collected from CMU-MOSI dataset. Initially the input data is preprocessed using Subaperture Keystone Transform Matched Filtering (SAKTMF) is used to clean unwanted data. Then, feature extraction is done by Two-Sided Offset Quaternion Linear Canonical Transform (TSOQLCT) to extract unimodal features likes acoustic, textual, visual. Then the selected features are given to ADRGAN classifying Sentiment Recognition and Sentiment Analysis likes positive, negative, neutral. In general, ADRGAN doesn’t express some adaption of optimization strategies for determining optimal parameters to assure accurate classification of Sentiment Recognition and Sentiment Analysis. Hence, Northern Goshawk Optimization Algorithm (GOA) is proposed to enhance weight parameter of ADRGAN, which precisely classifies the Sentiment Recognition and Sentiment Analysis in positive, negative and neutral. The proposed model is implemented and its efficiency is evaluated utilizing some performance metrics likes accuracy, precision, specificity, sensitivity,F1-score. The MDF-DRGAN-SR-SA method provides 25.85%, 26.79% and 27.63% higher accuracy; 35.66%, 34.97% and 26.57% higher precision; 28.18%, 29.52% and 25.68% higher specificity is compared with existing method such as Two-Level Multimodal Fusion for SA in Public Security (TMDF-SA-PS), Multimodal SA Depend on Adaptive Modality-Specific Weight Fusion Network (MFN-SA-AMW) and Multimodal SA Utilizing Multi-tensor Fusion Network and Cross-modal Modeling(MTFN-SA) respectively.

This paper proposes a multi-scale time-frequency representation fusion network (MTRFN) for target recognition in synthetic aperture radar (SAR) imagery. Leveraging the spectral characteristics of six radar sub-views, the model incorporates a multi-scale representation fusion (MRF) module to extract discriminative frequency-domain features from two types of radar sub-views with high learnability. Additionally, physical scattering characteristics in SAR images are captured via time-frequency domain analysis. To enhance feature integration, a gated fusion network performs adaptive feature concatenation. The MRF module integrates a lightweight residual block to reduce network complexity and employs a coordinate attention mechanism to prioritize salient targets in the frequency spectrum over background noise, aligning the model’s focus with physical scattering principles. Furthermore, the model introduces an angular additive margin loss function during classification to enhance intra-class compactness and inter-class separability while reducing computational overhead. Compared with existing interpretable methods, the proposed approach combines architectural transparency with physical interpretability, thereby lowering the risk of recognition errors. Extensive experiments conducted on four public datasets demonstrate that the proposed MTRFN significantly outperforms existing benchmark methods. Comparative experiments using heat maps further confirm that the proposed physical feature-guided module effectively directs the model’s attention toward the target rather than the background.

Although deep learning algrithms have outstanding performance in biometrics and been paid more and more attention, triditional features for should not be ignored. In this paper, fusion of partition local binary patterns (PLBP) and convolutional neural networks (CNNs) is investigated in three schemes. In serial fusion (SF) method, PLBP feature is extracted and reshaped as the input of CNNS. Decision fusion (DF) carries out the PLBP with nearest neighour classifer and CNNs seperatelly and weighted fuses the results. For feature fusion (FF), PLBP feature is reshaped and weighted fused with the feature map of CNNs. To examine the proposed methods, NCUT data set with 2040 images from 204 hands is augmented using PCA. The results indicate that when the PLBP and CNNs are merged in FF scheme with weights of 0.2 and 0.8, our fusion method reaches a state-of-the-art recognition rate of 99.95%.

The temporal and spatial information of electroencephalogram (EEG) are essential for the emotion recognition model to learn the discriminative features. Hence, we propose a novel hybrid spatial-temporal feature fusion neural network (STFFNN) to extract the discriminative features and integrate complementary information. The generated power topographic maps, which capture dependencies among the electrodes, are fed to convolutional neural network (CNN) for spatial feature learning. Furthermore, instance normalizations (INs) and batch normalizations (BNs) within the CNN are appropriately combined to alleviate the individual difference and preserve the domain-invariant information. Meanwhile, a feedforward network is adopted for temporal feature learning. Due to the high dimensionality of EEG features, we propose a grid-search-based configurational optimization method to robustly reduce the dimensionality. Finally, inspired by the multimodal fusion strategies that leverage the complementarity of data to obtain more robust predictions, we utilize a bidirectional long short-term memory (Bi-LSTM) network for temporal and spatial feature fusion. To validate the effectiveness of the proposed method, the tenfold cross-validation experiments and subject-dependent experiments are both conducted on the DEAP database. The experimental results demonstrate that the proposed method achieves outstanding performance in emotion recognition with arousal and valence level.

In this letter, a novel multi-cue fusion (MCF) network for automatic modulation recognition is proposed. To exploit the potentially non-trivial and informative contents that previous works ignored, multi-cue learning is injected into neural network design. The MCF network consists of a signal cue multi-stream (SCMS) module and a visual cue discrimination (VCD) module. The SCMS module based on Convolutional Neural Network (CNN) and Independently Recurrent Neural Network (IndRNN) is built for modeling spatial-temporal correlations from two signal cues (i.e., In-phase/Quadrature (I/Q) and amplitude-phase (A/Φ)), which aims to explore various differences and leverage the complements from multiple data-form. In VCD module, raw I/Q data is converted to constellation diagrams as the visual cue to exploit the structural information utilizing the feature extracting capability of CNN. Experimental results on RadioML2016.10a and RadioML2018 datasets achieve 97.8% and 96.1% respectively, which outperform the state-of-the-art works.

In haptic recognition, pressure information is usually represented as an image, and then used for feature extraction and classification. Deep learning that processes haptic information in end-to-end manner has attracted attention. This study proposes a multiorder attentional spatial interactive convolutional neural network (MoAS-CNN) for haptic recognition. The asymmetric dual-stream all convolutional neural network with integrated channel attention module is applied for automatic first-order feature extraction. Later on, the spatial interactive features based on the overall feature map are computed to improve the second-order description capability. Finally, the multiorder features are summed to improve the feature utilization efficiency. To validate the MoAS-CNN, we construct a haptic acquisition platform based on three-scale pressure arrays and collect haptic letter-shape (A–Z) datasets with complex contours. The recognition accuracies are 95.73% for 16 × 16, 98.37% for 20 × 20 and 98.65% for 32 × 32, which significantly exceeds the traditional first- and second-order CNNs and local SIFT feature.

Early and Late Level Fusion of Deep Convolutional Neural Networks for Visual Concept Recognition

In recent years, convolutional neural networks (CNNs) have been successfully used in image recognition and image classification. General CNNs only use a single image as feature extraction. If the quality of the obtained image is not good, it is easy to cause misjudgment or recognition error. Therefore, this study proposes the feature fusion of a dual-input CNN for the application of face gender classification. In order to improve the traditional feature fusion method, this paper also proposes a new feature fusion method, called the weighting fusion method, which can effectively improve the overall accuracy. In addition, in order to avoid the parameters of the traditional CNN being determined by the user, this paper uses a uniform experimental design (UED) instead of the user to set the network parameters. The experimental results show that in the dual-input CNN experiment, average accuracy rates of 99.98% and 99.11% on the CIA and MORPH data sets are achieved, respectively, which is superior to the traditional feature fusion method.

Deep neural networks (DNNs), e.g., convolutional neural network (CNN), are able to learn effective features from wafer maps for dimensional reduction and feature extraction. However, very large image data are needed to train DNNs to obtain high generalization performance. It is still a difficult task due to the lack of sufficient labeled images with various defects. This article proposes a semisupervised deep transfer learning algorithm called joint feature and label adversarial network (JFLAN). JFLAN uses CNNs to extract transferable features of wafer maps and then introduces a multilayer domain adaptation and pseudolabel learning block based on the generative adversarial network (GAN). This effectively reduces the distribution discrepancy and the among-class distance of the transferable features. Finally, JFLAN transfers knowledge from wafer image source data collected offline and then achieves the goal of significantly improved accuracy of wafer defect recognition and realizes online adaptive defect recognition. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The defect recognition on wafer maps plays a key role to recognize fault sources of the semiconductor manufacturing processes. Transfer learning is able to use the existing labeled data to assist in the classification of unlabeled data and is very effective to solve the problem of small samples and nonstationary generalization errors. In particular, the infusion of adversarial learning in transfer learning will provide a new idea for deep feature learning. This article provides a novel method based on transfer learning to implement wafer map defect recognition (WMDR) to quickly identify defect root causes for yield enhancement. This article provides a novel way for quality control of semiconductor manufacturing processes based on transfer and adversarial learning.

Activity recognition plays pivotal role in enhancing functionality for prosthetic devices, ensuring seamless integration with users' movements. However, the complexity arises from the diverse data sources, including acceleration, angular velocity, joint angles, orientation, electromyography (EMG), and marker data, necessitating a robust approach to overcome challenges in information integration. The primary challenge lies in the effective utilization of multiple sensor modalities, each with unique characteristics and potential noise sources. The proposed solution addresses this by employing advanced sensor fusion techniques, such as Kalman filtering, during data collection. Synchronization and resampling ensure temporal consistency, while noise reduction techniques, such as low-pass filters, mitigate signal distortions. To further refine the process, a hybrid optimization-based feature selection Adaptive Step Size in Marine Predators Algorithm (ASSMPA) is introduced, focusing on marker data features. ASSMPA synergizes Marine Predators Algorithm (MPA) and Pathfinder Algorithm (PFA) for optimal feature selection in marine predator pathfinding tasks. The Feature Fusion step integrates attention mechanisms to dynamically weigh the significance of different sensor modalities during the fusion process. This strategic fusion enhances the overall performance of the Multi-Modal Hierarchical Neural Network (MMHNN). The proposed model is implemented using Python.

Images are powerful tools with which to convey human emotions, with different images stimulating diverse emotions. Numerous factors affect the emotions stimulated by the image, and many researchers have previously focused on low-level features such as color, texture and so on. Inspired by the successful use of deep convolutional neural networks (CNN) in the visual recognition field, we used a data augmentation method for small data sets to gain the sufficient number of the training dataset. In this paper, we use low-level features (color and texture features) of the image to assist the extraction of advanced features (image object category features and deep emotion features of images), which are automatically learned by deep networks, to obtain more effective image sentiment features. Then, we use the stack sparse auto-encoding network to recognize the emotions evoked by the image. Finally, high-level semantic descriptive phrases including image emotions and objects are output. Our experiments are carried out on the IAPS and GAPED data sets of the dimension space and the artphoto data set of the discrete space. Compared with the traditional manual extraction methods and other existing models, our method is superior to in terms of test performance.

Emotion recognition has a wide range of applications in human-computer interaction, marketing, healthcare, and other fields. In recent years, the development of deep learning technology has provided new methods for emotion recognition. Prior to this, many emotion recognition methods have been proposed, including multimodal emotion recognition methods, but these methods ignore the mutual interference between different input modalities and pay little attention to the directional dialogue between speakers. Therefore, this article proposes a new multimodal emotion recognition method, including a cross modal context fusion module, an adaptive graph convolutional encoding module, and an emotion classification module. The cross modal context module includes a cross modal alignment module and a context fusion module, which are used to reduce the noise introduced by mutual interference between different input modalities. The adaptive graph convolution module constructs a dialogue relationship graph for extracting dependencies and self dependencies between speakers. Our model has surpassed some state-of-the-art methods on publicly available benchmark datasets and achieved high recognition accuracy.

An automated speech recognition system converts a speech signal into a compact, coded representation that correlates to a speech phoneme set. A number of different neural network pattern matching schemes may be used to perform the necessary speech coding. An integrated user interface guides a user unfamiliar with the details of speech recognition or neural networks to quickly develop and test a neural network for phoneme recognition. To train the neural network, digitized voice data containing known phonemes that the user wants the neural network to ultimately recognize are processed by the integrated user interface. The digitized speech is segmented into phonemes with each segment being labelled with a corresponding phoneme code. Based on a user selected transformation method and transformation parameters, each segment is transformed into a series of multiple dimension vectors representative of the speech characteristics of that segment. These vectors are iteratively presented to a neural network to train/adapt that neural network to consistently distinguish and recognize these vectors and assign an appropriate phoneme code to each vector. Simultaneous display of the digitized speech, segments, vector sets, and a representation of the trained neural network assist the user in visually confirming the acceptability of the phoneme training set. A user may also selectively audibly confirm the acceptability of the digitization scheme, the segments, and the transform vectors so that satisfactory training data are presented to the neural network. If the user finds a particular step or parameter produces an unacceptable result, the user may modify one or more of the parameters and verify whether the modification effected an improvement in performance. The trained neural network is also automatically tested by presenting a test speech signal to the integrated user interface and observing both audibly and visually automatic segmentation of the speech, transformation into multidimensional vectors, and the resulting neural network assigned phoneme codes. A method of decoding such phoneme codes using the neural network is also disclosed.

Many deep neural networks have been constructed for the pansharpening task. However, the differences between the high and low frequencies in images are not considered in some DNN-based pansharpening methods. As high and low frequencies have different information of images, it is difficult for the same network to learn and reconcile the two kinds of frequencies. Considering the aforementioned differences, we propose a new pansharpening network to fuse the high and low frequencies in low spatial resolution multispectral and panchromatic images separately. Specifically, a high and low frequency fusion network is constructed, which is composed of a high-frequency fusion network and a low-frequency fusion network. In the high-frequency fusion network, skip attention is introduced into U-Net to better retain the high frequencies in feature maps. The low-frequency fusion network uses the involution to capture the dependency among the channels of feature maps. Experiments on the GeoEye-1 dataset reveal that the proposed network outperforms some state-of-the-art methods. The code can be accessed at https://github.com/RSMagneto/HLF-Net.

The text embedded in scene images can be seen everywhere in our lives. However, recognizing text from natural scene images is still a challenge because of its diverse shapes and distorted patterns. Recently, advanced recognition networks generally treat scene text recognition as a sequence prediction task. Although achieving excellent performance, these recognition networks consider the feature map cells as independent individuals and update cells state without utilizing the information of their related cells. And the local receptive field of traditional convolutional neural network (CNN) makes a single cell that cannot cover the whole text region in an image. Due to these issues, the existing recognition networks cannot extract the global context information in a visual scene. To deal with the above problems, we propose a Multi-head Self-relation Network(MSRN) for scene text recognition in this paper. The MSRN consists of several multi-head self-relation layers, which are designed for extracting the global context information of a visual scene. Then the information of the related cells can be fused by multi-head self-relation layer. Furthermore, experiments over several public datasets demonstrate that our proposed recognition network achieves superior performance on several benchmark datasets including IC03, IC13, IC15, SVT-Perspective.