Emotion recognition using speech and neural structured learning to facilitate edge intelligence

In order to effectively improve the expressiveness and classification accuracy of fault signal characteristics of equipment, this paper proposed a fault diagnosis method based on Mel-Frequency Cepstrum Coefficient (MFCC) fusion and Support Vector Machines (SVM). First of all, the MFCC features, the Wavelet Packet Decomposition Energy features and the Zero-crossing rate (ZCR) features of the signal are separately extracted. Then, linearly combining the three features based on the MFCC features to obtain the MFCC fusion features. And the SVM classifier is used to classify the faults. Experiments show that compared with the traditional single MFCC features and Wavelet Packet Decomposition Energy features, the MFCC fusion features and SVM method have higher classification accuracy under the same noise environment.

Emotion is the psychological and physiological response of a person or animal to a specific stimulus or event, and it can also have an impact on cognition, decision-making, and behavior. Research on emotion recognition has wide applications in various fields, such as dialogue and communication with machines, and in the medical field. At present, most emotion recognition systems are based on PC or embedded platforms. Compared with traditional emotion recognition platforms, FPGA has good parallel computing ability and can process multiple data streams simultaneously, making it suitable for complex pattern recognition in emotion recognition. This article chooses ZYNQ as the development platform, combined with the collaborative ability of FPGA and ARM processors, to build a complete hardware application system that can carry neural networks. Secondly, in response to the problems of complex computation and high storage resource consumption in existing neural network computing frameworks, this paper adopts a lightweight neural network MobileNetV3 and implements and improves it. The model is encapsulated and ported using the Vivado HLS development tool and C++. The hardware IP core is called on the hardware platform to achieve ZYNQ based facial emotion classification and recognition, and the power consumption of hardware resources is analyzed.

Emotion detection, hence, is an important step toward making human-computer interaction a more enhanced process, where systems are made capable of identifying and responding to the emotional state of users. In fact, multimodal emotion detection systems in which both auditory and visual information are fused are emerging, and these approaches toward expressive emotional states are complementary and robust. Multimodal systems enhance the quality of interacting and, through many applications, can diagnose emotional disorders, monitor automotive safety, and improve human-robot interactions. In nature, the high-dimensional space and dynamic threats have resulted in obtaining low accuracy and high computational cost in applying the traditional models based on single-modality data. On the other hand, multimodal systems explore the synergy between audio and visual data, giving better performance and higher accuracy in inferring subtle emotional expressions. The latest improvement was done on these systems using recent advancements in transfer learning and deep learning techniques.That being said, this research Proposal devises a multimodal emotion recognition system integrating speech and face information through transfer learning for improved accuracy and robustness. Serving this purpose, the objectives of this research entail the effective comparison among different transfer-learning strategies, including the impact of pre-trained models in speech-based emotion recognition, and to introduce the role of voice activity detection in the process. Advanced neural network architectures like Spatial Transformer Networks and bidirectional LSTM in facial emotion recognition will also be tested. Early and late fusion strategies will also be used to find the best strategy for combining speech and facial data.This research will target several challenges that involve the complexity of data, balancing of the model performance-robustness balance, computational limitations, and standardization of evaluations in developing a working and robust emotion recognition system to enhance digital interaction and apply in practical areas. The goal is to create a system that oversteps the limitation of single-modality models through state-of-the-art advances in deep learning, as well as front-line improvements in transfer learning, in the manner of emotion detection performance.

SummaryEmotion recognition is challenging for understanding people and enhances human–computer interaction experiences, which contributes to the harmonious running of smart health care and other smart services. In this paper, several kinds of speech features such as Mel frequency cepstrum coefficient, pitch, and formant were extracted and combined in different ways to reflect the relationship between feature fusions and emotion recognition performance. In addition, we explored two methods, namely, support vector machine (SVM) and deep belief networks (DBNs), to classify six emotion status: anger, fear, joy, neutral status, sadness, and surprise. In the SVM‐based method, we used SVM multi‐classification algorithm to optimize the parameters of penalty factor and kernel function. With DBN, we adjusted different parameters to achieve the best performance when solving different emotions. Both gender‐dependent and gender‐independent experiments were conducted on the Chinese Academy of Sciences emotional speech database. The mean accuracy of SVM is 84.54%, and the mean accuracy of DBN is 94.6%. The experiments show that the DBN‐based approach has good potential for practical usage, and suitable feature fusions will further improve the performance of speech emotion recognition. Copyright © 2017 John Wiley & Sons, Ltd.

The speech signal consists of linguistic information and also paralinguistic one such as emotion. The modern automatic speech recognition systems have achieved high performance in neutral style speech recognition, but they cannot maintain their high recognition rate for spontaneous speech. So, emotion recognition is an important step toward emotional speech recognition. The accuracy of an emotion recognition system is dependent on different factors such as the type and number of emotional states and selected features, and also the type of classifier. In this paper, a modular neural-support vector machine (SVM) classifier is proposed, and its performance in emotion recognition is compared to Gaussian mixture model, multi-layer perceptron neural network, and C5.0-based classifiers. The most efficient features are also selected by using the analysis of variations method. It is noted that the proposed modular scheme is achieved through a comparative study of different features and characteristics of an individual emotional state with the aim of improving the recognition performance. Empirical results show that even by discarding 22% of features, the average emotion recognition accuracy can be improved by 2.2%. Also, the proposed modular neural-SVM classifier improves the recognition accuracy at least by 8% as compared to the simulated monolithic classifiers.

Emotions are a crucial aspect of human life and the researchers have tried to build an automatic emotion recognition system that helps to provide important real-world applications. The psychologists have shown that emotions differ across culture, considering this fact, we provide and describe the first audio-visual Arabic emotional dataset which called (AVANEmo). In this work we aim to fill the gap between studies of emotion recognition for Arabic content and other languages by provided an Arabic dataset which is a major and fundamental part of build emotion recognition application. Our dataset contains 3000 clips for video and audio data, and it covers six basic emotional labels (Happy, Sad, Angry, Surprise, Disgust, Neutral). Also, we provide some baseline experiments to measure the primitive performance for automated audio and visual emotion recognition application using the AVANEmo dataset. The best accuracy that we achieved was 54.5% and 57.9% using the audio and visual data respectively. The data will be available for distribution to researchers.

Cognitive emotion recognition using facial expression is the sub-field of social signal processing. It is broadly utilized in various fields, specifically human interaction and computer vision. Different investigators use learning approaches to construct an efficient system for automatic emotion recognition. However, predicting certain emotions like surprise, fear, sadness, happiness, disgust, and anger is still challenging in computer vision. However, deep learning has attained the interest of various researchers to resolve real-time problems like emotion recognition. This work concentrates on modelling and efficient Deep Belief Feature Learning (DBFL) for emotion recognition. The model uses a deep network layer to predict the feature information and further classification performance. The model gives 98% average prediction accuracy compared to various other approaches. The evaluation of the model is conducted over the public dataset, and the classification outcomes attained by the model demonstrates the significance of the proposed DBFL.

A vector quantized masked autoencoder for audiovisual speech emotion recognition

Emotion is a natural feeling which is distinguished from reasoning or knowledge, it is a strong feeling derived from one’s circumstance or surroundings. With the increase in man to machine interaction, speech analysis has become an integral part in reducing the gap between physical and digital world. An important sub field within this domain is the recognition of emotion in speech signals, which was traditionally studied in linguistics and psychology. Speech emotion recognition is a field having diverse applications. When implemented the Speech Emotion Recognition (SER) will be able to understand different human emotion such as anger, fear, happiness, sadness etc. Speech is a medium of expression of one’s perspective or feelings to other. Emotion recognition from audio signal requires feature extraction and classifier training. The feature vector consists of elements of the audio signal which characterize speaker specific features such as tone, pitch, energy, which is crucial to train the classifier model to recognize a particular emotion accurately. Thus, with the help of SER we can make conversations between human and computer more realistic and natural. Automatic Speech Emotion Recognition (SER) is a current research topic in the field of Human Computer Interaction (HCI) with wide range of applications. The speech features such as, Mel Frequency cepstrum coefficients (MFCC) and Mel Energy Spectrum Dynamic Coefficients (MEDC) are extracted from speech utterance. The Support Vector Machine (SVM) is used as classifier to classify different emotional states such as anger, happiness, sadness, neutral, fear, from Berlin emotional database.

Speech emotion recognition has become a challenging task in speech technology in order to ensure better and effective human-machine interaction. Improving the accuracy of speaker independent emotion recognition system by using a database containing the speech samples of limited number of utterances spoken by limited set of speakers is more challenging. This paper mainly discusses the use of integrated features and integrated models for improving the accuracy of the emotion recognition system. Integrated features are obtained by concatenating the probability with the normal system features. Integrated models are obtained by combining clustering and continuous density hidden Markov models. Integrated models with Mel frequency perceptual linear predictive cepstrum concatenated with probability provides better accuracy of 89% for recognising emotions using EMO-DB which contains speech samples of ten different utterances of ten different speakers in different emotions such as anger, boredom, disgust, fear, happy, neutral and sad by applying spectral analysis as a additional pre processing technique.

PurposeThe purpose of this paper is to provide a shorter time cost, high-accuracy fault diagnosis method for water pumps. Water pumps are widely used in industrial equipment and their fault diagnosis is gaining increasing attention. Considering the time-consuming empirical mode decomposition (EMD) method and the more efficient classification provided by the convolutional neural network (CNN) method, a novel classification method based on incomplete empirical mode decomposition (IEMD) and dual-input dual-channel convolutional neural network (DDCNN) composite data is proposed and applied to the fault diagnosis of water pumps.Design/methodology/approachThis paper proposes a data preprocessing method using IEMD combined with mel-frequency cepstrum coefficient (MFCC) and a neural network model of DDCNN. First, the sound signal is decomposed by IEMD to get numerous intrinsic mode functions (IMFs) and a residual (RES). Several IMFs and one RES are then extracted by MFCC features. Ultimately, the obtained features are split into two channels (IMFs one channel; RES one channel) and input into DDCNN.FindingsThe Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection (MIMII dataset) is used to verify the practicability of the method. Experimental results show that decomposition into an IMF is optimal when taking into account the real-time and accuracy of the diagnosis. Compared with EMD, 51.52% of data preprocessing time, 67.25% of network training time and 63.7% of test time are saved and also improve accuracy.Research limitations/implicationsThis method can achieve higher accuracy in fault diagnosis with a shorter time cost. Therefore, the fault diagnosis of equipment based on the sound signal in the factory has certain feasibility and research importance.Originality/valueThis method provides a feasible method for mechanical fault diagnosis based on sound signals in industrial applications.

Yuan cosmos is a virtual world linked and created by scientific and technological means, which is mapped and interacted with the real world, and a digital living space with a new social system. With the increasing popularity of data acquisition and production equipment, people are increasingly convenient to produce multimedia data such as images, graphics, audio, video, animation, and three-dimensional models. In addition to the rapid development of digital technology itself, the biological information technology related to digital technology also greatly promotes the emergence of the metauniverse. This paper aims to study the application of multimedia digital technology to the ecological scene design of metauniverse space, introduces the related concepts of metauniverse and multimedia digital technology, expounds the related methods of multimedia digital technology and neural network related algorithms, and then takes the three-dimensional simulation of the auditory system in the interactive multisensory simulation system of the constituent elements of metauniverse as an example. The mel-frequency cepstrum coefficient (MFCC) is used to simulate the auditory characteristics of the auditory periphery (cochlea) as the perceptual end of the model. A variety of bionic mechanisms are used in the model, such as designing the connection mode of neurons, learning state and release effect, and the regeneration mechanism of neurons. For the verification of the performance of the model, the speech sample database, including English words and phrases, is recorded and the speech content information recognized by the model by means of speech recognition is experimented. The experimental results show that, in terms of phrase accuracy, the DN-1 model improves 2.59% and the DN-2 model improves 2.77% compared with MFCC feature on the basis of mixed features. When only DBN features are used, the performance improvement rate of the developmental network model is small.

In today's world, accurately understanding and interpreting emotions in human-computer interaction is important. In this context, this study has adopted a detailed approach to the emotion recognition problem on both speech and text data using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. First, the problem of datasets with limited number of records and unbalanced distribution across classes was addressed. For this purpose, a dataset obtained from records created as improvised in the IEMOCAP dataset was used and data augmentation methods were applied for both speech and text data. Using datasets that were balanced by applying data augmentation, single-mode emotion recognition experiments were performed with models developed for Speech Emotion Recognition (SER) and Textual Emotion Recognition (TER). Subsequently, the features obtained from these two single modalities were combined with the intermediate fusion method to provide more comprehensive emotion recognition and accuracy, and the Bimodal Emotion Recognition (BiMER) system was developed. The ResNet50-CRNN+AT model, which we obtained the highest accuracy from the three different models developed for SER, creates the speech mode of BiMER, while the Bidirectional Encoder Representations from Transformers (BERT) model used for TER creates the text mode of BiMER. In this way, BiMER was supported with data augmentation methods and the robustness and generalization ability of the model were improved, reaching 88.33% accuracy. Finally, the developed BiMER system was implemented as a real-time web application using the Flask framework, and the capacity of this application to recognize emotions interactively through the user interface was tested.

In human–computer interaction, emotion recognition provides a deeper understanding of the user’s emotions, enabling empathetic and effective responses based on the user’s emotional state. While deep learning models have improved emotion recognition solutions, it is still an active area of research. One important limitation is that most emotion recognition systems use only text as input, ignoring features such as voice intonation. Another limitation is the limited number of datasets available for multimodal emotion recognition. In addition, most published datasets contain emotions that are simulated by professionals and produce limited results in real-world scenarios. In other languages, such as Spanish, hardly any datasets are available. Therefore, our contributions to emotion recognition are as follows. First, we compile and annotate a new corpus for multimodal emotion recognition in Spanish (Spanish MEACorpus 2023), which contains 13.16 h of speech divided into 5129 segments labeled by considering Ekman’s six basic emotions. The dataset is extracted from YouTube videos in natural environments. Second, we explore several deep learning models for emotion recognition using text- and audio-based features. Third, we evaluate different multimodal techniques to build a multimodal recognition system that improves the results of unimodal models, achieving a Macro F1-score of 87.745%, using late fusion with concatenation strategy approach.

In this work, spectral features extracted from sub-syllabic regions and pitch synchronous analysis are proposed for speech emotion recognition. Linear prediction cepstral coefficients, mel frequency cepstral coefficients and the features extracted from high amplitude regions of spectrum are used to represent emotion specific spectral information. These features are extracted from consonant, vowel and transition regions of each syllable to study the contribution of these regions toward recognition of emotions. Consonant, vowel and the transition regions are determined using vowel onset points. Spectral features extracted from each pitch cycle, are also used to recognize emotions present in speech. The emotions used in this study are: anger, fear, happy, neutral and sad. The emotion recognition performance using sub-syllabic speech segments are compared with the results of conventional block processing approach, where entire speech signal is processed frame by frame. The proposed emotion specific features are evaluated using simulated emotion speech corpus, IITKGP-SESC (Indian Institute of Technology, KharaGPur-Simulated Emotion Speech Corpus). The emotion recognition results obtained using IITKGP-SESC are compared with the results of Berlin emotion speech corpus. Emotion recognition systems are developed using Gaussian mixture models and auto-associative neural networks. The purpose of this study is to explore sub-syllabic regions to identify the emotions embedded in a speech signal, and if possible, to avoid processing of entire speech signal for emotion recognition without serious compromise in the performance.