Annotations and consistency detection for Chinese dual-mode emotional speech database

Speech emotion recognition (SER) is one of the latest challenge in human-computer interaction. In conventional SER classification methods, a single emotion label is outputted per one utterance as the estimation result. This is because conventional speech emotional databases which are used to train SER models have a single emotion label for one utterance. However, it is often the case that multiple emotions are expressed simultaneously with different intensities in human speech. In order to realize more natural SER than ever, existence of multiple emotions in one utterance should be taken into account. Therefore, we created an emotional speech database which contains multiple emotions and their intensities labels. The creation experiment was conducted by extracting speech utterance parts where emotions appear from existing video works. In addition, we evaluated the created database by performing statistical analysis on the database. As a result, 2,025 samples were obtained, of which 1,525 samples contained multiple emotions.

Egyptian Arabic speech emotion recognition using prosodic, spectral and wavelet features

Spoken emotion recognition is an interesting and challenging subject. In this paper, a new feature extraction method based on local Fisher discriminant analysis (LFDA) is proposed for spoken emotion recognition. LFDA is used to extract the low-dimensional discriminant embedded feature data from high-dimensional emotional speech features on spoken emotion recognition tasks. The performance of LFDA is compared with principal component analysis (PCA) and linear discriminant analysis (LDA). Experimental results on the emotional Chinese speech database demonstrate the promising performance of the proposed method.

Affective speech-to-speech translation (S2ST) is to preserve the affective state conveyed in the speaker's message. The ultimate goal of this study is to construct an affective S2ST system that has the ability to transform the emotional states of a spoken utterance from one language to another language. A universal automatic speech-emotion-recognition system is required to detect emotional state regardless of language. Therefore, this study investigates commonalities and differences of emotion perception across multi-languages. Thirty subjects from three countries, Japan, China and Vietnam, evaluate three emotional speech databases, Japanese, Chinese and German, in valence-activation space. The results reveal that directions from neutral to other emotions are similar among subjects groups. However, the estimated degree of emotional state depend on the expressed emotional styles. Moreover, neutral positions were significantly different among subjects groups. Thus, directions and distances from neutral to other emotions could be adopted as features to recognize emotional states for multi-languages.

Voice quality is considered to play an important role for the transmission of emotions in human speech communications. In this paper, we explored the acoustical characteristics of voice quality in the emotional speech signals based on numerical parameters, such as Jitter, RAP, Shimmer, APQ, NHR and SPI. In addition, the role of pitch, pitch range and normalized speech duration of the emotional speech was focused. Korean emotional speech database was collected from a professional actor. Nine sentences having different contents were respectively uttered with six different kinds of emotions: neutral, happiness, anger, sadness, fear and boredom. Jitter, RAP, Shimmer, APQ, NHR and SPI were computed respectively after extracting the voiced segment with the vowel /a/ from each emotional sentence. Pitch, pitch range and normalized speech duration of each emotional speech signal were also measured or computed. The statistical analysis based on the changes of these nine sets of different parameters was performed to characterize voice quality of the human emotional speeches.

This work investigates whether vocal emotion expressions of full-blown discrete emotions can be recognized cross-lingually. This study will enable us to get more information regarding nature and function of emotion. Furthermore, this work will help in developing a generalized vocal emotion recognition system, which will increase the efficiency required for human-machine interaction systems. An emotional speech database was created with 140 simulated utterances (20 per emotion) per speaker, consisting of short sentences of six full-blown discrete basic emotions and one 'no-emotion' (i.e. neutral) in five native languages (not dialects) of Assam. A new feature set is proposed based on Eigenvalues of Autocorrelation Matrix (EVAM) of each frame of utterance. The Gaussian Mixture Model is used as classifier. The performance of EVAM feature set is compared at two sampling frequencies (44.1 kHz and 8.1 kHz) and with additive white noise with signal-to-noise ratios of 0 db, 5 db, 10 db and 20 db.

Fuzzy Inference System (FIS) is used for pattern recognition and classification purposes in many fields such as emotion recognition. However, the performance of FIS is highly dependent on the radius of clusters which has a very important role for its recognition accuracy. Although many researcher initialize this parameter randomly which does not grantee the best performance of their systems. The purpose of this paper is to optimize FIS parameters in order to construct a high efficient system for speech emotion recognition. Therefore, an optimization algorithm based on particle swarm optimization technique is proposed for finding the best parameters of FIS classifier. In order to evaluate the proposed system it was tested using two emotional speech databases; Fujitsu and Berlin database. The simulation results show that the optimized system has high recognition accuracy for both languages with 97 % recognition accuracy for Japanese and 80 % for German database.

Semantic video annotation using ontologies has received a large attention from the scientific community in the recent years. Ontologies are being regarded as an appropriate tool to bridge the semantic gap. In this paper we present an overview of the state-of-the-art of approaches and algorithms that exploit ontologies to perform semantic video annotation and present an approach to automatically learn rules describing high-level concepts. This approach exploits the domain knowledge embedded into an ontology to learn a set of rules for semantic video annotation. The proposed technique is an adaptation of the First Order Inductive Learner (FOIL) technique to the Semantic Web Rule Language (SWRL) standard: Experiments have been performed in two different video domains: i) the TRECVID 2005 broadcast news collection, to detect events related to airplanes, such as taxiing, flying, landing and taking off; ii) surveillance videos, to detect if a person enters or exits a specific area. The promising experimental performance demonstrates the effectiveness and flexibility of the proposed framework.

Multimodal feature extraction and fusion for audio-visual speech recognition

This study deals with Speech Function and Mood used in the 2014 Indonesian Presidential Candidates Debate. This study was conducted by using descriptive qualitative method. The purpose of this study is to find out the types of Speech Function and Mood used in the debate based on Halliday’s theory. The data of this study were the utterances of both President Candidates, namely Jokowi Dodo and PrabowoSubianto. The findings show that both President candidates used all types of Speech Function in the debate, which Statement are 464 utterances (82.71%), Question are 51 utterances (9.09%), Command are 34 utterances (6.06%) and Offer are 12 utterances (2.14%). It is also found that they use congruent and metaphorical realization of Mood in debating. The use of Congruent Mood of declarative are 469 utterances (83.6%), Interrogative are 49 utterances (8.73%), and Imperative are 33 utterances (5.89%). Meanwhile the metaphorical realization of Declarative are 7 utterances (1.24%), Interrogative are 2 utterances (0.36%) and Imperative only 1 utterance (0.18%). Statement as the most dominant type of speech function in this debate shows thatJokowi and Prabowoare mostly giving information in order to deliver their future plans and ideas in the form of proposition, promise and persuasion, to strengthen the arguments and to convey messages logically in a limited time, so that the hearer/ people of Indonesia interested in choosing him as the next President. And the use of congruent or metaphorical realization in this debate implies that both Jokowi and Prabowo are able to give and to response arguments logically in different ways.

Purpose: We sought to establish normative growth curves for intelligibility development for the speech of typically developing children as revealed by objectively based orthographic transcription of elicited single-word and multiword utterances by naive listeners. We also examined sex differences, and we compared differences between single-word and multiword intelligibility growth.Method: One hundred sixty-four typically developing children (92 girls, 72 boys) contributed speech samples for this study. Children were between the ages of 30 and 47 months, and analyses examined 1-month age increments between these ages. Two different naive listeners heard each child and made orthographic transcriptions of child-produced words and sentences (n = 328 listeners). Average intelligibility scores for single-word productions and multiword productions were modeled using linear regression, which estimated normal-model quantile age trajectories for single- and multiword utterances.Results: We present growth curves showing steady linear change over time in 1-month increments from 30 to 47 months for 5th, 10th, 25th, 50th, 75th, 90th, and 95th percentiles. Results showed that boys did not differ from girls and that, prior to 35 months of age, single words were more intelligible than multiword productions. Starting at 41 months of age, the reverse was true. Multiword intelligibility grew at a faster rate than single-word intelligibility.Conclusions: Children make steady progress in intelligibility development through 47 months, and only a small number of children approach 100% intelligibility by this age. Intelligibility continues to develop past the fourth year of life. There is considerable variability among children with regard to intelligibility development.Supplemental Material S1. Technical details on integrated weighted intelligibility response for multiword utterances. Hustad, K. C., Mahr, T., Matzke, P. E. M., & Rathaus, P. J. (2020). Development of speech intelligibility between 30 and 47 months in typically developing children: A cross-sectional study of growth. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2020_JSLHR-20-00008

Computer-Based Social Anxiety Regulation in Virtual Reality Exposure Therapy

Computer visualization can effectively deliver instructions to a user whose task requires understanding of a real world scene. Consider the example of surgical telementoring, where a general surgeon performs an emergency surgery under the guidance of a remote mentor. The mentor guidance includes annotations of the operating field, which conventionally are displayed to the surgeon on a nearby monitor. However, this conventional visualization of mentor guidance requires the surgeon to look back and forth between the monitor and the operating field, which can lead to cognitive load, delays, or even medical errors. Another example is 3D acquisition of a real-world scene, where an operator must acquire multiple images of the scene from specific viewpoints to ensure appropriate scene coverage and thus achieve quality 3D reconstruction. The conventional approach is for the operator to plan the acquisition locations using conventional visualization tools, and then to try to execute the plan from memory, or with the help of a static map. Such approaches lead to incomplete coverage during acquisition, resulting in an inaccurate reconstruction of the 3D scene which can only be addressed at the high and sometimes prohibitive cost of repeating acquisition.Augmented reality (AR) promises to overcome the limitations of conventional out-of-context visualization of real world scenes by delivering visual guidance directly into the user's field of view, guidance that remains in-context throughout the completion of the task. In this thesis, we propose and validate several AR visual interfaces that provide effective visual guidance for task completion in the context of surgical telementoring and 3D scene acquisition.A first AR interface provides a mentee surgeon with visual guidance from a remote mentor using a simulated transparent display. A computer tablet suspended above the patient captures the operating field with its on-board video camera, the live video is sent to the mentor who annotates it, and the annotations are sent back to the mentee where they are displayed on the tablet, integrating the mentor-created annotations directly into the mentee's view of the operating field. We show through user studies that surgical task performance improves when using the AR surgical telementoring interface compared to when using the conventional visualization of the annotated operating field on a nearby monitor. A second AR surgical telementoring interface provides the mentee surgeon with visual guidance through an AR head-mounted display (AR HMD). We validate this approach in user studies with medical professionals in the context of practice cricothyrotomy and lower-limb fasciotomy procedures, and show improved performance over conventional surgical guidance. A comparison between our simulated transparent display and our AR HMD surgical telementoring interfaces reveals that the HMD has the advantages of reduced workspace encumbrance and of correct depth perception of annotations, whereas the transparent display has the advantage of reduced surgeon head and neck encumbrance and of annotation visualization quality. A third AR interface provides operator guidance for effective image-based modeling and rendering of real-world scenes. During the modeling phase, the AR interface builds and dynamically updates a map of the scene that is displayed to the user through an AR HMD, which leads to the efficient acquisition of a five-degree-of-freedom image-based model of large, complex indoor environments. During rendering, the interface guides the user towards the highest-density parts of the image-based model which result in the highest output image quality. We show through a study that first-time users of our interface can acquire a quality image-based model of a 13m $\times$ 10m indoor environment in 7 minutes.A fourth AR interface provides operator guidance for effective capture of a 3D scene in the context of photogrammetric reconstruction. The interface relies on an AR HMD with a tracked hand-held camera rig to construct a sufficient set of six-degrees-of-freedom camera acquisition poses and then to steer the user to align the camera with the prescribed poses quickly and accurately. We show through a study that first-time users of our interface are significantly more likely to achieve complete 3D reconstructions compared to conventional freehand acquisition. We then investigated the design space of AR HMD interfaces for mid-air pose alignment with an added ergonomics concern, which resulted in five candidate interfaces that sample this design space. A user study identified the aspects of the AR interface design that influence the ergonomics during extended use, informing AR HMD interface design for the important task of mid-air pose alignment.

To evaluate the quality of life after cochlear implantation in adult cochlear implant recipients and to assess the applicability of Nijmegen Cochlear implant Questionnaire (NCIQ). Thirty-two postlingually deaf patients, 19 males and 13 females, underwent cochlear implantation with the age at operation of 35.27 (18-66) during the period from 1996 to the end of 2006. NCIQ was used to investigate the application of the equipment, hearing ability, and emotional exchange ability. All of the patients used their cochlear implants for 16 h every day on average with an experience in using such implant of more than 6 months. Improvement was seen in sound recognition, speech recognition, confidence, daily life, social exchange, and emotional behavior ability at different degrees. The improvement of confidence was very obvious. The speech communication ability of those with a hearing deprivation time longer than 5 years was lower than that of those with a shorter hearing deprivation time. Age and educational level did not influence the efficiency. Cochlear implantation is a cost effective intervention in the postlingually deaf adults. NCIQ test can be used to evaluate the quality of life among postlingually deaf adult cochlear implant recipients in China.

Speech emotion recognition (SER) has become one of the significant approaches to improve human-robot interaction. In this paper, two methods are proposed which take into consideration the size of the databases along with other aspects of the models. The first model applied K nearest neighbors (KNN) algorithms with 1-30 Gammatone frequency cepstral coefficients (GTCCs) which is mainly proposed for relatively small databases. It achieved 95.3% overall recognition accuracy on Berlin Emotional Speech database (EMODB). The second model is mainly focused on relatively large databases, which adopted 1-30 GTCCs, delta 1-30 GTCCs, delta-delta 1- 30 GTCCs, spectral features and prosodic features as the feature set and used long short-term memory (LSTM) as the classifier. An overall accuracy of 87.5% is achieved with this model when applied to Chinese emotional speech database (CASIA).