Target speech detection and separation for humanoid robots in sparse dialogue with noisy home environments

Voice Activity Detection (VAD) is an important pre-processing step in a wide variety of speech processing systems. VAD should in a practical application be able to detect speech in both noisy and noise-free environments, while not introducing significant latency. In this work we propose using an adversarial multi-task learning method when training a supervised VAD. The method has been applied to the state-of-the-art VAD Waveform-based Voice Activity Detection. Additionally the performance of the VADis investigated under different algorithmic delays, which is an important factor in latency. Introducing adversarial multi-task learning to the model is observed to increase performance in terms of Area Under Curve (AUC), particularly in noisy environments, while the performance is not degraded at higher SNR levels. The adversarial multi-task learning is only applied in the training phase and thus introduces no additional cost in testing. Furthermore the correlation between performance and algorithmic delays is investigated, and it is observed that the VAD performance degradation is only moderate when lowering the algorithmic delay from 398 ms to 23 ms.

In recent years, DNN-based systems are extremely popular for performing automatic speech recognition (ASR) task. They have shown better performance than other methods. To perform ASR task efficiently, segmentation of the input data also has to be accurate. There are different kinds of methods of voice activity detection (VAD) including power based statistical methods. But to incorporate with the ASR system, it is better to perform the VAD task with the help of DNN-based method. In this paper, we investigate the superiority of DNN-based VAD over power based statistical methods. Not only that but also the effect of data augmentation on the performance of VAD in various conditions is also investigated. Performance of the VAD is evaluated using CENSREC-1-C data especially developed for evaluation of VAD task. The VAD trained with multi-condition data shows better performance than the baseline and other popular power based statistical VAD tools. We have also evaluated the performance for eval1 test dataset of CSJ corpus as well as its telephone variants.

Automatic speech recognition (ASR) system is used to recognize the text transcript from the given speech signal. Such speech signal can contain either isolated words or large vocabulary continuous speech (LVCS). Isolated words can be recognized with high accuracy in clean environment, but recognizing continuous words involves various parameters like speech corpus, speaker, environment noise, etc., that affects the accuracy of automatic speech recognition system directly. In the proposed work, hybrid feature extraction technique combines the perceptual linear predictive (PLP) and mel frequency cepstral coefficient (MFCC) to improve the accuracy of ASR in noisy environment. Voice activity and detection (VAD)-based frame dropping is used for improving the phonemes modeling by removing the pauses and distorted elements from the given speech signal. The proposed hybrid model with VAD is implemented by using self-generated speech corpus and shows relatively 12% increase in recognition rate compared with the state-of-the-art methodology.KeywordsAutomatic speech recognition (ASR)Perceptual linear predictive (PLP)Mel frequency cepstral coefficient (MFCC)Voice activity and detection (VAC)

An audio-visual speech recognition system (AVSR) integrates audio and visual information to perform speech recognition task. The AVSR has various applications in practice especially in natural language processing systems such as speech-to-text conversion, automatic translation and sentiment analysis. Decades ago, researchers tend to use Hidden Markov Model (HMM) to construct speech recognition system due to its good achievements in success recognition rate. However, HMM's training dataset is enormous in order to have sufficient linguistic coverage. Besides, its recognition rate under noisy environments is not satisfying. To overcome this deficiency, a Recurrent Neural Network (RNN) based AVSR is proposed. The proposed AVSR model consists of three components: 1) audio features extraction mechanism, 2) visual features extraction mechanism and 3) audio and visual features integration mechanism. The features integration mechanism combines the output features from both audio and visual extraction mechanisms to generate final classification results. In this research, the audio features mechanism is modelled by Mel-frequency Cepstrum Coefficient (MFCC) and further processed by RNN system, whereas the visual features mechanism is modelled by Haar-Cascade Detection with OpenCV and again, it is further processed by RNN system. Then, both of these extracted features were integrated by multimodal RNN-based features-integration mechanism. The performance in terms of the speech recognition rate and the robustness of the proposed AVSR system were evaluated using speech under clean environment and Signal Noise Ratio (SNR) levels ranging from -20 dB to 20 dB with 5 dB interval. On average, final speech recognition rate is 89% across different levels of SNR.

Voice activity detection is a fundamental problem in speech processing, which has been discussed for decades. However, it is a big challenge to determine the speech boundary in noisy environments because the corrupted speech is uncertain. In handing problems with noisy data, this study adopts a fuzzy neural network (FNN) to process the uncertainty. Furthermore, human speech perception is bimodal. We lip-read in noisy environments to improve intelligibility. This idea inspires us to adopt the visual information into the voice activity detection system. Based on the skin color segmentation, faces and mouths can be found in images. By analyzing the geometric shapes, the lip contour feature of speaker can be extracted. Then, the proposed fuzzy neural network considers not only audio but also visual information. Compared with the other voice activity detection, the proposed method for voice activity detection is more robust in the condition of low signal-to-noise ratio (SNR).

Speech signal is the primary mode of communication amongst humans. It is also used for the communication between human and machine, i.e., human-machine interaction (IoTs), communication with a range of electronic devices, e.g., Internet of Things (IoTs), and even with virtual assistants (VAs) or online search engines, e.g., Google etc. The use of speech signals for the communication over mobile phones, and smart phones based virtual assistants such as Cortana, Siri, Alexa etc. is well known. Usually, this communication involves some communication medium, which is susceptible to noise. The speech or audio signal used for the communication may be adversely affected by the noise that is added unwantedly over the communication channel or medium. It not only degrades the quality of communication and intelligibility of the message or content, but also leads to poor experience. The quality of speech signal may be degraded significantly, due to changes in the characteristics of speech signal, during the process from transmission to reception and the signal conversions involved at both ends. Hence there is a need for algorithms, signal processing methods, acoustic features and classifiers etc. for noise detection and noise reduction during the speech signal communication process. In this paper, a comprehensive review of various speech signal databases, speech signal processing methods used, acoustic features extracted and the classifiers used in different studies is presented. A comprehensive and comparative view of results obtained in different studies, and the applications is also presented. This study should be helpful to all the researchers working in the domains of speech signal processing in noisy environments or degraded speech etc.

An effect of noise on mental health indicator using voice

Robot capability of listening to several things at once by its own ears, that is, robot audition, is important in improving human-robot interaction. The critical issue in robot audition is real-time processing in noisy environments with high flexibility to support various kinds of robots and hardware configurations. This paper presents open-source robot audition software, called “HARK”, which includes sound source localization, separation, and automatic speech recognition (ASR). Since separated sounds suffer from spectral distortion due to separation, HARK generates a temporal-frequency map of reliability, called “missing feature mask”, for features of separated sounds. Then separated sounds are recognized by the Missing-Feature Theory (MFT) based ASR with missing feature masks. HARK is implemented on the middleware called “FlowDesigner” to share intermediate audio data, which provides real-time processing. HARK’s performance in recognition of noisy/simultaneous speech is shown by using three humanoid robots, Honda ASIMO, SIG2 and Robovie with different microphone layouts.

A robust and effective voice activity detection (VAD) algorithm is proposed for improving speech recognition performance in noisy environments. The approach is based on filtering the input channel to avoid high energy noisy components and then the determination of the speech/non-speech bispectra by means of third order auto-cumulants. This algorithm differs from many others in the way the decision rule is formulated (detection tests) and the domain used in this approach. Clear improvements in speech/non-speech discrimination accuracy demonstrate the effectiveness of the proposed VAD. It is shown that application of statistical detection test leads to a better separation of the speech and noise distributions, thus allowing a more effective discrimination and a tradeoff between complexity and performance. The algorithm also incorporates a previous noise reduction block improving the accuracy in detecting speech and non-speech. The experimental analysis carried out on the AURORA databases and tasks provides an extensive performance evaluation together with an exhaustive comparison to the standard VADs such as ITU G.729, GSM AMR and ETSI AFE for distributed speech recognition (DSR), and other recently reported VADs.

This paper proposes robust feature extraction for accurate voice activity detection (VAD). VAD is one of the principal modules for speech signal processing such as speech codec, speech enhancement, and speech recognition. Noisy environments contain nonstationary noises causing the accuracy of the VAD to drastically decline because the fluctuation of features in the noise intervals results in increased false alarm rates. In this paper, in order to improve the VAD performance, harmonic-weighted energy is proposed. This feature extraction method focuses on voiced speech intervals and weighted harmonic-to-noise ratios to determine the amount of the harmonicity to frame energy. For performance evaluation, the receiver operating characteristic curves and equal error rate are measured.

Voice activity detection (VAD) is an outstanding problem for speech transmission, enhancement and recognition. The variety and the varying nature of speech and background noise make it especially challenging. In the past years, many features emphasizing the differences between speech and noise have been proposed for their robustness. However an important problem in many areas of speech processing is the determination of presence of speech periods in a given signal. This task can be identified as a statistical hypothesis problem and its purpose is the determination to which category or class a given signal belongs. Also the classification task is often not as trivial as it appears since the increasing level of background noise leading to numerous detection errors. The selection of an adequate feature vector for signal detection and a robust decision rule is a challenging problem that affects the performance of VADs. Most algorithms are effective in numerous applications but often cause detection errors mainly due to the loss of discriminating power of the decision rule at lower SNRs. In this paper, it has been tried to extract the characteristics of noise by the VAD algorithm which can be used to smooth out the signal in silence part from the noisy environment. For further noise reduction signal then filtered in the wavelet domain using thresholding.

A robust voice activity detector (VAD) is expected to increase the accuracy of ASR in noisy environments. This study focuses on how to extract robust information for designing a robust VAD. To do so, we construct a noise eigenspace by the principal component analysis of the noise covariance matrix. Projecting noise speech onto the eigenspace, it is found that available information with higher SNR is generally located in the channels with smaller eigenvalues. According to this finding, the available components of the speech are obtained by sorting the noise eigenspace. Based on the extracted high-SNR components, we proposed a robust voice activity detector. The threshold for deciding the available channels is determined using a histogram method. A probability-weighted speech presence is used to increase the reliability of the VAD. The proposed VAD is evaluated using TIMIT database mixed with a number of noises. Experiments showed that our algorithm performs better than traditional VAD algorithms.KeywordsVoice activity detectionPrincipal component analysisAuto-segmentationLocal noise estimation

Improved likelihood ratio test based voice activity detector applied to speech recognition

Voice Activity Detection (VAD) provides the information whether an audio signal contains speech or not. Besides speech coding and transmission, there are many other applications in speech and audio processing that benefit from this information, and their performance is crucially dependent on the accuracy and robustness of the applied VAD. Various approaches to detect speech have been developed in the past, but when considering the challenging scenarios in which speech needs to be detected, e.g. hands-free communication in noisy environments or dialog in background music, there is still room for improvement. In this chapter, we describe the problem and the environments of VAD, discuss the procedure, examples for methods and their evaluation. Especially the more challenging application scenarios illustrate how superior human hearing can be compared to implementations of audio signal processing.

Voice activity detection (VAD) in noisy environments is a very important preprocessing scheme in speech communication technology, a field which includes speech recognition, speech coding, speech enhancement and captioning video contents. We have developed a VAD method for noisy environments based on the modulation spectrum. In Experiment 1, we investigate the optimal ranges of speech and modulation frequencies for the proposed algorithm by using the simulated data in the CENSREC-1-C corpus. Results show that when we combine an upper limit frequency between 1,000 and 2,000 Hz with a lower limit frequency of less than 300 Hz as speech frequency bands, error rates are lower than with other bands. Furthermore, when we use the frequency components of the modulation spectrum between 3–9, 3–11, 3–14, 3–18, 4–9, 4–11, 4–14, 4–18, 5–7, 5–9, 5–11, or 5–14 Hz, the proposed method performs VAD well. In Experiment 2, we use one of the best parameter settings from Experiment 1 and evaluate the real environment data in the CENSREC-1-C corpus by comparing our method with other conventional methods. Improvements were observed from the VAD results for each SNR condition and noise type.