Integrating Spectrotemporal Context into Features Based on Auditory Perception for Classification-based Speech Separation

Speech separation is a process of segregating the target speech from the noisy mixture. Human auditory system naturally has the capability to separate the speech from background noise; however the machine implementation of the same has been failed. Computational Auditory Scene Analysis (CASA) is a recent approach which models the human auditory system by computational means to separate the target speech from the noisy mixture. Most of the speech separation system based on CASA uses energy as one of the reliable feature to segregate the target speech from the noisy mixture. This research work proposes a new approach using adaptive energy selection threshold to segregate the target speech. The experimental result shows significant improvement in the signal to noise ratio as compared to the conventional energy threshold method.

To further improve the speech separation effect of deep neural networks (DNN), a DNN speech separation algorithm is proposed in this paper based on segmented masking target. The algorithm combines the advantages of IBM and IRM in different signal-to-noise ratio (SNR) regions to construct a segmented masking target that can adapt to changes in SNR as the training target of DNN. In addition, to improve the accuracy of IRM estimation, a two-step prior SNR is used for the effective calculation to further improve the speech separation performance of the DNN model. Finally, the simulation experiments show that the improved target in this paper has a better speech separation effect than IBM and IRM.

An arrangement is provided for an automatic speech recognition mechanism to adapt to an adverse acoustic environment. Some of the original training data, collected from an original acoustic environment, is played back in an adverse acoustic environment. The playback data is recorded in the adverse acoustic environment to generate recorded playback data. An existing speech model is then adapted with respect to the adverse acoustic environment based on the recorded playback data and/or the original training data.

In this contribution, a two-channel acoustic front-end for robust automatic speech recognition (ASR) in adverse acoustic environments is analyzed. The source signal extraction scheme combines a blocking matrix based on semi-blind source separation, which provides a continuously updated reference of all undesired components separated from the desired signal and its reflections, and a single-channel Wiener postfilter. The postfilter is directly derived from the obtained noise and interference reference signal and hence, generalizes well-known postfilter realizations. The proposed front-end and its integration into an ASR system are analyzed and evaluated with respect to keyword accuracy under reverberant conditions with unpredictable and nonstationary interferences, and for different target source distances. Evaluating a simplified front-end based on free-field assumptions, an ideal front-end, where knowledge about the true undesired components is assumed, and comparing the proposed scheme with the competitive approach of solely using multistyle training, demonstrates the importance of an adequate signal preprocessing for robust distant speech recognition.

The successful implementation of speech processing systems in the real world depends on its ability to handle adverse acoustic conditions with undesirable factors such as room reverberation and background noise. In this study, an extension to the established multiple sensors degenerate unmixing estimation technique (MENUET) algorithm for blind source separation is proposed based on the fuzzy c-means clustering to yield improvements in separation ability for underdetermined situations using a nonlinear microphone array. However, rather than test the blind source separation ability solely on reverberant conditions, this paper extends this to include a variety of simulated and real-world noisy environments. Results reported encouraging separation ability and improved perceptual quality of the separated sources for such adverse conditions. Not only does this establish this proposed methodology as a credible improvement to the system, but also implies further applicability in areas such as noise suppression in adverse acoustic environments.

In online speech separation, there is a trade-off between inherent latency and speech separation performance. When processing the current input audio, looking ahead to more future context usually brings better speech separation performance but increases the algorithm latency, and vice versa. In the requirements of extremely low latency, the future context is expensive for the algorithm latency and may not be available. In this work, we apply the contrastive predictive coding (CPC) method to the previously proposed online Skipping Memory (SkiM) speech separation model, which is a low-latency model for online speech separation. During the training stage, the SkiM model is required to predict the future memory states given the history memory. By using CPC training, the predictive SkiM model shows stronger causal sequence modeling capacity in the online speech separation task. In addition, we explore a local context codec (LCC) method to reduce the computational cost, and we make qualitative analyses on it. Our best online predictive SkiM equipped with CPC and LCC gets 15.5 dB SI-SNR improvement on WSJ02-mix benchmark with 3-ms actual latency tested on a single-core CPU, which should be the state-of-the-art results among causal models.

Traditional separation methods have limited ability to handle the speech separation problem in high reverberant and low signal-to-noise ratio (SNR) environments, and thus achieve unsatisfactory results. In this study, a convolutional neural network with temporal convolution and residual network (TC-ResNet) is proposed to realize speech separation in a complex acoustic environment. A simplified steered-response power phase transform, denoted as GSRP-PHAT, is employed to reduce the computational cost. The extracted features are reshaped to a special tensor as the system inputs and implements temporal convolution, which not only enlarges the receptive field of the convolution layer but also significantly reduces the network computational cost. Residual blocks are used to combine multiresolution features and accelerate the training procedure. A modified ideal ratio mask is applied as the training target. Simulation results demonstrate that the proposed microphone array speech separation algorithm based on TC-ResNet achieves a better performance in terms of distortion ratio, source-to-interference ratio, and short-time objective intelligibility in low SNR and high reverberant environments, particularly in untrained situations. This indicates that the proposed method has generalization to untrained conditions.

In this paper, a Blind Speech Separation (BSS) technique is introduced based on Independent Component Analysis (ICA) for underdetermined single microphone case. In general, ICA uses noisy speech from at least two microphones to separate speech and noise. But ICA fails to separate when only one stream of noisy speech is available. We use Log Spectral Magnitude Estimator based on Minimum Mean Square Error (LogMMSE) as a non-linear estimation technique to estimate the speech spectrum, which is used as the other input to ICA, with the noisy speech. The proposed method was tested for machinery, babble and traffic noise types mixed with speech at Signal to Noise Ratios (SNRs) of −5 dB, 0 dB and 5 dB. Objective and subjective results show high quality and intelligibility in the separated speech using the developed method.

This paper presents recognition results for a novel microphone solution for ASR in adverse acoustic environments. PARAT is an intelligent hearing protector and communication terminal implemented as a lightweight earplug. The inner microphone in PARAT captures the speech signal behind a seal in the auditory canal, thus impeding contamination from external noise sources. Experiments on a stereo speech database contaminated with noise show that the PARAT microphone compares favourably with a high-quality noise-cancelling close-talking microphone.

The primary objective of this thesis is to enhance the effectiveness and efficiency of speech representations, specifically improving noise robustness for downstream applications. Current speech representation learning frameworks, despite their advanced foundational knowledge and powerful speech understanding capabilities, fall short in critical areas essential for real-world applications, such as adaptability to noise, expressiveness, and computational efficiency. For instance, their performance varies greatly across different levels of noise corruption, showing high vulnerability to distortion from external influences. Moreover, learning for domain adaptation and performing inference are computationally intensive due to the large scale and complex design of the model structures. Hence, this thesis introduces innovative solutions to bridge these gaps, offering substantial improvements over existing methods. Adaptability to Noise: We address noise robustness by integrating Barlow Twins learning, an advanced regularization technique that strategically reduces channel redundancy and effectively disentangles speech features, as presented in chapter 3. This ensures that our models capture essential, noise-free features critical for accurate speech recognition, even in adverse acoustic environments, thereby ensuring more stable and robust performance to noise distortion. Expressiveness: To enhance the expressivity of pre-trained models, we employ a parameter-efficient fine-tuning approach that incorporates the proposed deep filter tuning with Feature-wise Linear Modulation (FiLM)-inspired integration, as detailed in Chapter 4. This method refines the handling of speech nuances, allowing the models to more effectively express important information through targeted feature extraction from FiLM. This adaptation better accommodates diverse vocal attributes and acoustic variations. Efficiency: Our proposed deep filter tuning strategy enables efficient adaptation of frozen pre-trained models with the aim of minimizing the number of trainable parameters. This approach optimizes computational resources and maximizes efficiency, particularly when faced with constraints in computational memory. In addition to the work, we also innovate in data compression through multi-band quantization, optimizing bit allocation by prioritizing perceptually significant features in chapter 5. This approach leverages psychoacoustic principles to enhance the efficiency of speech processing and is particularly beneficial for speech synthesis and voice conversion tasks, ensuring high fidelity in the outputs. Our methods have been tested across tasks such as automatic speech recognition and speech reconstruction, demonstrating significant enhancements in accuracy, robustness, and processing efficiency. These improvements underscore the potential of our approaches to revolutionize speech representation learning, making it more adaptive, scalable, and context-aware. By addressing the inherent limitations of current models, this research advances the field of speech representation learning, setting a new benchmark for future developments and applications in adaptive and efficient speech technology.

Playback attack detection for text-dependent speaker verification over telephone channels

We present a variety of new visual features in extension to the TED-LIUM corpus. We re-aligned the original TED talk audio transcriptions with official TED.com videos. By utilizing state-of-the-art models for face and facial landmarks detection, optical character recognition, object detection and classification, we extract four new visual features that can be used for Large-Vocabulary Continuous Speech Recognition (LVCSR) systems, including facial images, landmarks, text, and objects in the scenes. The facial images and landmarks can be used in combination with audio for audio-visual acoustic modeling where the visual modality provides robust features in adverse acoustic environments. The contextual information, i.e. extracted text and detected objects in the scene can be used as prior knowledge to create contextual language models. Experimental results showed the efficacy of using visual features on top of acoustic features for speech recognition in overlapping speech scenarios.

As the most widely-used spatial filtering approach for multi-channel speech separation, beamforming extracts the target speech signal arriving from a specific direction. An emerging alternative approach is multi-channel complex spectral mapping, which trains a deep neural network (DNN) to directly estimate the real and imaginary spectrograms of the target speech signal from those of the multi-channel noisy mixture. In this all-neural approach, the trained DNN itself becomes a nonlinear, time-varying spectrospatial filter. However, it remains unclear how this approach performs relative to commonly-used beamforming techniques on different array configurations and acoustic environments. This paper is devoted to examining this issue in a systematic way. Comprehensive evaluations show that multi-channel complex spectral mapping achieves separation performance comparable to or better than beamforming for different array geometries and speech separation tasks and reduces to monaural complex spectral mapping in single-channel conditions, demonstrating the general utility of this approach on multi-channel and single-channel speech separation. In addition, such an approach is computationally more efficient than widely-used mask-based beamforming. We conclude that this neural spectrospatial filter provides a strong alternative to traditional and mask-based beamforming.

In this paper we present results from our research on the impact of various soil types and processes on LWIR broadbandand hyperspectral signatures of landmines and the surrounding soil. These analyses utilized our digital hyperspectral models of the combined landmine-soil system for the computation of these signatures. These physics-based models incorporate models for external environmental processes and allow soil thermal parameters to be set as a function of the subsurface conditions (i.e., porosity, moisture content). Under this research effort, signature computations were conducted for various soil types as a function of the underlying soil conditions in order to examine the relative impact of these conditions on both the broadband and spectral LWIR signatures. Of particular interest were changes in spectral features and contrast changes due to soil water content. Results from the digital signature computations will be presented along with an analysis of the signature features. A comparison of the digital signature calculations to measured data will be included in this discussion. A brief description of the signature model will also be presented.

Considering that in different signal-to-noise ratio (SNR) regions, the amplification distortion caused by the overestimation of the gain function will have varying degrees of impact on the speech intelligibility, this paper proposes a deep neural network (DNN) speech separation with joint improved distortion constraints to further improve speech intelligibility. The algorithm first uses the segmented SNR to correct the overestimated gain function, and then combines the constraints of amplification distortion to further constrain the output amplitude of the estimated speech to obtain the final separated speech. Finally, the comparative experiments show that the proposed algorithm can significantly improve the speech quality and intelligibility of the separated speech.