Multimodal speech recognition: increasing accuracy using high speed video data

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Speech Recognition is frequently cited as a potential remedy to distraction resulting from drivers' operation of in-vehicle devices. This position typically assumes that the introduction of speech recognition will result in reduced cognitive workload and improved driving performance. Past research neither fully supports nor fully discounts this assumption. However, it is difficult to compare many of these studies, due to differences in device operation tasks, the pacing of those tasks, speech recognition system performance, and system interface designs. In an effort to directly address the effect of voice recognition on driver distraction, the present authors developed a capability to manipulate the performance characteristics of a speech recognition system through a Wizard of Oz speech recognition system and installed this system in a simulated driving environment. The sensitivity of the simulated driving environment and speech recognition accuracy manipulation were evaluated in an initial study comparing driver cognitive workload and driving performance during self-paced simulated operation of a personal digital assistant (PDA) during no PDA use, manual control of the PDA, and speech control of the PDA. In the Speech PDA condition, speech recognition accuracy was varied between drivers. Analysis of drivers' emergency braking response times and rated cognitive workload revealed significantly lower cognitive demand and better performance in the No PDA condition when compared to the Manual PDA condition. The Speech PDA condition resulted in response times and rated cognitive workload levels that were between the No PDA and Manual PDA conditions, but not significantly different from either of these conditions. Further analysis of emergency braking performance revealed a non-significant trend towards better performance in conjunction with higher speech recognition accuracy levels. The potential for reducing driver distraction through the careful development and evaluation of speech recognition systems is discussed.

The accuracy of automatic speech recognition (ASR) systems is generally evaluated using corpora of grammatically sound read speech or natural spontaneous speech. This prohibits an accurate estimation of the performance of the acoustic modeling part of ASR because the language modeling performance is inherently integrated in the overall performance metric. In this work, ASR and human speech recognition (HSR) accuracies are compared for null grammar sentences in different signal-to-noise ratios and vocabulary sizes-1000, 2000, 4000, and 8000. The results shed light on differences between ASR and HSR in relative significance of bottom-up word recognition and context awareness.

Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systems

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

In this work, we propose a novel feature extraction algorithm that improves the robustness of automatic speech recognition (ASR) systems in the presence of various types of noise. The proposed algorithm uses a new cepstral technique based on the differential power spectrum (DPS) instead of the power spectrum (PS), the algorithm replaces the logarithmic non linearity by the power function. In order to reduce cepstral coefficients mismatches between training and testing conditions, we used the mean and variance normalization, then we apply auto-regression movingaverage filtering (MVA) in the cepstral domain. The ASR experiments were conducted using two databases, the first is LASA digit database designed for recognition the isolated Arabic digits in the presence of different types of noise. The second is Aurora 2 noisy speech database designed to recognize connected English digits in various operating environments. The experimental results show a substantial improvement from the proposed algorithm over the baseline Mel Frequency Cepstral Coefficients (MFCC), the relative improvement is the 28.92% for LASA database and is the 44.43% for aurora 2 database. The performance of our proposed algorithm was tested and verified by extensive comparisons with the state-of-the-art noise-robust features in aurora 2.

Challenges of Automatic Speech Recognition for medical interviews - research for Polish language

Speaker Identification (SI) is a process of identifying a speaker automatically via a machine using the speaker’s voice. In SI, one speaker’s voice is compared with n- number of speakers’ templates within the reference database to find the best match among the potential speakers. Speakers are capable of changing their voice, though, such as their accent, which makes is more challenging to identify who is talking. In this paper, we extracted phonemes from a speaker’s voice recording and investigated the associated frequencies and amplitudes to be assist in identifying the person who is speaking. This paper demonstrates the importance of phonemes in both speech and voice recognition systems. The results demonstrate that we can use phonemes to help the machine identify a particular speaker, however, phonemes get better accuracy in speech recognition than speaker identification.

End-to-end speech recognition system is a major research field in speech recognition. The most typical model is the end-to-end speech recognition system based on CTC where RNN mining to and time sequence information are adopted, and a series of assumptions of HMM are discarded to obtain a good recognition rate. However, the CTC-based model is more dependent on the speech model and have a longer training cycle. Therefore, in the framework of traditional acoustic model, this paper proposes to train a feature extraction network of spectrogram based on attention mechanism by using prior knowledge. Firstly, it was spliced in the front end based on CTC model, and then the number of layers of cyclic neural network based on CTC model was reduced. Finally, it was combined to retrain. The experimental results show that the training time of the combined model is effectively reduced, and the accuracy of speech recognition is further improved.

We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. Using our new corpus, which includes the speech of elderly people living in three regions of Japan, we conducted speech recognition experiments using a variety of DNN-HNN acoustic models. As training data for our acoustic models, we examined whether a standard adult Japanese speech corpus (JNAS), an elderly speech corpus (S-JNAS) or a spontaneous speech corpus (CSJ) was most suitable, and whether or not adaptation to the dialect of each region improved recognition results. We adapted each of our three acoustic models to all of our speech data, and then re-adapt them using speech from each region. Without adaptation, the best recognition results were obtained when using the S-JNAS trained acoustic models (total corpus: 21.85% Word Error Rate). However, after adaptation of our acoustic models to our entire corpus, the CSJ trained models achieved the lowest WERs (entire corpus: 17.42%). Moreover, after readaptation to each regional dialect, the CSJ trained acoustic model with adaptation to regional speech data showed tendencies of improved recognition rates. We plan to collect more utterances from all over Japan, so that our corpus can be used as a key resource for elderly speech recognition in Japanese. We also hope to achieve further improvement in recognition performance for elderly speech.

Speech is the one of the most important communication channel among the humans. Speech recognition occupies prominent place in communication between the humans and machine. Several factors are affecting the accuracy of the speech recognition system. Much effort has been done to increase the accuracy of the speech recognition system. Still erroneous output is generated in current speech recognition system. Static pronunciation dictionary plays key role in the speech recognition accuracy. The required phoneme of the word changes to the phoneme of the some other word. Modification in the dictionary in the decoder of the speech recognition system reduces the number of the confusion pairs which automatically increase the accuracy. Hit rate is considerably increased and false alarms have been changed during the modification of the pronunciation dictionary. Also this proposed method observed the variations on different error measures such as F-measures, Error-Rate and WER by applying this proposed method.

In this paper we present a novel method designed for word-level visual speech recognition and intended for use in human-robot interaction. The ability of robots to understand natural human speech will significantly improve the quality of human-machine interaction. Despite outstanding breakthroughs achieved in this field in recent years this challenge remains unresolved. In current research we mainly focus on the visual part of the human speech, so-called automated lip-reading task, which becomes crucial for human-robot interaction in acoustically noisy environment. The developed method is based on the use of state-of-the-art artificial intelligence technologies and allowed to achieve an incredible 85.03% speech recognition accuracy using only video data. It is worth noting that the model training and testing of the method was carried out on a benchmarking LRW database recorded inthe-wild, and the presented results surpass many existing achieved by the researchers of the world speech recognition community.

Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articulatory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic features, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically analysed by using a statistical space distribution indicator called Maximum Articulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic features at the empirically found optimal fusion level achieves a notable performance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively. • Systematically compares the articulatory movement pattern mismatch between dysarthric and typical speech. This is achieved through the visualisation of the 3D point cloud of several real articulatory data samples and an analysis of the statistical space distribution across the dataset, employing the Maximum Articulator Motion Range (MAMR) as a metric. • Conduct comprehensive articulatory information analysis, including the computation of mutual information , to identify and select the most task-beneficial articulators, thereby enabling a more informed integration of articulatory features with acoustic features for optimal multimodal ADSR performance. • Investigate multimodal acoustic modelling for ADSR using real articulatory information in combination with acoustic features, encompassing both hand-crafted features and raw signal representations, e.g., raw waveform with non-parametric and parametric CNNs, magnitude spectrum, the raw source and filter components, and raw real and imaginary parts of the Fourier transform.

The performance of automatic speech recognition (ASR) may be degraded when accented speech is recognized because the speech has some linguistic differences from standard speech. Conventional accented speech recognition studies have utilized the accent embedding method, in which the accent embedding features are directly fed into the ASR network. Although the method improves the performance of accented speech recognition, it has some restrictions, such as increasing the computational costs. This study proposes an efficient method of training the ASR model for accented speech in a domain adversarial way based on the Domain Adversarial Neural Network (DANN). The DANN plays a role as a domain adaptation in which the training data and test data have different distributions. Thus, our approach is expected to construct a reliable ASR model for accented speech by reducing the distribution differences between accented speech and standard speech. DANN has three sub-networks: the feature extractor, the domain classifier, and the label predictor. To adjust the DANN for accented speech recognition, we constructed these three sub-networks independently, considering the characteristics of accented speech. In particular, we used an end-to-end framework based on Connectionist Temporal Classification (CTC) to develop the label predictor, a very important module that directly affects ASR results. To verify the efficiency of the proposed approach, we conducted several experiments of accented speech recognition for four English accents including Australian, Canadian, British (England), and Indian accents. The experimental results showed that the proposed DANN-based model outperformed the baseline model for all accents, indicating that the end-to-end domain adversarial training effectively reduced the distribution differences between accented speech and standard speech.

We investigate the use of Fourier plane nonlinear filtering for phase-encoded images. We investigate the performance of the nonlinear joint transform correlator and the nonlinearly transformed matched filter for phase-encoded images with different types of input noise. We use the peak-to-output-energy ratio, peak-to-sidelobe ratio, and discrimination ratio as the metrics for measuring the performances. We mathematically analyze the peak-to-output-energy ratio of the nonlinearly transformed matched filter for phase-encoded images with spatially nonoverlapping white noise. Computer simulations are provided to show the performance improvements of the nonlinear filtering techniques for the phase-encoded images. In comparison with linear filtering techniques, we find that the nonlinear filtering techniques substantially improve the performance metrics. From the computer-simulation results it can be seen that the nonlinear joint transform correlator performs better than the nonlinearly transformed matched filter in detecting phase-encoded targets in the presence of different types of noise, such as additive overlapping white noise, spatially nonoverlapping white background noise, spatially nonoverlapping colored background noise, and nontarget objects.