Representation Learning Based Speech Assistive System for Persons With Dysarthria.

Automatic speech recognition (ASR) for dysarthric speech is challenging. The acoustic characteristics of dysarthric speech are highly variable and there are often fewer distinguishing cues between phonetic tokens. Multimodal ASR utilises the data from other modalities to facilitate the task when a single acoustic modality proves insufficient. Articulatory information, which encapsulates knowledge about the speech production process, may constitute such a complementary modality. Although multimodal acoustic-articulatory ASR has received increasing attention recently, incorporating real articulatory data is under-explored for dysarthric speech recognition. This paper investigates the effectiveness of multimodal acoustic modelling using real dysarthric speech articulatory information in combination with acoustic features, especially raw signal representations which are more informative than classic features, leading to learning representations tailored to dysarthric ASR. In particular, various raw acoustic-articulatory multimodal dysarthric speech recognition systems are developed and compared with similar systems with hand-crafted features. Furthermore, the difference between dysarthric and typical speech in terms of articulatory information is systematically analysed by using a statistical space distribution indicator called Maximum Articulator Motion Range (MAMR). Additionally, we used mutual information analysis to investigate the robustness and phonetic information content of the articulatory features, offering insights that support feature selection and the ASR results. Experimental results on the widely used TORGO dysarthric speech dataset show that combining the articulatory and raw acoustic features at the empirically found optimal fusion level achieves a notable performance gain, leading to up to 7.6% and 12.8% relative word error rate (WER) reduction for dysarthric and typical speech, respectively. • Systematically compares the articulatory movement pattern mismatch between dysarthric and typical speech. This is achieved through the visualisation of the 3D point cloud of several real articulatory data samples and an analysis of the statistical space distribution across the dataset, employing the Maximum Articulator Motion Range (MAMR) as a metric. • Conduct comprehensive articulatory information analysis, including the computation of mutual information , to identify and select the most task-beneficial articulators, thereby enabling a more informed integration of articulatory features with acoustic features for optimal multimodal ADSR performance. • Investigate multimodal acoustic modelling for ADSR using real articulatory information in combination with acoustic features, encompassing both hand-crafted features and raw signal representations, e.g., raw waveform with non-parametric and parametric CNNs, magnitude spectrum, the raw source and filter components, and raw real and imaginary parts of the Fourier transform.

While dysarthric speech recognition can be a convenient interface for dysarthric speakers, it is hard to collect enough speech data to overcome the underestimation problem of acoustic models. In addition, there are lots of pronunciation variations in the collected database due to the paralysis of the articulator of dysarthric speakers. Thus, a discriminative training method is proposed for improving the performance of such resource-limited dysarthric speech recognition. The proposed method is applied to subspace Gaussian mixture modeling by incorporating pronunciation variations into a conventional minimum phone error discriminative training method.

This research focuses on the automatic recognition and assessment of dysarthric speech, a critical area for individuals with speech impairments resulting from neurological conditions. The study explores advanced speech processing techniques, including deep learning models, autoencoder-based speech enhancement, and speaker adaptation approaches, to improve dysarthric speech recognition and intelligibility assessment. By leveraging both dysarthric and healthy speech corpora, the research enhances automatic speech recognition (ASR) accuracy, enabling more effective communication support for individuals with dysarthria. The societal impact of this work is significant. Automatic intelligibility assessment can assist speech therapists in tracking the progression of speech disorders and tailoring therapy plans, leading to improved rehabilitation outcomes. Enhanced ASR systems can also empower individuals with dysarthria by facilitating better communication in everyday interactions, reducing social isolation, and improving access to education and employment. This research contributes to the broader goal of inclusive technology, ensuring speech-impaired individuals can interact effectively with digital and human communication interfaces.

This paper describes a deep neural network – hidden Markov model (DNN-HMM) human activity recognition system based on instrumented objects and studies compensation strategies to deal with object variability. The sensors, comprising an accelerometer, gyroscope, magnetometer and force-sensitive resistors (FSRs), are packaged in a coaster attached to the base of an object, here a mug. Results are presented for recognition of actions involved in manipulating a mug. Evaluations are performed using over 24 hours of data recordings containing sequences of actions, labelled without time-stamp information. We demonstrate the importance of data alignments. While the DNN-HMM system achieved error rate below 0.1% for matched train-test conditions, this increased up to 26.5% for highly mismatched conditions. The error rate averaged over all conditions was 1.4% when using multi-condition training and decreased to 0.8% by employing feature augmentation. The use of FSR feature compensation, specific to weight variability, resulted in 0.24% error rate.

Acoustic modelling for automatic dysarthric speech recognition (ADSR) is a challenging task. Data deficiency is a major problem and substantial differences between typical and dysarthric speech complicate the transfer learning. In this paper, we aim at building acoustic models using the raw magnitude spectra of the source and filter components for ADSR. The proposed multi-stream models consist of convolutional, recurrent and fully-connected layers allowing for pre-processing various information streams and fusing them at an optimal level of abstraction. We demonstrate that such a multi-stream processing leverages information encoded in the vocal tract and excitation components and leads to normalising nuisance factors such as speaker attributes and speaking style. This leads to a better handling of dysarthric speech that exhibits large inter- and intra-speaker variabilities and results in a notable performance gain. Furthermore, we analyse the learned convolutional filters and visualise the outputs of different layers after dimensionality reduction to demonstrate how the speaker-related attributes are normalised along the pipeline. We also compare the proposed multi-stream model with various systems based on MFCC, FBank, raw waveform and i-vector, and, study the training dynamics as well as usefulness of the feature normalisation and data augmentation via speed perturbation. On the widely used TORGO and UASpeech dysarthric speech corpora, the proposed approach leads to a competitive performance of up to 35.3% and 30.3% WERs for dysarthric speech, respectively.

We develop and open-source a morpheme-based decoder wrapper, Morpheme Lattice Dynamically Generating Decoder(MLDG-Decoder) for Uyghur Deep Neural Network-Hidden Markov Model(DNN-HMM) systems. The Morpheme Lattice Dynamically Generating Decoder employs an algorithm, named as “on-the-fly composition with Forbidden Epsilons Activated by Back-Off States(FEABOS),” to allow the back-off states and back-off transitions to play the role of a relay station during on-the-fly composition. The algorithm empowers the dynamically generated graph to constrain the morpheme sequences in the lattices as effectively as the static and fully composed graph does when a 4-Gram morpheme-based Language Model (LM) is used. Based on the deeper and wider neural network acoustic models and the Morpheme Lattice Dynamically Generating Decoder, we experiment with three morpheme-based decoding strategies for Deep Neural Network-Hidden Markov Model hybrid systems: using a Morpheme Lattice Dynamically Generating Decoder, the two-pass decoding based on rescoring, and the fully static and morpheme-based decoding. The results show that decoding based on a fully static decoding graph reduces the minimum Word Error Rate(WER) to 14.24%. Morpheme Lattice Dynamically Generating Decoder reduces Word Error Rate to 14.54% and maintains reasonable memory consumption.

Dysarthria is a neurological speech disorder that affects the speech intelligibility of the speaker. Speech assistive aids are developed to support their communication needs. Successful speech assistive aids are developed using automatic speech recognition systems trained using their own speech data. The effectiveness and usefulness of speech recognition systems depend on the amount of speech data used for training. However, collecting a large amount of dysarthric speech data is difficult. Data augmentation involves applying transformation techniques to increase the quantity of available speech data. Adding noise data is also one of the approaches to make such transformations and create a new volume of data. However, care should be taken while using noise data for the transformation of the dysarthric speech data since dysarthria on its own is disordered data, and adding even more distortion reduces its quality of it. However, by performing a proper analysis of the noisy data, noise can also be used as a source to create new samples of dysarthric speech data. This paper concentrates on identifying noise characteristics and finding the suitability of using noise as a source for data augmentation in dysarthric speech. With the noise-augmented dysarthric speech data, dysarthric speech recognition systems were trained to evaluate the quality of the augmented data. It was noted that for dysarthric speakers, especially with the severe category, the low-frequency noise selection approach has resulted in a lower WER than the without augmentation by 12.29%.

Dysarthric speech recognition is a challenging task due to acoustic variability and limited amount of available data. Diverse conditions of dysarthric speakers account for the acoustic variability, which make the variability difficult to be modeled precisely. This paper presents a variational auto-encoder based variability encoder (VAEVE) to explicitly encode such variability for dysarthric speech. The VAEVE makes use of both phoneme information and low-dimensional latent variable to reconstruct the input acoustic features, thereby the latent variable is forced to encode the phoneme-independent variability. Stochastic gradient variational Bayes algorithm is applied to model the distribution for generating variability encodings, which are further used as auxiliary features for DNN acoustic modeling. Experiment results conducted on the UASpeech corpus show that the VAEVE based variability encodings have complementary effect to the learning hidden unit contributions (LHUC) speaker adaptation. The systems using variability encodings consistently outperform the comparable baseline systems without using them, and" obtain absolute word error rate (WER) reduction by up to 2.2% on dysarthric speech with "Very lowintelligibility level, and up to 2% on the "Mixed" type of dysarthric speech with diverse or uncertain conditions.

Disabled individuals can realize many benefits from automatic speech recognition (ASR). To date, most ASR research has focused on normal speech. However, many individuals with physical disabilities also exhibit speech disorders. While limited research has been conducted focusing on dysarthric speech recognition, the preliminary results indicate that additional study is necessary. Recently, increasing attention has been given to multimodal speech recognition schemes that utilize multiple input sources-most commonly audio and video. This multimodal approach has been applied to normal speech with demonstrated effectiveness. Through studying the effect of audio and visual information in a human perception experiment, this study attempts to discover whether such an approach would be useful for dysarthric speech recognition. Results of a closed-vocabulary perception test are presented. In this test, 15 normal-hearing viewers were presented with video-tapes of three dysarthric speakers (with cerebral palsy) speaking a series of one-syllable nonsense words. These words differed only in the initial consonant. The words were presented in both audio-only and audio-visual modes. Perception rates in both modes were measured. The results are analyzed and compared to other studies of visual speech perception and dysarthric speech articulation.

The histogram equalization approach is an efficient feature normalization technique for noise robust automatic speech recognition. However, it suffers from performance degradation when some fundamental conditions are not satisfied in the test environment. To remedy these limitations of the original histogram equalization methods, class-based histogram equalization approach has been proposed. Although this approach showed substantial performance improvement under noise environments, it still suffers from performance degradation due to the overfitting problem when test data are insufficient. To address this issue, the proposed histogram equalization technique employs the Bayesian estimation method in the test cumulative distribution function estimation. It was reported in a previous study conducted on the Aurora-4 task that the proposed approach provided substantial performance gains in speech recognition systems based on the acoustic modeling of the Gaussian mixture model-hidden Markov model. In this work, the proposed approach was examined in speech recognition systems with deep neural network-hidden Markov model (DNN-HMM), the current mainstream speech recognition approach where it also showed meaningful performance improvement over the conventional maximum likelihood estimation-based method. The fusion of the proposed features with the mel-frequency cepstral coefficients provided additional performance gains in DNN-HMM systems, which otherwise suffer from performance degradation in the clean test condition.

Dysarthric speech recognition is essential for enhancing communication and accessibility for individuals with speech impairments, yet its development is hindered by a scarcity of robust, speaker-specific datasets. This study explores low-resource dysarthric speech recognition through cross-speaker transfer using synthetic data and parameter-efficient fine-tuning (PEFT). We integrate SpeechT5 text-to-speech (TTS) synthesis with x-vector speaker embeddings to generate speaker-specific dysarthric speech, enabling model adaptation while preserving pathological speech characteristics such as prosodic irregularities. Experiments on the TORGO dataset show that mixed cross-synthetic data with LoRA fine-tuning achieves a WER of 0.17, representing a 71.7% improvement over the standard model (0.60 WER) without fine-tuning the TTS model. However, cross-dataset generalisation remains challenging, yielding higher WERs on MINDS-14 (4.69) and AMI (0.96–3.83) datasets. Whilst synthetic data enhances in-domain recognition, further research is needed to improve cross-dataset generalisation and speaker adaptation, particularly for low-resource pathological speech settings.

Pronunciation variation is a major problem in disordered speech recognition. This paper focus on handling the pronunciation variations in dysarthric speech by forming speaker-specific lexicons. A novel approach is proposed for identifying mispronunciations made by each dysarthric speaker, using state-specific vector (SSV) of phone-cluster adaptive training (Phone-CAT) acoustic model. SSV is low-dimensional vector estimated for each tied-state where each element in a vector denotes the weight of a particular monophone. The SSV indicates the pronounced phone using its dominant weight. This property of SSV is exploited in adapting the pronunciation of a particular dysarthric speaker using speaker-specific lexicons. Experimental validation on Nemours database showed an average relative improvement of 9% across all the speakers compared to the system built with canonical lexicon. Index Terms: Dysarthric speech recognition, phone-CAT, lexical modeling, pronunciations, phone confusion matrix

The aim of our project is to develop a markerless system to detect falls and evaluate the frailty of elderly people at home. In previous work, we developed an algorithm detecting falls and daily life activities based on depth images provided by Microsoft's Kinect sensor. We also developed another algorithm based on the same features for gait analysis. However, an ambient system installed at home for frailty evaluation should be able to identify the individuals that one wishes to monitor. This paper proposes a method to identify individuals based on the depth images of gait sequences. The gait sequences are detected using previously presented results on activity recognition based on Hidden Markov Models (HMMs). The visibility of the person in the sequence is assessed from the likelihood of the sequence. We propose to perform the identification of the person from her height and gait in sequences in which she walks being fully visible. The gait pattern of the person is modeled using a HMM built from features of the trajectory of the centre of mass. A specific HMM is built for each person to be identified. This approach also allows us to identify unknown individuals who do not correspond to any of the built HMMs. We test the algorithm with 10 known and 2 unknown individuals. The results show that the presented method differentiates accurately enough the unknown and known individuals, and in the last case identifies correctly the individuals. In other words, our algorithm is able to identify the person of interest among other known (family, caregivers) or unknown persons (occasional individuals).

We present the OC16-CE80 Chinese-English mixlingual speech database which was released as a main resource for training, development and test for the Chinese-English mixlingual speech recognition (MixASR-CHEN) challenge on O-COCOSDA 2016. This database consists of 80 hours of speech signals recorded from more than 1,400 speakers, where the utterances are in Chinese but each involves one or several English words. Based on the database and another two free data resources (THCHS30 and the CMU dictionary), a speech recognition (ASR) baseline was constructed with the deep neural network-hidden Markov model (DNN-HMM) hybrid system. We then report the baseline results following the MixASR-CHEN evaluation rules and demonstrate that OC16-CE80 is a reasonable data resource for mixlingual research.

A speaker cluster-based speaker adaptive training (SAT) method under deep neural network-hidden Markov model (DNN-HMM) framework is presented in this paper. During training, speakers that are acoustically adjacent to each other are hierarchically clustered using an i-vector based distance metric. DNNs with speaker dependent layers are then adaptively trained for each cluster of speakers. Before decoding starts, an unseen speaker in test set is matched to the closest speaker cluster through comparing i-vector based distances. The previously trained DNN of the matched speaker cluster is used for decoding utterances of the test speaker. The performance of the proposed method on a large vocabulary spontaneous speech recognition task is evaluated on a training set of with 1500 hours of speech, and a test set of 24 speakers with 1774 utterances. Comparing to a speaker independent DNN with a baseline word error rate of 11.6%, a relative 6.8% reduction in word error rate is observed from the proposed method.