Train Your Classifier First: Cascade Neural Networks Training from Upper Layers to Lower Layers

End-to-end neural network systems for automatic speech recognition (ASR) are trained from acoustic features to text transcriptions. In contrast to modular ASR systems, which contain separately-trained components for acoustic modeling, pronunciation lexicon, and language modeling, the end-to-end paradigm is both conceptually simpler and has the potential benefit of training the entire system on the end task. However, such neural network models are more opaque: it is not clear how to interpret the role of different parts of the network and what information it learns during training. In this paper, we analyze the learned internal representations in an end-to-end ASR model. We evaluate the representation quality in terms of several classification tasks, comparing phonemes and graphemes, as well as different articulatory features. We study two languages (English and Arabic) and three datasets, finding remarkable consistency in how different properties are represented in different layers of the deep neural network.

Pretrained multilingual End-to-End (E2E) Automatic Speech Recognition (ASR) models demonstrate remarkable capabilities but struggle with domain-specific terminologies and underrepresented dialects. Fine-tuning requires expensive paired audio-transcript data, creating barriers to practical adaptation. This paper investigates language model (LM) integration via shallow fusion as an efficient, text-only adaptation method. We introduce Confidence Gated Fusion (CGF), a novel approach that dynamically determines the LM weight during decoding based on ASR model uncertainty, eliminating the expensive validation-set-dependent grid search required by traditional shallow fusion. We validate our approach using OpenAI’s Whisper across multiple model sizes on Arabic ASR, evaluating Modern Standard Arabic (MSA), Egyptian dialect (EGY), and a specialized judiciary domain. Integrating a domain-specific LM achieved up to 40.92% relative WER reduction on dialectal speech and 32.96% on the judiciary domain. Our CGF method achieved comparable performance to tuned baselines while requiring no hyperparameter optimization, with particular advantages in specialized domains (28.92% relative WER reduction on judiciary) and on smaller models where static weights often lack robustness.

Although over 300M around the world speak Bangla, scant work has been done in improving Bangla voice-to-text transcription due to Bangla being a low-resource language. However, with the introduction of the Bengali Common Voice 9.0 speech dataset, Automatic Speech Recognition (ASR) models can now be significantly improved. With 399hrs of speech recordings, Bengali Common Voice is the largest and most diversified open-source Bengali speech corpus in the world. In this paper, we outperform the State-of-the-Art (SOTA) pretrained Bengali ASR models by finetuning a pretrained wav2vec2 model on the common voice dataset. We also demonstrate how to significantly improve the performance of an ASR model by adding an n-gram language model as a post-processor. Finally, we do some experiments and hyperparameter tuning to generate a robust Bangla ASR model that is better than the existing ASR models.

This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

This research introduces a comprehensive strategy to enhance the performance of an existing automatic speech recognition (ASR) model, which has been previously documented in published articles. The study sets out to achieve several objectives. Firstly, it concentrates on updating the ASR model by retraining it with new datasets. This involves integrating samples from the latest Common Voice corpus release and data collected independently via the armspeech.com web application. Another key focus lies in optimizing the ASR model for near-real-time processing, intending to improve its speed and efficiency. The proposed adjustments to the model’s architecture aim to balance accuracy and processing speed, which is essential for applications requiring prompt speech recognition. Furthermore, the research explores the integration of Transformer models into the post-processing pipeline to introduce punctuation and capitalization into the ASR output. This step not only enhances the linguistic quality of transcriptions but also improves their readability and usability. In tandem with these advancements, the research presents a systematic approach to gathering, annotating, and storing datasets specifically tailored for punctuation and capitalization tasks. The methodology outlines the acquisition and organization of a dataset conducive to training Transformer models for these linguistic tasks. This comprehensive approach, which encompasses dataset enrichment, architectural modifications, and post-processing enhancements, aims to elevate the ASR model’s accuracy, speed, and linguistic refinement, with a particular focus on addressing the intricacies of the Armenian language. The research contributes valuable insights into the optimization of ASR systems, tackling both language-specific challenges and broader issues related to linguistic post-processing.

Toner charge distributions in corona charged toner layers on a grounded metal plate were investigated. The charge distributions were measured through image analysis of the toner particles displaced onto a filter, depending on their charge to diameter ratio, after being blown off from the toner inlet into an electric field and dispersed. It has been confirmed that the toner charge distributions became broader by corona charging. In order to clarify the cause, dual toner layers have been formed by using different color toners, and the charge distributions at the upper and lower layers have been measured separately. The results show that the lower toner layer has a wider charge distribution than that of the upper layer. This result is attributable to the fact that the lower parts of the toner layer are covered with the upper toner layer and difficult to corona charge. The toner particles in the lower layer located at the opening of the overlapped upper layer tend to be highly corona charged, while the toner particles around the opening in the upper layer receive only a small amount of the corona charge. It has thus been shown that difference in charge distribution between the upper and lower parts of the toner layer is caused by the following mechanisms: 1) The toner particles at the lower part of the layer are shielded from corona charging ions by the upper layer; 2) The toner particles in the lower layer, which are located at the opening of the charged particles in the upper layer, receive more charging ions by concentration of the electric field; 3) The toner particles at the upper layer receive fewer charging ions because of higher electric potential owing to a longer distance from the grounded plate.

The spiking neural network (SNN) using leaky-integrated-and-fire (LIF) neurons has been commonly used in automatic speech recognition (ASR) tasks. However, the LIF neuron is still relatively simple compared to that in the biological brain. Further research on more types of neurons with different scales of neuronal dynamics is necessary. Here we introduce four types of neuronal dynamics to post-process the sequential patterns generated from the spiking transformer to get the complex dynamic neuron improved spiking transformer neural network (DyTr-SNN). We found that the DyTr-SNN could handle the non-toy automatic speech recognition task well, representing a lower phoneme error rate, lower computational cost, and higher robustness. These results indicate that the further cooperation of SNNs and neural dynamics at the neuron and network scales might have much in store for the future, especially on the ASR tasks.

Early visual experience is essential for the maturation of visual functions in which the primary visual cortex plays crucial roles. The extraction of visual features based on response selectivity of individual neurons, a fundamental process in the cortex, is basically established by eye opening in rodents, suggesting that visual experience is required for the development of neural functions other than feature extraction. Here, we show that synchronized firing, which is important for visual information processing, occurs selectively in adjacent neurons sharing similar orientation or spatial frequency preferences in layers 2-4 (upper layer) of rat visual cortex. This feature-selective spike synchrony was rudimentary when the eyes opened and became prominent during the first few weeks after eye opening only in the presence of pattern vision. In contrast, synchronization in layers 5-6 (lower layer) was almost independent of orientation similarity and more weakly dependent on spatial frequency similarity compared with upper layer synchrony. Lower layer synchronization was strengthened during development after eye opening independently of visual experience as a whole. However, the feature selectivity of synchronization was regulated by visual inputs, whereas the inputs without contours were sufficient for this regulation. Therefore, we speculate that feature-selective synchronization in the upper layer may convey detailed information on visual objects to the higher-order cortex, whereas weakly feature-selective synchronization in the lower layer may covey rather rough visual information to the subcortical areas or higher-order cortex. A major role of visual experience may be to establish the specific neural circuits underlying highly feature-selective synchronization.SIGNIFICANCE STATEMENT The neuronal mechanisms underlying experience-dependent improvement of visual functions still remain unresolved. In this study, we investigated whether early visual experience contributes to the development of synchronized neural firing in the primary visual cortex, which plays important roles in visual information processing. We found that synchronized firing depends more remarkably on the similarity of preferred visual stimuli in the upper than lower layer neurons. Pattern vision during development was required for the establishment of spike synchrony in the upper but not the lower layer. These findings provide a new view regarding the role of sensory experience in the functional development of the cortex and the differences in the modes of information processing in the upper and lower cortical layers.

On days with clear skies in late August 2002 diurnal changes in the within-branch heterogeneity of photosynthetic photon flux density at the leaf surface (PPFDs) and leaf temperature (T leaf) were measured at natural leaf orientations in the upper and lower layers of a Fagus crenata crown. The PPFDs and T leaf measurements were converted to branch photosynthesis rates (P B; μmol s−1) using a photosynthetic model proposed by Farquhar et al. (Planta 149:78–90, 1980), an empirical stomatal conductance model suggested by Leuning et al. (Plant Cell Environ 18:339–335, 1995), and the total leaf area of the branches. To evaluate the importance of the variation in PPFDs and T leaf on photosynthesis calculations, P B calculated with the observed variation in PPFDs and T leaf was compared with estimates, based on the average (variation-free) values of PPFDs and T leaf, respectively. In both the layers, daily total P B values obtained with T leaf averaging were very close to those obtained with no averaging because of the weak inflection of the net photosynthesis rate (P n) to T leaf curves in the observed T leaf ranges (24.4–36.5 and 21.9–29.1°C in the upper and lower layers, respectively) and relatively small variation in within-branch T leaf at each time of day. This finding applied across potential climate conditions on fine days in August (T leaf range of 19.4–41.5 and 16.9–34.1°C in the upper and lower layers, respectively) and when the spatial scale was increased from branch to leaf layer, which increased the maximum variation in within-branch T leaf from 7.8 to 9.5°C and 4.5 to 5.5°C in the upper and lower layers, respectively. In contrast, averaging PPFDs caused 25–50% and 41–90% overestimation of daily total P B in the upper and lower layers, respectively, due to the sharp curvature in the PPFDs response curve to P n, and relatively large variation in within-branch PPFDs. Further, it led to overestimation of midday depression of P B in the upper layer, possibly because branch structural acclimation to incident light was neglected. Our results indicate that averaged values of T leaf could be used for the estimation of carbon gain at layer scale throughout August, but spatial variations in PPFDs need to be considered in detail for reliable estimates of carbon gain.

Solubility of magnesium hydrogen l-aspartate was examined. This salt is very easily soluble in water but hardly dissolves in organic solvents. In a mixture of water and organic solvents miscible with water, such as methanol, ethanol, propanol, and acetone, the salt distributes between the two layers, forming a dilute solution of the salt in the upper layer and a concentrated solution in the lower layer. The solubility of the slat in the upper layer increases exponentially with the molar fraction of water in the upper layer and the solubility is not affected by temperature. Analysis was carried out on the propanol-water-aspartate system in the upper and lower layers, three-componental phase diagram was prepared, and the Plait point, at which the upper and lower layers agreed, was calculated. It was also found that the exponential function established between the solubility in the upper layer and molar fraction of water thereof is also applicable to the lower layer.

End-to-end (E2E) models, including the attention-based encoder-decoder (AED) models, have achieved promising performance on the automatic speech recognition (ASR) task. However, the supervised training process of the E2E model needs a large amount of speech-text paired data. In contrast, self-supervised pre-training can pre-train the model on the unlabeled data and then fine-tune it on the limited labeled data to realize better performance. Most of the previous self-supervised pre-training methods focus on learning hidden representations from speech but ignore how to utilize the unpaired text. As a result, previous works often pre-train an acoustic encoder and then fine-tune it as a classification based ASR model, such as Connectionist Temporal Classification (CTC) based model, rather than an AED model. In this paper, we propose a self-supervised pre-training method for the AED model (SP-AED). The SP-AED method contains acoustic pre-training for the encoder, linguistic pre-training for the decoder, and an adaptive combination fine-tuning for the whole system. We first design a linguistic pre-training method for decoder by utilizing the text-only data. The decoder will be pre-trained as a noise-condition language model to learn the prior distribution of the text. Then, we pre-train the AED encoder with the wav2vec2.0 method with some modifications. Finally, we combine the pre-trained encoder and decoder and fine-tune them on the limited labeled data. We design an adaptive combination method during fine-tuning by modifying the decoder's input and output to prevent catastrophic forgetting. Experiments prove that compared with the random initialized models, the SP-AED pre-trained models can realize up to 17% relative improvement. And with similar model size or computational cost, we can get comparable results to other classification-based models on both English and Chinese corpus.

Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER). Previous works usually adopt end-to-end models and has strong dependency on Pseudo Paired Data and Original Paired Data. But when only pre-training on Pseudo Paired Data, previous models have negative effect on correction. While fine-tuning on Original Paired Data, the source side data must be transcribed by a well-trained ASR model, which takes a lot of time and not universal. In this paper, we propose UCorrect, an unsupervised Detector-Generator-Selector framework for ASR Error Correction. UCorrect has no dependency on the training data mentioned before. The whole procedure is first to detect whether the character is erroneous, then to generate some candidate characters and finally to select the most confident one to replace the error character. Experiments on the public AISHELL-1 dataset and WenetSpeech dataset show the effectiveness of UCorrect for ASR error correction: 1) it achieves significant WER reduction, achieves 6.83% even without fine-tuning and 14.29% after fine-tuning; 2) it outperforms the popular NAR correction models by a large margin with a competitive low latency; and 3) it is an universal method, as it reduces all WERs of the ASR model with different decoding strategies and reduces all WERs of ASR models trained on different scale datasets.

This article concerns comparative studies on the Automatic Speech Recognition (ASR) model incorporated with the Large Language Model (LLM) used for medical interviews. The proposed solution is tested on polish language benchmarks and dataset with medical interviews. The latest ASR technologies are based on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. Most of them work as end-to-end solutions. The presented approach in the case of the Whisper model shows a two-stage solution with End-To-End ASR and LLM working together in a pipeline. The ASR output is an input for LLM. The LLM is a component by which the output from ASR is corrected and improved. Comparative studies for automatic recognition of the Polish language between modern End-To-End deep learning architectures and the ASR hybrid model were performed. The medical interview tests were performed with two state-of-the-art ASR models: OpenAI Whisper incorporated with LLM and Scribe ElevenLabs. Additionally, the results were compared with five more end-to-end models (QuartzNet, FastConformer, Wav2Vec 2.0 XLSR and ESPnet Model Zoo) on Mozilla Common Voice and VoxPopuli databases. Tests were conducted for clean audio signal, signal with bandwidth limitation, and degraded. The tested models were evaluated on the basis of Word Error Rate (WER) and Character Error Rate (CER). The results show that the Whisper model performs by far the best among the open-source models. ElevenLabs Scribe model, on the other hand, performs best for Polish on both general benchmark and medical data.

The fluid dynamic processes affecting the mechanism and amount of assimilation of the roof of a magma chamber have been studied in a series of analogue laboratory experiments. An aqueous solution of $$Na_{2}CO_{3}$$ was first cooled and crystallized at the top of a laboratory tank to form a solid roof layer of prescribed composition. The tank was then drained and refilled with warmer solution, either $$Na_{2}CO_{3}$$ (of the same or different composition) or, in two experiments, $$KNO_{3}$$. The metal top in all cases was held below the eutectic temperature for the system $$Na_{2}CO_{3}-H_{2}O$$. The experiments were continued until substantial crystallization, and in some cases complete solidification, were achieved. Starting with a dense solid roof layer of $$Na_{2}CO_{3}10H_{2}O$$ and ice on the $$Na_{2}CO_{3}$$ side of the eutectic, and a warmer eutectic $$Na_{2}CO_{3}$$ solution below, there was rapid convective cooling of the liquid layer accompanied by melting of the roof, producing a compositionally dense liquid which convected downwards. When the roof was a solid of eutectic composition and the warm input fluid was much denser, a light stable liquid layer formed. The two liquid layers were separated by a double-diffusive interface through which heat was transported rapidly, but through which there was little mass transfer. As the lower layer cooled and crystallized at the bottom, its $$Na_{2}CO_{3}$$ concentration decreased, and the latent heat released by crystallization was transported upwards through the interface and contributed to the heat required for melting at the roof. In an experiment in which the mean compositions of the roof and the input solution were comparable, melting also led to the development of a stable upper liquid layer, as denser $$Na_{2}CO_{3}10H_{2}O$$ crystals fell off, leaving a lighter melt of eutectic composition behind. With a different denser solute ($$KNO_{3}$$) below, the system at first behaved similarly to the $$Na_{2}CO_{3}$$ experiment and produced a light roof layer, but later the densities of the two liquid layers evolved due to cooling and crystallization until they became equal, resulting in overturning and thorough mixing. The implication for a basaltic magma chamber, in which the magma produced by melting of the roof rocks is lighter than the magma below, is that the melted material will remain at the top of the chamber and be chemically isolated from the basaltic magma at the bottom. The process of AFC (assimilation with fractional crystallization) implies that the heat required for assimilation comes from the latent heat of crystallization, and that these two processes occur simultaneously in a single body of well-mixed magma. The melt released forms a light roof layer which may initially mix with some of the underlying basaltic magma, but then mixes very little. As a consequence, assimilation is separated from the crystallization in space and time. The interface between the upper and lower layers still allows rapid thermal transport vertically, so that crystallization at the floor supplies heat to assimilate the roof, but little roof material is incorporated in the lower layer as it crystallizes. At a later stage the contaminated magma at the roof crystallizes so that fractional crystallization follows assimilation, rather than the two processes occurring simultaneously from the same body of magma. True AFC is confined to the lower basaltic layer but is limited by the amount of material which can be transported across the double-diffusive interface which divides the basaltic layer from the melted roof layer. AFC does become important if large numbers of blocks of the roof fall through the upper layer and melt in the lower layer.

Subspace Gaussian mixture based language modeling for large vocabulary continuous speech recognition