Online Speech Recognition Using Multichannel Parallel Acoustic Score Computation and Deep Neural Network (DNN)- Based Voice-Activity Detector

This paper proposes a frame-level selective-decoding method by using both native acoustic models (AMs) and non-native AMs in order to construct a robust speech recognition system for non-native speech as well as native speech. To this end, we use two kinds of well-trained AMs: (a) AMs trained with a large amount of native speech (native AMs) and (b) AMs trained with a plenty amount of non-native speech (non-native AMs). First, each speech feature vector is decoded using native AMs and non-native AMs in parallel. And, we select proper AMs by comparing the likelihoods of the two AMs. Then, the next M frames of speech feature vectors are decoded by using the selected AMs, where M is a pre-defined parameter. The selection and the decoding procedures are repeated until an end of an utterance is encountered. From automatic speech recognition (ASR) experiments for English spoken by Korean speakers, it is shown that an ASR system employing the proposed method reduces an average word error rate (WER) by 16.6% and 41.3% for English spoken by Koreans and native English, respectively, when compared to an ASR system employing an utterance-level selective-decoding method.

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial animation parameters (FAPs) supported by the MPEG-4 standard for the visual representation of speech. We also describe a robust and automatic algorithm we have developed to extract FAPs from visual data, which does not require hand labeling or extensive training procedures. The principal component analysis (PCA) was performed on the FAPs in order to decrease the dimensionality of the visual feature vectors, and the derived projection weights were used as visual features in the audio-visual automatic speech recognition (ASR) experiments. Both single-stream and multistream hidden Markov models (HMMs) were used to model the ASR system, integrate audio and visual information, and perform a relatively large vocabulary (approximately 1000 words) speech recognition experiments. The experiments performed use clean audio data and audio data corrupted by stationary white Gaussian noise at various SNRs. The proposed system reduces the word error rate (WER) by 20% to 23% relatively to audio-only speech recognition WERs, at various SNRs (0–30 dB) with additive white Gaussian noise, and by 19% relatively to audio-only speech recognition WER under clean audio conditions.

A new speech corpus of super-elderly Japanese for acoustic modeling

Regional languages play an important role in cultural preservation and strengthening ethnic identity. Bugis, one of the most widely spoken regional languages, can be studied through speech recognition technology. This study analyzes the performance of two Deep Learning models, Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory (BiLSTM), for converting Bugis speech into text. The dataset consists of 2,500 audio recordings from native speakers along with their transcripts. The preprocessing stage includes trimming, denoising, and feature extraction using Mel-Frequency Cepstral Coefficients (MFCC). Evaluation using Word Error Rate (WER) and Bilingual Evaluation Understudy (BLEU) shows that BiLSTM outperforms LSTM, achieving a WER of 4.88% and a BLEU of 86.82%, compared to LSTM’s WER of 90.33% and BLEU of 3.86%. These findings indicate that BiLSTM is more effective in capturing contextual information, resulting in more accurate transcriptions. This research contributes to the preservation of the Bugis language and provides insights into the application of Deep Learning in low-resource speech recognition. Future work will explore attention mechanisms and larger datasets to further improve model performance

Conventional far-field automatic speech recognition (ASR) systems typically employ microphone array techniques for speech enhancement in order to improve robustness against noise or reverberation. However, such speech enhancement techniques do not always yield ASR accuracy improvement because the optimization criterion for speech enhancement is not directly relevant to the ASR objective. In this work, we develop new acoustic modeling techniques that optimize spatial filtering and long short-term memory (LSTM) layers from multi-channel (MC) input based on an ASR criterion directly. In contrast to conventional methods, we incorporate array processing knowledge into the acoustic model. Moreover, we initialize the network with beamformers' coefficients. We investigate effects of such MC neural networks through ASR experiments on the real-world far-field data where users are interacting with an ASR system in uncontrolled acoustic environments. We show that our MC acoustic model can reduce a word error rate (WER) by~16.5\% compared to a single channel ASR system with the traditional log-mel filter bank energy (LFBE) feature on average. Our result also shows that our network with the spatial filtering layer on two-channel input achieves a relative WER reduction of~9.5\% compared to conventional beamforming with seven microphones.

In cybersecurity, the rise of fileless malware poses a significant challenge to endpoint security. Traditional detection methods often fail against these sophisticated attacks, necessitating advanced techniques like deep learning models. This study highlights the limitations of Bi-Directional Long Short-Term Memory (BLSTM) models in dynamic malware analysis and proposes enhancements through Convolutional Long Short-Term Memory (ConvLSTM) architecture. BLSTM models process input sequences in forward and backward directions, combining the results into one output. While this dual-layer approach improves analysis, it is time-consuming, potentially increasing the risk of fileless malware attacks. A key limitation of BLSTM is the lack of parameter sharing between forward and backward directions. This reduces its ability to capture spatial and temporal features simultaneously, hindering effectiveness in detecting fileless malware. To address this, the ConvLSTM model consolidates feature extraction within a single LSTM cell layer. ConvLSTM breaks down samples into subsequence and uses timesteps for additional feature extraction, enabling spatial-temporal data analysis and improving malware prediction accuracy. The model was tested using a dynamic malware dataset. Unlike traditional LSTM, ConvLSTM integrates convolutional layers, allowing parameter sharing across both spatial and temporal dimensions. This reduces computational complexity and improves model performance in handling multidimensional data. The research re-simulated prior work with BLSTM using the same malware dataset. The Spyder app ran the event simulator and the ConvLSTM model's results replaced BLSTM's using identical parameters. Time, accuracy and loss were the main performance metrics. ConvLSTM outperformed BLSTM, achieving 98% detection accuracy compared to BLSTM's 90%. It also significantly reduced processing time, averaging 10 seconds, while BLSTM took 22 seconds. ConvLSTM experienced lower losses, averaging 10% per epoch versus BLSTM's 20%. In conclusion, ConvLSTM offers superior performance over BLSTM in fileless malware detection. Its enhanced computational efficiency and ability to quickly mitigate threats make it a robust solution for fortifying endpoint security against evolving cyber threats. ConvLSTM holds potential in strengthening defence mechanisms against sophisticated malware attacks, providing a proactive approach to safeguarding networks and data.

We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. Using our new corpus, which includes the speech of elderly people living in three regions of Japan, we conducted speech recognition experiments using a variety of DNN-HNN acoustic models. As training data for our acoustic models, we examined whether a standard adult Japanese speech corpus (JNAS), an elderly speech corpus (S-JNAS) or a spontaneous speech corpus (CSJ) was most suitable, and whether or not adaptation to the dialect of each region improved recognition results. We adapted each of our three acoustic models to all of our speech data, and then re-adapt them using speech from each region. Without adaptation, the best recognition results were obtained when using the S-JNAS trained acoustic models (total corpus: 21.85% Word Error Rate). However, after adaptation of our acoustic models to our entire corpus, the CSJ trained models achieved the lowest WERs (entire corpus: 17.42%). Moreover, after readaptation to each regional dialect, the CSJ trained acoustic model with adaptation to regional speech data showed tendencies of improved recognition rates. We plan to collect more utterances from all over Japan, so that our corpus can be used as a key resource for elderly speech recognition in Japanese. We also hope to achieve further improvement in recognition performance for elderly speech.

In real-world environments, automatic speech recognition (ASR) is highly affected by reverberation and background noise. A well-known strategy to reduce such adverse interferences in multi-microphone scenarios is microphone array acoustic beamforming. Recently, time-frequency (T-F) mask-based acoustic beamforming receives tremendous interest and has shown great benefits as a front-end for noise-robust ASR. However, the conventional neural network (NN) based T-F mask estimation approaches are only trained in parallel simulated speech corpus, which results in poor performance in the real data testing, where a data mismatch problem occurs. To make the NN-based mask estimation, termed as NN-mask, more robust against data mismatch problem, this paper proposes a bi-directional long short-term memory (BiLSTM) based teacher-student (T-S) learning scheme, termed as BiLSTM-TS, which can utilize the real data during student network training stage. Moreover, in order to further suppress the noise in the beamformed signal, we explore three different mask-based post-processing methods to find a better way to utilize the estimated masks from NN. The proposed approach is evaluated as a front-end for ASR on the CHiME-3 dataset. Experimental results show that the data mismatch problem can be reduced significantly by the proposed strategies, leading to relative 4% Word Error Rates (WER) reduction compared to conventional BiLSTM mask-based beamforming, in the real data test set.

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

PurposeOne of the paramount challenges in the field of structural health monitoring (SHM) is the precise assessment of structural damage. Recently, many methods have been proposed to diagnose damage in structures using time series data. Nevertheless, these techniques are restricted in their ability to capture the bidirectional temporal and spatial relationships among the data. To remedy these shortcomings, this study proposes an advanced deep learning (DL) technique that integrates one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks, termed BiLSTM-1DCNN.Design/methodology/approachThis study proposes an advanced deep learning (DL) technique that integrates one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks, termed BiLSTM-1DCNN. The novelty of BiLSTM-1DCNN comes in its effective fusion of 1DCNN’s local feature extraction and BiLSTM’s capacity to capture long-term information from both directions (forward and backward).FindingsValidation using a dataset from the Nam O steel truss bridge demonstrates that the proposed method surpasses conventional models, including 1DCNN (87.9%), LSTM (72.5%), BiLSTM (76.1%) and LSTM-1DCNN (91.1%), achieving an accuracy rate of 96.4%. This approach provides a significant advancement in SHM, paving the way for more accurate and reliable damage detection.Research limitations/implicationsThe study’s limitations include potential overfitting due to the model’s complexity and the high computational resources required. The BiLSTM-1DCNN model’s performance is also influenced by the quality and quantity of the training data as well as the specific characteristics of the dataset. Additionally, while the model performs well in detecting damage in the Nam O steel truss bridge, its generalizability to other types of structures and environmental conditions remains untested.Practical implicationsThe BiLSTM-1DCNN model offers significant practical benefits for structural health monitoring by enhancing damage detection accuracy. Its ability to effectively capture both local features and long-term temporal dependencies makes it a valuable tool for real-time monitoring and early detection of structural issues. This can lead to more timely maintenance actions, potentially reducing repair costs and improving safety. The model’s high accuracy in detecting damage can also support infrastructure management decisions, contributing to better resource allocation and enhanced overall safety of critical structures.Social implicationsThe BiLSTM-1DCNN model’s advanced damage detection capabilities have positive social implications, such as improving the safety and reliability of public infrastructure like bridges. Enhanced structural health monitoring can reduce the risk of catastrophic failures, protecting lives and reducing accidents. By providing more accurate and timely assessments, the model can also lead to more efficient use of maintenance and repairs, ensuring that resources are allocated effectively.Originality/valueThe originality of the BiLSTM-1DCNN model lies in its innovative integration of one-dimensional convolutional neural networks (1DCNN) with bidirectional long short-term memory (BiLSTM) networks. This unique combination leverages 1DCNN’s strength in local feature extraction and BiLSTM’s capacity to capture temporal dependencies in both directions, addressing the limitations of existing methods. The model’s high accuracy in detecting structural damage, as demonstrated by its performance on the Nam O steel truss bridge dataset, highlights its value in advancing structural health monitoring technologies and offers a significant improvement over traditional approaches.

The COVID-19 pandemic has triggered a global crisis in health and economic sectors, causing profound impacts on sea transport and trade. This research paper investigates the ramifications of the pandemic on maritime shipping prices, and, hence, their subsequent influence on the stocks of shipping companies. This global upheaval disrupted international trade significantly, resulting in an unprecedented demand surge for shipping services and a substantial spike in freight rates. This study is intended to propose a predictive method based on Multivariate Multi-step convolutional neural network - Bidirectional Long Short-Term Memory (Multivariate Multi-step CNNBiLSTM) networks in order to forecast the prices of three of the most prominent stocks of big organizations operating in maritime transport. The proposed method is composed of Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (BiLSTM), where the research utilizes CNN to help for feature extraction from the inputted data, alongside BiLSTM that forecasts the closing stock price for the upcoming five days, utilizing the extracted feature data. Hence, stock price prediction can be realized by applying a novel optimization strategy, which was founded on the Multivariate Multi-step CNN-BiLSTM model and utilizing the Adam optimizer. Prediction accuracy can be assessed by incorporating four metrics into the system: Root Mean Square Error, Mean Absolute Error, Mean Absolute Percentage Error, and Median Absolute Percentage Error. Experimental findings demonstrate that Multivariate Multi-step CNN-BiLSTM yields the most dependable stock price forecasts with the highest accuracy. The proposed prediction method, correctly applied, can yield economic benefits at the macr and micro levels; the prediction accuracy can help policy makers make better future outlook estimates in relation to inflation, gross domestic product (GDP), and unemployment levels that might be impacted by the volatile, uncontrolled, or unexpected fluctuations of stock prices of some leading economic sectors that are closely connected to global shipping and supply chain operations; thus, leading to serious impacts at the microeconomic level in relation to costs, supply and demand, and behavior of individual consumers and companies.

Speech recognition has become increasingly important in various real-world applications. However, Vietnamese presents unique linguistic challenges such as tones, syllabic structures, and complex morphology, which make speech recognition for this language significantly different from that of languages like English. In this paper, we propose a deep learning approach that combines Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) networks to recognize Vietnamese speech using the VIVOS dataset. The CNN component is employed to extract spatial features from audio spectrograms, while the BiLSTM captures the bidirectional temporal dependencies in speech signals. Experimental results show that the proposed CNN-BiLSTM model achieves a competitive Word Error Rate (WER) of 14.7%. These results highlight the potential of deep learning techniques in effectively recognizing tonal languages such as Vietnamese.

Prediction of PM2.5 concentration in urban agglomeration of China by hybrid network model

Tuberculosis is an infectious disease and requires serious treatment. Extrapulmonary Tuberculosis is detected using a microscope. Currently it will take a long time because the fluid preparations are viewed in a microscope one by one carefully and in the fluid preparations there are 150 fields of vision. Examination for Extra Pulmonary Tuberculosis by culture takes between 1-2 weeks or even more. Examination by biopsy will take a long time because the fluid preparations are looked at carefully under the microscope one by one. The image of Tuberculosis is expressed if in the image there is a bacillus object in red, and it turns out that apart from the bacillus object there are other objects also in red. So that examinations for tuberculosis can be more efficient, examinations using computer technology are needed. This research aims to compare the Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory (BiLSTM) classification methods in the detection of extra-pulmonary tuberculosis disease to obtain better accuracy results. This research carried out HSI color space transformation, segmentation using global thresholding, feature extraction using 13 features based on shape and texture using the Correlation Based Feature Selection (CFS) feature selection method. The results show that BiLSTM has the best accuracy with a value of 88.40% at the number of features = 3, namely Short Run High Gray-Level Emphasis, Run Length Nonuniformity, Minor axis length, while LSTM produces an accuracy of 63.19% at the number of features = 5. BiLSTM is capable of detecting opposite features, meaning that BiLSTM can detect opposite features in data sequences and BiLSTM's ability to understand multiple contexts, so it tends to provide more accurate results in some data classification tasks.

Code-switching is a commonly occurring phenomenon in many multilingual communities, wherein a speaker switches between languages within a single utterance. Conventional Word Error Rate (WER) is not sufficient for measuring the performance of code-mixed languages due to ambiguities in transcription, misspellings and borrowing of words from two different writing systems. These rendering errors artificially inflate the WER of an Automated Speech Recognition (ASR) system and complicate its evaluation. Furthermore, these errors make it harder to accurately evaluate modeling errors originating from code-switched language and acoustic models. In this work, we propose the use of a new metric, transliteration-optimized Word Error Rate (toWER) that smoothes out many of these irregularities by mapping all text to one writing system and demonstrate a correlation with the amount of code-switching present in a language. We also present a novel approach to acoustic and language modeling for bilingual code-switched Indic languages using the same transliteration approach to normalize the data for three types of language models, namely, a conventional n-gram language model, a maximum entropy based language model and a Long Short Term Memory (LSTM) language model, and a state-of-the-art Connectionist Temporal Classification (CTC) acoustic model. We demonstrate the robustness of the proposed approach on several Indic languages from Google Voice Search traffic with significant gains in ASR performance up to 10% relative over the state-of-the-art baseline.