Ripple Sparse Self-Attention for Monaural Speech Enhancement

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community Key Terms: Speech Enhancement, CHIME-7 Challenge ,Deep Neural Networks, UDASE Task, Audio Restoration, Subjective Evaluation, objective Evaluation

EnAbstract Background Improved articulatory placement through speech therapy may eliminate compensatory errors, improve velopharyngeal function, minimize perception of hypernasality, and improve speech intelligibility. Aim The aim of this work was to study the effect of preoperative speech therapy on the speech intelligibility in patients with residual velopharyngeal insufficiency in order to provide a better speech outcome in such patients. Patients and methods This study included 41 patients diagnosed with residual velopharyngeal insufficiency after cleft palate repair. Patients were divided into two groups: group I included 22 patients scheduled to undergo speech therapy 6 months preoperatively, and group II included 19 patients who did not receive preoperative speech therapy. Both groups underwent speech therapy for 6 months postoperatively. Evaluation was carried out through auditory perceptual assessment (APA) including the type and degree of open nasality, consonant precision, compensatory articulatory mechanisms (glottal articulation and pharyngealization of fricatives), audible nasal emission of air, and overall intelligibility of speech. All these elements are graded along a five-point scale in which 0 is normal and 4 is severe affection. Documentation of APA is performed by high fidelity speech and voice audio recording and endoscopy. Formal speech intelligibility testing was carried out using the Arabic Speech Intelligibility test, which is designed to provide an estimation of the overall speech intelligibility of children by providing a total score in percentage. Nasometry was performed for all patients using a Kay nasometer, which provides the ‘nasalance score’. Results On comparing the APA assessment of group I after 6 months of speech therapy postoperatively with the preoperative data, a highly significant decrease with regard to all parameters was revealed; however, in group II, the same comparison revealed a highly significant decrease as regards the degree of open nasality and nasal emission of air and a significant difference in terms of glottal articulation, pharyngealization of fricatives, and overall speech intelligibility. On comparing the two groups postoperatively, a significant difference in glottal articulation, pharyngealization of fricatives, and overall speech intelligibility, being more improved in group I, was revealed. Nasometry showed a significant difference between the postoperative results after speech therapy compared with the preoperative results, with a nonsignificant difference between the two groups postoperatively. In group I, the results of the Arabic Speech Intelligibility test showed a highly significant decrease in the number of patients with unintelligible speech and poor speech intelligibility, a significant decrease in number of patients with fair speech intelligibility, and a highly significant increase in the number of patients with good and excellent speech intelligibility. In group II, there was a significant decrease in the number of patients with unintelligible speech and poor speech intelligibility and a significant increase in the number of patients with fair, good, and excellent speech intelligibility. Conclusion and recommendations Speech therapy before surgery for residual velopharyngeal insufficiency can improve the results of postoperative therapy, with a better speech intelligibility outcome. Therefore, it is recommended to schedule a speech therapy program before secondary repair of the velopharyngeal valve in order to attain better speech intelligibility.

This study proposes a bi-directional recurrent neural network (Bi-RNN) post-processing method for speech enhancement (SE) at low signal-to noise ratios (SNR). Current speech enhancement solutions performed badly under low SNR situations. Loizou and Kim proposed a solution to reduce speech distortion errors in time-frequency (T-F) domain but it requires the knowledge of ground truth. As ground truth is unknown in real-life applications, the current study proposes to use a Bi-RNN to implement Loizou and Kim’s solution as a post-processing method for SE engines. Our solutions do not require prior knowledge of ground truth. The effectiveness of the proposed method is investigated with a spectral subtraction (SS) SE engine, a non-negative matrix factorization (NMF) SE engine, and a deep neural network ideal ratio mask (DNN-IRM) SE engine, under matched/mis-matched noise and different SNR conditions. Experimental results demonstrate that the proposed post-processing method effectively improved both perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) for all of these SE engines, especially at low SNR conditions.

Purpose This study (a) examined the effect of different levels of background noise on speech intelligibility and perceived listening effort in speakers with impaired and intact speech following treatment for head and neck cancer (HNC) and (b) determined the relative contribution of speech intelligibility, speaker group, and background noise to a measure of perceived listening effort. Method Ten speakers diagnosed with nasal, oral, or oropharyngeal HNC provided audio recordings of six sentences from the Sentence Intelligibility Test. All speakers were 100% intelligible in quiet: Five speakers with HNC exhibited mild speech imprecisions (speech impairment group), and five speakers with HNC demonstrated intact speech (HNC control group). Speech recordings were presented to 30 inexperienced listeners, who transcribed the sentences and rated perceived listening effort in quiet and two levels (+7 and +5 dB SNR) of background noise. Results Significant Group × Noise interactions were found for speech intelligibility and perceived listening effort. While no differences in speech intelligibility were found between the speaker groups in quiet, the results showed that, as the signal-to-noise ratio decreased, speakers with intact speech (HNC control) performed significantly better (greater intelligibility, less perceived listening effort) than those with speech imprecisions in the two noise conditions. Perceived listening effort was also shown to be associated with decreased speech intelligibility, imprecise speech, and increased background noise. Conclusions Speakers with HNC who are 100% intelligible in quiet but who exhibit some degree of imprecise speech are particularly vulnerable to the effects of increased background noise in comparison to those with intact speech. Results have implications for speech evaluations, counseling, and rehabilitation.

Competitive state-of-the-art automatic pathological speech intelligibility measures typically rely on regression training on a large number of features, require a large amount of healthy speech training data, or are applicable only to phonetically balanced scenarios where healthy and pathological speakers utter the same utterances. As a result, their performance in unseen data is unsatisfactory, and they cannot be used in low-resource languages or in phonetically unbalanced scenarios. To overcome these drawbacks, we propose a subspace-based intelligibility (SBI) measure. The SBI measure operates based on the hypothesis that dominant spectral patterns of pathological speech differ from intelligible speech (where the pathological and intelligible speech signals do not need to match in phonetic content), with the difference increasing as pathological speech intelligibility decreases. The SBI measure uses a minimal number of speech recordings to compute dominant spectral basis vectors spanning intelligible and pathological speech. The subspaces spanned by the intelligible and pathological spectral basis vectors are compared to each other through a subspace distance measure, which is directly used (i.e., without any training) as the pathological speech intelligibility estimate. Exploiting psychoacoustic evidence on the importance of spectral modulation cues to the perceived speech intelligibility and clinical evidence on the degradation of these cues in pathological speech, we show that the power of the proposed SBI measure lies in capturing the effect of spectral modulation degradation. To be able to additionally track possible degradations in the temporal structure of the pathological speech signal, we also propose two extensions of the SBI measure by incorporating short-time temporal information. Experimental results for different languages and speech pathologies show that the proposed intelligibility measures yield high and significant correlations with subjective intelligibility ratings, while not requiring any regression training or a large number of healthy speech recordings and being applicable to phonetically unbalanced scenarios.

In many deep learning based speech enhancement frameworks, the deep neural network (DNN) is usually trained under the mean square error (MSE) criterion to estimate the potential clean amplitudes. Yet, the MSE criterion is not directly connected with speech perception, which potentiates alternative training targets for further improvements in speech quality and intelligibility. This paper introduces maximizing the correlation between the enhanced speech and the reference, so as to further improve the speech quality and intelligibility of the enhanced speech synchronously. Based on the short-time Fourier transform amplitude spectra, the correlations are firstly calculated between the envelops along the time and frequency respectively. Then, the time and frequency envelop correlations are incorporated into the time-frequency correlation (TFC). Finally, the loss function is defined as 1 minus the TFC. In experiment, several loss functions containing the time and frequency envelop correlations are investigated. The system trained with the TFC loss outperforms the baseline trained with the MSE loss with promising improvements in both speech quality and intelligibility. Phoneme recognition results also confirm the effectiveness of maximizing the correlation for speech enhancement.

Graph-structured data arise ubiquitously in many application domains. A fundamental problem is to quantify their similarities. Graph kernels are often used for this purpose, which decompose graphs into substructures and compare these substructures. However, most of the existing graph kernels do not have the property of scale-adaptivity, i.e., they cannot compare graphs at multiple levels of granularities. Many real-world graphs such as molecules exhibit structure at varying levels of granularities. To tackle this problem, we propose a new graph kernel called Tree++ in this paper. At the heart of Tree++ is a graph kernel called the path-pattern graph kernel. The path-pattern graph kernel first builds a truncated BFS tree rooted at each vertex and then uses paths from the root to every vertex in the truncated BFS tree as features to represent graphs. The path-pattern graph kernel can only capture graph similarity at fine granularities. In order to capture graph similarity at coarse granularities, we incorporate a new concept called super path into it. The super path contains truncated BFS trees rooted at the vertices in a path. Our evaluation on a variety of real-world graphs demonstrates that Tree++ achieves the best classification accuracy compared with previous graph kernels.

Currently, almost all parallel implementations of programs fix the granularity at which parallelism is exploited at design time. Depending on the application structure and the parallel hardware structure, the programmer decides to exploit parallelism at a fine granularity or coarse granularity or some intermediate granularity, but this granularity is not changed at runtime. In this paper we argue that for many applications fixing the granularity in advance is not a good strategy. Instead it is advantageous to decide the granularity at which parallelism is exploited at runtime, as a function of the available hardware resources and as a function of the overheads associated with going to a finer granularity. We present experimental results from a parallel implementation of a geometric constraint satisfaction system to support our thesis. Our results show a significant advantage in using adaptive parallelism.

As transistor process technology approaches the nanometer scale, process variation significantly affects the design and optimization of high performance microprocessors. Prior studies have shown that chip operating frequency and leakage power can have large variations due to fluctuations in transistor gate length and sub-threshold voltage. In this work, we study the impact of process variation on microarchitecture soft error robustness, an increasing reliability design challenge in the billion-transistor chip era. We explore two techniques that can effectively mitigate the effect of design parameter variation while significantly enhancing microarchitecture soft error reliability. Our first technique is entry-based. It tolerates the deleterious impact of variable latency techniques on soft error reliability by reducing the quantity and residency cycle of vulnerable bits in the microarchitecture structure at a fine granularity. Our second technique is structure-based. It applies body biasing schemes to dynamically adapt transistor sub-threshold voltage (and hence device-level soft error robustness) to the program reliability characteristics at a coarse granularity. We also combine the two techniques which further produces improved results. Compared to existing process variation tolerant schemes, our proposed techniques achieve optimal trade-offs between reliability, performance, and power. To our knowledge, this paper presents the first study on characterizing and optimizing processor microarchitecture resilience to soft errors in light of process variation.

Malware family labels and key features used for the decision-making of Android malware detection models fall short of precise comprehension of malicious behaviors due to their coarse granularity. To solve these problems, in this paper, we first introduce the concept of the malicious behavior trajectory ( MBT ) and propose an innovative approach called ProMal . ProMal aims to automatically generate malware descriptions with fine granularity through extracted MBTs from malware for users. Specifically, a labeled dataset of MBTs is constructed through substantial human efforts to build a behavioral knowledge graph ( BxKG ). The BxKG is scalable and can be automatically updated using two strategies to ensure its completeness and timeliness: 1) taking into consideration the evolution of Android SDKs, and 2) mining new MBTs by leveraging the widely-used malware datasets. We highlight that the knowledge graph is essential in ProMal , which can reason new MBTs based on existing MBTs because of its structured data representation and semantic relation modeling, and thus helps effectively extract real MBTs in Android malware. We evaluated ProMal on a recent malware dataset where researcher-crafted malware descriptions are available, and the Precision, Recall, and F1-Score of MBT identification based on BxKG reached 96.97%, 91.43%, and 0.94, respectively, outperforming the state-of-the-art approaches. Taking MBTs identified from Android malware as inputs, precise, fine-grained, and human-readable descriptions can be generated using the large language model, whose readability and usability are verified through a user study. The generated descriptions play a significant role in interpreting and comprehending malware behaviors.

Previous algorithms for view-dependent level of detail provide local mesh refinements either at the finest granularity or at a fixed, coarse granularity. The former provides triangle-level adaptation, often at the expense of heavy CPU usage and low triangle rendering throughput; the latter improves CPU usage and rendering throughput by operating on groups of triangles.

Speech enhancement (SE) is crucial for improving the intelligibility and quality of noisy speech in applications such as speech recognition and assistive technologies. While traditional SE methods often fail in complex noise environments, deep learning-based approaches and multi-modal techniques have shown significant progress. However, existing multi-modal SE methods, such as audio-visual approaches, are limited in scenarios where visual data cannot be captured. EMG-SENet-basic explores Electromyography (EMG) signals as a robust complementary modality for SE, leveraging their resistance to environmental noise and potential for non-invasive acquisition. In addition, we propose an improved multi-modal SE framework named EMG-SENet-improved featuring a frame-level cross-attention mechanism for dynamic feature fusion, an adaptive weighting mechanism to better integrate amplitude information from both modalities, and the Con-TF-Mamba architecture to enhance local feature extraction in time and frequency domains. Furthermore, Temporal Masking, used to simulate frame loss, improves robustness under real-world conditions. Experimental results demonstrate the proposed method’s superior performance in enhancing SE quality and intelligibility, addressing key challenges in practical applications with degraded or incomplete data.

Altered nasality influences speech intelligibility. Automatic speech recognition (ASR) has proved suitable for quantifying speech intelligibility in patients with different degrees of nasal emissions. We investigated the influence of hyponasality on the results of speech recognition before and after nasal surgery using ASR. Speech recordings, nasal peak inspiratory flow and self-perception measurements were carried out in 20 German-speaking patients (8 women, 12 men; aged 38 ± 22 years) who underwent surgery for various nasal and sinus pathologies. The degree of speech intelligibility was quantified as the percentage of correctly recognized words of a standardized word chain by ASR (word recognition rate; WR). WR was measured 1 day before (t1), 1 day after with nasal packings (t2), and 3 months after (t3) surgery; nasal peak flow on t1 and t3. WR was calculated with program for the automatic evaluation of all kinds of speech disorders (PEAKS). WR as a parameter of speech intelligibility was significantly decreased immediately after surgery (t1 vs. t2 p < 0.01) but increased 3 months after surgery (t2 vs. t3 p < 0.01). WR showed no association with age or gender. There was no significant difference between WR at t1 and t3, despite a post-operative increase in nasal peak inspiratory flow measurements. The results show that ASR is capable of quantifying the influence of hyponasality on speech; nasal obstruction leads to significantly reduced WR and nasal peak flow cannot replace evaluation of nasality.

In this paper, we present a Speech Enhancement (SE) technique to improve intelligibility of speech perceived by Hearing Aid users using smartphone as an assistive device. We use the formant frequency information to improve the overall quality and intelligibility of the speech. The proposed SE method is based on new super Gaussian joint maximum a Posteriori (SGJMAP) estimator. Using the priori information of formant frequency locations, the derived gain function has " tradeoff" factors that allows the smartphone user to customize perceptual preference, by controlling the amount of noise suppression and speech distortion in real-time. The formant frequency information helps the hearing aid user to control the gains over the non-formant frequency band, allowing the HA users to attain more noise suppression while maintaining the speech intelligibility using a smartphone application. Objective intelligibility measures and subjective results reflect the usability of the developed SE application in noisy real world acoustic environment.