VAD-CLVA: Integrating CLIP with LLaVA for Voice Activity Detection

Real time magnetic resonance imaging (rtMRI) enables direct video capture of the moving vocal tract concurrent with audio signal providing valuable data for speech research. We consider a multimodal approach to voice activity detection (VAD) in the rtMRI recording that uses audio signal as well as MRI image sequence. The degraded quality of the audio recorded in the scanner motivates this multimodal scheme for robust VAD. Optimal regions in the MRI image are selected for performing VAD with a novel algorithm. VAD experiments using rtMRI data of two male and two female subjects show that VAD performance using optimally selected regions from MRI images is comparable to that using only audio signal. The optimal regions turn out to be parts of jaw, velum, glottis and lips. VAD performance using audio signal and MRI image sequence together is found to be significantly better (∼14% absolute improvement in VAD accuracy) than that using the audio only when the audio is contaminated with additive noise at low SNR. Index Terms: voice activity detection, vocal tract imaging, optimal region selection.

Mobile radio bandwidth can be efficiently utilized by not transmitting a talker's voice signal during pauses, which on the average accounts for 60% of a party's time in a conversation. A speech detector (SD) is needed to identify the speech-plus-noise and noise-only intervals within the talker's audio signal. In this way, both voice and data users can share the radio frequencies on demand.The SD should be simple, as each voice terminal must possess one. Deployed in a background of highly variable acoustic noise, the SD should be adaptive and robust. Also, the detector should produce acceptable speechquality and yet realize a bandwidth-compression gain close to the reciprocal of the intrinsic speech activity. Lastly, the detector's talkspurt/pause statistics should be such that the speech transmission is tolerant to variable delay induced by a dynamic-resource-sharing network.This thesis describes the design and development of a SD to meet the above requirements. Two signal features are used to make the binary decision: magnitude-energy and zero crossings. The SD also uses a "hangover" to admit weak utterances as speech and to produce continuous speech flow. The novel part of the SD is to use a signal-variability measure to adapt the parameters of the discrimination devices to the background noise characteristics.The SD has been implemented and developed on the 2920 single-chip digital signal processor. From observations, subjective evaluation, and objective measurements, the SD is shown to meet the said requirements.

Voice Activity Detection (VAD) provides the information whether an audio signal contains speech or not. Besides speech coding and transmission, there are many other applications in speech and audio processing that benefit from this information, and their performance is crucially dependent on the accuracy and robustness of the applied VAD. Various approaches to detect speech have been developed in the past, but when considering the challenging scenarios in which speech needs to be detected, e.g. hands-free communication in noisy environments or dialog in background music, there is still room for improvement. In this chapter, we describe the problem and the environments of VAD, discuss the procedure, examples for methods and their evaluation. Especially the more challenging application scenarios illustrate how superior human hearing can be compared to implementations of audio signal processing.

Voice Activity Detection (VAD) is a technique used to identify the presence of human voice in an audio signal. It is implemented as an always-on component in most speech processing applications. As speech is absent most of the time, this component typically dominates the overall average power consumption of the system (excluding microphone). The widespread usage in speech applications and the need for ultra low power VAD have led to a plethora of algorithms and implementations in the hardware domain, necessitating a comprehensive study and analysis to understand (real-time) requirements, different design parameters, testing strategies, but also to identify design trends, challenges and guidelines for future implementations and testing of VAD devices. A scoping review was conducted to identify the articles for hardware implementations of VAD from January 2010 - December 2021, the results of which are presented in this article. The results highlight a big design space being used for VAD along with a lack of standard testing methodology and usage of application-dependent performance metrics. An increased usage of filter-based feature extractors along with neural-network-based classifiers is observed. Due to lack of standardisation, no other trends can be established from the results. A set of rules and guidelines are therefore provided to facilitate the future development and benchmarking of VADs.

In recent decade, many state-of-the-art algorithms on image classification as well as audio classification have achieved noticeable successes with the development of deep convolutional neural network (CNN). However, most of the works only exploit single type of training data. In this paper, we present a study on classifying bird species by exploiting the combination of both visual (images) and audio (sounds) data using CNN, which has been sparsely treated so far. Specifically, we propose CNN-based multimodal learning models in three types of fusion strategies (early, middle, late) to settle the issues of combining training data cross domains. The advantage of our proposed method lies on the fact that We can utilize CNN not only to extract features from image and audio data (spectrogram) but also to combine the features across modalities. In the experiment, we train and evaluate the network structure on a comprehensive CUB-200-2011 standard data set combing our originally collected audio data set with respect to the data species. We observe that a model which utilizes the combination of both data outperforms models trained with only an either type of data. We also show that transfer learning can significantly increase the classification performance.

Recently QUT have released a set of annotated broadcast news videos (SAIVT-BNEWS) that we have made available at our website (https://www.qut.edu.au/research/saivt). This presentation will outline the dataset itself, covering 50 or so short news clips surrounding a single political event with many entities appearing in multuple records, and cover interesting research that QUT has, is currently, and is interested in performing on this dataset in the future. This presentation will cover existing published research, including image processing tasks like face detection and clustering; and speech processing tasks (including the use of visual speech) like speech detection, speaker recognition, and speaker diarisation. We have also started very interesting research on fusing multiple sources of information, including metadata, OCR, faces, speech, and scene detection to improve the performance of many techniques, but with a focus on improving the automatic extraction of entities (people, places, companies and organisations) from large volumes of audio-visual data, and this will also be addressed in this talk. As this dataset is publicly available for free to all researchers, QUT hopes that other researchers will make use of, and improve upon this dataset as well.

Voice activity detection is a fundamental problem in speech processing, which has been discussed for decades. However, it is a big challenge to determine the speech boundary in noisy environments because the corrupted speech is uncertain. In handing problems with noisy data, this study adopts a fuzzy neural network (FNN) to process the uncertainty. Furthermore, human speech perception is bimodal. We lip-read in noisy environments to improve intelligibility. This idea inspires us to adopt the visual information into the voice activity detection system. Based on the skin color segmentation, faces and mouths can be found in images. By analyzing the geometric shapes, the lip contour feature of speaker can be extracted. Then, the proposed fuzzy neural network considers not only audio but also visual information. Compared with the other voice activity detection, the proposed method for voice activity detection is more robust in the condition of low signal-to-noise ratio (SNR).

Ensemble of deep neural networks using acoustic environment classification for statistical model-based voice activity detection

There have been lots of previous studies on fluency evaluation of spontaneous speech. However, most of them focus on lexical cues, and little emphasis is placed on how diverse acoustic features and deep end-to-end models contribute to improving the performance. In this paper, we describe multi-layer neural network to investigate not only lexical features extracted from transcription, but also consider utterance-level acoustic features from audio data. We also conduct the experiments to investigate the performance of end-to-end approaches with mel-spectrogram in this task. As the speech fluency evaluation task, we evaluate our proposed method in two binary classification tasks of fluent speech detection and disfluent speech detection. Speech data of around 10 seconds duration each with the annotation of the three classes of “fluent,” “neutral,” and “disfluent” is used for evaluation. According to the two way splits of those three classes, the task of fluent speech detection is defined as binary classification of fluent vs. neutral and disfluent, while that of disfluent speech detection is defined as binary classification of fluent and neutral vs. disfluent. We then conduct experiments with the purpose of comparative evaluation of multi-layer neural network with diverse features as well as end-to-end models. For the fluent speech detection, in the comparison of utterance-level disfluency-based, prosodic, and acoustic features with multi-layer neural network, disfluency-based and prosodic features only are better. More specifically, the performance improved a lot when removing all of the acoustic features from the full set of features, while the performance is damaged a lot if fillers related features are removed. Overall, however, the end-to-end Transformer+VGGNet model with mel-spectrogram achieves the best results. For the disfluent speech detection, the multi-layer neural network using disfluency-based, prosodic, and acoustic features without fillers achieves the best results. The end-to-end Transformer+VGGNet architecture also obtains high scores, whereas it is exceeded by the best results with the multi-layer neural network with significant difference. Thus, unlike in the fluent speech detection, disfluency-based and prosodic features other than fillers are still necessary in the disfluent speech detection.

This paper presents an ultra-low-power voice activity detection (VAD) system to discriminate speech from non-speech parts of audio signals. The proposed VAD system uses level-crossing sampling for voice activity detection. The useless samples in the non-speech parts of the signal are eliminated due to the activity-dependent nature of this sampling scheme. A 40 ms moving window with a 30 ms overlap is exploited as a feature extraction block, within which the output samples of the level-crossing analog-to-digital converter (LC-ADC) are counted as the feature. The only variable used to distinguish speech and non-speech segments in the audio input signal is the number of LC-ADC output samples within a time window. The proposed system achieves an average of 91.02% speech hit rate and 82.64% non-speech hit rate over 12 noise types at −5, 0, 5, and 10 dB signal-to-noise ratios (SNR) over the TIMIT database. The proposed system including LC-ADC, feature extraction, and classification circuits was designed in 0.18 µm CMOS technology. Post-layout simulation results show a power consumption of 394.6 nW with a silicon area of 0.044 mm2, which makes it suitable as an always-on device in an automatic speech recognition system.

This research presents a novel methodology for detecting DeepFake voices, which is based on the effective classification of fake and real audio signals. To enhance the assessment of information in the audience, the voices of 58 politicians and public figures were compiled as fake and real audio files. In the study, fake audio samples were artificially generated, while real samples were obtained from authentic sources. The analysis of the audio signal structure employed Mel-Frequency Cepstral Coefficients (MFCC), Zero-Crossing Rate (ZCR) metrics, and data visualization techniques, including bar charts and histograms. During the research, the numerical distribution, lengths, MFCC features, and ZCR values of the fake and real audio samples were analyzed. LSTM and CNN models were tested for DeepFake voice detection, resulting in the LSTM model achieving 100% accuracy, while the CNN model was rated at 97.50% accuracy. The findings demonstrated that the LSTM model could accurately and reliably distinguish between fake and real audio, emphasizing the importance of assessing the authenticity of audio signals in light of the dangers posed by DeepFake technology. This research provides functional methodologies aimed at developing systems for visual individuals while also uncovering new ways to determine the authenticity of audio signals and demonstrating the effectiveness of applying modern deep learning technologies. The study emphasizes that DeepFake plays a significant role in assessing and identifying information in an audience and provides a foundation for future research and practice.

Audio-visual correlation learning aims at capturing and understanding natural phenomena between audio and visual data. The rapid growth of dl propelled the development of proposals that process audio-visual data and can be observed in the number of proposals in the past years. Thus encouraging the development of a comprehensive survey. Besides analyzing the models used in this context, we also discuss some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate objective functions frequently used and discuss how audio-visual data is exploited in the optimization process, i.e., the different methodologies for representing knowledge in the audio-visual domain. In fact, we focus on how human-understandable mechanisms, i.e., structured knowledge that reflects comprehensible knowledge, can guide the learning process. Most importantly, we provide a summarization of the recent progress of ()avcl and discuss the future research directions.

In this paper, we address the problem of multiple view data fusion in the presence of noise and interferences. Recent studies have approached this problem using kernel methods, by relying particularly on a product of kernels constructed separately for each view. From a graph theory point of view, we analyze this fusion approach in a discrete setting. More specifically, based on a statistical model for the connectivity between data points, we propose an algorithm for the selection of the kernel bandwidth, a parameter that, as we show, has important implications on the robustness of this fusion approach to interferences. Then, we consider the fusion of audio-visual speech signals measured by a single microphone and by a video camera pointed to the face of the speaker. Specifically, we address the task of voice activity detection, i.e., the detection of speech and nonspeech segments, in the presence of structured interferences such as keyboard taps and office noise. We propose an algorithm for voice activity detection based on the audio-visual signal. Simulation results show that the proposed algorithm outperforms competing fusion and voice activity detection approaches. In addition, we demonstrate that a proper selection of the kernel bandwidth indeed leads to improved performance.

The enormous amount of unstructured audio data available nowadays and the spread of its use as a data source in many applications are introducing new challenges to researchers in information and signal processing. The continuously growing size of digital audio information increases the difficulty of its access and management, thus hampering its practical usefulness. As a consequence, the need for content-based audio data parsing, indexing and retrieval techniques to make the digital information more readily available to the user is becoming ever more critical. The lack of proper indexing and retrieval systems is making de facto useless significant portions of existing audio information (and obviously audiovisual information in general). In fact, if generating digital content is easy and cheap, managing and structuring it to produce effective services is clearly not. This applies to the whole range of content providers and broadcasters which can amount to terabytes of audio and audiovisual data. It also applies to the audio content gathered in private collection of digital movies or music files stored in the hard disks of conventional personal computers. In summary, the goal of an audio indexing system will then be to automatically extract high-level information from the digital raw audio in order to provide new means to navigate and search in large audio databases. Since it is not possible to cover all applications of audio indexing, the basic concepts described in this chapter will be mainly illustrated on the specific problem of musical instrument recognition.

Accurate and effective voice activity detection (VAD) is a fundamental step for robust speech or speaker recognition. In this study, we proposed a hierarchical framework approach for VAD and speech enhancement. The modified Wiener filter (MWF) approach is utilized for noise reduction in the speech enhancement block. For the feature selection and voting block, several discriminating features were employed in a voting paradigm for the consideration of reliability and discriminative power. Effectiveness of the proposed approach is compared and evaluated to other VAD techniques by using two well-known databases, namely, TIMIT database and NOISEX-92 database. Experimental results show that the proposed method performs well under a variety of noisy conditions.