Particle Filter Algorithm for Single Speaker Tracking with Audio-Video Data Fusion

The paper proposes a transmission system for Motion-JPEG2000 video in mobile communications. In Motion JPEG2000, video data are compressed in frames according to JPEG2000 part1. In JPEG2000 part1 encoding, an input image is coded in the order of significant sub-bands, which are low-frequency bands. The proposed transmission system therefore assigns each sub-band data suitable weights according to their significance by using orthogonal multi-code CDMA so that bit errors are concentrated in the high-frequency sub-bands which are of little significance on image reconstruction. Concretely speaking, copies of significant sub-band data are made and the copied data are orthogonally multiplexed. The multiplexed data are averaged in the receiver so that the bit errors of significant sub-bands are reduced. On the other hand, the other sub-band data are not copied in order to increase transmission rates. Furthermore, audio data are transmitted with high quality by orthogonal multiplexing. The proposed system adaptively performs the above process for video and audio data. Moreover, these proposed systems are effective in the progressive order of LRCP (layer - resolution level - component - position) and of RLCP (resolution level - layer - component - position) on JPEG2000. Simulation results demonstrate that the proposed system can improve image quality compared with the conventional transmission system under the same BER.

Fibre channel is an emerging high-speed digital network technology that combines to make inroads into the avionics arena. The suitability of fibre channel for such applications is largely due to its flexibility in these key areas: Network topologies can be configured in point-to-point, arbitrated loop or switched fabric connections. The physical layer supports either copper or fiber optic implementations with a Bit Error Rate of less than 10<SUP>-12</SUP>. Multiple Classes of Service are available. Multiple Upper Level Protocols are supported. Multiple high speed data rates offer open ended growth paths providing speed negotiation within a single network. Current speeds supported by commercially available hardware are 1 and 2 Gbps providing effective data rates of 100 and 200 MBps respectively. Such networks lend themselves well to the transport of digital video and audio data. This paper summarizes an ANSI standard currently in the final approval cycle of the InterNational Committee for Information Technology Standardization (INCITS). This standard defines a flexible mechanism whereby digital video, audio and ancillary data are systematically packaged for transport over a fibre channel network. The basic mechanism, called a container, houses audio and video content functionally grouped as elements of the container called objects. Featured in this paper is a specific container mapping called Simple Parametric Digital Video (SPDV) developed particularly to address digital video in avionics systems. SPDV provides pixel-based video with associated ancillary data typically sourced by various sensors to be processed and/or distributed in the cockpit for presentation via high-resolution displays. Also highlighted in this paper is a streamlined Upper Level Protocol (ULP) called Frame Header Control Procedure (FHCP) targeted for avionics systems where the functionality of a more complex ULP is not required.

In the paper the features of protocols and the basic methods of organization of multimedia information transfer in telecommunication networks and distributed corporate safety networks are represented. The analysis of formats of basic elements of signals used in standards of multimedia information transfer is given.

While various methods for sound source localization have been developed, most of them utilize on the time difference of arrival (TDOA) between microphones or the measured head related transfer functions (HRTF). In case of a real robot implementation, the former has a merit of light computation load to estimate the sound direction but can not consider the effect of platform on TDOAs, while the latter can, because characteristics of robot platform are included in HRTF. However, the latter needs large resources for the HRTF database of a specific robot platform. We propose the compensation method which has the light computation load while the effect of platform on TDOA can be taken into account. The proposed method is used with spherical head related transfer function (SHRTF) on the assumption that robot platform, for example a robot head, installed microphones can be modeled to a sphere. We verify that the proposed method decreases the estimation error caused by the robot platform through the simulation.

Emotions are a crucial aspect of human life and the researchers have tried to build an automatic emotion recognition system that helps to provide important real-world applications. The psychologists have shown that emotions differ across culture, considering this fact, we provide and describe the first audio-visual Arabic emotional dataset which called (AVANEmo). In this work we aim to fill the gap between studies of emotion recognition for Arabic content and other languages by provided an Arabic dataset which is a major and fundamental part of build emotion recognition application. Our dataset contains 3000 clips for video and audio data, and it covers six basic emotional labels (Happy, Sad, Angry, Surprise, Disgust, Neutral). Also, we provide some baseline experiments to measure the primitive performance for automated audio and visual emotion recognition application using the AVANEmo dataset. The best accuracy that we achieved was 54.5% and 57.9% using the audio and visual data respectively. The data will be available for distribution to researchers.

Purpose: High-speed videoendoscopy (HSV) is an emerging, but barely used, endoscopy technique in the clinic to assess and diagnose voice disorders because of the lack of dedicated software to analyze the data. HSV allows to quantify the vocal fold oscillations by segmenting the glottal area. This challenging task has been tackled by various studies; however, the proposed approaches are mostly limited and not suitable for daily clinical routine.Method: We developed a user-friendly software in C# that allows the editing, motion correction, segmentation, and quantitative analysis of HSV data. We further provide pretrained deep neural networks for fully automatic glottis segmentation.Results: We freely provide our software Glottis Analysis Tools (GAT). Using GAT, we provide a general threshold-based region growing platform that enables the user to analyze data from various sources, such as in vivo recordings, ex vivo recordings, and high-speed footage of artificial vocal folds. Additionally, especially for in vivo recordings, we provide three robust neural networks at various speed and quality settings to allow a fully automatic glottis segmentation needed for application by untrained personnel. GAT further evaluates video and audio data in parallel and is able to extract various features from the video data, among others the glottal area waveform, that is, the changing glottal area over time. In total, GAT provides 79 unique quantitative analysis parameters for video- and audio-based signals. Many of these parameters have already been shown to reflect voice disorders, highlighting the clinical importance and usefulness of the GAT software.Conclusion: GAT is a unique tool to process HSV and audio data to determine quantitative, clinically relevant parameters for research, diagnosis, and treatment of laryngeal disorders.Supplemental Material S1. Supplemental figures: Figure S1. Neural network training metrics.Figure S2. Neural network evaluation.Figure S3. Example segmentations using different backbones from various source data.Figure S4. Audio analysis platform.Supplemental Material S2. Endoscopic footage with semi-automatic and fully automatic segmentation methods, multiple examples.Supplemental Material S3. GAT manual. Kist, A. M., Gomez, P., Dubrovskiy, D., Schlegel, P., Kunduk, M., Echternach, M., Patel, R., Semmler, M., Bohr, C., Durr, S., Schutzenberger, A., & Dollinger, M. (2021). A deep learning enhanced novel software tool for laryngeal dynamics analysis. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2021_JSLHR-20-00498

Content Based Video Retrieval (CBVR) is so popular these days, because of the increased utilization of video based analytical systems. Video based analytics is quite effective than image analysis, as a series of actions are captured by the video. This ends up with better decision making ability. The CBVR systems play an important role in boosting the human–computer interaction. This paper presents a multimodal CBVR that takes both the visual and audio information into account for retrieving relevant videos to the user. Two modules are employed by this work to deal with video and audio data. The video data is processed to detect the significant frame from shots and is achieved by Lion Optimization Algorithm (LOA). The features are extracted from the visual data and with respect to the audio data, MHEC and LPCC features are extracted. The extracted features are clustered by Kernelized Fuzzy C Mean (KFCM) algorithm. Finally, the feature database is formed and is utilized in the query matching process during the testing phase. The performance of the proposed work is tested in terms of precision, recall, F-measure and time consumption rates. The proposed CBVR system proves better performance than the existing approaches and is evident through attained results.

We present the hardware design of a high-fidelity, low-cost infrasonic-capable (6 Hz) passive acoustic monitoring (PAM) prototype instrument. The device allows temporal synchronisation using a variety of global navigation satellite system (GNSS) constellations. This allows the recordings made by independent spatially-distributed nodes to be synchronised in time for acoustic monitoring and localisation using, for example, time difference of arrival (TDOA). Each node is capable of sampling at 24-bit resolution with a sampling frequency that is adjustable between 8 kHz and 192 kHz. Audio samples are stored locally on a microSD card, along with timestamp and location metadata.

BackgroundRespectful, equitable, and supportive health care team interactions with patients and their companions are a key part of quality health care services. Despite the importance of communication and other aspects of engagement during inpatient postpartum care, little is known about how care is experienced during this period from families’ perspectives.MethodsThis study collected video and audio data with 15 birthing families (n = 9 English-speaking and n = 6 Spanish-speaking) and their health care team members during inpatient postpartum care in a southeastern United States academic medical center. This analysis quantifies health care team member presence in family hospital rooms, assesses linguistic appropriateness of health care team member verbal communication, and describes birthing parent and companion verbal reactions to health care team member interactions. A behavioral taxonomy was applied to identify these codes in the 12 h prior to hospital discharge. Additionally, we transcribed the birthing parent and companion verbal reactions for 10-minutes each time the health care team member departed their room. This content was inductively coded to identify topics and develop themes.ResultsA total of 160 h of video and audio data were coded across 15 participating families for this analysis. There were 19.9 h of missing data in the 12 h prior to hospital discharge due to equipment being turned off across five participants. At least one health care team member was present in the postnatal unit rooms within the observation period in 200 instances (median 13 times, range 5 to 19 times per participating family). Communication with Spanish-speaking birthing parents was linguistically appropriate for between 20.0% and 75.0% of interactions. Following health care team member presence, birthing parent and companion reactions were indeterminate 1 time, no verbal reaction 107 times, positive 25 times, confused 38 times, and negative 52 times. Many parents expressed more than one reaction. Reaction topics included clarity of information provided, postpartum pain management, hospital discharge coordination, health care team member conduct, and access to supplies.ConclusionThere are strengths in postpartum patient engagement as well as areas for health care team improvement. More appropriate communication including setting of clearer expectations surrounding various aspects of services on the postnatal unit may improve patient experiences. As a part of this improved quality of inpatient postpartum care, consistent utilization of interpretation services could decrease patient confusion, enable shared decision-making, and promote positive patient-provider relationships.

High-speed videoendoscopy is an important tool to study laryngeal dynamics, to quantify vocal fold oscillations, to diagnose voice impairments at laryngeal level and to monitor treatment progress. However, there is a significant lack of an open source, expandable research tool that features latest hardware and data analysis. In this work, we propose an open research platform termed OpenHSV that is based on state-of-the-art, commercially available equipment and features a fully automatic data analysis pipeline. A publicly available, user-friendly graphical user interface implemented in Python is used to interface the hardware. Video and audio data are recorded in synchrony and are subsequently fully automatically analyzed. Video segmentation of the glottal area is performed using efficient deep neural networks to derive glottal area waveform and glottal midline. Established quantitative, clinically relevant video and audio parameters were implemented and computed. In a preliminary clinical study, we recorded video and audio data from 28 healthy subjects. Analyzing these data in terms of image quality and derived quantitative parameters, we show the applicability, performance and usefulness of OpenHSV. Therefore, OpenHSV provides a valid, standardized access to high-speed videoendoscopy data acquisition and analysis for voice scientists, highlighting its use as a valuable research tool in understanding voice physiology. We envision that OpenHSV serves as basis for the next generation of clinical HSV systems.

Emotion recognition has been one of the cornerstones of human-computer interaction. Although decades of work has attacked the problem of automatic emotion recognition from either audio or video signals, the fusion of the two modalities is more recent. In this paper, we aim to tackle the problem when both audio and video data are available in a synchronized manner. We address the six basic human emotions, namely, anger, disgust, fear, happiness, sadness, and surprise. We employ an automatic face tracker to extract the different facial points of interest from a video. We then compute feature vectors for each video frame using distances and angles between the tracked points. For audio data, we use the pitch, energy and MFCC to derive feature vectors for each window as well as the entire audio signal. We use two standard techniques, GMM-based HMM and SVM, as the base classifiers. We then design a novel fusion method using the F-score of the base classifiers. We first demonstrate that our fusion approach can increase the accuracy of the base classifiers by as much as 5%. Finally, we show that our fusion-based bi-modal emotion recognition method achieves an overall accuracy of 54% on a publicly available database, which is an improvement upon the current state-of-the-art by 9%.

In recent years, the searching and indexing techniques for multimedia data are getting more attention in the area of multimedia databases. As many research works were done on the content-based retrieval of image and video data, less attention was received to the content-based retrieval of audio data. Audio is one of important multimedia information and there is a growing need for automatic audio indexing and retrieval techniques in recent years. Audio data contain abundant semantics and the audio signal processing can reduce computational complexity, so effective and efficient indexing and retrieval techniques for audio data are getting more attention. In this paper, problems of audio retrieval are discussed firstly. Then, main audio characteristics and features are introduced. Finally the combination of Support Vector Machine and General Model is described and the hybrid model is used in audio retrieval. Experiments show that the hybrid model is effective for audio classification.

Multipath propagation and reverberation of underwater acoustic signal affect the accuracy of localization estimation based on time difference of arrival (TDOA). To solve the problem, a novel algorithm is proposed to improve the accuracy of localization estimation of underwater acoustic source. Based on the signals received at an array of sensors, a general framework for acoustic source localization using particle filtering is proposed. A generic particle filtering framework was derived. The simulation results demonstrate the superiority of the proposed method. The resulting particle filter is shown to outperform traditional acoustic source localization method.

This paper presents a technique for enhanced localization of moving nodes in Wireless Sensor Networks (WSNs). The proposed technique is based on the use of Time Difference of Arrival (TDOA) along with Particle Filter (PF) method which is demonstrated to be capable of accurate detection since each particle in the PF represents a state which translates into the moving node location in the case of TDOA localization. The proposed technique outperforms other filtering methods, such as the Extended Kalman Filter (EKF) and the Unscented Kalman Filter (UKF), which are restricted to Gaussian distribution of the moving nodes. Numerical simulations verify the accuracy and robustness of the proposed TDOA-PF localization technique.

Estimation of Time-Varying Single Particle Tracking Models using Local Likelihood