OpenHSV: an open platform for laryngeal high-speed videoendoscopy

Purpose: High-speed videoendoscopy (HSV) is an emerging, but barely used, endoscopy technique in the clinic to assess and diagnose voice disorders because of the lack of dedicated software to analyze the data. HSV allows to quantify the vocal fold oscillations by segmenting the glottal area. This challenging task has been tackled by various studies; however, the proposed approaches are mostly limited and not suitable for daily clinical routine.Method: We developed a user-friendly software in C# that allows the editing, motion correction, segmentation, and quantitative analysis of HSV data. We further provide pretrained deep neural networks for fully automatic glottis segmentation.Results: We freely provide our software Glottis Analysis Tools (GAT). Using GAT, we provide a general threshold-based region growing platform that enables the user to analyze data from various sources, such as in vivo recordings, ex vivo recordings, and high-speed footage of artificial vocal folds. Additionally, especially for in vivo recordings, we provide three robust neural networks at various speed and quality settings to allow a fully automatic glottis segmentation needed for application by untrained personnel. GAT further evaluates video and audio data in parallel and is able to extract various features from the video data, among others the glottal area waveform, that is, the changing glottal area over time. In total, GAT provides 79 unique quantitative analysis parameters for video- and audio-based signals. Many of these parameters have already been shown to reflect voice disorders, highlighting the clinical importance and usefulness of the GAT software.Conclusion: GAT is a unique tool to process HSV and audio data to determine quantitative, clinically relevant parameters for research, diagnosis, and treatment of laryngeal disorders.Supplemental Material S1. Supplemental figures: Figure S1. Neural network training metrics.Figure S2. Neural network evaluation.Figure S3. Example segmentations using different backbones from various source data.Figure S4. Audio analysis platform.Supplemental Material S2. Endoscopic footage with semi-automatic and fully automatic segmentation methods, multiple examples.Supplemental Material S3. GAT manual. Kist, A. M., Gomez, P., Dubrovskiy, D., Schlegel, P., Kunduk, M., Echternach, M., Patel, R., Semmler, M., Bohr, C., Durr, S., Schutzenberger, A., & Dollinger, M. (2021). A deep learning enhanced novel software tool for laryngeal dynamics analysis. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2021_JSLHR-20-00498

Spatial Segmentation for Laryngeal High-Speed Videoendoscopy in Connected Speech

Hypercapnia produces a profound effect on respiratory drive and upper airway function to maintain airway patency. Previous work has evaluated the effects of hypercapnia on the sole arytenoid abductor, the posterior cricoarytenoid (PCA), using indirect measures of function, such as electromyography and direct nerve recording. Here we describe a novel method to evaluate PCA function in anesthetized animals and use this method to determine the effects of hypercapnia on PCA function. Eight dogs were anesthetized, and a laryngeal mask airway was used, in combination with high-speed videoendoscopy, to evaluate laryngeal function. A stepwise increase in inspired partial pressure of CO2 produced marked arytenoid abduction above 70-mmHg end-tidal CO2 (ETCO2 ) (P < 0.001). Glottic length increased above 80-mmHg ETCO2 (P < 0.02), and this lead to underrepresentation of changes in glottic area, if standard measures of glottic area (normalized glottic gap area) were used. Use of a known scale to determine absolute glottic area demonstrated no plateau with increasing ETCO2 up to 120 mmHg. Ventilatory parameters also continued to increase with no evidence of a maximal response. In a second anesthetic episode, repeated bursts of transient hypercapnia for 60 s with an ETCO2 of 90 mmHg produced a 43-55% increase in glottic area (P < 0.001) at or shortly after the end of the hypercapnic burst. A laryngeal mask airway can be used in combination with videoendoscopy to precisely determine changes in laryngeal dimensions with high temporal resolution. Absolute glottic area more precisely represents PCA function than normalized glottic gap area at moderate levels of hypercapnia.

In laryngeal high-speed videoendoscopy (HSV) the area between the vibrating vocal folds during phonation is of interest, being referred to as glottal area waveform (GAW). Varying camera resolution may influence parameters computed on the GAW and hence hinder the comparability between examinations. This study investigates the influence of spatial camera resolution on quantitative vocal fold vibratory function parameters obtained from the GAW. In total 40 HSV recordings during sustained phonation (20 healthy males and 20 healthy females) were investigated. A clinically used Photron Fastcam MC2 camera with a frame rate of 4000 fps and a spatial resolution of 512×256 pixels was applied. This initial resolution was reduced by pixel averaging to (1) a resolution of 256×128 and (2) to a resolution of 128×64 pixels, yielding three sets of recordings. The GAW was extracted and in total 50 vocal fold vibratory parameters representing different features of the GAW were computed. Statistical analyses using SPSS Statistics, version 21, was performed. 15 Parameters showing strong mathematical dependencies with other parameters were excluded from the main analysis but are given in the Supporting Information. Data analysis revealed clear influence of spatial resolution on GAW parameters. Fundamental period measures and period perturbation measures were the least affected. Amplitude perturbation measures and mechanical measures were most strongly influenced. Most glottal dynamic characteristics and symmetry measures deviated significantly. Most energy perturbation measures changed significantly in males but were mostly unaffected in females. In females 18 of 35 remaining parameters (51%) and in males 22 parameters (63%) changed significantly between spatial resolutions. This work represents the first step in studying the impact of video resolution on quantitative HSV parameters. Clear influences of spatial camera resolution on computed parameters were found. The study results suggest avoiding the use of the most strongly affected parameters. Further, the use of cameras with high resolution is recommended to analyze GAW measures in HSV data.

&lt;br&gt;&lt;b&gt;Introduction:&lt;/b&gt; Advances in computer image analysis have enabled the use of new functional imaging methods in the diagnosis of laryngeal diseases. Particularly interesting techniques of dynamic laryngeal imaging involve High Speed Videoendoscopy (HSV). This still-developed technique allows to overcome the limitations of laryngovideostroboscopy (LVS) and a more detailed analysis of the glottal function based on the image of the actual vibrations of the vocal folds. It also enables the determination of objective coefficients parameterizing phonatory vibrations of the vocal folds.&lt;/br&gt; &lt;br&gt;&lt;b&gt;Aim:&lt;/b&gt; The aim of this pilot study was to evaluate the use of a high-speed videoendoscopy set with laser illumination for the diagnosis of glottic pathology in ENT practice.&lt;/br&gt; &lt;br&gt;&lt;b&gt;Material and methods:&lt;/b&gt; The study included 40 patients who underwent LVS followed by HSV. The modern HSV examination kit - Advanced Larynx Imager System (ALIS), used for the first time in a clinical setting in Poland, is characterized by significantly improved, compared to the previously used high-speed cameras, operational parameters - a light head, the possibility of continuous lighting operation without excessive heating of the head tip, registration of the image in full color scale. Thanks to such modernization, the safety and course of the examination do not differ from laryngoscopy conducted with commonly used recorders. The device owes some of these improvements to a laser illuminator which was used for the first time as the main light source in a high-speed camera. In the study, two cases were selected to present the results of HSV and the analysis of the generated kymograms - a woman with no glottic pathology and a man with a polyp of the right vocal fold. In the first case, the HSV examination compared with the LVS revealed a discrete glottis functional disorder in the form of a tendency to hyperphonation. The patient with an organic lesion had a clearly visible irregularity of vocal fold vibrations, which also allowed to trace mucosal wave disturbances related to its reflection from the pathological structure of the glottis and the formation of a return wave, both on the fold affected by the lesion and, to a lesser extent, contralaterally. The glottic dysfunctions observed in the studied patients were confirmed in the generated kymograms and the graphs of the glottal width waveform (GWW), as well as in the parameters calculated on their basis, assessing the frequency and amplitude of phonatory vibrations.&lt;/br&gt; &lt;br&gt;&lt;b&gt;Conclusions:&lt;/b&gt; The use of high-speed videoendoscopy allows for a much more accurate assessment of the phonatory function of the glottis than in laryngovideostroboscopy. The presented HSV system allows for obtaining high quality kinematic images of the larynx, color fidelity, and contrast. The use of this technology in laryngological practice enables precise structural and functional assessment of the glottis and detection of discrete phonation disorders that elude the techniques used so far.&lt;/br&gt.

The goal of the study was to assess the variability of healthy vocal fold dynamics. Quantitative parameters describing mean regularity and vibration characteristics of vocal fold dynamics along the entire glottal axis were analyzed. Prospective study with 14 young healthy females with no history of voice problems. High-speed videoendoscopy was used to capture vocal fold vibratory characteristics while subjects were producing vowel /i/ with comfortable pitch and loudness over three different recording sessions. The analysis of the data was achieved by using the objective image analysis technique, phonovibrography. The objective analysis of the high-speed video data showed no statistically significant changes in endoscopic voice parameters for normal young female voices. Analysis suggests that endoscopic parameters obtained from phonovibrograms are consistent from day to day in normal voices. Further studies are needed to objectively rate the presence and degree of these parameters in disordered voices.

In this paper, a single speaker tracking mechanism with particle filter is presented for a reliable interaction with robot. To have a smooth tracking result, we use not only audio data but also video data. For sound observation, we use Time Delay Of Arrival (TDOA) between two microphones, and in order to expect high reliability in the likelihood of localization, we apply variable sensitivity values according to the angle of arrival for a triangle arrangement of microphones to calculate the likelihood for audio observation. For video observation, we use face detection routine of OpenCV which is widely available through the Internet. Together with audio and video data, we found out that more reliable source tracking is possible if we apply data fusion mechanism at every possible environment on a robot platform than single information.

The paper proposes a transmission system for Motion-JPEG2000 video in mobile communications. In Motion JPEG2000, video data are compressed in frames according to JPEG2000 part1. In JPEG2000 part1 encoding, an input image is coded in the order of significant sub-bands, which are low-frequency bands. The proposed transmission system therefore assigns each sub-band data suitable weights according to their significance by using orthogonal multi-code CDMA so that bit errors are concentrated in the high-frequency sub-bands which are of little significance on image reconstruction. Concretely speaking, copies of significant sub-band data are made and the copied data are orthogonally multiplexed. The multiplexed data are averaged in the receiver so that the bit errors of significant sub-bands are reduced. On the other hand, the other sub-band data are not copied in order to increase transmission rates. Furthermore, audio data are transmitted with high quality by orthogonal multiplexing. The proposed system adaptively performs the above process for video and audio data. Moreover, these proposed systems are effective in the progressive order of LRCP (layer - resolution level - component - position) and of RLCP (resolution level - layer - component - position) on JPEG2000. Simulation results demonstrate that the proposed system can improve image quality compared with the conventional transmission system under the same BER.

The dissertation aims to achieve two goals. First, it attempts to establish a new theoretical framework—the collaborative scientific conceptual change model, which explicitly attends to social factor and epistemic practices of science, to understand conceptual change. Second, it report the findings of a classroom study to investigate how to apply this theoretical framework to examine the trajectories of collaborative scientific conceptual change in a CSCL environment and provide pedagogical implications. Two simulations were designed to help students make connections between the macroscopic substances and the aperceptual microscopic entities and underlying processes. The reported study was focused on analyzing the aggregated data from all participants and the video and audio data from twenty focal groups’ collaborative activities and the process of their conceptual development in two classroom settings. Mixed quantitative and qualitative analyses were applied to analyze the video/audio data. The results found that, overall participants showed significant improvements from pretest to posttest on system understanding. Group and teacher effect as well as group variability were detected in both students’ posttest performance and their collaborative activities, and variability emerged in group interaction. Multiple data analyses found that attributes of collaborative discourse and epistemic practices made a difference in student learning. Generating warranted claims in discourse as well as the predicting, coordinating theory-evidence, and modifying knowledge in epistemic practices had an impact on student’s conceptual understanding. However, modifying knowledge was found negatively related to students’ learning effect. The case studies show how groups differed in using the computer tools as a medium to conduct collaborative discourse and epistemic practices. Only with certain combination of discourse features and epistemic practices can the group interaction lead to successful convergent understanding. The results of the study imply that the collaborative scientific conceptual change model is an effective framework to study conceptual change and the simulation environment may mediate the development of successful collaborative interactions (including collaborative discourse and epistemic practices) that lead to collaborative scientific conceptual change.

In the paper the features of protocols and the basic methods of organization of multimedia information transfer in telecommunication networks and distributed corporate safety networks are represented. The analysis of formats of basic elements of signals used in standards of multimedia information transfer is given.

Fibre channel is an emerging high-speed digital network technology that combines to make inroads into the avionics arena. The suitability of fibre channel for such applications is largely due to its flexibility in these key areas: Network topologies can be configured in point-to-point, arbitrated loop or switched fabric connections. The physical layer supports either copper or fiber optic implementations with a Bit Error Rate of less than 10<SUP>-12</SUP>. Multiple Classes of Service are available. Multiple Upper Level Protocols are supported. Multiple high speed data rates offer open ended growth paths providing speed negotiation within a single network. Current speeds supported by commercially available hardware are 1 and 2 Gbps providing effective data rates of 100 and 200 MBps respectively. Such networks lend themselves well to the transport of digital video and audio data. This paper summarizes an ANSI standard currently in the final approval cycle of the InterNational Committee for Information Technology Standardization (INCITS). This standard defines a flexible mechanism whereby digital video, audio and ancillary data are systematically packaged for transport over a fibre channel network. The basic mechanism, called a container, houses audio and video content functionally grouped as elements of the container called objects. Featured in this paper is a specific container mapping called Simple Parametric Digital Video (SPDV) developed particularly to address digital video in avionics systems. SPDV provides pixel-based video with associated ancillary data typically sourced by various sensors to be processed and/or distributed in the cockpit for presentation via high-resolution displays. Also highlighted in this paper is a streamlined Upper Level Protocol (ULP) called Frame Header Control Procedure (FHCP) targeted for avionics systems where the functionality of a more complex ULP is not required.

Social media is being employed to build support for social, economic, and political justice (Selander et al. 2016; Vaast et al. 2014) in movements such as occupy Wall Street, violence against women's movement and global sustainability movement. These movements have used social media in ways that goes beyond simple communications. As social media allows people to produce and share user generated content, they enable a certain set of affordances of these technologies (Bharati et al 2015; Bharati et al 2014). The social media affordances, available to both collective and individual actors, translate into capabilities afforded to social movements (Tufecki 2014). Research on connective action has examined the effects of digital action repertoires on interaction and engagement such as in the Tea Party and Occupy movements (Agarwal et al. 2014; Selander et al. 2016). Social media can also facilitate mobilization of movement and participation by new volunteers and oftentimes provides a transnational character by diffusing actions beyond the virtual (Van Laer and Van Aelst 2010). Conversation, an essential part of social movements, shapes “social life by altering individual and collective understandings, by creating and transforming social ties, by generating cultural materials that are then available for subsequent social interchange, and by establishing, obliterating, or shifting commitments on the part of participants&x201D; (Tilly 2002, p. 122). In a personal interaction that involves repeated organized interactions between individuals, typically, leads to shared values and trust. The role of social media technologies in furthering this conversation has to be studied and its' influence on social movement ascertained. A few scholars have started to investigate social media affordances and capabilities, especially focused on discourse, during contentious collective action. Still the research has been limited to studying mechanisms of participation, development of a sense of collective identity, creation of community, and framing of political discourse (Farrell 2012; Garrett 2006). Social media data on social movements can involve impersonal “like&x201D; and “share&x201D; to more engaged conversations. Twitter, Instagram, Facebook, and YouTube offer a wide communicative and discourse reach in networks with text, image, audio, and video data. This social media based big data, consisting mostly of unstructured data, comes in the form of social media posts, digital pictures and videos. The symposium will discuss how formal organizational structures and practices might be integrated with social media capabilities to reinforce and enhance social movement organizations as leaders in social change movements. It will explore the characteristics of social media discourse and assess the dynamics of social movements and, subsequent, impact on real-world protests. The symposium will also demonstrate how discourse analysis can be applied visually in order to understand communication patterns. The panel symposium will focus on theoretical and methodological challenges of social media analytics, big data and social movements. Panelists will engage the audience in an interactive discussion on: 1) Theoretical challenges: a. How and why are social movement recruitment and engagement mechanisms being impacted as a result of social media? b. How do we advance theory on social media and social movements when we are overwhelmed with social media based big data? c. What approach should we undertake if big data analysis contradicts most theories on social media and social movements? d. How do we address the issue of generalizability of social media and social movement research when data collection was limited to one social media platform, albeit involving big data? 2) Methodological challenges: a. What methodological approaches have worked in the analysis of social media, big data and social movements? b. How can discourse analysis be applied visually in order to understand communication patterns evident in social media-based big data? c. How can we employ social media analytics to investigate image and video data? d. What combinations of qualitative and quantitative methodologies be employed for big data and social media analytics in the context of social movements? e. What are the limitations of quantitative data analysis techniques, such as structural equation modeling, because of an extremely large sample size? f. What are the limitations of qualitative data analysis techniques as they become extremely labor intensive and, maybe even, impractical because of big data?

Effects of implant and vocal fold stiffness on voice production after medialization laryngoplasty in an MRI-based vocal fold model

Content Based Video Retrieval (CBVR) is so popular these days, because of the increased utilization of video based analytical systems. Video based analytics is quite effective than image analysis, as a series of actions are captured by the video. This ends up with better decision making ability. The CBVR systems play an important role in boosting the human–computer interaction. This paper presents a multimodal CBVR that takes both the visual and audio information into account for retrieving relevant videos to the user. Two modules are employed by this work to deal with video and audio data. The video data is processed to detect the significant frame from shots and is achieved by Lion Optimization Algorithm (LOA). The features are extracted from the visual data and with respect to the audio data, MHEC and LPCC features are extracted. The extracted features are clustered by Kernelized Fuzzy C Mean (KFCM) algorithm. Finally, the feature database is formed and is utilized in the query matching process during the testing phase. The performance of the proposed work is tested in terms of precision, recall, F-measure and time consumption rates. The proposed CBVR system proves better performance than the existing approaches and is evident through attained results.

Emotions are a crucial aspect of human life and the researchers have tried to build an automatic emotion recognition system that helps to provide important real-world applications. The psychologists have shown that emotions differ across culture, considering this fact, we provide and describe the first audio-visual Arabic emotional dataset which called (AVANEmo). In this work we aim to fill the gap between studies of emotion recognition for Arabic content and other languages by provided an Arabic dataset which is a major and fundamental part of build emotion recognition application. Our dataset contains 3000 clips for video and audio data, and it covers six basic emotional labels (Happy, Sad, Angry, Surprise, Disgust, Neutral). Also, we provide some baseline experiments to measure the primitive performance for automated audio and visual emotion recognition application using the AVANEmo dataset. The best accuracy that we achieved was 54.5% and 57.9% using the audio and visual data respectively. The data will be available for distribution to researchers.