Deep Multi-Modal Network Based Automated Depression Severity Estimation

Local line directional pattern for palmprint recognition

In this paper, local directional position pattern (LDPP) and local ternary pattern (LTP) are selected for facial recognition method which are having many advantages over previous techniques like local binary pattern (LBP) and local directional pattern (LDP). The selected techniques of LDPP and LTP are estrangement in their algorithms which help solely to extract features out of an image. LDPP is a revised form of LDP. In a typical LDP, only the top edge direction was taken into consideration, but strength sign of the pixel was not considered which may result in same code for opposite kind of edge pixel. This snag is overcome by LDPP which is further concatenated with LTP for better feature extraction. Once features are extracted they are trained using deep belief network. In the experimental work 10 images of each expression i.e. angry, surprise, disgust, neutral, sad, smile are selected. LDPP and LTP are concatenated followed by principal component analysis (PCA) and general discriminant analysis (GDA). Further for training, Deep Belief Network (DBN) is used which eventually increases the recognition rate and achieve accuracy of 95.3% which was 89.3% without concatenating.

In the paper the features of protocols and the basic methods of organization of multimedia information transfer in telecommunication networks and distributed corporate safety networks are represented. The analysis of formats of basic elements of signals used in standards of multimedia information transfer is given.

The paper proposes a transmission system for Motion-JPEG2000 video in mobile communications. In Motion JPEG2000, video data are compressed in frames according to JPEG2000 part1. In JPEG2000 part1 encoding, an input image is coded in the order of significant sub-bands, which are low-frequency bands. The proposed transmission system therefore assigns each sub-band data suitable weights according to their significance by using orthogonal multi-code CDMA so that bit errors are concentrated in the high-frequency sub-bands which are of little significance on image reconstruction. Concretely speaking, copies of significant sub-band data are made and the copied data are orthogonally multiplexed. The multiplexed data are averaged in the receiver so that the bit errors of significant sub-bands are reduced. On the other hand, the other sub-band data are not copied in order to increase transmission rates. Furthermore, audio data are transmitted with high quality by orthogonal multiplexing. The proposed system adaptively performs the above process for video and audio data. Moreover, these proposed systems are effective in the progressive order of LRCP (layer - resolution level - component - position) and of RLCP (resolution level - layer - component - position) on JPEG2000. Simulation results demonstrate that the proposed system can improve image quality compared with the conventional transmission system under the same BER.

In this paper, a single speaker tracking mechanism with particle filter is presented for a reliable interaction with robot. To have a smooth tracking result, we use not only audio data but also video data. For sound observation, we use Time Delay Of Arrival (TDOA) between two microphones, and in order to expect high reliability in the likelihood of localization, we apply variable sensitivity values according to the angle of arrival for a triangle arrangement of microphones to calculate the likelihood for audio observation. For video observation, we use face detection routine of OpenCV which is widely available through the Internet. Together with audio and video data, we found out that more reliable source tracking is possible if we apply data fusion mechanism at every possible environment on a robot platform than single information.

Weed identification techniques that are effective can lower the cost of weed control while also improving crop quality and output. Herbicides are currently the most frequent method of weed management. In order to deal with the issues of weed man-agement in agriculture, accurate crop and weed discrimination is essential. This work deals with the performance comparison of different classification algorithms namely Convolutional Neural Network(CNN), Graph Convolutional Networks(GCN), k-Nearest Neighbor(k-NN), and Support Vector Machine(SVM). Feature extraction algorithms, Local Directional Pattern(LDP) and Local Directional Relation Pattern(LDRP) are used alongside SVM and kNN. For data expansion, Generative Adversarial Network(GAN) was used, which generates synthetic samples of images from the dataset. The result of the study shows that CNN gives the maximum accuracy of 90.58% in ‘Crop and weed detection’ dataset when compared with other classification models used for investigation in our paper.

Purpose: High-speed videoendoscopy (HSV) is an emerging, but barely used, endoscopy technique in the clinic to assess and diagnose voice disorders because of the lack of dedicated software to analyze the data. HSV allows to quantify the vocal fold oscillations by segmenting the glottal area. This challenging task has been tackled by various studies; however, the proposed approaches are mostly limited and not suitable for daily clinical routine.Method: We developed a user-friendly software in C# that allows the editing, motion correction, segmentation, and quantitative analysis of HSV data. We further provide pretrained deep neural networks for fully automatic glottis segmentation.Results: We freely provide our software Glottis Analysis Tools (GAT). Using GAT, we provide a general threshold-based region growing platform that enables the user to analyze data from various sources, such as in vivo recordings, ex vivo recordings, and high-speed footage of artificial vocal folds. Additionally, especially for in vivo recordings, we provide three robust neural networks at various speed and quality settings to allow a fully automatic glottis segmentation needed for application by untrained personnel. GAT further evaluates video and audio data in parallel and is able to extract various features from the video data, among others the glottal area waveform, that is, the changing glottal area over time. In total, GAT provides 79 unique quantitative analysis parameters for video- and audio-based signals. Many of these parameters have already been shown to reflect voice disorders, highlighting the clinical importance and usefulness of the GAT software.Conclusion: GAT is a unique tool to process HSV and audio data to determine quantitative, clinically relevant parameters for research, diagnosis, and treatment of laryngeal disorders.Supplemental Material S1. Supplemental figures: Figure S1. Neural network training metrics.Figure S2. Neural network evaluation.Figure S3. Example segmentations using different backbones from various source data.Figure S4. Audio analysis platform.Supplemental Material S2. Endoscopic footage with semi-automatic and fully automatic segmentation methods, multiple examples.Supplemental Material S3. GAT manual. Kist, A. M., Gomez, P., Dubrovskiy, D., Schlegel, P., Kunduk, M., Echternach, M., Patel, R., Semmler, M., Bohr, C., Durr, S., Schutzenberger, A., & Dollinger, M. (2021). A deep learning enhanced novel software tool for laryngeal dynamics analysis. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2021_JSLHR-20-00498

Content Based Video Retrieval (CBVR) is so popular these days, because of the increased utilization of video based analytical systems. Video based analytics is quite effective than image analysis, as a series of actions are captured by the video. This ends up with better decision making ability. The CBVR systems play an important role in boosting the human–computer interaction. This paper presents a multimodal CBVR that takes both the visual and audio information into account for retrieving relevant videos to the user. Two modules are employed by this work to deal with video and audio data. The video data is processed to detect the significant frame from shots and is achieved by Lion Optimization Algorithm (LOA). The features are extracted from the visual data and with respect to the audio data, MHEC and LPCC features are extracted. The extracted features are clustered by Kernelized Fuzzy C Mean (KFCM) algorithm. Finally, the feature database is formed and is utilized in the query matching process during the testing phase. The performance of the proposed work is tested in terms of precision, recall, F-measure and time consumption rates. The proposed CBVR system proves better performance than the existing approaches and is evident through attained results.

Fibre channel is an emerging high-speed digital network technology that combines to make inroads into the avionics arena. The suitability of fibre channel for such applications is largely due to its flexibility in these key areas: Network topologies can be configured in point-to-point, arbitrated loop or switched fabric connections. The physical layer supports either copper or fiber optic implementations with a Bit Error Rate of less than 10<SUP>-12</SUP>. Multiple Classes of Service are available. Multiple Upper Level Protocols are supported. Multiple high speed data rates offer open ended growth paths providing speed negotiation within a single network. Current speeds supported by commercially available hardware are 1 and 2 Gbps providing effective data rates of 100 and 200 MBps respectively. Such networks lend themselves well to the transport of digital video and audio data. This paper summarizes an ANSI standard currently in the final approval cycle of the InterNational Committee for Information Technology Standardization (INCITS). This standard defines a flexible mechanism whereby digital video, audio and ancillary data are systematically packaged for transport over a fibre channel network. The basic mechanism, called a container, houses audio and video content functionally grouped as elements of the container called objects. Featured in this paper is a specific container mapping called Simple Parametric Digital Video (SPDV) developed particularly to address digital video in avionics systems. SPDV provides pixel-based video with associated ancillary data typically sourced by various sensors to be processed and/or distributed in the cockpit for presentation via high-resolution displays. Also highlighted in this paper is a streamlined Upper Level Protocol (ULP) called Frame Header Control Procedure (FHCP) targeted for avionics systems where the functionality of a more complex ULP is not required.

Recent advances in artificial intelligence combined with behavioral sciences have led to the development of cutting-edge tools for recognizing human emotions based on text, video, audio, and physiological data. However, these data sources are expensive, intrusive, and regulated, unlike plants, which have been shown to be sensitive to human steps and sounds. A methodology to use plants as human emotion detectors is proposed. Electrical signals from plants were tracked and labeled based on video data. The labeled data were then used for classification., and the MLP, biLSTM, MFCC-CNN, MFCC-ResNet, Random Forest, 1-Dimensional CNN, and biLSTM (without windowing) models were set using a grid search algorithm with cross-validation. Finally, the best-parameterized models were trained and used on the test set for classification. The performance of this methodology was measured via a case study with 54 participants who were watching an emotionally charged video; as ground truth, their facial emotions were simultaneously measured using facial emotion analysis. The Random Forest model shows the best performance, particularly in recognizing high-arousal emotions, achieving an overall weighted accuracy of 55.2% and demonstrating high weighted recall in emotions such as fear (61.0%) and happiness (60.4%). The MFCC-ResNet model offers decently balanced results, with AccuracyMFCC-ResNet=0.318 and RecallMFCC-ResNet=0.324. Regarding the MFCC-ResNet model, fear and anger were recognized with 75% and 50% recall, respectively. Thus, using plants as an emotion recognition tool seems worth investigating, addressing both cost and privacy concerns.

Facial expression recognition from video data is considered to be a very challenging task in the research areas of computer vision, image processing, and pattern recognition. A novel approach is pr...

Face recognition and facial expression recognition using local patterns is the order of the day. Local directional number pattern (LDNP) is one of the prominent descriptor for face recognition. LDNP assigns a 3 bit code for each pixel in the image. The resultant LDNP labeled image is divided into regions to form histogram based descriptor. The histogram bins of all the regions are concatenated to form the final descriptor. In contrast to LDNP, a dimensionality reduced local directional number pattern (DR-LDNP) is proposed in this paper. The proposed descriptor computes single code for each block. This is done by X-ORing of the LDNP codes obtained in a single block. During the process, restructuring of the patterns is done by slightly modifying the LDNP coding pattern constraints. The resultant DR-LDNP descriptor outperforms the existing methods. The experimentation is carried out on standard databases like FERET, YALE, ORL, Cohn–Kannade and JAFFEE and obtained good recognition rates compared to other methods.

Emotions are a crucial aspect of human life and the researchers have tried to build an automatic emotion recognition system that helps to provide important real-world applications. The psychologists have shown that emotions differ across culture, considering this fact, we provide and describe the first audio-visual Arabic emotional dataset which called (AVANEmo). In this work we aim to fill the gap between studies of emotion recognition for Arabic content and other languages by provided an Arabic dataset which is a major and fundamental part of build emotion recognition application. Our dataset contains 3000 clips for video and audio data, and it covers six basic emotional labels (Happy, Sad, Angry, Surprise, Disgust, Neutral). Also, we provide some baseline experiments to measure the primitive performance for automated audio and visual emotion recognition application using the AVANEmo dataset. The best accuracy that we achieved was 54.5% and 57.9% using the audio and visual data respectively. The data will be available for distribution to researchers.

With the widespread application of deep learning methods, multimodal techniques have also achieved rapid development. Since single-modal speech recognition may affect the accuracy of recognition results in noisy environments, multimodal fusion recognition gradually replaces the traditional single-modal recognition methods. In this paper, we mainly strengthen and pre-process audio and video data first, and use LSTM recurrent neural network for deep feature extraction of audio and video streams, which effectively solves the problem of long-term forgetting of general neural networks. The audio and video feature vectors are then fused by a fully connected neural network with linear connections. Compared with the speech recognition technique alone, this audiovisual fusion recognition method has a better recognition effect in the case of noise interference. Compared with the traditional audiovisual recognition method, the model simplifies the recognition work. Recognition experiments on the LRS2-BBC dataset show that the recognition accuracy of this method improves to a certain extent over that of other methods in a clean environment and greatly improves in noisy conditions.

BackgroundRespectful, equitable, and supportive health care team interactions with patients and their companions are a key part of quality health care services. Despite the importance of communication and other aspects of engagement during inpatient postpartum care, little is known about how care is experienced during this period from families’ perspectives.MethodsThis study collected video and audio data with 15 birthing families (n = 9 English-speaking and n = 6 Spanish-speaking) and their health care team members during inpatient postpartum care in a southeastern United States academic medical center. This analysis quantifies health care team member presence in family hospital rooms, assesses linguistic appropriateness of health care team member verbal communication, and describes birthing parent and companion verbal reactions to health care team member interactions. A behavioral taxonomy was applied to identify these codes in the 12 h prior to hospital discharge. Additionally, we transcribed the birthing parent and companion verbal reactions for 10-minutes each time the health care team member departed their room. This content was inductively coded to identify topics and develop themes.ResultsA total of 160 h of video and audio data were coded across 15 participating families for this analysis. There were 19.9 h of missing data in the 12 h prior to hospital discharge due to equipment being turned off across five participants. At least one health care team member was present in the postnatal unit rooms within the observation period in 200 instances (median 13 times, range 5 to 19 times per participating family). Communication with Spanish-speaking birthing parents was linguistically appropriate for between 20.0% and 75.0% of interactions. Following health care team member presence, birthing parent and companion reactions were indeterminate 1 time, no verbal reaction 107 times, positive 25 times, confused 38 times, and negative 52 times. Many parents expressed more than one reaction. Reaction topics included clarity of information provided, postpartum pain management, hospital discharge coordination, health care team member conduct, and access to supplies.ConclusionThere are strengths in postpartum patient engagement as well as areas for health care team improvement. More appropriate communication including setting of clearer expectations surrounding various aspects of services on the postnatal unit may improve patient experiences. As a part of this improved quality of inpatient postpartum care, consistent utilization of interpretation services could decrease patient confusion, enable shared decision-making, and promote positive patient-provider relationships.