Multimodal Neural Graph Memory Networks for Visual Question Answering

The study intended to explore the effect of different neural network algorithms in the electrocardiogram (ECG) classification of patients with congenital heart disease (CHD). Based on the single convolutional neural network (CNN) ECG algorithm and the recurrent neural network (RNN) ECG algorithm, a multimodal neural network (MNN) ECG algorithm was constructed utilizing the MIT-BIH database as training set and test set. Furthermore, the MNN ECG algorithm was optimized to establish an improved MNN (IMNN) algorithm, which was applied to the diagnosis of CHD patients. The CHD patients admitted between August 2016 and August 2019 were selected for analysis to compare the classification effect and accuracy rate of IMNN, MNN, CNN ECG, and RNN ECG algorithms. It was found that the RNN ECG algorithm had higher classification sensitivity and true positive rate in terms of normal or bundle (NB) branch block beat, supraventricular abnormal (SA) rhythm, abnormal ventricular (AV) beat, and fusion beat (FB) than the CNN ECG algorithm ( P < 0.05 ), and the classification sensitivity and true positive rate of IMNN algorithm in the four aspects were significantly higher than those of MNN algorithm ( P < 0.05 ). The classification accuracy of CNN ECG algorithm and RNN ECG algorithm was above 98%, while that of MNN algorithm and IMNN algorithm was better than that of CNN ECG algorithm and RNN ECG algorithm, and the accuracy rate can reach 98.5% or more. Moreover, the accuracy rate of the IMNN algorithm can reach more than 98%. In conclusion, IMNN not only has a good classification ability in the simulated environment but also performs well in the actual environment, which is worthy of clinical promotion.

AI‐based tools were developed in the existing works, which focused on one type of image data; either CXR's or computerized tomography (CT) scans for COVID‐19 prediction. There is a need for an AI‐based tool that predicts COVID‐19 detection from chest images such as Chest X‐ray (CXR) and CT scans given as inputs. This research gap is considered the core objective of the proposed work. In the proposed work, multimodal CNN architectures were developed based on the parameters and hyperparameters of neural networks. Nine experiments evaluate optimizers, learning rates, and the number of epochs. Based on the experimental results, suitable parameters are fixed for multimodal architecture development for COVID‐19 detection. We have constructed a bespoke convolutional neural network (CNN) architecture named multimodal covid network (MMCOVID‐NET) by varying the number of layers from two to seven, which can predict covid or normal images from both CXR's and CT scans. In the proposed work, we have experimented by constructing 24 models for COVID‐19 prediction. Among them, four models named MMCOVID‐NET‐I, MMCOVID‐NET‐II, MMCOVID‐NET‐III, and MMCOVID‐NET‐IV performed well by producing an accuracy of 100%. We obtained these results from a small dataset. So we repeated these experiments in a larger dataset. We inferred that MMCOVID‐NET‐III outperformed all the state‐of‐the‐art methods by producing an accuracy of 99.75%. The experiments carried out in this work conclude that the parameters and hyperparameters play a vital role in increasing or decreasing the model's performance.

While text classification can classify tweets, assessing whether a tweet is related to an ongoing flood event or not, based on its text, remains difficult. Inclusion of contextual hydrological information could improve the performance of such algorithms. Here, a multilingual multimodal neural network is designed that can effectively use both textual and hydrological information. The classification data was obtained from Twitter using flood-related keywords in English, French, Spanish and Indonesian. Subsequently, hydrological information was extracted from a global precipitation dataset based on the tweet's timestamp and locations mentioned in its text. Three experiments were performed analyzing precision, recall and F1-scores while comparing a neural network that uses hydrological information against a neural network that does not. Results showed that F1-scores improved significantly across all experiments. Most notably, when optimizing for precision the neural network with hydrological information could achieve a precision of 0.91 while the neural network without hydrological information failed to effectively optimize. Moreover, this study shows that including hydrological information can assist in the translation of the classification algorithm to unseen languages.

Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies. A primary challenge in modeling drug response prediction (DRP) with PDXs and neural networks (NNs) is the limited number of drug response samples. We investigate multimodal neural network (MM-Net) and data augmentation for DRP in PDXs. The MM-Net learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs). We explore whether combining WSIs with GE improves predictions as compared with models that use GE alone. We propose two data augmentation methods which allow us training multimodal and unimodal NNs without changing architectures with a single larger dataset: 1) combine single-drug and drug-pair treatments by homogenizing drug representations, and 2) augment drug-pairs which doubles the sample size of all drug-pair samples. Unimodal NNs which use GE are compared to assess the contribution of data augmentation. The NN that uses the original and the augmented drug-pair treatments as well as single-drug treatments outperforms NNs that ignore either the augmented drug-pairs or the single-drug treatments. In assessing the multimodal learning based on the MCC metric, MM-Net outperforms all the baselines. Our results show that data augmentation and integration of histology images with GE can improve prediction performance of drug response in PDXs.

Text visual question answering (TextVQA) targets at answering the question related to texts appearing in the given images, posing more challenges than VQA by requiring a deeper recognition and understanding of various shapes of human-readable scene texts as well as their meanings in different contexts. Existing works on TextVQA suffer from two weaknesses: i) scene texts and non-textual objects are processed separately and independently without considering their mutual interactions during the question understanding and answering process, ii) scene texts are encoded only through word embeddings without taking the corresponding visual appearance features as well as their potential relationships with other non-textual objects in the images into account. To overcome the weakness of existing works, we propose a novel multi-modal contextual graph neural network (MCG) model for TextVQA. The proposed MCG model can capture the relationships between visual features of scene texts and non-textual objects in the given images as well as utilize richer sources of multi-modal features to improve the model performance. In particular, we encode the scene texts into richer features containing textual, visual and positional features, then model the visual relations between scene texts and non-textual objects through a contextual graph neural network. Our extensive experiments on real-world dataset demonstrate the advantages of the proposed MCG model over baseline approaches.

Introduction. The article considers the solution of the problem of detecting fires at technological facilities in automatic mode. To solve this problem, an approach was chosen to create a convolutional neural network capable of operating on a real-time video stream. Aims and Purposes. The aim of this work is to create a neural network capable of detecting flames and smoke in the image from CCTV cameras. The purposes of the work include: selection of the optimal architecture of the neural network in accordance with the latest research in this field; speeding up the chosen architecture using quantization and filter thinning techniques. Methods. Different architectures of convolutional neural networks performing the task of detecting objects in an image are considered. Their performance and quality of work are compared. The YOLOv5 architecture, its target function, training methods and ways of speeding up work are considered. Results and discu ssion. The paper shows the results of training a convolutional neural network of the YOLOv5 architecture for the task of flame and smoke detection, as well as how the results change when applying neural network acceleration methods. It was determined that the use of such acceleration methods such as quantization and filter cleaning can significantly increase up the speed of the neural network, while almost no loss in accuracy of operation. Conclusions. As a result of the conducted work, the architecture of the neural network was determined for performing the task of detecting a fire source. Based on the chosen architecture, a neural network was trained to detect flames and smoke in the image. The speed of its work allows to process video stream in real-time without using graphic accelerator.

An Artificial Intelligence Algorithm that may blow your mind.

We propose a modular Deep Convolutional Neural Network (DCNN) architecture which has the property of block-like design and re-usage of parameters by certain blocks. We leverage networks from the ResNet family as the backbone for our proposed architecture. The proposed network architecture is composed of three primary blocks, i) common block ii) hidden blocks and iii) output block. The common block mainly focuses on learning low-level features which are afterward fed to ρ hidden blocks at a time, where ρ is the branching factor. The hidden blocks are placed in a row-wise manner and replicated ρ times depending on the computational availability. The output block takes features as an input from the common block and all the hidden blocks during both training and testing phases. In addition, each of the hidden blocks together with the common block is a stand-alone neural network, which implies that we have ρ neural networks coupled in a single framework. During the training phase, in every single epoch, each of the stand-alone neural networks (i.e. common block + hidden block) is trained in Round Robin fashion followed by fine-tuning the whole network graph. Inference can be performed using either the whole network graph or just any sub-graph of it. To substantiate the effectiveness of our proposed neural architecture we performed experiment on CIFAR-10 and CIFAR-100 datasets.

The variation in the distributions of recorded data between individuals leads to low classification accuracy. To address this issue, we introduce a multimodal fusion convolutional neural network (MFCNN). This network extracts common information from surface electromyography (sEMG) and accelerometer signals of different subjects using a two-stream convolutional neural network (CNN). To enhance the classification accuracy of a particular subject, a fine-tuning approach was implemented. The performance of the proposed method was assessed in four different scenarios, which include intersubject classification, intersubject classification when training data from multiple subjects, fine-tuned intersubject classification, and fine-tuned intersubject classification when training data from multiple subjects. The results demonstrate that in the intersubject scenario, when multiple subjects are available for training, the MFCNN achieves higher classification accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${p} &lt; 0.05$ </tex-math></inline-formula> ) than other neural networks and support vector machines (SVMs) that use sEMG signals [neural network (NN) and SVM], accelerometer signals (accNN and accSVM), sEMG and accelerometer signals [multimodal fusion nerual network (MFNN) and multimodal fusion support vector machine (MFSVM)] as inputs, as well as a CNN that uses sEMG signals as input after fine-tuning. Furthermore, compared with an MFCNN model trained with data from a single subject and an accCNN model trained with data from a single subject or multiple subjects, an MFCNN trained with multiple subjects demonstrated better performance on new subjects after fine-tuning ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${p} &lt; 0.05$ </tex-math></inline-formula> ). This method can learn common features among different subjects and improve the performance of classification among subjects. Our proposed method demonstrates the innovation of using a multimodal fusion approach and two-stream CNN to improve intersubject classification accuracy in upper limb movements.

When using traditional knowledge retrieval algorithms to analyze whether the feature input of words in multi-modal natural language library is symmetrical, the symmetry of words cannot be analyzed, resulting in inaccurate analysis results. A feature input symmetric algorithm of multi-modal natural language library based on BP (back propagation) neural network is proposed in this paper. A Chinese abstract generation method based on multi-modal neural network is used to extract Chinese abstracts from images in multi-modal natural language library. The Word Sense Disambiguation (WSD) Model is constructed by the BP neural network. After the word or text disambiguation is performed on the Chinese abstract in the multi-modal natural language library, the feature input symmetry problem in the multi-modal natural language library is analyzed according to the sentence similarity. The experimental results show that the proposed algorithm can effectively analyze the eigenvalue symmetry problem of the multi-modal natural language library. The maximum error rate of the analysis algorithm is 7%, the growth rate of the analysis speed is up to 50%, and the average analysis time is 540.56 s. It has the advantages of small error and high efficiency.

Wide-field small-aperture telescopes are generally used for transient observations in the optical band, as they can effectively capture images with moderate spatial resolution and depth in a cost-effective way. However, fluctuations in observational conditions and noise from various sources can result in numerous false optical transients, which significantly impede the timely detection of important time-domain astronomical events. This paper introduces a framework aimed at the rapid and efficient identification of optical transients. The proposed framework features a fast image-preprocessing pipeline utilizing graphics processing unit computing to identify transient candidates, alongside a multimodal neural network for obtaining transients from candidates according to not only image features but also properties of observation systems and celestial objects. Additionally, it includes a machine learning operations framework for managing both the neural networks and the image-preprocessing pipeline. Our framework not only automates data processing but also optimizes the parameters of the entire pipeline to yield satisfactory results. We have evaluated the performance of our framework using data obtained from the Ground-based Wide-Angle Camera Array (or GWAC). The results indicate that our framework significantly enhances the detection efficiency of optical transients. Our framework is provided as a Docker image, facilitating easy download and implementation in various projects.

College students are under increasing competition pressure, which has a negative impact on their mental health, as the pace of learning and life accelerates, as well as the increasingly difficult employment situation. As a result, emphasizing the importance of college students’ mental health and fully addressing it has become a top priority in the work of colleges and universities. However, some students and even teachers are currently unconcerned about mental illness, making it difficult for students with psychological abnormalities to receive timely detection and effective treatment. As a result, it is the responsibility of student management for colleges and universities to identify and intervene early in the mental health problems of college students. Through the use of multimodal data and neural network models, it is now possible to evaluate and predict the mental state of college students in real time, thanks to the advancement of intelligent technology. Therefore, a novel multimodal neural network model is proposed in this paper. Our model is divided into two branches in particular. The traditional mental health assessment and prediction algorithm, which is based on the improved BP neural network and the International Mental Health Scale SCL‐90, is one of the branches. Given how difficult it is to meet the requirements for the accuracy of college students’ mental health assessments using this method, our other branch is computer vision‐based facial emotion recognition of college students, which is used to aid in the evaluation of mental health assessments. Our model demonstrates competitive performance through simulation and comparative experiments.

Deep leaning in food safety and authenticity detection: An integrative review and future prospects

The surface-water hydrographs of rivers exhibit large variations due to many natural phenomena. One of the most commonly used approachs for interpolating and extending streamflow records is to fit observed data with an analytic power model. However, such analytic models may not adequately represent the flow process, because they are based on many simplifying assumptions about the natural phenomena that influence the river flow. This paper demonstrates how a neural network can be used as an adaptive model synthesizer as well as a predictor. Issues such as selecting an appropriate neural network architecture and a correct training algorithm as well as presenting data to neural networks are addressed using a constructive algorithm called the cascade-correlation algorithm. The neural-network approach is applied to the flow prediction of the Huron River at the Dexter sampling station, near Ann Arbor, Mich. Empirical comparisons are performed between the predictive capability of the neural network models and the most commonly used analytic nonlinear power model in terms of accuracy and convenience of use. Our preliminary results are quite encouraging. An analysis performed on the structure of the networks developed by the cascade-correlation algorithm shows that the neural networks are capable of adapting their complexity to match changes in the flow history and that the models developed by the neural-network approach are more complex than the power model.

In 2020, COVID-19 underscored the vulnerabilities of the healthcare system. Supply lines and hospital capacities presented unrealized complications for dealing with the virus's sudden outbreak. Being able to anticipate an influx of COVID-19 patients would allow for targeted responses by healthcare providers and policy makers. In this work we present a method for predicting COVID-19 case counts on a county-level granularity using neural networks. Utilizing multimodal network architectures, incorporating convolutional and LSTM layers, we were able to outperform a feed forward estimation of COVID-19 case counts. We test our network's performance on a variety of historical and geographical inputs, utilizing static anthropogeographic inputs, and through retraining the network on a span of historical positions. Our results indicate the models well are suited to forecast the spread of COVID-19 on a county-level basis, but our model is best utilized for targeted local applications preventing its overall portability on a national scale.