Improving Protein Gamma-Turn Prediction Using Inception Capsule Networks

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper, we use another version of the techniques to extend the results of [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] to the case where the entries of the data matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, justifies the mean field approximation in the infinite width limit for the deep untrained neural networks and the property of the macroscopic universality of random matrix theory in this case.

Capsule Networks (CapsNets) is a great approach for understanding data in the field of computer vision. CapsNets allow a deeper understanding of images compared to the traditional Convolutional Neural Networks. The first test for CapsNet was in digits recognition on the ‘MNIST’ dataset, where it successfully achieved high accuracy. CapsNets are reliable at deciphering overlapping digits. Deep Capsule Networks achieved state-of-the-art accuracy in CIFAR10 which isn’t achieved by shallow capsule networks. Despite all these accomplishments, Deep Capsule Networks are very slow due to the ‘Dynamic Routing’ algorithm. In this paper, Fast Embedded Capsule Network and Deep Fast Embedded Capsule Network are introduced, representing novel capsule network architectures that uses 1D convolution based dynamic routing with a fast element-wise multiplication transformation process. These architectures not only compete with the state-of-the-art methods in terms of accuracy in the capsule domain, but also excels in terms of speed, and reduced complexity. This is shown by the 58% reduction in the number of trainable parameters and 64% reduction in the average epoch time in the training process. Experimental results shows excellent and verified properties.

This paper introduces an experimental study of the heart disease datasets characteristics impact on the performance of classification algorithms in the aim of identifying the best algorithm for each dataset under its characteristics. The performance of five machine learning algorithms (logistic regression (LR), K-Nearest Neighbor (KNN), Decision tree (DT), Random Forest (RF), and support vector machine (SVM)), single layer neural network (ANN), and deep neural network (DNN), has been evaluated using five heart disease datasets under four data complexity measurement: number of samples (dataset size), number of features (dimension of dataset), Data sparsity measures, and correlation of features. All datasets have been processed and normalized then the mutual information-based feature selection method was used to solve the overfitting problem. The results show that in general, the machine learning especially the Random Forest algorithm achieves high classification accuracy than deep learning network. In other hand, the high sparsity and less mutual information of dataset has large impact on degradation of the performance of classification algorithms than other characteristics of data.

Finding methods to accurately predict the final infarct volumes for acute ischemic stroke patients with full or no recanalization would significantly help to evaluate the potential benefits of thrombolytic therapy. We proposed such a method by constructing a model of ensemble deep learning and machine learning using diffusion-weighted imaging (DWI) only. The proposed prediction model (named AUNet) combines an adaptive linear ensemble model (ALEM) of machine learning and a deep U-Net network with an accelerated non-local module (U-NL-Net) to learn voxel-wise and spatial features, respectively. Of 40 patients with acute ischemic stroke who received thrombolytic therapy, 17 were fully recanalized, 14 were not recanalized, and nine were partially recanalized. The AUNet was separately trained for full recanalization conditions (AUNetR) and no recanalization (AUNetN) as the best and worst outcomes of thrombolysis, respectively. AUNet performed significantly better in predicting the final infarct volumes in both the recanalization and non-recanalization conditions [area under the receiver operating characteristic curve (AUC) =0.898±0.022, recanalization; AUC =0.875±0.036, non-recanalization: Matthew's correlation coefficient (MCC) =0.863±0.033, recanalization; MCC =0.851±0.025, non-recanalization] than the fixed-thresholding method (AUC =0.776±0.021, P<0.0001, recanalization; AUC =0.692±0.023, P<0.0001, non-recanalization: MCC =0.742±0.035, recanalization; MCC =0.671±0.024, non-recanalization), the logistic regression method (AUC =0.797±0.023, P<0.003, recanalization; AUC =0.751±0.030, P<0.003, non-recanalization: MCC =0.762±0.035, recanalization; MCC =0.730±0.031, non-recanalization), and a recently developed convolutional neural network (AUC =0.814±0.013, P<0.003, recanalization; AUC =0.781±0.027, P<0.003, non-recanalization: MCC =792±0.022, recanalization; MCC =0.758±0.016, non-recanalization). The potential benefit of thrombolysis calculated from AUNetR and AUNetN showed large individual differences (from 12.81% to 239.73%). AUNet improved predictive accuracy over current state-of-the-art methods. More importantly, the accurate prediction of infarct volumes under different recanalization conditions may provide benefitial information for physicians in selecting suitable patients for thrombolytic therapy.

The capsule network is a novel architecture to encode feature attributes and spatial relationships of an image. By using the dynamic routing (DR) algorithm, a capsule network (CapsNet) model can be trained. However, the original three-layer CapsNet with the DR algorithm performs poorly on complex data sets, such as FashionMNIST, CIFAR-10, and CIFAR-100. This deficiency limits the wider application of capsule networks. In this article, we propose a deep capsule network model combined with a U-Net preprocessing module (DCN-UN). Local connection and weight-sharing strategies are adopted from convolutional neural networks to design a convolutional capsule layer in the DCN-UN model. This allows considerably reducing the number of parameters in the network model. Moreover, a greedy strategy is incorporated into the design of a mask DR (MDR) algorithm to improve the performance of network models. DCN-UN requires up to five times fewer parameters compared with the original CapsNet and other CapsNet-based models. The performance improvement achieved by the DCN-UN model with the MDR algorithm over the original CapsNet model with the DR algorithm is approximately 12% and 17% on the CIFAR-10 and CIFAR-100 data sets, respectively. The experimental results confirm that the proposed DCN-UN model allows preserving advantages of image reconstruction and equivariance mechanism in capsule networks. Moreover, an efficient initialization method is explored to enhance training stability and avoid gradient explosion.

Numerous methods for predicting γ-turns in proteins have been developed. However, the results they generally provided are not very good, with a Matthews correlation coefficient (MCC)≤0.18. Here, an attempt has been made to develop a method to improve the accuracy of γ-turn prediction. First, we employ the geometric mean metric as optimal criterion to evaluate the performance of support vector machine for the highly imbalanced γ-turn dataset. This metric tries to maximize both the sensitivity and the specificity while keeping them balanced. Second, a predictor to generate protein shape string by structure alignment against the protein structure database has been designed and the predicted shape string is introduced as new variable for γ-turn prediction. Based on this perception, we have developed a new method for γ-turn prediction. After training and testing the benchmark dataset of 320 non-homologous protein chains using a fivefold cross-validation technique, the present method achieves excellent performance. The overall prediction accuracy Qtotal can achieve 92.2% and the MCC is 0.38, which outperform the existing γ-turn prediction methods. Our results indicate that the protein shape string is useful for predicting protein tight turns and it is reasonable to use the dihedral angle information as a variable for machine learning to predict protein folding. The dataset used in this work and the software to generate predicted shape string from structure database can be obtained from anonymous ftp site ftp://cheminfo.tongji.edu.cn/GammaTurnPrediction/ freely.

Cancer is now the leading causes of death. During treatment, if an abnormal mass is detected by routine mammography after histological examination, a biopsy will be performed for further diagnosis. However, the time required to review and evaluate the biopsy slide is very long. An effective algorithm that can identify cancer tissues and reduce the rate of misdiagnosis is urgently needed. The current research methods mainly include: Capsule Network, and Deep Convolutional Neural Networks (CNNs) used in image classification, etc. This paper mainly uses Capsule Network, CNN, CNN trained on Histogram of Oriented Gradient (HOG) images, and Grey Level Co-occurrence Matrix (GLCM) to classify benign and malignant images of the tumor micro-biopsy images. The first step is to download the breast tumor micro-biopsy images from Breakhis. The second step is to preprocess the images, mainly image enhancement, augmentation, and HOG feature extraction. In the third step, the processed image data is used to train CNN and Capsule Network. The last step is to visualize the accuracy of model training and use confusion matrix to calculate the accuracy of image classification, recall rate, and so on. The results show that the accuracy of CNN with the help of transfer learning can reach 98.32%, the accuracy of CNN trained with HOG image is only 82.99%, the accuracy of CNN trained with GLCM is 92.79%, and the accuracy of capsule network can reach 92.39%.

Extracting and inserting knowledge into stacked denoising auto-encoders

Cyberbullying is the use of information technology networks by individuals’ to humiliate, tease, embarrass, taunt, defame and disparage a target without any face-to-face contact. Social media is the 'virtual playground' used by bullies with the upsurge of social networking sites such as Facebook, Instagram, YouTube and Twitter. It is critical to implement models and systems for automatic detection and resolution of bullying content available online as the ramifications can lead to a societal epidemic. This paper presents a deep neural model for cyberbullying detection in three different modalities of social data, namely textual, visual and info-graphic (text embedded along with an image). The all-in-one architecture, CapsNet–ConvNet, consists of a capsule network (CapsNet) deep neural network with dynamic routing for predicting the textual bullying content and a convolution neural network (ConvNet) for predicting the visual bullying content. The info-graphic content is discretized by separating text from the image using Google Lens of Google Photos app. The perceptron-based decision-level late fusion strategy for multimodal learning is used to dynamically combine the predictions of discrete modalities and output the final category as bullying or non-bullying type. Experimental evaluation is done on a mix-modal dataset which contains 10,000 comments and posts scrapped from YouTube, Instagram and Twitter. The proposed model achieves a superlative performance with the AUC–ROC of 0.98.

2'-O-methylation (2OM) of ribose is a widespread RNA modification that significantly impacts RNA stability, structure, and function. Accurately predicting 2OM sites is crucial for understanding RNA's biological functions and related pathologies. Traditional detection methods pose challenges such as resource intensiveness, potential RNA sample damage, and high costs. However, recent advancements in machine learning, particularly deep learning techniques, offer rapid and cost-effective prediction solutions. In this study, we introduce DeepR2OM, a novel method integrating feature selection and deep learning for 2OM sites prediction. DeepR2OM encodes sequences using eight RNA descriptors, employs feature selection algorithms to reduce dimensions, and then utilizes a deep learning network for training. After evaluating various deep learning architectures, we selected Convolutional Neural Network (CNN), Multi-Head Self-Attention mechanism, and Deep Neural Network (DNN) as our final prediction models. Experimental results demonstrate DeepR2OM's effectiveness, achieving 87.1% accuracy (ACC), 85.5% recall rate (Recall), 87.9% precision (PRE), and a Matthews correlation coefficient (MCC) of 75.7% on an independent test set. This tool serves as a valuable resource for exploring the functional and bioinformatic aspects of 2OM sites.

Long non-coding RNA(lncRNA) is one of the non-coding RNAs longer than 200 nucleotides and it has no protein encoding function. LncRNA plays a key role in many biological processes. Studying the RNA-binding protein (RBP) binding sites on the lncRNA chain helps to reveal epigenetic and post-transcriptional mechanisms, to explore the physiological and pathological processes of cancer, and to discover new therapeutic breakthroughs. To improve the recognition rate of RBP binding sites and reduce the experimental time and cost, many calculation methods based on domain knowledge to predict RBP binding sites have emerged. However, these prediction methods are independent of nucleotides and do not take into account nucleotide statistics. In this paper, we use a high-order statistical-based encoding scheme, then the encoded lncRNA sequences are fed into a hybrid deep learning architecture named AC-Caps. It consists of a joint processing layer(composed of attention mechanism and convolutional neural network) and a capsule network. The AC-Caps model was evaluated using 31 independent experimental data sets from 12 lncRNA-binding proteins. In experiments, our method achieves excellent performance, with an average area under the curve (AUC) of 0.967 and an average accuracy (ACC) of 92.5%, which are 0.014, 2.3%, 0.261, 28.9%, 0.189, and 21.8% higher than HOCCNNLB, iDeepS, and DeepBind, respectively. The results show that the AC-Caps method can reliably process the large-scale RBP binding site data on the lncRNA chain, and the prediction performance is better than existing deep-learning models. The source code of AC-Caps and the datasets used in this paper are available at https://github.com/JinmiaoS/AC-Caps .

Purpose: The aim of the article is to develop an algorithm for classifying cracks in the analyzed images using modern methods of deep machine learning and transfer learning based on pretrained convolutional neural network - Inception-ResNet-v2. Design/Methodology/Approach: Transfer learning based on the pretrained convolutional neural network was used to categorize the images. The fully conected layer of the Inception-ResNet-v2 network has been modified. The last layer was trained using a two-class (binary) linear SVM (Support Vector Machine). In total, 20,000 training cases (images) were used to train the fully connected layer within transfer learning process. The research analyzed the possibility of using the deep neural networks for quick and fully automatic identification of cracks / defects on the surface of analyzed parts. Findings: The results indicate that pretrained convolutional neural network using SVM to train a fully connected layer is a very effective solution for visual crack / fault detection. In the analyzed model, a positive classification was obtained at the level of 99.89%. Practical Implications: The model presented in the article can be used in quality control carried out by process monitoring. An effective model for identifying defective parts can be used in both logistics and production processes. Originality/Value: A novelty is the use of a freely available, deep neural network trained to classify 1000 categories of various images for binary categorization of faults (cracks). The algorithm was adjusted by replacing the primary, 1000-output fully connected layer in the Inception-ResNet-v2 network with a binary layer (2 categories). The fully connected layer has been trained using the classification version of the popular SVM learner, but thanks to the combination of this layer with the sophisticated fearure extraction ability of the pre-trained Inception-ResNet-v2 deep network, the resulting predictive model enables the classification of defects with a very high level of accuracy.

Accurate pollutant prediction is essential in fields such as meteorology, meteorological disasters, and climate change studies. In this study, long short-term memory (LSTM) and deep neural network (DNN) models were applied to six pollutants and comprehensive air-quality index (CAI) predictions from 2015 to 2020 in Korea. In addition, we used the network method to find the best data sources that provide factors affecting comprehensive air-quality index behaviors. This study had two steps: (1) predicting the six pollutants, including fine dust (PM10), fine particulate matter (PM2.5), ozone (O3), sulfurous acid gas (SO2), nitrogen dioxide (NO2), and carbon monoxide (CO) using the LSTM model; (2) forecasting the CAI using the six predicted pollutants in the first step as predictors of DNNs. The predictive ability of each model for the six pollutants and CAI prediction was evaluated by comparing it with the observed air-quality data. This study showed that combining a DNN model with the network method provided a high predictive power, and this combination could be a remarkable strength in CAI prediction. As the need for disaster management increases, it is anticipated that the LSTM and DNN models with the network method have ample potential to track the dynamics of air pollution behaviors.

A point cloud deep neural network metamodel method for aerodynamic prediction

As a transmission component, the gear has been obtained widespread attention. The remaining useful life (RUL) prediction of gear is critical to the prognostics health management (PHM) of gear transmission systems. The digital twin (DT) provides support for gear RUL prediction with the advantages of rich health information data and accurate health indicators (HI). This paper reviews digital twin-driven RUL prediction methods for gear performance degradation, from the view of digital twin-driven physical model-based and virtual model-based prediction method. From the view of the physical model-based one, it includes a prediction model based on gear crack, gear fatigue, gear surface scratch, gear tooth breakage, and gear permanent deformation. From the view of the digital twin-driven virtual model-based one, it includes non-deep learning methods and deep learning methods. Non-deep learning methods include the wiener process, gamma process, hidden Markov model (HMM), regression-based model, and proportional hazard model. Deep learning methods include deep neural networks (DNN), deep belief networks (DBN), convolutional neural networks (CNN), and recurrent neural networks (RNN). It mainly summarizes the performance degradation and life test of various models in gear and evaluates the advantages and disadvantages of various methods. In addition, it encourages future works.