Comparison of Unsupervised Learning Methods for Natural Image Processing

From an algorithmic complexity point of view, machine learning methods scale and generalize better using a few key features; using lots is computationally expensive, and overfitting can occur. High dimensional data is often counterintuitive to perceive and process, but unfortunately it is common for observed data to be in a representation of greater dimensionality than it requires. This gives rise to the notion of dimensionality reduction, a sub-field of machine learning that is motivated to find a descriptive low-dimensional representation of data. In this review, it is explored a way to perform dimensionality reduction, provided by a class of Latent Variable Models (LVMs). In particular, the aim is to establish the technical foundations required for understanding the Gaussian Process Latent Variable Model (GP-LVM), a probabilistic nonlinear dimensionality reduction model. The review is organized as follows: after an introduction to the problem of dimensionality reduction and LVMs, Principal Component Analysis (PCA) is recalled and it is reviewed its probabilistic equivalent that contributes to the derivation of GP-LVM. Then, GP-LVM is introduced, and briefly a remarkable extension of the latter, the Bayesian Gaussian Process Latent Variable Model (BGP-LVM) is described. Eventually, and the main advantages of using GP-LVM are summarized.

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

Data is one of the most important limiting factors of deep machine learning (ML) model in drilling applications. Though a big size of historical data can be available, high-quality cleaned and labeled data is usually limited. In this case study, we show that with limited labeled data, physics-based data augmentation combined with unsupervised learning significantly improves both stability and accuracy in bit wear ML model. It provides a pathway to overcome labeled data shortage and field data quality limitations. Labeled bit wear data is usually limited because only the final bit dull state can be labeled using dull photos for the entire drilling distance. To overcome this difficulty, an encoder-decoder unsupervised ML framework based on Bi-directional LSTM architecture is first applied to the data of entire drilling distance to extract and maximize data learnings. Then, a physics-based wear estimator is implemented in the learned latent space to augment labeled wear dataset, guided by true wear labels and offset-well data. Weighting factor in loss function is applied to optimize augmented and true datasets in final supervised learning step. The proposed real-time bit wear model is built on a combination of physics-based data augmentation and unsupervised data learning method. The model is applied on multiple field bit runs. Results show that by implementing the unsupervised data learnings only, the prediction accuracy is improved by 30% compared to the baseline ML model. By combining physics-based augmentation, the accuracy is further improved by 10%. More importantly, adding physics-based data augmentation significantly reduces prediction variance and unphysical wear outputs, therefore improving prediction stability by more than 30%. It should be noted that the prediction stability of AI/ML model is crucial in real-time application and decision making. The results show physics-based data augmentation not only increases the size of label dataset and prevents model overfitting, but also applies physics-based guidance to the ML model effectively. It is learned weighting factor plays a crucial role in balancing loss contributions from true wear labels and physics-based labels. While low weighting factor of physics-based labels diminishes the augmented data, high weighting factor disrespects the true wear labels leading to a high prediction bias. Overall, the bit wear model study shows physic-based data augmentation combined with unsupervised data learning can effectively improve model accuracy, stability, and overcome labeled data shortage difficulty. The proposed paper shows a case study for the bit wear ML model using a combination of physics-based data augmentation and unsupervised data learning. While labeled data is one of the major challenges of ML models in many drilling applications, this study provides a pathway to improve both accuracy and stability of deep ML models with limited labeled data.

Chapter 2 - Computational intelligence in healthcare and biosignal processing

Dimensionality reduction (DR) has been considered as one of the most significant tools for data analysis. One type of DR algorithms is based on latent variable models (LVM). LVM-based models can handle the preimage problem easily. In this paper we propose a new LVM-based DR model, named thin plate spline latent variable model (TPSLVM). Compared to the well-known Gaussian process latent variable model (GPLVM), our proposed TPSLVM is more powerful especially when the dimensionality of the latent space is low. Also, TPSLVM is robust to shift and rotation. This paper investigates two extensions of TPSLVM, i.e., the back-constrained TPSLVM (BC-TPSLVM) and TPSLVM with dynamics (TPSLVM-DM) as well as their combination BC-TPSLVM-DM. Experimental results show that TPSLVM and its extensions provide better data visualization and more efficient dimensionality reduction compared to PCA, GPLVM, ISOMAP, etc.

While machine learning and artificial intelligence have long been applied in networking research, the bulk of such works has focused on supervised learning. Recently there has been a rising trend of employing unsupervised machine learning using unstructured raw network data to improve network performance and provide services such as traffic engineering, anomaly detection, Internet traffic classification, and quality of service optimization. The interest in applying unsupervised learning techniques in networking emerges from their great success in other fields such as computer vision, natural language processing, speech recognition, and optimal control (e.g., for developing autonomous self-driving cars). Unsupervised learning is interesting since it can unconstrain us from the need of labeled data and manual handcrafted feature engineering thereby facilitating flexible, general, and automated methods of machine learning. The focus of this survey paper is to provide an overview of the applications of unsupervised learning in the domain of networking. We provide a comprehensive survey highlighting the recent advancements in unsupervised learning techniques and describe their applications for various learning tasks in the context of networking. We also provide a discussion on future directions and open research issues, while also identifying potential pitfalls. While a few survey papers focusing on the applications of machine learning in networking have previously been published, a survey of similar scope and breadth is missing in literature. Through this paper, we advance the state of knowledge by carefully synthesizing the insights from these survey papers while also providing contemporary coverage of recent advances.

Artificial Intelligence (AI) has advanced in a rapid and exponential manner and AI is now revolutionizing the medical field, the development of self-driving cars, and the management of financial operations. A major reason for such achievement is the efficiency and performance of AI models, which can be drastically boosted through new learning methods. One of the most promising avenues is the unsupervised representation learning method that also offers the choice of leaping over the traditional supervised learning. Instead of supervised learning, where labeled data is the base upon which training of models takes place, unsupervised learning allows AI systems to get new insights out of raw, unlabeled data without any human intervention. This method teaches the AI system how to represent the data in an organized manner, thus enabling it to uncover hidden features and relations without anyone helping it. In effect, AI models, powered by unsupervised representation learning, can be quite successful in areas like clustering, anomaly detection, and feature extraction, frequently beating traditional methods in terms of their speed and accuracy. .The skill of finding deep structures in a dataset can unexpectedly influence many areas of the sciences, e.g., it can help us create a better diagnostic system in medicine or use it for decision-making when investing in the stock market

ABSTRACTIn this article, speeded-up robust features (SURF) for each image have been calculated. Discrete Fourier transform (DFT) method has been applied to these SURF. High dimensions of these SURF–DFT feature vectors are reduced to low dimensions with large-margin nearest neighbor (LMNN), Gaussian process latent variable models (GPLVM), and neighborhood component analysis (NCA). When size reduction process was done, effect on the GPLVM, LMNN, and NCA of the 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 feature numbers has been examined. These features are classified by naive Bayes (NB) classifier. Thus, SURF_DFT_GPLVM_NB, SURF_DFT_NCA_NB, and SURF_DFT_LMNN_NB methods for gastric histopathological images have been developed. Classification results obtained with these methods have been compared. According to the obtained results, the highest classification result was obtained as 90.24% by using 4 features by SURF_DFT_GPLVM_NB method for second group images.

Machine learning has significantly impacted the analysis of biological images and is now an important part of many biological data analysis pipelines. A variety of biological and biomedical domain-related tasks is gaining benefit from image analysis and pattern recognition tools developed currently. Applications include diagnostic histopathology, environmental monitoring, synthetic biology, genomics, and proteomics. Particularly in the last decade, several deep learning and advanced computer vision methods such as convolutional neural networks (CNNs), typically trained in a supervised fashion, have started to be largely employed in biological image classification. Moreover, the advancement of automatic acquisition systems has been generating a massive amount of biological data, which requires to be analyzed by domain experts. However, the cost of manual annotation of such data has become a bottleneck, impairing the application of supervised machine learning algorithms. Biological images generally have an intrinsic high variability, whose identity is sometimes hard to assign and strongly dependent on the annotator’s expertise. In this context, a limited number of annotation-free (i.e., unsupervised) learning solutions have been proposed, typically based on hand-crafted features, specifically tailored for a certain biological domain. Nonetheless, a successful unsupervised learning approach must be accurate, and sufficiently robust to deal with different biological domains. This paper aims at providing a viable solution to these issues, proposing an unsupervised learning algorithm based on compressed deep features for image classification. We exploit features extracted from ImageNet pre-trained transformers and CNNs, further compressed with a customized β-Variational AutoEncoder (β-VAE), that we call reconstruction VAE (R-VAE). We test our algorithm on biological images coming from diverse domains characterized by high variability in shape and texture information and acquired with widely differing imaging platforms. Considered image datasets range from multi-cellular organisms (plankton, coral) to sub-cellular organelles (budding yeast vacuoles, human cells’ nuclei, etc.). Our results show that the compressed deep features extracted from different pre-trained vision models establish new unsupervised learning state-of-the-art performances for the investigated datasets.

Independent component analysis (ICA) is a powerful tool for redundancy reduction and nongaussian data analysis. And, artificial neural network (ANN), especially the self-organizing map (SOM) based on unsupervised learning is a kind of excellent method for pattern clustering and recognition. By combining ICA with ANN, a novel compound neural network for fault sources recognition was proposed. First, neural ICA algorithm was applied to fusion of multi-channel measurements by sensors. Moreover, further feature extraction was made. Thus, statistical features higher than second order were captured from the measurements. Second, a typical neural classifier such as the back-propagation (BP), the radial basis function (RBF) or the SOM network was trained for the final fault sources recognition. The results from contrast experiments in fault diagnosis of rotating machines show that the proposed compound neural network with ICA based feature extraction can recognize various fault sources at considerable accuracy, and be constructed in simpler way, both of which imply its great potential in fault diagnosis.

Introduction: Unsupervised machine learning (ML) can provide insight into a dataset without significant expert time for annotation and labeling, but was never used to assess the heterogeneous MitraClip population. Hypothesis: We hypothesized that unsupervised k-means ML could identify prognostically-distinct phenogroups in MitraClip patients without a priori knowledge of the dataset. Methods: Patients who underwent MitraClip (June 2014-September 2020) at Mayo Clinic sites were identified from the Mayo institutional NCDR database for baseline and follow-up data. Variables with ≥ 50% missing data were excluded. A k-means algorithm was used for clustering analysis. Input variables were grouped according to 4 distinct k-means determined clusters. Kaplan-Meier survival analysis was used to assess the survival of each cluster. Results: A total of 389 consecutive patients were included in the final analysis, and 95 unique variables were used. The mean age was 80.3±8.7 years; 256 (65.8%) were male. The mean STS MV Replace score was 9.1±5.7. Fifty-five (14.5%) patients died during the mean follow duration (185 days). Kaplan-Meier analysis showed significant survival differences among clusters (Figure 1). The cluster (cluster 2, n=120) with best survival performance (all-cause mortality 3.3%) had features of non-calcified mitral leaflets, nonischemic cardiomyopathy, and less than moderate tricuspid regurgitation. Cluster 3 (n=66) had the worst survival performance (all-cause mortality 27.3%) and opposite features than cluster 2. Conclusions: Unsupervised ML can identify distinct phenotype clusters with prognostic significance in MitraClip patients. Patients with non-calcified mitral leaflets, nonischemic cardiomyopathy, and less than moderate tricuspid regurgitation are intrinsically different from other MitraClip patients and should be considered the potential optimal candidates.

Enhancing forensic investigations: Identifying bloodstains on various substrates through ATR-FTIR spectroscopy combined with machine learning algorithms

The Gaussian process latent variable model (GP-LVM) has been identified to be an effective probabilistic approach for dimensionality reduction because it can obtain a low-dimensional manifold of a data set in an unsupervised fashion. Consequently, the GP-LVM is insufficient for supervised learning tasks (e.g., classification and regression) because it ignores the class label information for dimensionality reduction. In this paper, a supervised GP-LVM is developed for supervised learning tasks, and the maximum a posteriori algorithm is introduced to estimate positions of all samples in the latent variable space. We present experimental evidences suggesting that the supervised GP-LVM is able to use the class label information effectively, and thus, it outperforms the GP-LVM and the discriminative extension of the GP-LVM consistently. The comparison with some supervised classification methods, such as Gaussian process classification and support vector machines, is also given to illustrate the advantage of the proposed method.

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Accurate prediction of rehabilitation training effect is important for the development of personalized rehabilitation program. In order to improve the prediction accuracy, a multimodal data fusion model based on machine learning is constructed to integrate clinical data, image data and physiological signals for feature extraction and optimization. Supervised learning methods are used for classification prediction, and unsupervised learning is combined for data pattern recognition and dimensionality reduction analysis. The experimental results show that the CNN performs best in several indicators and has strong generalization ability. The research results can provide data-driven decision support for rehabilitation training, improve the objectivity and accuracy of rehabilitation assessment, and promote the development of intelligent rehabilitation medicine.