Fully Linear Graph Convolutional Networks for Semi-Supervised and Unsupervised Classification

Semi-supervised learning with a graph-based approach has gained prominence in machine learning, particularly in scenarios where labeling data involves substantial costs. Graph convolution networks (GCNs) have found widespread application in semi-supervised learning, predominantly on graph-structured data such as citation and social networks. However, a noticeable gap exists in the application of these methods to non-graph multi-view data, such as collections of images. In an effort to address this gap, we introduce two innovative deep semi-supervised multi-view classification models specifically tailored for non-graph data. Both models share a common architecture, leveraging GCNs and integrating a label smoothing constraint. The primary distinction lies in the construction of the consensus similarity graph. The first model directly reconstructs the consensus graph from different views using a specialized objective function designed for flexible graph-based semi-supervised classification. In contrast, the second model independently reconstructs individual graphs and subsequently adaptively merges them into a unified consensus graph. Our experiments encompass various multiple-view image datasets. The results consistently demonstrate the superior performance of our proposed approach compared to traditional fusion methods with GCNs. In this research, we present two approaches for tackling semi-supervised classification challenges involving multiple views. One method is named Semi-supervised Classification with a Unified Graph (SCUG), and the other is referred to as Semi-supervised Classification with a Fused Graph (SC-Fused). Both methods share a common semi-supervised classification process, utilizing the GCN framework and incorporating label smoothing. However, the key distinction lies in the construction of the similarity graph. Unlike traditional ad hoc graph construction approaches, our proposed methods, SCUG and SC-Fused, estimate the unified graph or individual graphs, respectively, alongside the labels. This results in more optimized graphs that benefit from data smoothing and the semi-supervised context.

Efficient modeling of high-dimensional data requires extracting only relevant dimensions through feature learning. Unsupervised feature learning has gained tremendous attention due to its unbiased approach, no need for prior knowledge or expensive manual processing, and ability to handle exponential data growth. Deep Autoencoder (AE) is a state-of-the-art deep neural network for unsupervised feature learning, which learns embedded-representations using a series of stacked layers. However, as the AE network gets deeper, these learned embedded-representations can deteriorate due to vanishing gradient, leading to performance degradation. This article presents ResNet Autoencoder (RAE) and its convolutional version (C-RAE) for unsupervised feature learning. The advantage of RAE and C-RAE is that it enables the user to add residual connections for increased network capacity without incurring the cost of degradation for unsupervised feature learning compared to standard AEs. While RAE and C-RAE inherit all the advantages of AEs, such as automated non-linear feature extraction and unsupervised learning, they also allow users to design larger networks without adverse effects on feature learning performance. We performed classification on learned embedded-representation to evaluate RAE and C-RAE. RAE and C-RAE were compared against AEs on MNIST, Fashion MNIST, and CIFAR10 datasets. When increasing the number of layers, C-RAE outperformed AE by showing significantly lower performance degradation of classification accuracy (less than 3%) compared to AE (33% to 65%). Further, C-RAE exhibited higher mean accuracy and lower variance of accuracy than standard AE. When comparing RAE and C-RAE with widely used feature learning methods (Convolutional AE, PCA, ICA, LLE, Factor Analysis, and SVD), C-RAE showed the highest accuracy.

Unsupervised feature learning refers to the problem of learning useful feature extraction functions from unlabeled data. Despite the great success of deep learning networks in this task in recent years, both for static and for sequential data, these systems can in general still not compete with the high performance of our brain at learning to extract useful representations from its sensory input. We propose the Neocortex-Inspired Locally Recurrent Neural Network: a new neural network for unsupervised feature learning in sequential data that brings ideas from the structure and function of the neocortex to the well-established fields of machine learning and neural networks. By mimicking connection patterns in the feedforward circuits of the neocortex, our system tries to generalize some of the ideas behind the success of convolutional neural networks to types of data other than images. To evaluate the performance of our system at extracting useful features, we have trained different classifiers using those and other learnt features as input and we have compared the obtained accuracies. Our system has shown to outperform other shallow feature learning systems in this task, both in terms of the accuracies achieved and in terms of how fast the classification task is learnt. The results obtained confirm our system as a state-of-the-art shallow feature learning system for sequential data, and suggest that extending it to or integrating it into deep architectures may lead to new successful networks that are competent at dealing with complex sequential tasks.

Spiking Neural Networks (SNNs) bear the potential of efficient unsupervised\nand continual learning capabilities because of their biological plausibility,\nbut their complexity still poses a serious research challenge to enable their\nenergy-efficient design for resource-constrained scenarios (like embedded\nsystems, IoT-Edge, etc.). We propose SpikeDyn, a comprehensive framework for\nenergy-efficient SNNs with continual and unsupervised learning capabilities in\ndynamic environments, for both the training and inference phases. It is\nachieved through the following multiple diverse mechanisms: 1) reduction of\nneuronal operations, by replacing the inhibitory neurons with direct lateral\ninhibitions; 2) a memory- and energy-constrained SNN model search algorithm\nthat employs analytical models to estimate the memory footprint and energy\nconsumption of different candidate SNN models and selects a Pareto-optimal SNN\nmodel; and 3) a lightweight continual and unsupervised learning algorithm that\nemploys adaptive learning rates, adaptive membrane threshold potential, weight\ndecay, and reduction of spurious updates. Our experimental results show that,\nfor a network with 400 excitatory neurons, our SpikeDyn reduces the energy\nconsumption on average by 51% for training and by 37% for inference, as\ncompared to the state-of-the-art. Due to the improved learning algorithm,\nSpikeDyn provides on avg. 21% accuracy improvement over the state-of-the-art,\nfor classifying the most recently learned task, and by 8% on average for the\npreviously learned tasks.\n

Joint sparse subspace clustering via fast [formula omitted]-norm constrained optimization

An unsupervised competitive learning algorithm based on the classical -means clustering algorithm is proposed. The proposed learning algorithm called the centroid neural network (CNN) estimates centroids of the related cluster groups in training date. This paper also explains algorithmic relationships among the CNN and some of the conventional unsupervised competitive learning algorithms including Kohonen's self-organizing map (SOM) and Kosko's differential competitive learning (DCL) algorithm. The CNN algorithm requires neither a predetermined schedule for learning coefficient nor a total number of iterations for clustering. The simulation results on clustering problems and image compression problems show that CNN converges much faster than conventional algorithms with compatible clustering quality while other algorithms may give unstable results depending on the initial values of the learning coefficient and the total number of iterations.

Modeling the joint distribution of high-dimensional data is a central task in unsupervised machine learning. In recent years, many interests have been attracted to developing learning models based on tensor networks, which have the advantages of a principle understanding of the expressive power using entanglement properties, and as a bridge connecting classical computation and quantum computation. Despite the great potential, however, existing tensor network models for unsupervised machine learning only work as a proof of principle, as their performance is much worse than the standard models such as restricted Boltzmann machines and neural networks. In this Letter, we present autoregressive matrix product states (AMPS), a tensor network model combining matrix product states from quantum many-body physics and autoregressive modeling from machine learning. Our model enjoys the exact calculation of normalized probability and unbiased sampling. We demonstrate the performance of our model using two applications, generative modeling on synthetic and real-world data, and reinforcement learning in statistical physics. Using extensive numerical experiments, we show that the proposed model significantly outperforms the existing tensor network models and the restricted Boltzmann machines, and is competitive with state-of-the-art neural network models.

The ultimate goal of precision medicine is to determine right treatment for right patients based on precise diagnosis. To achieve this goal, correct stratification of patients using molecular features and clinical phenotypes is crucial. During the long history of medical science, our understanding on disease classification has been improved greatly by chemistry and molecular biology. Nowadays, we gain access to large scale patient-derived data by high-throughput technologies, generating a greater need for data science including unsupervised learning and network modeling. Unsupervised learning methods such as clustering could be a better solution to stratify patients when there is a lack of predefined classifiers. In network modularity analysis, clustering methods can be also applied to elucidate the complex structure of biological and disease networks at the systems level. In this review, we went over the main points of clustering analysis and network modeling, particularly in the context of Traditional Chinese medicine (TCM). We showed that this approach can provide novel insights on the rationale of classification for TCM herbs. In a case study, using a modularity analysis of multipartite networks, we illustrated that the TCM classifications are associated with the chemical properties of the herb ingredients. We concluded that multipartite network modeling may become a suitable data integration tool for understanding the mechanisms of actions of traditional medicine.

In the rapidly advancing field of AI and ML, this paper explores their pivotal role in transforming cybersecurity. Highlighting the integration of sophisticated techniques like deep learning for intrusion detection and reinforcement learning for adaptive threat modeling, it emphasizes the shift towards AI-driven cybersecurity solutions. The study meticulously analyzes supervised and unsupervised learning's impact on threat detection accuracy and the dynamic capabilities of neural networks in real-time threat identification. It reveals how these methodologies enhance digital defenses against complex cyber threats, underscoring the theoretical underpinnings and practical applications of AI and ML in cybersecurity. The paper also discusses the challenges and future directions, contributing significant insights into the evolving landscape of cybersecurity technologies. This comprehensive research background sets the stage for understanding the unique contributions and potential of AI and ML in strengthening cybersecurity measures.

The process of merging data of different data sources is referred to as record linkage. A medical environment with increased preconditions on privacy protection demands the transformation of clear-text attributes like first name or date of birth into one-way encrypted pseudonyms. When performing an automated or privacy preserving record linkage there might be the need of a binary classification deciding whether two records should be classified as the same entity. The classification is the final of the four main phases of the record linkage process: Preprocessing, indexing, matching and classification. The choice of binary classification techniques in dependence of project specifications in particular data quality has not extensively been studied yet. The aim of this work is the introduction and evaluation of an automatable semi-supervised binary classification system applied within the field of record linkage capable of competing or even surpassing advanced automated techniques of the domain of unsupervised classification. This work describes the rationale leading to the model and the final implementation of an automatable semi-supervised binary classification system and the comparison of its classification performance to an advanced active learning approach out of the domain of unsupervised learning. The performance of both systems has been measured on a broad variety of artificial test sets (n = 400), based on real patient data, with distinct and unique characteristics. While the classification performance for both methods measured as F-measure was relatively close on test sets with maximum defined data quality, 0.996 for semi-supervised classification, 0.993 for unsupervised classification, it incrementally diverged for test sets of worse data quality dropping to 0.964 for semi-supervised classification and 0.803 for unsupervised classification. Aside from supplying a viable model for semi-supervised classification for automated probabilistic record linkage, the tests conducted on a large amount of test sets suggest that semi-supervised techniques might generally be capable of outperforming unsupervised techniques especially on data with lower levels of data quality.

Infinity Laplacian on graphs with gradient terms for image and data clustering

Attention reweighted sparse subspace clustering

Machine learning has significantly impacted the analysis of biological images and is now an important part of many biological data analysis pipelines. A variety of biological and biomedical domain-related tasks is gaining benefit from image analysis and pattern recognition tools developed currently. Applications include diagnostic histopathology, environmental monitoring, synthetic biology, genomics, and proteomics. Particularly in the last decade, several deep learning and advanced computer vision methods such as convolutional neural networks (CNNs), typically trained in a supervised fashion, have started to be largely employed in biological image classification. Moreover, the advancement of automatic acquisition systems has been generating a massive amount of biological data, which requires to be analyzed by domain experts. However, the cost of manual annotation of such data has become a bottleneck, impairing the application of supervised machine learning algorithms. Biological images generally have an intrinsic high variability, whose identity is sometimes hard to assign and strongly dependent on the annotator’s expertise. In this context, a limited number of annotation-free (i.e., unsupervised) learning solutions have been proposed, typically based on hand-crafted features, specifically tailored for a certain biological domain. Nonetheless, a successful unsupervised learning approach must be accurate, and sufficiently robust to deal with different biological domains. This paper aims at providing a viable solution to these issues, proposing an unsupervised learning algorithm based on compressed deep features for image classification. We exploit features extracted from ImageNet pre-trained transformers and CNNs, further compressed with a customized β-Variational AutoEncoder (β-VAE), that we call reconstruction VAE (R-VAE). We test our algorithm on biological images coming from diverse domains characterized by high variability in shape and texture information and acquired with widely differing imaging platforms. Considered image datasets range from multi-cellular organisms (plankton, coral) to sub-cellular organelles (budding yeast vacuoles, human cells’ nuclei, etc.). Our results show that the compressed deep features extracted from different pre-trained vision models establish new unsupervised learning state-of-the-art performances for the investigated datasets.

Graph Neural Networks (GNNs) have achieved great success on graph-structured data, and their applications on traditional data structures such as natural language processing and semi-supervised text classification have been extensively explored in recent years. While previous works only consider the text information while building the graph, heterogeneous information such as labels is ignored. In this paper, we consider incorporating the label information while building the graph by adding text-label-text paths, through which the supervision information will propagate among the graph more directly. Specifically, we treat labels as nodes in the graph which also contains text and word nodes, and then connect labels with texts belonging to that label. Through graph convolutions, label embeddings are jointly learned with text embeddings in the same latent semantic space. The newly incorporated label nodes will facilitate learning more accurate text embeddings by introducing the label information, and thus benefit the downstream text classification tasks. Extensive results on several benchmark datasets show that the proposed framework outperforms baseline methods by a significant margin.

Semi-supervised node classification on graph-structured data has many applications such as fraud detection, fake account and review detection, user’s private attribute inference in social networks, and community detection. Various methods such as pairwise Markov Random Fields (pMRF) and graph neural networks were developed for semi-supervised node classification. pMRF is more efficient than graph neural networks. However, existing pMRF-based methods are less accurate than graph neural networks, due to a key limitation that they assume a heuristics-based constant edge potential for all edges. In this work, we aim to address the key limitation of existing pMRF-based methods. In particular, we propose to learn edge potentials for pMRF. Our evaluation results on various types of graph datasets show that our optimized pMRF-based method consistently outperforms existing graph neural networks in terms of both accuracy and efficiency. Our results highlight that previous work may have underestimated the power of pMRF for semi-supervised node classification.