Self-Supervised 3D Behavior Representation Learning Based on Homotopic Hyperbolic Embedding.

Contrastive learning is an important technique in the field of machine learning for learning data representations. In the field of self-supervised visual representation learning, the strategy of positive and negative sample selection is key to improving the efficiency and effectiveness of model learning. Traditional self-supervised learning methods often employ random sampling strategies to select positive and negative samples, but this approach may result in uneven sample quality when dealing with complex datasets, thereby affecting the learning outcomes. To alleviate this problem, this study is dedicated to exploring more effective strategies for positive and negative sample selection and processing to optimize the self-supervised learning process. To this end, we propose an improved method of self-supervised learning called contrastive learning with enhanced diversity. On the one hand, this method utilizes the weight parameters of the DINO pre-trained model to initialize the feature extraction network of SimCLR, providing more accurate calculations of feature similarity. On the other hand, by setting a threshold on the feature similarity matrix and penalizing (subtracting 0.5 from) similarity scores that do not exceed this threshold, we reduce the excessive impact of high similarity scores on model training, thereby helping the model better distinguish between positive and negative samples. In downstream image classification tasks, we conducted detailed evaluations of the improved model, specifically including fine-tuning and linear evaluation aspects. Experiments show that the proposed approach improves the performance of the loss functions and improves the accuracy of the proposed SimCLR model.

Deep learning has become a valuable tool for the automation of certain medical image segmentation tasks, significantly relieving the workload of medical specialists. Some of these tasks require segmentation to be performed on a subset of the input dimensions, the most common case being 3D→2D. However, the performance of existing methods is strongly conditioned by the amount of labeled data available, as there is currently no data efficient method, e.g. transfer learning, that has been validated on these tasks. In this work, we propose a novel convolutional neural network (CNN) and self-supervised learning (SSL) method for label-efficient 3D→2D segmentation. The CNN is composed of a 3D encoder and a 2D decoder connected by novel 3D→2D blocks. The SSL method consists of reconstructing image pairs of modalities with different dimensionality. The approach has been validated in two tasks with clinical relevance: the en-face segmentation of geographic atrophy and reticular pseudodrusen in optical coherence tomography. Results on different datasets demonstrate that the proposed CNN significantly improves the state of the art in scenarios with limited labeled data by up to 8% in Dice score. Moreover, the proposed SSL method allows further improvement of this performance by up to 23%, and we show that the SSL is beneficial regardless of the network architecture. Our code is available at https://github.com/j-morano/multimodal-ssl-fpn.

Hyperbolic hierarchical knowledge graph embeddings for biological entities

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Adaptive self-supervised learning for sequential recommendation

Self-supervised learning methods (SSL) have achieved significant success via maximizing the mutual information between two augmented views, where cropping is a popular augmentation technique. Cropped regions are widely used to construct positive pairs, while the remained regions after cropping have rarely been explored in existing methods, although they together constitute the same image instance and both contribute to the description of the category. In this paper, we make the first attempt to demonstrate the importance of both regions in cropping from a complete perspective and the effectiveness of using both regions via designing a simple yet effective pretext task called Region Contrastive Learning (RegionCL). Technically, to construct the two kinds of regions, we randomly crop a region (called the paste view) from each input image with the same size and swap them between different images to compose new images together with the remained regions (called the canvas view). Then, instead of taking the new images as a whole for positive or negative samples, contrastive pairs are efficiently constructed from the regional perspective based on the following simple criteria, i.e., each view is (1) positive with views augmented from the same original image and (2) negative with views augmented from other images. With minor modifications to popular SSL methods, RegionCL exploits those abundant pairs and helps the model distinguish the regions features from both canvas and paste views, therefore learning better visual representations. Experiments on ImageNet, MS COCO, and Cityscapes demonstrate that RegionCL improves MoCov2, DenseCL, and SimSiam by large margins and achieves state-of-the-art performance on classification, detection, and segmentation tasks. The code is publicly available at https://github.com/Annbless/RegionCL .

Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. In either category, we provide a unified framework for methods as well as how these methods differ in each component under the framework. Our unified treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms. We also summarize different SSL settings and the corresponding datasets used in each setting. To facilitate methodological development and empirical comparison, we develop a standardized testbed for SSL in GNNs, including implementations of common baseline methods, datasets, and evaluation metrics.

Self-supervised learning provides a possible solution to extract effective visual representations from unlabeled histopathological images. However, existing methods either fail to make good use of domain-specific knowledge, or rely on side information like spatial proximity and magnification. In this paper, we propose CS-CO, a hybrid self-supervised visual representation learning method tailored for histopathological images, which integrates advantages of both generative and discriminative models. The proposed method consists of two self-supervised learning stages: cross-stain prediction (CS) and contrastive learning (CO), both of which are designed based on domain-specific knowledge and do not require side information. A novel data augmentation approach, stain vector perturbation, is specifically proposed to serve contrastive learning. Experimental results on the public dataset NCT-CRC-HE-100K demonstrate the superiority of the proposed method for histopathological image visual representation. Under the common linear evaluation protocol, our method achieves 0.915 eight-class classification accuracy with only 1,000 labeled data, which is about 1.3% higher than the fully-supervised ResNet18 classifier trained with the whole 89,434 labeled training data. Our code is available at https://github.com/easonyang1996/CS-CO.

Considering several patients screened due to COVID-19 pandemic, computer-aided detection has strong potential in assisting clinical workflow efficiency and reducing the incidence of infections among radiologists and healthcare providers. Since many confirmed COVID-19 cases present radiological findings of pneumonia, radiologic examinations can be useful for fast detection. Therefore, chest radiography can be used to fast screen COVID-19 during the patient triage, thereby determining the priority of patient's care to help saturated medical facilities in a pandemic situation. In this paper, we propose a new learning scheme called self-supervised transfer learning for detecting COVID-19 from chest X-ray (CXR) images. We compared six self-supervised learning (SSL) methods (Cross, BYOL, SimSiam, SimCLR, PIRL-jigsaw, and PIRL-rotation) with the proposed method. Additionally, we compared six pretrained DCNNs (ResNet18, ResNet50, ResNet101, CheXNet, DenseNet201, and InceptionV3) with the proposed method. We provide quantitative evaluation on the largest open COVID-19 CXR dataset and qualitative results for visual inspection. Our method achieved a harmonic mean (HM) score of 0.985, AUC of 0.999, and four-class accuracy of 0.953. We also used the visualization technique Grad-CAM++ to generate visual explanations of different classes of CXR images with the proposed method to increase the interpretability. Our method shows that the knowledge learned from natural images using transfer learning is beneficial for SSL of the CXR images and boosts the performance of representation learning for COVID-19 detection. Our method promises to reduce the incidence of infections among radiologists and healthcare providers.

Representation learning on temporal graphs has drawn considerable research attention owing to its fundamental importance in a wide spectrum of real-world applications. Though a number of studies succeed in obtaining time-dependent representations, it still faces significant challenges. On the one hand, most of the existing methods restrict the embedding space with a certain curvature. However, the underlying geometry in fact shifts among the positive curvature hyperspherical, zero curvature Euclidean and negative curvature hyperbolic spaces in the evolvement over time. On the other hand, these methods usually require abundant labels to learn temporal representations, and thereby notably limit their wide use in the unlabeled graphs of the real applications. To bridge this gap, we make the first attempt to study the problem of self-supervised temporal graph representation learning in the general Riemannian space, supporting the time-varying curvature to shift among hyperspherical, Euclidean and hyperbolic spaces. In this paper, we present a novel self-supervised Riemannian graph neural network (SelfRGNN). Specifically, we design a curvature-varying Riemannian GNN with a theoretically grounded time encoding, and formulate a functional curvature over time to model the evolvement shifting among the positive, zero and negative curvature spaces. To enable the self-supervised learning, we propose a novel reweighting self-contrastive approach, exploring the Riemannian space itself without augmentation, and propose an edge-based self-supervised curvature learning with the Ricci curvature. Extensive experiments show the superiority of SelfRGNN, and moreover, the case study shows the time-varying curvature of temporal graph in reality.

The field of machine learning has recently undergone significant advancements, driven by the development of sophisticated algorithms and the availability of large datasets. Among the various paradigms of machine learning, supervised learning has traditionally dominated, relying heavily on labeled data to train models. However, acquiring and annotating large amounts of labeled data is both time-consuming and costly, leading to a growing interest in self-supervised learning (SSL). This promising approach leverages unlabeled data to learn useful representations, which can then be fine-tuned for specific tasks with minimal labeled data Self-supervised learning creates learning signals from the data itself, thus eliminating the need for manual labeling. By solving auxiliary tasks, such as predicting the rotation angle of an image or filling in missing parts of data, SSL methods can learn robust feature representations that are useful for a wide range of downstream tasks. This paradigm reduces dependency on labeled data and exploits the abundance of unlabeled data available in various domains. The significance of SSL lies in its potential to democratize machine learning, making it more accessible and applicable across different fields. From natural language processing to computer vision and beyond, self-supervised techniques have demonstrated remarkable success in improving model performance while reducing reliance on extensive labeled datasets. As such, SSL represents a critical step towards more efficient and scalable machine learning solutions. This paper explores the development and implementation of a novel self-supervised learning solution based on machine learning principles. The proposed method achieves an accuracy of 97.6%, a mean absolute error (MAE) of 0.403, and a root mean square error (RMSE) of 0.203. A comprehensive overview of the underlying methodologies, the challenges faced, and the potential applications of this approach is provided. By examining the theoretical foundations and practical implementations, this study aims to contribute to the growing body of research in SSL and highlight its transformative impact on the machine learning landscape.Self-Supervised Learning.

The implementation of computer vision generally focuses on resolving a problem of one of two types of image data, whether it was natural image datasets or medical image datasets. Self-supervised learning (SSL), as the most recent approach in computer vision, is typically used to solved tasks related only to natural image datasets such as ImageNet, CIFAR-10, and PASCAL. This study focuses on analyzing the involvement of natural images in the SSL implementation when classifying medical images. We used the ResNet50 architecture comparing the results of human embryo image classification using the conventional transfer learning method and SSL methods. We apply three SSL approaches, namely Rotation, Jigsaw Puzzle, and SimCLRv1. The results of our research, with 1226 embryo images divided into three classes, show that the SSL implementations cannot be said to be efficient enough compared to the conventional model produced by ResNet50 with pretrained weight ImageNet. However, when the models were trained using random weight initialization, the Rotation model with an average accuracy of 78.15% and the Jigsaw Puzzle model with an average accuracy of 80.30% are significantly superior to the conventional model with average accuracy is 71.07%. Moreover, related to our embryos dataset, we discuss the differences between the SSL context-context contrastive learning approach and the context instance contrastive learning approach.

The artificial intelligence (AI) community has recently made tremendous progress in developing self-supervised learning (SSL) algorithms that can learn high-quality data representations from massive amounts of unlabeled data. These methods brought great results even to the fields outside of AI. Due to the joint efforts of researchers in various areas, new SSL methods come out daily. However, such a sheer number of publications make it difficult for beginners to see clearly how the subject progresses. This survey bridges this gap by carefully selecting a small portion of papers that we believe are milestones or essential work. We see these researches as the “dots” of SSL and connect them through how they evolve. Hopefully, by viewing the connections of these dots, readers will have a high-level picture of the development of SSL across multiple disciplines including natural language processing, computer vision, graph learning, audio processing, and protein learning.

A limited amount of training data is a well-known challenge in the application of deep learning methods. This issue is particularly relevant in biomedical signal processing, such as the analysis of electrocardiograms, due to the labor-intensive nature of data preparation, which requires the involvement of qualified specialists. Self-supervised learning methods, originally developed in such domains as natural language processing and computer vision, have emerged as a potential approach to addressing this challenge and are increasingly being explored in biomedical signal processing. However, direct adaptation of self-supervised learning techniques from other domains does not fully account for ECG-specific characteristics, such as quasi-periodicity, localized morphological features, and susceptibility to noise. This highlights the relevance of developing ECG-specific self-supervised learning methods. This study presents a novel self-supervised learning approach for pretraining neural networks on unlabeled ECG data. The proposed method is based on predicting the short consecutive signal segment using a preceding one and a learned representation vector. The representation extraction and prediction models are trained jointly on the MIMIC-ECG-IV dataset using backpropagation to minimize the mean squared error between the predicted and original signal segments. As an example of a downstream task, a linear binary classifier was trained on the PTB-XL dataset to diagnose pathological conditions using Lead I. The number of training examples for each diagnosis was limited to thirty-four samples. Firstly, the representation model was pre-trained on the unlabeled MIMIC-ECG-IV dataset, and then linear classifiers were trained on the learned representations for each selected diagnosis in PTB-XL. A comparison was also conducted with a randomly initialized representation model trained jointly with the classifier in a fully supervised manner. The proposed method was evaluated against adaptations of Contrastive Learning, Contrastive Predictive Coding, and Masked Autoencoders method. To ensure a controlled experimental setup, implementations of all considered methods were developed using a unified codebase and shared architectural components. Experimental results demonstrated a significant advantage of all self-supervised learning approaches over joint training of feature extraction and classification models. The proposed SSL method outperformed other tested approaches, particularly for diagnoses with subtle short-term morphological features, such as atrial fibrillation and flutter. These findings suggest the potential for further research in developing ECG-specific self-supervised learning methods as a promising approach to improving neural network performance in scenarios with limited labeled data.

Recent research progress in shadow detection has leveraged the development of remote sensing and computer vision. Since shadows of buildings, trees, bridges in one image can provide useful information about the scene to help people understand the shape, feature or estimate their locations and orientations of original objects, especially for damaged objects. In this study, a novel shadow detection algorithm for remote sensing imagery, called self-supervised learning method is proposed. The aim of this work is to generate shadow ratio threshold automatically without human interaction. To alleviate the traditional drawbacks of shadow detection, we fully combine supervised and unsupervised shadow detection method to suggest a self-supervised learning method, which supports us a strongly clue with establishing the relation of shadow and its original object. Subsequently, we benefit from gray-scale histogram to extract shadow segments, then shadow outlines are obtained. Finally, we assess the shadow detection performance of the proposed approach by comparing our results with the state-of-the-art methods. The results reveal the applicability and precision of the proposed self-supervised learning shadow detection technique.