GLARE: discovering hidden patterns in spaceflight transcriptome using representation learning

Product embeddings have been heavily investigated in the past few years, serving as the cornerstone for a broad range of machine learning applications in e-commerce. Despite the empirical success of product embeddings, little is known on how and why they work from the theoretical standpoint. Analogous results from the natural language processing (NLP) often rely on domain-specific properties that are not transferable to the e-commerce setting, and the downstream tasks often focus on different aspects of the embeddings. We take an e-commerce-oriented view of the product embeddings and reveal a complete theoretical view from both the representation learning and the learning theory perspective. We prove that product embeddings trained by the widely-adopted skip-gram negative sampling algorithm and its variants are sufficient dimension reduction regarding a critical product relatedness measure. The generalization performance in the downstream machine learning task is controlled by the alignment between the embeddings and the product relatedness measure. Following the theoretical discoveries, we conduct exploratory experiments that supports our theoretical insights for the product embeddings.

Lighting is a crucial environmental variable in poultry operations, but illumination during incubation is relatively understudied. The ability to stimulate development or immune performance using in ovo lighting is a promising approach for improving poultry health and welfare. This study investigated how monochromatic green light during incubation and vaccination method and timing affected chicken splenic gene expression patterns. We performed this study with 1,728 Hy-Line white layer eggs incubated under two light treatments during incubation: continuous dark and continuous green monochromatic light, over the entire incubation period. Half the eggs in each light treatment received in ovo vaccination, applied on embryonic day 18 (ED18). The remaining half were vaccinated by spraying on hatch day. After hatching, the light treatments followed the industry-standard lighting regimens. The study had six treatment groups with light–dark pairs for non-vaccinated, in ovo vaccinated, and post-hatch vaccinated. We assessed splenic gene expression at ED18 and at 7 days post-hatch (PH) in all the treatments. We isolated and sequenced 24 mRNA libraries on the Illumina platform, followed by bioinformatics and differential gene expression analyses. RNAseq analysis showed between 62 and 6,755 differentially expressed genes (DEGs) between comparisons, with the most prominent differences observed between ED and PH samples, followed by comparisons between vaccination methods. In contrast, light vs. dark treatments at ED showed limited effects on transcriptomic profiles. However, we observed a synergistic effect of lighting during incubation on post-hatch vaccination responses, with differentially expressed genes (DEGs) unique to the light treatment showing stimulation of cell proliferation with significance for immune activity (inferred from gene ontology terms). Gene ontology and pathway analysis indicated biological processes like cellular component organization or biogenesis, rhythmic process, developmental process, response to stimulus, and immune system processes were explained by the DEGs. While lighting is an important source of circadian stimulation, other controlled studies are required to clarify whether in ovo circadian entrainment plays a role in modulating immune responses.

In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing tasks, such as language modelling and machine translation. This suggests that neural models will also achieve good performance on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using a semantic rather than lexical matching. Although initial iterations of neural models do not outperform traditional lexical-matching baselines, the level of interest and effort in this area is increasing, potentially leading to a breakthrough. The popularity of the recent SIGIR 2016 workshop on Neural Information Retrieval provides evidence to the growing interest in neural models for IR. While recent tutorials have covered some aspects of deep learning for retrieval tasks, there is a significant scope for organizing a tutorial that focuses on the fundamentals of representation learning for text retrieval. The goal of this tutorial will be to introduce state-of-the-art neural embedding models and bridge the gap between these neural models with early representation learning approaches in IR (e.g., LSA). We will discuss some of the key challenges and insights in making these models work in practice, and demonstrate one of the toolsets available to researchers interested in this area.

Light is an important environmental regulator of plant secondary metabolism. Terpenoids, the most abundant secondary metabolites in plants, demonstrate a wide spectrum of biologically significant properties, encompassing antimicrobial, antioxidative, and analgesic activities. Litsea cubeba (Lour.) Pers., a core species within the Lauraceae family, exhibits notable pharmacological potential, including antimicrobial and antitumor effects. Here, we found that darkness treatment significantly suppressed terpenoid accumulation in L. cubeba fruits. To clarify the molecular mechanisms underlying the regulatory effect of light and darkness treatments on terpenoid biosynthesis, we conducted a comparative transcriptome profiling of L. cubeba fruits under light and darkness treatments. A total of 13,074 differentially expressed genes (DEGs) were identified among four sampling time points (L1-L2-L3-L4 vs. D1-D2-D3-D4). These genes were enriched in various pathways, with significant enrichment being observed in the terpenoid and other secondary metabolism pathways. Additionally, the enrichment of DEGs in L2 and D2 stages was further studied, and it was found that nine DEGs were significantly enriched in the monoterpene synthesis pathway. The weighted gene co-expression network analysis (WGCNA) showed that alcohol dehydrogenase (ADH), a key enzyme in terpenoid synthesis, had the same expression pattern as WRKY and NAC transcription factors, suggesting their involvement in the biosynthesis of terpenoids in L. cubeba. Expression profiling demonstrated that plastid-localized terpenoid pathway genes were markedly downregulated under darkness treatment. qRT-PCR validation of key genes (LcDXS3, LcHMGS1, LcMDS, and LcTPS19) confirmed the reliability of the transcriptome data, with LcDXS3 exhibiting pronounced declines in expression after 6 h (2.76-fold decrease) and 12 h (2.63-fold decrease) of darkness treatment. These findings provide novel insights into the photoregulatory mechanisms governing terpenoid metabolism in L. cubeba.

In this paper, we design and evaluate a new substructure-aware Graph Representation Learning (GRL) approach. GRL aims to map graph structure information into low-dimensional representations. While extensive efforts have been made for modeling global and/or local structure information, GRL can be improved by substructure information. Some recent studies exploit adversarial learning to incorporate substructure awareness, but hindered by unstable convergence. This study will address the major research question: is there a better way to integrate substructure awareness into GRL? As subsets of the graph structure, interested substructures (i.e., subgraph) are unique and representative for differentiating graphs, leading to the high correlation between the representation of the graph-level structure and substructures. Since mutual information (MI) is to evaluate the mutual dependence between two variables, we develop a MI inducted substructure-aware GRL method. We decompose the GRL pipeline into two stages: (1) node-level, where we introduce to maximize MI between the original and learned representation by the intuition that the original and learned representation should be highly correlated; (2) graph-level, where we preserve substructures by maximizing MI between the graph-level structure and substructure representation. Finally, we present extensive experimental results to demonstrate the improved performances of our method with real-world data.

In this paper, we propose an approach to learn optimal service representations that can be used in downstream data mining tasks by combining attributes and multiple relations from a heterogeneous information network and using meta-paths to model service relations. We construct a heterogeneous information network that connects services, mashups, and their attributes and derive latent relational knowledge from it for representation learning. We address two major challenges related to using such a heterogeneous network in representation learning: (1) how to effectively combine attribute information and network topology, (2) how to optimize the weighting scheme among those inputs to enhance the learning from relevant features and minimize distraction from noisy ones. Our approach can take advantage of partially or fully annotated data, and still works in a fully unsupervised setting. We conduct a comprehensive experimental study on a real world dataset, where we use clustering as a downstream task. In our experiments, Our model performs better than all competing approaches.

Self-supervised representation learning for visual pre-training has achieved remarkable success with sample (instance or pixel) discrimination and semantics discovery of instance, whereas there still exists a non-negligible gap between pre-trained model and downstream dense prediction tasks. Concretely, these downstream tasks require more accurate representation, in other words, the pixels from the same object must belong to a shared semantic category, which is lacking in the previous methods. In this work, we present Dense Semantic Contrast (DSC) for modeling semantic category decision boundaries at a dense level to meet the requirement of these tasks. Furthermore, we propose a dense cross-image semantic contrastive learning framework for multi-granularity representation learning. Specially, we explicitly explore the semantic structure of the dataset by mining relations among pixels from different perspectives. For intra-image relation modeling, we discover pixel neighbors from multiple views. And for inter-image relations, we enforce pixel representation from the same semantic class to be more similar than the representation from different classes in one mini-batch. Experimental results show that our DSC model outperforms state-of-the-art methods when transferring to downstream dense prediction tasks, including object detection, semantic segmentation, and instance segmentation. Code will be made available.

Local discriminative representation is needed in many medical image analysis tasks such as identifying sub-types of lesion or segmenting detailed components of anatomical structures. However, the commonly applied supervised representation learning methods require a large amount of annotated data, and unsupervised discriminative representation learning distinguishes different images by learning a global feature, both of which are not suitable for localized medical image analysis tasks. In order to avoid the limitations of these two methods, we introduce local discrimination into unsupervised representation learning in this work. The model contains two branches: one is an embedding branch which learns an embedding function to disperse dissimilar pixels over a low-dimensional hypersphere; and the other is a clustering branch which learns a clustering function to classify similar pixels into the same cluster. These two branches are trained simultaneously in a mutually beneficial pattern, and the learnt local discriminative representations are able to well measure the similarity of local image regions. These representations can be transferred to enhance various downstream tasks. Meanwhile, they can also be applied to cluster anatomical structures from unlabeled medical images under the guidance of topological priors from simulation or other structures with similar topological characteristics. The effectiveness and usefulness of the proposed method are demonstrated by enhancing various downstream tasks and clustering anatomical structures in retinal images and chest X-ray images.KeywordsUnsupervised representation learningLocal discriminationTopological priors

Unsupervised and self-supervised representation learning has become popular in recent years for learning useful features from unlabelled data. Representation learning has been mostly developed in the neural network literature, and other models for representation learning are surprisingly unexplored. In this work, we introduce and analyze several kernel-based representation learning approaches: Firstly, we define two kernel Self-Supervised Learning (SSL) models using contrastive loss functions and secondly, a Kernel Autoencoder (AE) model based on the idea of embedding and reconstructing data. We argue that the classical representer theorems for supervised kernel machines are not always applicable for (self-supervised) representation learning, and present new representer theorems, which show that the representations learned by our kernel models can be expressed in terms of kernel matrices. We further derive generalisation error bounds for representation learning with kernel SSL and AE, and empirically evaluate the performance of these methods in both small data regimes as well as in comparison with neural network based models.

Global land cover (GLC) products can be utilized to provide geographical supervision for remote sensing representation learning, which has significantly improved downstream tasks’ performance and decreased the demand of manual annotations. However, the time differences between remote sensing images and GLC products may introduce deviations in geographical supervision. In this paper, we propose a Geographical supervision Correction method (GeCo) for remote sensing representation learning. Deviated geographical supervision generated by GLC products can be corrected adaptively using the correction matrix during network pre-training and joint optimization process is designed to simultaneously update the correction matrix and network parameters. Additionally, we identify prior knowledge on geographical supervision to guide representation learning and restrict the correction process. The prior knowledge named “minor changes” implies that the geographical supervision may not change significantly, whereas the prior knowledge named “spatial aggregation” implies that land covers are aggregated in their spatial distribution. According to the prior knowledge, corresponding regularization terms are proposed to prevent abrupt changes in geographical supervision correction process and excessive smoothing of network outputs, thereby ensuring the adaptive correction process’s correctness. Experimental results demonstrate that our proposed method outperforms random initialization, ImageNet pre-training, and other representation learning methods on a variety of downstream tasks. In particular, when compared to the method that learns representations directly from deviated geographical supervision, it is proved that our method can eliminate the influence of deviations and further improve the effect of representation learning.

Current self-supervised approaches for skeleton action representation learning often focus on constrained scenarios, where videos and skeleton data are recorded in laboratory settings. When dealing with estimated skeleton data in realworld videos, such methods perform poorly due to the large variations across subjects and camera viewpoints. To address this issue, we introduce ViA, a novel View-Invariant Autoencoder for self-supervised skeleton action representation learning 1 . ViA leverages motion retargeting between different human performers as a pretext task, in order to disentangle the latent action-specific 'Motion' features on top of the visual representation of a 2D or 3D skeleton sequence. Such 'Motion' features are invariant to skeleton geometry and camera view and allow ViA to facilitate both, cross-subject and cross-view action classification tasks. We conduct a study focusing on transfer-learning for skeleton-based action recognition with self-supervised pre-training on real-world data (e.g., Posetics). Our results showcase that skeleton representations learned from ViA are generic enough to improve upon stateof-the-art action classification accuracy, not only on 3D laboratory datasets such as NTU-RGB+D 60 and NTU-RGB+D 120, but also on real-world datasets where only 2D data are accurately estimated, e.g., Toyota Smarthome, UAV-Human and Penn Action.

Multi-view classification with limited sample size and data augmentation is a very common machine learning (ML) problem in medicine. With limited data, a triplet network approach for two-stage representation learning has been proposed. However, effective training and verifying the features from the representation network for their suitability in subsequent classifiers are still unsolved problems. Although typical distance-based metrics for the training capture the overall class separability of the features, the performance according to these metrics does not always lead to an optimal classification. Consequently, an exhaustive tuning with all feature-classifier combinations is required to search for the best end result. To overcome this challenge, we developed a novel nearest-neighbor (NN) validation strategy based on the triplet metric. This strategy is supported by a theoretical foundation to provide the best selection of the features with a lower bound of the highest end performance. The proposed strategy is a transparent approach to identify whether to improve the features or the classifier. This avoids the need for repeated tuning. Our evaluations on real-world medical imaging tasks (i.e., radiation therapy delivery error prediction and sarcoma survival prediction) show that our strategy is superior to other common deep representation learning baselines [i.e., autoencoder (AE) and softmax]. The strategy addresses the issue of feature's interpretability which enables more holistic feature creation such that the medical experts can focus on specifying relevant data as opposed to tedious feature engineering.

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This article (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.

Microbes and sunlight convert terrigenous dissolved organic matter (DOM) in surface waters to greenhouse gases. Prior studies show contrasting results about how biological and photochemical processes interact to contribute to the degradation of DOM. In this study, DOM leached from the organic layer of tundra soil was exposed to natural sunlight or kept in the dark, incubated in the dark with the natural microbial community, and analysed for gene expression and DOM chemical composition. Microbial gene expression (metatranscriptomics) in light and dark treatments diverged substantially after 4 h. Gene expression suggested that sunlight exposure of DOM initially stimulated microbial growth by (i) replacing the function of enzymes that degrade higher molecular weight DOM such as enzymes for aromatic carbon degradation, oxygenation, and decarboxylation, and (ii) releasing low molecular weight compounds and inorganic nutrients from DOM. However, growth stimulation following sunlight exposure of DOM came at a cost. Sunlight depleted the pool of aromatic compounds that supported microbial growth in the dark treatment, ultimately causing slower growth in the light treatment over 5 days. These first measurements of microbial metatranscriptomic responses to photo-alteration of DOM provide a mechanistic explanation for how sunlight exposure of terrigenous DOM alters microbial processing and respiration of DOM.

With the surge of deep learning framework, various studies have attempted to address the challenges of sentiment analysis of tweets (data sparsity, under-specificity, noise, and multilingual content) through text and network-based representation learning approaches. However, limited studies on combining the benefits of textual and structural (graph) representations for sentiment analysis of tweets have been carried out. This study proposes a multi-view learning framework (end-to-end and ensemble-based) that leverages both text-based and graph-based representation learning approaches to enrich the tweet representation for sentiment classification. The efficacy of the proposed framework is evaluated over three datasets using suitable baseline counterparts. From various experimental studies, it is observed that combining both textual and structural views can achieve better performance of sentiment classification tasks than its counterparts.