Improvement of Attentional Mechanisms for Unsupervised Learning Models

IntroductionIn swine disease surveillance, obtaining labeled data for supervised learning models can be challenging because many farms lack standardized diagnostic routines and consistent health monitoring systems. Unsupervised learning is particularly suitable in such scenarios because it does not require labeled data, allowing for detecting anomalies without predefined labels. This study evaluates the effectiveness of unsupervised machine learning models in detecting anomalies in productivity indicators in swine breeding herds.MethodsAnomalies, defined as deviations from expected patterns, were identified in indicators such as abortions per 1000 sows, prenatal losses, preweaning mortality, total born, liveborn, culled sows per 1000 sows, and dead sows per 1000 sows. Three unsupervised models - Isolation Forest, Autoencoder, and K-Nearest Neighbors (KNN) - were applied to data from two swine production systems. The herd-week was used as the unit of analysis, and anomaly scores above the 75th percentile were used to flag anomalous weeks. A permutation test assessed differences between anomalous and non-anomalous weeks. Performance was evaluated using F1-score, precision, and recall, with true anomalous weeks defined as those coinciding with reported health challenges, including porcine reproductive and respiratory syndrome (PRRS) and Seneca Valley virus outbreaks. A total of 8,044 weeks were analyzed.ResultsThe models identified 336 anomalous weeks and 1,008 non-anomalous weeks in Production System 1, and 1,675 anomalous weeks and 5,025 non-anomalous weeks in Production System 2. The results from the permutation test revealed significant differences in productivity indicators between anomalous and non-anomalous weeks, especially during PRRS outbreaks, with more subtle changes observed during Seneca Valley virus outbreaks. The models performed well in detecting the PRRSV anomaly, achieving perfect precision (100%) across all models for both production systems. For anomalies like SVV the models showed lower performance compared to PRRSV.DiscussionThese findings suggest that unsupervised machine learning models are promising tools for early disease detection in swine herds, as they can identify anomalies in productivity data that may signal health challenges.

High-risk neuroblastoma is one of the most difficult to treat pediatric cancers, with a survival rate of only about 50% and significant long-term consequences from current chemotherapy. Preclinical models such as cell lines and mice are the backbone of drug development and experimental-mechanistic oncology, but the development of new treatments is hampered in part by a lack of understanding of the direct clinical relevance of such data collected in preclinical models. Despite this, few formal methods have been developed to determine how these various models represent/resemble primary patient tumors. Here, we present the first comprehensive single-cell RNA-seq analysis of neuroblastoma across an extensive cohort of patient tumors and a variety of preclinical model systems (n = 126 total samples assembled - the largest cohort of its kind). By building an innovative unsupervised machine learning method, which we term “automatic consensus nonnegative matrix factorization” (acNMF), we have integrated and contrasted the transcriptional landscapes of patient tumors with those of cell lines, patient-derived xenografts (PDX), and genetic mouse models (GEMM).Using these tools, we discovered the dominant adrenergic gene expression programs found in neuroblastoma patient tumors were preserved across all preclinical models. However, the presumptive chemo-resistant mesenchymal-like programs, while identifiable in cell lines, were primarily restricted to subpopulations of cancer-associated fibroblasts and Schwann-like cells in vivo. Surprisingly, a mesenchymal-like program could be acutely chemotherapy-induced in GEMM and was evident in pre-treated patient and PDX samples, suggesting a previously uncharacterized mechanism of therapy escape. In addition to these core findings, our computational tools were able to further delineate the classical neuroblastoma adrenergic and mesenchymal gene expression programs, discovering for example, novel subpopulations of cancer associated fibroblasts. These behaviors were conserved across tumors and most preclinical models, which we validated by RNA in situ hybridization, which is a high resolution ultra sensitive spatial transcriptomics technology. Our work cautions against overreliance on traditional preclinical models without recognizing their limitations and we offer a nuanced, high-resolution view of neuroblastoma pre-clinical systems for advancing therapeutic development. We have launched an open-source web resource, featuring this integrated map to aid the scientific community in further exploration of these data and hypothesis generation (available at http://pscb.stjude.org). Citation Format: Rich Chapple, Xueying Liu, Sivaraman Natarajan, Margaret I. Alexander, Yuna Kim, Anand Patel, Christy W. LaFlamme, Min Pan, William C. Wright, Hyeong-Min Lee, Yinwen Zhang, Meifen Lu, Selene C. Koo, Courtney Long, John Harper, Chandra Savage, Melissa D. Johnson, Thomas Confer, Walter J. Akers, Michael A. Dyer, Heather Sheppard, John Easton, Paul Geeleher. A novel unsupervised machine learning model applied to neuroblastoma single-cell RNA-seq data reveals a drug-induced mesenchymal-like gene expression program [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 868.

This paper proposes a scheme for analysing speech data inspired by the concept of working memory, it is uses wavelet analysis and unsupervised learning models. The scheme relies on splitting a sound stream in arbitrary chunks and producing feature streams by sequentially analysing each chunk with time-frequency methods. The purpose of this is to precisely detect the time of transitions as well as length of stable acoustic units that occur between them. The procedure uses two feature extraction stages to analyse the audio chunk and two types of unsupervised machine learning models, hierarchical clustering and Self-Organising Maps. The first pass looks at the whole chunk piece by piece looking for speech and silence parts, this stage takes the root mean square, the arithmetic mean, standard deviation from the samples of each piece and classifies the features using hierarchical clustering into speech and non-speech clusters. The second pass looks for stable patterns and transitions at the locations inferred from the results of the first pass, this step uses Harmonic and Daubechies wavelets for coefficient extraction. After the analysis procedures have been completed the chunk advances 2 seconds, the transient and stable feature vectors are saved within SOMs and a new cycle begins on a new chunk.

It is challenging to predict which patients who meet criteria for subcortical ischemic vascular disease (SIVD) will ultimately progress to subcortical vascular cognitive impairment (SVCI). We collected clinical information, neuropsychological assessments, T1 imaging, diffusion tensor imaging, and resting-state functional magnetic resonance imaging from 83 patients with SVCI and 53 age-matched patients with SIVD without cognitive impairment. We built an unsupervised machine learning model to isolate patients with SVCI. The model was validated using multimodal data from an external cohort comprising 45 patients with SVCI and 32 patients with SIVD without cognitive impairment. The accuracy, sensitivity, and specificity of the unsupervised machine learning model were 86.03%, 79.52%, and 96.23% and 80.52%, 71.11%, and 93.75% for internal and external cohort, respectively. We developed an accurate and accessible clinical tool which requires only data from routine imaging to predict patients at risk of progressing from SIVD to SVCI. Our unsupervised machine learning model provides an accurate and accessible clinical tool to predict patients at risk of progressing from subcortical ischemic vascular disease (SIVD) to subcortical vascular cognitive impairment (SVCI) and requires only data from imaging routinely used during the diagnosis of suspected SVCI. The model yields good accuracy, sensitivity, and specificity and is portable to other cohorts and to clinical practice to distinguish patients with SIVD at risk for progressing to SVCI. The model combines assessment of diffusion tensor imaging and functional magnetic resonance imaging measures in patients with SVCI to analyze whether the "disconnection hypothesis" contributes to functional and structural changes and to the clinical presentation of SVCI.

The development of Question Answering (QA) systems has become important because it reveals research issues that require insight from a variety of disciplines, including Artificial Intelligence, Information Extraction, Natural Language Processing, and Psychology. Our goal here is to develop a QA approach to answer yes/no questions relevant to civil laws in legal bar exams. A bar examination is intended to determine whether a candidate is qualified to practice law in a given jurisdiction. We have found that the development of a QA system for this task provides insight into the challenges of formalizing reasoning about legal text, and about how to exploit advances in computational linguistics. We separate our QA approach into two steps. The first step is to identify legal documents relevant to the exam questions; the second step is to answer the questions by analyzing the relevant documents. In our initial approach described here, the first step has been already solved for us: the appropriate articles for each question have been identified by legal experts. So here, we focus on the second task, which can be considered as a form of Recognizing Textual Entailment (RTE), where input to the system is a question sentence and its corresponding civil law article(s), and the output is a binary answer: whether the question sentence is entailed from the article(s). We propose a hybrid method, which combines simple rules and an unsupervised learning model using deep linguistic features. We first construct a knowledge base for negation and antonym words for the legal domain. We then identify potential premise and conclusion components of input questions and documents, based on text patterns and separating commas. We further classify the questions into easy and difficult ones, and develop a two-phase method for answering yes/no questions. We answer easy questions by negation/antonym detection. For more difficult questions, we adapt an unsupervised machine learning method based on morphological, syntactic, and lexical semantic analysis on identified premises and conclusions. This provides the basis to compare the semantic correlation between a question and a legal article. Our experimental results show reasonable performance, which improves the baseline system, and outperforms an SVM-based supervised machine learning model.

This study aimed to evaluate the prognosis of patients with COVID-19 and hypertension who were treated with angiotensin-converting enzyme inhibitor (ACEI)/angiotensin receptor B (ARB) drugs and to identify key features affecting patient prognosis using an unsupervised learning method. A large-scale clinical dataset, including patient information, medical history, and laboratory test results, was collected. Two hundred patients with COVID-19 and hypertension were included. After cluster analysis, patients were divided into good and poor prognosis groups. The unsupervised learning method was used to evaluate clinical characteristics and prognosis, and patients were divided into different prognosis groups. The improved wild dog optimization algorithm (IDOA) was used for feature selection and cluster analysis, followed by the IDOA-k-means algorithm. The impact of ACEI/ARB drugs on patient prognosis and key characteristics affecting patient prognosis were also analysed. Key features related to prognosis included baseline information and laboratory test results, while clinical symptoms and imaging results had low predictive power. The top six important features were age, hypertension grade, MuLBSTA, ACEI/ARB, NT-proBNP, and high-sensitivity troponin I. These features were consistent with the results of the unsupervised prediction model. A visualization system was developed based on these key features. Using unsupervised learning and the improved k-means algorithm, this study accurately analysed the prognosis of patients with COVID-19 and hypertension. The use of ACEI/ARB drugs was found to be a protective factor for poor clinical prognosis. Unsupervised learning methods can be used to differentiate patient populations and assess treatment effects. This study identified important features affecting patient prognosis and developed a visualization system with clinical significance for prognosis assessment and treatment decision-making.

The ultimate goal of precision medicine is to determine right treatment for right patients based on precise diagnosis. To achieve this goal, correct stratification of patients using molecular features and clinical phenotypes is crucial. During the long history of medical science, our understanding on disease classification has been improved greatly by chemistry and molecular biology. Nowadays, we gain access to large scale patient-derived data by high-throughput technologies, generating a greater need for data science including unsupervised learning and network modeling. Unsupervised learning methods such as clustering could be a better solution to stratify patients when there is a lack of predefined classifiers. In network modularity analysis, clustering methods can be also applied to elucidate the complex structure of biological and disease networks at the systems level. In this review, we went over the main points of clustering analysis and network modeling, particularly in the context of Traditional Chinese medicine (TCM). We showed that this approach can provide novel insights on the rationale of classification for TCM herbs. In a case study, using a modularity analysis of multipartite networks, we illustrated that the TCM classifications are associated with the chemical properties of the herb ingredients. We concluded that multipartite network modeling may become a suitable data integration tool for understanding the mechanisms of actions of traditional medicine.

A continuous supply of electricity is necessary to maintain an acceptable standard of life, and the power distribution system's overhead line components play a crucial role in this matter. In Pakistan, identifying defective parts often necessitates human involvement. An unmanned aerial vehicle was used to gather a collection of 10,343 photos to automate this procedure. Using supervised and unsupervised machine learning methods, a number of automated anomaly detection systems were created. Support vector machine, random forest, VGG16, and ResNet50 were used as supervised machine learning models, and a convolutional auto‐encoder was used as the unsupervised machine learning model. VGG16 achieved the best accuracy of 99.00% while random forest achieved the worst accuracy of 72.49%. The convolutional auto‐encoder was successful in distinguishing between normal and abnormal components. The aforementioned machine learning models can be put on unmanned aerial vehicles to immediately identify defective parts.

In unsupervised learning of image features without labels, especially on datasets with fine-grained object classes, it is often very difficult to tell if a given image belongs to one specific object class or another, even for human eyes. However, we can reliably tell if image C is more similar to image A than image B. In this work, we propose to explore how this relative order can be used to learn discriminative features with an unsupervised metric learning method. Instead of resorting to clustering or self-supervision to create pseudo labels for an absolute decision, which often suffers from high label error rates, we construct reliable relative orders for groups of image samples and learn a deep neural network to predict these relative orders. During training, this relative order prediction network and the feature embedding network are tightly coupled, providing mutual constraints to each other to improve overall metric learning performance in a cooperative manner. During testing, the predicted relative orders are used as constraints to optimize the generated features and refine their feature distance-based image retrieval results using a constrained optimization procedure. Our experimental results demonstrate that the proposed relative orders for unsupervised learning (ROUL) method is able to significantly improve the performance ofunsupervised deep metric learning.

Recently, many scholars have used attention mechanisms to achieve excellent performance results on various neural network applications. However, the attention mechanism also has shortcomings. Firstly, the high computational and storage consumption makes the attention mechanism difficult to apply on long sequences. Second, all tokens are involved in the computation of the attention map, which may increase the influence of noisy tokens on the results and lead to poor training results. Due to these two shortcomings, attention models are usually strictly limited to sequence length. Further, attention models have difficulty in exploiting their excellent properties for modelling long sequences. To solve the above problems, an efficient sparse attention mechanism (SSA) is proposed in this paper. SSA is based on two separate layers: the local layer and the global layer. These two layers jointly encode local sequence information and global context. This new sparse-attention patterns is powerful in accelerating reasoning. The experiments in this paper validate the effectiveness of the SSA mechanism by replacing the self-attentive structure with an SSA structure in a variety of transformer models. The SSA attention mechanism has achieved state-of-the-art performance on several major benchmarks. SSA was validated on a variety of datasets and models encompassing language translation, language modelling and image recognition. With a small improvement in accuracy, the inference calculation speed was increased by 24%.KeywordsTransformersSequenceLocal attentionGlobal attentionSparsity

Extreme learning machine (ELM) as well as its variants have been widely used in many fields for its good generalization performance and fast learning speed. Though distributed ELM can sufficiently process large-scale labeled training data, the current technology is not able to process partial labeled or unlabeled training data. Therefore, we propose a new unified distributed ELM with supervised, semi-supervised and unsupervised learning based on MapReduce framework, called U-DELM. The U-DELM method can be used to overcome the existing distributed ELM framework’s lack of ability to process partially labeled and unlabeled training data. We first compare the computation formulas of supervised, semi-supervised and unsupervised learning methods and found that the majority of expensive computations are decomposable. Next, MapReduce framework based U-DELM is proposed, which extracts three different matrices continued multiplications from the three computational formulas introduced above. After that, we transform the cumulative sums respectively to make them suitable for MapReduce. Then, the combination of the three computational formulas are used to solve the output weight in three different learning methods. Finally, by using benchmark and synthetic datasets, we are able to test and verify the efficiency and effectiveness of U-DELM on learning massive data. Results prove that U-DELM can achieve unified distribution on supervised, semi-supervised and unsupervised learning.

The advent of gravitational wave astronomy (GW) has revolutionized the observation of cataclysmic cosmic events, such as black hole mergers and neutron star collisions. The Laser Interferometer Gravitational-Wave Observatory (LIGO) has been at the forefront of these discoveries. However, the immense volume and complexity of gravitational wave data present significant challenges for traditional analysis methods. This paper investigates the growing synergy between artificial intelligence (AI) and GW science, emphasizing how AI enhances signal detection, noise reduction, and data interpretation. It begins with an overview of GW fundamentals and the role of machine learning in increasing detector sensitivity. Notable GW events observed by LIGO are discussed alongside persistent analytical challenges such as data quality, generalization, and computational constraints. A comprehensive performance review of AI techniques, including supervised learning, unsupervised learning, deep learning, and reinforcement learning is presented based on data spanning 2021 to 2024. Evaluation metrics include accuracy, precision, true positive rate (TPR), false positive rate (FPR), and computational efficiency. Findings indicate that deep learning and supervised learning outperform other approaches, particularly in enhancing TPR and minimizing FPR. While unsupervised and reinforcement learning models offer less precision, they demonstrate high efficiency and potential for real-time applications. The study also explores AI’s integration into next-generation detectors and waveform reconstruction techniques. Overall, the integration of AI into GW research significantly improves the reliability and speed of event detection, unlocking new possibilities for exploring the dynamic universe. This paper provides a comprehensive outlook on the transformative role of AI in shaping the future of GW astronomy.

Remote sensing framework for geological mapping via stacked autoencoders and clustering

Nowadays word embeddings, also known as word vectors, play an important role for many natural language processing (NLP) tasks. In general, these word embeddings are learned from unsupervised learning models (e.g. Word2Vec, GloVe) with a large unannotated corpus and they are independent with the task of their application. In this paper we aim to enrich word embeddings by adding more information from a specific task that is the aspect based sentiment analysis. We propose a model using a convolutional neural network that takes a labeled data set, the learned word embeddings from an unsupervised learning model (e.g. Word2Vec) as input and fine-tunes word embeddings to capture aspect category and sentiment information. We conduct experiments on restaurant review data (http://spidr-ursa.rutgers.edu/datasets/). Experimental results show that fine-tuned word embeddings outperform unsupervisedly learned word embeddings.

Artificial Intelligence (AI) has advanced in a rapid and exponential manner and AI is now revolutionizing the medical field, the development of self-driving cars, and the management of financial operations. A major reason for such achievement is the efficiency and performance of AI models, which can be drastically boosted through new learning methods. One of the most promising avenues is the unsupervised representation learning method that also offers the choice of leaping over the traditional supervised learning. Instead of supervised learning, where labeled data is the base upon which training of models takes place, unsupervised learning allows AI systems to get new insights out of raw, unlabeled data without any human intervention. This method teaches the AI system how to represent the data in an organized manner, thus enabling it to uncover hidden features and relations without anyone helping it. In effect, AI models, powered by unsupervised representation learning, can be quite successful in areas like clustering, anomaly detection, and feature extraction, frequently beating traditional methods in terms of their speed and accuracy. .The skill of finding deep structures in a dataset can unexpectedly influence many areas of the sciences, e.g., it can help us create a better diagnostic system in medicine or use it for decision-making when investing in the stock market