High-throughput sequencing technologies have greatly facilitated omics research in the past two decades. An enormous volume of omics data have been generated, with the potential to answer various key biological questions. Unsupervised learning technologies such as cluster analysis serve as a powerful tool to uncover hidden patterns and associations in omics data. However, considering the complex natural structure and special characteristics of omics data, it is usually inappropriate or inefficient to directly apply clustering methods designed for general purposes. To address these issues, we develop several clustering and biclustering algorithms specifically designed for omics data. %In this dissertation, we tried to address these concerns from multiple different aspects: The first project is about Poisson hurdle model-based clustering, which aims to deal with extra sparsity in microbiome count data. Extra sparsity is a huge problem when dealing with microbiome data and some other omics data. In this study, we present a novel model-based clustering framework based on Poisson hurdle models, designed specifically for the intricacies of sparse microbiome data. We describe an expectation-maximization algorithm and a modified version using simulated annealing to conduct the cluster analysis. Additionally, we offer algorithms for intuitive initialization and determining the optimal number of clusters. Simulation results demonstrate that our proposed methods provide better clustering results than alternative methods under a variety of settings. A practical application of our method to a sorghum rhizosphere microbiome dataset yielded intriguing biological insights. The second project is a Dirichlet-Multinomial biclustering algorithm for compositional microbiome data. Similar to (one-way) cluster analysis, biclustering is a popular tool to study the interactions between microbial sub-communities and environmental samples. In this project, we introduce a Dirichlet-Multinomial (DM) model-based checkerboard biclustering technique tailored for the simultaneous clustering of microbiome features and samples. This approach assumes the presence of a DM distribution mixture within each microbiome sample, and utilizes a combination of Expectation-Maximization and coordinate descent algorithm to solve for optimal parameters. Compared to conventional, general-purpose biclustering algorithms, our method consistently demonstrated superior clustering accuracy and enriched biological interpretation across diverse simulation settings and multiple real datasets. The third project studied and compared the effect of several feature selection techniques implemented in the biclustering framework. When dealing with biological data, usually tens of thousands of features are present in a single dataset, but most of them are likely non-informative and do not benefit our analysis. Including them in the analysis can be computationally inefficient and lead to overfitting issues. This project gives a review of implementing penalization methods in the biclustering framework to do feature selection. Penalization/regularization are broadly used in regression analysis, and have also been applied to one-way clustering. This project focuses on extending and applying penalization methods to bicluster analysis of biological data. We show two different classes of penalization methods, one on the with-cluster sum of squares (WCSS) and the other on the feature weights. By introducing sparsity into the feature space, we are able to simultaneously find feature groups/sample subtypes and identify non-informative features. The fourth project focused on clustering analysis using an over-clustering and merging structure. This idea originates from a problem we encountered in the third project: if the true feature classes show a highly imbalanced structure, K-means type of clustering or biclustering methods usually lose power. This is due to the spherical nature of K-means clusters, which lets it perform poorly when the class sizes differ a lot. Motivated by this drawback of K-means, and also by the simple fact that K-means can only produce convex clusters, we developed a novel K-means merging algorithm \emph{CavMerge} that is suitable for producing general-sized and imbalanced clustering results. This method is intuitive, free of parameter tuning, and computationally efficient. It starts from an over-clustered K-means result, and utilizes a property called log-concavity to perform merging. Operating under minimal local distributional assumptions, our algorithm demonstrates strong consistency and guarantees rapid convergence. Empirical studies on various simulated and real datasets demonstrate that our method yields more reliable clusters in comparison to current state-of-the-art algorithms.
Read more