Deep expectation-maximization network for unsupervised image segmentation and clustering

We propose a novel approach to investigate and implement unsupervised image content understanding and segmentation of color industrial images like medical imaging, forensic imaging, security and surveillance imaging, biotechnical imaging, biometrics, mineral and mining imaging, material science imaging, and many more. In this particular work, our focus will be on medical images only. The aim is to develop a computer aided diagnosis (CAD) system based on a newly developed Multidimensional Spatially Variant Finite Mixture Model (MSVFMM) using Markov Random Fields (MRF) Model. Unsupervised means automatic discovery of classes or clusters in images rather than generating the class or cluster descriptions from training image sets. The aim of this work is to produce precise segmentation of color medical images on the basis of subtle color and texture variation. Finer segmentation of images has tremendous potential in medical imaging where subtle information related to color and texture is required to analyze the image accurately. In this particular work, we have used CIE-Luv and Daubechies wavelet transforms as color and texture descriptors respectively. Using the combined effect of a CIE-Luv color model and Daubechies transforms, we can segment color medical images precisely in a meaningful manner. The evaluation of the results is done through comparison of the segmentation quality with another similar alternative approach and it is found that the proposed approach is capable of producing more faithful segmentation.

Segmentation is a low-level processing aimed at the partition of an image in disjoint regions, each one homogeneous with respect to some properties like intensity, texture, shape, etc. Such a task is needed in many high-level processing and applications in such diverse fields as remote-sensing, medical imaging, video coding, and industrial automation, just to name a few. In this work of thesis, we first deal with the problem of color based image segmentation, that is based on some form of homogeneity in the color/spectral properties of the image, and then move to the more complex task of texture based segmentation, where the aim is to recognize complex structures in the image which are typically non homogeneous in terms of spectral properties. Concerning color based segmentation, we focus on a specific class of Markov Random Field (MRF) probability models, namely the Tree-Structured Markov Random Fields (TS-MRF), recently proposed to provide a hierarchical multiscale description of images. Based on such a model, the unsupervised image segmentation is carried out by means of a sequence of nested class splits, where each class is modeled as a local binary MRF. As a first contribution, we first extend the model to generic tree structures removing the unnecessary binary constraint, and then devise a new TS-MRF unsupervised segmentation technique which improves upon the original algorithm by selecting a better tree structure and eliminating spurious classes. Such results are obtained by using the Mean-Shift procedure to estimate the number of pdf modes at each node, and to obtain a more reliable initial clustering for subsequent MRF optimization. To this end, we devise a new reliable and fast clustering algorithm based on the Mean-Shift technique, which makes use of a variable-bandwidth strategy based on the k-Nearest Neighbors (k-NN) technique, and is implemented with a speed-up strategy that cuts significantly the computational complexity. Experimental results on synthetic data and real life images from the remote sensing domain prove the potential of the proposed method. Turning to texture based segmentation, we present a novel multiscale texture model, and a related algorithm for the unsupervised segmentation of color images. Elementary textures are characterized by their spatial interactions with neighboring regions along selected directions. Such interactions are modeled in turn by means of a set of Markov chains, one for each direction, whose parameters are collected in a feature vector that synthetically describes the texture. Based on the feature vectors, the texture are then recursively merged, giving rise to larger and more complex textures, which appear at different scales of observation: accordingly, the model is named Hierarchical Multiple Markov Chain (H-MMC). The Texture Fragmentation and Reconstruction (TFR) algorithm, addresses the unsupervised segmentation problem based on the H-MMC model. The ``fragmentation'' step allows one to find the elementary textures of the model, while the ``reconstruction'' step defines the hierarchical image segmentation based on a probabilistic measure (texture score) which takes into account both region scale and inter-region interactions. The performance of the proposed method was assessed through the Prague segmentation benchmark, based on mosaics of real natural textures, and also tested on real-world natural images from the Berkeley dataset. Finally, an application of the TFR algorithm for the unsupervised segmentation of multiresolution remotely sensed images is proposed, based on the significant presence of textural content in last generation high-resolution satellite images. The obtained results further assess the potential of the proposed method for such real life applications.

This work proposes a fast background learning algorithm for foreground detection under changing illumination. Gaussian Mixture Model (GMM) is an effective statistical model in background learning. We first focus on Titterington's online EM algorithm that can be used for real-time unsupervised GMM learning, and then advocate a deterministic data assignment strategy to avoid Bayesian computation. The color of the foreground is apt to be influenced by the environmental illumination that usually produce undesirable effect for GMM updating, however, a collinear feature of pixel intensity under changing light is discovered in RGB color space. This feature is afterward used as a reliable clue to decide which part of mixture to update under changing light. A foreground detection step proposed in early version of this work is employed to extract foreground objects by comparing the estimated background model with the current video frame. Experiments have shown the proposed method is able to achieve satisfactory static background images of scenes as well as is also superior to some mainstream methods in detection performance under both indoor and outdoor scenes.

The Gaussian mixture model (GMM) is an important background model of background subtraction methods in moving object detection, and is fit to deal with gradual changes of illumination. To handle sharp changes, hierarchical GMM (HGMM) is proposed as a generic solution which uses state models without temporal correlation on different scales. A new on-line EM algorithm is devised to model new states quickly and accurately. Experiments show that the presented method brings fast adaptation to sharp changes of illumination.

A novel neural network-based technique for segmentation of single-channel magnetic resonance images is presented. The segmentation of single-channel magnetic resonance images is a daunting task due to the relatively little information available at each pixel site. The proposed algorithm is based on unsupervised clustering by means of a Kohonen Self-Organizing Map: unsupervised segmentation algorithms are highly desirable in order to eliminate intra- and interobserver variability. Particular attention has been devoted to the choice of suitable features, in order to ensure an accurate and reliable segmentation: in particular, a feature set extracted from the neighborhood of each pixel has been evaluated. The proposed technique has been tested on simulated magnetic resonance images to assess its stability against the presence of noise and intensity inhomogeneities. Moreover, it has been tested on real magnetic resonance images of both volunteers and brain tumor patients. The preliminary results presented make the proposed technique a promising alternative for the segmentation of single-channel magnetic resonance images and encourage further investigation.

A novel color image segmentation method based on finite Gaussian mixture model is proposed in this paper. First, we use EM algorithm to estimate the distribution of input image data and the number of mixture components is automatically determined by MML criterion. Then the segmentation is carried out by clustering each pixel into appropriate component according to maximum likelihood (ML) criterion. The advantage of our method lies in its ability of less relying on initialization and segmenting images in a totally unsupervised manner. Experimental results show that our segmentation method can obtain better results than other methods.

Masked graph autoencoder (MGAE) has emerged as a promising self-supervised graph pre-training (SGP) paradigm due to its simplicity and effectiveness. However, existing efforts perform the mask-then-reconstruct operation in the raw data space as is done in computer vision (CV) and natural language processing (NLP) areas, while neglecting the important non-Euclidean property of graph data. As a result, the highly unstable local structures largely increase the uncertainty in inferring masked data and decrease the reliability of the exploited self-supervision signals, leading to inferior representations for downstream evaluations. To address this issue, we propose a novel SGP method termed Robust mAsked gRaph autoEncoder (RARE) to improve the certainty in inferring masked data and the reliability of the self-supervision mechanism by further masking and reconstructing node samples in the high-order latent feature space. Through both theoretical and empirical analyses, we have discovered that performing a joint mask-then-reconstruct strategy in both latent feature and raw data spaces could yield improved stability and performance. To this end, we elaborately design a masked latent feature completion scheme, which predicts latent features of masked nodes under the guidance of high-order sample correlations that are hard to be observed from the raw data perspective. Specifically, we first adopt a latent feature predictor to predict the masked latent features from the visible ones. Next, we encode the raw data of masked samples with a momentum graph encoder and subsequently employ the resulting representations to improve the predicted results through latent feature matching. Extensive experiments on seventeen datasets have demonstrated the effectiveness and robustness of RARE against state-of-the-art (SOTA) competitors across three downstream tasks. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/WxTu/RARE</uri> .

This thesis investigates the integration of machine learning techniques into computational fluid dynamics (CFD) to improve simulation accuracy and computational performance. It proposes the use of the Gaussian Mixture Model (GMM) and unsupervised clustering as novel tools for mesh adaptation and flow feature detection. The methodology applies GMM to segment the computational domain into regions with distinct physical behaviour, viscous/turbulent and inviscid/irrotational. This clustering-based p-adaptation approach removes the need for predefined thresholds, enabling a fully automated and localised adjustment of the solution's polynomial degree. As a result, mesh refinement becomes more efficient and precise. A series of numerical experiments, including simulations of flow past a cylinder at varying Reynolds numbers, demonstrate the ability of the method to balance computational cost and solution fidelity. The framework is also tested on real-world engineering cases, such as wind turbine wake simulations, confirming its scalability and robustness. The results show substantial gains in efficiency and adaptability, establishing this approach as a promising solution for a wide range of CFD applications. By embedding machine learning into high-order solvers, this work advances the state-of-the-art and opens new directions for adaptive simulation strategies. It also lays the groundwork for future research in error-driven refinement, 3D turbulence modeling, and the broader application of AI in fluid mechanics. Keywords: Computational fluid dynamics, machine learning, Gaussian Mixture Model, p-adaptation, flow feature detection, mesh refinement, high-order methods. RESUMEN Esta tesis examina la integración de técnicas de aprendizaje automático en la dinámica de fluidos computacional (CFD) para mejorar la precisión de las simulaciones y reducir el coste computacional. Se propone el uso del Modelo de Mezclas Gaussianas (GMM) y métodos de agrupamiento no supervisado para desarrollar nuevas estrategias de adaptación de malla y detección de regiones del flujo sin recurrir a umbrales predefinidos. La metodología aplica GMM para segmentar automáticamente el dominio en regiones con diferentes características físicas, como zonas viscosas/turbulentas y no viscosas/irrotacionales. Esta clasificación guía un proceso de p-adaptación, ajustando el orden polinomial de la solución numérica de forma localizada, lo que permite un refinamiento automático, preciso y eficiente. Diversos experimentos numéricos, incluyendo simulaciones del flujo alrededor de un cilindro para distintos números de Reynolds, demuestran que el enfoque propuesto equilibra de manera eficaz el coste computacional y la calidad de la solución. Además, su aplicabilidad se valida en casos reales como las estelas de aerogeneradores, evidenciando la escalabilidad del método. Los resultados confirman que estas técnicas mejoran la eficiencia y adaptabilidad de los solucionadores de alto orden, lo que las convierte en herramientas prometedoras para aplicaciones complejas en CFD. La tesis contribuye así al estado del arte al integrar aprendizaje automático con métodos numéricos avanzados, y sienta las bases para investigaciones futuras sobre refinamiento adaptativo basado en errores, simulaciones tridimensionales de turbulencia y uso extendido de IA en mecánica de fluidos. Palabras clave: dinámica de fluidos computacional, aprendizaje automático, Modelo de Mezclas Gaussianas, p-adaptación, detección de características, refinamiento de mallas, métodos de alto orden. RÉSUMÉ Cette thèse explore l’intégration des techniques d’apprentissage automatique dans la dynamique des fluides numérique (CFD) afin d’améliorer la précision des simulations et la surcharge de calcul. En utilisant le modèle de mélanges gaussiens (GMM) et des méthodes de regroupement non supervisé, cette recherche introduit des approches innovantes pour l’adaptation du maillage et la détection des structures d’écoulement. La méthodologie se concentre sur l’application du GMM pour partitionner les champs d’écoulement en régions distinctes, régions distinctes, différenciant les zones visqueuses/turbulentes des zones inviscides/irrotationnelles. Cette technique de p-adaptation basée sur le regroupement élimine le besoin de seuils prédéfinis, permettant ainsi un raffinement du maillage automatisé, efficace et précis. De nombreuses expériences numériques sur divers cas tests, notamment l’écoulement autour d’un cylindre à différents nombres de Reynolds, démontrent la capacité de la méthode proposée à équilibrer le coût computationnel et la fidélité de la solution. Des applications en ingénierie, telles que la simulation du sillage d’une éolienne, illustrent davantage l’évolutivité des méthodes proposées. Les résultats montrent des améliorations significatives en termes du coût computationnel et d’adaptabilité, rendant ces techniques particulièrement adaptées à un large éventail d’applications en CFD. Ce travail contribue à l’état de l’art en intégrant de manière fluide l’apprentissage automatique dans les solveurs CFD d’ordre élevé, ouvrant ainsi la voie à de futures avancées dans les méthodes de simulation adaptative. Les méthodologies proposées établissent une base pour l’exploration du raffinement basé sur l’erreur, la modélisation tridimensionnelle de la turbulence et l’adoption plus large de l’apprentissage automatique dans la recherche en dynamique des fluides.

Anomaly detection in videos is challenging due to the scarcity and variance in positive samples. Current anomaly detection methods can be categorized into reconstruction models and future frame prediction-based models. However, reconstruction models might be exceptionally adapted to abnormal events due to the learning capacity and generalization ability of deep neural networks, whereas prediction-based methods can be sensitive to noise. In this study, we propose an anomaly detection model based on the latent feature space, which combines advantages from both sides. We argue that the constraints in the latent feature space can promote reconstruction; moreover, the optical flow is also considered to introduce temporal information. We use SPyNet for accurate and efficient optical flow estimation. We extensively validate our method on the UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The results demonstrated the feasibility of the proposed method and the benefit of utilizing information in the latent feature space.

Unsupervised deep learning and semi-automatic data labeling in weed discrimination

The traditional multi-resolution Markov random field (MRMRF) model uses two-component Markov random field model on each resolution, and requires training data to estimate the necessary model parameters, which is unsuitable for unsupervised image segmentation. Under this circumstance, a new multi-resolution Markov random field model with variable potential for unsupervised texture image segmentation is presented. The new model solves this problem by introducing a variable potential function for multi-level logistic distribution (MLL) model on each scale. Using this method, the new model can automatically estimate model parameters and produce accurate unsupervised segmentation results. The results obtained on synthetic texture images and remote sensing images demonstrate that a better segmentation is achieved by our model than the traditional MRMRF model.

Fuzzy C-Means (FCM) clustering and Gaussian Mixture Model (GMM) are two popular tools for data processing. In this study the unsupervised algorithm combining FCM clustering in the Kernel Space (KFCM) and GMM is presented. First, a trick is applied to the FCM algorithm. Then, the number of clusters is chosen automatically in the kernel space. On the basis of obtained starting parameters, i.e. number of mixture components, mean vector, covariance matrices and mixing proportion coefficients, the final GMM parameters are estimated. For this estimation the Expectation Maximization (EM) algorithm is used. The presented methodology - combination of KFCM and GMM methods named unsupervised cascade clustering - constitutes the basic step in Ewing's sarcoma segmentation. On this basis the voxels intensity values describing segmented tumour and surrounded healthy tissue are defined and fuzzy connectedness analysis is performed. The obtained mixture parameters estimation results are compared with the results obtained using two different methods described in literature.

Conventional unsupervised image segmentation methods use color and geometric information and apply clustering algorithms over pixels. They preserve object boundaries well but often suffer from over-segmentation due to noise and artifacts in the images. In this paper, we contribute on a preprocessing step for image smoothing, which alleviates the burden of conventional unsupervised image segmentation and enhance their performance. Our approach relies on a convolutional autoencoder (CAE) with the total variation loss (TVL) for unsupervised learning. We show that, after our CAE-TVL preprocessing step, the over-segmentation effect is significantly reduced using the same unsupervised image segmentation methods. We evaluate our approach using the BSDS500 image segmentation benchmark dataset and show the performance enhancement introduced by our approach in terms of both increased segmentation accuracy and reduced computation time. We examine the robustness of the trained CAE and show that it is directly applicable to other natural scene images.

The demand for accurate and reliable unsupervised image segmentation methods is high. Regardless of whether we are faced with a problem for which we do not have a usable training dataset, or whether it is not possible to obtain one, we still need to be able to extract the desired information from images. In such cases, we are usually gently pushed towards the best possible clustering method, as it is often more robust than simple traditional image processing methods. We investigate the usefulness of combining two clustering methods for unsupervised image segmentation. We use the mixture models to extract the color and spatial image features based on the obtained output segments. Then we construct a similarity matrix (adjacency matrix) based on these features to perform spectral clustering. In between, we propose a label noise correction using Markov random fields. We investigate the usefulness of our method on many hand-crafted images of different objects with different shapes, colorization, and noise. Compared to other clustering methods, our proposal performs better, with 10% higher accuracy. Compared to state-of-the-art supervised image segmentation methods based on deep convolutional neural networks, our proposal proves to be competitive.

Non-Gaussian triplet Markov fields (TMF) model is suitable for dealing with multi-class segmentation of non-stationary and non-Gaussian synthetic aperture radar (SAR) images. Considering the complexity of the model and algorithm, as well as the requirement of real-time, and robust and efficient processing of SAR images, a fast algorithm based on TMF for unsupervised multi-class segmentation of SAR images is proposed in this paper. For the speckle noise in SAR images, numerical characteristic, threshold selection and QuadTree decomposition criterion are researched firstly. With the new method, a SAR image can quickly be mapped into an edge-based pixon-representation, which results in a coarse decomposition in smooth regions, and a fine decomposition in edges. Then by combining TMF model with the pixon-representation of SAR image, a new potential energy function of TMF based on pixon-representation is derived. Finally, the segmentation is finished by Bayesian maximum posterior mode (MPM). The effectiveness of the fast TMF algorithm is demonstrated by applying it to simulated data and real SAR images.