Laplacian eigenmaps for multimodal groupwise image registration

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Image registration is an elementary task in medical image processing and analysis, which can be divided into monomodal and multimodal. Direct 3D multimodal registration in volumetric medical images can provide more insight into the interpretation of subsequent image processing applications than 2D methods. This paper is dedicated to the development of a 3D multimodal image registration algorithm based on a viscous fluid model associated with the Bhattacharyya distance. In our approach, a modified Navier-Stoke's equation is exploited as the foundation of the multimodal image registration framework. The hopscotch method is numerically implemented to solve the velocity field, whose values at the explicit locations are first computed and the values at the implicit positions are solved by transposition. The differential of the Bhattacharyya distance is incorporated into the body force function, which is the main driving force for deformation, to enable multimodal registration. A variety of simulated and real brain MR images were utilized to assess the proposed 3D multimodal image registration system. Preliminary experimental results indicated that our algorithm produced high registration accuracy in various registration scenarios and outperformed other competing methods in many multimodal image registration tasks.Clinical Relevance- This facilitates the disease diagnosis and treatment planning that requires accurate 3D multimodal image registration without massive image data and extensive training regardless of the imaging modality.

Chapter 19 - Dimensionality Reduction and Latent Variable Modeling

Multimodal medical image deformable registration is a critical foundation for liver tumor interventional therapy, assisting doctors in path planning and ablation efficacy evaluation. Research on this topic is still limited, especially regarding the challenges in liver multimodal deformable registration, such as significant image intensity differences and large organ deformations. In this paper, we propose DMCR--a novel multimodal multiscale image registration model based on a dual-stream Mamba with contrastive representation. During the encoding stage, images from different modalities are fed into distinct feature branches constructed using the Mamba architecture. In the registration stage, we propose a multiscale registration approach based on a decoding strategy, wherein the deformation field is progressively propagated and fused across multiple scales. Furthermore, we introduce a modality-invariant contrastive loss to guide the model in capturing intrinsic image features during the encoding stage, while disregarding modality-specific details, thus enhancing the effectiveness of subsequent registration. We trained and tested our model on multimodal liver ablation datasets from three different medical centers. The results demonstrate that our proposed model achieves superior registration performance compared to state-of-the-art registration methods and also exhibits better generalization capability.Clinical relevance- This model effectively performs multi-modal image deformable registration, assisting physicians in facilitating interventional procedures and evaluating multimodal therapeutic outcomes.

Over the past few decades, fast<strong>-</strong>growth has occurred in the area of medical image acquisition devices, and physicians now rely on the utilization of medical images for the diagnosis, treatment plans, and surgical guidance. Researchers have classified medical images according to two structures: anatomical and functional structures. Due to this classification, the data obtained from two or more images of the same object frequently provide complementary and more abundant information through a process known as multimodal medical model registration. Image registration is spatially mapping the coordinate system of the two images obtained from a different viewpoint and utilizing various sensors. Several automatic multimodal medical image registration algorithms have been introduced based on types of medical images and their applications to increase the reliability, robustness, and accuracy. Due to the diversity in imaging and the different demands<strong> </strong>for applications, there is no single registration algorithm that can do that. This paper introduces a novel method for developing a multimodal medical image registration system that can select the most accepted registration algorithm from a group of registration algorithms for a variety of input datasets. The method described here is based on a machine learning technique that selects the most promising candidate. Several experiments have been conducted, and the results reveal that the novel approach leads to considerably faster reliability, accuracy, and more robustness registration algorithm selection.

Systematic comparison of unsupervised feature extraction methods for enhancing discrimination accuracy of brown rice flour adulteration in LIBS

Multi-modal image registration has been a challenging task in medical images because of the complex intensity relationship between images to be aligned. Registration methods often rely on the statistical intensity relationship between the images which suffers from problems such as statistical insufficiency. The proposed registration method works based on extracting structural features by utilizing the complex phase and gradient-based information. By employing structural relationships between different modalities instead of complex similarity measures, the multi-modal registration problem is converted into a mono-modal one. Therefore, conventional mono-modal similarity measures can be utilized to evaluate the registration results. This new registration paradigm has been tested on magnetic resonance (MR) brain images of different modes. The method has been evaluated based on target registration error (TRE) to determine alignment accuracy. Quantitative results demonstrate that the proposed method is capable of achieving comparable registration accuracy compared to the conventional mutual information.

We present an instrument based on commodity embedded hardware, that implements an automatic procedure for early skin-cancer screening using dynamic thermal imaging. The procedure leverages image segmentation in the visible range and real-time multimodal registration to compute the temperature recovery curve (TRC) of suspicious skin lesions using thermal infrared video. The instrument implements two algorithms that infer the malignancy of the lesion from the computed TRCs. The first algorithm assumes that the TRCs are deterministic and infers the malignancy from the distance between the TRC of the suspicious lesion and its surrounding skin, which is assumed to be healthy tissue. The second algorithm models the TRC of the lesion as a random process and uses detection theory to statistically infer its malignancy from the eigenfunctions and corresponding eigenvalues of its covariance function. We built a prototype of the instrument using a Raspberry Pi 3 model B+ board, which acquires a visible-range image of the lesion at the beginning of the procedure and performs image segmentation in 62ms. Operating on a 400×400-pixel region-of-interest within the infrared video, the board performs frame-by-frame multimodal image registration and generates the TRCs in real time at more than 37 frames per second, thus eliminating the need to store video data for off-line processing. The statistical detection algorithm, which yields the best results, runs in 1.07s at the end of the procedure, and achieves a sensitivity of 98% and a specificity of 95% on a dataset of 116 volunteer subjects.

In recent years, a variety of nonlinear dimensionality reduction techniques (NLDR) have been proposed in the literature. They aim to address the limitations of traditional techniques such as PCA and classical scaling. Most of these techniques assume that the data of interest lie on an embedded non-linear manifold within the higher-dimensional space. They provide a mapping from the high-dimensional space to the low-dimensional embedding and may be viewed, in the context of machine learning, as a preliminary feature extraction step, after which pattern recognition algorithms are applied. Laplacian Eigenmaps (LE) is a nonlinear graph-based dimensionality reduction method. It has been successfully applied in many practical problems such as face recognition. However the construction of LE graph suffers, similarly to other graph-based DR techniques from the following issues: (1) the neighborhood graph is artificially defined in advance, and thus does not necessary benefit the desired DR task; (2) the graph is built using the nearest neighbor criterion which tends to work poorly due to the high-dimensionality of original space; and (3) its computation depends on two parameters whose values are generally uneasy to assign, the neighborhood size and the heat kernel parameter. To address the above-mentioned problems, for the particular case of the LPP method (a linear version of LE), L. Zhang et al.1 have developed a novel DR algorithm whose idea is to integrate graph construction with specific DR process into a unified framework. This algorithm results in an optimized graph rather than a predefined one.

Principal component analysis (PCA) is the most widely used dimensionality reduction technique in the biological sciences, and is commonly employed to create 2D visualizations of geometric morphometric data. However, interesting biological information may be lost or misrepresented in these plots due to PCA’s inability to summarize nonlinear dependencies between variables. Nonlinear alternative methods exist, but their effectiveness has never been tested on morphometric data. Here, the performance of PCA on the task of visualizing morphometric variation is compared to four nonlinear techniques: Sammon Mapping, Isomap, Locally Linear Embedding, and Laplacian Eigenmaps. The performance of methods is assessed on the basis of global and local preservation of pairwise distances for a variety of simulated and empirical datasets. The relative performance of PCA varies in function of the distribution of variation, complexity, and size of datasets. Overall, nonlinear methods show superior preservation of small differences between morphologies compared to PCA.

Real-time multi-modal rigid registration based on a novel symmetric-SIFT descriptor

The recent explosion in procurement and availability of high-dimensional gene- and protein-expression profile datasets for cancer diagnostics has necessitated the development of sophisticated machine learning tools with which to analyze them. A major limitation in the ability to accurate classify these high-dimensional datasets stems from the 'curse of dimensionality', occurring in situations where the number of genes or peptides significantly exceeds the total number of patient samples. Previous attempts at dealing with this issue have mostly centered on the use of a dimensionality reduction (DR) scheme, Principal Component Analysis (PCA), to obtain a low-dimensional projection of the high-dimensional data. However, linear PCA and other linear DR methods, which rely on Euclidean distances to estimate object similarity, do not account for the inherent underlying nonlinear structure associated with most biomedical data. The motivation behind this work is to identify the appropriate DR methods for analysis of high-dimensional gene- and protein-expression studies. Towards this end, we empirically and rigorously compare three nonlinear (Isomap, Locally Linear Embedding, Laplacian Eigenmaps) and three linear DR schemes (PCA, Linear Discriminant Analysis, Multidimensional Scaling) with the intent of determining a reduced subspace representation in which the individual object classes are more easily discriminable.

In many applications, one would like to perform calculations on smooth manifolds of dimension d embedded in a high-dimensional space of dimension D. Often, a continuous description of such manifold is not known, and instead it is sampled by a set of scattered points in high dimensions. This poses a serious challenge. In this thesis, we approximate the point-set manifold as an overlapping set of smooth parametric descriptions, whose geometric structure is revealed by statistical learning methods, and then parametrized by meshfree methods. This approach avoids any global parameterization, and hence is applicable to manifolds of any genus and complex geometry. It combines four ingredients: (1) partitioning of the point set into subregions of trivial topology, (2) the automatic detection of the local geometric structure of the manifold by nonlinear dimensionality reduction techniques, (3) the local parameterization of the manifold using smooth meshfree (here local maximum-entropy) approximants, and (4) patching together the local representations by means of a partition of unity. In this thesis we show the generality, flexibility, and accuracy of the method in four different problems. First, we exercise it in the context of Kirchhoff-Love thin shells, (d=2, D=3). We test our methodology against classical linear and non linear benchmarks in thin-shell analysis, and highlight its ability to handle point-set surfaces of complex topology and geometry. We then tackle problems of much higher dimensionality. We perform reduced order modeling in the context of finite deformation elastodynamics, considering a nonlinear reduced configuration space, in contrast with classical linear approaches based on Principal Component Analysis (d=2, D=10000's). We further quantitatively unveil the geometric structure of the motility strategy of a family of micro-organisms called Euglenids from experimental videos (d=1, D~30000's). Finally, in the context of enhanced sampling in molecular dynamics, we automatically construct collective variables for the molecular conformational dynamics (d=1...6, D~30,1000's).

Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques

With respect to the demands of adaptive and 4D-radiotherapy applications, an algorithm is proposed for a fully automatic, multimodality deformable registration that follows the concept of translational relocation of regularly distributed image subvolumes governed by local anatomical features. Thereby, the problem of global deformable registration is broken down to multiple independent local registration steps which allows for straightforward parallelization of the algorithm. In a subsequent step, possible local misregistrations are corrected for by minimization of the elastic energy of the displacement field under consideration of image information. The final displacement field results from interpolation of the subvolume shift vectors. The algorithm can employ as a similarity measure both the correlation coefficient and mutual information. The latter allows the application to intermodality deformable registration problems. The typical calculation time on a modern multiprocessor PC is well below 1 min, which facilitates almost-interactive, "online" usage. CT-to-MRI and CT-to-cone-beam-CT registrations of head-and-neck data sets are presented, as well as inhale-to-exhale registrations of lung CT data sets. For quantitative evaluation of registration accuracy, a virtual thorax phantom was developed; additionally, a landmark-based evaluation on four lung respiratory-correlated CT data sets was performed. This consistently resulted in average registration residuals on the order of the voxel size or less (3D-residuals approximately 1-2 mm). Summarizing, the presented algorithm allows an accurate multimodality deformable registration with calculation times well below 1 min, and thus bears promise as a versatile basic tool in adaptive and 4D-radiotherapy applications.