Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis

In this study, we propose a novel point cloud based 3D registration and segmentation framework using reinforcement learning. An artificial agent, implemented as a distinct actor based on value networks, is trained to predict the optimal piece-wise linear transformation of a point cloud for the joint tasks of registration and segmentation. The actor network estimates a set of plausible actions and the value network aims to select the optimal action for the current observation. Point-wise features that comprise spatial positions (and surface normal vectors in the case of structured meshes), and their corresponding image features, are used to encode the observation and represent the underlying 3D volume. The actor and value networks are applied iteratively to estimate a sequence of transformations that enable accurate delineation of object boundaries. The proposed approach was extensively evaluated in both segmentation and registration tasks using a variety of challenging clinical datasets. Our method has fewer trainable parameters and lower computational complexity compared to the 3D U-Net, and it is independent of the volume resolution. We show that the proposed method is applicable to mono- and multi-modal segmentation tasks, achieving significant improvements over the state-of-the-art for the latter. The flexibility of the proposed framework is further demonstrated for a multi-modal registration application. As we learn to predict actions rather than a target, the proposed method is more robust compared to the 3D U-Net when dealing with previously unseen datasets, acquired using different protocols or modalities. As a result, the proposed method provides a promising multi-purpose segmentation and registration framework, particular in the context of image-guided interventions.

Accurate image registration is a necessary prerequisite for many diagnostic and therapy planning procedures where complementary information from different images has to be combined. The design of robust and reliable non-parametric registration schemes is currently a very active research area. Modern approaches combine the pure registration scheme with other image processing routines such that both ingredients may benefit from each other. One of the new approaches is the combination of segmentation and registration (segistration). Here, the segmentation part guides the registration to its desired configuration, whereas on the other hand the registration leads to an automatic segmentation. By joining these image processing methods it is possible to overcome some of the pitfalls of the individual methods. Here, we focus on the benefits for the registration task. In the current work, we present a novel unified framework for non-parametric registration combined with energy-based segmentation through active contours. In the literature, one may find various ways to combine these image processing routines. Here, we present the most promising approaches within the general framework. It is based on a single variational formulation of both the registration and the segmentation part. The performance tests are carried out for magnetic resonance (MR) images of the brain, and they demonstrate the potential of the proposed methods.

This paper presents a 3D non-rigid registration algorithm between histological and MR images of the prostate with cancer. To compensate for the loss of 3D integrity in the histology sectioning process, series of 2D histological slices are first reconstructed into a 3D histological volume. After that, the 3D histology-MRI registration is obtained by maximizing a) landmark similarity and b) cancer region overlap between the two images. The former aims to capture distortions at prostate boundary and internal blob-like structures; and the latter aims to capture distortions specifically at cancer regions. In particular, landmark similarities, the former, is maximized by an annealing process, where correspondences between the automatically-detected boundary and internal landmarks are iteratively established in a fuzzy-to-deterministic fashion. Cancer region overlap, the latter, is maximized in a joint cancer segmentation and registration framework, where the two interleaved problems - segmentation and registration - inform each other in an iterative fashion. Registration accuracy is established by comparing against human-rater-defined landmarks and by comparing with other methods. The ultimate goal of this registration is to warp the histologically-defined cancer ground truth into MRI, for more thoroughly understanding MRI signal characteristics of the prostate cancerous tissue, which will promote the MRI-based prostate cancer diagnosis in the future studies.

This paper introduces a fuzzy framework for the simultaneous segmentation and registration of MRI datasets. The framework utilizes prior information which may be available about the class center and class's pixels distribution through the datasets. The algorithm is evaluated using phantom and real medical MRI brain volume. The results show that the algorithm has considerable accuracy for segmentation and affine registration. The algorithm needs a small number of iterations to reach convergence compared with other similar algorithms.

Medical image segmentation and registration are two tasks to analyze the anatomical structures in clinical research. Still, deep-learning solutions utilizing the connections between segmentation and registration remain underdiscovered. This article designs a joint learning framework named RSegNet that can realize concurrent deformable registration and segmentation by minimizing an integrated loss function, including three parts: diffeomorphic registration loss, segmentation similarity loss, and dual-consistency supervision loss. The probabilistic diffeomorphic registration branch could benefit from the auxiliary segmentations available from the segmentation branch to achieve anatomical consistency and better deformation regularity by dual-consistency supervision. Simultaneously, the segmentation performance could also be improved by data augmentation based on the registration with well-behaved diffeomorphic guarantees. Experiments on the human brain 3-D magnetic resonance images have been implemented to demonstrate the effectiveness of our approach. We trained and validated RSegNet with 1000 images and tested its performances on four public datasets, which shows that our method successfully yields concurrent improvements of both segmentation and registration compared with separately trained networks. Specifically, our method can increase the accuracy of segmentation and registration by 7.0% and 1.4%, respectively, in terms of Dice scores. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Registration and segmentation of medical images are two significant tasks in medical research and clinical application. However, most existing approaches consider these two tasks independently while neglecting the potential association between them. Therefore, we suggest a new approach that combines these two tasks into one joint deep learning framework, boosting registration, and segmentation performance by introducing dual-consistency supervision. Besides, our framework could generate outputs within 1 s by taking an affinely aligned medical image pair as input, which is suitable for time-critical requirements in a clinic. We tested it on four public datasets and achieved state-of-the-art performance to demonstrate the proposed method’s feasibility and robustness. Furthermore, our proposed RSegNet is a general learning framework suitable for various image modalities and anatomical structures. Hence, we expect our framework to serve as a practical clinical tool to speed up medical image analysis procedures and improve diagnostic accuracy.