Chapter 4 - 4D medical image analysis: a systematic study on applications, challenges, and future research directions

The current state of the art in three dimensional oncologic imaging: An overview

Three-dimensional (3D) medical images are prone to overlap, and there are some problems, such as low detection efficiency and inconsistent with the actual situation. Therefore, a 3D medical image surface reconstruction method based on data mining and machine learning is proposed. The 3D medical images were classified according to different ways, the information frame of 3D medical images was established and the surface overlapping information model of 3D images was given. Based on this information framework, the nonlinear function of overlapping area information of 3D medical images was constructed. The weight of the nonlinear function was used to calculate the input and output results of overlapping area information. Combined with the input mode of 3D medical image information, the error between the information output and the expected output was set. The nonlinear function weight of the overlapping area information of 3D medical images was modified by using the learning rate and the use time of the overlapping area information, and the influence factors of the overlapping information detection were obtained by increasing the situation terms, so as to complete the detection of the surface reconstruction information of 3D medical images. The experimental results show that the information detection results of the proposed method fit well with the actual situation, and the information detection efficiency is high.

Three-dimensional (3D) medical images, such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), are essential for clinical applications. However, the need for diverse and comprehensive representations is particularly pronounced when considering the variability across different organs, diagnostic tasks, and imaging modalities. How to effectively interpret the intricate contextual information and extract meaningful insights from these images remains an open challenge to the community. While current self-supervised learning methods have shown potential, they often consider an image as a whole thereby overlooking the extensive, complex relationships among local regions from one or multiple images. In this work, we introduce a pioneering method for learning 3D medical image representations through an autoregressive pre-training framework. Our approach sequences various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence. By employing an autoregressive sequence modeling task, we predict the next visual token in the sequence, which allows our model to deeply understand and integrate the contextual information inherent in 3D medical images. Additionally, we implement a random startup strategy to avoid overestimating token relationships and to enhance the robustness of learning. The effectiveness of our approach is demonstrated by the superior performance over others on nine downstream tasks in public datasets.

Three-dimensional (3D) medical image segmentation is used to segment the target (a lesion or an organ) in 3D medical images. Through this process, 3D target information is obtained; hence, this technology is an important auxiliary tool for medical diagnosis. Although some methods have proved to be successful for two-dimensional (2D) image segmentation, their direct use in the 3D case has been unsatisfactory. To obtain more precise tumor segmentation results from 3D MR images, in this paper, we propose a method known as the 3D shape-weighted level set method (3D-SLSM). The proposed method first converts the LSM, which is superior with respect to 2D image segmentation, into a 3D algorithm that is suitable for overall calculations in 3D image models, and which improves the efficiency and accuracy of calculations. A 3D shape-weighted value is then added for each 3D-SLSM iterative process according to the changes in volume. Besides increasing the convergence rate and eliminating background noise, this shape-weighted value also brings the segmented contour closer to the actual tumor margins. To perform a quantitative analysis of 3D-SLSM and to examine its feasibility in clinical applications, we have divided our experiments into computer-simulated sequence images and actual breast MRI cases. Subsequently, we simultaneously compared various existing 3D segmentation methods. The experimental results demonstrated that 3D-SLSM exhibited precise segmentation results for both types of experimental images. In addition, 3D-SLSM showed better results for quantitative data compared with existing 3D segmentation methods.

Lesion detection or segmentation on 3D medical image (e.g., MRI, CT-PET or hybrid modal) is a challenging task which aims to automatically mark the abnormal region on given input image. The methods to tackle the problem often construct a U-shape autoencoders to map the input to a compressed vector in latent space and predict the final probability map. However, approaches for 3D multimodal image segmentation are still not fully explored and localization accuracy can be improved by designing proper structure based on prior. In this paper, we propose a supervised contrastive approach for 3D multi-modality medical image segmentation. A hybrid 3D-2D multi-modality fusion model is designed based on the input image prior. The 3D segmentation problem is converted to a sequence prediction task by 2D decoder with contrastive modeling. The combined framework (HDC, Hybrid Dimension Contrastive) achieves comparable or even greater performance to 3D counterpart while memory efficient. We conduct extensive experiments on two multimodal datasets of brain image BraTS and our epilepsy MRI-PET. The validation results show the effectiveness of our framework.

An efficient interactive multi-label segmentation tool for 2D and 3D medical images using fully connected conditional random field

The endometrial–myometrial junction: a fresh look at a busy crossing

Introduction: Most medical specialties rely on 3D image data either for diagnosis, surgical planning, surgical navigation, education or patient-clinician communication. Surprisingly, conventional medical image workstations do not promote proper visual insight nor visual collaboration between professionals. Redesigning conventional applications into visually rich and interactive platforms that seamlessly bring together professionals, devices and 3D medical images, may foster better healthcare by promoting visual insights and boost visual memory through collaboration [1–3]. To this end, we envision portable, accessible, and easy to deploy interactive visualization settings to be commonly adopted in clinical environments. This entails adopting novel technologies (e.g., interactive surfaces, augmented reality and virtual reality) into different healthcare workspaces. Consequently, designing novel medical user interfaces and medical user experiences can offer more natural ways to interact with and engage in collaborative tasks around 3D images. Challenges and approaches: Moving from conventional workstations to interactive visualization platforms brings about new and interesting challenges. By interacting with healthcare professionals, visiting their workspaces, analyzing their requirements and suggesting new ways to interact with 3D medical images, we have identified the major hurdles. For one (i) it is difficult to create and nurture a network of healthcare professionals interested on taking time off their office hours or after-hours to engage in research on a regular basis; (ii) a linguistic barrier between healthcare professionals and computer scientists, as clinicians and health technicians usually provide “big picture scenarios” lacking details crucial to interface design, requiring extra effort to develop a working set of specifications; (iii) many clinicians overlook the technical complexity hidden from the view of users, believing in easy-to-deploy “automagical” solutions; and (iv) identifying current healthcare practices that could be improved by adopting interactive technologies. In order for healthcare professionals to realize that interactive visualization systems may bring added value over traditional practices, it is necessary to develop both well-designed and easy-to-learn medical user interfaces that promote more natural user experiences. In addition, these interactive visualization systems must also be cost-effective, accessible, portable, novel, yet familiar to enable professionals to collaboratively work either in loco or remotely. While developing such approaches is never an easy task [4], the potential rewards make it a very worthwhile endeavor.

The ever-increasing importance of accelerated information processing, communica-tion, and storing are major requirements within the big-data era revolution. With the extensive rise in data availability, handy information acquisition, and growing data rate, a critical challenge emerges in efficient handling. Even with advanced technical hardware developments and multiple Graphics Processing Units (GPUs) availability, this demand is still highly promoted to utilise these technologies effectively. Health-care systems are one of the domains yielding explosive data growth. Especially when considering their modern scanners abilities, which annually produce higher-resolution and more densely sampled medical images, with increasing requirements for massive storage capacity. The bottleneck in data transmission and storage would essentially be handled with an effective compression method. Since medical information is critical and imposes an influential role in diagnosis accuracy, it is strongly encouraged to guarantee exact reconstruction with no loss in quality, which is the main objective of any lossless compression algorithm. Given the revolutionary impact of Deep Learning (DL) methods in solving many tasks while achieving the state of the art results, includ-ing data compression, this opens tremendous opportunities for contributions. While considerable efforts have been made to address lossy performance using learning-based approaches, less attention was paid to address lossless compression. This PhD thesis investigates and proposes novel learning-based approaches for compressing 3D medical images losslessly.Firstly, we formulate the lossless compression task as a supervised sequential prediction problem, whereby a model learns a projection function to predict a target voxel given sequence of samples from its spatially surrounding voxels. Using such 3D local sampling information efficiently exploits spatial similarities and redundancies in a volumetric medical context by utilising such a prediction paradigm. The proposed NN-based data predictor is trained to minimise the differences with the original data values while the residual errors are encoded using arithmetic coding to allow lossless reconstruction.Following this, we explore the effectiveness of Recurrent Neural Networks (RNNs) as a 3D predictor for learning the mapping function from the spatial medical domain (16 bit-depths). We analyse Long Short-Term Memory (LSTM) models’ generalisabil-ity and robustness in capturing the 3D spatial dependencies of a voxel’s neighbourhood while utilising samples taken from various scanning settings. We evaluate our proposed MedZip models in compressing unseen Computerized Tomography (CT) and Magnetic Resonance Imaging (MRI) modalities losslessly, compared to other state-of-the-art lossless compression standards.This work investigates input configurations and sampling schemes for a many-to-one sequence prediction model, specifically for compressing 3D medical images (16 bit-depths) losslessly. The main objective is to determine the optimal practice for enabling the proposed LSTM model to achieve a high compression ratio and fast encoding-decoding performance. A solution for a non-deterministic environments problem was also proposed, allowing models to run in parallel form without much compression performance drop. Compared to well-known lossless codecs, experimental evaluations were carried out on datasets acquired by different hospitals, representing different body segments, and have distinct scanning modalities (i.e. CT and MRI).To conclude, we present a novel data-driven sampling scheme utilising weighted gradient scores for training LSTM prediction-based models. The objective is to determine whether some training samples are significantly more informative than others, specifically in medical domains where samples are available on a scale of billions. The effectiveness of models trained on the presented importance sampling scheme was evaluated compared to alternative strategies such as uniform, Gaussian, and sliced-based sampling.

Precise localization of anatomical structures in 3D medical images can support several tasks such as image registration, organ segmentation, lesion quantification and abnormality detection. This work proposes a novel method, based on deep reinforcement learning, to actively learn to localize an object in the volumetric scene. Given the parameterization of the sought object, an intelligent agent learns to optimize the parameters by performing a sequence of simple control actions. We show the applicability of our method by localizing boxes (9 degrees of freedom) on a set of acquired MRI scans of the brain region. We achieve high speed and high accuracy detection results, with robustness to challenging cases. This method can be applied to a broad range of problems and easily generalized to other type of imaging modalities.

In three-dimensional (3D) medical ultrasound imaging with two-dimensional (2D) arrays, sparse 2D arrays have been studied to reduce the number of active channels. Among them, sparse 2D arrays with regular or uniform arrangements of elements have advantages of low side lobe energy and uniform field responses over the entire field of view. This paper presents two uniform sparse array models: sparse rectangular arrays (SRAs) on a rectangular grid and sparse spiral arrays (SSAs) on a sunflower grid. Both arrays can be easily implemented on the commercially available or the custom-made arrays. To suppress the overall grating lobe levels, the transmit (Tx) and receive (Rx) array pairs of both the array models are designed not to have grating lobes at the same locations in the Tx/Rx beam patterns, for which the theoretical design rules are also proposed. Computer simulation results indicate that the proposed array pairs for both the SRAs and the SSAs achieve peak grating lobe levels below –40 dB using about a quarter of the number of elements in the dense rectangular array while maintaining similar beam widths to that of the dense array pair.

Three-dimensional (3D) medical image segmentation plays a crucial role in medical care applications. Although various two-dimensional (2D) and 3D neural network models have been applied to 3D medical image segmentation and achieved impressive results, a trade-off remains between efficiency and accuracy. To address this issue, a novel mixture convolutional network (MixConvNet) is proposed, in which traditional 2D/3D convolutional blocks are replaced with novel MixConv blocks. In the MixConv block, 3D convolution is decomposed into a mixture of 2D convolutions from different views. Therefore, the MixConv block fully utilizes the advantages of 2D convolution and maintains the learning ability of 3D convolution. It acts as 3D convolutions and thus can process volumetric input directly and learn intra-slice features, which are absent in the traditional 2D convolutional block. By contrast, the proposed MixConv block only contains 2D convolutions; hence, it has significantly fewer trainable parameters and less computation budget than a block containing 3D convolutions. Furthermore, the proposed MixConvNet is pre-trained with small input patches and fine-tuned with large input patches to improve segmentation performance further. In experiments on the Decathlon Heart dataset and Sliver07 dataset, the proposed MixConvNet outperformed the state-of-the-art methods such as UNet3D, VNet, and nnUnet.

The problem of the quantization noise effect in coefficients of discrete wavelet transform (DWT) filters for 3D medical imaging is analyzed in this paper. We proposed the coefficients quantizing method which allows reducing hardware implementation costs. We derived formulas for determining the minimum wavelet coefficients bit-width that provide high quality in terms of the peak signal-to-noise ratio of 3D medical image processing using DWT. The experiments with 3D tomographic images confirmed the accuracy of theoretical analysis results. All data are presented in the fixed-point format which allows the possibility for the efficient implementation of image denoising and compression on modern devices such as field-programmable gate arrays and application-specific integrated circuits.

Medical imaging using different modalities has many problems. The main ones are low informativeness, various distortion noises, and a large amount of information. Fusion, denoising, and visual data compression are used to solve them in practice. Discrete wavelet transform is one way to implement various fusion, denoising, and compression methods for 2D and 3D medical image processing. Medical imaging systems produce increasingly accurate images with scanning technology and digital devices development. These images have improved quality using both higher spatial resolutions and color bit-depth. Processing a large volume of medical imaging data requires considerable resources and processing time. Modern wavelet-based devices for medical image processing do not meet the current performance demand. Hardware accelerators are being designed to solve this problem. This paper proposes new (field-programmable gate array) FPGA accelerators using wavelet processing (WP) with scaled filter coefficients (SFC) and parallel computing in residue number system (RNS) to improve the performance of high-quality 3D medical image WP systems. The computational complexity is reduced using the developed WP method with SFC and the proposed wavelet filter coefficients scaling algorithm. Parallel computing is organized in RNS using moduli sets of a particular type. Hardware implementation of 3D medical image WP using the proposed FPGA accelerators increases device performance by 2.89-3.59 times, increasing the hardware resources by 1.18-3.29 times compared to state-of-the-art solutions. The device performance improvement is achieved while maintaining high-quality 3D medical image processing in peak signal-to-noise ratio terms.

In recent years, 4D medical image involving structural and motion information of tissue has attracted increasing attention. The key to the 4D image reconstruction is to stack the 2D slices based on matching the aligned motion states. In this study, the distribution of the 2D slices with the different motion states is modeled as a manifold graph, and the reconstruction is turned to be the graph alignment. An embedding-alignment fusion-based graph convolution network (GCN) with a mixed-learning strategy is proposed to align the graphs. Herein, the embedding and alignment processes of graphs interact with each other to realize a precise alignment with retaining the manifold distribution. The mixed strategy of self- and semi-supervised learning makes the alignment sparse to avoid the mismatching caused by outliers in the graph. In the experiment, the proposed 4D reconstruction approach is validated on the different modalities including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Ultrasound (US). We evaluate the reconstruction accuracy and compare it with those of state-of-the-art methods. The experiment results demonstrate that our approach can reconstruct a more accurate 4D image.