Brain Tumor Segmentation and Survival Prediction Using Automatic Hard Mining in 3D CNN Architecture

Convolutional neural networks have achieved excellent results in automatic medical image segmentation. In this study, we proposed a novel three-dimensional (3D) multipath DenseNet for generating the accurate glioblastoma (GBM) tumor contour from four multimodal pre-operative MR images. We hypothesized that the multipath architecture could achieve more accurate segmentation than a singlepath architecture. Two hundred and fifty-eight GBM patients were included in this study. Each patient had four MR images (T1-weighted, contrast-enhanced T1-weighted, T2-weighted, and FLAIR) and the manually segmented tumor contour. We built a 3D multipath DenseNet that could be trained to achieve an end-to-end mapping from four MR images to the corresponding GBM tumor contour. A 3D singlepath DenseNet was also built for comparison. Both DenseNets were based on the encoder-decoder architecture. All four images were concatenated and fed into a single encoder path in the singlepath DenseNet, while each input image had its own encoder path in the multipath DenseNet. The patient cohort was randomly split into a training set of 180 patients, a validation set of 39 patients, and a testing set of 39 patients. Model performance was evaluated using the Dice similarity coefficient (DSC), average surface distance (ASD), and 95% Hausdorff distance (HD95% ). Wilcoxon signed-rank tests were conducted to assess statistical significances. The singlepath DenseNet achieved the DSC of 0.911±0.060, ASD of 1.3±0.7mm, and HD95% of 5.2±7.1mm, while the multipath DenseNet achieved the DSC of 0.922±0.041, ASD of 1.1±0.5mm, and HD95% of 3.9±3.3mm. The P-values of all Wilcoxon signed-rank tests were less than 0.05. Both DenseNets generated GBM tumor contours in good agreement with the manually segmented contours from multimodal MR images. The multipath DenseNet achieved more accurate tumor segmentation than the singlepath DenseNet. Here presented the 3D multipath DenseNet that demonstrated an improved accuracy over comparable algorithms in the clinical task of GBM tumor segmentation.

Organ delineation is crucial to diagnosis and therapy, while it is also labor-intensive and observer-dependent. Dual energy CT (DECT) provides additional image contrast than conventional single energy CT (SECT), which may facilitate automatic organ segmentation. This work aims to develop an automatic multi-organ segmentation approach using deep learning for head-and-neck region on DECT. We proposed a mask scoring regional convolutional neural network (R-CNN) where comprehensive features are firstly learnt from two independent pyramid networks and are then combined via deep attention strategy to highlight the informative ones extracted from both two channels of low and high energy CT. To perform multi-organ segmentation and avoid misclassification, a mask scoring subnetwork was integrated into the Mask R-CNN framework to build the correlation between the class of potential detected organ’s region-of-interest (ROI) and the shape of that organ’s segmentation within that ROI. We evaluated our model on DECT images from 127 head-and-neck cancer patients (66 training, 61 testing) with manual contours of 19 organs as training target and ground truth. For large- and mid-sized organs such as brain and parotid, the proposed method successfully achieved average Dice similarity coefficient (DSC) larger than 0.8. For small-sized organs with very low contrast such as chiasm, cochlea, lens and optic nerves, the DSCs ranged between around 0.5 and 0.8. With the proposed method, using DECT images outperforms using SECT in almost all 19 organs with statistical significance in DSC (p<0.05). Meanwhile, by using the DECT, the proposed method is also significantly superior to a recently developed FCN-based method in most of organs in terms of DSC and the 95th percentile Hausdorff distance. Quantitative results demonstrated the feasibility of the proposed method, the superiority of using DECT to SECT, and the advantage of the proposed R-CNN over FCN on the head-and-neck patient study. The proposed method has the potential to facilitate the current head-and-neck cancer radiation therapy workflow in treatment planning.

Accurate brain tumor segmentation is crucial for clinical assessment, follow-up, and subsequent treatment of gliomas. While convolutional neural networks (CNN) have become state of the art in this task, most proposed models either use 2D architectures ignoring 3D contextual information or 3D models requiring large memory capacity and extensive learning databases. In this study, an ensemble of two kinds of U-Net-like models based on both 3D and 2.5D convolutions is proposed to segment multimodal magnetic resonance images (MRI). The 3D model uses concatenated data in a modified U-Net architecture. In contrast, the 2.5D model is based on a multi-input strategy to extract low-level features from each modality independently and on a new 2.5D Multi-View Inception block that aims to merge features from different views of a 3D image aggregating multi-scale features. The Asymmetric Ensemble of Asymmetric U-Net (AE AU-Net) based on both is designed to find a balance between increasing multi-scale and 3D contextual information extraction and keeping memory consumption low. Experiments on 2019 dataset show that our model improves enhancing tumor sub-region segmentation. Overall, performance is comparable with state-of-the-art results, although with less learning data or memory requirements. In addition, we provide voxel-wise and structure-wise uncertainties of the segmentation results, and we have established qualitative and quantitative relationships between uncertainty and prediction errors. Dice similarity coefficient for the whole tumor, tumor core, and tumor enhancing regions on BraTS 2019 validation dataset were 0.902, 0.815, and 0.773. We also applied our method in BraTS 2018 with corresponding Dice score values of 0.908, 0.838, and 0.800.

High Resolution peripheral Quantitative Computed Tomography (HR-pQCT) imaging for studying bone disease has become increasingly common. However, due to the bone inhomogeneity and the noise characteristics, segmentation of HR-pQCT data remains a challenging task. In this work we propose a novel segmentation technique of the cortical bone based on fuzzy energy active contours model. A novel approach as well as a new formulation of the fuzzy membership function are proposed to deal with the HR-pQCT inhomogeneity and separate the cortical bone from the trabecular one. Results show the efficiency and the high accuracy of the proposed approach compared to different existing techniques in terms of Dice similarity coefficient (DSC). Proposed method provides high result (DSC: 90.14±1.64%) compared to Burghardt (DSC: 85.86±3,16%) and FEBAC (DSC: 85.5±4.52%).

A cascade of fully convolutional neural networks is proposed to segment multi-modal Magnetic Resonance (MR) images with brain tumor into background and three hierarchical regions: whole tumor, tumor core and enhancing tumor core. The cascade is designed to decompose the multi-class segmentation problem into a sequence of three binary segmentation problems according to the subregion hierarchy. The whole tumor is segmented in the first step and the bounding box of the result is used for the tumor core segmentation in the second step. The enhancing tumor core is then segmented based on the bounding box of the tumor core segmentation result. Our networks consist of multiple layers of anisotropic and dilated convolution filters, and they are combined with multi-view fusion to reduce false positives. Residual connections and multi-scale predictions are employed in these networks to boost the segmentation performance. Experiments with BraTS 2017 validation set show that the proposed method achieved average Dice scores of 0.7859, 0.9050, 0.8378 for enhancing tumor core, whole tumor and tumor core, respectively. The corresponding values for BraTS 2017 testing set were 0.7831, 0.8739, and 0.7748, respectively.

BackgroundThe evaluation of automatic segmentation algorithms is commonly performed using geometric metrics. An analysis based on dosimetric parameters might be more relevant in clinical practice but is often lacking in the literature. The aim of this study was to investigate the impact of state-of-the-art 3D U-Net-generated organ delineations on dose optimization in radiation therapy (RT) for prostate cancer patients.MethodsA database of 69 computed tomography images with prostate, bladder, and rectum delineations was used for single-label 3D U-Net training with dice similarity coefficient (DSC)-based loss. Volumetric modulated arc therapy (VMAT) plans have been generated for both manual and automatic segmentations with the same optimization settings. These were chosen to give consistent plans when applying perturbations to the manual segmentations. Contours were evaluated in terms of DSC, average and 95% Hausdorff distance (HD). Dose distributions were evaluated with the manual segmentation as reference using dose volume histogram (DVH) parameters and a 3%/3 mm gamma-criterion with 10% dose cut-off. A Pearson correlation coefficient between DSC and dosimetric metrics, i.e. gamma index and DVH parameters, has been calculated.Results3D U-Net-based segmentation achieved a DSC of 0.87 (0.03) for prostate, 0.97 (0.01) for bladder and 0.89 (0.04) for rectum. The mean and 95% HD were below 1.6 (0.4) and below 5 (4) mm, respectively. The DVH parameters, V_{60/65/70,{mathrm{Gy}}} for the bladder and V_{50/65/70,{mathrm{Gy}}} for the rectum, showed agreement between dose distributions within pm , 5% and pm ,2%, respectively. The D_{98/2%} and V_{95%}, for prostate and its 3 mm expansion (surrogate clinical target volume) showed agreement with the reference dose distribution within 2% and 3 Gy with the exception of one case. The average gamma pass-rate was 85%. The comparison between geometric and dosimetric metrics showed no strong statistically significant correlation.ConclusionsThe 3D U-Net developed for this work achieved state-of-the-art geometrical performance. Analysis based on clinically relevant DVH parameters of VMAT plans demonstrated neither excessive dose increase to OARs nor substantial under/over-dosage of the target in all but one case. Yet the gamma analysis indicated several cases with low pass rates. The study highlighted the importance of adding dosimetric analysis to the standard geometric evaluation.

In this paper, a novel method to automatically segment colorectal cancer from 3D MR images based on combination of 3D fully convolutional neural networks (3D-FCNNs) and 3D level-set is proposed. The 3D-level set is incorporated in the 3D-FCNNs aiming at: i) a fine-tuning of the training phase; ii) a refinement of the outputs during the testing phase by integrating smoothing function and prior information in a post-processing step. The proposed method is assessed and compared with 3D-FCNNs without 3D-level set (3D-FCNNs alone) in terms of Dice Similarity Coefficient (DSC) as a performance metric. The proposed method showed higher DSC than 3D-FCNNs alone on both training and testing data set as, (0.91813 vs 0.8568) and (0.9378 vs 0.86238), respectively. Our results on 3D colorectal MRI data demonstrated that the proposed method gives better and accurate segmentation results than 3D-FCNNs alone.

Accurate and automated brain tumor segmentation using multi modal MR images is essential for the evaluation of the disease progression in order to improve disease diagnosis and treatment planning. We present a new fully automated method for high graded brain tumor segmentation combining sparse autoencoder and multimodal Fuzzy C-means clustering. The approach utilizes multimodal MRI contrast: T1, T2, FLAIR and T1c (contrast-enhanced) for 15 high graded glioma (HGG) subjects. The objective of the proposed study is to segment tumor tissues from HGG including edema and tumor core within edema. The segmentation was performed on the training data of the multimodal brain tumor image segmentation benchmark 2015. Sparse autoencoder, which is an unsupervised learning algorithm, was used to automatically learn features from unlabeled dataset of tumor in order to segment edema. Followed by edema segmentation, tumor core was segmented from edema using multimodal FCM clustering. Evaluating the performance of the segmentation results with the ground truth yields high dice score (DS) of 0.9866±0.01 and 0.9843±0.01 for edema and tumor core respectively and high Jaccard similarity (JS) of 0.9738±0.02 and 0.9692±0.02 for edema and tumor core respectively; showed high accuracy in segmenting the complex tumor structures from multi-contrast MR scans of HGG patients. We also compared our methodology in terms of segmentation efficiency with some recent techniques reported in proceedings of MICCAI-BRATS challenge 2015.

Automated segmentation of brain tumour from multimodal MR images is pivotal for the analysis and monitoring of disease progression. As gliomas are malignant and heterogeneous, efficient and accurate segmentation techniques are used for the successful delineation of tumours into intra-tumoural classes. Deep learning algorithms outperform on tasks of semantic segmentation as opposed to the more conventional, context-based computer vision approaches. Extensively used for biomedical image segmentation, Convolutional Neural Networks have significantly improved the state-of-the-art accuracy on the task of brain tumour segmentation. In this paper, we propose an ensemble of two segmentation networks: a 3D CNN and a U-Net, in a significant yet straightforward combinative technique that results in better and accurate predictions. Both models were trained separately on the BraTS-19 challenge dataset and evaluated to yield segmentation maps which considerably differed from each other in terms of segmented tumour sub-regions and were ensembled variably to achieve the final prediction. The suggested ensemble achieved dice scores of 0.750, 0.906 and 0.846 for enhancing tumour, whole tumour, and tumour core, respectively, on the validation set, performing favourably in comparison to the state-of-the-art architectures currently available.

Accelerated multi-modal magnetic resonance imaging (MRI) offers an effective solution for fast MRI, excelling in reconstructing the target modality from its under-sampled counterpart with the assistance of a reference modality. Current deep learning methods for accelerated MRI primarily utilize convolutional neural networks (CNNs) or Transformers to extract features and integrate various types of data. While CNNs are efficient, their limited receptive fields constrain their ability to capture global context. Meanwhile, Transformers excel at modeling global information but are challenged by their quadratic computational complexity. Recent advancements in the State Space Model (SSM), particularly Mamba, offer a promising solution to this issue by enabling global awareness with linear complexity. Thus, we propose MR-FusionMamba to perform efficient multi-modal MRI reconstruction, which integrates Mamba blocks into dual U-shaped networks and extends their functionality to support dual inputs. To improve the data-consistency, we apply the Range-Null Decomposition theorem to our network. We evaluate our method with extensive experiments on the BraTS dataset, demonstrating that our method achieves state-of-the-art (SOTA) performance.

MSFCN-multiple supervised fully convolutional networks for the osteosarcoma segmentation of CT images

Purpose: To investigate the feasibility of using DIR to propagate the manually contoured rectum and bladder from the 1st insertion to the new CT images on subsequent insertions and evaluate the segmentation performance. Methods: Ten cervical cancer patients, who were treated by T&R brachytherapy in 3–4 insertions, were retrospectively collected. In each insertion, rectum and bladder were manually delineated on the planning CT by a physicist and verified by a radiation oncologist. Using VelocityAI (Velocity Medical Solutions, Atlanta, GA), a rigid registration was firstly employed to match the bony structures between the first insertion and each of the following insertions, then a multi-pass B-spine DIR was carried out to further map the sub volume that encompasses rectum and bladder. The resultant deformation fields propagated contours, and dice similarity coefficient (DSC) was used to quantitatively evaluate the agreement between the propagated contours and the manually-delineated organs. For the 3rd insertion, we also evaluated if the segmentation performance could be improved by propagating the contours from the most recent insertion, i.e., the 2nd insertion. Results: On average, the contour propagation took about 1 minute. The average and standard deviation of DSC over all insertions and patients was 0.67±0.10 (range: 0.44–0.81) for rectum, and 0.78±0.07 (range: 0.63–0.87) for bladder. For the 3rd insertion, propagating contours from the 2nd insertion could improve the segmentation performance in terms of DSC from 0.63±0.10 to 0.72±0.08 for rectum, and from 0.77±0.07 to 0.79±0.06 for bladder. A Wilcoxon signed rank test indicated that the improvement was statistically significant for rectum (p = 0.004). Conclusion: The preliminary results demonstrate that deformable image registration could efficiently and accurately propagate rectum and bladder contours between CT images in different T&R brachytherapy fractions. We are incorporating the propagated contours into our learning-based method to further segment these organs.

Convolutional neural networks (CNNs) show potential for delineating cancers on contrast-enhanced MRI (ce-MRI) but there are clinical scenarios in which administration of contrast is not desirable. We investigated performance of the CNN for delineating primary nasopharyngeal carcinoma (NPC) on non-contrast-enhanced images and compared the performance to that on ce-MRI. We retrospectively analyzed primary NPC in 195 patients using a well-established CNN, U-Net, for tumor delineation on the non-contrast-enhanced fat-suppressed (fs)-T2W, ce-T1W and ce-fs-T1W images. The CNN-derived delineations were compared to manual delineations to obtain Dice similarity coefficient (DSC) and average surface distance (ASD). The DSC and ASD on fs-T2W were compared to those on ce-MRI. Primary tumor volumes (PTVs) of CNN-derived delineations were compared to that of manual delineations. The CNN for NPC delineation on fs-T2W images showed similar DSC (0.71 ± 0.09) and ASD (0.21 ± 0.48cm) to those on ce-T1W images (0.71 ± 0.09 and 0.17 ± 0.19cm, respectively) (p > 0.05), and lower DSC but similar ASD to ce-fs-T1W images (0.73 ± 0.09, p < 0.001; and 0.17 ± 0.20cm, p > 0.05). The CNN overestimated PTVs on all sequences (p < 0.001). The CNN showed promise for NPC delineation on fs-T2W images in cases where it is desirable to avoid contrast agent injection. The CNN overestimated PTVs on all sequences.

Cerebral infarct volume observed from follow-up non contrast CT (NCCT) scans is an important radiologic outcome measurement of the effectiveness of acute ischemic patient treatment. Post-treatment infarct typically includes ischemic infarct only. But in around 10% of ischemic stroke patients, intracerebral hemorrhage is present along with ischemic infarction. Post-treatment infarct is currently segmented manually, making it tedious and error prone. In order to measure post-treatment infarct volume more efficiently from follow-up NCCT images, a novel joint segmentation approach is proposed for segmenting ischemic infarct and hemorrhage simultaneously, which makes use of multi-region time-implicit contour evolution combined with random forest (RF) learned semantic information. The quantitative evaluation using 16 patient images shows that the proposed segmentation approach is delivering favorable results compared to the gold standard of manual segmentation in terms of dice similarity coefficient (DSC) and the mean (MAD) and maximum (MAXD) absolute surface distance. To the best of our knowledge, this paper reports the first attempt of simultaneously segmenting ischemic infarct and hemorrhage from follow-up NCCT images of post-treatment acute stroke patients.

BackgroundThis study aimed to (1) develop a fully residual deep convolutional neural network (CNN)-based segmentation software for computed tomography image segmentation of the male pelvic region and (2) demonstrate its efficiency in the male pelvic region.MethodsA total of 470 prostate cancer patients who had undergone intensity-modulated radiotherapy or volumetric-modulated arc therapy were enrolled. Our model was based on FusionNet, a fully residual deep CNN developed to semantically segment biological images. To develop the CNN-based segmentation software, 450 patients were randomly selected and separated into the training, validation and testing groups (270, 90, and 90 patients, respectively). In Experiment 1, to determine the optimal model, we first assessed the segmentation accuracy according to the size of the training dataset (90, 180, and 270 patients). In Experiment 2, the effect of varying the number of training labels on segmentation accuracy was evaluated. After determining the optimal model, in Experiment 3, the developed software was used on the remaining 20 datasets to assess the segmentation accuracy. The volumetric dice similarity coefficient (DSC) and the 95th-percentile Hausdorff distance (95%HD) were calculated to evaluate the segmentation accuracy for each organ in Experiment 3.ResultsIn Experiment 1, the median DSC for the prostate were 0.61 for dataset 1 (90 patients), 0.86 for dataset 2 (180 patients), and 0.86 for dataset 3 (270 patients), respectively. The median DSCs for all the organs increased significantly when the number of training cases increased from 90 to 180 but did not improve upon further increase from 180 to 270. The number of labels applied during training had a little effect on the DSCs in Experiment 2. The optimal model was built by 270 patients and four organs. In Experiment 3, the median of the DSC and the 95%HD values were 0.82 and 3.23 mm for prostate; 0.71 and 3.82 mm for seminal vesicles; 0.89 and 2.65 mm for the rectum; 0.95 and 4.18 mm for the bladder, respectively.ConclusionsWe have developed a CNN-based segmentation software for the male pelvic region and demonstrated that the CNN-based segmentation software is efficient for the male pelvic region.