A Semi-supervised Learning Method for Automatic Nuclei Segmentation Using Generative Adversarial Networks

Automated nuclei segmentation and classification are the keys to analyze and understand the cellular characteristics and functionality, supporting computer-aided digital pathology in disease diagnosis. However, the task still remains challenging due to the intrinsic variations in size, intensity, and morphology of different types of nuclei. Herein, we propose a self-guided ordinal regression neural network for simultaneous nuclear segmentation and classification that can exploit the intrinsic characteristics of nuclei and focus on highly uncertain areas during training. The proposed network formulates nuclei segmentation as an ordinal regression learning by introducing a distance decreasing discretization strategy, which stratifies nuclei in a way that inner regions forming a regular shape of nuclei are separated from outer regions forming an irregular shape. It also adopts a self-guided training strategy to adaptively adjust the weights associated with nuclear pixels, depending on the difficulty of the pixels that is assessed by the network itself. To evaluate the performance of the proposed network, we employ large-scale multi-tissue datasets with 276349 exhaustively annotated nuclei. We show that the proposed network achieves the state-of-the-art performance in both nuclei segmentation and classification in comparison to several methods that are recently developed for segmentation and/or classification.

The number of nuclei on a pathology image assists pathologists in consistent diagnosis of breast cancer. Currently, most pathologists make a diagnosis based on a rough estimation of the number of nuclei on pathology images. Because of the rough estimation, the diagnosis is not objective. To assist pathologists to make a consistent, objective and fast diagnosis, it is necessary to develop a computer system to automatically recognize and count several kinds of nuclei. We have developed an algorithm for the automatic segmentation and counting of nuclei in breast cancer pathology images. In the development of the algorithm, we proposed two novel methods: an adaptive-sized hybrid neural network for the automatic segmentation of nuclei, insulin-like growth factor-II messenger RNAs and other structures, and the combined use of both the focused gradient filter and the watersheds algorithm for segmentation of overlapped nuclei.

Generative Adversarial Network is the topic of interest in today’s research in the field of image processing and computer vision. A basic GAN model was introduced by Ian Goodfellow et al. in 2014. After that advancement in the field of research in GAN models has been application specific. In computer vision and image to image translation GANs are playing very effective role either in the case of face detection and recognition or in image resolution enhancement and image augmentation. This paper represents a concise overview of various GAN models along with their features and applications. Pix2Pix and conditional GAN models work upon paired datasets while other models like cycle GAN, discover GAN, dual GAN, info GAN, deep convolutional GAN etc. work upon unpaired datasets. Various image datasets which are commonly used for training of generator and discriminator networks are also discussed in this paper. Since partial mode collapse is a common problem to occur during training process for all models, therefore various normalization techniques are also preferred during the training of generator and discriminator networks. As the advancements in GAN models are increasing at a very fast rate, soon these models will also be preferred in commercial applications.

With the use of computer-aided diagnostic systems, the automatic detection and segmentation of the cell nuclei have become essential in pathology due to cellular nuclei counting and nuclear pleomorphism analysis are critical for the classification and grading of breast cancer histopathology. This work describes a methodology for automatic detection and segmentation of cellular nuclei in breast cancer histopathology images obtained from the BreakHis database, the Standford tissue microarray database, and the Breast Cancer Cell Segmentation database. The proposed scheme is based on the characterization of Hematoxylin and Eosin (H&E) staining, size, and shape features. In addition, we use the information obtained from morphological transformations and adaptive intensity adjustments to detect and separate each cell nucleus detected in the image. The segmentation was carried out by testing the proposed methodology in a histological breast cancer database that provides the associated groundtruth segmentation. Subsequently, the Sørensen-Dice similarity coefficient was calculated to analyze the suitability of the results.Clinical relevance- In this work, the detection and segmentation of cell nuclei in breast cancer histological images are carried out automatically. The method can identify cell nuclei regardless of variations in the level of staining and image magnification. Moreover, a granulometric analysis of the components allows identifying cell clumps and segment them into individual cell nuclei. Improved identification of cell nuclei under different image conditions was demonstrated to reach a sensitivity average of 0.76 ± 0.12. The results provide a base for further and complex processes such as cell counting, feature analysis, and nuclear pleomorphism, which are relevant tasks in the evaluation and diagnostic performed by the expert pathologist.

This paper presents a method for automatic segmentation of nuclei in phase-contrast images using the intensity, convexity and texture of the nuclei. The proposed method consists of three main stages: preprocessing, h-maxima transformation-based marker controlled watershed segmentation ( h-TMC), and texture analysis. In the preprocessing stage, a top-hat filter is used to increase the contrast and suppress the non-uniform illumination, shading, and other imaging artifacts in the input image. The nuclei segmentation stage consists of a distance transformation, h-maxima transformation and watershed segmentation. These transformations utilize the intensity information and the convexity property of the nucleus for the purpose of detecting a single marker in every nucleus; these markers are then used in the h-TMC watershed algorithm to obtain segments of the nuclei. However, dust particles, imaging artifacts, or prolonged cell cytoplasm may falsely be segmented as nuclei at this stage, and thus may lead to an inaccurate analysis of the cell image. In order to identify and remove these non-nuclei segments, in the third stage a texture analysis is performed, that uses six of the Haralick measures along with the AdaBoost algorithm. The novelty of the proposed method is that it introduces a systematic framework that utilizes intensity, convexity, and texture information to achieve a high accuracy for automatic segmentation of nuclei in the phase-contrast images. Extensive experiments are performed demonstrating the superior performance ( precision = 0.948; recall = 0.924; F1-measure = 0.936; validation based on ∼4850 manually-labeled nuclei) of the proposed method.

Nucleus segmentation in histopathology images holds great clinical value for disease analysis. With the advancements in deep learning and continuous iteration of devices, existing nucleus segmentation algorithms can achieve satisfactory performance. However, these fully supervised nucleus segmentation methods require pixel-level annotation information, which requires a significant amount of time and effort. To ease this burden of annotations, we propose an envelope enhancement segmentation Network (EESNet) for weak supervison nucleus segmentation. Specifically, we first employ a Voronoi diagram and adaptive k-means clustering algorithm to derive two kinds of informative pixel-level pseudo labels,which are used for pixel-wise segmentation. Furthermore, we introduce an envelope enhancement network to provide rich envelope structure information to the segmentation network, which helps mitigate the problem of inaccurate boundary segmentation caused by point annotations. Finally, our network is verified on a mainstream nucleus segmentation datasets (consep) and achieves satisfactory performance.

Dysplasia is a common oral premalignant lesion type that can be classified as mild, moderate and severe. The diagnosis of different types of dysplasia is made by a pathologist through complex and time consuming histological image analysis. The use of computer-aided diagnosis can be applied as a tool to aid and enhance the pathologists decisions. Recently, deep learning based methods are earning more and more attention and have been successfully applied to nuclei segmentation problems in several scenarios. In this paper, we present a method for automated nuclei segmentation on dysplastic oral tissues histological images using convolutional neural networks. We also evaluated the impact of color normalization techniques applied to the automated nuclei segmentation task on hematoxylin-eosin stained histological images. The proposed method achieves the best results overall when validated against other segmentation methods using a dataset composed of mice tongue histopathological images.

In automatic cancer detection, nuclei segmentation is a very essential step which enables the classification task simpler and computationally more efficient. However, automatic nuclei detection is fraught with the problems of inter-class variability of nuclei size and shapes. In this research article, a novel unsupervised edge detection technique, is proposed for segmenting the nuclei regions in liver cancer Hematoxylin and Eosin (H&E) stained histopathology images. In this novel edge detection technique, the notion of computing local standard deviation is incorporated, instead of computing gradients. Since, local standard deviation value is correlated with the edge information of image, this novel method can extract the nuclei edges efficiently, even at multiscale. The edge-detected image is further converted into a binary image by employing Ostu (IEEE Trans Syst Man Cybern 9(1):62–66, 1979)’s thresholding operation. Subsequently, an adaptive morphological filter is also employed in order to refine the final segmented image. The proposed nuclei segmentation method is also tested on a well-recognized multi-organ dataset, in order to check its effectiveness over wide variety of dataset. The visual results of both datasets indicate that the proposed segmentation method overcomes the limitations of existing unsupervised methods, moreover, its performance is comparable with the same of recent deep neural models like DIST, HoverNet, etc. Furthermore, three quality metrics are computed in order to measure the performance of several nuclei segmentation methods quantitatively. The mean value of quality metrics reveals that proposed segmentation method indeed outperformed other existing nuclei segmentation methods.

Purpose:Image guided radiotherapy provides images not only for patientpositioning but also for online adaptive radiotherapy. Accurate delineationof organs-at-risk (OARs) on Head and Neck (H&N) CT and MR images isvaluable to both initial treatment planning and adaptive planning, butmanual contouring is laborious and inconsistent. A novel method based on thegenerative adversarial network (GAN) with shapeconstraint (SC-GAN) is developed for fullyautomated H&N OARs segmentation on CT and low field MRI.Methods and material:A deep supervised fully convolutional DenseNet is employed as thesegmentation network for voxel-wise prediction. A CNN based discriminatornetwork is then utilized to correct predicted errors and image-levelinconsistency between the prediction and ground truth. An additional shaperepresentation loss between the prediction and ground truth in the latentshape space is integrated into the segmentation and adversarial lossfunctions to reduce false positivity and constrain the predicted shapes. Theproposed segmentation method was first benchmarked on a public H&N CTdatabase including 32 patients, and then on 25 0.35T MR images obtained froman MR guided radiotherapy system. The OARs include brainstem, opticalchiasm, larynx (MR only), mandible, pharynx (MR only), parotid glands (bothleft and right), optical nerves (both left and right), and submandibularglands (both left and right, CT only). The performance of the proposedSC-GAN was compared with GAN alone and GAN with the shape constraint (SC)but without the Densenet (SC-GAN-ResNet) to quantify the contributions ofshape constraint and DenseNet in the deep neural network segmentation.Results:The proposed SC-GAN slightly but consistently improve thesegmentation accuracy on the benchmark H&N CT images compared with ourprevious deep segmentation network, which outperformed other publishedmethods on the same or similar CT H&N dataset. On the low field MRdataset, the following average Dice’s indices were obtained usingimproved SC-GAN: 0.916 (Brainstem), 0.589 (Optical chiasm), 0.816(Mandible), 0.703 (Optical nerves), 0.799 (Larynx), 0.706 (Pharynx), and0.845 (Parotid glands). The average surface distances ranged from 0.68mm(Brainstem) to 1.70mm (Larynx). The 95% surface distance ranged from 1.48mm(Left optical nerve) to 3.92mm (Larynx). Compared with CT, using 95% surfacedistance evaluation, the automated segmentation accuracy is higher on MR forthe brainstem, optical chiasm, optical nerves and parotids, and lower forthe mandible. The SC-GAN performance is superior to SC-GAN-ResNet, which ismore accurate than GAN alone on both the CT and MR datasets. Thesegmentation time for one patient is 14 seconds using a single GPU.Conclusion:The performance of our previous shape constrained fully convolutionalneural networks for H&N segmentation is further improved byincorporating GAN and DenseNet. With the novel segmentation method, weshowed that the low field MR images acquired on a MR guided radiationradiotherapy system can support accurate and fully automated segmentation ofboth bony and soft tissue OARs for adaptive radiotherapy.

To reduce external disturbances and achieve high vertical resolution, the scanning time for white-light interference microscopy is very short. Because capturing high-resolution (HR) images is time consuming, low-resolution (LR) images are acquired instead. However, HR images are more desirable because they contain more details. To ensure high vertical resolution and high image resolution, one feasible solution is to process the scanned LR images to HR images by single image super-resolution (SISR). In this paper, an interference image super-resolution (IISR) model based on a generative adversarial network (GAN) is proposed. The generator is based on the enhanced super-resolution generative adversarial network (ESRGAN) architecture. With the aim of acquiring more realistic images, the discriminator network is designed using a modified DenseNet architecture, in which the pooling layers are replaced with dilated convolutional layers. The perceptual loss is optimized, and the content loss is upgraded to a continuously differentiable piecewise function. Various microscopy images are tested, including images with and without interference fringes. The IISR model has been proven to restore LR images to HR images. The comparative experiments prove that the proposed model achieves better visual quality than other models, preserving more realistic details.

As a promising generative modeling method, Generative Adversarial Networks are a deep-learning-based generative model, in which two networks, namely the generative network and the discriminant network, contest with each other in a game during which the generator “learns” and get trained to be able to fool the discriminator of believing a specific image is (superficially) authentic. It is widely used to generate pictures based on existing data. Based on the MNIST data set, this paper studies and analyzes the application of Generative Adversarial Networks in the generation of handwritten digital images. In addition, by studying the Deep Convolutional Generative Adversarial Network and the Conditional Generation Adversarial Network model, we compared the connections and differences between these three models in training the MNIST dataset to generate handwritten digits algorithms, and gave an optimization method for generating handwritten digits.

Semi-supervised learning with GAN for automatic defect detection from images

Cervical cancer is identified as the fourth most recurrent cancer among women across the globe. The cancer is treatable, if identified at the early stage. Pap smear test is the most common and the best tool for initial screening of cancer. Pap smear cell level image analysis is an open issue. The limitation of the analysis is due to the complexity of the cell structure. The smear cell image is composed of cytoplasm and nucleus. The shape and structure of the nucleus determines the cancer prevalence. Segmentation of nucleus is an important step in cancer detection. There are various methods developed for nucleus segmentation. The article proposes multithresholding algorithm to segment cytoplasm and nucleus region from the background. Morphological operations are used for correcting the segmented output. Support vector machine classifier is used for classifying the smear cell as normal or abnormal based on the extracted features of the segmented output. The obtained accuracy of the classifier, sensitivity and specificity for single smear cell are 99.66%, 99.85% and 99.17%.

A data augmentation approach to train fully convolutional networks for left ventricle segmentation

To address the problems that existing recognition networks rely on a large amount of labeled data and the perceptual field is limited to the local area of convolution, and the lack of understanding of contextual information, this paper proposes a self-attentive semantic segmentation method based on generative adversarial networks. The method is based on generating adversarial networks, constructing semantic segmentation networks and discriminators, where the semantic segmentation network uses Resnet101 as the backbone to connect the spatial pyramid pooling module of PSPNet and adopts the cross-attention method in order to overcome the problem of too many parameters of classical attention models. The model was simulated in the publicly available PASCAL VOC 2012 dataset, and the results showed that the MIoU of the model reached 73.1%, 74.4%, and 75.1% for 1/8, 1/4, and 1/2 with labels, respectively, in the first semi-supervised experiments without improving the segmentation network compared to the control group, which were 3.6%, 2.3%, and 1.3% higher, respectively. The MIoU value reached 75.4% after improving the segmentation network, which proved the superiority and effectiveness of this model.