Boundary-Aware Supervoxel-Level Iteratively Refined Interactive 3D Image Segmentation With Multi-Agent Reinforcement Learning.

Interactive segmentation methods utilize user-provided positive and negative clicks to guide the model in accurately segmenting target objects. Compared to fully automatic medical image segmentation, these methods can achieve higher segmentation accuracy with limited image data, demonstrating significant potential in clinical applications. Typically, for each new click provided by the user, conventional interactive segmentation methods reprocess the entire network by re-inputting the click into the segmentation model, which greatly increases the user’s interaction burden and deviates from the intended goal of interactive segmentation tasks. To address this issue, we propose an efficient segmentation network, ESM-Net, for interactive medical image segmentation. It obtains high-quality segmentation masks based on the user’s initial clicks, reducing the complexity of subsequent refinement steps. Recent studies have demonstrated the strong performance of the Mamba model in various vision tasks; however, its application in interactive segmentation remains unexplored. In our study, we incorporate the Mamba module into our framework for the first time and enhance its spatial representation capabilities by developing a Spatial Augmented Convolution (SAC) module. These components are combined as the fundamental building blocks of our network. Furthermore, we designed a novel and efficient segmentation head to fuse multi-scale features extracted from the encoder, optimizing the generation of the predicted segmentation masks. Through comprehensive experiments, our method achieved state-of-the-art performance on three medical image datasets. Specifically, we achieved 1.43 NoC@90 on the Kvasir-SEG dataset, 1.57 NoC@90 on the CVC-ClinicDB polyp segmentation dataset, and 1.03 NoC@90 on the ADAM retinal disk segmentation dataset. The assessments on these three medical image datasets highlight the effectiveness of our approach in interactive medical image segmentation.

Interactive image segmentation is a challenging task and receives increasing attention recently; however, two major drawbacks exist in interactive segmentation approaches. First, the segmentation performance of region of interest (ROI)-based methods is sensitive to the initial ROI: different ROIs may produce results with great difference. Second, most seed-based methods need intense interactions, and are not applicable in many cases. In this paper, we generalize the neutro-connectedness (NC) to be independent of top-down priors of objects and to model image topology with indeterminacy measurement on image regions, propose a novel method for determining object and background regions, which is applied to exclude isolated background regions and enforce label consistency, and put forward a hybrid interactive segmentation method, NC Cut (NC-Cut), which can overcome the above two problems by utilizing both pixelwise appearance information and region-based NC properties. We evaluate the proposed NC-Cut by employing two image data sets (265 images), and demonstrate that the proposed approach outperforms the state-of-the-art interactive image segmentation methods (Grabcut, MILCut, One-Cut, MGCmaxsum, and pPBC).

Convolutional neural networks (CNNs) are the backbone of most modern interactive segmentation algorithms. However, the limited receptive field of CNNs restricts their ability to capture long-range semantic relationships. Recently, transformers have gained significant attention for their capacity to capture long-range dependencies. Nevertheless, CNNs still outperform Transformer in extracting local information. An effective interactive segmentation algorithm should accurately capture fine-grained local details alongside global semantic relationships. Therefore, we propose GLClick, a global–local click-based interactive image segmentation model that integrates local and global information through a novel fusion mechanism. We design an efficient global–local feature fusion module (GLFM) that integrates fine-grained features from various layers of ResNet50 with those from the Transformer feature pyramid. This approach maintains ResNet50’s ability to extract local features while effectively leveraging the Transformer to capture global context. Additionally, we enhance the multi-layer perceptron (MLP) to improve performance. Extensive experiments on diverse benchmark datasets demonstrate significant improvements in interactive image segmentation, confirming the effectiveness of our approach. Moreover, we conduct experiments on medical image datasets, further illustrating the model’s versatility and effectiveness across different domains.

One of the most important goals in image segmentation is the process of separating the object parts from the image background. Image segmentation is also a fundamental stage in the development of other image applications such as object recognition, target tracking, computer vision, and biomedical image processing. Interactive image segmentation methods with additional user interaction are still popular in research. Interactive image segmentation aims to provide additional information through simple interactions, especially in images with complex objects. Interactive image segmentation with region merging processes has drawbacks, one of which is suboptimal region splitting due to soft color shades, blurred contours, and uneven lighting, referred to in this study as ambiguous regions. However, in the fuzzy region initialization stage after obtaining values from the marker process, there is a possibility of missing or suboptimal determination of fuzzy regions. This is because it only takes the highest gray level value for the background marker and the lowest gray level value for the object marker. In this study, fuzzy region merging using hierarchical clustering is proposed to find optimal initialization for fuzzy regions in image segmentation. Based on the experimental results, the proposed method can achieve optimal segmentation with an average misclassification error value of 2.62% for Natural Images and 9.33% for Dental Images.

Interactive image segmentation enables users to interact minimally with a machine, facilitating the gradual refinement of the segmentation mask for a target of interest. Previous studies have demonstrated impressive performance in extracting a single target mask through interactive segmentation. However, existing methods often overlook the information cues from previously interacted objects, despite realworld scenarios frequently involving multiple objects in the same category, leading to increased redundant user effort. To this end, this chapter introduces a new interactive image segmentation framework for multiple objects within the same category in a single image. Specifically, this work leverages Transformers to extract interactionfocused visual features from the image and the interactions, thereby obtaining a satisfactory mask of the target as an exemplar. For multiple objects, an exemplar -informed module is proposed to enhance the learning of similarities among the objects of the target category. To combine features from different modules, cross-attention blocks are incorporated, followed by a feature fusion module. Extensive experiments conducted on different datasets demonstrate that the proposed framework achieves superior performance compared to previous methods. Particularly, it reduces users’ labor by around 15%, requiring two fewer clicks to achieve target IoUs of 85% and 90% for multiple objects. These results further highlight the proposed method’s potential as a flexible and practical annotation tool.

Semi-automated segmentation or more commonly known as interactive image segmentation is an algorithm that extracts a region of interest (ROI) from an image based on the input information from the user. The said algorithm will be repetitively fed with such input information until required region of interest is successfully segmented. To accelerate this segmentation procedure as well as enhancing the result, pre-processing steps can be applied. The application of superpixel is an example of such pre-processing step. Superpixel can be defined as a collection of pixels that share common features such as texture and colours. Though employed as pre-processing step in many interactive segmentation algorithms, to date, no study has been conducted to assess the effects of such incorporations on the segmentation algorithms. Thus, this study aims to address this issue. In this study, five different types of superpixels ranging from watershed, density, graph, clustering and energy optimization categories are evaluated. The superpixels generated by these five algorithms will be used on two interactive image segmentation algorithms: i) Maximal Similarity based Region Merging (MSRM) and ii) Graph-Based Manifold Ranking (GBMR) with single and multiple strokes on various images from the Berkeley image dataset. The result of testing had shown that MSRM achieved better result compared to GBMR in both single and multiple input strokes using SEEDS superpixel algorithm. This study summary concluded that at different superpixel algorithms produced different results and that it is not possible to single out one particular superpixel algorithm that can work well for all the interactive segmentation algorithms. As such, the key to achieving a decent segmentation result lies in choosing the right superpixel algorithms for a given interactive segmentation algorithm.

This study investigates the impact of image complexity on the accuracy of interactive image segmentation algorithms. Image complexity plays a crucial role in segmentation performance, yet previous studies have primarily relied on subjective methods, leaving a gap in understanding how objective measures impact accuracy. The purpose of this research is to explore the relationship between image complexity and segmentation performance and to propose an adaptive approach for improving accuracy based on complexity measures. The study utilizes objective measures, namely entropy and fractal dimension, to quantify image complexity. An interactive image segmentation algorithm is employed, with a bounding box as the background and strokes as the foreground annotations. The number of strokes is dynamically adjusted based on complexity measures, ensuring a tailored segmentation approach. Comparative evaluations are conducted to assess the effectiveness of dynamic and fixed stroke allocation strategies. The principal results reveal a significant influence of image complexity on segmentation accuracy. The dynamic stroke allocation strategy outperforms fixed allocation, highlighting the importance of adapting to complexity. Moreover, the optimal combination of strokes and superpixel sizes is explored, providing valuable insights for practitioners. The findings emphasize the need to consider image complexity in segmentation algorithms to achieve accurate results. In conclusion, this study contributes to the understanding of the relationship between image complexity and interactive image segmentation. By introducing a dynamic stroke allocation approach and evaluating different configurations, the research provides insights into optimizing accuracy based on image complexity. The adaptive strategy improves segmentation performance and guides the development of robust algorithms. Future research can further refine the adaptive approach, explore additional complexity measures, and incorporate advanced machine learning techniques to enhance interactive image segmentation. Overall, this study advances the field by highlighting the importance of image complexity, providing guidance for practitioners, and paving the way for more efficient segmentation algorithms.

Interactive segmentation offers greater clinical potential in computational pathology compared to traditional automatic segmentation. By incorporating interactive input, it addresses the limitations of fully automatic segmentation models, which often fail to meet pathologists' requirements and rely heavily on large-scale, pixel-level annotated datasets. However, current interactive segmentation methods struggle to balance interaction cost and segmentation performance, and they fail to adapt effectively to slide-level segmentation, a task that is even more crucial in routine pathology analysis. In this study, we propose a Prototype-Guided Interactive Segmentation (ProGIS) framework for pathological image segmentation, designed to deliver precise segmentation results efficiently with minimal interaction signals. ProGIS identifies all same-type tissue connected components in a single interaction and supports multi-class segmentation without predefined categories during inference. Moreover, ProGIS can be easily adapted for slide-level interactive segmentation. Specifically, ProGIS consists of three modules: Prototype Initialization, Prototype Navigation, and Local Refinement. First, the Prototype Initialization module identifies categorical prototypes, which are then utilized in the Prototype Navigation module to identify all tissue connected components belonging to the same type. The local refinement module further refines the segmentation results using detailed correction signals to ensure the accuracy of challenging-to-distinguish regions. We evaluate our framework on two regions of interest level and two slide-level pathological segmentation datasets, achieving new state-of-the-art performance with fewer interactions than existing methods. Our code is available at https://github.com/JSGe-AI/ProGIS.

Image segmentation is a necessary but difficult task in many image processing applications. Unlike conventional auto-segmentation, semi-supervised image segmentation involves a moderate amount of user interaction. In this paper, a novel interactive image segmentation method based on diffusion maps is proposed, which can better account for the distribution of pixels in feature space. Our method replaces the pixel features of Euclidean distance with diffusion maps and performs feature optimization and dimensionality reduction in order to achieve a final feature vector, with the assistance of user interaction. On this basis, L0 gradient minimization is applied to achieve steady segmentation results. Finally, the K-means clustering algorithm is applied to perfect the segmentation. Experiments show that our method can obtain better segmentation results than several state-of-the-art interactive image segmentation methods.

Nuclear forensics relies on different signatures to identify the source of nuclear material. Such signatures include crystalline structure, chemical composition, and particle morphology. One way to quantify morphology in electron microscope imagery is through image segmentation, where pixels are assigned to several partitions (or groups) that correspond to particles, grains, and other objects of interest within the image. Once pixels are assigned to segments, it is relatively straightforward to quantify other quantities of interest, such as grain size, circularity, etc. However, the range and diversity of microscope images make it difficult to obtain an accurate segmentation automatically. The accuracy of segmentation can be improved through supervised learning, but this requires many images to be manually segmented. Another way to improve the accuracy is to use interactive segmentation. Interactive segmentation requires a human to provide image-specific user input to improve performance. However, the amount of user input (effort) is generally far less than is required for supervised learning. In this paper, we investigate several parallelization strategies to automatically explore the user input parameter space of interactive segmentation algorithms across a large number of images. Specifically, we investigate four different parallelization mechanisms in a high-performance computing (HPC) environment and use the Amdahl fraction to evaluate efficiency on multiple processor cores across multiple nodes. Ultimately, the parallelization strategy that was most efficient utilized the message passing interface integrated with the segmentation and quantification code. This strategy had an Amdahl fraction of 0.985 and a performance of about 0.251 s/image. These results indicate that the parameter space of interactive segmentation algorithms can be efficiently explored using HPC. This opens the door to future work where user input is reduced and where interactive image segmentation algorithms are automatically applied to large image sets.

In this paper, the origin of random walk with restart (RWR) and its generalization are described. It is well known that the random walk (RW) and the anisotropic diffusion models share the same energy functional, i.e., the former provides a steady-state solution and the latter gives a flow solution. In contrast, the theoretical background of the RWR scheme is different from that of the diffusion-reaction equation, although the restarting term of the RWR plays a role similar to the reaction term of the diffusion-reaction equation. The behaviors of the two approaches with respect to outliers reveal that they possess different attributes in terms of data propagation. This observation leads to the derivation of a new energy functional, where both volumetric heat capacity and thermal conductivity are considered together, and provides a common framework that unifies both the RW and the RWR approaches, in addition to other regularization methods. The proposed framework allows the RWR to be generalized (GRWR) in semilocal and nonlocal forms. The experimental results demonstrate the superiority of GRWR over existing regularization approaches in terms of depth map up-sampling and interactive image segmentation.

It is hard for current interactive stereo image segmentation methods to deal with large scale images with fast feedback after each interaction. In this paper, we present an interactive stereo image segmentation method. Different from current methods, our method introduces a multi-scale graph structure for fast graph cut optimization. Besides, we use GPU parallel computing to handle single instruction multiple data tasks involved in the segmentation. Compared with state-of-the-art methods, our approach significantly accelerates segmentation speed. In the meanwhile, our method obtains segmentation accuracy comparable with state-of-the-art methods.

Image segmentation for object to extract data from ultrasound acquired is an essential preprocessing step for the effective diagnosis. Various image segmentation methods have been studied. In this study, interactive image segmentation method by graph cut algorithm is proposed to develop a variety of applications of vascular ultrasound imaging and diagnostics. General imaging and vascular ultrasound imaging segmentation by entering constrain condition such as foreground and background. In the future it will be able to develop new ultrasound diagnostics.

The image segmentation of histopathological tissue images has always been a challenge due to the overlapping of tissue color distributions, the complexity of extracellular texture and the large image size. In this paper, we introduce a new region-merging algorithm, namely, the Regional Pattern Merging (RPM) for interactive color image segmentation and annotation, by efficiently retrieving and applying the user’s prior knowledge of stroke-based interaction. Low-level color/texture features of each region are used to compose a regional pattern adapted to differentiating a foreground object from the background scene. This iterative region-merging is based on a modified Region Adjacency Graph (RAG) model built from initial segmented results of the mean shift to speed up the merging process. The foreground region of interest (ROI) is segmented by the reduction of the background region and discrimination of uncertain regions. We then compare our method against state-of-the-art interactive image segmentation algorithms in both natural images and histological images. Taking into account the homogeneity of both color and texture, the resulting semi-supervised classification and interactive segmentation capture histological structures more completely than other intensity or color-based methods. Experimental results show that the merging of the RAG model runs in a linear time according to the number of graph edges, which is essentially faster than both traditional graph-based and region-based methods.

Many medical image analysis systems require segmentation of the structures of interest as a first step. For scans with gross pathology, automatic segmentation methods may fail. The authors' aim is to develop a versatile, fast, and reliable interactive system to segment anatomical structures. In this study, this system was used for segmenting lungs in challenging thoracic computed tomography (CT) scans. In volumetric thoracic CT scans, the chest is segmented and divided into 3D volumes of interest (VOIs), containing voxels with similar densities. These VOIs are automatically labeled as either lung tissue or nonlung tissue. The automatic labeling results can be corrected using an interactive or a supervised interactive approach. When using the supervised interactive system, the user is shown the classification results per slice, whereupon he/she can adjust incorrect labels. The system is retrained continuously, taking the corrections and approvals of the user into account. In this way, the system learns to make a better distinction between lung tissue and nonlung tissue. When using the interactive framework without supervised learning, the user corrects all incorrectly labeled VOIs manually. Both interactive segmentation tools were tested on 32 volumetric CT scans of pigs, mice and humans, containing pulmonary abnormalities. On average, supervised interactive lung segmentation took under 9 min of user interaction. Algorithm computing time was 2 min on average, but can easily be reduced. On average, 2.0% of all VOIs in a scan had to be relabeled. Lung segmentation using the interactive segmentation method took on average 13 min and involved relabeling 3.0% of all VOIs on average. The resulting segmentations correspond well to manual delineations of eight axial slices per scan, with an average Dice similarity coefficient of 0.933. The authors have developed two fast and reliable methods for interactive lung segmentation in challenging chest CT images. Both systems do not require prior knowledge of the scans under consideration and work on a variety of scans.