Interactive medical image segmentation via a point-based interaction

Interactive segmentation methods utilize user-provided positive and negative clicks to guide the model in accurately segmenting target objects. Compared to fully automatic medical image segmentation, these methods can achieve higher segmentation accuracy with limited image data, demonstrating significant potential in clinical applications. Typically, for each new click provided by the user, conventional interactive segmentation methods reprocess the entire network by re-inputting the click into the segmentation model, which greatly increases the user’s interaction burden and deviates from the intended goal of interactive segmentation tasks. To address this issue, we propose an efficient segmentation network, ESM-Net, for interactive medical image segmentation. It obtains high-quality segmentation masks based on the user’s initial clicks, reducing the complexity of subsequent refinement steps. Recent studies have demonstrated the strong performance of the Mamba model in various vision tasks; however, its application in interactive segmentation remains unexplored. In our study, we incorporate the Mamba module into our framework for the first time and enhance its spatial representation capabilities by developing a Spatial Augmented Convolution (SAC) module. These components are combined as the fundamental building blocks of our network. Furthermore, we designed a novel and efficient segmentation head to fuse multi-scale features extracted from the encoder, optimizing the generation of the predicted segmentation masks. Through comprehensive experiments, our method achieved state-of-the-art performance on three medical image datasets. Specifically, we achieved 1.43 NoC@90 on the Kvasir-SEG dataset, 1.57 NoC@90 on the CVC-ClinicDB polyp segmentation dataset, and 1.03 NoC@90 on the ADAM retinal disk segmentation dataset. The assessments on these three medical image datasets highlight the effectiveness of our approach in interactive medical image segmentation.

Interactive 3D object segmentation is an important and challenging activity in medical imaging, although it is tedious and error-prone to be done. Automatic segmentation methods aim to replace the user altogether, but require user interaction to produce training data sets of segmented masks and to make error corrections. We propose a complete framework for interactive medical image segmentation, which reduces user effort by automatically providing an initial segmentation result. We develop a Statistical Seed Model (SSM) to this end, that improves from seed sets selected by robot users when reconstructing masks of previously segmented images. The SSM outputs a seed set that may be used to automatically delineate a new test image. The seeds provide both an implicit object shape constraint and a flexible way of interactively correcting segmentation. We demonstrate that our framework decreases the amount of user interaction by a factor of three, when segmenting MR-images of the cerebellum.

Snakes have become a standard image analysis technique with several variants now in common use. We have developed a software package called “United Snakes”. It unifies the most important snake variants, including finite difference, B-spline, and Hermite polynomial snakes, within the framework of a general finite element formulation with a choice of shape functions. Furthermore, we have incorporated into united snakes a recently proposed snake-like technique known as “livewire”, via a method for imposing hard constraints on snakes. Here, we demonstrate that the combination of techniques in united snakes yields generality, accuracy, ease of use, and robustness in several medical image analysis applications, including the segmentation of neuronal dendrites in EM images, dynamic chest image analysis, and the quantification of growth plates.

Deep learning-based segmentation algorithms for medical image require massive training datasets with accurate annotations, which is costly since it takes much human effort to manually labeling from scratch. Therefore, interactive image segmentation is important and may greatly improve the efficiency and accuracy of medical image labeling. Some interactive segmentation methods (e.g. Deep Extreme Cut and Deepgrow) may improve the labeling through minimal interactive input. However these methods only utilize the initial manually input information, while existing segmentation results (such as annotations produced by nonprofessionals or conventional segmentation algorithms) cannot be utilized. In this paper, an interactive segmentation method is proposed to make use of both existing segmentation results and human interactive information to optimize the segmentation results progressively. In this framework, the user only needs to click on the foreground or background of the target individual on the medical image, the algorithm could adaptively learn the correlation between them, and automatically completes the segmentation of the target. The main contributions of this paper are: (1) We adjusted and applied a convolutional neural network which takes medical image data and user's clicks information as input to achieve more accurate segmentation of medical images. (2) We designed an iterative training strategy to realize the applicability of the model to deal with different number of clicks data input. (3) We designed an algorithm based on false positive and false negative regions to simulate the user's clicks, so as to provide enough training data. By applying the proposed method, users can easily extract the region of interest or modify the segmentation results by multiple clicks. The experimental results of 6 medical image segmentation tasks show that the proposed method achieves more accurate segmentation results by at most five clicks.

Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints. While iterative in nature, most existing interactive segmentation methods tend to ignore the dynamics of successive interactions and take each interaction independently. We here propose to model iterative interactive image segmentation with a Markov decision process (MDP) and solve it with reinforcement learning (RL) where each voxel is treated as an agent. Considering the large exploration space for voxel-wise prediction and the dependence among neighboring voxels for the segmentation tasks, multi-agent reinforcement learning is adopted, where the voxel-level policy is shared among agents. Considering that boundary voxels are more important for segmentation, we further introduce a boundary-aware reward, which consists of a global reward in the form of relative cross-entropy gain, to update the policy in a constrained direction, and a boundary reward in the form of relative weight, to emphasize the correctness of boundary predictions. To combine the advantages of different types of interactions, i. e., simple and efficient for point-clicking, and stable and robust for scribbles, we propose a supervoxel-clicking based interaction design. Experimental results on four benchmark datasets have shown that the proposed method significantly outperforms the state-of-the-arts, with the advantage of fewer interactions, higher accuracy, and enhanced robustness.

In this paper, a novel interactive medical image segmentation method called SMOPL is proposed. This method only needs marking some pixels on foreground region for segmentation. To do this, SMOPL characterize the inherent correlations among foreground and background pixels as Hilbert-Schmidt independence. By maximizing the independence and minimizing the smoothness of labels on instance neighbor graph simultaneously, SMOPL gets the sufficiently smooth confidences of both positive and negative classes in absence of negative training examples. Then a image segmentation can be obtained by assigning each pixel to the label for which the greatest confidence is calculated. Experiments on real-world medical images show that SMOPL is robust to get a high-quality segmentation with only positive label examples.

This thesis presents an intuitive, fast and accurate interactive segmentation method for visualizing and analyzing 3D medical images. This method combines a general deformable subdivision surface model with a novel sketch-line user initialization process. The model is simply and precisely initialized with a few quick sketch lines drawn across the width of the target object on several key slices of the volume image. The smooth surface constructed using these lines is extremely close to the shape of the object boundary, making the model's task of snapping to this boundary much simpler and hence more likely to succeed in noisy images with minimal user editing. This subdivision-surface based deformable model provides a foundation for precise user steering/editing capabilities and all of the simple, intuitive user interactions are seamlessly integrated with advanced visualization capabilities. Furthermore, to demonstrate its efficiency and accuracy, this new model has been used to segment objects from several 3D data sets.

Convolutional neural networks (CNNs) are the backbone of most modern interactive segmentation algorithms. However, the limited receptive field of CNNs restricts their ability to capture long-range semantic relationships. Recently, transformers have gained significant attention for their capacity to capture long-range dependencies. Nevertheless, CNNs still outperform Transformer in extracting local information. An effective interactive segmentation algorithm should accurately capture fine-grained local details alongside global semantic relationships. Therefore, we propose GLClick, a global–local click-based interactive image segmentation model that integrates local and global information through a novel fusion mechanism. We design an efficient global–local feature fusion module (GLFM) that integrates fine-grained features from various layers of ResNet50 with those from the Transformer feature pyramid. This approach maintains ResNet50’s ability to extract local features while effectively leveraging the Transformer to capture global context. Additionally, we enhance the multi-layer perceptron (MLP) to improve performance. Extensive experiments on diverse benchmark datasets demonstrate significant improvements in interactive image segmentation, confirming the effectiveness of our approach. Moreover, we conduct experiments on medical image datasets, further illustrating the model’s versatility and effectiveness across different domains.

Interactive image segmentation is a challenging task and receives increasing attention recently; however, two major drawbacks exist in interactive segmentation approaches. First, the segmentation performance of region of interest (ROI)-based methods is sensitive to the initial ROI: different ROIs may produce results with great difference. Second, most seed-based methods need intense interactions, and are not applicable in many cases. In this paper, we generalize the neutro-connectedness (NC) to be independent of top-down priors of objects and to model image topology with indeterminacy measurement on image regions, propose a novel method for determining object and background regions, which is applied to exclude isolated background regions and enforce label consistency, and put forward a hybrid interactive segmentation method, NC Cut (NC-Cut), which can overcome the above two problems by utilizing both pixelwise appearance information and region-based NC properties. We evaluate the proposed NC-Cut by employing two image data sets (265 images), and demonstrate that the proposed approach outperforms the state-of-the-art interactive image segmentation methods (Grabcut, MILCut, One-Cut, MGCmaxsum, and pPBC).

Interactive image segmentation enables users to interact minimally with a machine, facilitating the gradual refinement of the segmentation mask for a target of interest. Previous studies have demonstrated impressive performance in extracting a single target mask through interactive segmentation. However, existing methods often overlook the information cues from previously interacted objects, despite realworld scenarios frequently involving multiple objects in the same category, leading to increased redundant user effort. To this end, this chapter introduces a new interactive image segmentation framework for multiple objects within the same category in a single image. Specifically, this work leverages Transformers to extract interactionfocused visual features from the image and the interactions, thereby obtaining a satisfactory mask of the target as an exemplar. For multiple objects, an exemplar -informed module is proposed to enhance the learning of similarities among the objects of the target category. To combine features from different modules, cross-attention blocks are incorporated, followed by a feature fusion module. Extensive experiments conducted on different datasets demonstrate that the proposed framework achieves superior performance compared to previous methods. Particularly, it reduces users’ labor by around 15%, requiring two fewer clicks to achieve target IoUs of 85% and 90% for multiple objects. These results further highlight the proposed method’s potential as a flexible and practical annotation tool.

Semi-automated segmentation or more commonly known as interactive image segmentation is an algorithm that extracts a region of interest (ROI) from an image based on the input information from the user. The said algorithm will be repetitively fed with such input information until required region of interest is successfully segmented. To accelerate this segmentation procedure as well as enhancing the result, pre-processing steps can be applied. The application of superpixel is an example of such pre-processing step. Superpixel can be defined as a collection of pixels that share common features such as texture and colours. Though employed as pre-processing step in many interactive segmentation algorithms, to date, no study has been conducted to assess the effects of such incorporations on the segmentation algorithms. Thus, this study aims to address this issue. In this study, five different types of superpixels ranging from watershed, density, graph, clustering and energy optimization categories are evaluated. The superpixels generated by these five algorithms will be used on two interactive image segmentation algorithms: i) Maximal Similarity based Region Merging (MSRM) and ii) Graph-Based Manifold Ranking (GBMR) with single and multiple strokes on various images from the Berkeley image dataset. The result of testing had shown that MSRM achieved better result compared to GBMR in both single and multiple input strokes using SEEDS superpixel algorithm. This study summary concluded that at different superpixel algorithms produced different results and that it is not possible to single out one particular superpixel algorithm that can work well for all the interactive segmentation algorithms. As such, the key to achieving a decent segmentation result lies in choosing the right superpixel algorithms for a given interactive segmentation algorithm.

MotivationOver the last decades, image processing and analysis have become one of the key technologies in systems biology and medicine. The quantification of anatomical structures and dynamic processes in living systems is essential for understanding the complex underlying mechanisms and allows, i.e. the construction of spatio-temporal models that illuminate the interplay between architecture and function. Recently, deep learning significantly improved the performance of traditional image analysis in cases where imaging techniques provide large amounts of data. However, if only a few images are available or qualified annotations are expensive to produce, the applicability of deep learning is still limited.ResultsWe present a novel approach that combines machine learning-based interactive image segmentation using supervoxels with a clustering method for the automated identification of similarly colored images in large image sets which enables a guided reuse of interactively trained classifiers. Our approach solves the problem of deteriorated segmentation and quantification accuracy when reusing trained classifiers which is due to significant color variability prevalent and often unavoidable in biological and medical images. This increase in efficiency improves the suitability of interactive segmentation for larger image sets, enabling efficient quantification or the rapid generation of training data for deep learning with minimal effort. The presented methods are applicable for almost any image type and represent a useful tool for image analysis tasks in general.Availability and implementationThe presented methods are implemented in our image processing software TiQuant which is freely available at tiquant.hoehme.com.Supplementary informationSupplementary data are available at Bioinformatics online.

This study investigates the impact of image complexity on the accuracy of interactive image segmentation algorithms. Image complexity plays a crucial role in segmentation performance, yet previous studies have primarily relied on subjective methods, leaving a gap in understanding how objective measures impact accuracy. The purpose of this research is to explore the relationship between image complexity and segmentation performance and to propose an adaptive approach for improving accuracy based on complexity measures. The study utilizes objective measures, namely entropy and fractal dimension, to quantify image complexity. An interactive image segmentation algorithm is employed, with a bounding box as the background and strokes as the foreground annotations. The number of strokes is dynamically adjusted based on complexity measures, ensuring a tailored segmentation approach. Comparative evaluations are conducted to assess the effectiveness of dynamic and fixed stroke allocation strategies. The principal results reveal a significant influence of image complexity on segmentation accuracy. The dynamic stroke allocation strategy outperforms fixed allocation, highlighting the importance of adapting to complexity. Moreover, the optimal combination of strokes and superpixel sizes is explored, providing valuable insights for practitioners. The findings emphasize the need to consider image complexity in segmentation algorithms to achieve accurate results. In conclusion, this study contributes to the understanding of the relationship between image complexity and interactive image segmentation. By introducing a dynamic stroke allocation approach and evaluating different configurations, the research provides insights into optimizing accuracy based on image complexity. The adaptive strategy improves segmentation performance and guides the development of robust algorithms. Future research can further refine the adaptive approach, explore additional complexity measures, and incorporate advanced machine learning techniques to enhance interactive image segmentation. Overall, this study advances the field by highlighting the importance of image complexity, providing guidance for practitioners, and paving the way for more efficient segmentation algorithms.

We combine a pen and pressure-sensitive tablet input device, and a sketch-based user initialization process, with a general subdivision-curve Snake to create an intuitive, fast, accurate, interactive model-based segmentation method. Using the pen input device, the Snake is quickly and precisely initialized with a series of sketch lines such that it is extremely close to the position and shape of the target object boundary, making the Snake's task much simpler and hence more likely to succeed in noisy images with minimal user editing. The user may also use pen pressure levels to easily impart knowledge of object edge strength to the model, and the low degree-of-freedom subdivision curve Snake provides powerful control and editing capabilities. We apply our Snake to the segmentation of several 2D medical images.

In this paper, the origin of random walk with restart (RWR) and its generalization are described. It is well known that the random walk (RW) and the anisotropic diffusion models share the same energy functional, i.e., the former provides a steady-state solution and the latter gives a flow solution. In contrast, the theoretical background of the RWR scheme is different from that of the diffusion-reaction equation, although the restarting term of the RWR plays a role similar to the reaction term of the diffusion-reaction equation. The behaviors of the two approaches with respect to outliers reveal that they possess different attributes in terms of data propagation. This observation leads to the derivation of a new energy functional, where both volumetric heat capacity and thermal conductivity are considered together, and provides a common framework that unifies both the RW and the RWR approaches, in addition to other regularization methods. The proposed framework allows the RWR to be generalized (GRWR) in semilocal and nonlocal forms. The experimental results demonstrate the superiority of GRWR over existing regularization approaches in terms of depth map up-sampling and interactive image segmentation.