Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Hidden Markov models (HMMs) have previously been successfully applied to subsurface threat detection using ground penetrating radar (GPR) data. However, parameter estimation in most HMM-based landmine detection approaches is difficult since object locations are typically well known for the 2-D coordinates on the Earth's surface but are not well known for object depths underneath the ground/time of arrival in a GPR A-scan. As a result, in a standard expectation maximization HMM (EM-HMM), all depths corresponding to a particular alarm location may be labeled as target sequences although the characteristics of data from different depths are substantially different. In this paper, an alternate HMM approach is developed using a multiple-instance learning (MIL) framework that considers an unordered set of HMM sequences at a particular alarm location, where the set of sequences is defined as positive if at least one of the sequences is a target sequence; otherwise, the set is defined as negative. Using the MIL framework, a collection of these sets (bags), along with their labels is used to train the target and nontarget HMMs simultaneously. The model parameters are inferred using variational Bayes, making the model tractable and computationally efficient. Experimental results on two synthetic and two landmine data sets show that the proposed approach performs better than a standard EM-HMM.

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

For the problem of object-based image retrieval, in this paper a novel semi-supervised multiple instance learning algorithm is presented. In the framework of multiple instance learning, this algorithm regards the whole image as a bag, and low-level visual feature of the segmented regions as instances. Firstly, the algorithm clusters the instances in two sets, one of which is composed of instances in positive bags and the other is composed of instances in negative bags, so as to find potential positive instances and feature data of bag structure. Then their respective similarities are measured by radial basis function, and an alpha coefficient is introduced in bag similarity measure as the trade-off between the two similarities. Experiments on SIVAL dataset show that this algorithm is feasible and the performance is superior to other algorithms.

Object proposals are a series of candidate segments containing objects of interest, which are taken as preprocessing and widely applied in various vision tasks. However, most of existing saliency approaches only utilize the proposals to compute a location prior. In this paper, we naturally take the proposals as the bags of instances of multiple instance learning (MIL), where the instances are the superpixels contained in the proposals, and formulate saliency detection problem as a MIL task (i.e., predict the labels of instances using the classifier in the MIL framework). This method allows some flexibility in finding a decision boundary based on the bag-level representations and can identify salient superpixels from ambiguous proposals. In addition, we introduce the MIL to an optimization mechanism, which iteratively updates training bags from easy to complex ones to learn a strong model. The significant improvement can be consistently achieved when applying the optimization model to existing saliency approaches. Extensive experiments demonstrate that the proposed algorithms perform favorably against the stateof- art saliency detection methods on several benchmark datasets.

Combining big data and deep learning can analyze large-scale breast cancer pathology images for auxiliary diagnosis. Furthermore, Whole Slide Images (WSIs) of breast cancer pathology offer detailed tissue feature information, which supports the accurate identification of malignant lesions. Current approaches combine Self-Supervised Learning (SSL) and Multiple Instance Learning (MIL) for WSI analysis, aiming to address the issues of billion-level pixels in a single WSI and the lack of precise annotations. However, pseudo-labels produced by SSL frequently lack accuracy, and MIL fails to effectively integrate global information at the WSI level, resulting in performance bottlenecks. This paper proposes the Multi-granularity Self-supervised Second-order MIL (MS2M) to tackle these issues. MS2M first achieves instance-level fine-grained feature learning through multi-granularity SSL and optimizes instance-level representations using bag-level labels within the MIL framework. Then, the transformer captures long-range dependencies between instances. When combined with second-order (covariance) pooling, it also captures high-order relational information. This process generates a robust bag-level representation. MS2M achieves accuracies of 0.9845 and 0.9719 on the CAMELYON16 and private breast cancer WSI datasets, respectively, outperforming existing methods.

Instance segmentation and panoptic segmentation is being paid more and more attention in recent years. In comparison with bounding box based object detection and semantic segmentation, instance segmentation can provide more analytical results at pixel level. Given the insight that pixels belonging to one instance have one or more common attributes of current instance, we bring up an one-stage instance segmentation network named Common Attribute Support Network (CASNet), which realizes instance segmentation by predicting and clustering common attributes. CASNet is designed in the manner of fully convolutional and can implement training and inference from end to end. And CASNet manages predicting the instance without overlaps and holes, which problem exists in most of current instance segmentation algorithms. Furthermore, it can be easily extended to panoptic segmentation through minor modifications with little computation overhead. CASNet builds a bridge between semantic and instance segmentation from finding pixel class ID to obtaining class and instance ID by operations on common attribute. Through experiment for instance and panoptic segmentation, CASNet gets mAP 32.8% and PQ 59.0% on Cityscapes validation dataset by joint training, and mAP 36.3% and PQ 66.1% by separated training mode. For panoptic segmentation, CASNet gets state-of-the-art performance on the Cityscapes validation dataset.

Accurate organ-level segmentation is essential for achieving high-throughput, non-destructive, and automated plant phenotyping. To address the challenge of intelligent acquisition of phenotypic parameters in tomato plants, we propose TSINet, an end-to-end dual-task segmentation network designed for effective and precise semantic labeling and instance recognition of tomato point clouds, based on the Pheno4D dataset. TSINet adopts an encoder–decoder architecture, where a shared encoder incorporates four Geometry-Aware Adaptive Feature Extraction Blocks (GAFEBs) to effectively capture local structures and geometric relationships in raw point clouds. Two parallel decoder branches are employed to independently decode shared high-level features for the respective segmentation tasks. Additionally, a Dual Attention-Based Feature Enhancement Module (DAFEM) is introduced to further enrich feature representations. The experimental results demonstrate that TSINet achieves superior performance in both semantic and instance segmentation, particularly excelling in challenging categories such as stems and large-scale instances. Specifically, TSINet achieves 97.00% mean precision, 96.17% recall, 96.57% F1-score, and 93.43% IoU in semantic segmentation and 81.54% mPrec, 81.69% mRec, 81.60% mCov, and 86.40% mWCov in instance segmentation. Compared with state-of-the-art methods, TSINet achieves balanced improvements across all metrics, significantly reducing false positives and false negatives while enhancing spatial completeness and segmentation accuracy. Furthermore, we conducted ablation studies and generalization tests to systematically validate the effectiveness of each TSINet component and the overall robustness of the model. This study provides an effective technological approach for high-throughput automated phenotyping of tomato plants, contributing to the advancement of intelligent agricultural management.

In this work, we propose a single deep neural network for panoptic segmentation, for which the goal is to provide each individual pixel of an input image with a class label, as in semantic segmentation, as well as a unique identifier for specific objects in an image, following instance segmentation. Our network makes joint semantic and instance segmentation predictions and combines these to form an output in the panoptic format. This has two main benefits: firstly, the entire panoptic prediction is made in one pass, reducing the required computation time and resources; secondly, by learning the tasks jointly, information is shared between the two tasks, thereby improving performance. Our network is evaluated on two street scene datasets: Cityscapes and Mapillary Vistas. By leveraging information exchange and improving the merging heuristics, we increase the performance of the single network, and achieve a score of 23.9 on the Panoptic Quality (PQ) metric on Mapillary Vistas validation, with an input resolution of 640 x 900 pixels. On Cityscapes validation, our method achieves a PQ score of 45.9 with an input resolution of 512 x 1024 pixels. Moreover, our method decreases the prediction time by a factor of 2 with respect to separate networks.

Automatic computer-aided detection plays an important role in biomedical image analysis. Many studies have focused on weak supervised learning as annotation tasks are time-consuming and tedious. Compared with pixel-wise annotation by particular software on the scanned digital high-resolution images, an alternative method of marking out of suspicious regions on microscopy slides is significantly more convenient for pathologists. Additionally, with a focus on dysplasias in the central area, there is a high likelihood of the similar tissues to be found around in clusters. In this paper, for weak annotation on microscopy images, we propose an efficient Foveated Imaging based Multiple Instance Learning (FIMIL) framework to classify weakly-labeled microscopy images. The model also provides multi-scale algorithm for arbitrary image size, in which the patches with highest possibility to contain dysplasia are considered as ”fixation points” in the image. The developed model combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) for dysplasias detection with only image-level labeling. The benchmark tests are carried out on the marked regions of 40x magnified whole-slide cytology images and the normal/abnormal label and their corresponding possibilities are predicted. Evaluated on the real-life clinical data, our proposed model shows high accuracy and efficiency by weakly-supervised learning. 1

Due to the trade-off between spatial resolution of the imagery and satellite revisit times, the research topics using multiresolution remote sensing data have attracted much attention. In recent years, the techniques for improving the visibility of multi-resolution imagery have been proposed, including pan-sharpening and super-resolution. However, there are relatively few studies on the techniques of improving the model performance on low-resolution imagery by referring to detailed information from the high-resolution imagery during training time. To tackle this type of task, domain adaptation has been proposed in the field of computer vision to adapt a model trained on one dataset to another with different properties. Yet, domain adaptation for multi-resolution data is difficult due to the scale variation in addition to differences from the camera sensors. In this study, we propose a new approach for multi-resolution modeling that combines the major techniques, semi-supervised domain adaptation (SSDA) and multiple instance learning (MIL). Under the MIL framework, a large scene can be regarded as a bag of instances (e.g., image patches), and information from different receptive field sizes can be exploited. We conducted experiments on a dataset of Japanese oak wilt, which is known to have severe forest damage with two different optical satellite imagery, SPOT6&7 satellite imageries (1.5m) and Pleiades-1B satellite imagery (0.5m). The proposed method improves the discrimination accuracy of the low-resolution model compared to the standard SSDA technique. This obtained result reveals the potential usefulness of MIL for effective multi-resolution modeling.

PurposeThe purpose of this paper is to design a robust tracking algorithm which is suitable for the real-time requirement and solves the mistake labeling issue in the appearance model of trackers with the spare features.Design/methodology/approachThis paper proposes a tracker to select the most discriminative randomly projected ferns and integrates a coarse-to-fine search strategy in this framework. First, the authors exploit multiple instance boosting learning to maximize the bag likelihood and select randomly projected fern from feature pool to degrade the effect of mistake labeling. Second, a coarse-to-fine search approach is first integrated into the framework of multiple instance learning (MIL) for less detections.FindingsThe quantitative and qualitative experiments demonstrate that the tracker has shown favorable performance in efficiency and effective among the competitors of tracking algorithms.Originality/valueThe proposed method selects the feature from the compressive domain by MIL AnyBoost and integrates the coarse-to-fine search strategy first to reduce the burden of detection. This paper designs a tracker with high speed and favorable results which is more suitable for real-time scene.

In this study, we present a new dataset of knotcovered oak planks. It contains 1500 images that have 1 to 11 knots per image, along with mask and bounding-box annotations. The data set was evaluated using deep machine learning methods, and it has been found that instance segmentation models are superior in this task, achieving 59% Box-IoU versus 49% Box-IoU using semantic segmentation. Instance segmentation performed better to detect knots by segmenting instances with an accuracy of 90%, while semantic segmentation detected konts with an accuracy of 89%.

Annotating cancerous regions in whole-slide images (WSIs) of pathology samples plays a critical role in clinical diagnosis, biomedical research, and machine learning algorithms development. However, generating exhaustive and accurate annotations is labor-intensive, challenging, and costly. Drawing only coarse and approximate annotations is a much easier task, less costly, and it alleviates pathologists’ workload. In this paper, we study the problem of refining these approximate annotations in digital pathology to obtain more accurate ones. Some previous works have explored obtaining machine learning models from these inaccurate annotations, but few of them tackle the refinement problem where the mislabeled regions should be explicitly identified and corrected, and all of them require a – often very large – number of training samples. We present a method, named Label Cleaning Multiple Instance Learning (LC-MIL), to refine coarse annotations on a single WSI without the need for external training data. Patches cropped from a WSI with inaccurate labels are processed jointly within a multiple instance learning framework, mitigating their impact on the predictive model and refining the segmentation. Our experiments on a heterogeneous WSI set with breast cancer lymph node metastasis, liver cancer, and colorectal cancer samples show that LC-MIL significantly refines the coarse annotations, out-performing state-of-the-art alternatives, even while learning from a single slide. Moreover, we demonstrate how real annotations drawn by pathologists can be efficiently refined and improved by the proposed approach. All these results demonstrate that LC-MIL is a promising, lightweight tool to provide fine-grained annotations from coarsely annotated pathology sets.

Localizing objects of interest in images when provided with only image-level labels is a challenging visual recognition task. Previous efforts have required carefully designed features and have difficulty in handling images with cluttered backgrounds. Up-scaling to large datasets also poses a challenge to applying these methods to real applications. In this paper, we propose an efficient and effective learning framework called MILinear, which is able to learn an object localization model from large-scale data without using bounding box annotations. We integrate rich general prior knowledge into a learning model using a large pre-trained convolutional network. Moreover, to reduce ambiguity in positive images, we present a bag-splitting algorithm that iteratively generates new negative bags from positive ones. We evaluate the proposed approach on the challenging Pascal VOC 2007 dataset, and our method outperforms other state-of-the-art methods by a large margin; some results are even comparable to fully supervised models trained with bounding box annotations. To further demonstrate scalability, we also present detection results on the ILSVRC 2013 detection dataset, and our method outperforms supervised deformable part-based model without using box annotations.