Weakly Supervised Large Scale Object Localization with Multiple Instance Learning and Bag Splitting.

Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel-wise masks or bounding box annotations, is an important problem to alleviate the data-hungry nature of deep learning. In this article, we tackle this challenging problem by aggregating the image-level information of all training images into a large knowledge graph and exploiting semantic relationships from this graph. Specifically, our effort starts with some generic segment-based object proposals (SOP) without category priors. We propose a multiple instance learning (MIL) framework, which can be trained in an end-to-end manner using training images with image-level labels. For each proposal, this MIL framework can simultaneously compute probability distributions and category-aware semantic features, with which we can formulate a large undirected graph. The category of background is also included in this graph to remove the massive noisy object proposals. An optimal multi-way cut of this graph can thus assign a reliable category label to each proposal. The denoised SOP with assigned category labels can be viewed as pseudo instance segmentation of training images, which are used to train fully supervised models. The proposed approach achieves state-of-the-art performance for both weakly supervised instance segmentation and semantic segmentation. The code is available at https://github.com/yun-liu/LIID.

Many computer-aided diagnosis (CAD) problems can be best modelled as a multiple-instance learning (MIL) problem with unbalanced data, i.e., the training data typically consists of a few positive bags, and a very large number of negative instances. Existing MIL algorithms are much too computationally expensive for these datasets. We describe CH, a framework for learning a Convex Hull representation of multiple instances that is significantly faster than existing MIL algorithms. Our CH framework applies to any standard hyperplane-based learning algorithm, and for some algorithms, is guaranteed to find the global optimal solution. Experimental studies on two different CAD applications further demonstrate that the proposed algorithm significantly improves diagnostic accuracy when compared to both MIL and traditional classifiers. Although not designed for standard MIL problems (which have both positive and negative bags and relatively balanced datasets), comparisons against other MIL methods on benchmark problems also indicate that the proposed method is competitive with the state-of-the-art.

Point annotations are considerably more time-efficient than bounding box annotations. However, how to use cheap point annotations to boost the performance of semi-supervised object detection is still an open question. In this work, we present Point-Teaching, a weakly- and semi-supervised object detection framework to fully utilize the point annotations. Specifically, we propose a Hungarian-based point-matching method to generate pseudo labels for point-annotated images. We further propose multiple instance learning (MIL) approaches at the level of images and points to supervise the object detector with point annotations. Finally, we propose a simple data augmentation, named Point-Guided Copy-Paste, to reduce the impact of those unmatched points. Experiments demonstrate the effectiveness of our method on a few datasets and various data regimes. In particular, Point-Teaching outperforms the previous best method Group R-CNN by 3.1 AP with 5% fully labeled data and 2.3 AP with 30% fully labeled data on the MS COCO dataset. We believe that our proposed framework can largely lower the bar of learning accurate object detectors and pave the way for its broader applications. The code is available at https://github.com/YongtaoGe/Point-Teaching.

In this chapter, we show an autonomous learning scheme to automatically build visual semantic concept models from video sequences or the searched data of Internet search engines without any manual labeling work. First of all, system users specify some specific concept models to be learned automatically. Example videos or images can be obtained from the large video databases based on the result of keyword search on the automatic speech recognition transcripts. Another alternative method is to gather them by using the Internet search engines. Then, we propose to model the searched results as a term of “Quasi-Positive Bags” in the Multiple-Instance Learning (MIL). We call this as the generalized MIL (GMIL). In some of the scenarios, there is also no “Negative Bags” in the GMIL. We propose an algorithm called “Bag K-Means” to find out the maximum Diverse Density (DD) without the existence of negative bags. A cost function is found as K-Means with special “Bag Distance”. We also show a solution called “Uncertain Labeling Density” (ULD) which describes the target density distribution of instances in the case of quasi-positive bags. A “Bag Fuzzy K-Means” is presented to get the maximum of ULD. Utilizing this generalized MIL with ULD framework, the model for a particular concept can then be learned through general supervised learning methods. Experiments show that our algorithm get correct models for the concepts we are interested in.

For the problem of object-based image retrieval, in this paper a novel semi-supervised multiple instance learning algorithm is presented. In the framework of multiple instance learning, this algorithm regards the whole image as a bag, and low-level visual feature of the segmented regions as instances. Firstly, the algorithm clusters the instances in two sets, one of which is composed of instances in positive bags and the other is composed of instances in negative bags, so as to find potential positive instances and feature data of bag structure. Then their respective similarities are measured by radial basis function, and an alpha coefficient is introduced in bag similarity measure as the trade-off between the two similarities. Experiments on SIVAL dataset show that this algorithm is feasible and the performance is superior to other algorithms.

Recently, multiple instance learning (MIL) tends to be a popular method for hyperspectral target representation since it does not require the detailed information about the target. However, the data imbalance problem occurred in MIL usually has negative effects on the representation of hyperspectral target. To solve the problem, this paper propose a bag-level data balance-inducing multiple instance learning method for hyperspectral target representation. The method aims to balance the positive and negative bags by synthesizing new positive bags since the number of positive bags in the training data set is much smaller than that of negative bags. Experimental results show that the proposed method can improve the target representation ability and enhance the target detection performance.

Bone fracture diagnosis from radiographs is a crucial yet time-consuming job requiring professional interpretation. Typical fully supervised learning methods require expensive and often scarce annotation data, especially in medical imaging. We propose a weakly supervised learning (WSL) approach for automatic fracture classification that mimics the holistic strategy of human radiologist. In our method, Vision Transformer (ViT) models (including Base, Large, distillation, and self-supervised variants) are trained using only image-level labels without any bounding box annotations. Despite the absence of localized supervision, our WSL ViT-based models achieve high fracture detection accuracy on the FracAtlas dataset of musculoskeletal X-rays. More importantly, the learned self-attention maps provide human interpretable heatmaps highlighting suspect regions, effectively bridging classification and localization. We demonstrate that our approach detects fractures like a human radiologist, scanning the entire image and focusing on abnormal patterns through learned attention. Experimental results showed that our WSL ViT-based architecture outperforms recent CNN-based methods in classification accuracy while also producing reliable visual explanations. Our best-performing model, a distilled ViT (DeiT) variant, achieved 94% classification accuracy with ROC AUC of 0.93. This work establishes a promising step toward accurate and interpretable fracture detection with minimal supervision. Potentially reducing the need for expensive localization annotations and aiding clinical decision-making.

Accurate diagnosis of hematologic malignancies from peripheral blood smears (PBSs) requires integrating cellular morphology and composition across numerous white blood cells. Existing computational approaches predominantly automate single-cell classifications and do not provide holistic, slide-level diagnostic predictions. We present a framework that employs a high-performance cell-based encoder (DeepHeme) for feature extraction, integrated with our weakly supervised, attention-based multiple instance learning (MIL) model, termed CAREMIL (Cell AggRegation, Explainable, Multiple Instance Learning). Through comprehensive evaluations of leading image encoders and MIL architectures, the combination of DeepHeme and CAREMIL demonstrated superior performance on disease classification tasks. CAREMIL functions as a robust aggregation mechanism, consistently outperforming established slide-level MIL methods (gated MIL and Dual-stream MIL Network) across multiple encoder types. The most pronounced performance gains were observed with out-of-domain encoders, including ImageNet-pretrained and open-source pathology foundation models (UNI2 and Virchow2). CAREMIL combined with DeepHeme achieves the highest diagnostic accuracy across acute myeloid leukemia (AML), myelodysplastic syndromes (MDS), and hairy cell leukemia (HCL), with AUROCs of 0.999, 0.891, and 0.945, respectively, and successfully identifies AML even in cases with minimal or absent circulating blasts. Attention values assigned by CAREMIL highlight diagnostically relevant cells and reveal disease-specific morphometric patterns, enabling biological interpretability and case-level insights. The framework remains resilient to individual cell misclassifications and does not require explicit cell-level supervision. These findings establish CAREMIL as an effective and interpretable MIL framework for hematologic slide diagnosis, extendable to bone marrow aspirates, cytology, and other liquid biopsy specimens, supporting a shift toward quantitative, morphology-informed hematologic diagnostics.

Automatic computer-aided detection plays an important role in biomedical image analysis. Many studies have focused on weak supervised learning as annotation tasks are time-consuming and tedious. Compared with pixel-wise annotation by particular software on the scanned digital high-resolution images, an alternative method of marking out of suspicious regions on microscopy slides is significantly more convenient for pathologists. Additionally, with a focus on dysplasias in the central area, there is a high likelihood of the similar tissues to be found around in clusters. In this paper, for weak annotation on microscopy images, we propose an efficient Foveated Imaging based Multiple Instance Learning (FIMIL) framework to classify weakly-labeled microscopy images. The model also provides multi-scale algorithm for arbitrary image size, in which the patches with highest possibility to contain dysplasia are considered as ”fixation points” in the image. The developed model combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) for dysplasias detection with only image-level labeling. The benchmark tests are carried out on the marked regions of 40x magnified whole-slide cytology images and the normal/abnormal label and their corresponding possibilities are predicted. Evaluated on the real-life clinical data, our proposed model shows high accuracy and efficiency by weakly-supervised learning. 1

The well-built dataset is a pre-requisite for object categorization. However, the processes of collecting and labeling the images are laborious and monotonous. In this paper, we focus on an automatic labeling of images by using a bounding box for each visual object. We propose a two-stage localization approach for image labeling which combines the Efficient Subwindow Search scheme with Multiple Instance Learning. We firstly detect the object coarsely by the the Efficient Subwindow Search scheme, and then we finely localize the object by Multiple Instance learning. Our approach has two advantages, one is to speed up the object search, and the other is to locate the object precisely in a tighter box than the Efficient Subwindow Search scheme. We evaluate the image labeling performance by the detection precision and the detection consistency with the ground truth label. Our approach is simple and fast in object localization. The experiment results demonstrate that our approach is more effective and accurate than the BOW model in the precision and consistency of detection.

The detection of induced pluripotent stem cell (iPSC) colonies often needs the precise extraction of the colony features. However, existing computerized systems relied on segmentation of contours by preprocessing for classifying the colony conditions were task-extensive. To maximize the efficiency in categorizing colony conditions, we propose a multiple instance learning (MIL) in weakly supervised settings. It is designed in a single model to produce weak segmentation and classification of colonies without using finely labeled samples. As a single model, we employ a U-net-like convolution neural network (CNN) to train on binary image-level labels for MIL colonies classification. Furthermore, to specify the object of interest we used a simple post-processing method. The proposed approach is compared over conventional methods using five-fold cross-validation and receiver operating characteristic (ROC) curve. The maximum accuracy of the MIL-net is 95%, which is 15% higher than the conventional methods. Furthermore, the ability to interpret the location of the iPSC colonies based on the image level label without using a pixel-wise ground truth image is more appealing and cost-effective in colony condition recognition.

Human pose estimation has an important impact on a wide range of applications, from human-computer interface to surveillance and content-based video retrieval. For human pose estimation, joint obstructions and overlapping upon human bodies result in departed pose estimation. To address these problems, by integrating priors of the structure of human bodies, we present a novel structure-aware network to discreetly consider such priors during the training of the network. Typically, learning such constraints is a challenging task. Instead, we propose generative adversarial networks as our learning model in which we design two residual Multiple-Instance Learning (MIL) models with identical architecture—one is used as the generator, and the other one is used as the discriminator. The discriminator task is to distinguish the actual poses from the fake ones. If the pose generator generates results that the discriminator is not able to distinguish from the real ones, then the model has successfully learned the priors. In the proposed model, the discriminator differentiates the ground-truth heatmaps from the generated ones, and later the adversarial loss back-propagates to the generator. Such procedure assists the generator to learn reasonable body configurations and is proved to be advantageous to improve the pose estimation accuracy. Meanwhile, we propose a novel function for MIL. It is an adjustable structure for both instance selection and modeling to appropriately pass the information between instances in a single bag. In the proposed residual MIL neural network, the pooling action adequately updates the instance contribution to its bag. The proposed adversarial residual multi-instance neural network that is based on pooling has been validated on two datasets for the human pose estimation task and successfully outperforms the other state-of-the-art models. The code will be made available on https://github.com/pshams55/AMIL.

Weakly-supervised action localization requires training a model to localize the action segments in the video given only video level action label. It can be solved under the Multiple Instance Learning (MIL) framework, where a bag (video) contains multiple instances (action segments). Since only the bag's label is known, the main challenge is assigning which key instances within the bag to trigger the bag's label. Most previous models use attention-based approaches applying attentions to generate the bag's representation from instances, and then train it via the bag's classification. These models, however, implicitly violate the MIL assumption that instances in negative bags should be uniformly negative. In this work, we explicitly model the key instances assignment as a hidden variable and adopt an Expectation-Maximization (EM) framework. We derive two pseudo-label generation schemes to model the E and M process and iteratively optimize the likelihood lower bound. We show that our EM-MIL approach more accurately models both the learning objective and the MIL assumptions. It achieves state-of-the-art performance on two standard benchmarks, THUMOS14 and ActivityNet1.2.

Panoptic segmentation aims to partition an image to object instances and semantic content for thing and stuff categories, respectively. To date, learning weakly supervised panoptic segmentation (WSPS) with only image-level labels remains unexplored. In this paper, we propose an efficient jointly thing-and-stuff mining (JTSM) framework for WSPS. To this end, we design a novel mask of interest pooling (MoIPool) to extract fixed-size pixel-accurate feature maps of arbitrary-shape segmentations. MoIPool enables a panoptic mining branch to leverage multiple instance learning (MIL) to recognize things and stuff segmentation in a unified manner. We further refine segmentation masks with parallel instance and semantic segmentation branches via self-training, which collaborates the mined masks from panoptic mining with bottom-up object evidence as pseudo-ground-truth labels to improve spatial coherence and contour localization. Experimental results demonstrate the effectiveness of JTSM on PASCAL VOC and MS COCO. As a by-product, we achieve competitive results for weakly supervised object detection and instance segmentation. This work is a first step towards tackling challenge panoptic segmentation task with only image-level labels.

MIRSVM: Multi-instance support vector machine with bag representatives