Weakly supervised instance segmentation using multi-prior fusion

In this paper, we introduce a new large-scale sidewalk dataset called SideGuide that could potentially help impaired people. Unlike most previous datasets, which are focused on road environments, we paid attention to sidewalks, where understanding the environment could provide the potential for improved walking of humans, especially impaired people. Concretely, we interviewed impaired people and carefully selected target objects from the interviewees' feedback (objects they encounter on sidewalks). We then acquired two different types of data: crowd-sourced data and stereo data. We labeled target objects at instance-level (i.e., bounding box and polygon mask) and generated a ground-truth disparity map for the stereo data. SideGuide consists of 350K images with bounding box annotation, 100K images with a polygon mask, and 180K stereo pairs with the ground-truth disparity. We analyzed our dataset by performing baseline analysis for object detection, instance segmentation, and stereo matching tasks. In addition, we developed a prototype that recognizes the target objects and measures distances, which could potentially assist people with disabilities. The prototype suggests the possibility of practical application of our dataset in real life.

Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel-wise masks or bounding box annotations, is an important problem to alleviate the data-hungry nature of deep learning. In this article, we tackle this challenging problem by aggregating the image-level information of all training images into a large knowledge graph and exploiting semantic relationships from this graph. Specifically, our effort starts with some generic segment-based object proposals (SOP) without category priors. We propose a multiple instance learning (MIL) framework, which can be trained in an end-to-end manner using training images with image-level labels. For each proposal, this MIL framework can simultaneously compute probability distributions and category-aware semantic features, with which we can formulate a large undirected graph. The category of background is also included in this graph to remove the massive noisy object proposals. An optimal multi-way cut of this graph can thus assign a reliable category label to each proposal. The denoised SOP with assigned category labels can be viewed as pseudo instance segmentation of training images, which are used to train fully supervised models. The proposed approach achieves state-of-the-art performance for both weakly supervised instance segmentation and semantic segmentation. The code is available at https://github.com/yun-liu/LIID.

Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> egmenting <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> eyond the <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ounding <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b> ox ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S3B-Net</b> ). Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.

For few-shot object detection has attracted wide attention. Plateau wildlife detection is a typical problem of few-shot object detection. The solution of this problem can reduce the manpower and material resources of collecting and marking large-scale wildlife data sets. Deep neural network can significantly improve the efficiency of wildlife detection, and the key step is to obtain a large amount of data with bounding box annotations. However, there are few plateau wildlife pictures with labeling, which is insufficient to support the training of deep neural network. In this paper, we propose a target detection method based on few-shot learning to detect plateau wildlife images, where each animal category has only a small number of annotated bounding boxes (no more than 10). we use two-stage training method to solve the task of small samples detection task. Based on the two-stage target detector Faster R-CNN, we established a few shot target detection model. The training is divided into two stages. The first stage is to train the model on the basic set, and the second stage is to adjust the balance set of the new class and the base class on the last layer. In order to further improve the detection effect, the two-stage training adopts data enhancement methods to expand the sample data. Experimental results show that this model can achieve better detection effect on plateau wildlife pictures with a small number of labeled samples.

Brain tumor detection from Magnetic Resonance Imaging (MRI) scans is a critical clinical diagnosis application in which timely and accurate detection directly influences treatment planning and patient survival. Traditional machine learning algorithms and deep learning algorithms, although effective in classification, fall short in precise tumor location identification and real-time appropriateness. This study proposes an improved detection framework on YOLOv8 with Adaptive Hybrid Transfer Learning (AHTL) for multiclass identification of brain tumors, including glioma, meningioma, pituitary tumor, and normal conditions. The Kaggle and Hayatabad Medical Complex dataset was preprocessed with normalization, bounding box annotation, and augmentation for improving the dataset's robustness. YOLOv8 model architecture, promoted by AHTL, is based on leveraging pretrained information from COCO with adaptive finetuning on MRI features to yield improved generalization across variation in tumor shape, size, and location. 50 epochs were used for training with optimization using momentum and early stopping to avoid overfitting. Experimental performance demonstrates the effectiveness of the proposed system with precision of 0.93, recall of 0.91, F1-score of 0.92, and mAP@0.5 of 0.94, outperforming standard CNN-based classifiers, segmentation models, and YOLO frameworks. Visual output with bounding boxes also enhanced interpretability to enable radiologists to validate AI-based predictions. Briefly, the novel YOLOv8 + AHTL approach provides a computationally effective, accurate, and clinically interpretable brain tumor detection method that has filled the gap between research experimentation and actual clinical practice. This performance serves as good support for decision making in health care.

The Segment Anything Model (SAM) is a recently developed all-range foundation model for image segmentation. It can use sparse manual prompts such as bounding boxes to generate pixel-level segmentation in natural images but struggles in medical images such as low-contrast, noisy ultrasound images. We propose a refined test-phase prompt augmentation technique designed to improve SAM's performance in medical image segmentation. The method couples multi-box prompt augmentation and an aleatoric uncertainty-based false-negative (FN) and false-positive (FP) correction (FNPC) strategy. We evaluate the method on two ultrasound datasets and show improvement in SAM's performance and robustness to inaccurate prompts, without the necessity for further training or tuning. Moreover, we present the Single-Slice-to-Volume (SS2V) method, enabling 3D pixel-level segmentation using only the bounding box annotation from a single 2D slice. Our results allow efficient use of SAM in even noisy, low-contrast medical images. The source code has been released at: https://github.com/MedICL-VU/FNPC-SAM.

Automatic License Plate Recognition (ALPR) has been the focus of many\nresearches in the past years. In general, ALPR is divided into the following\nproblems: detection of on-track vehicles, license plates detection, segmention\nof license plate characters and optical character recognition (OCR). Even\nthough commercial solutions are available for controlled acquisition\nconditions, e.g., the entrance of a parking lot, ALPR is still an open problem\nwhen dealing with data acquired from uncontrolled environments, such as roads\nand highways when relying only on imaging sensors. Due to the multiple\norientations and scales of the license plates captured by the camera, a very\nchallenging task of the ALPR is the License Plate Character Segmentation (LPCS)\nstep, which effectiveness is required to be (near) optimal to achieve a high\nrecognition rate by the OCR. To tackle the LPCS problem, this work proposes a\nnovel benchmark composed of a dataset designed to focus specifically on the\ncharacter segmentation step of the ALPR within an evaluation protocol.\nFurthermore, we propose the Jaccard-Centroid coefficient, a new evaluation\nmeasure more suitable than the Jaccard coefficient regarding the location of\nthe bounding box within the ground-truth annotation. The dataset is composed of\n2,000 Brazilian license plates consisting of 14,000 alphanumeric symbols and\ntheir corresponding bounding box annotations. We also present a new\nstraightforward approach to perform LPCS efficiently. Finally, we provide an\nexperimental evaluation for the dataset based on four LPCS approaches and\ndemonstrate the importance of character segmentation for achieving an accurate\nOCR.\n

Video description is one of the most challenging problems in vision and language understanding due to the large variability both on the video and language side. Models, hence, typically shortcut the difficulty in recognition and generate plausible sentences that are based on priors but are not necessarily grounded in the video. In this work, we explicitly link the sentence to the evidence in the video by annotating each noun phrase in a sentence with the corresponding bounding box in one of the frames of a video. Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase. This allows training video description models with this data, and importantly, evaluate how grounded or "true" such model are to the video they describe. To generate grounded captions, we propose a novel video description model which is able to exploit these bounding box annotations. We demonstrate the effectiveness of our model on our dataset, but also show how it can be applied to image description on the Flickr30k Entities dataset. We achieve state-of-the-art performance on video description, video paragraph description, and image description and demonstrate our generated sentences are better grounded in the video.

Fisheye object detection is challenging due to the fisheye distortion, which inclines objects to different extents and pushes extensive irrelevant pixels into the predicted horizontal bounding box (HBB). To address the problems above, we establish a new fisheye object detection dataset (named FishOBB) with compact oriented bounding box (OBB) annotations, as well as an OBB-customized mosaic augmentation technology. To our knowledge, there are very few fisheye datasets labeled by OBB, especially the open source forword view dataset like ours. Besides, we provide a fisheye object detection baseline (named FDA-YOLO) with two fisheye adaption units. Concretely, we first design a distortion orientation aggregation (DOA) unit guided by polar sampling to capture distortion-aware fisheye features. On the other hand, to transfer HBB-based detection models to OBB-based counterparts, we propose an oriented anchor attention unit. It automatically weights the unbalanced positive/negative samples and facilitates convergence for multi-anchor models. Finally, we demonstrate that the two adaption units can be easily integrated into various anchor-based YOLO methods, e.g., ScaledYOLOv4 and YOLOv7, contributing to superior performance to existing state-of-the-art (SoTA) solutions in the proposed dataset. Meanwhile, our method has also achieved SoTA performance on other popular datasets like WEPDTOF. The dataset and code are released at https://github.com/lukanightfever/FishOBB .

Object detection in aerial images is an active yet challenging task in computer vision because of the bird’s-eye view perspective, the highly complex backgrounds, and the variant appearances of objects. Especially when detecting densely packed objects in aerial images, methods relying on horizontal proposals for common object detection often introduce mismatches between the Region of Interests (RoIs) and objects. This leads to the common misalignment between the final object classification confidence and localization accuracy. In this paper, we propose a RoI Transformer to address these problems. The core idea of RoI Transformer is to apply spatial transformations on RoIs and learn the transformation parameters under the supervision of oriented bounding box (OBB) annotations. RoI Transformer is with lightweight and can be easily embedded into detectors for oriented object detection. Simply apply the RoI Transformer to light head RCNN has achieved state-of-the-art performances on two common and challenging aerial datasets, i.e., DOTA and HRSC2016, with a neglectable reduction to detection speed. Our RoI Transformer exceeds the deformable Position Sensitive RoI pooling when oriented bounding-box annotations are available. Extensive experiments have also validated the flexibility and effectiveness of our RoI Transformer.

Localizing objects of interest in images when provided with only image-level labels is a challenging visual recognition task. Previous efforts have required carefully designed features and have difficulty in handling images with cluttered backgrounds. Up-scaling to large datasets also poses a challenge to applying these methods to real applications. In this paper, we propose an efficient and effective learning framework called MILinear, which is able to learn an object localization model from large-scale data without using bounding box annotations. We integrate rich general prior knowledge into a learning model using a large pre-trained convolutional network. Moreover, to reduce ambiguity in positive images, we present a bag-splitting algorithm that iteratively generates new negative bags from positive ones. We evaluate the proposed approach on the challenging Pascal VOC 2007 dataset, and our method outperforms other state-of-the-art methods by a large margin; some results are even comparable to fully supervised models trained with bounding box annotations. To further demonstrate scalability, we also present detection results on the ILSVRC 2013 detection dataset, and our method outperforms supervised deformable part-based model without using box annotations.

Instance Segmentation and Localization of Strawberries in Farm Conditions for Automatic Fruit Harvesting

In this study, we present a new dataset of knotcovered oak planks. It contains 1500 images that have 1 to 11 knots per image, along with mask and bounding-box annotations. The data set was evaluated using deep machine learning methods, and it has been found that instance segmentation models are superior in this task, achieving 59% Box-IoU versus 49% Box-IoU using semantic segmentation. Instance segmentation performed better to detect knots by segmenting instances with an accuracy of 90%, while semantic segmentation detected konts with an accuracy of 89%.

For efficient human-agent interaction, an agent should proactively recognize their target user and prepare for upcoming interactions. We formulate this challenging problem as a novel task of jointly forecasting a person’s intent to interact with the agent, their attitude towards the agent and the action they will perform, from the agent’s (egocentric) perspective. We propose SocialEgoNet - a graph-based spatiotemporal framework that exploits task dependencies through a hierarchical multitask learning approach. SocialEgoNet uses whole-body skeletons (keypoints from face, hands and body) extracted from only 1 second of video input for high inference speed. For evaluation, we augment an existing egocentric human-agent interaction dataset with new class labels and bounding box annotations. Extensive experiments on this augmented dataset, named JPL-Social, demonstrate real-time inference and superior performance (average accuracy across all tasks: 83.15%) of our model outperforming several competitive baselines. The additional annotations and code are available at github.com/biantongfei/SocialEgoNet.

Bounding boxes have been widely implemented into aerial object detection for its simplicity. They perform instance-level location with the coordinates and orientation for each target. But the defects such as coarse edge information impede semantic interpretation in earth observation. Besides, in terms of the aerial imaging instruments, it's essential to recognize the exterior appearance and contour of the objects. In this paper, we propose a novel aerial instance segmentation method termed adaptive RoI extraction network (ARE-Net) which bridges the gap of accurately delineating instances under the complex back-ground of aerial images. To exert instance segmentation under the proprietary property, e.g., complex background, densely distributed instances, of aerial images, RoIs are pooled from multi-level feature maps and integral region proposals. On this basis, global attention RoI extractor (GA-RoIE) and perceptual RoI extractor (PRoIE) are respectively introduced for detection branch and mask branch to perform adaptive RoI extraction for aerial images. Meanwhile, to reconcile the probability distribution regional distribution of pixel-wise prediction in aerial images, we present the Adaptive Compound loss function to improve the integrating degree of the predicted binary mask to ground truth mask. Additionally, we adopt RegNetx with Deformable Convolution to optimize ARE-Net, and name it as R-ARE-Net. Despite implementing pixel-wise prediction, comprehensive experiments on iSAID and NWPU VHR-10 instance segmentation dataset still have verified the effectiveness and efficiency of ARE-Net and R-ARE-Net. Experimental results indicate that our proposed methods receive the highest AP value (38.0% AP on iSAID and 64.2% AP on NWPU VHR-10 instance segmentation dataset) and lowest FLOPs and Parameters consumption (~46% reduced FLOPs and 61.5% reduced Parameters than SCNet) among the mainstream methods. Besides, the false alarms, missing segmentations, poorly predicted masks, and under-segmentations that appeared in the mainstream methods can be avoided to some extend for R-ARE-Net.