- Research Article
10
- 10.1016/j.eswa.2018.06.043
A Multi-strategy Region Proposal Network
- Jun 28, 2018
- Expert Systems with Applications
- Yu-Peng Chen + 3 more +3
A Multi-strategy Region Proposal Network
Deep Network greatly accelerates the development of object detection. Recent advances in object detection are mainly attributed to the combination of deep network and region proposal methods [1, 2, 3]. However, the accuracy of object detection on the complicated datasets is still not satisfied, especially on small object detection. This is mainly because of the coarseness of the convolution feature maps. In this paper, we design a new strategy for generating region proposals and propose a new localization method for object detection. Compared with previous baseline detectors such as Fast R-CNN [4] and Faster R-CNN [5], Our method makes use of the adjacent-level feature maps at all scales to generate region proposals and also adopts the cascaded region proposal network (RPN) to fine-tune the location of the bounding box. Compared with other state-of-the-art methods, our method achieves the best recall and object detection accuracy.
A Multi-strategy Region Proposal Network
A Multi-strategy Region Proposal Network
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.
State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features-using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.
Read moreHyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection
Almost all of the current top-performing object detection networks employ region proposals to guide the search for object instances. State-of-the-art region proposal methods usually need several thousand proposals to get high recall, thus hurting the detection efficiency. Although the latest Region Proposal Network method gets promising detection accuracy with several hundred proposals, it still struggles in small-size object detection and precise localization (e.g., large IoU thresholds), mainly due to the coarseness of its feature maps. In this paper, we present a deep hierarchical network, namely HyperNet, for handling region proposal generation and object detection jointly. Our HyperNet is primarily based on an elaborately designed Hyper Feature which aggregates hierarchical feature maps first and then compresses them into a uniform space. The Hyper Features well incorporate deep but highly semantic, intermediate but really complementary, and shallow but naturally high-resolution features of the image, thus enabling us to construct HyperNet by sharing them both in generating proposals and detecting objects via an end-to-end joint training strategy. For the deep VGG16 model, our method achieves completely leading recall and state-of-the-art object detection accuracy on PASCAL VOC 2007 and 2012 using only 100 proposals per image. It runs with a speed of 5 fps (including all steps) on a GPU, thus having the potential for real-time processing.
Read moreCross-layer fusion feature network for material defect detection
Object detection involves solving two main problems: identifying the object and its location. This transforms the problem into a classification and localization problem. Currently, research shows that a convolutional neural network (CNN) can be used to solve the classification problem, and the object detection module incorporating a region proposal method can be used to locate the object. Although a CNN based on region proposals can achieve high recall, which improves detection accuracy, its performance cannot meet the actual requirements for small-size object detection and precise localization. This is mainly due to the feature maps extracted from the CNN and the quality of the region proposals. We present a cross-layer fusion feature network (CLFF-Net) for both high-quality region proposal generation and accurate object detection. The CLFF-Net is based on the cross-layer fusion feature that extracts hierarchical feature maps and then aggregates them into a unified space. The fused feature map appropriately combines deep layer semantic information, middle layer supplemental information, and shallow layer location information for an image to build the CLFF-Net, which is shared for both generating region proposals and detecting objects via end-to-end training. Extensive experiments using a casting dataset demonstrate its promising performance compared to state-of-the-art approaches.
Read moreText Detection by Faster R-CNN with Multiple Region Proposal Networks
We propose an end-to-end consistently trainable text detection method based on the Faster R-CNN. The original Faster R-CNN is an end-to-end CNN for fast and accurate object detection. By considering the characteristics of texts, a novel architecture that make use of its ability on object detection is proposed. Although the original Faster R-CNN generates region of interests (RoIs) by a region proposal network (RPN) using the feature map of the last convolutional layer, the proposed method generates RoIs by multiple RPNs using the feature maps of multiple convolutional layers. This method uses multiresolution feature maps to detect texts of various sizes simultaneously. To aggregate the RoIs, we introduce RoI-merge layer, and this layer enables to select valid RoIs from multiple RPNs effectively. In addition, a training strategy is proposed for realizing end-to-end training and making each RPN be specialized in text region size. Experimental results using ICDAR2013/2015 RRC test dataset show that the proposed Multi-RPN method improved detection scores and kept almost the same detection speed as compared to the original Faster R-CNN and recent methods.
Read moreAutomatic Breast Cancer Lesion Detection and Classification in Mammograms Using Faster R-CNN Deep Learning Network
This paper studies various existing Computer Aided Diagnostics System (CAD) for Mammography and proposes a CAD system for breast cancer lesion with an advanced object detection architecture and validates with an experimental dataset. Mammography is an important tool in the early detection of breast cancers which makes the patient to realize the changes much earlier than they are able to feel the changes in their breasts. CAD system uses digital mammography images and searches for abnormality present in breast such as well-defined or circumscribed mass, calcification, architectural distortion and asymmetry. Feature extraction by Machine learning methods require domain knowledge which is challenging and time-consuming process. Whereas deep learning methods adaptively extract the features by learning from the expert annotated input data. Deep learning methods like convolutional neural networks (CNN) have been a tremendous success in various imaging tasks like image detection, recognition and classification. This paper proposes a CAD system for automatic detection and classification of breast cancer lesion in mammograms based on Faster R-CNN, an advanced, successful object detection framework having region proposal generation and classifier layers. The system generated a mAP (mean Average Precision) value of 0.857 for the testing set, which is the combined performance accuracy of both classification and object detection. Aims: Automatic detection and classification of Breast Cancer lesion in mammograms using Faster R-CNN deep learning network. Methods and Materials: This paper proposes a CAD system for detection and classification of breast cancer lesion in mammograms based on Faster R-CNN, an advanced, successful object detection framework having region proposal generation and classifier layers. The proposed CAD system uses mini-MIAS database (mini-Mammographic Image Analysis Society) with 115 images to train the model. The original MIAS database has 330 digitised mammographic images at 50 micron resolution, which has been reduced to 200 micron resolution with Portable Gray Map (PGM) format file of 1024 x 1024 pixels. Conclusions: This paper considered mini-MIAS database as the database which requires minimal data pre-processing steps with a limited hardware. The enhancement to this CAD system is planning to use CBIS-DDSM database, which is a part of the original DDSM (Digital Database for Screening Mammography) database with annotations by a trained mammographer. The CBIS-DDSM database includes the mammographic images which have been decompressed and converted into DICOM format.
Read moreAn Enhanced Region Proposal Network for object detection using deep learning method.
Faster Region-based Convolutional Network (Faster R-CNN) is a state-of-the-art object detection method. However, the object detection effect of Faster R-CNN is not good based on the Region Proposal Network (RPN). Inspired by RPN of Faster R-CNN, we propose a novel proposal generation method called Enhanced Region Proposal Network (ERPN). Four improvements are presented in ERPN. Firstly, our proposed deconvolutional feature pyramid network (DFPN) is introduced to improve the quality of region proposals. Secondly, novel anchor boxes are designed with interspersed scales and adaptive aspect ratios. Thereafter, the capability of object localization is increased. Thirdly, a particle swarm optimization (PSO) based support vector machine (SVM), termed PSO-SVM, is developed to distinguish the positive and negative anchor boxes. Fourthly, the classification part of multi-task loss function in RPN is improved. Consequently, the effect of classification loss is strengthened. In this study, our proposed ERPN is compared with five object detection methods on both PASCAL VOC and COCO data sets. For the VGG-16 model, our ERPN obtains 78.6% mAP on VOC 2007 data set, 74.4% mAP on VOC 2012 data set and 31.7% on COCO data set. The performance of ERPN is the best among the comparison object detection methods. Furthermore, the detection speed of ERPN is 5.8 fps. Additionally, ERPN obtains good effect on small object detection.
Read moreSmall object intelligent Detection method based on Adaptive Cascading Context
With the technology advances, deep learning-based object detection has made unprecedented progress. However, the small spatial ratio of object pixels affects the effective extraction of deep details features, resulting in poor detection results in small object detection. To improve the accuracy of small object detection, an adaptive Cascading Context small (ACC) object detection method is proposed based on YOLOv5. Firstly, a separate shallow layer feature was proposed to obtain more detailed information beneficial to small object detection. Secondly, an adaptive cascade method is proposed to fuse the output features of the three layers of the pyramid to adaptively filter negative semantic information, while fusing with shallow features to solve the problem of low classification accuracy caused by insufficient semantic information of shallow features. Finally, an adaptive context model is proposed to use a deformable convolution to obtain spatial context features of shallow small objects, associating the targets with the background, thereby improving the accuracy of small object detection. The experimental results show that the detection accuracy of the proposed method has been improved by 6.12%, 3.35%, 3.33%, and 5.2%, respectively, compared with the source code on the PASCAL VOC, NWPU VHR-10, KITTI, and RSOD datasets, which fully demonstrate the effectiveness of our method in small object detection.
Read moreTFPN: Twin Feature Pyramid Networks for Object Detection
FPN (Feature Pyramid Networks) is one of the most popular object detection networks, which can improve small object detection by enhancing shallow features. However, limited attention has been paid to the improvement of large object detection via deeper feature enhancement. One existing approach merges the feature maps of different layers into a new feature map for object detection, but can lead to increased noise and loss of information. The other approach adds a bottom-up structure after the feature pyramid of FPN, which superimposes the information from shallow layers into the deep feature map but weakens the strength of FPN in detecting small objects. To address these challenges, this paper proposes TFPN (Twin Feature Pyramid Networks), which consists of (1) FPN+, a bottom-up structure that improves large object detection; (2) TPS, a Twin Pyramid Structure that improves medium object detection; and (3) innovative integration of these two with FPN, which can significantly improve the detection accuracy of large and medium objects while maintaining the advantage of FPN in small object detection. Extensive experiments using the MSCOCO object detection datasets and the BDD100K automatic driving dataset demonstrate that TFPN significantly improves over existing models, achieving up to 2.2 improvement in detection accuracy (e.g., 36.3 for FPN vs. 38.5 for TFPN on COCO Val-17). Our method can obtain the same accuracy as FPN with ResNet-101 based on ResNet-50 and needs fewer parameters.
Read moreSmall Moving Object Detection Based on Sequence Confidence Method in UAV Video
In the small object detection on the UAV (Unmanned Aerial Vehicle) platform, the confidence description of the moving object is proposed to improve the accuracy, robustness and reliable tracking method of the object detection. Due to the low resolution and slow motion of small moving object in aerial video, and the image is easily subject to illumination and camera jitter noise, and the correlation between video sequences is neglected, it is prone to false detection of moving object and low detection accuracy, the characteristics of poor robustness. For the UAV video with small moving object, the algorithm uses the ORB operator to extract reliable global feature points for each frame of the video, and then performs global motion compensation on the motion background through the affine transformation model and calculates the difference image. The energy accurately detects the small object, and then describes the confidence of the moving object. The n-step back-off method is used to increase the correlation information between the video sequences. The proposed method is to evaluate the video captured on the airborne aircraft, and has done a lot of experiments and tests. For the object as small as 25 pixels, the method still has better performance, and our method can be realized by parallel computing. Real-time, processing 1280 × 720 frames at around 45 fps.
Read moreCentered Multi-Task Generative Adversarial Network for Small Object Detection
Despite the breakthroughs in accuracy and efficiency of object detection using deep neural networks, the performance of small object detection is far from satisfactory. Gaze estimation has developed significantly due to the development of visual sensors. Combining object detection with gaze estimation can significantly improve the performance of small object detection. This paper presents a centered multi-task generative adversarial network (CMTGAN), which combines small object detection and gaze estimation. To achieve this, we propose a generative adversarial network (GAN) capable of image super-resolution and two-stage small object detection. We exploit a generator in CMTGAN for image super-resolution and a discriminator for object detection. We introduce an artificial texture loss into the generator to retain the original feature of small objects. We also use a centered mask in the generator to make the network focus on the central part of images where small objects are more likely to appear in our method. We propose a discriminator with detection loss for two-stage small object detection, which can be adapted to other GANs for object detection. Compared with existing interpolation methods, the super-resolution images generated by CMTGAN are more explicit and contain more information. Experiments show that our method exhibits a better detection performance than mainstream methods.
Read moreDeep learning for small object detection in images
[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT REQUEST OF AUTHOR.] With the rapid development of deep learning in computer vision, especially deep convolutional neural networks (CNNs), significant advances have been made in recent years on object recognition and detection in images. Highly accurate detection results have been achieved for large objects, whereas detection accuracy on small objects remains to be low. This dissertation focuses on investigating deep learning methods for small object detection in images and proposing new methods with improved performance. First, we conducted a comprehensive review of existing deep learning methods for small object detections, in which we summarized and categorized major techniques and models, identified major challenges, and listed some future research directions. Existing techniques were categorized into using contextual information, combining multiple feature maps, creating sufficient positive examples, and balancing foreground and background examples. Methods developed in four related areas, generic object detection, face detection, object detection in aerial imagery, and segmentation, were summarized and compared. In addition, the performances of several leading deep learning methods for small object detection, including YOLOv3, Faster R-CNN, and SSD, were evaluated based on three large benchmark image datasets of small objects. Experimental results showed that Faster R-CNN performed the best, while YOLOv3 was a close second. Furthermore, a new deep learning method, called Retina-context Net, was proposed and outperformed state-of-the art one-stage deep learning models, including SSD, YOLOv3 and RetinaNet, on the COCO and SUN benchmark datasets. Secondly, we created a new dataset for bird detection, called Little Birds in Aerial Imagery (LBAI), from real-life aerial imagery. LBAI contains birds with sizes ranging from 10 by 10 pixels to 40 by 40 pixels. We adapted and applied several state-of-the-art deep learning models to LBAI, including object detection models such as YOLOv2, SSH, and Tiny Face, and instance segmentation models such as U-Net and Mask R-CNN. Our empirical results illustrated the strength and weakness of these methods, showing that SSH performed the best for easy cases, whereas Tiny Face performed the best for hard cases with cluttered backgrounds. Among small instance segmentation methods, U-Net achieved slightly better performance than Mask R-CNN. Thirdly, we proposed a new graph neural network-based object detection algorithm, called GODM, to take the spatial information of candidate objects into consideration in small object detection. Instead of detecting small objects independently as the existing deep learning methods do, GODM treats the candidate bounding boxes generated by existing object detectors as nodes and creates edges based on the spatial or semantic relationship between the candidate bounding boxes. GODM contains four major components: node feature generation, graph generation, node class labelling, and graph convolutional neural network model. Several graph generation methods were proposed. Experimental results on the LBDA dataset show that GODM outperformed existing state-of-the-art object detector Faster R-CNN significantly, up to 12% better in accuracy. Finally, we proposed a new computer vision-based grass analysis using machine learning. To deal with the variation of lighting condition, a two-stage segmentation strategy is proposed for grass coverage computation based on a blackboard background. On a real world dataset we collected from natural environments, the proposed method was robust to varying environments, lighting, and colors. For grass detection and coverage computation, the error rate was just 3%.
Read moreA Multi-Scale Spatial Attention Region Proposal Network for High-Resolution Optical Remote Sensing Imagery
Deep convolutional neural networks (DCNNs) are driving progress in object detection of high-resolution remote sensing images. Region proposal generation, as one of the key steps in object detection, has also become the focus of research. High-resolution remote sensing images usually contain various sizes of objects and complex background, small objects are easy to miss or be mis-identified in object detection. If the recall rate of region proposal of small objects and multi-scale objects can be improved, it will bring an improvement on the performance of the accuracy in object detection. Spatial attention is the ability to focus on local features in images and can improve the learning efficiency of DCNNs. This study proposes a multi-scale spatial attention region proposal network (MSA-RPN) for high-resolution optical remote sensing imagery. The MSA-RPN is an end-to-end deep learning network with a backbone network of ResNet. It deploys three novel modules to fulfill its task. First, the Scale-specific Feature Gate (SFG) focuses on features of objects by processing multi-scale features extracted from the backbone network. Second, the spatial attention-guided model (SAGM) obtains spatial information of objects from the multi-scale attention maps. Third, the Selective Strong Attention Maps Model (SSAMM) adaptively selects sliding windows according to the loss values from the system’s feedback, and sends the windowed samples to the spatial attention decoder. Finally, the candidate regions and their corresponding confidences can be obtained. We evaluate the proposed network in a public dataset LEVIR and compare with several state-of-the-art methods. The proposed MSA-RPN yields a higher recall rate of region proposal generation, especially for small targets in remote sensing images.
Read moreSmall-objectness sensitive detection based on shifted single shot detector
We present a small object sensitive method for object detection. Our method is built based on SSD (Single Shot MultiBox Detector (Liu et al. 2016)), a simple but effective deep neural network for image object detection. The discrete nature of anchor mechanism used in SSD, however, may cause misdetection for the small objects located at gaps between the anchor boxes. SSD performs better for small object detection after circular shifts of the input image. Therefore, auxiliary feature maps are generated by conducting circular shifts over lower extra feature maps in SSD for small-object detection, which is equivalent to shifting the objects in order to fit the locations of anchor boxes. We call our proposed system Shifted SSD. Moreover, pinpoint accuracy of localization is of vital importance to small objects detection. Hence, two novel methods called Smooth NMS and IoU-Prediction module are proposed to obtain more precise locations. Then for video sequences, we generate trajectory hypothesis to obtain predicted locations in a new frame for further improved performance. Experiments conducted on PASCAL VOC 2007, along with MS COCO, KITTI and our small object video datasets, validate that both mAP and recall are improved with different degrees and the speed is almost the same as SSD.
Read moreAdvances and Challenges in Small Object Detection: A Comparative Analysis of State-of-the-Art Models and Future Directions
Object detection involves the precise and efficient identification and localization of multiple predefined object categories within images. With the advent of deep learning, both the accuracy and efficiency of object detection have significantly improved. Nevertheless, challenges remain in optimizing the performance of mainstream detection algorithms, improving the accuracy of small object detection, enabling multi-class detection, and developing lightweight models. In response to these challenges, this paper provides a comprehensive literature review, analyzing approaches to enhance mainstream object detection by exploring advancements in backbone networks, expanding the visual receptive field, feature fusion, and various training strategies. We also evaluate the performance of leading detection models across established datasets, identifying current limitations and proposing future research directions. These directions include enhancing small object representation in datasets, enriching semantic information, and improving model interpretability. Small object detection remains a critical focus in computer vision, and we anticipate the continued development of algorithms with higher accuracy and efficiency
Read more