Construct Effective Geometry Aware Feature Pyramid Network for Multi-Scale Object Detection

Semantic segmentation of remote sensing images plays an important role in a\nwide range of applications including land resource management, biosphere\nmonitoring and urban planning. Although the accuracy of semantic segmentation\nin remote sensing images has been increased significantly by deep convolutional\nneural networks, several limitations exist in standard models. First, for\nencoder-decoder architectures such as U-Net, the utilization of multi-scale\nfeatures causes the underuse of information, where low-level features and\nhigh-level features are concatenated directly without any refinement. Second,\nlong-range dependencies of feature maps are insufficiently explored, resulting\nin sub-optimal feature representations associated with each semantic class.\nThird, even though the dot-product attention mechanism has been introduced and\nutilized in semantic segmentation to model long-range dependencies, the large\ntime and space demands of attention impede the actual usage of attention in\napplication scenarios with large-scale input. This paper proposed a\nMulti-Attention-Network (MANet) to address these issues by extracting\ncontextual dependencies through multiple efficient attention modules. A novel\nattention mechanism of kernel attention with linear complexity is proposed to\nalleviate the large computational demand in attention. Based on kernel\nattention and channel attention, we integrate local feature maps extracted by\nResNeXt-101 with their corresponding global dependencies and reweight\ninterdependent channel maps adaptively. Numerical experiments on three\nlarge-scale fine resolution remote sensing images captured by different\nsatellite sensors demonstrate the superior performance of the proposed MANet,\noutperforming the DeepLab V3+, PSPNet, FastFCN, DANet, OCRNet, and other\nbenchmark approaches.\n

Feature pyramid network is widely used in advanced object detection. By simply changing the network connection, the performance of small object detection can be greatly improved without increasing the amount of calculation of the original model. However, the algorithm still has some shortcomings. Therefore, a new attention feature pyramid network (attention PFN) is proposed. Firstly, an improved receptive field module is added, which can make full use of global and local context information. Secondly, the connection mode of the pyramid is further optimized. Deconvolution is used to replace the nearest neighbor interpolation in top-down up-sampling and a channel attention module is added to the horizontal connection to highlight important context information. Finally, adaptive fusion and spatial feature balance are used for each feature pyramid, so that the network can learn the weights of different feature layers. Each pyramid layer contains more discrimination information. Attention PFN is tested on Pascal VOC and MS COCO datasets, respectively. The experiment results revealed that the proposed method has better performance than the original algorithm. Therefore, the attention PFN is an effective algorithm.

The effective use of multi-scale features remains an open problem for object detection tasks. Recently, proposed object detectors have usually used Feature Pyramid Networks (FPN) to fuse multi-scale features. Since Feature Pyramid Networks use a relatively simple feature map fusion approach, it can lead to the loss or misalignment of semantic information in the fusion process. Several works have demonstrated that using a bottom-up structure in a Feature Pyramid Network can shorten the information path between lower layers and the topmost feature, allowing an adequate exchange of semantic information from different layers. We further enhance the bottom-up path by proposing a multi-scale residual aggregation Feature Pyramid Network (MSRA-FPN), which uses a unidirectional cross-layer residual module to aggregate features from multiple layers bottom-up in a triangular structure to the topmost layer. In addition, we introduce a Residual Squeeze and Excitation Module to mitigate the aliasing effects that occur when features from different layers are aggregated. MSRA-FPN enhances the semantic information of the high-level feature maps, mitigates the information decay during feature fusion, and enhances the detection capability of the model for large objects. It is experimentally demonstrated that our proposed MSRA-FPN improves the performance of the three baseline models by 0.5–1.9% on the PASCAL VOC dataset and is also quite competitive with other state-of-the-art FPN methods. On the MS COCO dataset, our proposed method can also improve the performance of the baseline model by 0.8% and the baseline model’s performance for large object detection by 1.8%. To further validate the effectiveness of MSRA-FPN for large object detection, we constructed the Thangka Figure Dataset and conducted comparative experiments. It is experimentally demonstrated that our proposed method improves the performance of the baseline model by 2.9–4.7% on this dataset and can reach up to 71.2%.

Improving the precision of weakly supervised multiscale objects localization is of significant challenge in computer vision, especially when tackling the small objects. in this paper, we propose to integrate the feature pyramid network (FPN) with convolutional neural network (CNN) for weakly supervised object localization, where the FPN is built upon the outputs of different layers of the CNN. Then, we upsample the high-level maps by nearest-neighbor interpolation and fuse with the low-level maps in the FPN to produce multi-scale fused maps which features of both high resolution and strong semantics. Finally, we produce class activation maps by each layer of the FPN and gain multiple prediction scores by wildcat spatial pooling. To acquire more precise localization, we select the class activation map that corresponds to the highest score across all multiscale maps for object localization. In particular, we choose the maximum response regions of the class activation map for pointwise localization and choose the largest connected component above the threshold in the class activation map for bounding box localization. By applying the proposed strategy over PASCAL VOC dataset and MS COCO dataset, it is demonstrated that our strategy is highly effective in improving the precision of weakly supervised object localization as compared with some of the state-of-the-art weakly supervised methods.

Object detection is increasingly in demand in IoT service applications. Deep learning based object detection algorithms are now in fashion. As the most popular multi-scale object detection network at present, Feature Pyramid Network achieves feature augmentation by fusing features of neighboring layers. It is widely used in the most advanced object detectors to detect objects of different scales. In this paper, we propose a new attention mechanism guided bidirectional feature pyramid architecture named AgBFPN to enhance the transfer of semantic and spatial information between each feature map. We design Channel Attention Guided Fusion(CAGF) Module and Spatial Attention Guided Fusion(SAGF) Module to enhance feature fusion. The CAGF mitigates the loss of information induced by channel reduction and better transfers the semantic information from high-level to low-level features. The SAGF passes the rich spatial information of shallow features into deep features. Our experiments show that AgBFPN achieves higher Average Precision for multi-scale object detection.

The synthetic aperture radar (SAR) has a special ability to detect objects in any climate and weather conditions. Consequently, SAR images are widely used in maritime transportation safety and fishery law enforcement for maritime object detection. Currently, deep-learning models are being extensively used for the detection of objects from images. Among them, the feature pyramid network (FPN) uses pyramids for representing semantic information regardless of the scale and has an improved accuracy of object detection. It is also suitable for the detection of multiple small ship objects in SAR images. This study aims to resolve the problems associated with small-object and multi-object ship detection in complex scenarios e.g., when a ship nears the port, by proposing a detection method based on an optimized FPN model. The feature pyramid model is first embedded in a traditional region proposal network (RPN) and mapped into a new feature space for object identification. Subsequently, the k-means clustering algorithm based on the shape similar distance (SSD) measure is used to optimize the FPN. Initial anchor boxes and tests are created using the SAR ship dataset. Experimental results show that the proposed algorithm for object detection shows an accuracy of 98.62%. Compared with Yolo, the RPN based on VGG/ResNet, FPN based on VGG/ResNet, and other models in complex scenarios, the proposed model shows a higher accuracy rate and better overall performance.

Multi-scale feature representations are widely exploited by both the one-stage object detectors and the two-stage object detectors in recent years. Object instances in optical remote sensing images change greatly in scale, orientation, and shape. Therefore, the detection of multi-scale objects, especially densely small objects, is a challenging problem in optical remote sensing images. To alleviate the problem arising from scale variation across object instances, we propose an Atrous Feature Pyramid Network (AFPN) to construct more effective feature pyramids network. The proposed network is an enhanced Faster R-CNN with a novel feature fusing module called Atrous Feature Fusing Module (AFFM). The multi-level features generated by the feature fusion module will be fed to detectors respectively to predict the detection results. The proposed network can not only detect objects more accurately in optical remote sensing images, but also improve the detection accuracy of natural images. Our network can achieve 96.89% mAP on NWPU VHR-10 dataset, 96.75% on RSOD dataset, which ensures the generalization ability of the method on larger and harder datasets.KeywordsOptical remote sensing imagesObject detectionMulti-scale feature representations

In general object detection, scale variation is always a big challenge. At present, feature pyramid networks are employed in numerous methods to alleviate the problems caused by large scale range of objects in object detection, which makes use of multi-level features extracted from the backbone for top-down upsampling and fusion to acquire a set of multi-scale depth image features. However, the feature pyramid network proposed by Ghiasi et al. adopts a simple fusion method, which fails to consider the fusion feature context, and therefore, it is difficult to acquire good features. In addition, the fusion of multi-scale features directly by traditional upsampling is prone to feature misalignment and loss of details. In this paper, an adaptive feature pyramid network is proposed based on the feature pyramid network to alleviate the foregoing potential problems, which includes two major designs, i.e., adaptive feature upsampling and adaptive feature fusion. The adaptive feature upsampling aims to predict a group of sampling points of each pixel through some models, and constitute feature representation of the pixel by feature combination of sampling points, while adaptive feature fusion is to construct pixel-level fusion weights between fusion features through attention mechanism. The experimental results verified the effectiveness of the method proposed in this paper. On the public object detection dataset MS-COCO test-dev, Faster R-CNN model achieved performance improvement of 1.2 AP by virtue of the adaptive feature pyramid network, and FCOS model could achieve performance improvement of 1.0 AP. What's more, the experiments also validated that the adaptive feature pyramid network proposed herein was more accurate for object localization.

Since small objects occupy less pixels in the image and are difficult to recognize. Small object detection has always been a research difficulty in the field of computer vision. Aiming at the problems of low sensitivity and poor detection performance of YOLOv3 for small objects. AFYOLO, which is more sensitive to small objects detection was proposed in this paper. Firstly, the DenseNet module is introduced into the low-level layers of backbone to enhance the transmission ability of objects information. At the same time, a new mechanism combining channel attention and spatial attention is introduced to improve the feature extraction ability of the backbone. Secondly, a new feature pyramid network (FPN) is proposed to better obtain the features of small objects. Finally, ablation studies on ImageNet classification task and MS-COCO object detection task verify the effectiveness of the proposed attention module and FPN. The results on Wider Face datasets show that the AP of the proposed method is 11.89%higher than that of YOLOv3 and 8.59%higher than that of YOLOv4. All of results show that AFYOLO has better ability for small object detection.

HIGHLIGHTSWe propose a balanced feature pyramid network model to achieve automatic recognition of agricultural pests.We propose and improved feature pyramid module to address the problems of the large semantic gap before feature fusion in FPN (feature pyramid network), information loss during feature fusion, and how to utilize complementary information to strengthen features after feature fusion.Our method is evaluated on the pest dataset and achieves better recognition results, the mAP (mean Average Precision) reaches 90.04%, and the final mAP after combining with the data enhancement strategy reaches 92.56%.Abstract.In order to ensure the steady improvement of the quality and efficiency of agricultural production, the precise prevention and control of agriculture pests should be given top priority. Based on the Faster R-CNN model, we propose a feature-augmented recognition model (FARM) for the detection of agricultural pests to improve the professional level of pest control. Firstly, we select VoVNet as the backbone network to extract rich visual information. Secondly, in view of the existing problems of FPN (Feature Pyramid Network), we propose a feature pyramid enhancement network to improve it. Finally, we use an adaptive anchor box network to train anchor boxes to avoid the problem of unbalanced positive and negative samples caused by artificial parameter adjustment. The experimental result shows that our method can achieve better recognition results; the mAP on the agricultural pest dataset reaches 90.04%, and the final mAP after combining with the data enhancement strategy reaches 92.56%. Keywords: Adaptive anchor box, Agricultural pests, Feature augmented, Feature pyramid.

Attention mechanisms have been widely used to address the common multi-scale challenges in remote sensing imagery for tasks such as object detection and instance segmentation. However, they often require the design of task-specific network architectures tailored to different data targets. To better accommodate objects of varying scales and downstream task requirements, we propose a dynamic search framework based on a Mixture-of-Experts model. In this framework, each module is treated as an expert, and their combinations are flexibly adjusted to ensure that the pyramid structure adapts to the needs of different-scale tasks. Specifically, we design a Scalable Bi-Directional Feature Pyramid Network (SBFPN), which incorporates various hybrid attention mechanisms to dynamically enhance feature fusion across different layers. This approach not only captures long-range dependencies within the image but also suppresses noise and interference from complex backgrounds. The scalable feature pyramid structure we construct utilizes sparse feature fusion within the same layers, while also computing self-attention weights between nodes and retaining or trimming skip connections across layers. We conduct experiments on object detection, instance segmentation, and panoptic segmentation tasks. This structure can be transferred to YOLO-based and R-CNN-based networks, achieving superior detection and segmentation performance. On the Airbus Ship dataset, the mAP for detection and segmentation increased from 71.3% and 62.4% to 82.7% and 71.1%, respectively, demonstrating the effectiveness of our proposed method. Code is available at https://github.com/chaibosong/SAFPN.

In the neck part of a two-stage object detection network, feature fusion is generally carried out in either a top-down or bottom-up manner. However, two types of imbalance may exist: feature imbalance in the neck of the model and gradient imbalance in the region of interest extraction layer due to the scale changes of objects. The deeper the network is, the more abstract the learned features are, that is to say, more semantic information can be extracted. However, the extracted image background, spatial location, and other resolution information are less. In contrast, the shallow part can learn little semantic information, but a lot of spatial location information. We propose the Both Ends to Centre to Multiple Layers (BEtM) feature fusion method to solve the feature imbalance problem in the neck and a Multi-level Region of Interest Feature Extraction (MRoIE) layer to solve the gradient imbalance problem. In combination with the Region-based Convolutional Neural Network (R-CNN) framework, our Balanced Feature Fusion (BFF) method offers significantly improved network performance compared with the Faster R-CNN architecture. On the MS COCO 2017 dataset, it achieves an average precision (AP) that is 1.9 points and 3.2 points higher than those of the Feature Pyramid Network (FPN) Faster R-CNN framework and the Generic Region of Interest Extractor (GRoIE) framework, respectively.

HighlightsA feature pyramid network with Scaling Feature Fusion and Enhanced Expansion Feature module is constructed to overcome the shortcomings of feature pyramids for small target detection.Noise addition, cropping, and other data augmentation methods for expanding data sets.Multi-channel Denoising Feature Fusion is constructed in the detection module to solve the inconsistencies inside the feature pyramid.Combine with YOLOv5's Backbone to complete the feature anchor box selection and detection can quickly and accurately complete the grape leaf disease detection.ABSTRACT.Grape leaf diseases all over the world cannot be ignored; it is one of the important causes of agricultural economic damage. In order to control the cost of manual detection of grape diseases and to overcome the impact on detection accuracy due to the large variation of target scales during automatic detection of grape leaves, the feature pyramid method is commonly used to solve this problem, but it may destroy the feature consistency of grape leaves at different scales. In addition, in practical applications, the commonly used methods are difficult to improve the detection accuracy of multi-scale grape leaves while ensuring detection. In this paper, we apply the improved YOLOv5 algorithm, which is used to detect the images of grapevine powdery mildew, grape mosaic virus disease, and grape downy mildew, as well as the images of healthy grape leaves. The proposed algorithm is based on the improved feature pyramid network SE-FPN, which added Scaling Feature Fusion (SFF) to enhance feature map information and Enhanced Expansion Feature (EEF) to improve multi-scale accuracy prediction. And replacing the original Path Aggregation Network (PANet) detection module with the Multi-channel Denoising Feature Fusion (MDFF) network improves the scale invariance of features and reduces the inference cost. By experimentally comparing the original YOLOv5 network and the YOLOv5 network with other improved methods applied, considering the following metrics: mean Average Precision (mAP), accuracy, recall, and mean target detection loss, mean classification loss, and runtime of the validation set, the results conclude that the improved model proposed in this paper performs the best, which increases the accuracy by 5.96% and the recall by 4.69% compared to the original YOLOv5 model. The algorithm provides good technical support for real-time detection and control of grapevine diseases. Keywords: Detection module, Disease identification, Feature pyramid network, Grape leaf disease, YOLO.

Object detection is challenging in large-scale images captured by unmanned aerial vehicles (UAVs), especially when detecting small objects with significant scale variation. Most solutions employ the fusion of different scale features by building multi-scale feature pyramids to ensure that the detail and semantic information are abundant. Although feature fusion benefits object detection, it still requires the long-range dependencies information necessary for small objects with significant scale variation detection. We propose a simple yet effective scale enhancement pyramid network (SEPNet) to address these problems. A SEPNet consists of a context enhancement module (CEM) and feature alignment module (FAM). Technically, the CEM combines multi-scale atrous convolution and multi-branch grouped convolution to model global relationships. Additionally, it enhances object feature representation, preventing features with lost spatial information from flowing into the feature pyramid network (FPN). The FAM adaptively learns offsets of pixels to preserve feature consistency. The FAM aims to adjust the location of sampling points in the convolutional kernel, effectively alleviating information conflict caused by the fusion of adjacent features. Results indicate that the SEPNet achieves an AP score of 18.9% on VisDrone, which is 7.1% higher than the AP score of state-of-the-art detectors RetinaNet achieves an AP score of 81.5% on PASCAL VOC.

YOLOv3 is a deep learning-based real-time object detector and is mainly used in applications such as video surveillance and autonomous vehicles. In this paper, we proposed an improved YOLOv3 (You Only Look Once version 3) applied Duplex FPN, which enhanced large object detection by utilizing low-level feature information. The conventional YOLOv3 improved the small object detection performance by applying FPN (Feature Pyramid Networks) structure to YOLOv2. However, YOLOv3 with an FPN structure specialized in detecting small objects, so it is difficult to detect large objects. Therefore, this paper proposed an improved YOLOv3 applied Duplex FPN, which can utilize low-level location information in high-level feature maps instead of the existing FPN structure of YOLOv3. This improved the detection accuracy of large objects. Also, an extra detection layer was added to the top-level feature map to prevent failure of detection of parts of large objects. Further, dimension clusters of each detection layer were reassigned to learn quickly how to accurately detect objects. The proposed method was compared and analyzed in the PASCAL VOC dataset. The experimental results showed that the bounding box accuracy of large objects improved owing to the Duplex FPN and extra detection layer, and the proposed method succeeded in detecting large objects that the existing YOLOv3 did not.