Generative Adversarial Networks for Improving Object Detection in Camouflaged Images

Spiking Neural Networks (SNNs), as a biologically inspired computational model, have garnered significant attention in object detection and image classification due to their event-driven mechanism and low-power characteristics. However, in object detection tasks, the residual structures in conventional networks introduce nonspiking operations, posing a critical challenge for SNNs. To address this issue, we propose a multi-layer feature cascade fusion SNN (MFCF-SNN) for object detection. During feature extraction, our novel multi-level cascaded feature extraction module replaces residual connections with cascade operations, eliminating nonspiking computations while enhancing gradient propagation to deeper layers. For downsampling, we introduce a pooling-convolution module that combines max-pooling and spiking convolution, effectively preserving feature information and improving gradient flow. These two modules collectively ensure pure spike-based computation while facilitating deep network training, thereby enhancing detection accuracy. Experimental results on the PASCAL VOC 2012 and SSDD datasets demonstrate state-of-the-art performance, validating the effectiveness of our approach in advancing SNN-based object detection.

Object detection is a core computer vision task that aims to localize and classify categories for various objects in an image. With the development of convolutional neural networks, deep learning methods have been widely used in the object detection task, achieving promising performance compared to traditional methods. However, designing a well-performing detection network is inefficient. It consumes too much hardware resources and time to trial, and it also heavily relies on expert knowledge. To efficiently design the neural network architecture, there has been a growing interest in automatically designing neural network architecture by Neural Architecture Search (NAS). In this paper, we propose a Memory-Efficient Multi-Agent Neural Architecture Search (MEMA-NAS) framework in end-to-end object detection neural network. Specifically, we introduce the multi-agent learning to search holistic architecture of the detection network. In this way, a lot of GPU memory is saved, allowing us to search each module’s architecture of the detection network simultaneously. To find a better tradeoff between the precision and computational costs, we add the resource constraint in our method. Search experiments on multiple datasets show that MEMA-NAS achieves state-of-the-art results in search efficiency and precision.

Generally, most approaches using methods such as cropping, rotating, and flipping achieve more data to train models for improving the accuracy of detection and segmentation. However, due to the difficulties of labeling such data especially semantic segmentation data, those traditional data augmentation methodologies cannot help a lot when the training set is really limited. In this paper, a model named OFA-Net (One For All Network) is proposed to combine object detection and semantic segmentation tasks. Meanwhile, using a strategy called “1-N Alternation” to train the OFA-Net model, which can make a fusion of features from detection and segmentation data. The results show that object detection data can be recruited to better the segmentation accuracy performance, and furthermore, segmentation data assist a lot to enhance the confidence of predictions for object detection. Finally, the OFA-Net model is trained without traditional data augmentation methodologies and tested on the KITTI test server. The model works well on the KITTI Road Segmentation challenge and can do a good job on the object detection task.

Deep Consensus Network: Aggregating predictions to improve object detection in microscopy images

Object detection is a fundamental task in computer vision, which is usually based on convolutional neural networks (CNNs). While it is difficult to be deployed in embedded devices due to the huge storage and computing consumptions, binary neural networks (BNNs) can execute object detection with limited resources. However, the extreme quantification in BNN causes diversity of feature representation loss, which eventually influences the object detection performance. In this paper, we propose a method balancing Information Retention and Deviation Control to achieve effective object detection, named IR-DC Net. On the one hand, we introduce the KL-Divergence to compose multiple entropy for maximizing the available information. On the other hand, we design a lightweight convolutional module to generate scale factors dynamically for minimizing the deviation between binary and real convolution. The experiments on PASCAL VOC, COCO2014, KITTI, and VisDrone datasets show that our method improved the accuracy in comparison with previous binary neural networks.

Deep Convolutional Neural Networks (CNN) are the state-of-the-art performers for the object detection task. It is well known that object detection requires more com- putation and memory than image classification. In this work, we propose LCDet, a fully-convolutional neural net- work for generic object detection that aims to work in em- bedded systems. We design and develop an end-to-end TensorFlow(TF)-based model. The detection works by a single forward pass through the network. Additionally, we employ 8-bit quantization on the learned weights. As a use case, we choose face detection and train the proposed model on images containing a varying number of faces of different sizes. We evaluate the face detection perfor- mance on publicly available dataset FDDB and Widerface. Our experimental results show that the proposed method achieves comparative accuracy comparing with state-of- the-art CNN-based face detection methods while reducing the model size by 3× and memory-BW by 3 - 4× compar- ing with one of the best real-time CNN-based object de- tector YOLO [23]. Our 8-bit fixed-point TF-model pro- vides additional 4× memory reduction while keeping the accuracy nearly as good as the floating point model and achieves 20× performance gain compared to the floating point model. Thus the proposed model is amenable for em- bedded implementations and is generic to be extended to any number of categories of objects.

Deep Learning Models or Neural Networks are loosely modeled based on the Human brain. Neural Networks consist of thousands of simple processing units that are densely interconnected, and are a means of performing machine learning. They aim to recognize patterns in data that enable a machine to perform some tasks based on analyzing several examples. One popular field of Deep Learning is Computer Vision that aims to simulate the complex visual system of human beings. An object detection model for instance, might be fed on thousands of images of objects to find visual patterns in the images that consistently correlate with particular labels. However, human visual perception is based on both visual and non-visual modalities of data. This paper aims to propose a Neural Network for Object detection that use various modalities of data, specifically Images and Text, to perform the object detection task

Much research on object detection focuses on building better model architectures and detection algorithms. Changing the model architecture, however, comes at the cost of adding more complexity to inference, making models slower. Data augmentation, on the other hand, doesn’t add any inference complexity, but is insufficiently studied in object detection for two reasons. First it is more difficult to design plausible augmentation strategies for object detection than for classification, because one must handle the complexity of bounding boxes if geometric transformations are applied. Secondly, data augmentation attracts less research attention perhaps because it is believed to add less value and to transfer poorly compared to advances in network architectures.This paper serves two main purposes. First, we propose to use AutoAugment [3] to design better data augmentation strategies for object detection because it can address the difficulty of designing them. Second, we use the method to assess the value of data augmentation in object detection and compare it against the value of architectures. Our investigation into data augmentation for object detection identifies two surprising results. First, by changing the data augmentation strategy to our method, AutoAugment for detection, we can improve RetinaNet with a ResNet-50 backbone from 36.7 to 39.0 mAP on COCO, a difference of +2.3 mAP. This gain exceeds the gain achieved by switching the backbone from ResNet-50 to ResNet-101 (+2.1 mAP), which incurs additional training and inference costs. The second surprising finding is that our strategies found on the COCO dataset transfer well to the PASCAL dataset to improve accuracy by +2.7 mAP. These results together with our systematic studies of data augmentation call into question previous assumptions about the role and transferability of architectures versus data augmentation. In particular, changing the augmentation may lead to performance gains that are equally transferable as changing the underlying architecture.

Convolutional neural network based methods have dominated object detection in recent years, which can be divided into the one-stage approach and the two-stage approach. In general, the two-stage approach (e.g., Faster R-CNN) achieves high accuracy, while the one-stage approach (e.g., SSD) has the advantage of high efficiency. To inherit the merits of both while overcoming their disadvantages, we propose a novel single-shot based detector, namely RefineDet++, which achieves better accuracy than two-stage methods and maintains comparable efficiency of one-stage methods. The proposed RefineDet++ consists of two inter-connected modules: the anchor refinement module and the alignment detection module. Specifically, the former module aims to (1) filter out negative anchors to reduce search space for the subsequent classifier, and (2) coarsely adjust the locations and sizes of anchors to provide better initialization for the subsequent regressor. The latter module takes (1) the refined anchors as the input from the former module with (2) a newly designed alignment convolution operation to further improve the regression accuracy and predict multi-class label. Meanwhile, we design a transfer connection block to transfer the features in the anchor refinement module to predict locations, sizes and class labels of objects in the object detection module. The multi-task loss function enables us to train the whole network in an end-to-end way. Extensive experiments on PASCAL VOC and MS COCO demonstrate that RefineDet++ achieves state-of-the-art detection accuracy with high efficiency.

In the recent generations, object detection (OD) has played an important role ranging from navigation of autonomous vehicles to video data compression. In the area of OD, deep learning (DL) has played a vital role in detecting the objects automatically. Breakthroughs have been achieved in the field of object detection with the simultaneous advancements in the field of DL. In this article, the author proposes a region-based neural network for detecting desired object in which it overcomes the computational complexity acquired by convolutional neural networks and provides a better accuracy. For training larger datasets and reusability, the deep learning principles are also proposed.

Application of Deep Learning for Object Detection

360°cameras have gained popularity over the last few years. In this paper, we propose two fundamental techniques-Field-of-View IoU (FoV-IoU) and 360Augmentation for object detection in 360° images. Although most object detection neural networks designed for perspective images are applicable to 360° images in equirectangular projection (ERP) format, their performance deteriorates owing to the distortion in ERP images. Our method can be readily integrated with existing perspective object detectors and significantly improves the performance. The FoV-IoU computes the intersection-over-union of two Field-of-View bounding boxes in a spherical image which could be used for training, inference, and evaluation while 360Augmentation is a data augmentation technique specific to 360° object detection task which randomly rotates a spherical image and solves the bias due to the sphere-to-plane projection. We conduct extensive experiments on the 360° indoor dataset with different types of perspective object detectors and show the consistent effectiveness of our method.

This paper proposes a data augmentation strategy for improving on-road object detection based on a deep neural network. The method uses a single camera and detects objects based on an optimized deep neural network for a driving environment. The strategy also uses a single-shot multi-box detector (SSD) for object detection, which is a state-of-the-art deep-learning algorithm. The performance is improved by using data augmentation for an advanced driver assist system (ADAS) specific to on-road object recognition. The problem of object detection is first analyzed based on a deep neural network in the ADAS domain, and then representative object detection methods that use deep neural networks are surveyed. A restricted random crop process is suggested for detecting small objects in an image, and then a patch resampling strategy is proposed for solving the long tail property in an on-road dataset. The proposed ADAS domain-specific data augmentation method is adjusted for the original object detection method based on a deep neural network. The object detection results were evaluated using an embedded board on the KITTI benchmark dataset, and the suggested data augmentation method improves the average precision by 30%.

Deep learning has revolutionized the performance of classification and object detection, but meanwhile demands sufficient labeled data for training. Given insufficient data, while many techniques have been developed to help combat overfitting, the challenge remains if one tries to train deep networks, especially in the ill-posed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">extremely low data regimes</i> : only a small set of labeled data are available, and nothing – including unlabeled data – else. Such regimes arise from practical situations where not only data labeling but also data collection itself is expensive. We propose a deep adversarial data augmentation (DADA) technique to address the problem, in which we elaborately formulate data augmentation as a problem of training a class-conditional and supervised generative adversarial network (GAN). Specifically, a new discriminator loss is proposed to fit the goal of data augmentation, through which both real and augmented samples are enforced to contribute to and be consistent in finding the decision boundaries. Tailored training techniques are developed accordingly. To quantitatively validate its effectiveness, we first perform extensive simulations to show that DADA substantially outperforms both traditional data augmentation and a few GAN-based options. We then extend experiments to three real-world small labeled classification datasets where existing data augmentation and/or transfer learning strategies are either less effective or infeasible. We also demonstrate that DADA to can be extended to the detection task. We improve the pedestrian synthesis work by substitute for our discriminator and training scheme. Validation experiment shows that DADA can improve the detection mean average precision (mAP) compared with some traditional data augmentation techniques in object detection. Source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SchafferZhang/DADA</uri> .

A major challenge in applying deep learning to medical imaging is the paucity of annotated data. This study explores the use of synthetic images for data augmentation to address the challenge of limited annotated data in colonoscopy lesion classification. We demonstrate that synthetic colonoscopy images generated by Generative Adversarial Network (GAN) inversion can be used as training data to improve polyp classification performance by deep learning models. We invert pairs of images with the same label to a semantically rich and disentangled latent space and manipulate latent representations to produce new synthetic images. These synthetic images maintain the same label as the input pairs. We perform image modality translation (style transfer) between white light and narrow-band imaging (NBI). We also generate realistic synthetic lesion images by interpolating between original training images to increase the variety of lesion shapes in the training dataset. Our experiments show that GAN inversion can produce multiple colonoscopy data augmentations that improve the downstream polyp classification performance by 2.7% in F1-score and 4.9% in sensitivity over other methods, including state-of-the-art data augmentation. Testing on unseen out-of-domain data also showcased an improvement of 2.9% in F1-score and 2.7% in sensitivity. This approach outperforms other colonoscopy data augmentation techniques and does not require re-training multiple generative models. It also effectively uses information from diverse public datasets, even those not specifically designed for the targeted downstream task, resulting in strong domain generalizability. Project code and model: https://github.com/DurrLab/GAN-Inversion.