Cross-domain infrared target recognition based on simulation

Automatic target recognition (ATR) is a challenging task for several computer vision applications. It requires efficient, accurate, and robust methods for target detection and target identification. Deep learning has shown great success in many computer vision applications involving color RGB images. However, the performance of these networks in ATR with infrared sensor data needs further investigation. In this paper, we propose a multistage automatic target detection and recognition (ATDR) system that performs both target detection and target classification on infrared (IR) imagery using deep learning. Our system processes large IR image frames where targets take <1 % of the total number of pixels. First, we train a state-of-the-art object detector you only look once (YOLO) to localize all potential targets in the input image frame. Then, we train a convolutional neural network (CNN) to identify these detections as targets or false alarms. In this second phase, we adapt and analyze the performance of three CNN architectures: a compact and fully connected CNN, VGG16 with batch normalization, and a wide residual neural network (WRN). We also explore the use of a loss function that optimizes directly the area under the receiver operating characteristic (ROC) curve (AUC), and adapt it to our ATR application. To enhance the robustness of the proposed ATR to perturbation and variations introduced during the detection stage, we train our CNN classifiers on automatically detected targets using YOLO, in addition to ground truth bounding boxes and apply selected data augmentation techniques. To simulate real testing environments, where the spatial location of the targets within the image frame is unknown, only YOLO-detected boxes are used during validation. We evaluate our ATDR on a real benchmark dataset that includes different vehicles captured at different resolutions. Our experiments have shown that YOLO can detect most of the targets at the expense of generating a high number of false alarms. We show that the VGG-16 network with batch normalization, which is the best performing model, can correctly identify the classes of the targets, as well as classify the majority of YOLO’s false detections into an additional nontarget class. We also show that the proposed training modification to optimize an AUC-based loss function for ATR proved to be advantageous mainly in identifying difficult targets.

This paper presents a technique for automatic airborne target recognition and tracking in forward-looking infrared (FLIR) images with a complex background. An image splitting and merging method is applied for detecting target signals. The presence of a complex background due to clouds and sun glint generates clutter in the image with the resulting possibility of false alarms. A Bayesian classifier trained using the NMI (normalized moment of inertia) feature is proposed for efficient clutter rejection. After classification, target candidates are entered into a tracking filter. As an efficient and robust multi-target tracking filter in cluttered environments, the JDC-JIHPDAF is proposed. Experimental results using a wide range of real FLIR images ensure reliable classification and automatic target recognition performance.

This paper proposes an automatic target recognition (ATR) method for synthetic aperture radar (SAR) images based on information-decoupled representation. A typical SAR image of a ground target can be divided into three parts: target region, shadow and background. From the aspect of SAR target recognition, the target region and shadow contain discriminative information. However, they also include some confusing information because of the similarities of different targets. The background mainly contains redundant information, which has little contribution to the target recognition. Because the target segmentation may impair the discriminative information in the target region, the relatively simpler shadow segmentation is performed to separate the shadow region for information decoupling. Then, the information-decoupled representations are generated, i.e., the target image, shadow and original image. The background is retained in the target image, which represents the coupling of target backscattering and background. The original image and generated target image are classified using the sparse representation-based classification (SRC). Then, their classification results are combined by a score-level fusion for target recognition. The shadow image is not used because of its lower discriminability and possible segmentation errors. To evaluate the performance of the proposed method, extensive experiments are conducted on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset under both standard operating condition (SOC) and various extended operating conditions (EOCs). The proposed method can correctly classify 10 classes of targets with the percentage of correct classification (PCC) of 94.88% under SOC. With the PCCs of 93.15% and 75.03% under configuration variance and 45° depression angle, respectively, the superiority of the proposed is demonstrated in comparison with other methods. The robustness of the proposed method to both uniform and nonuniform shadow segmentation errors is validated with the PCCs over 93%. Moreover, with the maximum average precision of 0.9580, the proposed method is more effective than the reference methods on outlier rejection.

High-performance neural networks for synthetic aperture radar (SAR) automatic target recognition (ATR) often encounter the challenge of data scarcity. The lack of sufficient labeled SAR image datasets leads to the consideration of using simulated data to supplement the dataset. On the one hand, electromagnetic computation simulations provide high amplitude accuracy but are inefficient for large-scale datasets due to their complex computations and physical models. On the other hand, ray tracing simulations offer high geometric accuracy and computational efficiency but struggle with low amplitude correctness, hindering accurate numerical feature extraction. Furthermore, the emergence of generative adversarial networks (GANs) provides a way to generate simulated datasets, trying to balance computational efficiency with image quality. Nevertheless, the simulated SAR images generated based on random noise lack constraints, and it is also difficult to generate images that exceed the parameter conditions of the real image’s training set. Hence, it is essential to integrate physics-based simulation techniques into GANs to enhance the generalization ability of the imaging parameters. In this paper, we present the SingleScene-SAR Simulator, an efficient framework for SAR image simulation that operates under limited real SAR data. This simulator integrates rasterized shooting and bouncing rays (SBR) with cycle GAN, effectively achieving both amplitude correctness and geometric accuracy. The simulated images are appropriate for augmenting datasets in target recognition networks. Firstly, the SingleScene-SAR Simulator employs a rasterized SBR algorithm to generate radar cross section (RCS) images of target models. Secondly, a specific training pattern for cycle GAN is established to translate noisy RCS images into simulated SAR images that closely resemble real ones. Finally, these simulated images are utilized for data augmentation. Experimental results based on the constructed dataset show that with only one scene SAR image containing 30 target chips, the SingleScene-SAR Simulator can efficiently produce simulated SAR images that exhibit high similarity in both spatial and statistical distributions compared with real images. By employing simulated SAR images for data augmentation, the accuracy of target recognition networks can be consistently and significantly enhanced.

Image dehazing has always been a challenging topic in image processing. The development of deep learning methods, especially the generative adversarial networks (GAN), provides a new way for image dehazing. In recent years, many deep learning methods based on GAN have been applied to image dehazing. However, GAN has two problems in image dehazing. Firstly, For haze image, haze not only reduces the quality of the image but also blurs the details of the image. For GAN network, it is difficult for the generator to restore the details of the whole image while removing the haze. Secondly, GAN model is defined as a minimax problem, which weakens the loss function. It is difficult to distinguish whether GAN is making progress in the training process. Therefore, we propose a guided generative adversarial dehazing network (GGADN). Different from other generation adversarial networks, GGADN adds a guided module on the generator. The guided module verifies the network of each layer of the generator. At the same time, the details of the map generated by each layer are strengthened. Network training is based on the pre-trained VGG feature model and L1-regularized gradient prior which is developed by new loss function parameters. From the dehazing results of synthetic images and real images, the proposed method is better than the state-of-the-art dehazing methods.

Image synthesis in magnetic resonance (MR) imaging has been an active area of research for more than ten years. MR image synthesis can be used to create images that were not acquired or replace images that are corrupted by artifacts, which can be of great benefit in automatic image analysis. Although synthetic images have been used with success in many applications, it is quite often true that they do not look like real images. In practice, an expert can usually distinguish synthetic images from real ones. Generative adversarial networks (GANs) have significantly improved the realism of synthetic images. However, we argue that further improvements can be made through the introduction of noise in the synthesis process, which better models the actual imaging process. Accordingly, we propose a novel approach that incorporates randomness into the model in order to better approximate the distribution of real MR images. Results show that the proposed method has comparable accuracy with the state-of-the-art approaches as measured by multiple similarity measurements while also being able to control the noise level in synthetic images. To further demonstrate the superiority of this model, we present results from a human observer study on synthetic images, which shows that our results capture the essential features of real MR images.

Automatic target recognition (ATR) of synthetic aperture radar (SAR) images is performed on either global or local features. The global features can be extracted and classified with high efficiency. However, they lack the reasoning capability thus can hardly work well under the extended operation conditions (EOCs). The local features are often more difficult to extract and classify but they provide reasoning capability for EOC target recognition. To combine the efficiency and robustness in an ATR system, a hierarchical fusion of the global and local features is proposed for SAR ATR in this paper. As the global features, the random projection features can be efficiently extracted and effectively classified by sparse representation-based classification (SRC). The physically relevant local descriptors, i.e., attributed scattering centers (ASCs), are employed for local reasoning to handle various EOCs like noise corruption, resolution variance, and partial occlusion. A one-to-one correspondence between the test and template ASC sets is built by the Hungarian algorithm. Then, the local reasoning is performed by evaluating individual matched pairs as well as the false alarms and missing alarms. For the test image to be recognized, it is first classified by the global classifier, i.e., SRC. Once a reliable decision is made, the whole recognition process terminates. When the decision is not reliable enough, it is passed to the local classifier, where a further classification by ASC matching is carried out. Therefore, by the hierarchical fusion strategy, the efficiency of global features and the robustness of local descriptors to various EOCs can be maintained jointly in the ATR system. Extensive experiments on the moving and stationary target acquisition and recognition data set demonstrate that the proposed method achieves superior effectiveness and robustness under both SOC and typical EOCs, i.e., noise corruption, resolution variance, and partial occlusion, compared with some other SAR ATR methods.

PurposeTo develop generative adversarial networks (GANs) that synthesize realistic anterior segment optical coherence tomography (AS-OCT) images and evaluate deep learning (DL) models that are trained on real and synthetic datasets for detecting angle closure.MethodsThe GAN architecture was adopted and trained on the dataset with AS-OCT images collected from the Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, synthesizing open- and closed-angle AS-OCT images. A visual Turing test with two glaucoma specialists was performed to assess the image quality of real and synthetic images. DL models, trained on either real or synthetic datasets, were developed. Using the clinicians’ grading of the AS-OCT images as the reference standard, we compared the diagnostic performance of open-angle vs. closed-angle detection of DL models and the AS-OCT parameter, defined as a trabecular-iris space area 750 µm anterior to the scleral spur (TISA750), in a small independent validation dataset.ResultsThe GAN training included 28,643 AS-OCT anterior chamber angle (ACA) images. The real and synthetic datasets for DL model training have an equal distribution of open- and closed-angle images (all with 10,000 images each). The independent validation dataset included 238 open-angle and 243 closed-angle AS-OCT ACA images. The image quality of real versus synthetic AS-OCT images was similar, as assessed by the two glaucoma specialists, except for the scleral spur visibility. For the independent validation dataset, both DL models achieved higher areas under the curve compared with TISA750. Two DL models had areas under the curve of 0.97 (95% confidence interval, 0.96–0.99) and 0.94 (95% confidence interval, 0.92–0.96).ConclusionsThe GAN synthetic AS-OCT images appeared to be of good quality, according to the glaucoma specialists. The DL models, trained on all-synthetic AS-OCT images, can achieve high diagnostic performance.Translational RelevanceThe GANs can generate realistic AS-OCT images, which can also be used to train DL models.

Many deep learning (DL) frameworks have demonstrated state-of-the-art performance in the super-resolution (SR) task of magnetic resonance imaging, but most performances have been achieved with simulated low-resolution (LR) images rather than LR images from real acquisition. Due to the limited generalizability of the SR network, enhancement is not guaranteed for real LR images because of the unreality of the training LR images. In this study, we proposed a DL-based SR framework with an emphasis on data construction to achieve better performance on real LR MR images. The framework comprised two steps: (a) downsampling training using a generative adversarial network (GAN) to construct more realistic and perfectly matched LR/high-resolution (HR) pairs. The downsampling GAN input was real LR and HR images. The generator translated the HR images to LR images and the discriminator distinguished the patch-level difference between the synthetic and real LR images. (b) SR training was performed using an enhance4d deep super-resolution network (EDSR). In the controlled experiments, three EDSRs were trained using our proposed method, Gaussian blur, and k-space zero-filling. As for the data, liver MR images were obtained from 24 patients using breath-hold serial LR and HR scans (only HR images were used in the conventional methods). The k-space zero-filling group delivered almost zero enhancement on the real LR images and the Gaussian group produced a considerable number of artifacts. The proposed method exhibited significantly better resolution enhancement and fewer artifacts compared with the other two networks. Our method outperformed the Gaussian method by an improvement of 0.111 ± 0.016 in the structural similarity index and 2.76 ± 0.98 dB in the peak signal-to-noise ratio. The blind/reference-less image spatial quality evaluator metric of the conventional Gaussian method and proposed method were 46.6 ± 4.2 and 34.1 ± 2.4, respectively.

This article explores the application of Generative Adversarial Networks (GANs) in improving subsurface rock analysis. GANs generate realistic images of rock features, enhancing the accuracy and efficiency of estimating rock properties. The objective is to advance the field of subsurface rock analysis, offering insights into the potential of GAN techniques. The article reviews recent developments in data-driven rock reconstruction technology, specifically focusing on GANs. It highlights GANs’ ability to generate and estimate rock features like porosity and permeability. These GAN methods are complemented by discriminators to select or reject features, making them integral to the reconstruction model. The article provides a framework for engineers and researchers to effectively utilize these techniques for rock reconstruction, offering an assessment of their strengths and limitations. Digital rock reconstruction, aided by GANs, has promising implications for site selection and recovery. GANs enhance image resolution, improving the field of view and dynamism of parameters in rock analysis. However, GANs heavily rely on high-quality data, which poses a limitation. To address this, conditional GANs have been proposed. The article comprehensively reviews the latest developments in GAN methods for digital rock reconstruction, offering valuable insights for engineers and researchers using GANs for accurate subsurface rock analysis. It also proposes ways to enhance these methods and advance the field. The novelty of this article lies in its exploration of Generative Adversarial Networks (GANs) in the context of subsurface rock analysis. It emphasizes the potential of GAN techniques to generate realistic rock images and estimate properties like porosity and permeability. Additionally, the article discusses the integration of GANs into the digital rock concept, highlighting their role in improving predictive models and supporting decision-making in the oil and gas industry's resource extraction and production strategies.

Deep learning has become the main way of the object detection task for autonomous vehicles. Meanwhile, this method typically requires vast amounts of training data to reach their full potential. However, collecting the data from real world and labeling manually is an expensive, time-consuming and error-prone process. Synthetic image has the potential to replace real image for training neural networks, because image creation and labeling annotations are free in this way. For the network trained by synthetic images, the reality gap between real and synthetic images is the main obstacle to use it in the real world. And most previous works are only devoted to generate synthetic images with a good performance on model training, but lack of analysis of the domain gap that affects the performance. This work designs a method of generating the real and synthetic images to analyze and reduce the reality gap between synthetic and real images for car detection. Firstly, this work put one single car with no-background in a random background image to generate real single car and synthetic single car images. In order to further reduce the domain gap in content level, this method keeps the car distribution in synthetic images is similar with the distribution of car in real world. For the purpose of reducing the domain gap in appearance level, the parameters of camera model are same as the camera parameters of image collecting cars and the image is rendered by using the PBRT(Physically Based Ray Tracing) when we generated the synthetic images. Secondly, by training the neural network of instance segmentation with different datasets, the across validation result proves that the reality gap between synthetic and real images is no more than the domain gap between real images. Thirdly, the training results of datasets with different samples diversity show that the diversity of the samples yields better generalization between different datasets for car detection which can effectively reduce the domain gap.

Synthetic aperture radar (SAR) image target detection and recognition (SAR-TDR) tasks have become research hot spots in the remote sensing application. These targets include ships, vehicles, aircraft, oil tanks, bridges, and so on. However, with the rapid development of SAR technology and increasingly complex electromagnetic environment, complex characteristics of SAR images bring severe challenges to the accurate SAR-TDR via traditional physical models or manual feature-extraction-based machine learning methods. In recent years, deep learning (DL), as a powerful automatic feature extraction algorithm, has been widely used in the computer vision domain. More specifically, DL has also been introduced into the SAR-TDR tasks and has effectively achieved good performance in terms of accuracy, real-time processing, etc. With the rapid development of DL, SAR image processing, and practical requirements of SAR-TDR in civilian and military domains, it is crucial to conduct a systematic survey on SAR-TDR in the past few years. In this survey article, we mainly conduct a systematic overview of DL-based SAR-TDR literature on two tasks, i.e., target recognition (e.g., ground vehicles, ships, and aircraft) and target detection (e.g., ships, aircraft, change detection, sea surface oil spills, and oil tanks). More specifically, our related works about these topics are also presented to verify the effectiveness of DL-based methods. First, several DL methods (e.g., convolutional neural networks, recurrent neural networks, autoencoders, and generative adversarial networks), commonly used in SAR-TDR, are briefly introduced. Then, a systematic review of DL-based SAR-TDR (including our related works) is presented. Finally, the current challenges and future possible research directions are deeply analyzed and discussed.

Many applications reported in artificial neural networks are associated with military problems. This paper reviews concepts associated with the processing of military data to find and recognize targets -- automatic target recognition (ATR). A general-purpose automatic target recognition system does not exist. The work presented here is demonstrated on military data, but it can only be considered proof of principle until systems are fielded and proven `under- fire.' ATR data can be in the form of nonimaging one-dimensional sensor returns, such as ultra-high range-resolution radar returns (UHRR) for air-to-air automatic target recognition and vibration signatures from a laser radar for recognition of ground targets. The ATR data can be two-dimensional images. The most common ATR images are infrared, but current systems must also deal with synthetic aperture radar (SAR) images. Finally, the data can be three-dimensional, such as sequences of multiple exposures taken over time from a nonstationary world. Targets move, as do sensors, and that movement can be exploited by the ATR. Hyperspectral data, which are views of the same piece of the world looking at different spectral bands, is another example of multiple image data; the third dimension is now wavelength and not time. ATR system design usually consists of four stages. The first stage is to select the sensor or sensors to produce the target measurements. The next stage is the preprocessing of the data and the location of regions of interest within the data (segmentation). The human retina is a ruthless preprocessor. Physiologically motivated preprocessing and segmentation is demonstrated along with supervised and unsupervised artificial neural segmentation techniques. The third design step is feature extraction and selection: the extraction of a set of numbers which characterize regions of the data. The last step is the processing of the features for decision making (classification). The area of classification is where most ATR related neural network research has been accomplished. The relation of neural classifiers to Bayesian techniques is emphasized along with the more recent use of feature sequences to enhance classification. © (1995) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Multiview synthetic aperture radar (SAR) images contain much richer information for automatic target recognition (ATR) than a single-view one. It is desirable to establish a reasonable multiview ATR scheme and design effective ATR algorithm to thoroughly learn and extract that classification information, so that superior SAR ATR performance can be achieved. Hence, a general processing framework applicable for a multiview SAR ATR pattern is first given in this paper, which can provide an effective approach to ATR system design. Then, a new ATR method using a multiview deep feature learning network is designed based on the proposed multiview ATR framework. The proposed neural network is with a multiple input parallel topology and some distinct deep feature learning modules, with which significant classification features, the intra-view and inter-view features existing in the input multiview SAR images, will be learned simultaneously and thoroughly. Therefore, the proposed multiview deep feature learning network can achieve an excellent SAR ATR performance. Experimental results have shown the superiorities of the proposed multiview SAR ATR method under various operating conditions.

Mass production of high-quality synthetic SAR training imagery is essential for boosting the performance of deep-learning (DL)-based SAR automatic target recognition (ATR) algorithms in an open-world environment. To address this problem, we exploit both the widely used Moving and Stationary Target Acquisition and Recognition (MSTAR) SAR dataset and the Synthetic and Measured Paired Labeled Experiment (SAMPLE) dataset, which consists of selected samples from the MSTAR dataset and their computer-generated synthetic counterparts. A series of data augmentation experiments are carried out. First, the sparsity of the scattering centers of the targets is exploited for new target pose synthesis. Additionally, training data with various clutter backgrounds are synthesized via clutter transfer, so that the neural networks are better prepared to cope with background changes in the test samples. To effectively augment the synthetic SAR imagery in the SAMPLE dataset, a novel contrast-based data augmentation technique is proposed. To improve the robustness of neural networks against out-of-distribution (OOD) samples, the SAR images of ground military vehicles collected by the self-developed MiniSAR system are used as the training data for the adversarial outlier exposure procedure. Simulation results show that the proposed data augmentation methods are effective in improving both the target classification accuracy and the OOD detection performance. The purpose of this work is to establish the foundation for large-scale, open-field implementation of DL-based SAR-ATR systems, which is not only of great value in the sense of theoretical research, but is also potentially meaningful in the aspect of military application.