Evaluation of automated target detection using image fusion

Automatic target recognition (ATR) is a challenging task for several computer vision applications. It requires efficient, accurate, and robust methods for target detection and target identification. Deep learning has shown great success in many computer vision applications involving color RGB images. However, the performance of these networks in ATR with infrared sensor data needs further investigation. In this paper, we propose a multistage automatic target detection and recognition (ATDR) system that performs both target detection and target classification on infrared (IR) imagery using deep learning. Our system processes large IR image frames where targets take <1 % of the total number of pixels. First, we train a state-of-the-art object detector you only look once (YOLO) to localize all potential targets in the input image frame. Then, we train a convolutional neural network (CNN) to identify these detections as targets or false alarms. In this second phase, we adapt and analyze the performance of three CNN architectures: a compact and fully connected CNN, VGG16 with batch normalization, and a wide residual neural network (WRN). We also explore the use of a loss function that optimizes directly the area under the receiver operating characteristic (ROC) curve (AUC), and adapt it to our ATR application. To enhance the robustness of the proposed ATR to perturbation and variations introduced during the detection stage, we train our CNN classifiers on automatically detected targets using YOLO, in addition to ground truth bounding boxes and apply selected data augmentation techniques. To simulate real testing environments, where the spatial location of the targets within the image frame is unknown, only YOLO-detected boxes are used during validation. We evaluate our ATDR on a real benchmark dataset that includes different vehicles captured at different resolutions. Our experiments have shown that YOLO can detect most of the targets at the expense of generating a high number of false alarms. We show that the VGG-16 network with batch normalization, which is the best performing model, can correctly identify the classes of the targets, as well as classify the majority of YOLO’s false detections into an additional nontarget class. We also show that the proposed training modification to optimize an AUC-based loss function for ATR proved to be advantageous mainly in identifying difficult targets.

Performance of automatic target recognition (ATR) systems depends on numerous factors including the mission description, operating conditions, sensor modality, and ATR algorithm itself. Performance prediction models sensitive to these factors could be applied to ATR algorithm design, mission planning, sensor resource management, and data collection design for algorithm verification. Ideally, such a model would return measures of performance (MOPs) such as probability of detection (Pd), correct classification (Pc), and false alarm (Pfa), all as a function of the relevant predictor variables. Here we discuss the challenges of model-based and data-based approaches to performance prediction, concentrating especially on the synthetic aperture radar (SAR) modality. Our principal conclusion for model-based performance models (predictive models derived from fundamental physics- and statistics-based considerations) is that analytical progress can be made for performance of ATR system components, but that performance prediction for an entire ATR system under realistic conditions will likely require the combined use of Monte Carlo simulations, analytical development, and careful comparison to MOPs from real experiments. The latter are valuable for their high-fidelity, but have a limited range of applicability. Our principal conclusion for data-based performance models (that fit empirically derived MOPs) offer a potentially important means for extending the utility of empirical results. However, great care must be taken in their construction due to the necessarily sparse sampling of operating conditions, the high-dimensionality of the input space, and the diverse character of the predictor variables. Also the applicability of such models for extrapolation is an open question.

In this paper, probabilistic fusion of multi-sensor data is applied to mine detection. Probabilistic fusion combines information in the form of scores from automatic target recognition (ATR) algorithms for each sensor. This fusion method has previously demonstrated improved mine detection performance when used with multi-sensor data from the Mine Hunter/Killer system. The sensor suite includes a ground- penetrating radar, metal detectors, and an IR camera; data were collected at a prepared test site. Results of applying the probabilistic fusion method to recent MH/K multi-sensor data using various new ATR algorithms are presented and analyzed in detail. Changes in detection performance are quantified for different combinations of the various ATR algorithms and sensors. It is shown that fusion improves mine detection performance even when the individual sensor and ATR algorithms have very different performance levels. This implies that multi-sensor approaches to mien detection should continue to be pursued.

For the Automatic Target Recognition (ATR) algorithm, the quality of the input image sequence can be a major determining factor as to the ATR algorithm's ability to recognize an object. Based on quality, an image can be easy to recognize, barely recognizable or even mangled beyond recognition. If a determination of the image quality can be made prior to entering the ATR algorithm, then a confidence factor can be applied to the probability of recognition. This confidence factor can be used to rate sensors; to improve quality through selectively preprocessing image sequences prior to applying ATR; or to limit the problem space by determining which image sequences need not be processed by the ATR algorithm. It could even determine when human intervention is needed. To get a flavor for the scope of the image quality problem, this paper reviews analog and digital forms of image degradation. It looks at traditional quality metric approaches such as peak signal-to-noise ratio. It examines a newer metric based on human vision data, a metric introduced by the Institute for Telecommunication Sciences. These objective quality metrics can be used as confidence factors primarily in ATR systems that use image sequences degraded due to transmission systems. However, to determine the quality metric, a transmission system needs the original input image sequence and the degraded output image sequence. This paper suggests a more general approach to determining quality using analysis of spatial and temporal vectors where the original input sequence is not explicitly given. This novel approach would be useful where there is no transmission system but where the ATR system is part of the sensor, on-board a mobile platform. The results of this work are demonstrated on a few standard image sequences.

In this paper we empirically examine the probability of detection (PD) and false alarm rate (FAR) for crash site detectionusing polarimetry to discriminate between aircraft target signatures within natural clutter. To date, the search and rescueprogram has tried several automatic target recognition (ATR) algorithms from the literature. While PD seems reasonablewith these algorithms, the FAR is too high (—iO's per square kilometer). The objective of our analysis is to determine if thisis a limitation of the ATR algorithms tried, or if this is the best that can be hoped for given the polarimetric statistics of thetarget and clutter.Keywords: Search and Rescue, Polarimetry, Cameron Transformation, Synthetic Aperture Radar 1. INTRODUCTION Beaconless search and rescue utilizes a synthetic aperture radar (SAR) system to search for downed aircraft in remoteregions. Low frequency SAR is capable of quickly imaging large areas and penetrating bad weather and foliage cover.Coupled to a computer processing system, that includes automatic crash site detection algorithms, SAR has the potential tobecome a significantly useful search tool. We are currently determining the fundamental limits on the probability ofdetection versus the false alarm rate for crash site detection using radar polarimetry. In this paper we present our analysis ofdata from Half Moon Bay (HMB), California, Teton National Park, Wyoming, and from scale-model tests done by SystemPlanning Corporation (SPC), Virginia.

The Deputy Under Secretary of Defense for Science and Technology (DUSD/S&T), as part of their ongoing ATR Program, has sponsored an effort to develop and demonstrate methods for evaluating ATR algorithms that utilize multiple data sources, i.e., fusion-based ATR. The AFRL COMPASE Center has formed a strong ATR evaluation team and this paper presents results from this program, focusing on the human-in-the-loop, i.e. assisted image exploitation. Reliance on Automated Target Recognition (ATR) technology is essential to the future success of Intelligence, Surveillance, and Reconnaissance (ISR) missions. Often, ATR technology is designed to aid the analyst, but the final decision rests with the human. Traditionally, evaluation of ATR systems has focused mainly on the performance of the algorithm. Assessing the benefits of ATR assistance for the user raises interesting methodological challenges. We will review the critical issues associated with evaluations of human-in-the-loop ATR systems and present a methodology for conducting these evaluations. Experimental design issues addressed in this discussion include training, learning effects, and human factors issues. The evaluation process becomes increasingly complex when data fusion is introduced. Even in the absence of ATR assistance, the simultaneous exploitation of multiple frames of co-registered imagery is not well understood. We will explore how the methodology developed for exploitation of a single source of data can be extended to the fusion setting.

The development of tools for automated analysis of data collected for mine countermeasures (MCM) search, classify and map (SCM) has predominantly focused on automated target recognition for individual mines. As MCM target detection becomes more automated, detecting spatial patterns in the positioning of mines should follow. Natural restrictions on mine deployment such as depth and bottom contour as well as mine deployment methodology and protocol effectively create mine line patterns that are typical to coastal areas. We have developed a mine line detection algorithm that uses nonparametric spatial analysis to identify mine-like contacts (MLCs) that fit into mine line patterns dictated by standard mine laying protocol in coastal regions. In general, a set of MLCs reported from a mine detection sensor will consist of both actual mines and false alarms; the algorithm classifies every MLC in this set into one of three classes: fixed — the MLCψfits into a mine line pattern with fixed mine spacing; random -the MLCψfits into a mine line pattern with random mine spacing; or false alarm-ψ-the MLCψdoes not fit into a mine line pattern and is subsequently not considered a mine. The latest version of the algorithm utilizes nonparametric techniques and requires no knowledge of either the detection sensor's probability of detection (P d ) or probability of false alarm (P fa ). A nonparametric anomaly detector first identifies regions with unusually high MLC density for more detailed analysis. Suspected regions are subsequently analyzed for unusually regular patterns of MLCs. The probabilities that the observed target densities and target patterns could occur simply by chance are estimated by comparison with results from nearby areas using basic information about mine deployment. To ensure that a mine line's full extent is captured, the algorithm attempts to grow a line after it is detected. And a more detailed probability analysis distinguishes random mine lines from fixed-spaced mine lines. Extensive Matlab simulations characterize the algorithm's performance across a wide range of operating environments and simulated minefield configurations. In total, ten replications each of 1,367 different configurations have been evaluated. These configurations include three different water depths (10 ft., 6 ft. and 200 ft.), six false notional alarm rates (1, 2,5,10,15 and 20 false alarms per sq. NM), three false alarm distributions (uniform, sloped, and striped), three sensor probabilities of detections (0.4, 0.6 and 0.9), three target confidence scores (0.6,0.8 and 1.0) and four false alarm confidence scores (0.5, 0.6, 0.8 and 1.0). The range of simulation parameters was selected in part as an attempt to push the algorithm past its acceptable performance limits. However, the algorithm performed remarkably well even under the worst conditions. For MLC sets with false alarm rates as high as 20 per square nautical mile, the algorithm correctly classifies over 98% of the MLCs as mines and detects 97% of the mine lines. Furthermore, it reduces the false alarm rate from 20 to 0.2 per square nautical mile with a false mine line rate of 0.14 false lines per square nautical mile. The algorithm achieves even lower false alarm rates when the initial false alarm rate is less than 20. The automated mine line detection discovers many obscure lines that are virtually undetectable to the naked eye.

This paper smmnarizes a preliminary investigation into a space-based multi-class system-oriented High Range Resolution (HRR) Automatic Target Recognition (ATR) algorithm. Recent conflicts have highlighted the need to detect, track and recognize time critical moving ground targets in all-weather standoff (airborne and space-based) sensor platforms. Current sensor technologies based on Moving Target Indication (MTI) and/or Synthetic Aperture Radar (SAR) imaging are inadequate. While MT1 provides the ability to detect and track ground moving targets, it lacks target discrimination capability due to limited bandwidth. SAR has target discrimination capability for stationary targets, but target motion will blur the image. As a result, ATR of ground moving targets using HRR radar has recently received increased interest within the Air Force. The baseline algorithms presented in this paper are template-based mean squared error classifiers. The paper includes a brief introduction into recent HRR ATR efforts at the Air Force Research Laboratory Sensors Directorate's ATR Assessment Branch (AFRL/SNAA). Motivations and limitations for space- based sensors are discussed along with potential sensor fusion algorithms. The paper concludes by discussing the advantages of HRR ATR for feature aided

Automatic detection and recognition of targets by means of passive IR sensors suffer from limitations due to lack of sufficient contrast between the targets and their background, and among the facets of a target. In this paper the results of a suite of polarization-sensitive automatic target detection and recognition algorithms on sets of simulated and real polarimetric IR imagery are presented. A custom designed Polarimetric IR (PIR) imaging sensor is used for collecting real polarimetric target data-three of the four Stokes parameters under a variety of conditions. Then a set of novel algorithms are designed and tested that uses the target and background Stokes parameters for detection, segmentation and classification of targets. The empirical performance results are obtained in terms of the probabilities of detection, false alarm rate, segmentation accuracy, and recognition probabilities as functions of number of pixels on target, aspect and depression angles and under several background conditions (clutter densities) on the polarimetric and non-polarimetrirc data. These results show that a noticeable improvement over the non-polarimetric ATR can be achieved.

A crucial deficiency of lossy blockwise image compression is the generation of local artifacts such as ringing defects, obscuration of fine detail, and blocking effect (BE). To date, few published reports of image quality measures (IQMs) have addressed the detection of such errors in a realistic, efficient manner. Exceptions are feature-based IQMs, perceptual IQMs, error detection templates, and quantification of BE that support its reduction in JPEG-and wavelet-compressed imagery. In this paper, we present an enhanced suite of IQMs that emphasize detection of local, feature-specific errors that corrupt visual appearance or numerical integrity of decompressed digital imagery. By the term visual appearance is meant subjective error, in contrast with objectively quantified effects of compression on individual pixel values and their spatial interrelationships. Subjective error is of key importance in human viewing applications, for examples, Internet video. Objective error is primarily of interest in object recognition applications such as automated target recognition (ATR), where implementational concerns involve the effect of compression or decompression algorithms on probability of detection (Pd) and rate of false alarms (Rfa). Analysis of results presented herein emphasizes application- specific quantification of local compression errors. In particular, introduction of extraneous detail (e.g., ringing defects of BE) or obscuration of source detail (e.g., texture masking) adversely impact both subjective and objective error of a decompressed image. Blocking effect is primarily a visual problem, but can confound ATR filters when a target spans a block boundary. Introduction of point or cluster errors primarily degrades ATR filter performance, but can also produce noticeable degradation of fine detail for human visual evaluation of decompressed imagery. In practice, error and performance analysis is supported by examples of ATR imagery including airborne and underwater mono-and multi-spectral images.

Automatic Target Recognition (ATR) is a key element of expeditionary Mine Countermeasures (MCM) and port protection operations. Most existing approaches to ATR are currently based on high resolution sonar sensors, which provide enough information to obtain satisfactory detection and classification performance for the large World War 2 mine types (e.g., 2 m long cylinder). False alarm rates, however, are still unacceptably high for modern mines, which constrains the way operations are undertaken and often requires either confirmation or re-evaluation by a human operator. The introduction of new AUV-mounted Synthetic Aperture Sonars (SAS) increases the resolution, quality and range of acquired sonar images, which broadens the set of machine vision and computer image analysis techniques that can be used for underwater ATR operations. In this paper we study the impact that the increased quality and resolution have on performance gains and false alarm reduction. A number of classification algorithms are selected to represent the pool of existing approaches to target detection and classification, and their performances are estimated using both simulated and real image data in order to quantify the benefits associated to the new SAS technology. Evolution and near-future plans are discussed, introducing emerging bio-sonar sensors, anomaly detectors and autonomous AUV systems.

For decades, broadband sonar has been a key tool in the underwater detection and classification of man-made targets. The DoD’s recent focus to clean up discarded WWII-era munitions has pushed a flurry of research and advancements in the areas of target scattering, environmental noise modeling, signal processing, and automatic target recognition (ATR). It is the interdisciplinary nature of this problem that makes it incredibly challenging, namely, the fact that the ability to classify an object is heavily influenced by the operational choices (grazing angle and vehicle altitude), surrounding environmental factors (target type, burial state, sea state, and bottom topography), and the processing tools (data products, feature-set, and ATR training/testing). This talk will outline the key steps underlying the process of trying to detect and classify underwater unexploded ordnance (UXO). It begins by pushing raw time domain data, which can be generated either experimentally or via models, through a set of signal processing tools to create data products (acoustic color and SAS images). These data products are fed through feature extraction processes, followed by ATR algorithms, in order to arrive at the final classification of an object. Examples to be presented include objects of interest to the remediation of underwater UXO. [Research supported by SERDP and ONR.].

For new military aircraft a specification of sensor characteristics and performance is required at an early stage in the design cycle, well before testing of a prototype. In the early days of military aviation the Johnson criteria [1] were used to determine the sensor resolution required for target recognition by a human. In the present day sensor data are processed by computer using various Automatic Target Recognition (ATR) algorithms. However there is no accepted method for predicting the sensor resolution and SNR required for reliable ATR and hence there is risk that any chosen sensor may fail to support the required ATR performance. This paper reports a study into the use of publicly-available CAD models for aircraft to address this requirement. The study considers the worst-case confusion between two views of 15 different aircraft types. For simplicity only rotations by an angle θ about the Z (vertical) axis are considered. Firstly the sensor resolution and noise level is fixed. Then for each aircraft type and view angle an ensemble of synthetic silhouettes are generated. Using these ensembles, a-posteriori distributions of 5 standard scale-invariant shape features (eccentricity, orientation, solidity, circularity and bounding box aspect ratio) are estimated for each view angle θ. The performance of ATR at the given resolution and noise level is predicted by estimating the Bayes Error Rate [2] when deciding between each aircraft type and the 14 non-matching types using these features. The worst-case confusion in terms of erroneous aircraft type and view angle is identified. The sensor resolution is then changed and the above process repeated to investigate the effect of varying sensor resolution on performance. As expected, high sensor resolution leads to low probability of misclassification, even in the worst-case. Reduction in resolution and increasing noise level causes the Bayes Error Rate to rise quickly. The Bayes Error Rate gives a fundamental limit to the reliability of classification, irrespective of the actual type of classification algorithm used. The predictions from the model are confirmed by testing against a standard classifier for specific discrimination examples. Further development of the approach presented is expected to yield a method for specifying sensor resolution requirements for specific ATR problems.

Most of the research on developing automatic target recognition (ATR) algorithms for acoustic-seismic landmine detection platforms has been focused on using geometric features, such as size and shape, of anomaly to distinguish between mines and clutter. This approach has achieved some success especially in detecting larger anti-tank mines. However, for smaller anti-personnel landmines, the difference in geometric features between mines and clutter can be very small, if any. To improve the detection vs. false alarm rates, it is necessary to incorporate other features into the ATR process. It has been observed from the collected acoustic data that areas with buried mines reveal more complicated surface vibration structures, such as the ring-like pattern, at certain frequencies than what a one-dimensional lumped mass-spring-dashpot model can describe. In this paper, we utilize the distributed mine/soil interaction model developed by the University of Mississippi to describe the surface vibration patterns. We develop a modified Hankel transform to extract features from areas under interrogation. Under such transform, concentration of energy is closely related to an object's physical properties. The frequency at which the energy concentration occurs corresponds to the object's natural frequency, while the corresponding Bessel basis captures its mode shape. After de-noising the transformed data, we use the frequencies, Bessel bases, and magnitudes of the energy concentrations, together with other geometric features, to form the feature vectors. We tested these features on a dataset consisting of anti-tank and anti-personnel mines as well as blank areas and metallic and non-metallic clutter. Classifiers designed based on the combined geometric and model-based features perform significantly better than those based on the geometric features alone.

To quantify the agility of a synthetic aperture radar (SAR) system using an automatic target recognition (ATR) system to detect targets obscured by foliage, an ultra-wideband, UHF- band, polarimetric SAR was constructed by ERIM under ARPA funding and installed on a Navy P-3 aircraft controlled by the Naval Air Warfare Center. The system was implemented as an upgrade to the existing X-, L-, and C-band SAR system already on this aircraft. A series of experiments funded by ARPA and Wright Laboratory were undertaken in 1995 to investigate foliage penetration (FOPEN). In this paper, the data and ground truth collected and their utility for investigations of FOPEN phenomenology and ATR algorithms will be presented. These data are being placed into a database for distribution to ATR algorithm developers. The characteristics of the P-3 UWB SAR will be discussed. The image formation technique used will be presented, along with the RFI suppression techniques used. Of particular interest will be the technique used for the required motion compensation. Results from recent investigations using the P-3 UWB SAR data will be discussed.