An ATR architecture for algorithm development and testing

For the Automatic Target Recognition (ATR) algorithm, the quality of the input image sequence can be a major determining factor as to the ATR algorithm's ability to recognize an object. Based on quality, an image can be easy to recognize, barely recognizable or even mangled beyond recognition. If a determination of the image quality can be made prior to entering the ATR algorithm, then a confidence factor can be applied to the probability of recognition. This confidence factor can be used to rate sensors; to improve quality through selectively preprocessing image sequences prior to applying ATR; or to limit the problem space by determining which image sequences need not be processed by the ATR algorithm. It could even determine when human intervention is needed. To get a flavor for the scope of the image quality problem, this paper reviews analog and digital forms of image degradation. It looks at traditional quality metric approaches such as peak signal-to-noise ratio. It examines a newer metric based on human vision data, a metric introduced by the Institute for Telecommunication Sciences. These objective quality metrics can be used as confidence factors primarily in ATR systems that use image sequences degraded due to transmission systems. However, to determine the quality metric, a transmission system needs the original input image sequence and the degraded output image sequence. This paper suggests a more general approach to determining quality using analysis of spatial and temporal vectors where the original input sequence is not explicitly given. This novel approach would be useful where there is no transmission system but where the ATR system is part of the sensor, on-board a mobile platform. The results of this work are demonstrated on a few standard image sequences.

In this paper, probabilistic fusion of multi-sensor data is applied to mine detection. Probabilistic fusion combines information in the form of scores from automatic target recognition (ATR) algorithms for each sensor. This fusion method has previously demonstrated improved mine detection performance when used with multi-sensor data from the Mine Hunter/Killer system. The sensor suite includes a ground- penetrating radar, metal detectors, and an IR camera; data were collected at a prepared test site. Results of applying the probabilistic fusion method to recent MH/K multi-sensor data using various new ATR algorithms are presented and analyzed in detail. Changes in detection performance are quantified for different combinations of the various ATR algorithms and sensors. It is shown that fusion improves mine detection performance even when the individual sensor and ATR algorithms have very different performance levels. This implies that multi-sensor approaches to mien detection should continue to be pursued.

Performance of automatic target recognition (ATR) systems depends on numerous factors including the mission description, operating conditions, sensor modality, and ATR algorithm itself. Performance prediction models sensitive to these factors could be applied to ATR algorithm design, mission planning, sensor resource management, and data collection design for algorithm verification. Ideally, such a model would return measures of performance (MOPs) such as probability of detection (Pd), correct classification (Pc), and false alarm (Pfa), all as a function of the relevant predictor variables. Here we discuss the challenges of model-based and data-based approaches to performance prediction, concentrating especially on the synthetic aperture radar (SAR) modality. Our principal conclusion for model-based performance models (predictive models derived from fundamental physics- and statistics-based considerations) is that analytical progress can be made for performance of ATR system components, but that performance prediction for an entire ATR system under realistic conditions will likely require the combined use of Monte Carlo simulations, analytical development, and careful comparison to MOPs from real experiments. The latter are valuable for their high-fidelity, but have a limited range of applicability. Our principal conclusion for data-based performance models (that fit empirically derived MOPs) offer a potentially important means for extending the utility of empirical results. However, great care must be taken in their construction due to the necessarily sparse sampling of operating conditions, the high-dimensionality of the input space, and the diverse character of the predictor variables. Also the applicability of such models for extrapolation is an open question.

A test system with four cameras in the infrared and visual spectra is under development at FFI (The Norwegian Defence Research Establishment). The system can be mounted on a high speed jet aircraft, but may also be used in a land-based version. It can be used for image acquisition as well as for development and test of automatic target recognition (ATR) algorithms. The sensors on board generate large amounts of data, and the scene may be rather cluttered or include anomalies (e.g. sun glare). This means we need image processing and pattern recognition algorithms which are robust, fast (real-time), and able to handle complex scenes. Algorithms based on order statistics are known to be robust and reliable. However, they are in general computationally heavy, and thus often unsuitable for real time applications. But approximations to order statistics do exist. Median of medians is one example. This is a technique where an approximation of the median of a sequence is found by first dividing the sequence in subsequences, and then calculating median (of medians) recursively. The algorithm is very efficient, the processing time is of order O(n). By utilizing such techniques for estimating image statistics, the computational challenge can be overcome. In this paper we present strategies for how approximations to order statistics can be applied for developing robust and fast algorithms for image processing, especially visualization and segmentation.

Combining the capability of an automated target recognition (ATR) system with the expertise of human operators has the potential to improve combined human-machine target-recognition performance. The objective of this work was to develop empirically validated guidelines for the display of information provided by an ATR system. Motivated by the premise that operators will have higher trust in an ATR system that justifies its decisions, this work explored alternative ATR displays that provided different types and amounts of information in support of the system's target decisions. Operators' usage of and preference for the displays were contrasted for low- and high-accuracy simulated ATR systems representative, respectively, of current and future technologies. Performance results suggested that the amount of supporting information that is displayed must be congruent with the ATR system's accuracy. Operators expressed a clear preference for an ATR system that provides a confidence rating for its target decisions. Based on the results design guidelines are suggested for displaying ATR information to help operators balance confidence in their own judgments with trust in the ATR's decisions.

Automatic target recognition (ATR) is a challenging task for several computer vision applications. It requires efficient, accurate, and robust methods for target detection and target identification. Deep learning has shown great success in many computer vision applications involving color RGB images. However, the performance of these networks in ATR with infrared sensor data needs further investigation. In this paper, we propose a multistage automatic target detection and recognition (ATDR) system that performs both target detection and target classification on infrared (IR) imagery using deep learning. Our system processes large IR image frames where targets take <1 % of the total number of pixels. First, we train a state-of-the-art object detector you only look once (YOLO) to localize all potential targets in the input image frame. Then, we train a convolutional neural network (CNN) to identify these detections as targets or false alarms. In this second phase, we adapt and analyze the performance of three CNN architectures: a compact and fully connected CNN, VGG16 with batch normalization, and a wide residual neural network (WRN). We also explore the use of a loss function that optimizes directly the area under the receiver operating characteristic (ROC) curve (AUC), and adapt it to our ATR application. To enhance the robustness of the proposed ATR to perturbation and variations introduced during the detection stage, we train our CNN classifiers on automatically detected targets using YOLO, in addition to ground truth bounding boxes and apply selected data augmentation techniques. To simulate real testing environments, where the spatial location of the targets within the image frame is unknown, only YOLO-detected boxes are used during validation. We evaluate our ATDR on a real benchmark dataset that includes different vehicles captured at different resolutions. Our experiments have shown that YOLO can detect most of the targets at the expense of generating a high number of false alarms. We show that the VGG-16 network with batch normalization, which is the best performing model, can correctly identify the classes of the targets, as well as classify the majority of YOLO’s false detections into an additional nontarget class. We also show that the proposed training modification to optimize an AUC-based loss function for ATR proved to be advantageous mainly in identifying difficult targets.

The Deputy Under Secretary of Defense for Science and Technology (DUSD/S&T), as part of their ongoing ATR Program, has sponsored an effort to develop and demonstrate methods for evaluating ATR algorithms that utilize multiple data sources, i.e., fusion-based ATR. The AFRL COMPASE Center has formed a strong ATR evaluation team and this paper presents results from this program, focusing on the human-in-the-loop, i.e. assisted image exploitation. Reliance on Automated Target Recognition (ATR) technology is essential to the future success of Intelligence, Surveillance, and Reconnaissance (ISR) missions. Often, ATR technology is designed to aid the analyst, but the final decision rests with the human. Traditionally, evaluation of ATR systems has focused mainly on the performance of the algorithm. Assessing the benefits of ATR assistance for the user raises interesting methodological challenges. We will review the critical issues associated with evaluations of human-in-the-loop ATR systems and present a methodology for conducting these evaluations. Experimental design issues addressed in this discussion include training, learning effects, and human factors issues. The evaluation process becomes increasingly complex when data fusion is introduced. Even in the absence of ATR assistance, the simultaneous exploitation of multiple frames of co-registered imagery is not well understood. We will explore how the methodology developed for exploitation of a single source of data can be extended to the fusion setting.

To quantify the agility of a synthetic aperture radar (SAR) system using an automatic target recognition (ATR) system to detect targets obscured by foliage, an ultra-wideband, UHF- band, polarimetric SAR was constructed by ERIM under ARPA funding and installed on a Navy P-3 aircraft controlled by the Naval Air Warfare Center. The system was implemented as an upgrade to the existing X-, L-, and C-band SAR system already on this aircraft. A series of experiments funded by ARPA and Wright Laboratory were undertaken in 1995 to investigate foliage penetration (FOPEN). In this paper, the data and ground truth collected and their utility for investigations of FOPEN phenomenology and ATR algorithms will be presented. These data are being placed into a database for distribution to ATR algorithm developers. The characteristics of the P-3 UWB SAR will be discussed. The image formation technique used will be presented, along with the RFI suppression techniques used. Of particular interest will be the technique used for the required motion compensation. Results from recent investigations using the P-3 UWB SAR data will be discussed.

A test system with four cameras in the infrared and visual spectra is under development at FFI (The Norwegian Defence Research Establishment). The system may be mounted on a jet aircraft or may be used in a land-based version. It can be used for image acquisition or for testing of automatic target recognition (ATR) algorithms. The sensors on board generate large amounts of data, and the scene may be rather cluttered or include anomalies (e.g. sun glare). This means we need algorithms which are robust, fast, able to handle complex scenes, and data from up to four sensors simultaneously. Typically, estimates of mean and covariance are needed for the processing. However, the common maximum likelihood (ML) estimates are in general too sensitive towards outliers. Algorithms based on order statistics are known to be robust and reliable. However, they are computationally very heavy. But approximations to order statistics do exist. Median of medians is one example. This is a technique where an approximation of the median of a sequence is found by first dividing the sequence in subsequences, and then calculating median (of medians) recursively. This technique can be applied for estimating the mean as well as the standard deviation. In this paper we extend this method for estimating the covariance matrix and the mean vector, and discuss the strategy with respect to robustness and computational efficiency. Applications for use in image processing and pattern recognition are given.

Reliance on Automated Target Recognition (ATR) technology is essential to the future success of Intelligence, Surveillance, and Reconnaissance (ISR) missions. Although benefits may be realized through ATR processing of a single data source, fusion of information across multiple images and multiple sensors promises significant performance gains. A major challenge, as ATR fusion technologies mature, is the establishment of sound methods for evaluating ATR performance in the context of data fusion. The Deputy Under Secretary of Defense for Science and Technology (DUSD/S&T), as part of their ongoing ATR Program, has sponsored an effort to develop and demonstrate methods for evaluating ATR algorithms that utilize multiple data source, i.e., fusion-based ATR. This paper presents results from this program, focusing on the target detection and cueing aspect of the problem. The first step in assessing target detection performance is to relate the ground truth to the ATR decisions. Once the ATR decisions have been mapped to ground truth, the second step in the evaluation is to characterize ATR performance. A common approach is to vary the confidence threshold of the ATR and compute the Probability of Detection (PD) and the False Alarm Rate (FAR) associated with each threshold. Varying the threshold, therefore, produces an empirical performance curve relating detection performance to false alarms. Various statistical methods have been developed, largely in the medical imaging literature, to model this curve so that statistical inferences are possible. One approach, based on signal detection theory, generalizes the Receiver Operator Characteristic (ROC) curve. Under this approach, the Free Response Operating Characteristic (FROC) curve models performance for search problems. The FROC model is appropriate when multiple detections are possible and the number of false alarms is unconstrained. The parameterization of the FROC model provides a natural method for characterizing both the operational environment and the ability of the ATR algorithm to detect targets. One parameter of the FROC model indicates the complexity of the clutter by characterizing the propensity for false alarms. The second parameter quantifies the separability between clutter and targets. Thus, the FROC model provides a framework for modeling and predicting ATR performance in multiple environments. This paper presents the FROC model for single sensor data and generalizes the model to handle the fusion case.

Following the tendency of increasingly using imaging sensors in military aircraft, future combat airplane pilots will need onboard artificial intelligence for aiding them in image interpretation and target designation. This document presents a system which is able to simulate high-resolution artificial SAR imagery and thereby facilitates automatic target recognition (ATR) algorithm development. The system provides a comprehensive interface that allows dynamically requesting imagery depending on the location and heading of a simulated carrier platform. Landscapes, structures and target signatures are generated based on digital terrain data and target models. An assessment of dissimilar database preparations for sensor simulation was done with respect to the different properties of SAR imaging compared to optical imaging. The document presents selected results for specific landscape elements. Post-processing algorithms for overcoming weaknesses of digital terrain databases and improving image realism are presented. Simulated sensor imagery is useful in a wide range of applications, two of which are training of ATR algorithms and sensor simulation in flight simulation environments. Using an existing ATR method as an example, the applicability and the influences of synthetic imagery on ATR training are shown and first approaches on how to validate the correctness of the imagery are explained. The integration of the system into a flight simulator in the context of interfacing and control topics serves as a concluding example.

This paper deals with several original contributions to an automatic target recognition (ATR) system, which is applied to underwater mine classification. The contributions concentrate on feature selection and object classification. First, a sophisticated filter method is designed for the feature selection. This filter method utilizes a novel feature relevance measure, the composite relevance measure (CRM). Feature relevance measures in the literature (e.g., mutual information and relief weight) evaluate the features only with respect to certain aspects. The CRM is a combination of several measures so that it is able to provide a more comprehensive assessment of the features. Both linear and nonlinear combinations of these measures are taken into account. A wide range of classifiers is able to provide satisfactory classification results by using the features selected according to the CRM. Second, in the step of object classification, an ensemble learning scheme in the framework of the Dempster-Shafer theory is introduced to fuse the results obtained by different classifiers. This fusion can improve the classification performance. We propose a reasonable construction of the basic belief assignment (BBA). The BBA considers both the reliability of the classifiers and the support of individual classifiers provided to the hypotheses about the types of test objects. Finally, this ATR system is applied to real synthetic aperture sonar imagery to evaluate its performance.

Automatic target recognition (ATR) of synthetic aperture radar (SAR) images is performed on either global or local features. The global features can be extracted and classified with high efficiency. However, they lack the reasoning capability thus can hardly work well under the extended operation conditions (EOCs). The local features are often more difficult to extract and classify but they provide reasoning capability for EOC target recognition. To combine the efficiency and robustness in an ATR system, a hierarchical fusion of the global and local features is proposed for SAR ATR in this paper. As the global features, the random projection features can be efficiently extracted and effectively classified by sparse representation-based classification (SRC). The physically relevant local descriptors, i.e., attributed scattering centers (ASCs), are employed for local reasoning to handle various EOCs like noise corruption, resolution variance, and partial occlusion. A one-to-one correspondence between the test and template ASC sets is built by the Hungarian algorithm. Then, the local reasoning is performed by evaluating individual matched pairs as well as the false alarms and missing alarms. For the test image to be recognized, it is first classified by the global classifier, i.e., SRC. Once a reliable decision is made, the whole recognition process terminates. When the decision is not reliable enough, it is passed to the local classifier, where a further classification by ASC matching is carried out. Therefore, by the hierarchical fusion strategy, the efficiency of global features and the robustness of local descriptors to various EOCs can be maintained jointly in the ATR system. Extensive experiments on the moving and stationary target acquisition and recognition data set demonstrate that the proposed method achieves superior effectiveness and robustness under both SOC and typical EOCs, i.e., noise corruption, resolution variance, and partial occlusion, compared with some other SAR ATR methods.

This paper smmnarizes a preliminary investigation into a space-based multi-class system-oriented High Range Resolution (HRR) Automatic Target Recognition (ATR) algorithm. Recent conflicts have highlighted the need to detect, track and recognize time critical moving ground targets in all-weather standoff (airborne and space-based) sensor platforms. Current sensor technologies based on Moving Target Indication (MTI) and/or Synthetic Aperture Radar (SAR) imaging are inadequate. While MT1 provides the ability to detect and track ground moving targets, it lacks target discrimination capability due to limited bandwidth. SAR has target discrimination capability for stationary targets, but target motion will blur the image. As a result, ATR of ground moving targets using HRR radar has recently received increased interest within the Air Force. The baseline algorithms presented in this paper are template-based mean squared error classifiers. The paper includes a brief introduction into recent HRR ATR efforts at the Air Force Research Laboratory Sensors Directorate's ATR Assessment Branch (AFRL/SNAA). Motivations and limitations for space- based sensors are discussed along with potential sensor fusion algorithms. The paper concludes by discussing the advantages of HRR ATR for feature aided

Proper evaluation of a pattern recognition system in the lab is paramount to its success in the field. In most commercial pattern recognition applications, such as breast cancer detection, optical character recognition, and industrial quality assurance, the boundaries and expectation of the system are well defined. This is due, at least in part, to an excellent understanding of the problem and data space for these applications. For these functions, a method for rigorous evaluation is well understood. However, the size and complexity of the data and problem spaces for automatic target recognition (ATR) systems is enormous. The consequences are that a complete understanding of how an ATR system will perform in practice is extraordinarily difficult to estimate. Thus the act of evaluating an ATR system becomes as important as its design. This paper compiles and reports the techniques used to evaluate ATR system performance. It surveys the specific difficulties associated with ATR performance estimation as well as approaches used to mitigate these obstacles.