Automatic target recognition and tracking in forward-looking infrared image sequences with a complex background

Forward Looking Infrared (FLIR) automatic target recognition (ATR) systems depend upon the capacity of the atmosphere to propagate thermal radiation over long distances. To date, not much research has been conducted on analyzing and mitigating the effects of the atmosphere on FLIR ATR performance, even though the atmosphere is often the limiting factor in long-range target detection and recognition. The atmosphere can also cause frame-to-frame inconsistencies in the scene, affecting the ability to detect and track moving targets. When image quality is limited by turbulence, increasing the aperture size or improving the focal plane array cannot improve ATR performance. Traditional single frame image enhancement does not solve the problem. A new approach is described for reducing the effects of turbulence. It is implemented under a lucky-region-imaging framework using short integration time and spatial domain processing. It is designed to preserve important target and scene structure. Unlike previous Fourier-based approaches originating from the astronomical community, this new approach is intended for real-time processing from a moving platform, with ground as the background. The system produces a video stream with minimal delay. A new video quality measure (VQM_turb) is presented for quantifying the success of turbulence mitigation on real data where no reference imagery is available. The VQM_turb is the core of the innovation because it allows a wide range of algorithms to be quantitatively compared. An algorithm can be chosen, and then tuned, to best-fit available processing power, latency requirements, scenarios and sensor characteristics.

Over the last two decades, researchers investigated various approaches for detection and classification of targets in forward looking infrared (FLIR) imagery using correlation based techniques. In this paper, a novel technique is proposed to recognize and track single as well as multiple identical and/or dissimilar targets in real life FLIR sequences using a combination of extended maximum average correlation height (EMACH) and polynomial distance classifier correlation filter (PDCCF). The EMACH filters are used for the detection stage and PDCCF filter is used for the classification stage for improving the detection and discrimination capability. The EMACH and PDCCF filters are trained a priori using target images with expected size and orientation variations. In the first step, the input scene is correlated with all the detection filters (one for each desired or expected target class) and the resulting correlation outputs are combined. The regions of interest (ROI) are selected from the input scene based on the regions with higher correlation peak values in the combined correlation output. In the second step, PDCCF filter is applied to these ROIs to identify target types and reject clutters/backgrounds based on a distance measure and a threshold. Moving target detection and tracking is accomplished by applying this technique independently to all incoming image frames. Independent tracking of target(s) from one frame to the other allows the system to handle complicated situations such as a target disappearing in few frames and then reappearing in later frames. This method has been found to yield robust performance for challenging FLIR imagery in terms of faster and accurate detection and classification as well as tracking of the targets.

This paper proposes a method to detect objects of arbitrary poses and sizes from a complex forward looking infrared (FLIR) image scene exploiting image correlation technique along with the preprocessing of the scene using a class of morphological operators. This presented automatic target recognition (ATR) algorithm consists of two steps. In the first step, the image is preprocessed, by employing morphological reconstruction operators, to remove the background as well as clutter and to intensify the presence of both low or high contrast targets. This step also involves in finding the possible candidate target regions or region of interests (ROIs) and passing those ROIs to the second step for classification. The second step exploits template-matching technique such as minimax distance transform correlation filter (MDTCF) to identify the true target from the false alarms in the pre-selected ROIs after classification. The MDTCF minimizes the average squared distance from the filtered true-class training images to a filtered reference image while maximizing the mean squared distance of the filtered false-class training images to this filtered reference image. This approach increases the separation between the false-class correlation outputs and the true-class correlation outputs. Classification is performed using the squared distance of a filtered test image to the chosen filtered reference image. The proposed technique has been tested with real life FLIR image sequences supplied by the Army Missile Command (AMCOM). Experimental results, obtained with these real FLIR image sequences, illustrating a wide variety of target and clutter variability, demonstrate the effectiveness and robustness of the proposed method.

Automatic target recognition (ATR) is a challenging task for several computer vision applications. It requires efficient, accurate, and robust methods for target detection and target identification. Deep learning has shown great success in many computer vision applications involving color RGB images. However, the performance of these networks in ATR with infrared sensor data needs further investigation. In this paper, we propose a multistage automatic target detection and recognition (ATDR) system that performs both target detection and target classification on infrared (IR) imagery using deep learning. Our system processes large IR image frames where targets take <1 % of the total number of pixels. First, we train a state-of-the-art object detector you only look once (YOLO) to localize all potential targets in the input image frame. Then, we train a convolutional neural network (CNN) to identify these detections as targets or false alarms. In this second phase, we adapt and analyze the performance of three CNN architectures: a compact and fully connected CNN, VGG16 with batch normalization, and a wide residual neural network (WRN). We also explore the use of a loss function that optimizes directly the area under the receiver operating characteristic (ROC) curve (AUC), and adapt it to our ATR application. To enhance the robustness of the proposed ATR to perturbation and variations introduced during the detection stage, we train our CNN classifiers on automatically detected targets using YOLO, in addition to ground truth bounding boxes and apply selected data augmentation techniques. To simulate real testing environments, where the spatial location of the targets within the image frame is unknown, only YOLO-detected boxes are used during validation. We evaluate our ATDR on a real benchmark dataset that includes different vehicles captured at different resolutions. Our experiments have shown that YOLO can detect most of the targets at the expense of generating a high number of false alarms. We show that the VGG-16 network with batch normalization, which is the best performing model, can correctly identify the classes of the targets, as well as classify the majority of YOLO’s false detections into an additional nontarget class. We also show that the proposed training modification to optimize an AUC-based loss function for ATR proved to be advantageous mainly in identifying difficult targets.

Performance of automatic target recognition (ATR) systems depends on numerous factors including the mission description, operating conditions, sensor modality, and ATR algorithm itself. Performance prediction models sensitive to these factors could be applied to ATR algorithm design, mission planning, sensor resource management, and data collection design for algorithm verification. Ideally, such a model would return measures of performance (MOPs) such as probability of detection (Pd), correct classification (Pc), and false alarm (Pfa), all as a function of the relevant predictor variables. Here we discuss the challenges of model-based and data-based approaches to performance prediction, concentrating especially on the synthetic aperture radar (SAR) modality. Our principal conclusion for model-based performance models (predictive models derived from fundamental physics- and statistics-based considerations) is that analytical progress can be made for performance of ATR system components, but that performance prediction for an entire ATR system under realistic conditions will likely require the combined use of Monte Carlo simulations, analytical development, and careful comparison to MOPs from real experiments. The latter are valuable for their high-fidelity, but have a limited range of applicability. Our principal conclusion for data-based performance models (that fit empirically derived MOPs) offer a potentially important means for extending the utility of empirical results. However, great care must be taken in their construction due to the necessarily sparse sampling of operating conditions, the high-dimensionality of the input space, and the diverse character of the predictor variables. Also the applicability of such models for extrapolation is an open question.

For new military aircraft a specification of sensor characteristics and performance is required at an early stage in the design cycle, well before testing of a prototype. In the early days of military aviation the Johnson criteria [1] were used to determine the sensor resolution required for target recognition by a human. In the present day sensor data are processed by computer using various Automatic Target Recognition (ATR) algorithms. However there is no accepted method for predicting the sensor resolution and SNR required for reliable ATR and hence there is risk that any chosen sensor may fail to support the required ATR performance. This paper reports a study into the use of publicly-available CAD models for aircraft to address this requirement. The study considers the worst-case confusion between two views of 15 different aircraft types. For simplicity only rotations by an angle θ about the Z (vertical) axis are considered. Firstly the sensor resolution and noise level is fixed. Then for each aircraft type and view angle an ensemble of synthetic silhouettes are generated. Using these ensembles, a-posteriori distributions of 5 standard scale-invariant shape features (eccentricity, orientation, solidity, circularity and bounding box aspect ratio) are estimated for each view angle θ. The performance of ATR at the given resolution and noise level is predicted by estimating the Bayes Error Rate [2] when deciding between each aircraft type and the 14 non-matching types using these features. The worst-case confusion in terms of erroneous aircraft type and view angle is identified. The sensor resolution is then changed and the above process repeated to investigate the effect of varying sensor resolution on performance. As expected, high sensor resolution leads to low probability of misclassification, even in the worst-case. Reduction in resolution and increasing noise level causes the Bayes Error Rate to rise quickly. The Bayes Error Rate gives a fundamental limit to the reliability of classification, irrespective of the actual type of classification algorithm used. The predictions from the model are confirmed by testing against a standard classifier for specific discrimination examples. Further development of the approach presented is expected to yield a method for specifying sensor resolution requirements for specific ATR problems.

We propose a method for automatic target detection and tracking in forward-looking infrared (FLIR) image sequences. We use morphological connected operators to extract and track targets of interest and remove undesirable clutter. The design of these operators is based on general size, connectivity, and motion criteria, using spatial intraframe and temporal interframe information. In a first step, an image sequence is filtered on a frame-by-frame basis to remove background and residual clutter and to enhance the presence of targets. Detections extracted from the first step are passed to a second step for motion-based analysis. This step exploits the spatiotemporal correlation of the data, stated in terms of a connectivity criterion along the time dimension. The proposed method is suitable for piplined implementation or time progressive coding/transmission, since only a few frames are considered at a time. Experimental results, obtained with real FLIR image sequences, illustrating a wide variety of target and clutter variability, demonstrate the effectiveness and robustness of the proposed method.

We present a performance analysis of an indexing module contained in an automatic target recognition (ATR) system developed at Worcester Polytechnic Institute (WPI). The linear signal decomposition/direction of arrival (LSD/DOA) technique provides a low computational-cost indexing function for pose in ATR applications. The LSD/DOA technique forms a reduced-size, essential-information object data set which implicitly incorporates target and sensor variation specific data. To control the computational costs, the system consists of two independent components: a computationally expensive off-line component which forms the object representation and a computationally inexpensive on-line object recognition component. The size of the stored data set may also be adjusted providing a means to trade off complexity versus performance. Synthetic aperture radar (SAR) data collected as part of the Moving and Stationary Target Acquisition and Recognition (MSTAR) program has been released to the public, providing an opportunity for ATR performance assessment against a standard high quality data set. We present the results of a number of simulated tests against the MSTAR public database to demonstrate the performance of the LSD/DOA index module on a given target along with the performance of the WPI ATR itself as a function of the depression angle and target configuration.

In distributed underwater signal processing for area surveillance and sanitization during regional conflicts, it is often necessary to transmit raw imagery data to a remote processing station for detection-report confirmation and more sophisticated automatic target recognition (ATR) processing. Because of the limited bandwidth available for wireless transmission, image compression is of paramount importance. Furthermore, it is equally crucial that image coding algorithms be evaluated according to some meaningful criteria. Instead of assessing the performance of image compression algorithms in terms of peak signal-to-noise ratio (PSNR) or normalized mean-squared error (NMSE), we resort to a more meaningful performance metric that reflects human and operational factors-ATR performance. We develop a novel image compression algorithm that achieves the minimal information state by a combination of subimage-specific transformation, principal component analysis, and vector quantization (VQ). We quantify the performance of image coding by extracting key parameters or features from the original and reconstructed images and by comparing the ATR performances using separate test sets that contain both mines and mine-like clutter. We achieve a compression ratio of up to 57:1 with minimal sacrifice in P/sub D/ and P/sub FA/.

In recent years, convolutional neural network (CNN) has been increasingly considered as a promising technology for military and homeland security applications. The fusion of CNN and Support vector machine (SVM), a popular traditional machine learning approach, has received intensive attention in the field of synthetic aperture radar (SAR) automatic target recognition (ATR). This paper, firstly, discusses the effects of some preprocessing and image enhancement methods on the performance of SAR ATR, starting with the pre-trained AlexNet in a transfer-learning based approach. Secondly, the architecture of AlexNet is modified to form a new model suitable for SAR ATR. Finally, we propose a hybrid model associated with the success of the learning feature of our CNN model and the ability of SVM to process high-dimensional dataset effectively. To evaluate the proposed method, experiments are performed on the Moving and Stationary Target Acquisition and Recognition (MSTAR) public database. The comparative results demonstrate that these preprocessing and enhancement methods prior to the deep-learning process are not necessary since the feature representation ability of AlexNet is already powerful. Furthermore, experimental results on the benchmark MSTAR dataset illustrate the effectiveness of the proposed new model. On classification of ten-class targets, the commonly used translation augmentation for training data has been performed. By combining the CNN and SVM, the classification accuracy percentages can be slightly improved for our proposed new model.

Automatic target recognition (ATR) is an important part for many computer vision applications. Despite the extensive research which has been carried out in this area for many years, there is no ATR system which performs well on all applications. Recently, different object recognition frameworks have been proposed which yield a high performance in baseline databases. However, our experiments showed that they can fail in real-world scenarios, when dealing with a limited number of data samples. In this paper, we propose a new ATR system, based on deep convolutional neural network (DCNN), to detect the targets in forward looking infrared (FLIR) scenes and recognize their classes. In our proposed ATR framework, a fully convolutional network is trained to map the input FLIR imagery data to a fixed stride correspondingly-sized target score map. The potential targets are identified by applying a threshold on the target score map. Finally, the corresponding regions centered at these target points are fed to a DCNN to classify them into different target types while at the same time rejecting the false alarms. The proposed architecture achieves a significantly better performance in comparison with that of the state-of-the-art methods on two large FLIR image databases.

Many applications reported in artificial neural networks are associated with military problems. This paper reviews concepts associated with the processing of military data to find and recognize targets -- automatic target recognition (ATR). A general-purpose automatic target recognition system does not exist. The work presented here is demonstrated on military data, but it can only be considered proof of principle until systems are fielded and proven `under- fire.' ATR data can be in the form of nonimaging one-dimensional sensor returns, such as ultra-high range-resolution radar returns (UHRR) for air-to-air automatic target recognition and vibration signatures from a laser radar for recognition of ground targets. The ATR data can be two-dimensional images. The most common ATR images are infrared, but current systems must also deal with synthetic aperture radar (SAR) images. Finally, the data can be three-dimensional, such as sequences of multiple exposures taken over time from a nonstationary world. Targets move, as do sensors, and that movement can be exploited by the ATR. Hyperspectral data, which are views of the same piece of the world looking at different spectral bands, is another example of multiple image data; the third dimension is now wavelength and not time. ATR system design usually consists of four stages. The first stage is to select the sensor or sensors to produce the target measurements. The next stage is the preprocessing of the data and the location of regions of interest within the data (segmentation). The human retina is a ruthless preprocessor. Physiologically motivated preprocessing and segmentation is demonstrated along with supervised and unsupervised artificial neural segmentation techniques. The third design step is feature extraction and selection: the extraction of a set of numbers which characterize regions of the data. The last step is the processing of the features for decision making (classification). The area of classification is where most ATR related neural network research has been accomplished. The relation of neural classifiers to Bayesian techniques is emphasized along with the more recent use of feature sequences to enhance classification. © (1995) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

The 3-D automatic target recognition (ATR) has many advantages over its 2-D counterpart, but there are several constraints in the context of small low-cost unmanned aerial vehicles (UAVs). These limitations include the requirement for active rather than passive monitoring, high equipment costs, sensor packaging size, and processing burden. We, therefore, propose a new structure from motion (SfM) 3-D ATR architecture that exploits the UAV's onboard sensors, i.e., the visual band camera, gyroscope, and accelerometer, and meets the requirements of a small UAV system. We tested the proposed 3-D SfM ATR using simulated UAV reconnaissance scenarios and found that the performance was better than classic 3-D light detection and ranging (LIDAR) ATR, combining the advantages of 3-D LIDAR ATR and passive 2-D ATR. The main advantages of the proposed architecture include the rapid processing, target pose invariance, small template size, passive scene sensing, and inexpensive equipment. We implemented the SfM module under two keypoint detection, description and matching schemes, with the 3-D ATR module exploiting several current techniques. By comparing SfM 3-D ATR, 3-D LIDAR ATR, and 2-D ATR, we confirmed the superior performance of our new architecture.

Multiview synthetic aperture radar (SAR) images contain much richer information for automatic target recognition (ATR) than a single-view one. It is desirable to establish a reasonable multiview ATR scheme and design effective ATR algorithm to thoroughly learn and extract that classification information, so that superior SAR ATR performance can be achieved. Hence, a general processing framework applicable for a multiview SAR ATR pattern is first given in this paper, which can provide an effective approach to ATR system design. Then, a new ATR method using a multiview deep feature learning network is designed based on the proposed multiview ATR framework. The proposed neural network is with a multiple input parallel topology and some distinct deep feature learning modules, with which significant classification features, the intra-view and inter-view features existing in the input multiview SAR images, will be learned simultaneously and thoroughly. Therefore, the proposed multiview deep feature learning network can achieve an excellent SAR ATR performance. Experimental results have shown the superiorities of the proposed multiview SAR ATR method under various operating conditions.

Proper evaluation of a pattern recognition system in the lab is paramount to its success in the field. In most commercial pattern recognition applications, such as breast cancer detection, optical character recognition, and industrial quality assurance, the boundaries and expectation of the system are well defined. This is due, at least in part, to an excellent understanding of the problem and data space for these applications. For these functions, a method for rigorous evaluation is well understood. However, the size and complexity of the data and problem spaces for automatic target recognition (ATR) systems is enormous. The consequences are that a complete understanding of how an ATR system will perform in practice is extraordinarily difficult to estimate. Thus the act of evaluating an ATR system becomes as important as its design. This paper compiles and reports the techniques used to evaluate ATR system performance. It surveys the specific difficulties associated with ATR performance estimation as well as approaches used to mitigate these obstacles.