Improved algorithm of thermal image target detection based on Yolo v4

Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain

The domain of this research is the use of computer vision methodologies in utilizing radiation for smart robotic applications for driving assistance. Radiation can be emitted by an object, reflected or transmitted. Understanding the nature and the properties of the radiation forming an image is essential in interpreting the information in that image which can then be used by a machine e.g. a smart vehicle to make a decision and perform an action. Throughout this work, different types of images are used to help a robotic vehicle make a decision and perform a certain action. This work presents three smart robotic applications; the first one deals with polarization images, the second one deals with thermal images and the third one deals with visible images. Each type of these images is formed by light (radiation) but in a way different from other types where the information embedded in an image depends on the way it was formed and how the light was generated. For polarization imaging, a direct method utilizing shading and polarization for unambiguous shape recovery without the need for nonlinear optimization routines is proposed. The proposed method utilizes simultaneously polarization and shading to find the surface normals, thus eliminating the reconstruction ambiguity. This can be useful to help a smart vehicle gain knowledge about the terrain surface geometry. Regarding thermal imaging, an automatic method for constructing an annotated thermal imaging pedestrian dataset is proposed. This is done by transferring detections from registered visible images simultaneously captured at day-time where pedestrian detection is well developed in visible images. Histogram of Oriented Gradients (HOG) features are extracted from the constructed dataset and then fed to a discriminatively trained deformable part based classifier that can be used to detect pedestrians at night. The resulting classifier was tested for night driving assistance and succeeded in detecting pedestrians even in the situations where visible imaging pedestrian detectors failed because of low light or glare of oncoming traffic. For visible images, a new feature based on HOG is proposed to be used for pedestrian detection. The proposed feature was augmented to two state of the art pedestrian detectors; the discriminatively trained Deformable Part based models (DPM) and the Integral Channel Features (ICF) using fast feature pyramids. The proposed approach is based on computing the image mixed partial derivatives to be used to redefine the gradients of some pixels and to reweigh the vote at all

Development of an online prediction system for soil organic matter and soil moisture content based on multi-modal fusion

One way to identify human faces is by using facial features or facial patterns based on visible images. However, this approach depends on the lighting conditions. In a poor illumination environment, it is difficult to recognize the face images. Thermal images do not rely on the illumination condition; thus can overcome this problem. Unlike the visible image, the thermal images lack visual information. Therefore, matching between the thermal and visible images (cross-spectral matching) is vital in face recognition. This paper proposed a phase correlation method, namely phase-only correlation (POC) and band-limited POC (BLPOC), for a cross-spectral matching of face images. First, the Fourier transform was applied to face images in both spectral domains, and the phase correlation between the two images was calculated. The experimental results showed that, from 100 visible and thermal image (acquired by FLIR C2 camera) pairs, the BLPOC resulted in significant correlation values compared to POC.

Object identification is a fundamental and important problem in computer vision. Although impressive results have been obtained for medium to large objects in visible images, the performance on small objects, including mini/micro-UAVs (Unmanned Air Vehicle) in visible and thermal images, has received less attention. Therefore, due to the different appearances of these small flying objects, the identification of these objects, especially under poor early illumination conditions is currently a challenge. To overcome this challenge, we propose an increase in the resolution of the captured images by adding high frequency details which allows to produce better visual quality. In this work, we propose a Fusion Network for Super Resolution of UAVs Visible and Thermal Images (FNSR), which aims to produce a Hight Resolution image (HR) using visible and infrared (IR) reference images (Ref). FNSR is made up of five related modules, including a multi-scale texture extractor for feature extraction from visible and thermal images, a spatial fusion module, a relevance embedding module, a features transfer block and a spatial decoder module. We introduce channel attention mechanisms to highlight the most promising features, as well as a Spatial Total Variation (STV) loss to preserve spatial information. Experimental results on image pairs (IR/Visible) show that the FNSR is capable of producing high quality super-resolved images. Compared to the State-Of-The-Art (SOTA), it presents better performances.

Images acquired by different sensors exhibit different characteristics because of the varied imaging mechanisms of sensors. The fusion of visible and infrared images is valuable for specific image applications. While infrared images provide stronger object features under poor illumination and smoke interference, visible images have rich texture features and color information about the target. This study uses dual optical fusion as an example to explore fusion detection methods at different levels and proposes a multimodal decision-level fusion detection method based on category probability sets (CPROS). YOLOv8—a single-mode detector with good detection performance—was chosen as the benchmark. Next, we innovatively introduced the improved Yager formula and proposed a simple non-learning fusion strategy based on CPROS, which can combine the detection results of multiple modes and effectively improve target confidence. We validated the proposed algorithm using the VEDAI public dataset, which was captured from a drone perspective. The results showed that the mean average precision (mAP) of YOLOv8 using the CPROS method was 8.6% and 16.4% higher than that of the YOLOv8 detection single-mode dataset. The proposed method significantly reduces the missed detection rate (MR) and number of false detections per image (FPPI), and it can be generalized.

Combination the corresponding advantage of infrared and visible images, the fusion of visible-light and near-infrared features is becoming one of the key research directions of uncontrolled face recognition. In this paper, a new face fusion recognition method based on near-infrared and visible-light imaging is proposed to make full use of the complementarity in two modalities. Firstly, for near infrared imaging, the low frequency part is retained for discrete cosine transform (DCT) coefficients and the block histogram of local binary pattern (LBP) is jointly applied to represent the discriminative features. Then, LBP features from visible faces are gotten to make up the incapable of local representation in near-infrared imaging. Finally, the three features are sent to the novel two-state fusion module to leverage the performance of face recognition. The experimental results show that the combination of DCT and LBP can extract the complementary features from infrared and visible images, and the two-stage fusion strategy lifts the robustness of the multi-modality face recognition. Especially, the recognition rate of the proposed fusion recognition algorithm has been improved significantly, with regard to the other methods based on statistical feature fusion under the circumstance of small samples.

Pine wilt disease (PWD) can cause destructive death in many species of pine trees within a short period. The recognition of infected pine trees in unmanned aerial vehicle (UAV) forest images is a key technology for automatic monitoring and early warning of pests. This paper collected UAV visible and multispectral images of Korean pines (Pinus koraiensis) and Chinese pines (P. tabulaeformis) infected by PWD and divided the PWD infection into early, middle, and late stages. With the open-source annotation tool, LabelImg, we labeled the category of infected pine trees at each stage. After coordinate-correction preprocessing of the ground truth, the Korean pine and Chinese pine datasets were established. As a means of detecting infected pine trees of PWD and determining different infection stages, a multi-band image-fusion infected pine tree detector (MFTD) based on deep learning was proposed. Firstly, the Halfway Fusion mode was adopted to fuse the network based on four YOLOv5 variants. Simultaneously, the Backbone network was initially designed as a dual branching network that includes visible and multispectral subnets. Moreover, the features of visible and multispectral images were extracted. To fully utilize the features of visible and multispectral images, a multi-band feature fusion transformer (MFFT) with a multi-head attention mechanism and a feed-forward network was constructed to enhance the information correlation between visible and multispectral feature maps. Finally, following the MFFT module, the two feature maps were fused and input into Neck and Head to predict the categories and positions of infected pine trees. The best-performing MFTD model achieved the highest detection accuracy with mean average precision values (mAP@50) of 88.5% and 86.8% on Korean pine and Chinese pine datasets, respectively, which improved by 8.6% and 10.8% compared to the original YOLOv5 models trained only with visible images. In addition, the average precision values (AP@50) are 87.2%, 93.5%, and 84.8% for early, middle, and late stages on the KP dataset and 81.2%, 92.9%, and 86.2% on the CP dataset. Furthermore, the largest improvement is observed in the early stage with 14.3% and 11.6%, respectively. The results show that MFTD can accurately detect the infected pine trees, especially those at the early stage, and improve the early warning ability of PWD.

Contactless detection of a target’s temperature of normal range (30°C ~ 150°C), which is based on its visible light images rather than infrared images, is a promising technology. Visible light imaging is primarily based on the visible light reflected from a target, rather than the thermal radiation of its own. The main challenge of contactless normal temperature detection through visible light images exists in the interference introduced by the variation of incident illumination. To solve this, a Retinex convolutional neural network (Retinex-CNN) was proposed in this paper, which was based on the convolutional neural network and Retinex algorithm. This network reduces the interference introduced by illumination variations and effectively improves the accuracy of the temperature detection based on visible light images. The temperature detection results of the Retinex-CNN shows that this network exhibits favorable generalization capability in terms of illumination variations, that is, it is still able to accurately detect temperature as a target is imaged in an illumination condition which is significantly different from that of the images in the training set. In this case, the Retinex-CNN obtains an average absolute error of 2.6°C, a value that is 6.89°C lower than that obtained by the CNN.

Deep-learning-based object detection is widely used in unmanned aerial vehicle (UAV) systems. However, existing methods can only perform the object detection algorithm on visible images. In many scenarios, the infrared image performs better than the visible image because it can represent more invisible features. This paper proposes an object detection algorithm using image fusion to fully use the advantages of both visible light images and infrared images. Moreover, we optimize and re-design the standard object detection algorithm, YOLO V2, to improve its performance on embedded platforms. By evaluating the performance of the proposed method, we found that the recognition accuracy rate is 97.43%, while the recognition accuracy rate of the visible image is 91.09% and the recognition accuracy rate of the infrared image is 91.39%. The experimental results indicate that the proposed method can increase the recognition accuracy rate of the visible image by 6.34% and increase the recognition accuracy rate of the infrared image by 6.04%.

We propose a novel human detection system with thermal and visible image sensors using a new joint bilateral filter. There is a limitation to the performance of devices which attempt to detect humans by using only one sensor. Therefore, we use two sensors: an infrared sensor and a color image sensor. To fuse the best features of both types of sensor in order to achieve superior performance in human detection, we use low-level image fusion. To carry out this image fusion process, we propose a new joint bilateral filter which is useful for fusion of the edge information in the visible image and the white region in the thermal image. This image fusion process effectively removes the regions of human shadow which is trouble of background subtraction. To achieve this effect, results from different pairs of fused images that are taken sequentially are subtracted to detect the human effectively. Experimental results show that effective and easy human detection can be achieved by the proposed algorithm.

Automatic recognition of scenes with power line wires in real life aerial images using DCT-based features

With the continuous development of science and technology, cylindrical lithium batteries, as new energy batteries, are widely used in many fields. In the production process of lithium batteries, various defects may occur. To detect the defects of lithium batteries, a detection algorithm based on convolutional neural networks is proposed in this paper. Firstly, image preprocessing is introduced on the collected lithium battery dataset. Secondly, the K-means clustering algorithm is used on the processed dataset to generate anchor boxes for lithium battery defect detection. Then the detection network YOLOv3 is trained with the given dataset. Finally, the detection network YOLOv3 is applied to output the type and location information of the defect. The experimental results show that the mean average precision (mAP) value of the detection algorithm on the lithium battery validation dataset reaches 94% and the detection speed is 25 frames per second. The proposed algorithm can effectively locate and classify the bottom defects of the lithium battery.

External insulation of coastal power grids transmitting offshore wind power faces significant threats from salt fog flashovers. Current arc monitoring and early warning technologies for flashover are severely inadequate. Research on salt fog discharge processes and determining the threshold at the flashover brink for transmission equipment external insulation is crucial for ensuring the safe operation of coastal grids delivering offshore wind power. Fiber Bragg Grating (FBG), with its advantages of compact size, excellent insulation, and fast response, enables effective discharge monitoring and identification of the critical flashover state on external insulation surfaces. In this study, FBGs were embedded at the interfaces of typical external insulation specimens, including silicone rubber plates and epoxy resin plates, to conduct contaminated AC salt fog discharge tests. Synchronized measurements of visible light images, infrared thermal images, and FBG interface temperature were conducted to investigate the discharge physical processes on silicone rubber insulating surfaces and the flashover threshold based on FBG temperature rise rate. The results indicate that discharge process can be divided into three phases: arc initiation, extension, and flashover based on the characteristics of arc visible light images. By comparing arc locations in infrared and visible light images with the corresponding FBG interface temperature rise, the arc phase criterion of FBG interface temperature rise rate and position were proposed. Furthermore, through multiple experiments, it has been found that flashover occurs when both interface temperatures reached above 4.6 × 10−2 °C/s. This study provides a novel research methodology for physical process of external insulation discharge and flashover warning in coastal salt fog environments.

Unmanned aerial vehicles (UAVs) provide rich thermal imagery data crucial for vision-related tasks such as feature extraction, image enhancement, and data synthesis. Acquiring high-resolution (HR) thermal images for UAVs remains challenging due to sensor hardware constraints. Guided super-resolution (SR) methods leverage HR visible images to enhance thermal images reconstruction from low-resolution (LR) inputs. However, these methods demonstrate limitations in restoring fine grained textural details. To solve the problem, we propose a hierarchical cross-modal mamba with multi-head reconstruction network (HCM-Mamba) for UAV thermal images SR. The framework introduces a residual state-space block (RSSB) that directly models long-range spatial dependencies, enabling robust contextual correlation learning for sparse, small targets in complex aerial scenes. To alleviate the problem of modal differences between thermal and visible images, we propose a cross-modal integration framework, which includes a hierarchical cross-modal bridging encoder (HCB-Encoder) (interaction, refinement, and enhancement). The complementary features are hierarchically aligned and integrated through a dual attention mechanism and multi-scale aggregation. In order to address the problems of structure distortion and detail loss in traditional end-to-end architectures, we propose a multi head collaborative reconstruction mechanism that ensures coherent optimization of global structure and local texture while decoupling the feature learning process. Extensive experiments have shown that HCM-Mamba has achieved significant improvements in multiple evaluation metrics, maintaining structure integrity, and restoring perceived real details under real-world UAV imaging conditions.