Studies on Image Stitching Algorithms in Machine Vision Inspection of Solar Panel

ABSTRACTReliable feature point matching is a pivotal step in the integrated 3D reconstruction of aerial and ground images. Nonetheless, the substantial viewpoint and scale differences between aerial and ground images introduce significant challenges in feature point matching. To address this issue, this study presents a robust matching method for feature points from these two types of images, employing partitioned attention aggregation along with a coarse‐to‐fine hierarchical matching approach, enabling efficient matching of large‐scale aerial‐ground images even with hardware limitations. In the coarse matching phase, region segmentation is carried out on the ground image based on its depth map, which is computed using a monocular depth estimation network. Subsequently, a graph neural network serves as an intermediary for information exchange between features, and partitioned attention aggregation is performed while adhering to the constraints imposed by the segmented image regions. During this process, feature points are grouped to form distinct subgraphs, enhancing the structural similarity between corresponding graphs when there are significant depth and viewpoint changes. This also reduces redundant connections within graph neural networks, leading to efficient feature matching. Corresponding local patches are generated from the full‐sized aerial‐ground images using the initial matches, and further feature point extraction and matching from these patches yield accurate point matches. The experimental results indicate the proposed method performs better than state‐of‐the‐art matching methods. Specifically, compared with the best‐performing method among the comparative ones, our method enhances the average number of correct matches and the average matching precision by approximately 6‐fold and 36%, respectively.

Road visual navigation relies on accurate road models. This study was aimed at proposing an improved scale-invariant feature transform (SIFT) algorithm for recovering depth information from farmland road images, which would provide a reliable path for visual navigation. The mean image of pixel value in five channels (R, G, B, S and V) were treated as the inspected image and the feature points of the inspected image were extracted by the Canny algorithm, for achieving precise location of the feature points and ensuring the uniformity and density of the feature points. The mean value of the pixels in 5×5 neighborhood around the feature point at an interval of 45o in eight directions was then treated as the feature vector, and the differences of the feature vectors were calculated for preliminary matching of the left and right image feature points. In order to achieve the depth information of farmland road images, the energy method of feature points was used for eliminating the mismatched points. Experiments with a binocular stereo vision system were conducted and the results showed that the matching accuracy and time consuming for depth recovery when using the improved SIFT algorithm were 96.48% and 5.6 s, respectively, with the accuracy for depth recovery of –7.17%-2.97% in a certain sight distance. The mean uniformity, time consuming and matching accuracy for all the 60 images under various climates and road conditions were 50%-70%, 5.0-6.5 s, and higher than 88%, respectively, indicating that performance for achieving the feature points (e.g., uniformity, matching accuracy, and algorithm real-time) of the improved SIFT algorithm were superior to that of conventional SIFT algorithm. This study provides an important reference for navigation technology of agricultural equipment based on machine vision. Keywords: scale-invariant feature transform (sift), feature matching, canny operator, energy method of feature point, farmland road, depth recovery, visual navigation DOI: 10.25165/j.ijabe.20191204.4821 Citation: Yao L J, Hu D, Yang Z D, Li H B, Qian M B. Depth recovery for unstructured farmland road image using an improved SIFT algorithm. Int J Agric & Biol Eng, 2019; 12(4): 141–147.

Image stitching refers to the combination of multiple images with a certain over lappingarea rate acquired by the image acquisition device into one image, which has a full picture of each image. Image stitching technology is an important part in the field of digital image processing. With the increasing requirements for image quality and the limited resolution of ordinary cameras, it is impossible to obtain large angle of view and clear image at the same time. In this paper, the key technologies involved in pavement crack image stitching are intensively studied, including image feature point detection and matching, internal and external camera parameter estimation and optimization, and image fusion. The work done in this paper and the main innovations include: (1)Two main feature detection and description algorithms, SIFT algorithm and ORBalgorithm, are studied. In view of the fact that the SIFT algorithm has redundancy when the Gaussian pyramid is established, this paper appropriately reduces the number of groups and maintains the number and stability of feature point detection. Combined with the advantages and disadvantages of the improved SIFT algorithm and ORB algorithm, and the performance of the two algorithms in the detection of pavement cracks, the improvedSIF'T algorithm is used as the feature point detection algorithm, and rBRlEF algorithm for describing the feature points in the ORB algorithm is used as the feature point description algorithm. The description algorithm is called the BSIF'T algorithm. Experiments show that the robustness of BSIFT algorithm is similar to that of SIFT algorithm, which is better than ORB algorithm and faster than SIF'T algorithm. (2)Image fiusion algorithms are studied, including best seam-line search and image fusion. Aiming at the ghost phenomenon that may occur in overlapping regions, this paper proposes a best seam-line search algorithm based on distance transform.

At present, there are many methods to realize the matching of specified images with features, and the basic components include image feature point detection, feature description, and image matching. Based on this background, this article has done different research and exploration around these three aspects. The image feature point detection method is firstly studied, which commonly include image edge information-based feature detection method, corner information-based detection method, and various interest operators. However, all of the traditional detection methods are involved in problems of large computation burden and time consumption. In order to solve this problem, a feature detection method based on image grayscale information-FAST operator is used in this paper, which is combined with decision tree theory to effectively improve the speed of extracting image feature points. Then, the feature point description method BRIEF operator is studied, which is a local expression of detected image feature points based on descriptors. Since the descriptor does not have rotation invariance, the detection operator is endowed by a direction that is proposed in this paper, and then the local feature description is conducted on the feature descriptor to generate a binary string array containing direction information. Finally, the feature matching machine learning method is analyzed, and the nearest search method is used to find the nearest feature point pair in Euclidean distance, of which the calculation burden is small. The simulation results show that the proposed nearest neighbor search and matching machine learning algorithm has higher matching accuracy and faster calculation speed compared with the classical feature matching algorithm, which has great advantages in processing a large number of array images captured by the light field camera.

Based on scale-invariant feature transform (SIFT) and mean seamless cloning (MVSC), an image stitching algorithm is presented, to improve the quality of the panoramic stiching image. Using SIFT algorithm to extract between benchmark images (await matched image) and follow-up images(with the baseline image match the image) of the feature points, identifying locations and directions, using 128 dimensional vector to describe features point. Using the nearest neighbor method to achieve two images feature point matching, identify overlap regions. Using SIFT algorithm to provide benchmark images and follow-up images to determine the source cloning domain and target cloning domain of the MVSC. Using the mean value coordinates to achieve the pixel to interpolat from the source cloning domain to target cloning domain. Finally, using MVSC algorithm to achieve the two images stitching. Experiment results shows that this method with regard to image rotation, perspective changes and image scaling to have a very good stitching results, stitching image is complete information, the quality of the image is high.

As the front end of visual simultaneous localization and mapping (vSLAM), the feature points matching of the visual odometry (VO) provides important robot pose judgment, while there are mismatches in the feature points matching due to inaccurate information obtained from cameras. In the process of feature points matching, VO initializes the matching of feature points through two adjacent frames of pictures, as SLAM runs, the cumulative error will gradually increase, resulting in mapping and positioning results inaccurate. Aiming to reduce the feature points matching error during initialization, an evidential fusion method for feature points matching is put forward. On the basis of descriptor similarity in Oriented FAST and Rotated BRIEF SLAM (ORB-SLAM) and pixel similarity around feature points, we treat the similarity index as an evidence, and take Belief Jensen-Shannon (BJS) divergence to measure distance between different evidences. The proposed evidential method considers the uncertainties brought by different indexes in accord with the feature points matching, and obtains suitable matching points reducing false matches through the combination of evidences. BJS divergence measure is carried out to measure the discrepancy and conflict degree among the evidences. After that, the reliability of evidence could be obtained to modify the information volume of the evidences. Finally, an example of feature points matching shows that the evidential method is feasible for feature points matching.

Ego-motion estimation, as one of the core technologies of unmanned systems, is widely used in autonomous robot navigation, unmanned driving, augmented reality and other fields. With the development of computer vision, there has been considerable interest in ego-motion estimation with visual navigation. One of the core technologies in Visual navigation is using the matching feature points between consecutive image frames to estimate pose. Since the feature-based method performed under the assumption of a static environment, it susceptive to the dynamic targets. Visual navigation in the dynamic environment has become an important research issue. This paper proposed a practical and robust features selection algorithm of visual navigation which avoids using the feature points on dynamic objects. Firstly, according to the instance segmentation of deep neural network, the objects are classified into potential dynamic and static categories. Subsequently, the matching features on the potential moving objects are used to update vehicle state respectively, meanwhile, the relevant reprojection error of other feature points on the background could be calculated. Eventually, the result of whether the target is moving or not will be judged by the reprojection error, and the features on dynamic targets are removed. To illustrate the effectiveness of the features selection method in the dynamic environment, the proposed algorithm is merged into an MSCKF based on tri-focal tensor geometry, and it has been evaluated in a public dataset. Experimental results demonstrated the effectiveness of the proposed method.

Converting tower-mounted videos from perspective to orthographic view is beneficial for their integration with maps and remote sensing images and can provide a clearer and more real-time data source for earth observation. This paper addresses the issue of low geometric accuracy in orthographic video generation by proposing a method that incorporates 3D GIS view matching. Firstly, a geometric alignment model between video frames and 3D GIS views is established through camera parameter mapping. Then, feature point detection and matching algorithms are employed to associate image coordinates with corresponding 3D spatial coordinates. Finally, an orthographic video map is generated based on the color point cloud. The results show that (1) for tower-based video, a 3D GIS constructed from publicly available DEMs and high-resolution remote sensing imagery can meet the spatialization needs of large-scale tower-mounted video data. (2) The feature point matching algorithm based on deep learning effectively achieves accurate matching between video frames and 3D GIS views. (3) Compared with the traditional method, such as the camera parameters method, the orthographic video map generated by this method has advantages in terms of geometric mapping accuracy and visualization effect. In the mountainous area, the RMSE of the control points is reduced from 137.70 m to 7.72 m. In the flat area, it is reduced from 13.52 m to 8.10 m. The proposed method can provide a near-real-time orthographic video map for smart cities, natural resource monitoring, emergency rescue, and other fields.

For the implementation of stereo image-based visual servoing algorithm in the eye-in-hand robotics applications, one of the main concerns is the accurate point feature detection and matching algorithm. Since the visual servoing is carried out in the textureless environment, the feature detection process is even more challenging. To fulfill the requirement of a robust and reliable point feature detection process, in this paper we present the novel deep learning-based algorithm. The approach based on convolutional neural networks and algorithm for detection of manufacturing entities is proposed and detected regions of interest are utilized for the improvement of the point feature detection algorithm. The proposed algorithm is experimentally evaluated in real-world settings by using wheeled nonholonomic mobile robot RAICO equipped with stereo vision system. The experimental results show the improvement of 58% in the accuracy of matched point features in the images obtained during the visual servoing process. Moreover, with the implementation of the proposed deep learning-based approach, the number of successful experimental runs has increased by 80%.

ABSTRACTDigital multimedia makes fabricating and copying much easier than ever before. Therefore, it demands efficient and automatic techniques to identify and verify the content of digital multimedia. Image authentication is such a technique to automatically identify whether the query image is a fabrication or a simple copy of the original one. In this paper, we propose a perceptual image authentication technique based on clustering and matching of feature points of images. Feature points are first extracted from images with the k‐largest local total variations and clustered using Fuzzy C‐means clustering algorithm. Then, feature points in the query image and the anchor image are matched into pairs in zigzag ordering along the diagonals of the images cluster by cluster. In the mean time, the outliers of feature points are removed. Then, the system decisions about the authenticity of images are determined by the majority vote of whether three types of distance between matched feature point pairs are larger than their respective thresholds. The three types of distance include the following: (i) histogram‐weighted distance, which is proposed in this paper; (ii) the normalized Euclidean distance; and (iii) the Hausdorff distance. The geometric transform between the query image and the anchor image is estimated, and the query image is registered. The possible tampered image blocks are detected, and the percentage of the tampered area is roughly estimated. The experimental results show the effectiveness and robustness of the proposed image authentication system. Copyright © 2011 John Wiley & Sons, Ltd.

To resolve multi-sensor remote sensing images registration, a new approach which combines feature and regional similarity measure is proposed. This approach firstly adopts SIFT algorithm to match feature points and constructs initial affine transformation function; next uses normalized cross-correlation coefficient (NCC) to define regional similarity measure, and improves NCC with initial affine transformation parameters in order to get further matching points from those unmatched in SIFT algorithm. Then, after gross filtering the effective matching feature points, builds the model for image registration by all the matching feature points. This method combines the advantages of feature matching and regional matching, and solves the problem that the multi-sensor remote sensing images are difficult to match correctly because of geometric and radiometric differences. Experiments show that this method has strong robustness, and are of higher registration accuracy than single registration method.

Compared to ordinary camera images, aerial images captured by UAVs mostly consist of small targets in the distance, with complex image information and rich detail. However, aerial images can be easily affected by factors such as lighting differences and atmospheric conditions, which leads to noticeable color variations between images. Traditional image stitching algorithms applied to drone aerial images often suffer from insufficient accuracy. To improve image stitching accuracy and efficiency, this paper proposes an improved ORB (Oriented Fast and Rotated Brief) feature extraction algorithm. The traditional ORB algorithm has shortcomings in rotation invariance, uneven distribution, and few feature points in the image to be registered. Firstly, this paper enhances the FAST feature point detection algorithm by setting an adaptive threshold based on the image's grayscale value to extract more feature point data. Secondly, the selection strategy of surrounding pixels is optimized during the feature point validation process to reduce the calculation time wasted on false feature points. Finally, the gray centroid method is used to add the main direction to the feature points found by the FAST algorithm to enhance matching accuracy under image rotation. Moreover, this paper optimizes the preprocessing stage of the algorithm to quickly solve the image transformation matrix, and thereby further improve the accuracy of image stitching. The L-M (Levenberg-Marquardt) method is combined with RANSAC to refine the obtained transformation matrix and improve the stitching accuracy. Experimental results demonstrate that the optimized algorithm applied to drone aerial image stitching achieves shorter processing times, higher matching accuracy, and better stitching performance.

Robust and rapid image dense matching is the key to large-scale three-dimensional (3D) reconstruction for multiple Unmanned Aerial Vehicle (UAV) images. However, the following problems must be addressed: (1) the amount of UAV image data is very large, but ordinary computer memory is limited; (2) the patch-based multi-view stereo-matching algorithm (PMVS) does not work well for narrow-baseline cases, and its computing efficiency is relatively low, and thus, it is difficult to meet the UAV photogrammetry’s requirements of convenience and speed. This paper proposes an Image-grouping and Self-Adaptive Patch-based Multi-View Stereo-matching algorithm (IG-SAPMVS) for multiple UAV imagery. First, multiple UAV images were grouped reasonably by a certain grouping strategy. Second, image dense matching was performed in each group and included three processes. (1) Initial feature-matching consists of two steps: The first was feature point detection and matching, which made some improvements to PMVS, according to the characteristics of UAV imagery. The second was edge point detection and matching, which aimed to control matching propagation during the expansion process; (2) The second process was matching propagation based on the self-adaptive patch. Initial patches were built that were centered by the obtained 3D seed points, and these were repeatedly expanded. The patches were prevented from crossing the discontinuous terrain by using the edge constraint, and the extent size and shape of the patches could automatically adapt to the terrain relief; (3) The third process was filtering the erroneous matching points. Taken the overlap problem between each group of 3D dense point clouds into account, the matching results were merged into a whole. Experiments conducted on three sets of typical UAV images with different texture features demonstrate that the proposed algorithm can address a large amount of UAV image data almost without computer memory restrictions, and the processing efficiency is significantly better than that of the PMVS algorithm and the matching accuracy is equal to that of the state-of-the-art PMVS algorithm.

Most of the existing visual SLAM schemes rely solely on point or line features to estimate camera trajectory. In some scenes such as texture missing or motion blurring, it’s difficult to find a sufficient number of reliable features, resulting in low positioning accuracy. To extract more features, an RPL-SLAM solution is proposed to extract point features and line features respectively. Ulteriorly, the depth information of RGBD image is used to restore the 3D information of point and line features, improving the accuracy of camera track positioning. RPL-SLAM scheme mainly includes three modules: tracking, local mapping and loop detection. Tracking module extends the application of line feature on the basis of point feature extraction and matching. A SLD line segment extraction algorithm which can eliminate the micro segments and a DBM segment matching algorithm based on word bag are proposed respectively. These two algorithms improve the matching efficiency while ensuring the matching accuracy, and effectively track and locate the camera of each frame. In the local mapping and loop detection module, the Plucker coordinates are applied to express the spatial line and define the re-projection error of the straight line, so that the back-end optimization error model of point-line fusion is unified to solve the instability problem in optimization. RPL-SLAM is tested on TUM RGBD and ICL-NUIM data set respectively, and compared with ORB-SLAM2. The result shows that RPL-SLAM can effectively improve the accuracy of pose estimation and map reconstruction while maintaining real-time performance by fusing point-line features with depth images.

Feature Point (FP) detection is a fundamental step in computer vision tasks. Although FP detectors are mostly designed to support Low Dynamic Range (LDR) images as input, interest in High Dynamic Range (HDR) images has increased recently due to their higher precision to register overexposed and underexposed areas in an image. As the detection of FPs is strongly dependent on the illumination of the scene, HDR images have the potential to be more robust than LDR images during FP detection. Known FP detectors, however, do not use the full potential of HDR images. In addition, few works have evaluated the performance of HDR images in this context. In this paper, we propose a modification of FP detectors aiming to improve FP detection on HDR images. To this end, we design a local mask based on the Coefficient of Variation (CV) of sets of pixels, creating thus a new step in FP detection. We compare our approach with popular FP detection methods using a standard evaluation metric, Repeatability Rate (RR) of FPs, and also Uniformity as a proposed new criterion. A dataset of images from scenes affected by camera transformations and substantial illumination changes was used as input. Experimental results show that our proposed algorithms give better Uniformity and RR in most HDR images from the dataset when compared to standard FP detectors. Moreover, they indicate that HDR images present a great potential to be explored in applications that rely on FP detection.