UAV Low Altitude Photogrammetry for Power Line Inspection

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

The maintenance of high-voltage power lines rights-of-way due to vegetation intrusions is important for electric power distribution companies for safe and secure delivery of electricity. However, the monitoring becomes more challenging if power line corridor (PLC) exists in complex environment such as mountainous terrains or forests. To overcome these challenges, this paper aims to provide an automated method for extraction of individual pylons and monitoring of vegetation near the PLC in hilly terrain. The proposed method starts off by dividing the large dataset into small manageable datasets. A voxel grid is formed on each dataset to separate power lines from pylons and vegetation. The power line points are converted into a binary image to get the individual spans. These span points are used to find nearby vegetation and pylon points and individual pylons and vegetation are further separated using a statistical analysis. Finally, the height and location of extracted vegetation with reference to power lines are estimated and separated into danger and clearance zones. The experiment on two large Australian datasets shows that the proposed method provides high completeness and correctness of 96.5% and 99% for pylons, respectively. Moreover, the growing vegetation beneath and around the PLC that can harm the power lines is identified.

Power line mapping technique using all-terrain mobile laser scanning

A power line segmentation model in aerial images based on an efficient multibranch concatenation network

Power lines are critical infrastructure components in power grid systems. Strand breakage is a kind of serious defect of power lines that can directly impact the reliability and safety of power supply. Due to the slender morphology of power lines and the difficulty in acquiring sufficient sample data, strand breakage detection remains a challenging task. Moreover, power grid corporations prefer to detect these defects on-site during power line inspection using unmanned aerial vehicles (UAVs), rather than transmitting all of the inspection data to the central server for offline processing which causes sluggish response and huge communication burden. According to the above challenges and requirements, this paper proposes a novel method for detecting broken strands on power lines in images captured by UAVs. The method features a multi-stage light-weight pipeline that includes power line segmentation, power line local image patch cropping, and patch classification. A power line segmentation network is designed to segment power lines from the background; thus, local image patches can be cropped along the power lines which preserve the detailed features of power lines. Subsequently, the patch classification network recognizes broken strands in the image patches. Both the power line segmentation network and the patch classification network are designed to be light-weight, enabling efficient online processing. Since the power line segmentation network can be trained with normal power line images that are easy to obtain and the compact patch classification network can be trained with relatively few positive samples using a multi-task learning strategy, the proposed method is relatively data efficient. Experimental results show that, trained on limited sample data, the proposed method can achieve an F1-score of 0.8, which is superior to current state-of-the-art object detectors. The average inference speed on an embedded computer is about 11.5 images per second. Therefore, the proposed method offers a promising solution for conducting real-time on-site power line defect detection with computing sources carried by UAVs.

Rockfall event is common on rocky terrain, and it threatens people. Recent advances in remote sensing technologies allow us to model the terrain with high accuracy and resolution. In this study, we show the application of modeling a rockfall site in three-dimensional (3D) using an unmanned aerial vehicle. The acquisition of information about discontinuities in the rock masses using 3D point clouds is vital to characterize rockfall dynamics. We obtained 3D spatial information and point clouds of a rocky slope from acquired images. We extracted 10 discontinuity sets from the 3D model as they have significant effects on the failure mechanism. Details in mapping discontinuities have improved significantly with the help of point clouds. The results are beneficial for modeling the terrain of inaccessible areas with high resolution and obtaining information from those areas. This paper shows how can point cloud analysis be applied to characterize discontinuity sets.

Recently, intelligent unmanned aerial vehicles (UAVs) have shown great advantages of flexibility and productivity in power line inspection, wherein robust detection of power lines from aerial images for automatic power line following navigation is required. However, identifying power lines accurately from a cluttered background is challenging due to the limited resolution of onboard cameras and the noisy environment. In this letter, we propose a novel power line detection method, denoted by CODNet, for the application of UAV navigation. Unlike existing works, the proposed method can extract features of power lines from cluttered backgrounds automatically and predict centers and orientations of power lines in the scene simultaneously. Besides, we introduce a new clustering method to summarize the average location and orientation of detected power lines as a guide for the automatic navigation of UAVs. Finally, experimental results demonstrate both the effectiveness and the superiority of the CODNet.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

As a new method for power line inspection, unmanned aerial vehicle (UAV)-based inspection is efficient and low-cost. Since the flight speed of UAV is fast, target auto-tracking and photographing of the airborne equipments including visual-light camera, infrared thermal imager and ultraviolet imager, is a very important and useful function. In this work, a power line inspection system suitable for UAV-based inspection is built, and based on this system, an auto-tracking algorithm used for UAV power line inspection is proposed. Theories including coordinate transformation of three coordinate systems (geodetic coordinate system, geocentric rectangular coordinate system and NEU coordinate system) and spherical approximation method of treating arc length as chord length are applied to this auto-tracking algorithm. Through this algorithm, airborne cameras' shooting angle can be automatically modified to keep tracking and photographing power line during the inspection, using the real-time captured GPS data of UAV. The requirements on GPS positioning accuracy for the algorithm are also discussed in this work.

Deep learning on a three dimensional (3D) point cloud has attracted a great deal of attention. In this paper, we propose a binarization method for latent representation in a 3D point cloud auto-encoder. The proposed method is based on z-score normalization and deterministic binarization. In the experiment, the proposed method increased data compression efficiency 32-fold, while slightly decreasing the generalization ability of the point cloud reconstruction.

Stereo cameras, laser rangers and other time-of-flight ranging devices are utilized with increasing frequency as they can provide information in the 3D plane. The ability to perform real-time registration of the 3D point clouds obtained from these sensors is important in many applications. However, the tasks of locating accurate and dependable correspondences between point clouds and registration can be quite slow. Furthermore, any algorithm must be robust against artifacts in 3D range data as sensor motion, reflection and refraction are commonplace. The SIFT feature detector is a robust algorithm used to locate features, but cannot be extended directly to the 3D range point clouds since it requires dense pixel information, whereas the range voxels are sparsely distributed. This paper proposes an approach which enables SIFT application to locate scale and rotation invariant features in 3D point clouds. The algorithm then utilizes the known point correspondence registration algorithm in order to achieve real-time registration of 3D point clouds.KeywordsPoint CloudScale Invariant Feature TransformIterative Close Point AlgorithmScale Invariant Feature Transform FeatureRANSAC AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Surface defect identification is a crucial task in many manufacturing systems, including automotive, aircraft, steel rolling, and precast concrete. Although image-based surface defect identification methods have been proposed, these methods usually have two limitations: images may lose partial information, such as depths of surface defects, and their precision is vulnerable to many factors, such as the inspection angle, light, color, noise, etc. Given that a three-dimensional (3D) point cloud can precisely represent the multidimensional structure of surface defects, we aim to detect and classify surface defects using a 3D point cloud. This has two major challenges: (i) the defects are often sparsely distributed over the surface, which makes their features prone to be hidden by the normal surface and (ii) different permutations and transformations of 3D point cloud may represent the same surface, so the proposed model needs to be permutation and transformation invariant. In this paper, a two-step surface defect identification approach is developed to investigate the defects’ patterns in 3D point cloud data. The proposed approach consists of an unsupervised method for defect detection and a multi-view deep learning model for defect classification, which can keep track of the features from both defective and non-defective regions. We prove that the proposed approach is invariant to different permutations and transformations. Two case studies are conducted for defect identification on the surfaces of synthetic aircraft fuselage and the real precast concrete specimen, respectively. The results show that our approach receives the best defect detection and classification accuracy compared with other benchmark methods.

Abstract. A series of panoramic images are usually used to generate a 720° panorama image. Although panoramic images are typically used for establishing tour guiding systems, in this research, we demonstrate the potential of using panoramic images acquired from multiple sites to create not only 720° panorama, but also three-dimensional (3D) point clouds and 3D indoor models. Since 3D modeling is one of the goals of this research, the location of the panoramic sites needed to be carefully planned in order to maintain a robust result for close-range photogrammetry. After the images are acquired, panoramic images are processed into 720° panoramas, and these panoramas which can be used directly as panorama guiding systems or other applications. In addition to these straightforward applications, interior orientation parameters can also be estimated while generating 720° panorama. These parameters are focal length, principle point, and lens radial distortion. The panoramic images can then be processed with closerange photogrammetry procedures to extract the exterior orientation parameters and generate 3D point clouds. In this research, VisaulSFM, a structure from motion software is used to estimate the exterior orientation, and CMVS toolkit is used to generate 3D point clouds. Next, the 3D point clouds are used as references to create building interior models. In this research, Trimble Sketchup was used to build the model, and the 3D point cloud was added to the determining of locations of building objects using plane finding procedure. In the texturing process, the panorama images are used as the data source for creating model textures. This 3D indoor model was used as an Augmented Reality model replacing a guide map or a floor plan commonly used in an on-line touring guide system. The 3D indoor model generating procedure has been utilized in two research projects: a cultural heritage site at Kinmen, and Taipei Main Station pedestrian zone guidance and navigation system. The results presented in this paper demonstrate the potential of using panoramic images to generate 3D point clouds and 3D models. However, it is currently a manual and labor-intensive process. A research is being carried out to Increase the degree of automation of these procedures.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Estimation of plant parameters is prerequisite for fast and precise management of tomato crops on commercial farm level. Destructive estimation methods may be accurate but not suitable when large farm size, spatio-temporal variability or short period of time to obtain information are considered. Therefore, this study was aimed to estimate leaf area (LA) in mature tomato plants from reconstructed geometric plant model based on 3-dimensional (3D) point cloud. A 2D laser scanner was placed on a programable linear conveyor system to move at controlled forward speed. To achieve a high density 3D point cloud of whole plant, close range (< 2 m) scanning was performed at 10 mm/s forward speed. Plant parameters capturing points per plant (PPP), plant height, plant volume, and LA were extracted from the 3D point clouds. Destructive method was utilized in laboratory for reference LA data collection, whereas other required reference data (plant height and canopy area) were measured in-situ. Linear regression models were developed and cross validated to estimate LA using variables from the 3D point clouds and reference LA. Among the models, plant height showed strong correlation with reference LA (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.90, RMSE = 40 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and followed by PPP (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.89, RMSE = 45 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ), voxel grid approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.56, RMSE = 58 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and slicing approach (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.44, RMSE = 70 cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ). Results indicate that mobile laser scanning could be a reliable non-destructive LA estimation tool for tomato and may be adapted by the commercial tomato growers for profitable management decisions.