Real-Time Obstacle Detection Based on Monocular Vision for Unmanned Surface Vehicles

Object tracking is an important basis for the autonomous navigation of unmanned surface vehicles. However, several problems still must be addressed for a wide applicating of object tracking in unmanned surface vehicles. First, if multiple objects of the same classification exist in the same field of view, then stable extraction of an object is difficult. Second, in an environment with a complex background and large changes in object shape, the tracking accuracy is low, and object tracking errors and tracking loss can easily occur. Third, much time is required to detect a high-resolution real-time video stream, not meeting the delay requirement of the photoelectric servo stable tracking. To resolve these problems, this paper proposes an object detection-tracking algorithm based on a radar-photoelectric system. The algorithm combines an object detection algorithm with an object tracking algorithm and involves the following steps. First, a first-frame object extraction algorithm is used to extract the tracking object from the first frame. Second, a region of interest (ROI)-prediction algorithm is used to predict ROIs and detect objects in these ROIs. This algorithm can effectively solve the above problems in marine tests. When multiple objects of the same classification exist in the same field of view, the algorithm can extract the radar-guided object stably. When faced with a complex background and a large change in object shape, the algorithm substantially improves the accuracy and robustness of object tracking. Compared with the conventional object detection algorithm, the time consumption of this algorithm is reduced by 25.8%.

There are many unknown obstacles in the sea, so the autonomous navigation of unmanned surface vehicle needs to avoid them as soon as possible even under the condition of low control ability of the controller. To solve the problem, by combining the dynamic collision avoidance algorithm and tracking control, a dynamic collision avoidance control method in the unknown ocean environment is presented. In this article, in consideration of the unknown ocean environment and real-time dynamic obstacle avoidance problem, the collision avoidance controller using a velocity resolution method and backstepping tracking controller based on unmanned surface vehicle maneuvering motion model is designed. Simulation results show that the method is effective and accurate and can provide the reference for the unmanned surface vehicle intelligent collision avoidance control technology.

Robust detection and recognition of water surfaces are critical for autonomous navigation of unmanned surface vehicles (USVs), since any none-water region is likely an obstacle posing a potential danger to the sailing vehicle. A novel water region visual detection method is proposed in this paper. First, the input image pixels are clustered into different regions and each pixel is assigned a label tag and a confidence value by adaptive multistage segmentation algorithm. Then the resulting label map and associated confidence map are fed into a convolutional neural network (CNN) as training samples to train the network online. Finally, the online trained CNN is used to segment the input image again but with greater precision and stronger robustness. Compared with other deep-learning image segmentation algorithms, the proposed method has two advantages. Firstly, it dispenses with the need of manual labeling training samples which is a costly and painful task. Secondly, it allows real-time online training for CNN, making the network adaptive to the navigational environment. Another contribution of this work relates to the training process of neuro network. An effective network training method is designed to learn from the imperfect training data. We present the experiments in the lake with a various scene and demonstrate that our proposed method could be applied to recognize the water region in the unknown navigation environment automatically.

SummaryEffective and intelligent path planning algorithms designed for operation in a dynamic marine environment are essential for the safe operation of unmanned surface vehicles (USVs). Most of the current research deals with the ‘dynamic problem’ by basing solutions on the nonpractical assumption that each USV has a robust communication channel to obtain essential information such as position and velocity of marine vehicles. In this paper, a Kalman filter‐based predictive path planning algorithm is proposed. The algorithm has been designed to predict the trajectories of moving ships, and the USV's own position in real time and accordingly assesses collision risk. For path planning, a weighted fast marching square method is proposed and developed to search for the optimal path. The path can be optimised for mission requirements such as minimum distance to travel and the most safety path by adjusting weighting parameters. The proposed algorithm has been validated using a number of simulations that include practical environmental aspects. The results show that the algorithms can sufficiently deal with complex traffic environments and that the generated practical path is suited for both unmanned and manned vessels. Copyright © 2015 John Wiley & Sons, Ltd.

Water segmentation is essential for autonomous navigation of Unmanned Surface Vehicles (USV) in inland waterways. However, most of the existing methods are suitable for maritime environments. Due to their poor real-time performance and a high false-positive rate of water surface reflection and water-sky interference in inland waterways, we propose a radarassisted real-time water segmentation network (RaRWS) to solve the above problems. We obtain pseudo-waterlines based on radar data by deleting points, fitting them multiple times, and generating a radar mask. While simplifying the encoder backbone network, we use the Attention Refinement Module (ARM) to fuse radar masks to improve detection and waterside accuracy. In addition, a Feature Fusion Module (FFM) is introduced to help the decoder fuse high- and low-level features and further fuse radar data. RaRWS is tested in both normal and bad weather. The results show that RaRWS can achieve higher performance compared with the current state-of-the-art methods (F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> is above 99%), while gaining real-time performance (41.3fps).

The identification and classification of obstacles in navigable and non-navigable regions, as well as the measurement of distances, are crucial topics of investigation in the field of autonomous navigation for unmanned surface vehicles (USVs). Currently, USVs mostly rely on LiDAR and ultrasound technology for the purpose of detecting impediments that exist on water surfaces. However, it is worth noting that these approaches lack the capability to accurately discern the precise nature or classification of those obstacles. Nevertheless, the limited optical range of unmanned vessels hinders their ability to comprehensively perceive the entirety of the surrounding information. A cooperative USV-UAV system is proposed to ensure the visual perception ability of USVs. The multi-object recognition, semantic segmentation, and obstacle ranging through USV and unmanned aerial vehicle (UAV) perspectives are selected to validate the performance of a cooperative USV-UAV system. The you only look once-X (YOLOX) model, the proportional–integral–derivative-NET (PIDNet) model, and distance measurements based on a monocular camera are utilized to realize these problems. The results indicate that by integrating the viewpoints of USVs and UAVs, a collaborative USV-UAV system, employing the aforementioned methods, can successfully detect and classify different objects surrounding the USV. Additionally, it can differentiate between navigable and non-navigable regions for unmanned vessels through visual recognition, while accurately determining the distance between the USV and obstacles.

With the development of technologies related to unmanned vehicles, including unmanned aerial vehicle and unmanned surface vehicle, unmanned vehicles play a pivotal role in maritime management and cruise activities. Meanwhile, lots of evidence shows that a single unmanned vehicle is not qualified for operation under complex environment, whereas the unmanned aerial vehicle–unmanned surface vehicle formation consists of multiple unmanned aerial vehicles and unmanned surface vehicles, which can enhance the performance of the single unmanned vehicle significantly. However, the cooperative operation performance of the unmanned aerial vehicle–unmanned surface vehicle formation is constrained by its communication capacity. Consequently, a cooperative communication framework is designed for the unmanned aerial vehicle–unmanned surface vehicle formation. A distributed dynamic network topology is constructed for fulfilling effective communication for the unmanned aerial vehicle–unmanned surface vehicle formation based on ad hoc network. Experiments are carried out by employing the built unmanned aerial vehicle–unmanned surface vehicle formation platform. Results verify the superiority of the designed cooperative communication framework for the unmanned aerial vehicle–unmanned surface vehicle formation.

Unmanned surface vehicles have the advantages of maneuverability, concealment, wide activity area and low cost of use. Therefore, they have broad application prospects. This makes unmanned surface vehicles a research hotspot at home and abroad, and the sensing technology is the basis for the unmanned surface vehicles to perform tasks. The perception technology based on optical vision has the advantages of convenient application, relatively low cost, easy data acquisition and large amount of information, and has been widely studied by scholars at home and abroad. This paper mainly discusses the research of optical vision in unmanned surface vehicles from five aspects: Firstly, the water surface image preprocessing based on unmanned surface vehicles, mainly including water surface image defogging enhancement research; second is the use of light vision target detection; the next is the surface target tracking methods. Finally, the light vision research of unmanned surface vehicles is summarized and forecasted.

Lentic waters, such as lakes, lagoons, reservoirs, and wetlands are characterized by their absence of current. In recent decades, they have been threatened by pollution and scarcity due to various environmental factors. Therefore, they require frequent monitoring to ensure their health and purity, especially to control the proliferation of harmful cyanobacteria (pollutants). Machine Vision Systems (MVS) on board Autonomous Surface Vehicles (ASVs) is a good option for automatic image processing in this context. ASVs must navigate safely, and obstacle detection is essential. In addition, the segmentation of pollutants in water is crucial. We propose an architecture based on convolutional neural networks that integrates both object detection and semantic segmentation. The goal is to simultaneously extract all available global information to detect objects and amorphous textures (cyanobacterial patches and water bodies), considering their variations in size, pose, and appearance. The architecture includes two branches: object detection and semantic segmentation, sharing the same backbone and neck. We evaluate the model on our dataset and the results show that it can holistically understand lentic water scenes with high accuracy, and the integration of the attention mechanism improves its overall performance.

This paper proposes a method for fused navigation of an unmanned marine surface vehicle (UMSV). The method also detects the outlier and interference of global positioning system (GPS) simultaneously. The method fuses available sensor measurements through extended Kalman filter (EKF) to find the location and attitude of the vehicle. The method uses error covariance of estimated location for detection of GPS outlier and interference. When outlier and interference of the GPS is detected, the method excludes GPS data from navigation process. The measurements to be fused for the navigation are position by GPS, acceleration, angular rate, magnetic field, linear velocity, and range and bearing to acoustic beacons. The method is tested through simulated data and measurement data produced through navigation of a vehicle. The results show that the method detects GPS outlier and interference as well as the GPS recovery, which frees the UMSV navigation from the problem of GPS abnormality. The method is applicable to the vehicle navigation where the UMSV is out of reach from the GPS jamming detection network such as GAARDIAN (GNSS Availability Accuracy Reliability and Integrity Assessment for Timing and Navigation) or jammer detection and location (JLOC) system.

Sea-sky line detection (SSLD) is pivotal for applications such as unmanned surface vehicles (USVs) navigation and maritime target detection. However, existing algorithms are susceptible to interference from adverse weather, illumination change, and the presence of waves, leading to poor detection accuracy and robustness. To address these challenges, we propose a novel SSLD algorithm based on a deep semantic segmentation network. First, we integrate the strengths of convolutional neural networks (CNNs) and Transformers in a lightweight block named efficient vision transformer (E-ViT). This block enables efficient interaction and aggregation of local and global features with lower computational overhead. Building upon E-ViT, we develop an encoder module that significantly improves the accuracy of semantic segmentation while maintaining the network’s lightweight. Then, we design a robust postprocessing module, which leverages semantic information to effectively remove interferences and filter out candidate points for the sea-sky line, thereby achieving high-precision SSLD. Finally, we construct a well-labeled maritime scene dataset with diverse complex attributes to validate the proposed algorithm. Experimental results demonstrate that our method outperforms several state-of-the-art algorithms in terms of both accuracy and robustness in complex maritime scenarios.

Oil and gas companies are increasingly being forced to conduct their sub-sea pipeline inspections in increasingly more hostile and deeper waters. Typically, Remotely Operated Vehicles (ROV) are used for sub-sea pipeline inspections and these are controlled by experienced ROV operators from a surface support vessel, which tend to make ROV operations costly. Conversely, an Autonomous Underwater Vehicle (AUV) can perform sub-sea pipeline inspections without the need for a human operator. This paper sets-out to describe the development of a novel Hybrid AUV (HAUV) system, composed of two unmanned vehicles and designed specifically for sub-sea pipeline inspection. The first vehicle is a compact Unmanned Surface Vehicle (USV), which tracks a sub-sea pipeline from the sea surface. The second vehicle is a HAUV, which is connected to the USV via a fibre optic tether. Both vehicles operate together to automatically inspect sub-sea pipelines. The complete system is controlled by a remote control station on the surface. At this early stage of development, its capability is limited due to its size and battery power source but lab-scale units have been developed and have undergone field testing. Our team study is ongoing and endeavours to refine the HAUV system-capabilities enabling us to offer a cost-effective approach for sub-sea pipeline inspection in the near future.

Efficient cleaning of floating waste using an intelligent unmanned surface vehicle is an important development trend in inland water governance. Path planning is the core of the decision-making module for unmanned surface vehicle waste cleaning and is key to achieving autonomous operation of the unmanned surface vehicle. However, due to the complexity and dynamic changes of the water surface environment, unmanned surface vehicle path planning methods for floating waste face challenges such as small size, uncertainty, and uneven distribution of floating waste. In response to the above issues, this article studies the problem of insufficient integration and low efficiency between existing path planning algorithms and target-perception modules, and designs an efficient overall path planning method for floating-waste cleaning by an unmanned surface vehicle. This method transforms the path planning problem of floating-waste cleaning unmanned surface vehicle into a Traveling Salesman Problem by setting global patrol points and tracking local targets, and proposes an improved ant colony algorithm, IACO, to solve the Traveling Salesman Problem. This article is based on the TSPLIB dataset and practical applications for experiments. The experimental results show that the proposed method has average optimal path lengths of 75.930 m, 446.555 m, and 703.759 m on the Ulysses22, eil51, and st70 datasets, respectively, which are reduced by 0.355 m, 4.108 m, and 13.575 m compared to the benchmark.

A COLREGs-compliant guidance strategy for an underactuated unmanned surface vehicle combining potential field with grid map

A consensus algorithm for coordinated control of unmanned surface vehicles (USVs) is studied. The USVs have different driving directions and initial velocities, so the consensus algorithm is needed to coordinate the directions and velocities of the USVs. Firstly, based on the first order linear model of USVs, get the first order nonlinear differential equations of high frequency, and the simulate environment of USVs. Then, based on the Vicsek model, the consensus algorithm in cooperative control of USVs is proposed. Aiming at the problem that USV's angular velocity lags behind the rudder angle, this paper improves the traditional PD control method, and adopts cascade PID control. So, USV can reach the same driving direction in a shorter time. Finally, simulations prove that the algorithm is feasible and compare the difference of efficiency between the PD controller and the cascade PID controller.