Vision-Based Impedance Control of an Aerial Manipulator Using a Nonlinear Observer

This paper presents a novel approach for estimating the ego-motion of a vehicle in dynamic and unknown environments using tightly-coupled inertial and visual sensors. To improve the accuracy and robustness, we exploit the combination of point and line features to aid navigation. The mathematical framework is based on trifocal geometry among image triplets, which is simple and unified for point and line features. For the fusion algorithm design, we employ the Extended Kalman Filter (EKF) for error state prediction and covariance propagation, and the Sigma Point Kalman Filter (SPKF) for robust measurement updating in the presence of high nonlinearities. The outdoor and indoor experiments show that the combination of point and line features improves the estimation accuracy and robustness compared to the algorithm using point features alone.

In this article, an admittance control strategy for aerial manipulators is proposed to achieve contact force tracking in unknown dynamic environments. First, considering the impact of variations in unknown environments on force tracking performance, an adaptive variable stiffness feature is incorporated into an advanced admittance model. The stiffness coefficient is dynamically adjusted using position and force feedback to generate the desired reference trajectory. Second, to address the issue of reference trajectory tracking under disturbances, a pose controller composed of a disturbance observer and barrier Lyapunov function is utilized to achieve stable tracking performance. In the absence of prior knowledge of disturbances, the state variables converge to a constrained range within a finite time, without introducing excessively high control gains. Finally, the stability of the proposed strategy is rigorously analyzed via Lyapunov tools. Both simulations and real-world experimental investigations are conducted to demonstrate the feasibility of the control strategy, highlighting its robust performance in maintaining a stable contact force during interaction with unknown dynamic environments.

This paper proposes a novel aerial manipulator with front cutting effector (AMFCE) to address the aerial physical interaction (APhI) problem. First, the system uncertainty and external disturbance during the system movement and contact operation are estimated by modeling the entire robot and contact position. Next, based on the established model, the nonlinear disturbance observer (NDO) is used to estimate and compensate the unknown external disturbance of the system and the uncertainty of the model parameters in real time. Then, the nonsingular terminal synovial membrane control method is used to suppress the part that is difficult to estimate. Finally, a controller which is suitable for the movement and operation of the entire system is designed. The controller’s performance is verified through experiments, and the results show that the design, modeling, and control of the entire system can achieve the APhI.

The aerial manipulator (AM) is a systematic operational robotic platform in high standard on algorithm robustness. Directly deploying the algorithms to the practical system will take numerous trial and error costs and even cause destructive results. In this paper, a new modular simulation platform is designed to evaluate aerial manipulation related algorithms before deploying. In addition, to realize a fully autonomous aerial grasping, a series of algorithm modules consisting a complete workflow are designed and integrated in the simulation platform, including perception, planning and control modules. This framework empowers the AM to autonomously grasp remote targets without colliding with surrounding obstacles relying only on on-board sensors. Benefiting from its modular design, this software architecture can be easily extended with additional algorithms. Finally, several simulations are performed to verify the effectiveness of the proposed system.

Unmanned Aerial Vehicles (UAVs) installed with a gripper is an effective and robust way to grab the wanted object from inaccessible locations. In this study, we develop a novel control mechanism to regulate the nonlinear dynamics of the aerial manipulator. In this research, hex-rotor UAV is chosen in order to fulfill the mission requirement in terms of size and weight of the object. It is equipped with a manipulator and the gimbal-based camera that will help to see the desired object and then transport it. The aerial vehicle has six-degrees-of-freedom (6-DOF) and the installed manipulator has 4-DOF which in total makes the 10-DOF aerial manipulator vehicle. At the time of clutching the desired object to eliminate or reduce the external noise, and stabilize the dynamic behavior of the aerial manipulator, we need a robust and efficient controller. To solve the aforementioned problems, this study develops a hybrid control mechanism that tracks and controls the altitude and attitude of UAV after clutching the desired object. The main contribution of this study is to design a control mechanism that includes Model Reference Adaptive Control with an Integrator (MRACI) in conjunction with regulation, pole-placement and tracking (RST) control algorithm. On one hand, the simulation results using MATLAB demonstrate the efficiency of the proposed control mechanism. On the other hand, to cross verify the validity of the proposed control algorithm, we perform the experiment by clutching the desired object at hovering and normal flight operation.

Unmanned Aerial Manipulators (UAM) are gaining attention within the unmanned aerial systems research community. They can be used for aerial manipulation tasks such as object retrieval from confined and hard-to-reach spaces. The coupled dynamics between the arm and the base and also the limited flight time would require the development and implementation of optimal motion planning and robust flight control strategies. In this paper, we propose a novel integrated planning and control strategy for object retrieval. A new kinodynamic version of the Rapidly-exploring Random Tree (RRT*), called Lazy-Steering-RRT*, is developed for planning UAM&#x2019;s motion from its start to a pre-grasp state, while keeping the motion of the arm to a minimum. This planning can be carried out on the fly by using Machine-Learning-based techniques to construct the edges in the search tree in a time-efficient way. This facilitates re-planning, as the environment is gradually sensed by limited-range sensors onboard. Once the UAM reaches the pre-grasp state at the end of the motion cycle, an RRT approach is then utilized, where motions of the base and the arm are coordinated for reaching and grasping the object. A novel partitioned control approach, composed of model predictive and PID controllers, is utilized for the UAM to track the planned trajectory. The overall motion planning and control algorithms have been implemented in simulation using Multibody Physics Engines and a number of representative simulation runs are presented. Our results show that the approach is effective in successfully executing the object retrieval task even in confined spaces. <i>Note to Practitioners</i>&#x2014;This research was motivated by the problem of retrieving an object from a confined space such as that in a collapsed building using an unmanned aerial manipulator. Our approach has the following advantages over existing methods: a) Planning and re-planning can be carried out on the fly, b) No prior information about the environment is needed, c) High-frequency energy-optimal control of the UAM is carried out, and d) the planner and the controller are coupled. In this paper, we suggest some novel approaches on both planning and control aspects of the navigation and also on the integration of these two modules to provide a navigation package that meets all the requirements in the retrieval task. Furthermore, our algorithm is computationally-light, therefore, well suited for on the fly planning and control.

At present, most of the visual Simultaneous Localization and Mapping (SLAM) systems rely on the surrounding point features to achieve acceptable localization and mapping. However, the number of point features is insufficient in the low-texture environments, so the performance of these SLAM systems will be significantly reduced. In this research, a PLISLAM system that integrates point features, line features and Inertial Measurement Unit (IMU) is proposed to implement high-precision positioning and mapping for dynamic vehicles. Specifically, a state-of-the-art SLAM scheme ORBSLAM3 is built at first. Then, its theoretical formulation is derived step by step to handle the environmental line features and the Bundle Adjustment (BA) is integrated to optimize the data. Finally, the system performance is verified through the EuRoC dataset, the results demonstrate its accuracy could be improved by adding the line features especially in scenes with rich line features.

For the visual-inertial system, the key frame loss always emerges in the rotation with the weak texture and strong illumination, which seriously affects the camera positioning accuracy. To solve this problem, a tight-coupling-based visual fusion scheme leveraging line features is proposed in our work, called VinsFusion-Line. It first deeply integrates line features, inertial measurement unit(IMU) pre-integration, and stereo visual observations through a tightly-coupled framework. And the infinitely extended line features provide more geometric and structural information, compared with the base point and line feature tracing methods. Hence, the positioning accuracy is improved greatly. In order to further make the calculation of 3D spatial line features faster and more efficient, Plücker coordinates and straight lines are used to represent the features. In addition, the camera state is optimized by a multi-source cost function that embeds both IMU pre-integration residuals and line feature reprojection errors. This design enables tight-coupled fusion of IMU and binocular visual information, rather than relying solely on point features as in existing methods. Finally, experiments on the public data show that under the strong light and weak texture, the binocular VinsFusion-Line system with infinite extension line in our work has better positioning accuracy than the single-purpose Vins-Fusion system.

Most mainstream visual-inertial SLAM systems rely on the point feature method of front-end images for motion estimation and localization. However, the SLAM system based on the point feature method, in the case of weakly textured environments, may not be able to estimate the motion due to the lack of point features. For this reason, the introduction of line features has received extensive attention. Line features in the environment can improve the localization accuracy of SLAM, but the extraction and matching of line features are relatively time-consuming and cannot satisfy real-time requirements. Based on the EDLines algorithm, we propose a fast visual-inertial odometry that fuses point and line features. The improved EDLines algorithm is used to process line features. By adjusting the internal parameters of the algorithm, the extraction time of line features is decreased while the extraction effect is the same. And the line features screening strategy is proposed to filter out invalid short line segments and merge adjacent line segments. We use the more effective line features after screening for graph matching and back-end optimization, which reduces the calculation amount of line features matching and ensures the accuracy. The proposed algorithm has been tested on the public datasets, and the comparison results with PL-VIO and VINS-Mono show that the algorithm in this paper improved real-time performance and localization accuracy.

Most of the existing visual SLAM schemes rely solely on point or line features to estimate camera trajectory. In some scenes such as texture missing or motion blurring, it’s difficult to find a sufficient number of reliable features, resulting in low positioning accuracy. To extract more features, an RPL-SLAM solution is proposed to extract point features and line features respectively. Ulteriorly, the depth information of RGBD image is used to restore the 3D information of point and line features, improving the accuracy of camera track positioning. RPL-SLAM scheme mainly includes three modules: tracking, local mapping and loop detection. Tracking module extends the application of line feature on the basis of point feature extraction and matching. A SLD line segment extraction algorithm which can eliminate the micro segments and a DBM segment matching algorithm based on word bag are proposed respectively. These two algorithms improve the matching efficiency while ensuring the matching accuracy, and effectively track and locate the camera of each frame. In the local mapping and loop detection module, the Plucker coordinates are applied to express the spatial line and define the re-projection error of the straight line, so that the back-end optimization error model of point-line fusion is unified to solve the instability problem in optimization. RPL-SLAM is tested on TUM RGBD and ICL-NUIM data set respectively, and compared with ORB-SLAM2. The result shows that RPL-SLAM can effectively improve the accuracy of pose estimation and map reconstruction while maintaining real-time performance by fusing point-line features with depth images.

PurposeThis paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step.Design/methodology/approachThe prototype system’s process can be divided into two stages: the offline preparation stage and online execution stage. In the offline preparation stage, planning results (assembly sequence, parts position, rotation, etc.) and image features [gradient and oriented FAST and rotated BRIEF (ORB)features] are extracted automatically from the assembly planning process. In the online execution stage, too, image features are extracted and matched with those generated offline to compute the camera pose, and planning results stored in XML files are parsed to generate the assembly instructions for manipulators. In the prototype system, the working range of template matching algorithm, LINE-MOD, is first extended by using depth information; then, a fast and robust marker-less tracker that combines the modified LINE-MOD algorithm and ORB tracker is designed to update the camera pose continuously. Furthermore, to track the camera pose stably, a tracking strategy according to the characteristic of assembly is presented herein.FindingsThe tracking accuracy and time of the proposed marker-less tracking approach were evaluated, and the results showed that the tracking method could run at 30 fps and the position and pose tracking accuracy was slightly superior to ARToolKit.Originality/valueThe main contributions of this work are as follows: First, the authors present a coarse-to-fine marker-less tracking method that uses modified state-of-the-art template matching algorithm, LINE-MOD, to find the coarse camera pose. Then, a feature point tracker ORB is activated to calculate the accurate camera pose. The whole tracking pipeline needs, on average, 24.35 ms for each frame, which can satisfy the real-time requirement for AR assembly. On basis of this algorithm, the authors present a generic tracking strategy according to the characteristics of the assembly and develop a generic AR-based assembly assistance platform. Second, the authors present a feature point mismatch-eliminating rule based on the orientation vector. By obtaining stable matching feature points, the proposed system can achieve accurate tracking results. The evaluation of the camera position and pose tracking accuracy result show that the study’s method is slightly superior to ARToolKit markers.

Image registration is an important basis of image processing, which is of great significance in image mosaicking, target recognition, and change detection. Aiming at the automatic registration problem of multi-angle optical images for ground scenes, a registration method combining point features and line features to register images is proposed. Firstly, the LSD (Line Segment Detector) algorithm is used to extract line features of images. The obtained line segments whose length are less than a given threshold are eliminated by a visual significant algorithm. Then, an affine transform model obtained by estimating a Gaussian mixture model (GMM) is applied to the image to be matched. Lastly, Harris point features are utilized in fine matching to overcome shortages of methods based on line features. In experiments, the proposed algorithm is compared with popular feature-based registration algorithms. The results indicate that the proposed algorithm in this work has obvious advantages in terms of registration accuracy and reliability for optical images acquired at different angles.

Visual Simultaneous Localization and Mapping (SLAM) technologies based on point features achieve high positioning accuracy and complete map construction. However, despite their time efficiency and accuracy, such SLAM systems are prone to instability and even failure in poor texture environments. In this paper, line features are integrated with point features to enhance the robustness and reliability of stereo SLAM systems in poor texture environments. Firstly, method Edge Drawing lines (EDlines) is applied to reduce the line feature detection time. Meanwhile, the proposed method improves the reliability of features by eliminating outliers of line features based on the entropy scale and geometric constraints. Furthermore, this paper proposes a novel Bags of Word (BoW) model combining the point and line features to improve the accuracy and robustness of loop detection used in SLAM. The proposed PL-BoW technique achieves this by taking into account the co-occurrence information and spatial proximity of visual words. Experiments using the KITTI and EuRoC datasets demonstrate that the proposed stereo Point and EDlines SLAM (PEL-SLAM) achieves high accuracy consistently, including in challenging environments difficult to sense accurately. The processing time of the proposed method is reduced by 9.9% and 4.5% when compared to the Point and Line SLAM (PL-SLAM) and Point and stereo Point and Line based Visual Odometry (sPLVO) methods, respectively.

PurposeThis paper aims to enable the unmanned aerial vehicles to inspect the surface condition of wind turbine in close range when the global positioning system signal is not reliable, and further improve its intelligence. So a visual-inertial odometry with point and line features is developed.Design/methodology/approachVisual front-end combining point and line features, as well as its purification strategies, are first presented to improve the robustness of feature tracking in low-textured scene and rapidity of segment detector. Additionally, the inertial measurement is integrated between keyframes as constrain to reduce tracking error existed in visual-only system. Second, the graph-based visual-inertial back-end is constructed. To parameterize line features effectively, the infinite line representation not sensitive to outdoor light is employed, in which Plücker and Cayley are selected for line re-projection and nonlinear optimization. Furthermore, Jacobians of the line re-projection errors are analytically derived for better accuracy.FindingsExperiments are performed in various scenes of the wind farm. The results demonstrate that the tight-coupled visual-inertial odometry with point and line features is more precise on all the samples than conventional algorithms in complex wind farm environments. Additionally, the constructed line feature map can be used in the following research for autonomous navigation.Originality/valueThe proposed visual-inertial odometry works robustly in strong electromagnetic interference, low-textured and illumination-change wind farm.

When mobile robots perform their missions, the self-localization is needed basically. Several past researches established how to obtain their location information from the environment by using a distance sensor or a camera. However, these methods have map-making problem when the environment changes and localization problem while the robot moves from sensing features has typical affine and occlusion characteristics. This paper presents a localizer for mobile robot that travels around indoor environments. Our module uses the only one sensor, a single camera looking up the ceiling. There is no efficient enough SLAM* (Simultaneous Localization And Mapping) algorithm working on embedded system. The initial difficulty of vision based SLAM is computational complexity to acquire reliable feature on their algorithm. To reduce the computational complexity, we use the ceiling segmentation to extract line features of ceiling area. Line features are extracted from the boundaries between the ceiling and walls. The line features have advantages over point features for its robustness to environmental variation and structural information helpful to data association. Extended Kalman Filter is used to estimate the pose of a robot and build the ceiling map with line features. The experiment is practiced in our indoor test bed and the proposed algorithm is proved by the experimental results.*SLAM: Simultaneous localization and mapping is a technique used by robots and autonomous vehicles to build up a map within an unknown environment or to update a map within a known environment while at the same time keeping track of their current location.