This paper introduces a novel deep reinforcement learning framework to tackle the problem of visual servoing of nonholonomic mobile robots. The visual control of nonholonomic mobile robots becomes particularly challenging within the classical paradigm of visual servoing, mainly due to motion and visibility constraints, which makes it impossible to reach a given desired pose for certain configurations without losing essential visual information from the camera field of view. Previous work has demonstrated the effectiveness of deep reinforcement learning in addressing various vision-based robotics tasks. In light of this, we propose a framework that integrates deep recurrent policies, intrinsic motivation, and a novel auxiliary task that leverages the interaction matrix, the core of classical visual servoing approaches, to address the problem of vision-based control of nonholonomic robotic systems. Firstly, we analyze the influence of the nonholonomic constraints on control policy learning. Subsequently, we validate and evaluate our approach in both simulated and real-world environments. Our approach exhibits an emergent control behavior that enables the robot to accurately attain the desired pose while maintaining the desired visual content within the camera’s field of view. The proposed method outperforms the state-of-the-art approaches, demonstrating its effectiveness, robustness, and accuracy. • A DRL method is proposed to address the visual control of nonholonomic mobile robots. • A novel auxiliary task for prediction motion perceptibility values from input images. • Validation and evaluation of the proposed approach in both simulated and real-world environments.

Environments lacking geometric features (e.g., tunnels and long straight corridors) are challenging for LiDAR-based odometry algorithms because LiDAR point clouds degenerate in such environments. For wheeled robots, a wheel kinematic model (i.e., wheel odometry) can improve the reliability of the odometry estimation. However, the kinematic model suffers from complex motions (e.g., wheel slippage, lateral movement) in the case of skid-steering robots particularly because this robot model rotates by skidding its wheels. Furthermore, these errors change nonlinearly when the wheel slippage is large (e.g., drifting) and are subject to terrain-dependent parameters. To simultaneously tackle point cloud degeneration and the kinematic model errors, we developed a LiDAR-IMU-wheel odometry algorithm incorporating online training of a neural network that learns the kinematic model of wheeled robots with nonlinearity. We propose to train the neural network online on a factor graph along with robot states, allowing the learning-based kinematic model to adapt to the current terrain condition. The proposed method jointly solves online training of the neural network and LiDAR-IMU-wheel odometry on a unified factor graph to retain the consistency of all those constraints. Through experiments, we first verified that the proposed network adapted to a changing environment, resulting in an accurate odometry estimation across different environments. We then confirmed that the proposed odometry estimation algorithm was robust against point cloud degeneration and nonlinearity (e.g., large wheel slippage by drifting) of the kinematic model. The summary video is available at: https://www.youtube.com/watch?v=CvRVhdda7Cw • Our neural network-based kinematic model for wheeled robots is trained online on a factor graph. • Online learning of the network and LiDAR-IMU-wheel odometry are simultaneously solved on the unified factor graph to retain both consistency. • Our network balances accuracy and computational cost in online learning to achieve real-time calculation. • The neural adaptive odometry factor provides constraints regarding robot motion and online learning, for factor graph optimization. • Online learning can give terrain adaptability to the learning-based kinematic model.