3D LIDAR–camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization

For applications in robotics and autonomous vehicles, fusing 3D LiDAR, RGB camera, and thermal camera can improve the perception in day and night environments. To fuse the sensors, accurate extrinsic calibration is important. However, most existing methods use offline calibration, which is tedious, as it requires the use of special targets and human intervention. Meanwhile, among the online calibration methods, little attention has been paid to LiDAR and thermal camera online calibration. Thus, in this paper, an online extrinsic calibration method is proposed to solve the problem, called RGBDTCalibNet. It can achieve the extrinsic calibration between a 3D LiDAR and an RGB camera, as well as a 3D LiDAR and a thermal camera. It leverages CNN-based deep learning methods for feature extraction and feature matching. The proposed network only takes a pair of images and point cloud as the input and outputs the extrinsic parameters in a single shot. It is trained and tested on two datasets: KITTI360 and our own dataset collected in the NTU campus, using two types of LiDAR: Velodyne HDL-64 and Livox Horizon. Both demonstrate the effectiveness, robustness and accuracy of RGBDTCalibNet. The code will be open-sourced at https://github.com/sanatmharolkar/RGBDTCalibNet.git

A consistently fast and accurate algorithm for estimating camera pose from point correspondences

New exact and heuristic algorithms for general production and delivery integration

This paper presents a novel extrinsic calibration algorithm between a binocular stereo vision system and a 2D LIDAR (laser range finder). Extrinsic calibration of these heterogeneous sensors is required to fuse information obtained separately by vision sensor and LIDAR in the context of intelligent vehicle. By placing a planar chessboard at different positions and orientations in front of the sensors, the proposed method solves the problem based on 3D reconstruction of the chessboard and geometric constraints between views from the stereovision system and the LIDAR. The three principle steps of the approach are: 3D corner points triangulation, 3D plane least-squares estimation, solving extrinsic parameters by applying a non-linear optimization algorithm based on the geometric constraints. To evaluate the performance of the algorithm, experiments based on computer simulation and real data are performed. The proposed approach is also compared with a popular calibration method to show its advantages.

3D LiDARs and 2D cameras are increasingly being used alongside each other in sensor rigs for perception tasks. Before these sensors can be used to gather meaningful data, however, their extrinsics (and intrinsics) need to be accurately calibrated, as the performance of the sensor rig is extremely sensitive to these calibration parameters. A vast majority of existing calibration techniques require significant amounts of data and/or calibration targets and human effort, severely impacting their applicability in large-scale production systems. We address this gap with CalibNet: a self-supervised deep network capable of automatically estimating the 6-DoF rigid body transformation between a 3D LiDAR and a 2D camera in real-time. CalibNet alleviates the need for calibration targets, thereby resulting in significant savings in calibration efforts. During training, the network only takes as input a LiDAR point cloud, the corresponding monocular image, and the camera calibration matrix K. At train time, we do not impose direct supervision (i.e., we do not directly regress to the calibration parameters, for example). Instead, we train the network to predict calibration parameters that maximize the geometric and photometric consistency of the input images and point clouds. CalibNet learns to iteratively solve the underlying geometric problem and accurately predicts extrinsic calibration parameters for a wide range of mis-calibrations, without requiring retraining or domain adaptation. The project page is hosted at https://epiception.github.io/CalibNet

We study the minimum-cost flow problem on dynamic networks with nonlinear cost functions that depend on time and edge flow. A general procedure for solving the problem using the time-expanded network is described. The main properties of dynamic flows on networks with concave cost functions are studied. We propose an algorithm for finding the optimal flow in networks with exactly one source; its running time is polynomial for a fixed number of sinks. Some details concerning the correctness of the algorithm and its computational complexity are discussed.

We consider a Cournot oligopoly game, where firms produce an homogenous good and the demand and cost functions are nonlinear. These features make the classical best reply solution difficult to be obtained, even if players have full information about their environment. We propose two different kinds of repeated games based on a lower degree of rationality of the firms, on a reduced information set and reduced computational capabilities. The first adjustment mechanism is called “Local Monopolistic Approximation” (LMA). First firms get the correct local estimate of the demand function and then they use such estimate in a linear approximation of the demand function where the effects of the competitors’ outputs are ignored. On the basis of this subjective demand function they solve their profit maximization problem. By using the second adjustment process, that belongs to a class of adaptive mechanisms known in the literature as “Gradient Dynamics” (GD), firms do not solve any optimization problem, but they adjust their production in the direction indicated by their (correct) estimate of the marginal profit. Both these repeated games may converge to a Cournot–Nash equilibrium, i.e. to the equilibrium of the best reply dynamics. We compare the properties of the two different dynamical systems that describe the time evolution of the oligopoly games under the two adjustment mechanisms, and we analyze the conditions that lead to non-convergence and complex dynamic behaviors. The paper extends the results of other authors that consider similar adjustment processes assuming linear cost functions or linear demand functions.

In this paper, the problem of adaptive fuzzy decentralized optimal control is investigated for a class of nonlinear large-scale systems in strict-feedback form. The considered nonlinear large-scale systems contain the unknown nonlinear functions and unmeasured states. By utilizing the fuzzy logic systems to approximate the unknown nonlinear functions and cost functions, a fuzzy state observer is established to estimate the unmeasured states. The control design is divided into two phases. First, by using the state observer and the backstepping design technique, a feedforward decentralized controller with parameters adaptive laws is designed, by which the original controlled strict-feedback nonlinear large-scale system is transformed into an equivalent affine nonlinear large-scale system. Second, by using adaptive dynamic programming theory, a feedback decentralized optimal controller is developed for the equivalent affine nonlinear system. The whole adaptive fuzzy decentralized optimal control scheme consists of a feedforward decentralized controller and a feedback decentralized optimal controller. It is shown that the proposed adaptive fuzzy decentralized optimal control approach can guarantee that all the signals in the closed-loop system are bounded, and the tracking errors converge to a small neighborhood of zero. In addition, the proposed control approach can guarantee that the cost functions are minimized. Simulation results are given to demonstrate the effectiveness of the proposed control approach.

To obtain the 6 DOF extrinsic parameters (rotation and translation matrix) between a 3D ranging sensor and a thermal camera, previous methods require a high-resolution 3D ranging sensor to reliably detect features. Although sparse 3D LiDARs are widely used on autonomous robots, to the best of our knowledge, the extrinsic calibration between a sparse 3D LiDAR (particularly Velodyne VLP-16) and a thermal camera has not been considered in the literature. In this paper, we present a two-step method to address the problem, where a monocular visual camera is used to assist the process. The proposed method decomposes the problem into two steps: extrinsic calibration between a sparse 3D LiDAR and a visual camera; extrinsic calibration between a visual camera and a thermal camera. Experiments are conducted to demonstrate the effectiveness of the proposed two-step method.

A variety of LQR-RRT kinodynamic motion planners are built on the idea of solving a two point boundary value problem in an LQR manner for affine systems. These planners can also be used for controllable nonlinear systems only if its linearized model at the equilibrium state is also controllable, and the cost function reflects only a time/control trade-off. We propose a class of RRT planners based on the SDRE (State Dependent Riccati Equation) control paradigm. The SDRE control is used both for finding the nearest state in the tree and for the tree expansion. By solving an LQR tracking problem for nonlinear systems within the SDRE framework, instead of a two point boundary value problem, the proposed planners deal with a wider range of controllable nonlinear systems and cost functions. We compare the proposed planners with LQR-RRT-like algorithms by observing the results obtained from the three specific benchmark examples.

This paper investigates fuzzy adaptive output feedback optimal control problem for a class of strict-feedback nonlinear systems. With the help of fuzzy logic systems approximating the unknown nonlinear functions and cost function, the unmeasured states are estimated by designing fuzzy adaptive state observer. Combining state observer with backstepping design technique, a feedforward controller is designed. Based on the designed feedforward control strategy, the controlled nonlinear system can be converted to an equivalence nonlinear system in affine-form. Finally, a fuzzy adaptive optimal controller with parameters adaptive laws is developed. The whole control scheme consists of a feedforward controller and a feedback optimal controller. It is shown that the proposed output feedback optimal control approach can guarantee that all signals in the closed-loop system are bounded, and the system output can track the reference signal. In addition, the proposed control approach can guarantee cost function is the smallest. Simulation results are given to demonstrate the effectiveness of the proposed control approach.

Recent advances in applying deep learning to planning have shown that Deep Reactive Policies (DRPs) can be powerful for fast decision-making in complex environments. However, an important limitation of current DRP-based approaches is either the need of optimal planners to be used as ground truth in a supervised learning setting or the sample complexity of high-variance policy gradient estimators, which are particularly troublesome in continuous state-action domains. In order to overcome those limitations, we introduce a framework for training DRPs in continuous stochastic spaces via gradient-based policy search. The general approach is to explicitly encode a parametric policy as a deep neural network, and to formulate the probabilistic planning problem as an optimization task in a stochastic computation graph by exploiting the re-parameterization of the transition probability densities; the optimization is then solved by leveraging gradient descent algorithms that are able to handle non-convex objective functions. We benchmark our approach against stochastic planning domains exhibiting arbitrary differentiable nonlinear transition and cost functions (e.g., Reservoir Control, HVAC and Navigation). Results show that DRPs with more than 125,000 continuous action parameters can be optimized by our approach for problems with 30 state fluents and 30 action fluents on inexpensive hardware under 6 minutes. Also, we observed a speedup of 5 orders of magnitude in the average inference time per decision step of DRPs when compared to other state-of-the-art online gradient-based planners when the same level of solution quality is required.

We present a bit rate maximising per-tone equalisation (BM-PTEQ) cost function that is based on an exact subchannel SNR as a function of per-tone equaliser in discrete multitone (DMT) systems. We then introduce the proposed BM-PTEQ criterion whose derivation for solution is shown to inherit from the methodology of the existing bit rate maximising time-domain equalisation (BM-TEQ). By solving a nonlinear BM-PTEQ cost function, an adaptive BM-PTEQ approach based on a recursive Levenberg-Marquardt (RLM) algorithm is presented with the adaptive inverse square-root (iQR) algorithm for DMT-based systems. Simulation results confirm that the performance of the proposed adaptive iQR RLM-based BM-PTEQ converges close to the performance of the proposed BM-PTEQ. Moreover, the performance of both these proposed BM-PTEQ algorithms is improved as compared with the BM-TEQ.

Three approaches are proposed for the design of a hybrid railway power substation taking into account the control of the storage system over 1 year. The first one is based on a piecewise linearization of the cost function and solved with linear programming. The others decompose the whole problem in 2 levels with lower dimensionality. Collaborative optimization solves it with a double‐loop scheme while Benders decomposition allows a sequential resolution. This last gives the same solution as linear programming in about 5 times more computing time and can be used with a nonlinear cost function.

This paper proposes a new method for solving economic dispatch problems in electrical power systems through the application of multithread programming. Multi-Thread Interior Point Barrier Algorithm, an efficient quadratic programming algorithm, is used to solve economic dispatch problem. The barrier algorithm also caters for any combination of polynomial functions such as polynomial cost functions. Nonlinear characteristics of electrical power system such as generators limitations, transmission losses and nonlinear cost functions are considered in the economic dispatch formulation. Exact transmission loss formula is used in the equality constraint in the economic dispatch equations. The transmission loss coefficients are calculated by using a multi-thread programming. To ensure the optimal dispatch satisfies the line flow limits, line flow constraints has been added to formulation as a security constraint. This optimization problem enable us to solve economic dispatch problems faster, so it is suitable for unit commitment problems and the other power system problems which need to get reasonable speed. IEEE 30, 57, 118 bus case studies are presented and tested with the proposed method.