Toward advanced vision systems : low-cost fusion of lidar and camera for its and infrastructure condition assessments
The integration of Internet of Things (IoT) technology, which includes LiDAR sensors, is changing traffic operation, safety, and infrastructure assessment. LiDAR sensors, along with other IoT devices such as cameras and GPS units, are embedded within transportation infrastructures to provide real-time, precise data about traffic conditions and road safety. This wealth of data enables dynamic traffic control, facilitates the immediate identification of safety hazards, and supports proactive maintenance strategies for roads and bridges. Furthermore, the study focuses on leveraging machine learning to develop low-cost vision systems that can effectively utilize data streaming from infrastructure-mounted sensors to enhance traffic operations, safety and infrastructure condition monitoring. Immediate access to high-resolution traffic parameters from surveillance videos allows for a more dynamic approach to traffic control, incident detection, and congestion management. While cameras combined with CV techniques are vital in analyzing real-time traffic scenarios, another vision-based sensor, Light Detection and Ranging (LiDAR), with its precise laser scanning, also brings transformative advantages to transportation systems. LiDAR revolutionizes transportation systems by delivering higher accuracy and comprehensive coverage data for monitoring traffic complexities. However, the limitations including requirements for complex camera calibration and the incomplete point cloud information made the usage of these sensors costly and impeded its widespread adoption. Explored the advantages of both sensors, it is necessary to develop a low- cost vision-based system, that fuses the capabilities of both camera and LiDAR respectively. Such a system would combine the distinctive advantages of both sensors to surpass their constraints separately and would work as a tool to optimize the collection process and improve transportation safety. The author proposed three innovative objectives based on the previous research gaps. Firstly, the research investigates the distinct benefits of the camera and LiDAR in extracting high- resolution traffic data from a singular sensory input, and it features the automatic annotation of 3D bounding boxes in the LiDAR domain. Second, the study designed a framework that fuses camera and LiDAR technology, capable of executing real-time object recognition and depth perception simultaneously. Third, the study developed a machine-learning framework to assess transportation infrastructure utilizing a single image that in daytime lighting condition. The first object includes two aspects of both camera and LiDAR, the specific objectives are: 1). Implements a 2D homography technique for perspective transformation of traffic scenes to a bird-eye view which reduces the effect of partial occlusion and improves the accuracy of speed and acceleration data collection; CCTV cameras are automatically calibrated to convert pixel distances to real-world distances; Used the concept of spatial-temporal neighbors to improve the tracking results and thereby address the challenge posed by misassignment of vehicle trajectories. 2). We propose an innovative and effective framework for obtaining full point cloud representations of partially occluded objects using just a single LiDAR system, thus addressing the limitation of requiring multiple LiDAR units for comprehensive data acquisition; We introduce a novel framework for generating 3D bounding box annotations without relying on human annotators. This development is crucial in streamlining the labor-intensive and time-consuming process of annotating LiDAR data for traffic detection and classification models; The study pioneers the application of zero-shot learning techniques for vehicle and pedestrian detection, classification, and counting. 3). Develops a framework for spatial-temporal fusion of 360-degree video and LiDAR data. This framework enables us to associate each bounding box proposal in 2D video domain with actual distances or depths generated from mobile LiDAR data; Define a novel architecture that extends the output prediction vectors of Darknet-like backbone with information about depth. We also introduce a new lost that enables the network to simultaneously generate bounding box proposals and corresponding depth of each object in a single shot; Generated a large database of annotations for machine learning model development and comparative analysis. 4). Generate a big dataset of transportation infrastructure that collected from camera and LiDAR. Developed an innovative framework that integrates transportation infrastructure image and class information, with LiDAR-derived information, such as depth, and reflectivity intensity data through a novel Double U-Net architecture, enabling pixel-level reflectivity predictions for detailed infrastructure assessment.
Read more