Implementation of stereo vision algorithm on signal processing hardware for depth estimation

The advanced driver assistance system (ADAS) for adaptive cruise control and collision avoidance is strongly dependent upon the robust image recognition technology such as lane detection, vehicle/pedestrian detection, and traffic sign recognition. However, the conventional ADAS cannot realize more advanced collision evasion in real environments due to the absence of intelligent vehicle/pedestrian behavior analysis. Moreover, accurate distance estimation is essential in ADAS applications and semiglobal matching (SGM) is most widely adopted for high accuracy, but its system-on-chip (SoC) implementation is difficult due to the massive external memory bandwidth. In this paper, an ADAS SoC with behavior analysis with Artificial Intelligence functions and hardware implementation of SGM is proposed. The proposed SoC has dual-mode operations of highperformance operation for intelligent ADAS with real-time SGM in D-Mode (d-mode) and ultralow-power operation for black box system in parking-mode. It features: 1) task-level pipelined SGM processor to reduce external memory bandwidth by 85.8%; 2) region-of-interest generation processor to reduce 86.2% of computation; 3) mixed-mode intention prediction engine for dualmode intelligence; and 4) dynamic voltage and frequency scaling control to save 36.2% of power in d-mode. The proposed ADAS processor achieves 862 GOPS/W energy efficiency and 31.4GOPS/ mm(2) area efficiency, which are 1.53x and 1.75x improvements than the state of the art, with 30 frames/s throughput under 720p stereo inputs.

Depth information can be obtained using stereo matching algorithms, which compute the horizontal displacement (disparity) of the corresponding points and convert to depth information using the triangular relation. However, the matching process is challenging with presence of textureless regions. This paper proposes a novel disparity refinement method for stereo matching based on Semi-global Matching (SGM) algorithm for texture less images. To be brief, the SGM algorithm is a high-performance matching algorithm of a stereo image pair and it reaches a trade off between matching quality and computing complexity. The main contribution comes from the improvement of matching quality on texture less regions. At the end of SGM, a right-to-left consistency check is performed to remove the invalid pixels in the disparity map. The proposed method is added after the right-to-left consistent check. We assume that the texture less regions in the original stereo pairs are planar. We employ edge detection to extract the textureless regions. The fitting method is employed in horizontal and vertical directions respectively. If the distance of two adjacent edge pixels is sufficiently large, the corresponding line segment will be considered as texturelss. For every texturelss line segment, the RANSAC algorithm is performed on the corresponding line segment in the disparity map. We use the well-known Middlebury dataset to compare our method with the normal SGM and other matching algorithms. It shows that our method performs well for most texture less stereo pairs.

Monocular depth estimation is a fundamental yet challenging task in computer vision as depth information will be lost when 3D scenes are mapped to 2D images. Although deep learning-based methods have led to considerable improvements for this task in a single image, most existing approaches still fail to overcome this limitation. Supervised learning methods model depth estimation as a regression problem and, as a result, require large amounts of ground truth depth data for training in actual scenarios. Unsupervised learning methods treat depth estimation as the synthesis of a new disparity map, which means that rectified stereo image pairs need to be used as the training dataset. Aiming to solve such problem, we present an encoder-decoder based framework, which infers depth maps from monocular video snippets in an unsupervised manner. First, we design an unsupervised learning scheme for the monocular depth estimation task based on the basic principles of structure from motion (SfM) and it only uses adjacent video clips rather than paired training data as supervision. Second, our method predicts two confidence masks to improve the robustness of the depth estimation model to avoid the occlusion problem. Finally, we leverage the largest scale and minimum depth loss instead of the multiscale and average loss to improve the accuracy of depth estimation. The experimental results on the benchmark KITTI dataset for depth estimation show that our method outperforms competing unsupervised methods.

Vision-based future vehicle localization provides intuitive trajectory prediction, serving as a critical foundation for Advanced Driving Assistance Systems (ADAS) to formulate collision avoidance decisions. Among existing approaches, ego-view trajectory prediction has proven effective for driver monitoring and intervention in vision-based localization. This method aligns closely with human perceptual processing, making it essential for the Driver-in-the-Loop (DIL) development stage in modern ADAS. However, most existing ego-view trajectory prediction approaches rely on two-dimensional image-based predictions, creating a gap with human three-dimensional perception. This disparity negatively impacts the accuracy and timeliness of driver decision-making and intervention. In this paper, we propose MFV3DL (Monocular Vision Method for Future Vehicle 3D Localization), a dual-stream framework integrating 2D image trajectory prediction and depth prediction to achieve future vehicle 3D localization. To enhance accuracy, we leverage Multi-Object Tracking and Segmentation (MOTS) results and depth estimation as inputs for the dual-stream architecture. Additionally, we introduce a Related Information Fusion (RIF) unit to enable cross-modal interaction between the two streams. For depth stream predictions, we propose a ConvLSTM-based depth prediction method. Experimental results on the KITTI dataset demonstrate that MFV3DL outperforms state-of-the-art methods. In diverse driving scenarios, MFV3DL achieves superior 3D visualization results compared to 2D trajectory-based predictions. Baseline comparisons and ablation studies further validate that the proposed ConvLSTM-based depth prediction enhances the dual-stream architecture and RIF unit for 3D localization tasks.

In stereo vision, disparity map after stereo matching using projection model can get the depth information and 3D information of the original image. In order to improve the accuracy and speed of stereo matching, improving the quality of disparity map becomes the core problem of stereo matching. Firstly, the paper extracted edge information of binocular images using Canny, Scharr and Sobel edge detection algorithm. On this basis, the paper use SGBM algorithm(Semi-global block Matching) and DP algorithm(Dynamic Programming)and CSCA algorithm(Cross-scale Cost Aggregation) respectively calculated the disparity map and its error rate. After analysis, the combination of Canny detection algorithm and CSCA algorithm can eliminate a large number of number of irrelevant information and reduce the amount of data processing. The real scene image is selected to validate the method .Based on the above research results, the method improved the quality of disparity map.

Nowadays, the number of road accident in Malaysia is increasing expeditiously. One of the ways to reduce the number of road accident is through the development of the advanced driving assistance system (ADAS) by professional engineers. Several ADAS system has been proposed by taking into consideration the delay tolerance and the accuracy of the system itself. In this work, a traffic sign recognition system has been developed to increase the safety of the road users by installing the system inside the car for driver’s awareness. TensorFlow algorithm has been considered in this work for object recognition through machine learning due to its high accuracy. The algorithm is embedded in the Raspberry Pi 3 for processing and analysis to detect the traffic sign from the real-time video recording from Raspberry Pi camera NoIR. This work aims to study the accuracy, delay and reliability of the developed system using a Raspberry Pi 3 processor considering several scenarios related to the state of the environment and the condition of the traffic signs. A real-time testbed implementation has been conducted considering twenty different traffic signs and the results show that the system has more than 90% accuracy and is reliable with an acceptable delay.

In numerous applications, such as collision detection, Robotic handling, Robotic-based manufacturing facility, and Advanced Driver Assistance Systems (ADAS), depth estimation is crucial and the most significant task. Radar, Ultrasonic, Lidar technologies both operate by reflecting radio or sound wave or laser beams respectively. Stereo cameras used for depth estimation are costly and increase the cost of the system. There are a few Monocular camera depth estimate approaches that have evolved using mathematical calculation. One approach uses pixel to depth estimate mapping and the other uses the geometry of the road, the contact point of the vehicle on road, and camera properties. The proposed solution would implement depth detection using a monocular camera with deep learning. The objective of the proposed solution is to detect the depth from a monocular image and calibrate it with the actual depth. The pixel to distance data which derived from camera properties is used to run through varied hidden layers and nodes to conclude the implementation using Deep Learning (DL). Evidently, train and test data learn and converge more quickly on the deeper models which has more hidden layers and nodes.

Pedestrian detection constitutes a fundamental component of Advanced Driver Assistance Systems (ADAS). playing a pivotal role in ensuring road safety and reducing the risk of accidents. With the emergence of deep learning methodologies, significant progress has been made in the field of pedestrian detection, leading to the development of state-of-the-art detectors that exhibit impressive accuracy and efficiency in real-world scenarios. However, despite these advancements, there are ongoing challenges that need to be addressed to further enhance the performance and robustness of pedestrian detection systems, particularly in adverse environmental conditions such as varying lighting conditions, and adverse weather. Additionally, there is a growing demand for pedestrian detection systems to incorporate distance estimation capabilities, with an extension to include cyclists and riders, who are also crucial for ensuring overall road safety. To address these challenges, our research focuses on creating a stereovision system using a Raspberry Pi 4, designed to detect pedestrians, cyclists, and riders, and to estimate their 3D distances in real-time. The initial phase of our study involves enhancing the performance of the YOLOv5s through a fine-tuning process using a custom dataset and leveraging advanced augmentation techniques. Through rigorous evaluation and comparison with the original YOLOv5s model, we demonstrate a significant improvement in detection accuracy, exceeding 79%. The second phase of our study focuses on real-time depth estimation using stereo camera calibration techniques and triangulation methods. Through triangulation techniques involving object detection results, the actual distance estimation algorithm is validated in real-time for a single person. Thus, our research presents a comprehensive approach to person detection and distance estimation, leveraging the synergies between deep learning and stereovision technology.

Stereo vision finds a wide range of applications in automotive, object detection, robot navigation, agriculture mapping, and others. Stereo matching is a stereo algorithm targeted to identify the corresponding pixels from two or more images. This study shows the implementation of stereo matching using the Sum of Absolute Difference (SAD) algorithm to extract the object's depth or disparity from stereo images. Our key objective revolves around the implementation of the stereo matching algorithm for a small field-programmable gate array (FPGA)—requiring relatively few resources while maintaining the processing speed as well as disparity map. For meeting this requirement, we used small window buffers to compute the stereo matching. The occluded pixels were reduced by introducing secondary consistency checking implementation. From the results of experiments performed using the Zynq UltraScale+ ZCU102 FPGA board with SDSoC compiler, the processing speed for computing the stereo matching algorithm with a 4×4 window buffer was 0.038 s for an image size of 486×720 pixels and 0.051 s for 375×1242 pixels resolution. The proposed design needed 1% each of BRAM and FF and 7% of LUT. An 18% reduction in the pixel errors has been observed when employing the secondary consistency matching on the post-processing.

Stereo matching is a difficult and challenging task due to many uncontrollable factors that affect the results. These factors include the non-ideal radiometric conditions and the presence of weak-textured regions. A stereo matching algorithm including cost computation, cost aggregation, disparity computation, and disparity refinement is proposed to overcome these limitations. First, census, gradient, and HSL measures are combined as the pixel-wise matching cost to reduce the radiometric distortions. Second, a second-order smoothness constraint based on angle direction priors is utilized to improve the matching accuracy in the weak-textured regions. Then, the matching cost and smoothness constraint are applied to semi-global matching. Third, a winner-takes-all strategy is adopted to calculate the initial disparity maps. Finally, a coarse-to-fine simple linear iterative clustering superpixel segmentation is proposed to split the image into some regions with similar colors, adjacent pixels, and similar disparity planes. Through the segmentation result, boundary lines are classified into occluded, hinge, and coplanar lines. Slanted plane iterative optimization is performed to obtain the optimized disparity maps according to the segmentation result and boundary lines classification. The algorithm is evaluated in the Middlebury and KITTI datasets. The experimental results show that the method has good performance under non-ideal radiometric conditions and in the weak-textured regions. The disparity maps obtained by the method have a lower mismatching rate compared with other algorithms.

Autonomous and automated driving development has been making quick progress within the last years and many large automotive companies have announced initiatives to offer self-driving cars soon, some as early as 2020. During the first decades, these automated cars will have to share the road with human drivers. A way to increase safety and comfort for these automated vehicles and improve current advanced driver-assistance systems (ADAS), is to take the expected future behavior of other traffic participants into account. This increases the time a vehicle can use to react to changes in its environment and therefore allows for more comfortable or safe actions to be chosen. Relevant research areas to achieve this goal include computer vision, machine learning, situational awareness, decision making and many more. In this thesis, we contribute and evaluate new ideas for some of the challenges in the toolchain required for predicting the behavior of other traffic participants and deciding what to do with these. We used supervised machine learning techniques for both reactive predictions (short-term) and motivation-based predictions (long-term) and contribute to the question how the data used for machine learning can be labeled, comparing manual to semi-automated to automated approaches. This is especially interesting when subjective data is involved, e.g. at which point in time a motivation arose in a driver, even though he did not act on it yet. How to use and what to actually do with predictions in vehicles equipped with ADAS or automated vehicles is the next contribution. We evaluated many heuristics and a new, complex decision model based on decision theory. As use case, an adaptive cruise control system was enhanced by our prototype implementation of predictors and decision algorithms. We used an automated prototype research car in a case study and compared the effects on safety and comfort using objective data evaluation and subjective feedback from the study participants. We are able to show that even simple prediction and decision algorithms are able to improve the current status quo considerably and that the more advanced models work even better, but at the cost of substantial complexity increases.

Real-time 3D sensing plays a critical role in robotic navigation, video surveillance and human-computer interaction, etc. When computing 3D structures of dynamic scenes from stereo sequences, spatiotemporal stereo and scene flow methods can produce temporally coherent disparity. However, most existing methods do not utilize the previous disparity map sufficiently to compute the next disparity map, and the searching space of correspondences limits the speed of disparity computation for each image pair. This paper proposes an effective scheme to predict disparity maps from stereo sequences. In particular, we apply a robust 3D registration algorithm based on the angular-invariant feature to estimate the ego-motion of the stereo rig between consecutive frames, and present the transformation between consecutive disparity maps. The scheme can produce a sequence of temporally coherent disparity maps rapidly. We apply the new scheme to real outdoor scenes, and thorough empirical studies indicate the effectiveness of the new scheme for practical applications.KeywordsImage PairPattern AnalConsecutive FrameIterative Close PointStereo MatchThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Abstract: This project introduces a low-cost prototype of an Advanced Driver Assistance System (ADAS) that integrates Doppler radar with computer vision to enhance vehicle safety. The system is designed to detect obstacles in real time, estimate the speed of nearby vehicles, and recognize road lanes, thereby assisting drivers in navigating complex traffic situations. The Doppler radar sensor provides reliable velocity measurements even in poor visibility conditions such as fog or rain, while the vision module, based on lane detection algorithms, helps in maintaining proper road alignment. Together, these components form a sensor fusion framework that reduces accident risks by delivering accurate and timely driving assistance. The hardware implementation is based on affordable and widely available components, including the HB100 Doppler radar module, a standard camera, and a Raspberry Pi 4. The Raspberry Pi acts as the central processing unit, executing both radar signal processing and computer vision tasks. By eliminating the need for expensive automotive-grade systems, this design makes ADAS functionalities more accessible for low-cost applications, particularly in regions where advanced safety systems are not yet common. Testing in a simulated environment has shown promising results, demonstrating that the system performs effectively across varied weather and lighting conditions. Radar ensures robust speed detection regardless of external factors, while the camera enables contextual awareness of road structures. This synergy enhances reliability and sets the stage for future improvements. Further advancements may include machine learning-based object classification, integration of additional sensors like LiDAR, and optimization for real-world deployment. The outcomes suggest that combining Doppler radar and embedded vision can provide a practical pathway toward safer and smarter driving technologies..

Digitalization of ancient architectures is one of the effective means for the preservation of heritage structures, with 3D reconstruction based on computer vision being a key component of such digitalization techniques. However, Chinese ancient architectures are located in mountainous areas, and existing 3D reconstruction methods fall short in restoring the local structures of these architectures. This paper proposes a self-attention-guided unsupervised single image-based depth estimation method, providing innovative technical support for the reconstruction of local structures in Chinese ancient architectures. First, an attention module is constructed based on features extracted from architectural images learned by the encoder, and then embedded into the encoder-decoder to capture the interdependencies across local features. Second, a disparity map is generated using the loss constraint network, including reconstruction matching, smoothness of the disparity, and left-right disparity consistency. Third, an unsupervised architecture based on binocular image pairs is constructed to remove any potential adverse effects due to unknown scale or estimated pose errors. Finally, with the known baseline distance and camera focal length, the disparity map is converted into the depth map to perform the end-to-end depth estimation from a single image. Experiments on the our architecture dataset validates our method, and it performs well also well on KITTI.

Recent works have demonstrated superior results for depth estimation from a stereo pair of images using convolutional neural networks. However, these methods require large amounts of computational resources and are not suited to real-time applications on edge devices. In this work, we propose a novel method for real-time stereo matching on edge devices, which consists of an efficient backbone for feature extraction, an attention-aware feature aggregation, and a cascaded 3D CNN architecture for multi-scale disparity estimation. The efficient backbone is designed to generate multi-scale feature maps with constrained computational power. The multi-scale feature maps are further adaptively aggregated via the proposed attention-aware feature aggregation module to improve representational capacity of features. Multi-scale cost volumes are constructed using aggregated feature maps and regularized using a cascaded 3D CNN architecture to estimate disparity maps in anytime settings. The network infers a disparity map at low resolution and then progressively refines the disparity maps at higher resolutions by calculating the disparity residuals. Because of the efficient extraction and aggregation of informative features, the proposed method can achieve accurate depth estimation in real-time inference. Experimental results demonstrated that the proposed method processed stereo image pairs with resolution 1242 \(\times \) 375 at 12–33 fps on an NVIDIA Jetson TX2 module and achieved competitive accuracy in depth estimation. The code is available at https://github.com/JiaRenChang/RealtimeStereo.