Video-Based Driver Assistance--From Basic Functions to Applications

This paper presents stereo vision-based vehicle detection approach on the road using a road feature and disparity histogram. It is not easy to detect only vehicles robustly on the road in various traffic situations, for example, a nonflat road or a multiple-obstacle situation. This paper focuses on the improvement of vehicle detection performance in various real traffic situations. The approach consists of three steps, namely obstacle localization, obstacle segmentation, and vehicle verification. First, we extract a road feature from v-disparity maps binarized using the most frequent values in each row and column, and adopt the extracted road feature as an obstacle criterion in column detection. However, many obstacles still coexist in each localized obstacle area. Thus, we divide the localized obstacle area into multiple obstacles using a disparity histogram and remerge the divided obstacles using four criteria parameters, namely the obstacle size, distance, and angle between the divided obstacles, and the difference of disparity values. Finally, we verify the vehicles using a depth map and gray image to improve the performance. We verify the performance of our proposed method by conducting experiments in various real traffic situations. The average recall rate of vehicle detection is 95.5%.

Traffic congestion has been causing many setbacks and challenges in the major developing and developed smart cities in India and all over the world. The typical traffic light sequence works on the specific switching of Red, Green and Yellow lights in a particular way with stipulated time irrespective of peak traffic hour and situation, this makes typical traffic light sequence method inefficient for present situation where density of traffic is exponentially increasing over the time. In this work, we propose and develop a design of Intelligent Traffic Light Control System to manage the road traffic according to the real time traffic situation. This work is intended to propose methods of dynamic time allocation for traffic lights according to traffic density and situation. In this paper, we acquired real time traffic density using image sensors and obstacle detector, where according to the traffic density, traffic controller dynamically generates “ON” and “OFF” time of traffic light sequence. The EDGE Spartan 6 FPGA (Field Programmable Gate Array) development board is programmed as controller to implement the proposed work and MATLAB programming in used to capture real time traffic image and generate dynamic time for the traffic light sequence to control the traffic situation in real time.

Since the 1990s, there have been various lane detection systems designed to suit various road conditions such as highways, urban and rural roads. Current reserach has shown to predominantly detect only 1 lane marking set in real time and is unable to provide additional lanes for support in situations such as lane closures, road work conditions and car accidents that may obstruct the driving lane. These driver assistance systems are limited in their ability to assist the driver in these conditions. In this paper we propose a method to determine the markings of 2 lanes which can be used in conjunction with a detection system to detect obstacles present in front of the driver on the road. We first determine a suitable threshold for the perspective image to extract these road markings and signs, then apply morphological transformations to counter possible ‘deviations’ that may arise in this feature extraction technique. This method provides a robust approach to lane detection and works considerably well in various weather conditions. The resulting images show that the method developed can be used for lane-departure warning, as well as for obsatcle detection, in driver assistance.

Advanced Driver Assistance Systems (ADAS) and Collision Avoidance Systems (CAS) are the primary modules of modern human-centric and autonomous driving applications, such as forward and rear-end collision warnings. To enhance the performance of ADAS and CAS systems in foggy weather, an Efficient and Cost-Effective Vehicle Detection and Tracking with Driver Assistance (ECE-VDTDA) system is proposed. The proposed ECE-VDTDA system comprises vehicle detection, tracking, and driver assistance modules. An optimized SimYOLO-V5s_WIOU vehicle detection algorithm is proposed, based on the SimSPPF module, the baseline You Only Look Once (YOLO) algorithm (YOLO-V5s), and the Wise Intersection Over Union (WIOU) localization loss function. State-of-the-art Deep-SORT, Strong-SORT, and optimized Deep-SORT algorithms are utilized for vehicle tracking. The vehicle detection and tracking performance of the ECE-VDTDA system is rigorously evaluated on DAWN, foggy driving, foggy cityscapes, BDD100K, web-collected, and self-collected foggy weather datasets. Optimized SimYOLO-V5s_WIOU algorithm outperformed on the foggy driving dataset with a 17.45% increase in mAP50, and foggy cityscapes dataset with a 0.32%, 1.05%, 1.58%, 2%, 0.54% increase in the multiclass mAP50, mAP50-95, F1 score, precision, and recall scores, respectively, compared to the baseline YOLO-V5s. Furthermore, the SimYOLO-V5s_WIOU algorithm also outperformed the state-of-the-art methods and enables Deep-SORT, Strong-SORT, and optimized Deep-SORT vehicle tracking algorithms to track vehicles with high confidence. The driver assistance module of the ECE-VDTDA system helps prevent imminent road collisions in foggy weather by estimating distance, speed, and time-to-collision and by issuing timely collision warnings. The experimental results demonstrate the robustness and computational efficiency of the proposed ECE-VDTDA system.

Traffic systems are man-made systems in which the human component as a complex factor plays a significant part. [1] The aim of the research was to offer a method which is applicable for setting the real traffic situations at a signalized intersection against the simulated ones, using a traffic model with similar statistical features. The simplest technical apparatus has been used to monitor the real traffic situations. Only one video camera is needed for data collection and it is located in the observable area. Hence time-dependent parameters have been collected which are available for the analysis of the traffic situations. In the case of both traffic models the analysis is based on a behavior model of inhomogeneous driver-populations and an estimation method. The behavior model of inhomogeneous driver-populations handles the time-dependent parameters of driver behavior as the most important predictors, describes the movement and the behavior of the inhomogeneous driver-populations approaching a signalized intersection, passing the amber signal light. The estimation method developed using the model parameters opens the door to estimating the composition of drivers and measuring the capacity of an intersection on a more sensitive scale.

Currently available driver assistance systems (i) warn the driver based on vehicle state sensors (e.g., door open, outside temperature near or below the freezing point), (ii) offer route guidance information (navigation systems based on GPS and digital road maps), or—in some critical situations—(iii) even actively influence vehicle handling under carefully delimited conditions (anti-blocking-system, electronic-stability-program). This contribution reports about investigations to combine passive GPS- and map-based route guidance with model-based machine vision in order to automatically assess or even execute driving maneuvers in inner-city traffic situations. Information provided by todays route guidance systems is treated as a generic description of lane structure. The schematic description of lane structures extractable from commercially available standard digital maps is automatically instantiated by a machine vision approach which interprets video image sequences recorded by cameras from within a driving vehicle. The resulting model of the lane structure in front of the vehicle is subsequently exploited in order to control vehicle maneuvers in real-time as a proof of principal system competences. The machine-vision-based execution of driving maneuvers can at any time be overridden by the driver.

The objective of this research is to develop an advanced driver assistance system with lane departure warning and forward collision warning functions. The main input of this system is a CMOS camera, which is used to acquire roadway image in front of vehicle. In order to extract lane markings and vehicles from roadway image, the image processing methods such as coordinate systems transformation, object detection and object tracking are applied to recognize the lane boundaries and the preceding vehicles. In lane marking recognition, gray scale statistics, dynamic range of interesting (ROI) and featured-based approaches are used to detect lane boundaries successfully. For vehicle recognition, a Sobel edge-enhancement filter and optical flow algorithm are used to implement vehicle detection and tracking. Besides, by means of the roadway and headway physical estimation models, the circumstances of lane departure and forward collision can be detected. The experimental results indicate that the hardware and the implemented algorithm used in this research are able to recognize the lane markings and headway distance precisely, and meet the requirements of real-time computing and high reliability.

Before widespread automated driving, driver assistance systems become more common and can deliver safety benefits. This study focuses on current and future lane keeping assistance (LKA) systems and their potential safety effects by analysing real-world crash data and LKA's possibilities to prevent fatal passenger car crashes. The analysed data includes in-depth analysis of 364 fatal head-on and single-vehicle crashes in Finland. According to statistical analysis, 27% of these crashes could potentially have been prevented had currently available LKA been deployed. In this study, many issues are examined related to the lane markings' visibility, infrastructure, weather, and driver-related risks, which rationalise the results, compliment previous research, and highlight possibilities to increase the safety potential of LKA. Related to the lane markings' visibility, safety potential could be improved by allocating more resources to road maintenance and to snow clearance. Advanced LKA enabling exploiting digital lane markings could hypothetically almost double the safety potential as weather conditions or the visibility of lane markings would not restrict LKA's operation. While the driver is still in charge of driving tasks, driver-related risks are difficult to manage and the crashes caused intendedly or due to driver's attack of illness are not preventable by current LKA systems.

In order to get a better understanding of how driving with advanced driver assistance (ADA) systems effects traffic flow in terms of safety, throughput and environment in practice a field operational test (FOT), called "the assisted driver" was conducted by the dutch road authority Rijkswaterstaat in The Netherlands. The main component of this project was the so-called full traffic FOT in which 20 cars, equipped with adaptive cruise control (ACC) and lane departure warning, were driven in mixed traffic for five months. During this period a vast amount of data was collected by installed data-loggers in order to perform an objective impact assessment. The results are quite promising. Driving with ACC and LDW improves traffic safety with approximately 8% and fuel consumption decreases with 3%. Associated emissions can decrease up to 10% and there seems to be no direct negative effect for throughput.

Lane marking detection is a basic task of Driver Assistance Systems (DAS) and Autonomous Land Vehicle (ALV). In order to improve the accuracy of lane marking detection, we design a rough lane marking locating method based on predecessors’ work. Considering the characteristic of lane markings, we extract Haar-like features of lane marking regions and train a strong cascade classifier by Adaboost Algorithm. The classifier is simple in principle and can fast locate the possible areas of lane markings accurately. Experimental results show that our method performs well.KeywordsLane marking locatingAdaboost algorithmHaar-like features

Research and/or Engineering Questions/Objective The vehicle of the future will support its driver by advising him regarding potential hazards. Essential prerequisite therefore is the sensor based perception of the traffic situation. For the recognition of traffic related objects, camera based sensors, deepness cameras, vehicle sensors as well as radar and lidar sensors are used. For the future development of ADAS the fusion of multiple sensor data to a consistent environmental picture will play a key role. The evaluation approach of real world driving tests will no longer be sufficient due to the complexity of the system interactions. New simulation methods are needed to evaluate ADAS by using virtual test driving with realistic vehicle behavior and complex traffic environment. Methodology Therefore it is important to integrate camera based components in a “closed loop”-simulation platform to be able to test sensor data fusion technologies under realistic conditions. To test new driver assistance systems in a simulation environment today animation data is filmed, subsequently this data is used to test an image processing algorithm or a fusion algorithm. But this method cannot be applied if wide-angle cameras such as cameras with fisheye lenses will be used. Within a research frame work for autonomous driving functions a new simulation technology was developed to integrate virtual cameras beside the well know environment sensor in the vehicle dynamic simulation CarMaker. For this purpose the real-time animation was extended with a sophisticated virtual camera model so called “VideoDataStream” to generate simultaneous video data (also PMD for 3D images). The camera positions as well as the camera properties could be applied individually. Additionally it is possible to freely define the type of the camera lens (e.g. fisheye) with lens settings like opening angle and the typical lens failures (e.g. distortion and vignetting). With this new technology it is possible that e.g. camera and radar data can be provided time and place synchronal for the fusion algorithm which should be tested! Results The video data could be used for evaluating image processing and sensor data fusion in Model-/Software-/Hardware-in-the-Loop applications within virtual test driving conditions. Here the created method and examples of image based perception of the vehicle environment as well as sensor data fusion algorithms shall be presented. Among others this covers first of all the recognition of traffic lanes, traffic signs and other traffic partners as well as the fusion of the single information up to a comprehensive environment picture. A further field of application will be the conjunction with navigation systems and digital maps, by which the virtual vehicle supports the navigation system with related GPS position and gets back the “MPP—Most Probable Path” with the “electronic horizon”, which is a type of predictive sensor, with all related preview information in front of the vehicle which are defined in the ADASIS protocol. Conclusion By using the introduced method the capability and efficiency of function development and testing in the area of Advanced Driver Assistant Systems will significantly be improved. Due to a powerful simulation environment a broad range of validation tests can be shifted into simulation because also complex test scenarios can be replicated and the tests are reproducible. The simulation data can be provided time and place synchronal, which is absolutely important, e.g. for a fusion algorithm which should be tested.

A New Human-Computer-Interface for High-Speed-Maglev Train Traffic Supervision

About 1.17 million people die in road crashes around the world each year. It is estimated that up to 30% of these fatalities are caused by fatigue and inattention. There are systems able to detect what is happening outside of the car, e.g., lane tracking, obstacle detection, pedestrian detection etc. Further on, there are also means for monitoring the actions of the driver. A natural step is to fuse the available data from within and outside of the car, and suggest a suitable response. This paper discusses driver assistance systems, lists a set of necessary core competencies of such a system and in particular presents a system for force-feedback in the steering wheel when crossing lanes. The presented system utilises a robust lane tracker which is experimentally evaluated for the purpose of driver assistance. In addition, preliminary results from simultaneous driver monitoring and lane tracking are presented that indicates a good correlation between the two, i.e. the driver's gaze direction and the structure of the road. These data can in turn be used for more advanced driver assistance systems in the future.

Summary form only given. One of the most significant large-scale deployment of intelligent systems in our daily life nowadays involves driver assistance in smart cars. Driver assistance systems use sensors to monitor the car surroundings or interior. They warn the driver in case of pending danger, and even exert automatic vehicle control if necessary. As such, they have major potential to reduce accidents. My talk starts with an overview of currently available driver assistance. I move on to our longstanding research on active pedestrian protection, presenting, among others, a prototype system for automatic vehicle braking and evasive steering. I conclude with some thoughts on future autonomous cars.

Automatic multi-vehicle tracking using video cameras: An improved CAMShift approach