Robust real-time eye detection and tracking under variable lighting conditions and various face orientations

Most eye trackers based on active IR illumination require distinctive bright pupil effect to work well. In this paper, we present a new real time eye tracking methodology that works under variable and realistic lighting conditions and various face orientations. By combining the conventional appearance based object recognition method (SVM) and object tracking method (mean shift) with Kalman filtering based on active IR illumination, our technique is able to benefit from the strengths of different techniques and overcome their respective limitations. Experimental study shows significant improvement of our technique over the existing techniques.

The process of eye tracking has been subject of intensive research during the last years. The main aim consists in the determination of the point of gaze or the eye movement relative to the head. In this work, we use a static video camera to realize the tracking of the eyes movement mainly due to its important features such as low cost, fully automated and easy application. The proposed approach encompasses a solution to the problems of eye detection and eye tracking. Our eye detection method employs a set of cascaded Haar classifiers that enable the fast recognition of eye regions in a real-time sequence of images. On the other hand, the eye tracking method uses particle filtering that is robust in clutter environments and allows the fusion of sensor data as well as the tracking of nonlinear systems. The results demonstrate that the eye tracking system is capable of robustly track the eyes under variable light conditions and under several different face orientations.

New luminescent coatings are increasingly being used in highway tunnels to address inadequate internal lighting conditions. However, there is currently a lack of scientifically reliable methods to evaluate the effectiveness of these paints in improving lighting conditions, reducing driver psychological stress, and quantifying these impacts. This study utilized new luminescent coatings to improve tunnel lighting conditions, conducting real-vehicle tests to measure drivers’ physiological parameters including pupil diameter and heart rate. It examined the mechanisms through which variations in lighting conditions within highway tunnels affect the psychological workload of drivers. A hierarchical analysis–fuzzy comprehensive evaluation (AHP-FCE) method was adopted to develop a quantitative evaluation system for highway tunnel driving psychological load. The results indicate that variations in tunnel lighting substantially influence the psychological workload experienced by drivers during operation. The new luminescent coatings effectively enhanced tunnel lighting conditions, increasing average brightness and illuminance by 37.71% and 40.95%, respectively. Following lighting improvements, pupil diameter variation rates during tunnel entry and exit phases decreased by 35.37% and 10.06%, respectively, while heart rate variation rates decreased by 12.50% and 4.36%. Quantitative analysis of driver mental load revealed a comprehensive score of 0.6230 before lighting enhancement, which decreased to 0.2702 after improvements. This research introduces an innovative integrative framework that combines physiological parameter monitoring with the AHP-FCE method to quantitatively assess the psychological workload experienced by drivers in tunnel environments. This approach addresses a significant gap in the literature concerning the quantitative relationship between tunnel lighting optimization and drivers’ psychological workload responses.

Driver fatigue is a leading cause of road accidents, particularly during long-distance driving, and necessitates reliable, real-time detection methods to prevent safety compromises. This study presents a drowsiness detection system designed to provide early warnings by analyzing eye-blinking patterns through real-time facial recognition and eye tracking using an in-vehicle camera. The system leverages MediaPipe to extract facial landmarks and compute the Eye Aspect Ratio (EAR), a geometric measure that quantifies eye openness. By employing a threshold-based decision mechanism, the system triggers audible alerts when EAR values indicate signs of drowsiness, such as prolonged or frequent eye closures. Evaluated on a dataset of 667 driver images, the system achieved an accuracy of 85% in drowsiness detection, with an optimal EAR threshold of 0.289 identified for maximizing performance. The approach demonstrates robustness against common real-world challenges, including variations in lighting conditions and camera positioning, offering a non-intrusive, adaptive, and highly reliable solution. By focusing on eye-based biometric features derived from facial landmarks, this system contributes an original and practical advancement in driver monitoring technology, with significant potential to reduce fatigue-related accidents and enhance overall road safety.

This paper describes a robust method for discriminating 3D face pose (face orientation) from a video sequence featuring views of a human head under variable lighting and facial expression conditions. The wavelet transform is used to decompose the image into multiresolution face images containing both spatial and spatial-frequency information. Principal component analysis (PCA) is used to project a low-resolution sub-band face pose onto a pose eigen-space where the first 3 eigen coefficients are found most sensitive to pose and follow a trajectory as the pose changes. Any unknown pose of an query image can then be estimated by finding the Euclidean distance of the first 3 eigen coefficients of the query image from the estimated trajectory. The wavelet transform reduces the computational load on the PCA and makes the algorithm robust against illumination changes and facial expression. An accuracy of 84% was obtained for test images unseen during training under different environment conditions, facial expressions, and even different human subjects.

Abstract. This paper addresses the significant challenge faced by robotic vision systems in detecting and sorting objects accurately under varying lighting conditions. Such variations in light can lead to decreased detection accuracy and inefficiencies in automated sorting processes. The paper employs a combination of literature review and experimental validation to investigate the effectiveness of advanced image processing techniques and control algorithms. Specifically, it explores the application of CLAHE adaptive compensation, Gaussian Blur, custom ROI, and PID controllers within a visual object sorting system to improve its robustness under diverse lighting conditions. The use of CLAHE and Gaussian Blur effectively compensates for uneven lighting, while custom ROI and PID controllers further optimize the system's response to fluctuating conditions.

Realistic representation of objects requires models which can synthesize the image of an object under all possible viewing conditions. We propose to learn these models from examples. Methods for learning surface geometry and albedo from one or more images under fixed posed and varying lighting conditions are described. Singular value decomposition (SVD) is used to determine shape, albedo, and lighting conditions up to an unknown 3×3 matrix, which is sufficient for recognition. The use of class-specific knowledge and the integrability constraint to determine this matrix is explored. We show that when the integrability constraint is applied to objects with varying albedo it leads to an ambiguity in depth estimation similar to the bas relief ambiguity. The integrability constraint, however, is useful for resolving ambiguities which arise in current photometric theories.KeywordsSingular Value DecompositionObject ClassLighting VariationReflectance FunctionIntegrability ConstraintThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Computer vision techniques are becoming increasingly popular for monitoring pig behavior. For instance, object detection models allow us to detect the presence of pigs, their location, and their posture. The performance of object detection models can be affected by variations in lighting conditions (e.g., intensity, spectrum, and uniformity). Furthermore, lighting conditions can influence pigs' active and resting behavior. In the context of experiments testing different lighting conditions, a detection model was developed to detect the location and postures of group-housed growing-finishing pigs. The objective of this paper is to validate the model developed using YOLOv8 detecting standing, sitting, sternal lying, and lateral lying pigs. Training, validation, and test datasets included annotation of pigs from 10 to 24wk of age in 10 different light settings; varying in intensity, spectrum, and uniformity. Pig detection was comparable for the different lighting conditions, despite a slightly lower posture agreement for warm light and uneven light distribution, likely due to a less clear contrast between pigs and their background and the presence of shadows. The detection reached a mean average precision (mAP) of 89.4%. Standing was the best-detected posture with the highest precision, sensitivity, and F1 score, while the sensitivity and F1 score of sitting was the lowest. This lower performance resulted from confusion of sitting with sternal lying and standing, as a consequence of the top camera view and a low occurrence of sitting pigs in the annotated dataset. This issue is inherent to pig behavior and could be tackled using data augmentation. Some confusion was reported between types of lying due to occlusion by pen mates or pigs' own bodies, and grouping both types of lying postures resulted in an improvement in the detection (mAP = 97.0%). Therefore, comparing resting postures (both lying types) to active postures could lead to a more reliable interpretation of pigs' behavior. Some detection errors were observed, e.g., two detections for the same pig were generated due to posture uncertainty, dirt on cameras detected as a pig, and undetected pigs due to occlusion. The localization accuracy measured by the intersection over union was higher than 95.5% for 75% of the dataset, meaning that the location of predicted pigs was very close to annotated pigs. Tracking individual pigs revealed challenges with ID changes and switches between pen mates, requiring further work.

The advancement of eye tracking technology, including improved portability, usability, and affordability, has facilitated the transition of eye trackers from controlled laboratory settings to diverse real-world environments. One significant challenge in these real-world settings is the wide variation in illumination levels, potentially affecting eye tracking accuracy. Despite anecdotal evidence suggesting the sensitivity of head-mounted eye trackers to environmental light conditions, systematic research on this topic has been limited. This study introduces a standardized protocol for investigating the influence of lighting conditions on the accuracy of head-mounted eye trackers. We examined the performance of SMI Eye Tracking Glasses 2 Wireless under three distinct illumination levels: Low (Dark), Moderate (Normal), and High (Bright). Our experiment involved 9 participants performing accuracy tests under each lighting condition. We measured gaze accuracy while varying the illumination level during both the calibration and data recording phases. Results indicate that eye tracking accuracy is affected by the illumination level used during data collection. Best accuracy was achieved when data recording was performed under the same illumination level as the calibration. Notably, calibration under moderate illumination led to the most consistent performance across all lighting conditions. These findings provide valuable insights for researchers and practitioners using head-mounted eye trackers in varying lighting environments. The standardized protocol presented can be applied to evaluate other eye tracking systems, contributing to developing more robust mobile eye tracking techniques. Future research should investigate the generalizability of these results across different eye tracking technologies and explore potential software solutions to mitigate illumination-related accuracy issues.

To achieve accurate detection of the pineapple fruit picking area and pose under complex backgrounds and varying lighting conditions, this study proposes a pineapple keypoint detection model (LTHRNet) based on an improved LiteHRNet. Image data of pineapple fruits under different lighting conditions were collected, and six keypoints were defined to characterize the morphological features of the fruit. In the model design, LTHRNet incorporates the LKA_Stem module to enhance initial feature extraction, the D-Mixer module to capture both global and local feature relationships, and the MS-FFN module to achieve multi-scale feature fusion. In addition, the model employs parallel sub-networks with different resolutions to maintain high-resolution feature information and improve the precision and spatial accuracy of keypoint detection. Experimental results show that LTHRNet performs well in pineapple keypoint detection. It achieves 93.5% and 95.1% in KAP0.5 and KAR0.5, respectively, outperforming other models in terms of detection accuracy and robustness under challenging lighting and occlusion conditions, with a detection speed of 21.1 fps. For pose estimation, the average offset angle (AOA) of LTHRNet is 2.37°, which is significantly lower than that of other models. In summary, the proposed LTHRNet model demonstrates high accuracy and strong robustness in pineapple keypoint detection and pose estimation, providing reliable keypoint localization and pose estimation data for pineapple harvesting, while also offering an effective reference for pose recognition in other fruit-picking tasks.

Acquiring high-quality images in diverse lighting conditions is challenging, particularly in the constantly evolving driving landscape. From the harsh brilliance of daylight to the subtle shadows of nighttime scenarios, the complexity of capturing accurate images intensifies. Therefore, addressing exposure issues across various lighting conditions is imperative. This study introduces a novel multi-task end-to-end dynamic illumination adjustment approach termed CPGA-DIA. By incorporating feature selection and gamma factor correction techniques, our method effectively handles extreme variations in light conditions for both low-light image enhancement and exposure correction. Comprehensive experimental evaluations demonstrate superior performance to existing processes, showcasing our method’s adaptability and robustness in real-world driving scenarios. Furthermore, our approach extends its capabilities to include the control and estimation of explainable factors related to changing lightness, enabling our model to understand better the environmental conditions that influence image perception and clarity. This research contributes to advancing image capture methodologies, particularly in dynamic lighting conditions, with promising implications for applications ranging from photography to various other fields.

Accurate grape detection in orchards is a core link in realizing automated harvesting. To address the challenges in orchard environments, such as complex grape backgrounds, variable lighting conditions, and dense occlusion of fruits, this study proposes a highly robust real-time grape detection model for orchard scenarios, namely Grapevine Ultra-Lightweight YOLO (GrapeUL-YOLO). Based on YOLOv11, this model enhances detection performance through three innovative designs: firstly, it adopts a Cross-Scale Residual Feature Backbone (CSRB) as the feature extraction network, combining downsampling operation with modules such as C3k2_SP and SPPELAN, which reduces computational complexity while retaining multi-scale features of grapes from small clusters to entire clusters; secondly, it constructs an Adaptive Bidirectional Fusion Network (ABFN) in the detection Neck, and through CARAFE content-aware upsampling and a bidirectional cross-scale concatenation mechanism, it strengthens the interaction between spatial details and semantic information, thereby improving the feature fusion capability in scenes with dense occlusion; thirdly, it designs a shape-adaptive detection Head, which uses customized elliptical anchor boxes to match the natural shape of grapes and detects grape targets of different sizes according to scale division. Experimental results show that on the Embrapa WGISD dataset, the mAP@0.5 of GrapeUL-YOLO reaches 0.912, and the mAP@0.5:0.95 is 0.576, both outperforming 9 mainstream models including CenterNet and YOLOv11; meanwhile, the model has only 5.11M parameters and an average detection time of 16.9ms per image, achieving a balance between high precision and lightweight, and providing an efficient solution for automated grape detection and harvesting in orchards.

Over the last several years there has been a renewed interest in the automation of harvesting of fruits and vegetables. The two major challenges in the automation of harvesting are the recognition of the fruit and its detachment from the tree. In this paper, a machine vision algorithm for the recognition of citrus fruits is presented. The algorithm consists of segmentation, region labeling, size filtering, perimeter extraction and circle detection. Evaluation of the algorithm consisted of images taken inside the canopy (varying lighting condition) and on the canopy surface. Results showed that more than 90% of the fruits were detected in the 110 images tested. In addition, the proposed segmentation was able to deal with varying lighting condition and the circle detection method proved to be effective in detecting fruits in clusters. The development of this algorithm with its capability of detecting fruits in varying lighting condition and occlusion would enhance the overall performance of robotic fruit harvesting.

Many previous works on photometric stereo have shown how to recover the shape and reflectance properties of an object using multiple images taken under a fixed viewpoint and variable lighting conditions. However, most of them only dealt with a single point light source in each image. In this paper, we show how to perform photometric stereo with four images which are taken under distant but general lighting conditions. Our method is based on the representation that uses low-order spherical harmonics for Lambertian objects. Attached shadows are considered in this representation. We show that the lighting conditions can be estimated regardless of object shape and reflectance properties. The estimated illumination conditions can then help to recover the shape and reflectance properties.

This paper proposes a method of tracking an object for a person-following mobile robot by combining a monocular camera and a laser scanner, where each sensor can supplement the weaknesses of the other sensor. For human-robot interaction, a mobile robot needs to maintain a distance between a moving person and itself. Maintaining distance consists of two parts: object tracking and person-following. Object tracking consists of particle filtering and online learning using shape features which are extracted from an image. A monocular camera easily fails to track a person due to a narrow field-of-view and influence of illumination changes, and has therefore been used together with a laser scanner. After constructing the geometric relation between the differently oriented sensors, the proposed method demonstrates its robustness in tracking and following a person with a success rate of 94.7% in indoor environments with varying lighting conditions and even when a moving object is located between the robot and the person.