Integrating Motion and Illumination Models for 3D Tracking

The use of video sequences for face recognition has been relatively less studied than image-based approaches. In this paper, we present a framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting and shape in generating an image using a perspective camera. This result can be used to estimate the pose and illumination conditions for each frame of the probe sequence. Then, using a 3D face model, we synthesize images corresponding to the pose and illumination conditions estimated in the probe sequences. Similarity between the synthesized images and the probe video is computed by integrating over the entire sequence. The method can handle situations where the pose and lighting conditions in the training and testing data are completely disjoint.

In human-human communication we use verbal, vocal and non-verbal signals to communicate with others. Facial expressions are a form of non-verbal communication, recognizing them helps to improve the human-machine interaction. This paper proposes a system for pose- and illumination-invariant recognition of facial expressions using near-infrared camera images and precise 3D shape registration. Precise 3D shape information of the human face can be computed by means of Constrained Local Models (CLM), which fits a dense model to an unseen image in an iterative manner. We used a multi-class SVM to classify the acquired 3D shape into different emotion categories. Results surpassed human performance and show poseinvariant performance. Varying lighting conditions can influence the fitting process and reduce the recognition precision. We built a near-infrared and visible light camera array to test the method with different illuminations. Results shows that the near-infrared camera configuration is suitable for robust and reliable facial expression recognition with changing lighting conditions.

The automatic recognition of human faces presents a significant challenge to the pattern recognition research community. Typically, human faces are very similar in structure with minor differences from person to person. They are actually within one class of "human face". Furthermore, lighting conditions change, while facial expressions and pose variations further complicate the face recognition task as one of the difficult problems in pattern analysis. This paper proposes a novel concept: namely, that faces can be recognized using a line edge map (LEM). The LEM, a compact face feature, is generated for face coding and recognition. A thorough investigation of the proposed concept is conducted which covers all aspects of human face recognition, i.e. face recognition under (1) controlled/ideal conditions and size variations, (2) varying lighting conditions, (3) varying facial expressions, and (4) varying pose. The system performance is also compared with the eigenface method, one of the best face recognition techniques, and with reported experimental results of other methods. A face pre-filtering technique is proposed to speed up the search process. It is a very encouraging to find that the proposed face recognition technique has performed better than the eigenface method in most of the comparison experiments. This research demonstrates that the LEM, together with the proposed generic line-segment Hausdorff distance measure, provides a new method for face coding and recognition.

The paper aims to propose a new approach towards human posture reconstruction and animation from monocular video sequences that contain any kind of human postures and movements. This is a way towards low cost motion capture and at the same time it avoids many limitations of those classical methods. A parameterized human skeleton model based on anatomy is adopted where the angular constraints are encoded in the joints. Criterion Function is defined to represent the residuals between feature points in the monocular image and the corresponding points resulted from projecting the human model to the projection plane. By transforming each segment of the human model to achieve the minimum value of the Criterion Function, the proper human posture that resembles the one represented by the monocular image can be generated. Different kinds of adjustments are utilized to adjust the body parts into the proper locations and orientations in 3D space without camera calibration. In order to find the optimal solution effectively in a high-dimensional parameter space by considering all the parameters simultaneously, the method of Genetic Algorithms is proposed. A procedure is developed to recover the whole body posture, and then a human animation system is developed to animate a series of human movements from monocular image sequences, during which information between consecutive frames is considered to improve the accuracy. Our technique makes it feasible to reconstruct any possible human postures, and experimental results from many monocular images and video sequences are encouraging.

Visual information extracted from RGB images has been successfully used for mobile robot localization. The main difficulty with localization using RGB images is that visual features from RGB images are not completely invariant against changes in viewpoints and lighting conditions. This problem can be overcome using features from RGB-D images. In this paper, we evaluate two depth features, depth patches and histograms of oriented normal vectors, extracted from RGB-D images for localization of a mobile robot and demonstrate that robust localization is possible under varying lighting conditions.

This study evaluates the generalization capabilities of computer vision (CV) models for detecting cow locations in indoor free-stall barns, with a focus on quantifying the resources required for effective deployment across different farms where camera angles and lighting conditions vary. Given the increasing reliance of precision livestock farming (PLF) on object localization techniques to monitor livestock health and optimize resource management, understanding how to deploy CV models with minimal resources while maintaining high performance is critical. To assess generalization, we investigated three key hypotheses: (1) Model generalization is equally influenced by changes in lighting conditions and camera angles; (2) Higher model complexity guarantees better generalization performance; and (3) Fine-tuning with custom initial weights trained on related tasks enhances final performance compared to using pre-trained weights. The study utilized the newly released COws LOcalization (COLO) dataset, comprising 1,254 images and 11,818 cow instances captured from five distinct imaging settings with varying camera angles and lighting conditions. The first hypothesis was tested by training models on images from specific camera angles and evaluating performance on new angles. The second hypothesis was examined by comparing the performance of five different model sizes (e.g., YOLO11n with 2.6M parameters vs. YOLO11x with 56.9M parameters) on the same dataset. The third hypothesis was evaluated by pre-training a model on a subset of images from a specific barn and fine-tuning it on another subset from a similar setting to simulate transferring the training outcome to a different location within the same barn. Results indicate that changes in lighting conditions have a negligible impact on model generalization, whereas shifting from a top-view to a side-view perspective significantly reduces performance (P < 0.001). Model complexity does not always correlate with improved generalization, as optimal complexity depends on the dataset and task. Additionally, fine-tuning with custom initial weights benefits small, simple models but provides no significant (P > 0.05) advantage for larger models compared to training from scratch. As artificial intelligence adoption grows, alongside rising concerns about its environmental impact, this study provides practical guidelines for PLF researchers to optimize CV model deployment with minimal resource consumption. It also highlights key generalization challenges and leverages the COLO dataset to drive further research in resource-efficient livestock monitoring.

Video tracking uses the semantic information between video image sequences to process, analyze and study the target to achieve target tracking. In response to the problems of sharp changes in target position, large deformation, and occlusion due to similar background interference, changes in lighting conditions, and different shapes of targets in the target tracking process, the tracking algorithm is less accurate and less robust in target appearance. An improved target tracking algorithm based on multi-domain network (MDNet) is proposed to solve this problem. By adding the Image-Align layer to the video target tracking task, a more accurate target value is obtained; applying the Directed Acyclic Graph-Recurrent Neural Network (DAG-RNN) by combining it with a convolutional neural network and modeling image neighborhood context dependencies for the target region to be tracked, we improve the problem that conventional networks only perform multi-layer extraction for the appearance features of the target, thus resulting in poor robustness to changes in target appearance; ROI Align layer is added after the convolution layer to speed up target feature extraction.

Enabling FIRA medium-sized soccer robots to recognize target objects according to color information requires that competing teams manually set the range of colors according to ambient lighting conditions prior to games. This color information is used to differentiate features of target objects, such as the ball, the goals, and the field. Constructing a color-feature model such as this is extremely time-consuming and the resulting model is unable to adapt dynamically to changes in lighting conditions. This study applied a look-up table method to execute RGB-HSV color space conversion to accelerate video processing. A particle swarm optimization (PSO) scheme was developed to detect the color-feature parameters of the target objects in the HSV color space. This enables the automatic completion of color-feature modeling and the construction of the knowledge model required by the robot for object recognition. Experiment results demonstrate that the proposed method is capable of enhancing the robustness of the robot vision system in determining changes in lighting conditions. In addition, the manpower and time required to set robot parameters prior to games were reduced significantly.

Over the recent years, a great deal of effort has been made to age estimation from face images. It has been reported that age can be accurately estimated under controlled environment such as frontal faces, no expression, and static lighting conditions. However, it is not straightforward to achieve the same accuracy level in real-world environment because of considerable variations in camera settings, facial poses, and illumination conditions. In this paper, we apply a recently-proposed machine learning technique called covariate shift adaptation to alleviating lighting condition change between laboratory and practical environment. Through real-world age estimation experiments, we demonstrate the usefulness of our proposed method.

Robust real-time eye detection and tracking under variable lighting conditions and various face orientations

Accuracy of skin segmentation algorithms is highly sensitive to changes in lighting conditions. When the lighting condition in a scene is different from that in the training examples, miss-classification rate of the skin segmentation algorithms is high. Using color constancy approach we aim to compensate for skin color variations to achieve accurate skin color segmentation. Skin color constancy is realized in an unsupervised manner by using the color changes observed on a face for different illuminations to drive the model. By training on few faces of different ethnicities, our model is able to generalize the color mapping for any unseen ethnicity. The color changes observed are used to learn the color mapping from one lighting condition to the other. These mappings are represented in a low dimensional subspace to obtain basis vector fields. Using these basis vector fields we can model the nonlinear color changes to transform skin colors in arbitrary lighting conditions to a reference lighting condition. We show the proof of concept of unsupervised skin color constancy on faces from the PIE database. Skin segmentation is performed on the color compensated faces using a Skin Distribution Map (SDM), which is trained on skin colors in reference lighting condition.

Computer vision techniques are becoming increasingly popular for monitoring pig behavior. For instance, object detection models allow us to detect the presence of pigs, their location, and their posture. The performance of object detection models can be affected by variations in lighting conditions (e.g., intensity, spectrum, and uniformity). Furthermore, lighting conditions can influence pigs' active and resting behavior. In the context of experiments testing different lighting conditions, a detection model was developed to detect the location and postures of group-housed growing-finishing pigs. The objective of this paper is to validate the model developed using YOLOv8 detecting standing, sitting, sternal lying, and lateral lying pigs. Training, validation, and test datasets included annotation of pigs from 10 to 24wk of age in 10 different light settings; varying in intensity, spectrum, and uniformity. Pig detection was comparable for the different lighting conditions, despite a slightly lower posture agreement for warm light and uneven light distribution, likely due to a less clear contrast between pigs and their background and the presence of shadows. The detection reached a mean average precision (mAP) of 89.4%. Standing was the best-detected posture with the highest precision, sensitivity, and F1 score, while the sensitivity and F1 score of sitting was the lowest. This lower performance resulted from confusion of sitting with sternal lying and standing, as a consequence of the top camera view and a low occurrence of sitting pigs in the annotated dataset. This issue is inherent to pig behavior and could be tackled using data augmentation. Some confusion was reported between types of lying due to occlusion by pen mates or pigs' own bodies, and grouping both types of lying postures resulted in an improvement in the detection (mAP = 97.0%). Therefore, comparing resting postures (both lying types) to active postures could lead to a more reliable interpretation of pigs' behavior. Some detection errors were observed, e.g., two detections for the same pig were generated due to posture uncertainty, dirt on cameras detected as a pig, and undetected pigs due to occlusion. The localization accuracy measured by the intersection over union was higher than 95.5% for 75% of the dataset, meaning that the location of predicted pigs was very close to annotated pigs. Tracking individual pigs revealed challenges with ID changes and switches between pen mates, requiring further work.

The saliency feature is a key factor in achieving vision-based tracking for multi-UAV control. However, due to the complex and variable environments encountered during multi-UAV operations—such as changes in lighting conditions and scale variations—the UAV’s visual features may degrade, especially under high-speed movement, ultimately resulting in failure of the vision tracking task and reducing the stability and robustness of swarm flight. Therefore, this paper proposes an adaptive active light source system based on light intensity matching to address the issue of visual feature loss caused by environmental light intensity and scale variations in multi-UAV collaborative navigation. The system consists of three components: an environment sensing and control module, a variable active light source module, and a light source power module. This paper first designs the overall framework of the active light source system, detailing the functions of each module and their collaborative working principles. Furthermore, optimization experiments are conducted on the variable active light source module. By comparing the recognition effects of the variable active light source module under different parameters, the best configuration is selected. In addition, to improve the robustness of the active light source system under different lighting conditions, this paper also constructs a light source color matching model based on light intensity matching. By collecting and comparing visible light images of different color light sources under various intensities and constructing the light intensity matching model using the comprehensive peak signal-to-noise ratio parameter, the model is optimized to ensure the best vision tracking performance under different lighting conditions. Finally, to validate the effectiveness of the proposed active light source system, quantitative and qualitative recognition comparison experiments were conducted in eight different scenarios with UAVs equipped with active light sources. The experimental results show that the UAV equipped with an active light source has improved the recall of yoloV7 and RT-DETR recognition algorithms by 30% and 29.6%, the mAP50 by 21% and 19.5%, and the recognition accuracy by 13.1% and 13.6, respectively. Qualitative experiments also demonstrated that the active light source effectively improved the recognition success rate under low lighting conditions. Extensive qualitative and quantitative experiments confirm that the UAV active light source system based on light intensity matching proposed in this paper effectively enhances the effectiveness and robustness of vision-based tracking for multi-UAVs, particularly in complex and variable environments. This research provides an efficient and computationally effective solution for vision-based multi-UAV systems, further enhancing the visual tracking capabilities of multi-UAVs under complex conditions.

To achieve accurate detection of the pineapple fruit picking area and pose under complex backgrounds and varying lighting conditions, this study proposes a pineapple keypoint detection model (LTHRNet) based on an improved LiteHRNet. Image data of pineapple fruits under different lighting conditions were collected, and six keypoints were defined to characterize the morphological features of the fruit. In the model design, LTHRNet incorporates the LKA_Stem module to enhance initial feature extraction, the D-Mixer module to capture both global and local feature relationships, and the MS-FFN module to achieve multi-scale feature fusion. In addition, the model employs parallel sub-networks with different resolutions to maintain high-resolution feature information and improve the precision and spatial accuracy of keypoint detection. Experimental results show that LTHRNet performs well in pineapple keypoint detection. It achieves 93.5% and 95.1% in KAP0.5 and KAR0.5, respectively, outperforming other models in terms of detection accuracy and robustness under challenging lighting and occlusion conditions, with a detection speed of 21.1 fps. For pose estimation, the average offset angle (AOA) of LTHRNet is 2.37°, which is significantly lower than that of other models. In summary, the proposed LTHRNet model demonstrates high accuracy and strong robustness in pineapple keypoint detection and pose estimation, providing reliable keypoint localization and pose estimation data for pineapple harvesting, while also offering an effective reference for pose recognition in other fruit-picking tasks.

This paper presents an innovative method for skin detection under changing lighting conditions. The proposed algorithm employs a Gaussian skin color model to classify skin pixels. This Gaussian model is adapted beforehand to accommodate the current lighting condition. AT first, the color temperature of the light source is estimated from a skin sample. Then the skin sample is going through a color constancy process to produce a corrected skin sample. The means of the raw and corrected skin samples are utilized to transform the Gaussian model. Experiments, using different adaptive skin detection methods and the proposed algorithm, have been conducted. Test results assert the effectiveness of the proposed method for skin detection under changing lighting conditions.