Stereo Endoscopic Camera Pose Optimal Estimation by Structure Similarity Index Measure Integration.

Human motion recognition is traditionally approached by either recognizing basic motions from features derived from video input or by interpreting complex motions by applying a high-level hierarchy of motion primitives. The former method is usually limited to rather simple motions while the latter requires human expert knowledge to build up a suitable hierarchy. In this paper we propose a new approach that uses the strength of both methods while overcoming their respective limitations. Our approach is able to recognize the motion units within complex motion sequences. The recognition process applies Hidden Markov Models (HMM) based on features consisting of optical flow gradient histograms. For each primitive motion unit we train one HMM and then concatenate these primitive motion units to form complex motion sequences. Modeling sequences with HMMs allows for a very flexible combination of motion units into motion sequences. They can either be combined in a restrictive rule-based formulation using predefined grammars or be more flexibly combined using a statistical model of sequence probabilities. In this paper we are mainly interested in the comparison of the optical flow features with marker-based features, therefore we do not use a motion grammar. We apply our approach to 24 motion units forming five complex motion sequences as they appear in a real-world kitchen tasks. The results show that the proposed approach allows for a very fast low-level recognition of human motion units without the need for any complex reconstruction, post processing or pose estimation. Straight-forward characteristic flow fields in combination with HMM sequence modeling are sufficient to reliably recognize complex motions even with an unrestricted search. Our results show that this search already achieves 13.1 % recognition error rate. We compare HMM models based on the optical flow features to those derived from a marker-based system. Our recognition results indicate that optical flow features achieve a competitive performance.

Current action recognition algorithms in ice hockey do not fully exploit the temporal cues available in video. To solve this challenge, we introduce a two-stream network utilizing player pose sequences and optical flow features for recognizing hockey actions. Player pose sequences are compact representations consisting of frame by frame human and stick joint locations and angles between joints. The optical flow features are obtained by a state-of-the-art optical flow algorithm. The player pose sequences are processed by a two-layered Long short-term memory (LSTM) network. The LSTM output is fused with optical flow features processed by a convolutional neural network (CNN). Experimental results demonstrate the efficacy of the method by achieving 90.48% test accuracy on the HARPET (Hockey Action Recognition Pose Estimation, Temporal) dataset thus surpassing current benchmark by 5%. The network performs better than the current benchmark in segregating similar classes like passing and shooting. It achieves a 90% reduction in parameters and 80% reduction in floating point operations per second (FLOPs) than the benchmark on the HARPET dataset, thus furthering the effectiveness of the network.

Marker-based 3D motion capture systems, widely used for gait analysis, are accurate but have disadvantages such as cost and accessibility. Whereas markerless pose estimation has emerged as a convenient and cost-effective alternative for gait analysis, challenges remain in achieving optimal accuracy. Given the limited research on the effects of camera location and orientation on data collection accuracy, this study investigates how camera placement affects gait assessment accuracy utilizing five smartphones. This study aimed to explore the differences in data collection accuracy between marker-based systems and pose estimation, as well as to assess the impact of camera location and orientation on accuracy in pose estimation. The results showed that the differences in joint angles between pose estimation and marker-based systems are below 5°, an acceptable level for gait analysis, with a strong correlation between the two datasets supporting the effectiveness of pose estimation in gait analysis. In addition, hip and knee angles were accurately measured at the front diagonal of the subject and ankle angle at the lateral side. This research highlights the significance of careful camera placement for reliable gait analysis using pose estimation, serving as a concise reference to guide future efforts in enhancing the quantitative accuracy of gait analysis.

For robot-assisted assembly of complex optical systems, the alignment is facilitated by an accurate pose estimation of its components. However, wavefront-based pose estimation is typically ill-conditioned due to the inherent geometry of conventional industrially manufactured optical components. Therefore, we propose a novel approach in this paper to increase wavefront-based pose estimation accuracy via the design of freeform optics. For this purpose, an optimization problem is derived that parameterizes the component's surfaces by a predetermined freeform surface model. To show the efficacy of our approach, we provide simulation results to compare the pose estimation accuracy for a variety of optical designs. As an application example for the resulting improved pose estimation, a hand-eye calibration of a wavefront sensor is performed. This calibration originates from the field of robotics and represents the identification of a sensor coordinate system with respect to a global reference frame. For quantitative evaluation, the calibrating results are first presented with the aid of simulation data. Finally, the practical feasibility is demonstrated using a conventional industrial robot and additively manufactured freeform lenses.

Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

We propose and demonstrate an efficient pose estimation method, specifically small tilt angle variation, for visual servo control system design using a monocular camera, eliminating the need for iterative computations. This paper addresses the challenges associated with accurately estimating the relative pose between a chaser, equipped with a monocular camera, and a target that provides visual cues, particularly in scenarios where the captured image shows only minimal variations in the target's tilt angle. Traditional approaches often rely on direct feedback methods from image data or designs for pose estimation and control separately, necessitating computationally expensive iterations for pose estimation or complex implementation of visual servoing. To address these challenges, we adopt an extended Kalman filter, known as a robust nonlinear observer, and propose a non‐iterative approach that integrates motion dynamics models and control inputs into pose estimation from image data. This approach enables efficient integration of pose estimation and control solely relying on a monocular camera, reducing computational costs and improving robustness. Experimental validation using the ball‐and‐plate system demonstrates the effectiveness of our proposed method. In our setup, the camera is deliberately positioned directly above the pose estimation target, creating a scenario with minimal image variation in the tilt angle. Despite this condition, our approach achieves efficient and accurate pose estimation, demonstrating its feasibility for high‐speed pose estimation and control without iterative computations.

Recognition of facial expression has many applications including human-computer interaction, human emotion analysis, personality development, cognitive science, health-care, virtual reality, image retrieval, etc. In this paper we propose a new method for recognition of facial expression using local region specific mean optical flow and local binary pattern feature descriptor with support vector machine classification. In general, facial expression recognition techniques divide the face into regular grid (holistic representation) and the facial features are extracted. However, in this paper we divide the face into domain specific local regions. At first a robust optical flow is utilized to get mean optical flow in different directions for each local region which considers both local statistic motion information and its spatial location. The features are used only from the key frames; which are detected based on maximal mean optical flow magnitude within a sequence w.r.t. neutral frame. Now, the region specific local binary pattern is extracted from key frame and concatenated with mean optical flow features. The performance of the proposed facial expression recognition system has been validated on CK+ facial expression dataset.

To address the degradation of visual simultaneous localization and mapping (VSLAM) pose estimation accuracy caused by extensive moving objects and the resulting contamination of feature points in indoor dynamic environments, this study proposes a dynamic feature point rejection strategy that integrates a You Only Look Once (YOLO) family detector, YOLO11, with the Lucas–Kanade (LK) optical flow method within the ORB_SLAM3 framework. First, YOLO11 is employed to detect potentially dynamic targets such as pedestrians in the image, and ORB features extracted within the detection bounding boxes are labeled as high-risk dynamic points. Subsequently, based on pyramidal LK optical flow, the motion consistency of feature points is analyzed across multiple frames, and a secondary discrimination is performed on features located at the edges and in the vicinity of the detection boxes. In this way, feature points that conform to a static motion model are maximally preserved, thereby alleviating the excessive removal and misclassification of static points caused by oversized detection boxes. On this basis, a new map is reconstructed using the filtered static feature points, and the pose estimation process is further refined. Experiments on multiple indoor dynamic scenarios demonstrate that, compared with the original ORB_SLAM3, the proposed method effectively suppresses dynamic interference, enhances the stability of feature points, and significantly improves pose estimation accuracy and trajectory continuity, thus providing a valuable reference for robust VSLAM applications in dynamic environments.

This paper proposes a novel perceptual video quality assessment metric for streamed videos using optical flow statistical features. We analyze the impact of network losses on the decoded videos and the resulting error propagation. We show that the statistical features of the optical flow of the corrupted frames can be used to measure the distortion in the received video. We show that this approach is suitable for videos with complex motion patterns. Our technique does not make any assumptions on the coding conditions, network loss patterns or error concealment techniques. The proposed approach is pixel-based and relies only on the inconsistency of the optical flow of the corrupted frames. We validate our proposed quality metric by testing it on a variety of coded sequences subject to network losses from the recently proposed LIVE mobile database. Our results show that the proposed metric can estimate perceptual quality of channel-induced distortions at the frame and sequence levels. For the test videos, we report Pearson's and Spearman's correlation coefficients with the temporal mean opinion scores (MOSs) reported in the database. The results show average correlations of 0.91 and 0.92 for the test sequences, respectively.

Fast and accurate pose estimation is essential for the local motion control of robots such as drones. At present, camera-based motion capture (Mocap) systems are mostly used by robots. However, this kind of Mocap system is easily affected by light noise and camera occlusion, and the cost of common commercial Mocap systems is high. To address these challenges, we propose Easy Rocap, a low-cost, open-source robot motion capture system, which can quickly and robustly capture the accurate position and orientation of the robot. Firstly, based on training a real-time object detector, an object-filtering algorithm using class and confidence is designed to eliminate false detections. Secondly, multiple-object tracking (MOT) is applied to maintain the continuity of the trajectories, and the epipolar constraint is applied to multi-view correspondences. Finally, the calibrated multi-view cameras are used to calculate the 3D coordinates of the markers and effectively estimate the 3D pose of the target robot. Our system takes in real-time multi-camera data streams, making it easy to integrate into the robot system. In the simulation scenario experiment, the average position estimation error of the method is less than 0.008 m, and the average orientation error is less than 0.65 degrees. In the real scenario experiment, we compared the localization results of our method with the advanced LiDAR-Inertial Simultaneous Localization and Mapping (SLAM) algorithm. According to the experimental results, SLAM generates drifts during turns, while our method can overcome the drifts and accumulated errors of SLAM, making the trajectory more stable and accurate. In addition, the pose estimation speed of our system can reach 30 Hz.

Bai, Y., 2019. Target detection method of underwater moving image based on optical flow characteristics. In: Guido-Aldana, P.A. and Mulahasan, S. (eds.), Advances in Water Resources and Exploration. Journal of Coastal Research, Special Issue No. 93, pp. 668–673. Coconut Creek (Florida), ISSN 0749-0208.The detection and recognition ability of underwater moving image target is improved, and an automatic target detection method based on optical flow feature is proposed. The image acquisition model of underwater moving image target detection is constructed, and the local information entropy feature of underwater moving target image is detected by optical flow field scale decomposition method. The contrast feature of the distribution area of the optical flow field of the underwater moving image is extracted, and the information fusion model of the underwater moving target image is constructed according to the statistical value of the inter-frame difference of the underwater moving image. The center position of the candidate target is obtained, and the adaptive feature extraction of the underwater moving image target is carried out by using the correlation fusion method between the pixel neighborhoods, and the automatic detection of the underwater moving image target is realized according to the optical flow feature distribution. The simulation results show that the proposed method has good adaptability, high accuracy and strong anti-optical flow interference ability in underwater moving image target recognition.

The 6D pose estimation of objects plays a crucial role in robot sorting technology. To simultaneously improve the accuracy and efficiency of pose estimation during the grasping process, this study proposes an innovative deep learning pose estimation method based on PVN3D. Specifically, this study deeply optimized the backbone network of the image extraction model, using dense connections and learnable grouped convolutions to improve model accuracy and reduce model complexity. At the same time, the core idea of convolutional neural networks is integrated into the point cloud feature extraction process, which uses dynamic convolution techniques to process point cloud features, making the processing of point cloud data more efficient and flexible. In addition, by introducing a parameter-free attention mechanism, the accuracy of the model has been further improved. Through extensive experimental verification on the publicly available LineMOD, YCB video datasets and other three core datasets under the BOP benchmark, the method proposed in this paper demonstrates significant advantages in both computational efficiency and estimation accuracy. It not only significantly improves the accuracy of pose estimation but also significantly enhances real-time performance.

Device-Free Localization (DFL) is a promising technique for applications in living environments and disaster sites, as it does not require users to carry any radio devices. Multipath-assisted Radio Tomographic Imaging (multipath-RTI) is one effective method for enabling DFL reducing the number of anchor nodes. This method maps a person’s location in a target area by using numerous wireless links within an radio sensor network. However, improving the localization accuracy in DFL poses a significant challenge, particularly in optimizing the regularization parameters required to solve the inverse problem. As usual, regularization parameters are heuristically selected. In this study, we propose a method for identifying the optimal regularization parameters through simulations using Bayesian optimization, thereby improving localization accuracy. The experimental results show that the proposed method can effectively improve localization accuracy—achieving a 10% reduction in average error—which highlights its applicability in practical DFL systems.

Driver distraction remains a critical road safety concern, necessitating robust activity recognition techniques in Advanced Driver Assistance Systems (ADAS). In this work, we present a novel three-stage approach that leverages the information from RGB and infrared (IR) video inputs to classify driver activities. The strategy is based on extracting multimodal features from RGB and IR video frames. We consider optical flow features for RGB frames as they can capture motion patterns associated with potential distractions. In the case of IR frames, temporal gradients (TG) are utilised because they can indicate dynamic changes in IR illumination. In the next stage, we feed the optical flow and TG features into separate convolutional networks individually to exploit the temporal relationship between consecutive frames within each modality. The features extracted from the individual models are then fused using a dedicated fusion module. In the last stage, the fused features are fed into a classification network based on the Inflated 3D (I3D) architecture to classify the driver’s activity and identify instances of distraction. We evaluate the proposed approach on a publicly available driver monitoring dataset. The results demonstrate superior performance compared to baseline methods that utilise only RGB or IR features individually. This highlights the effectiveness of multimodal fusion in reliably addressing real-world driver monitoring challenges.

UVMO: Deep unsupervised visual reconstruction-based multimodal-assisted odometry