A neural network based real-time robot tracking controller using position sensitive detectors

Designing a real-time visual tracking system to catch goldfishes is a very complex task because a fish’s precise relative 3D positions inside a water tank are difficult to determine. In this paper, we present a new visual servo control system using stereo vision to acquire the 3D positions of goldfishes. The continuously adaptive mean shift (CAMSHIFT) algorithm and optical flow vision tracking methods are applied to provide feedback of a tracked fish’s real-time position and motion trajectory in a water tank at a high frame-rate. Two low-cost, well calibrated cameras are used to track the goldfishes and a 5-axis robot manipulator is used to catch a specified goldfish. The designed visual tracking and servo system is less sensitive to lighting influences and thus performs more efficiently. Experiments of the proposed method yielded very good results as the real-time 3D vision successfully tracked two fishes and guided the manipulator to catching one of them in a net attached to its end-effector.

This paper proposes a real-time visual hand tracking and posture estimation system to guide a robotic arm in gripping gestures. We present an original 26 degree-of-freedom model of the hand, for which forward and inverse kinematics formulations have been developed. The hand, wearing a dark glove marked with colored cues, is tracked in real time in the image while completing a grasping task. The position of the color markers in the image is used to reconstruct the hand's pose. Occlusions are handled by position prediction and validation. A robot gripper can then be guided by the different configurations of the hand with respect to the object to be grasped, provided the relationship between the hand and the gripper is known. Successful results for the teleoperated control of a robot manipulator in simulation are presented.

This paper presents the design overview and the post-synthesis simulation results of a digital co-processor for real-time visual tracking implemented in a Virtex-E field programmable gate array. The hardware description of the system was made in VHDL and the simulations show that the system performs up to 79 frames per second in a half-resolution VGA image format (320 × 240 pixels). The co-processor calculates edge/corner detection, stationary background and noise filtering, and the distance transform algorithm; these optimized operations implemented in the FPGA and the Hausdorff distance algorithm programmed in a general purpose processor implement a real-time visual tracking system.

This study develops a highly reliable real-time visual tracking system capable of tracking a moving target in cluttered environment. The proposed system architecture can be divided into three main parts - the target detector, the target information extractor and the target tracker. The first part detects any moving object in the image, using the moving edges technique. Next, by integrating the information from the target detector and the attractable snake model, the second part automatically extracts the target's contour, illumination and color information to enable the target to be tracked. Finally, by adopting a new hybrid matching method, the third part enables the system to continuously track the target with great robustness and adaptability. Notably, this hybrid matching method incorporates the template matching method with a modified contour matching method. Furthermore, the visual tracking system developed in this study is established on a low-cost commercial camera platform, which can be controlled by a personal computer through an RS232 serial port. Finally, several experiments, including object tracking and human head tracking, are conducted to verify the robustness of the tracking system.

MTN optimal control of MIMO non-affine nonlinear time-varying discrete systems for tracking only by output feedback

There is tradeoff between magnification and viewing width when recognizing features on a moving object, for example, the eyes or a mouth of a human who is moving around. To recognize an image feature precisely or correctly, we need to have enough resolution of the image. It is not easy, however, to keep the feature area steady and centred in the camera's image if the object with the feature is moving fast or far away. This paper presents a visual tracking method for feature recognition of moving human face. We use two cameras with different viewing-angles to catch information both with lower resolution for tracking a moving object and with higher resolution for object recognition. The statistical information of the difference image between successive frames is used to estimate the motion speed of a moving object. To demonstrate the idea of this visual tracking system, we show its application to our anthropomorphic agent system called VSA (visual software agent), which is an advanced human interface system with a realistic human facial image and a speech communication function.

The increase in popularity of fitness and at-home fitness has illuminated a major deficiency with traditional fitness tracking; the absence of immediate motion analysis and corrective feedback. Many conventional methods of instruction, as well as wearables, lack the system feedback of prompting users for proper movements, which can lead to many real workout injuries. The proposed system is the "Real-Time Exercise Tracking System". The proposed project is an intelligent method of intervention that uses computer vision and deep learning to provide a solution to the problem of lacking immediate feedback. The objective of the system is to accurately assess the form a user is exercising in, and properly provide the user with effective feedback on their form in real-time. The system utilizes a combination of MediaPipe for estimating accurate user pose, and a machine learning model to assesses the required movements of the user. The application provides the user with real-time voice and visual feedback on motion awareness. The application will assess the user's exercise awareness in terms of AI-based form correction, real-time performance analytics, propose new workouts based on previous fitness activities, and mitigate risk of injuries with alerts. The project will contribute to filling the gap between unmonitored home activity and professional fitness education, thus creating a safer, more effective, and happy fitness experience for users. Keywords – Real-Time Tracking, Pose Estimation, Computer Vision, MediaPipe, Fitness Technology, Deep Learning, Form Correction

Position-sensitive detectors (PSDs) based on the lateral photo effect have been widely used in diverse applications including optical engineering, aerospace, and military fields. With increasing demands for long-working-distance, low-energy-consumption, and weak-signal-sensing systems, the poor responsivity of conventional silicon-based PSDs has become a bottleneck limiting their applications. Herein, we propose a high-performance passive PSD based on a graphene–Si heterostructure. The graphene is adapted as a photon-absorbing and charge-separation layer working together with Si as a junction, while the high mobility provides promising ultra-long carrier diffusion length and facilitates a large active area of the device. A PSD with a working area of 8 mm×8 mm is demonstrated to present excellent position sensitivity to weak light at the nanowatt level (much better than the limit of microwatts of Si P-I-N PSDs). More importantly, it shows very fast response and low degree of nonlinearity of ∼3%, and extends the operating wavelength to the near-infrared region (1319 and 1550 nm). This work therefore provides a new strategy for high-performance and broadband PSDs.

12 In this paper, a new measuring system is proposed which can measure 6-DOF motion of rigid bodies. Its measurement principle is based on detection of laser beam reflected from a specially fabricated mirror whose shape is a triangular pyramid having an equilateral cross-sectional shape. The 3- facet mirror is mounted on the object whose 6-DOF displacement is to be measured. The measurement is operated by a laser-based optical system composed of a 3-facet mirror, a laser source, three position-sensitive detectors (PSD). In the sensor system, three PSDs are located at three corner points of a triangular formation, which is an equilateral triangular formation lying parallel to the reference plane. From this arrangement, 6-DOF displacement of any object can be simply determined. In this paper, we model the relationship between the 6-DOF displacement of the object and the outputs of three PSDs. A series of experiments is performed to demonstrate the effectiveness of the proposed method. The experimental results show that the proposed sensing system can be an effective means of obtaining 3D position and orientation of arbitrary objects.

Real-time target tracking is a usual task for humans despite the neural delays during the nervous system’s axonal transfer and neural processing. A plausible explanation is that the human brain employs predictive mechanisms to compensate for the delay. Inspired by the brain, this paper adopts a prediction network based on spiking neural networks (SNNs) to implement a real-time tracking task on a neuromorphic chip with low power consumption. The SNN-based prediction network outperforms the long short-term memory (LSTM) network on a small dataset and reduces 90% to 98% computations compared with LSTM. The quantized SNN-based network is deployed on a neuromorphic chip, and it takes 25ms and only 442 626nJ for a single prediction. The tracking performance of the system is also verified in real-life scenarios. Furthermore, the proposed real-time target tracking system can be easily ported to other neuromorphic platforms.

A 2-D tetralateral position-sensitive detector (PSD) is able to provide continuous position measurements with high position resolution, fast response, and low cost. However, it suffers from the nonlinearity problem, which leads to inaccurate measurements. In this paper, two linearity indices are proposed as measures of linearity for its position measurement. Then, a systematic procedure is developed to design new formulas to achieve any degree of linearity through the use of the linearity indices. As an example, a simple estimation formula for the 2-D tetralateral PSD has been derived based on the procedure. Compared to the conventional formula, the linearity of the measurement is greatly improved when the new formula is used. To verify its effectiveness, the new formula is implemented in analog circuits and tested in our experiment. Similar linearity improvement for other 2-D PSDs can also be achieved by following the same design procedure.

Position-sensitive detectors based on the lateral photovoltaic effect have been widely used in optical engineering for the measurement of position, distance, and angles. However, self-powered ultraviolet position-sensitive detectors with high sensitivity and fast response are still lacking due to the difficulty associated with the fabrication of p-type wide bandgap semiconductors, which hinders their further design and enhancement. Here, the influence of band structures and interfacial transport properties on the performance of self-powered ultraviolet position-sensitive detectors based on PrNiO3/Nb:SrTiO3p–n junctions is systematically investigated. Large position sensitivity and fast relaxation time of the lateral photovoltaic effect were observed up to 400 K in the perovskite-based ultraviolet position-sensitive detectors. Hall effect measurements revealed that the transport of photoexcited carriers occurs mainly through the interface of the PrNiO3/Nb:SrTiO3 junctions, resulting in a fast response and a stable photovoltaic effect. This study presents insights and avenues for designing self-powered perovskite oxide ultraviolet position-sensitive detectors with enhanced photoelectric performance.

A combined system comprising the LINIAC TrueBeam (Varian Medical Systems, Palo Alt, CA) and a new real-time tumor-tracking radiotherapy system, SyncTraX FX4® (Shimadzu Co., Kyoto, Japan), was installed in our institution. It consists of four pairs of an X-ray tube and a flat panel detector. The system was assessed on beam-on time delay between TrueBeam and SyncTraX FX4 and fluoroscopic dose during a real-time tracking. Delay time was measured by using a tumor-tracking radiotherapy phantom (CALIB PHANTOM ASSY, Shimadzu Co., Kyoto, Japan), in the cases of flattening filtered (FF) 6 MV photon beam, flattening filter-free (FFF) 6 MV photon beam, FF-10 MV photon beam, and FFF-10 MV photon beam with the LINAC TrueBeam. Half-value layer (HVL) in mm Al, effective kVp, and air-kerma rate during fluoroscopy were measured using a solid-state detector for the tube voltage (70–110 kV) and the current (50–100 mA). The LINAC delayed from the real-time tumor tracking system on beam-on by 140.9 ± 8.5, 119.8 ± 3.8, 126.1 ± 3.2 and 116.8 ± 9.7 ms for FF-6 MV, FFF-6 MV, FF-10 MV and FFF-10 MV, respectively. The HVL, effective kVp and air-kerma rates from X-ray tube #1 (X-ray tube #1 and #2 were embedded in the patient’s head side floor) were 4.98 ± 0.00 mm, 111.2 ± 0.1 kV and 9.14 ± 0.04 mGy/min for 110 kV X-ray at 100 mA. The HVL, effective kVp and air-kerma rates from X-ray tube #3 (X-ray tube #3 and #4 were embedded in the patient’s feet side floor), were 5.20 ± 0.00 mm, 110.0 ± 0.1 kV and 11.87 ± 0.06 mGy/min for 110 kV X-ray at 100 mA. These tube voltage and current are the maximum conditions of this study. The time delay of the real-time system is longer than the old system that used image intensifiers. The air-kerma rate from X-ray tube #3 was higher than that from X-ray tube #1.

Traffic congestion is a serious issue in developing nations. Light configurations are automatically modified by smart traffic light systems in response to traffic circumstances in real-time. For the system to adjust correctly, information on traffic density would be required. A vehicle counting system that can be used to determine the number of vehicles on busy highways with the aid of neural networks. YOLO (You Only Look Once), an object detection technique based on neural networks, is used by this system to identify cars. Simple Online and Real-time Tracking (SORT) algorithms are used to count and categories the vehicles in traffic recordings. We will determine the direction of vehicle movement after counting and attempt to apply the type of vehicle count, such as how many cars, how many bicycle and so on. The proposed neural network structure is better suited for real-time vehicle tracking because the computational complexity is reduced.

This paper proposes a real-time location tracking system using a GPS module for different mobile devices and multiple users. It focuses on the management and observation of a majority of people can be foretold. Traditional RF, wireless ATM networks, or wireless cellular radio networks-based location tracking services encounter challenges. These include poor radio link quality and resource consumption (i.e., radio bandwidth) due to the users' mobility, thus necessitating managing their position. Our proposal can service a real-time location tracking service for multiple parties using TCP/IP wireless Internet protocol and GPS application in a mobile environment. The user may acquire and manage location information of specified subjects, who require individual care in real-time. Such users include those requiting specific protective measures, children, and the elderly. The proposed GPS and wireless network-based real-time location tracking system may be extended to various application domains that detect user location information and utilize this anytime, anywhere.