An FPGA Co-processor for Real-Time Visual Tracking

A real-time visual servo tracking system for an industrial robot has been developed. The position sensitive detector or PSD, instead of the CCD, is used as a real time vision sensor due to its fast response (The position is converted to analog current). A neural network learns the complex association between the object position and its sensor reading and uses it to track that object. It also turns out that this scheme lends itself to a convenient way to teach a workpath for the robot. Furthermore, for real-time use of the neural net, a novel architecture has been developed based on the concept of input space partitioning and local learning. It exhibits characteristics of fast processing and learning as well as optimal usage of hidden neurons. >

Designing a real-time visual tracking system to catch goldfishes is a very complex task because a fish’s precise relative 3D positions inside a water tank are difficult to determine. In this paper, we present a new visual servo control system using stereo vision to acquire the 3D positions of goldfishes. The continuously adaptive mean shift (CAMSHIFT) algorithm and optical flow vision tracking methods are applied to provide feedback of a tracked fish’s real-time position and motion trajectory in a water tank at a high frame-rate. Two low-cost, well calibrated cameras are used to track the goldfishes and a 5-axis robot manipulator is used to catch a specified goldfish. The designed visual tracking and servo system is less sensitive to lighting influences and thus performs more efficiently. Experiments of the proposed method yielded very good results as the real-time 3D vision successfully tracked two fishes and guided the manipulator to catching one of them in a net attached to its end-effector.

This paper proposes a real-time visual hand tracking and posture estimation system to guide a robotic arm in gripping gestures. We present an original 26 degree-of-freedom model of the hand, for which forward and inverse kinematics formulations have been developed. The hand, wearing a dark glove marked with colored cues, is tracked in real time in the image while completing a grasping task. The position of the color markers in the image is used to reconstruct the hand's pose. Occlusions are handled by position prediction and validation. A robot gripper can then be guided by the different configurations of the hand with respect to the object to be grasped, provided the relationship between the hand and the gripper is known. Successful results for the teleoperated control of a robot manipulator in simulation are presented.

This study develops a highly reliable real-time visual tracking system capable of tracking a moving target in cluttered environment. The proposed system architecture can be divided into three main parts - the target detector, the target information extractor and the target tracker. The first part detects any moving object in the image, using the moving edges technique. Next, by integrating the information from the target detector and the attractable snake model, the second part automatically extracts the target's contour, illumination and color information to enable the target to be tracked. Finally, by adopting a new hybrid matching method, the third part enables the system to continuously track the target with great robustness and adaptability. Notably, this hybrid matching method incorporates the template matching method with a modified contour matching method. Furthermore, the visual tracking system developed in this study is established on a low-cost commercial camera platform, which can be controlled by a personal computer through an RS232 serial port. Finally, several experiments, including object tracking and human head tracking, are conducted to verify the robustness of the tracking system.

The increase in popularity of fitness and at-home fitness has illuminated a major deficiency with traditional fitness tracking; the absence of immediate motion analysis and corrective feedback. Many conventional methods of instruction, as well as wearables, lack the system feedback of prompting users for proper movements, which can lead to many real workout injuries. The proposed system is the "Real-Time Exercise Tracking System". The proposed project is an intelligent method of intervention that uses computer vision and deep learning to provide a solution to the problem of lacking immediate feedback. The objective of the system is to accurately assess the form a user is exercising in, and properly provide the user with effective feedback on their form in real-time. The system utilizes a combination of MediaPipe for estimating accurate user pose, and a machine learning model to assesses the required movements of the user. The application provides the user with real-time voice and visual feedback on motion awareness. The application will assess the user's exercise awareness in terms of AI-based form correction, real-time performance analytics, propose new workouts based on previous fitness activities, and mitigate risk of injuries with alerts. The project will contribute to filling the gap between unmonitored home activity and professional fitness education, thus creating a safer, more effective, and happy fitness experience for users. Keywords – Real-Time Tracking, Pose Estimation, Computer Vision, MediaPipe, Fitness Technology, Deep Learning, Form Correction

A combined system comprising the LINIAC TrueBeam (Varian Medical Systems, Palo Alt, CA) and a new real-time tumor-tracking radiotherapy system, SyncTraX FX4® (Shimadzu Co., Kyoto, Japan), was installed in our institution. It consists of four pairs of an X-ray tube and a flat panel detector. The system was assessed on beam-on time delay between TrueBeam and SyncTraX FX4 and fluoroscopic dose during a real-time tracking. Delay time was measured by using a tumor-tracking radiotherapy phantom (CALIB PHANTOM ASSY, Shimadzu Co., Kyoto, Japan), in the cases of flattening filtered (FF) 6 MV photon beam, flattening filter-free (FFF) 6 MV photon beam, FF-10 MV photon beam, and FFF-10 MV photon beam with the LINAC TrueBeam. Half-value layer (HVL) in mm Al, effective kVp, and air-kerma rate during fluoroscopy were measured using a solid-state detector for the tube voltage (70–110 kV) and the current (50–100 mA). The LINAC delayed from the real-time tumor tracking system on beam-on by 140.9 ± 8.5, 119.8 ± 3.8, 126.1 ± 3.2 and 116.8 ± 9.7 ms for FF-6 MV, FFF-6 MV, FF-10 MV and FFF-10 MV, respectively. The HVL, effective kVp and air-kerma rates from X-ray tube #1 (X-ray tube #1 and #2 were embedded in the patient’s head side floor) were 4.98 ± 0.00 mm, 111.2 ± 0.1 kV and 9.14 ± 0.04 mGy/min for 110 kV X-ray at 100 mA. The HVL, effective kVp and air-kerma rates from X-ray tube #3 (X-ray tube #3 and #4 were embedded in the patient’s feet side floor), were 5.20 ± 0.00 mm, 110.0 ± 0.1 kV and 11.87 ± 0.06 mGy/min for 110 kV X-ray at 100 mA. These tube voltage and current are the maximum conditions of this study. The time delay of the real-time system is longer than the old system that used image intensifiers. The air-kerma rate from X-ray tube #3 was higher than that from X-ray tube #1.

Real-time target tracking is a usual task for humans despite the neural delays during the nervous system’s axonal transfer and neural processing. A plausible explanation is that the human brain employs predictive mechanisms to compensate for the delay. Inspired by the brain, this paper adopts a prediction network based on spiking neural networks (SNNs) to implement a real-time tracking task on a neuromorphic chip with low power consumption. The SNN-based prediction network outperforms the long short-term memory (LSTM) network on a small dataset and reduces 90% to 98% computations compared with LSTM. The quantized SNN-based network is deployed on a neuromorphic chip, and it takes 25ms and only 442 626nJ for a single prediction. The tracking performance of the system is also verified in real-life scenarios. Furthermore, the proposed real-time target tracking system can be easily ported to other neuromorphic platforms.

This paper proposes a real-time location tracking system using a GPS module for different mobile devices and multiple users. It focuses on the management and observation of a majority of people can be foretold. Traditional RF, wireless ATM networks, or wireless cellular radio networks-based location tracking services encounter challenges. These include poor radio link quality and resource consumption (i.e., radio bandwidth) due to the users' mobility, thus necessitating managing their position. Our proposal can service a real-time location tracking service for multiple parties using TCP/IP wireless Internet protocol and GPS application in a mobile environment. The user may acquire and manage location information of specified subjects, who require individual care in real-time. Such users include those requiting specific protective measures, children, and the elderly. The proposed GPS and wireless network-based real-time location tracking system may be extended to various application domains that detect user location information and utilize this anytime, anywhere.

We report on the design of a real-time tracking system using precise space and time information of the hit. Tracking detectors with about 150 ps time resolution are already available and intense R&D is in progress to improve the performance. We describe the implementation of a real-time tracking algorithm in commercial FPGAs for a prototype tracking system based on 8 planes of silicon sensors used as a case study. We demonstrate that the use of the additional time information in the tracking algorithm allows to suppress noise hits while reducing fake track combinations and improving real-time track reconstruction. The system provides offline-like quality tracks with sub-microsecond latency and it is capable to determine the time of the track with picosecond resolution assuming 10 ps resolution for the hits.

Motion estimation in videos is a computationally intensive process. A popular strategy for dealing with such a high processing load is to accelerate algorithms with dedicated hardware such as graphic processor units (GPU), field programmable gate arrays (FPGA), and digital signal processors (DSP). Previous approaches addressed the problem using accelerators together with a general purpose processor, such as acorn RISC machines (ARM). In this work, we present a co-processing architecture using FPGA and DSP. A portable platform for motion estimation based on sparse feature point detection and tracking is developed for real-time embedded systems and smart video sensors applications. A Harris corner detection IP core is designed with a customized fine grain pipeline on a Virtex-4 FPGA. The detected feature points are then tracked using the Lucas---Kanade algorithm in a DSP that acts as a co-processor for the FPGA. The hybrid system offers a throughput of 160 frames per second (fps) for VGA image resolution. We have also tested the benefits of our proposed solution (FPGA + DSP) in comparison with two other traditional architectures and co-processing strategies: hybrid ARM + DSP and DSP only. The proposed FPGA + DSP system offers a speedup of about 20 times and 3 times over ARM + DSP and DSP only configurations, respectively. A comparison of the Harris feature detection algorithm performance between different embedded processors (DSP, ARM, and FPGA) reveals that the DSP offers the best performance when scaling up from QVGA to VGA resolutions.

This paper presents an investigation of the development of a real-time object tracking system that utilizes computer vision techniques and incorporates pan-tilt control mechanisms. The system architecture includes hardware components such as a camera and a pan-tilt mechanism, and software components such as object detection and tracking algorithms. To achieve efficient and accurate tracking, the Kalman filter, which uses a state-space model, was included in the object tracking process. The proposed system is implemented and evaluated through experiments to demonstrate its feasibility and effectiveness in real-world scenarios. The results indicate that the proposed system is capable of achieving real-time and accurate object tracking, demonstrating its potential for practical applications in various fields.

Color feature tracking is based on pattern matching algorithms where the appearance of the target is compared with a reference model in successive images and the position of the target is estimated. The major drawback of these methods is that such operations require expensive computation power. It is bottleneck to implement real-time color feature tracking system. The probabilistic tracking methods have been shown to be robust and versatile for a modest computational cost. However, the probabilistic tracking methods break down easily when the object moves very fast because these methods search only the regions of interest based on the probability density function to estimate the position of the moving object. In this paper, we propose a real-time color feature tracking circuit. We propose a window-based image processing structure to improve the processing speed of the tracking circuit. The tracking circuit searches all regions of the image to perform a matching operation in order to estimate the position of the moving object. The main results of our work are that we have designed and implemented a physically feasible hardware circuit to improve the processing speed of the operations required for real-time color feature tracking. Therefore, this work has resulted in the development of a real-time color feature tracking system employing an FPGA (field programmable gate array) implemented circuit designed by VHDL (the VHSIC hardware description tanguage). Its performance has been measured to compare with the equivalent software implementation.

Visual-based Simultaneous Localization and Mapping (vSLAM) is indispensable for intelligent mobile robots and autonomous vehicles. However, the scene rigidity assumption typical in vSLAM systems limits the widespread use of most vSLAM under dynamic environments in the real world. We propose Real-time Visual Semantic Tracking and Mapping (RTS-vSLAM), implemented based on RGB-D-based ORB-SLAM2, which uses both semantic information and geometric methods to detect moving objects and outlier features. RTS-vSLAM adopts a novel generic system architecture that enables the use of many kinds of semantic segmentation methods on the remote server and can be employed on embedded system devices, head-mounted devices, and low-performance robots, where GPUs are not mounted on, via requesting service from the server. It also generates many kinds of maps to support navigation and more complex tasks in real-time. The proposed system is evaluated using the TUM RGB-D dataset and compared tracking errors with the state-of-the-art vSLAM methods under dynamic environments. RTS-vSLAM outperforms the others in tracking accuracy in a highly dynamic scenario and reduces the time delays greatly.KeywordsSLAMDynamic SLAMTrackingMappingRTS-vSLAM

A low-cost real-time two-dimensional PC-based ball tracking and control laboratory system is developed to teach real-time control system development with a real physical system instead of computer simulation. The system can be used for a single-student or a two-student competition for the best control algorithm. Training can be given on pattern recognition, modelling, stepper motor control, control strategies and other aspects.

To measure respiratory motion of the thoracic wall region in dogs using a real-time motion tracking system and compare the amount of respiratory motion between dogs positioned with and without a vacuum-formable cushion. 8 healthy adult mixed-breed dogs (median weight, 23 kg). Dogs were anesthetized and positioned in sternal and dorsal recumbency with and without a vacuum-formable cushion. Three-dimensional movement of anatomic landmarks was measured with a real-time motion capture system that tracked the locations of infrared light-emitting diodes attached externally to the dorsal or ventral and lateral aspects of the thoracic wall. Dogs positioned in sternal recumbency had significantly less cranial-to-caudal and left-to-right respiratory motion at the lateral aspect of the thoracic wall, compared with dogs positioned in dorsal recumbency, whether or not a cushion was used. For dogs treated in sternal recumbency, use of a cushion significantly increased the peak displacement vector (overall movement in 3-D space) for 3 of 4 marker locations on the dorsal thoracic wall. As respiratory rate increased, respiratory motion at the lateral and ventral aspects of the thoracic wall decreased when data for all dogs in dorsal recumbency were evaluated together. Associations between respiratory rate and respiratory motion suggested that the use of rapid, shallow ventilation may be beneficial for dogs undergoing highly conformal radiation treatment. These results provide a basis for further research on respiratory motion in anesthetized dogs.