Real-Time Object Detection and Recognition on Low-Compute Humanoid Robots using Deep Learning

Object detection and tracking is one of the key applications of wireless sensor networks (WSNs). The key issues associated with this application include network lifetime, object detection and localization accuracy. To ensure the high quality of the service, there should be a trade-off between energy efficiency and detection accuracy, which is challenging in a resource-constrained WSN. Most researchers have enhanced the application lifetime while achieving target detection accuracy at the cost of high node density. They neither considered the system cost nor the object localization accuracy. Some researchers focused on object detection accuracy while achieving energy efficiency by limiting the detection to a predefined target trajectory. In particular, some researchers only focused on node clustering and node scheduling for energy efficiency. In this study, we proposed a mobile object detection and tracking framework named the Energy Efficient Object Detection and Tracking Framework (EEODTF) for heterogeneous WSNs, which minimizes energy consumption during tracking while not affecting the object detection and localization accuracy. It focuses on achieving energy efficiency via node optimization, mobile node trajectory optimization, node clustering, data reporting optimization and detection optimization. We compared the performance of the EEODTF with the Energy Efficient Tracking and Localization of Object (EETLO) model and the Particle-Swarm-Optimization-based Energy Efficient Target Tracking Model (PSOEETTM). It was found that the EEODTF is more energy efficient than the EETLO and PSOEETTM models.

Abstract: In this study, we present a radar detection system for real-time object localization based on ultrasonic sensor technology. The system combines ultrasonic sensors, a servo motor, and an Arduino microcontroller to send ultrasonic waves, detect their echoes, and compute distances with high accuracy. This information is processed to generate a spatial map of the environment around it, allowing for precise object detection and localization in real time. The system works by utilizing the servo motor to spin the ultrasonic sensor, taking distance readings over an array of angles. The data is processed and presented, giving a real-time visualization of the objects found in the surroundings. This method guarantees high precision with simplicity and affordability. The flexibility of the system to various environmental conditions makes it a perfect solution for a variety of applications such as robotics, security systems, surveillance, obstacle detection, and industrial automation. One of the major benefits of this system is its cost-effectiveness and ease of construction, making it affordable for hobbyists, researchers, and industry experts. The modular design also guarantees flexibility, making it possible to customize it for particular applications or integrate it with sophisticated systems. Our study emphasizes the potential of ultrasonic sensor technology in object detection systems, opening doors to novel applications in automation, navigation, and safety systems.

From object detection to semantic segmentation, deep learning has achieved many groundbreaking results in recent years. However, due to the increasing complexity, the execution of neural networks on embedded platforms is greatly hindered. This has motivated the development of several neural network minimisation techniques, amongst which pruning has gained a lot of focus. In this work, we perform a case study on a series of methods with the goal of finding a small model that could run fast on embedded devices. First, we suggest a simple, but effective, ranking criterion for filter pruning called Mean Weight. Then, we combine this new criterion with a threshold-aware layer-sensitive filter pruning method, called T-sensitive pruning, to gain high accuracy. Further, the pruning algorithm follows a structured filter pruning approach that removes all selected filters and their dependencies from the DNN model, leading to less computations, and thus low inference time in lower-end CPUs. To validate the effectiveness of the proposed method, we perform experiments on three different datasets (with 3, 101, and 1000 classes) and two different deep neural networks (i.e., SICK-Net and MobileNet V1). We have obtained speedups of up to 13x on lower-end CPUs (Armv8) with less than 1% drop in accuracy. This satisfies the goal of transferring deep neural networks to embedded hardware while attaining a good trade-off between inference time and accuracy.

Digitizing large histopathology archives requires processing millions of scanned whole slide images that must be validated rapidly. Automated organ-of-origin classification can accelerate quality control and enable early detection of mislabeled specimens. We developed a deep learning model that classifies the organ of origin from H&E-stained slides using a single low-resolution thumbnail per slide in under one second. For training, we used thumbnails from 16,624 slides from the TCGA and CPTAC archives, which contain mostly primary tumor resections. The images were categorized into 14 classes based on the most common primary sites in TCGA: Bladder, Brain, Breast, Colorectal, Kidney, Liver, Lung, Pancreas, Prostate, Skin, Stomach, Thyroid gland, Uterus, and Other (encompassing the remaining tissue types). We evaluated our approach on two independent external cohorts: a 5-class cohort with 2,857 slides (Colorectal, Kidney, Liver, Pancreas, Prostate) and a comprehensive 14-class cohort (12,348 slides). The model achieved 90% balanced accuracy for the 5-class cohort and 62% for the full 14-class cohort. Notably, when considering only the predictions with high confidence, 53% of the large cohort could be classified with 74% balanced accuracy. Manual review of high-confidence misclassifications suggested that some may reflect errors in the ground truth rather than model error. Mean model inference time was 0.2s per slide on an NVIDIA L4 GPU. Our deep learning approach demonstrates high classification performance with very low inference time, indicating its potential for real-time and cost-effective quality control in digital pathology.

In Convolutional Neural Networks (CNN), the need for low inference time per batch is crucial for real-time applications. To improve the inference time, we present a method (CoIn) that benefits from the use of multiple devices that execute simultaneously. Our method achieves the goal of low inference time by partitioning images of a batch on diverse micro-architectures. The strategy for partitioning is based on offline profiling on the target devices. We have validated our partitioning technique on CPUs, GPUs and FPGAs that include memory-constrained devices in which case, a re-partitioning technique is applied. An average speedup of 1.39x and 1.5x is seen with CPU-GPU and CPU-GPU-FPGA co-execution respectively. In comparison with the approach of the state-of-the-art, CoIn has an average speedup of 1.62x across all networks.

Facial emotion recognition and pose estimation have been topics of great research interest in the past, especially in the last two decades. While several methods have achieved commendable results in both domains, most of them have focused on the fidelity and accuracy of predictions. Moreover, we believe that combining facial emotion recognition with body pose/posture estimation provides a more comprehensive understanding of the actions and behavior of a person. We propose an end-to-end pipeline that combines both i) pose estimation and ii) facial emotion recognition to analyze human behaviour. We achieve real-time performance due to optimal network size and reduced number of parameters compared to existing techniques, with a minimal trade-off in accuracy. We train the networks separately and combine them at inference time to form an end-to-end pipeline. Our technique has a low inference time, achieves real-time performance without compromising in accuracy, which can be beneficial for real-world applications such as surveillance, remote supervision, and proctoring systems.

Real-time localization and classification of fast-moving objects are crucial in various applications. Traditional imaging approaches face significant challenges, including large data requirements, limited update rates, motion blur, and restrictions in non-visible wavelengths. This paper proposes an image-free method based on complementary single-pixel detection and centralized geometric moments, which effectively integrates target localization and classification into a unified framework. By employing only four specific illumination patterns, the method can simultaneously determine the centroid position and shape of the target at an update rate of up to 5.55 kHz. Theoretical simulations verify the robustness of the proposed method under similarity transformations. Experimental results indicate that the proposed system achieves accurate real-time target localization and classification under diverse conditions, with an RMSE for centroid localization below 0.5 pixels and 93.3% classification accuracy for 30 different objects. The proposed method demonstrates strong adaptability to complicated environments. It holds significant potential for applications in target tracking, character recognition, industrial automation, and the development of optoelectronic neural networks for advanced optical computing tasks.

Many deep learning (DL)-based detectors have been developed for optical remote sensing object detection in recent years. However, most of the recent detectors are developed toward the pursuit of a higher accuracy, but little toward a balance between accuracy, deployability and inference time, which hinders the practical application for these detectors, especially in embedded devices. In order to achieve a higher detection accuracy and reduce the computational consumption and inference time simultaneously, a novel convolutional network named YOLO-DSD was developed based on YOLOv4. Firstly, a new feature extraction module, a dense residual (DenseRes) block, was proposed in a backbone network by utilizing a series-connected residual structure with the same topology for improving feature extraction while reducing the computational consumption and inference time. Secondly, convolution layer–batch normalization layer–leaky ReLu (CBL) ×5 modules in the neck, named S-CBL×5, were improved with a short-cut connection in order to mitigate feature loss. Finally, a low-cost novel attention mechanism called a dual channel attention (DCA) block was introduced to each S-CBL×5 for a better representation of features. The experimental results in the DIOR dataset indicate that YOLO-DSD outperforms YOLOv4 by increasing mAP0.5 from 71.3% to 73.0%, with a 23.9% and 29.7% reduction in Params and Flops, respectively, but a 50.2% improvement in FPS. In the RSOD dataset, the mAP0.5 of YOLO-DSD is increased from 90.0~94.0% to 92.6~95.5% under different input sizes. Compared with the SOTA detectors, YOLO-DSD achieves a better balance between the accuracy, deployability and inference time.

Airport aprons serve as central hubs for various airport operations, including aircraft parking, refueling, maintenance, passenger boarding and deboarding, as well as cargo loading and unloading. Accurate and reliable monitoring of these operations is critical for maintaining operational efficiency and safety. However, airport apron environments are highly dynamic and complex, with aircraft, ground equipment, and personnel in constant motion. The <i>FastGate</i> project aims to develop a digital twin of the apron for monitoring and optimization purposes. The state of the art of the object detection and localization methods are studied, and existing datasets is analyzed. However, the existing solutions either lack full 3D detection and localization capabilities or are unsuitable for the use case at the airport aprons. This paper introduces a novel pipeline for real-time object detection and localization for airport aprons, by combining object detection and segmentation in 2D images, 2D and 3D sensors data fusion, object localization through 3D point cloud, and object orientation detection with temporal data. The apron is surveyed using a RGB camera and LiDAR sensor, to enable the automatic object detection and localization to provide real-time information about vehicles, personnel, and other objects present at the airport apron. The proposed approach integrates camera and LiDAR data for real-time object detection and localization, offering comprehensive scene awareness and enabling precise object detection and localization on airport aprons. The implemented technical solution is evaluated in terms of accuracy, localization precision, and time efficiency.

Depalletizing is a challenging task for manipulation robots. Key to successful application are not only robustness of the approach, but also achievable cycle times in order to keep up with the rest of the process. In this paper, we propose a system for depalletizing and a complete pipeline for detecting and localizing objects as well as verifying that the found object does not deviate from the known object model, e.g., if it is not the object to pick. In order to achieve high robustness (e.g., with respect to different lighting conditions) and generality with respect to the objects to pick, our approach is based on multi-resolution surfel models. All components (both software and hardware) allow operation at high frame rates and, thus, allow for low cycle times. In experiments, we demonstrate depalletizing of automotive and other prefabricated parts with both high reliability (w.r.t. success rates) and efficiency (w.r.t. low cycle times).

Dexterous Gripper for In-Hand Manipulation with Embedded Object Localization Algorithm

Computer vision (CV) and human-computer interaction (HCI) are essential in many technological fields. Researchers in CV are particularly interested in real-time object detection techniques, which have a wide range of applications, including inspection systems. In this study, we design and implement real-time object detection and recognition systems using the single-shoot detector (SSD) algorithm and deep learning techniques with pre-trained models. The system can detect static and moving objects in real-time and recognize the object's class. The primary goals of this research were to investigate and develop a real-time object detection system that employs deep learning and neural systems for real-time object detection and recognition. In addition, we evaluated the free available, pre-trained models with the SSD algorithm on various types of datasets to determine which models have high accuracy and speed when detecting an object. Moreover, the system is required to be operational on reasonable equipment. We tried and evaluated several deep learning structures and techniques during the coding procedure and developed and proposed a highly accurate and efficient object detection system. This system utilizes freely available datasets such as MS Common Objects in Context (COCO), PASCAL VOC, and Kitti. We evaluated our system's accuracy using various metrics such as precision and recall. The proposed system achieved a high accuracy of 97% while detecting and recognizing real-time objects.

Detection of objects precisely in video surveillance system is vital for many applications such as abnormal activity detection, congestion analysis, person identification, military fields. For detection of objects, background subtraction and temporal detection methods are used. In background subtraction method moving and static objects which enters into a camera viewing region has been detected. In this method reference frame will be captured during camera initialization. It uses a reference frame as background, detects present frame and takes the difference between two frames. In this paper objects are detected in multiple regions separately from single camera instead of using multiple cameras to monitor different objects in different regions. And also user will have control to select required single or multiple regions to detect the object. This will reduces the overall cost of the system and increases the processing speed of the system if we choose a particular region. Accuracy of the object detection in different regions increases since it allows using different parameters like thresholding separately. So that it monitors different activities at the same time. Also detection of objects based on comparison, object of interest from stored data base is presented using single static camera view with good real time performance.

ABSTRACTIn robotics, navigating a humanoid robot through a cluttered environment is challenging. The present study aims to enhance the footstep and determine optimal paths regarding the robot's route length. The objective function for navigation of multiple humanoid robots is presented to optimize the route length and travel time. A hybrid technique using a probabilistic roadmap (PRM) and firefly algorithm (FA) is presented for humanoid robot navigation in a cluttered environment with static and dynamic obstacles. Sensory information, such as barrier range in the left, right, and front directions, is fed into the PRM framework that allows the humanoid robot to walk steadily with an initial steering angle. It finds the shortest path using the Bellman–Ford algorithm. The FA technique is used for efficient guidance and footstep modification in a cluttered environment to find a smooth and optimized path. To avoid static obstacles, the suggested hybrid technique provides optimum steering angles and ensures the minimum route length by taking the output of PRM as its input. A 3D simulator and a real‐world environment have been used for simulation and experiment in a cluttered environment utilizing the developed model and standalone methods. The humanoid robot achieves the target in all scenarios, but the FA‐tuned PRM technique is advantageous to this purpose, as shown by the convergence curve, route length, and travel duration. Multiple humanoid robot navigation has an additional self‐collision issue, which is eliminated by employing a dining philosopher controller as the base technique. In addition, the proposed controller is evaluated in contrast to the existing technique. The developed strategy ensures effectiveness and efficacy depending on these findings.

Object localization is a sub-field of computer vision-based object recognition technology that identifies object classes and locations. Studies on safety management are still in their infancy, particularly those aimed at lowering occupational fatalities and accidents at indoor construction sites. In comparison to manual procedures, this study suggests an improved discriminative object localization (IDOL) algorithm to aid safety managers with visualization to improve indoor construction site safety management. The IDOL algorithm employs Grad-CAM visualization images from the EfficientNet-B7 classification network to automatically identify internal characteristics pertinent to the set of classes evaluated by the network model without the need for further annotation. To evaluate the performance of the presented algorithm in the study, localization accuracy in 2D coordinates and localization error in 3D coordinates of the IDOL algorithm and YOLOv5 object detection model, a leading object detection method in the current research area, are compared. The comparison findings demonstrate that the IDOL algorithm provides a higher localization accuracy with more precise coordinates than the YOLOv5 model over both 2D images and 3D point cloud coordinates. The results of the study indicate that the IDOL algorithm achieved improved localization performance over the existing YOLOv5 object detection model and, thus, is able to assist with visualization of indoor construction sites in order to enhance safety management.