Cluster-Based Flight Path Construction for Drone-Assisted Pear Pollination Using RGB-D Image Processing

Simple SummaryOver the past decades, Integrated Pest Management (IPM) strategies have been widely adopted in commercial fruit production in Europe, supporting natural pest control as an ecosystem service. At the same time, there has been a growing awareness of the importance of pollinating insects, leading to the concept of Integrated Pest and Pollinator Management (IPPM). Here we present the outcomes of a 4-year case study as a valuable illustration of an IPPM strategy in a commercial intensive pear orchard. We show how the added-value of local biodiversity measures can be visualized in front of growers, linking ecological measures to economic benefits. This scientifically-based as well as practice-oriented demonstrative case study supports the acceptance and adoption of IPPM principles in commercial intensive pear production cultivation.Recently, the concept of Integrated Pest Management (IPM) was further extended into Integrated Pest and Pollinator Management (IPPM). Implementation of IPPM strategies entails the combination of actions for pest and pollinator management providing complementary or synergistic benefits for yield and/or quality of the harvest. The aim of this study was to examine IPPM elements (i.e., mixed hedgerow, nesting boxes for mason bees, Osmia spp.) and demonstrate their impact in the practical context of modern commercial fruit cultivation in a 4-year case study in an intensive ‘Conference’ pear orchard. The outcomes of visual observations during transect walks and molecular analysis of pollen collected by mason bees, showed the importance of additional floral resources for the presence of mason bees and other pollinating insects in the orchard environment. Pear quality assessments indicated that insect-mediated pollination had a significant positive impact, with a tendency for higher quality pears in the close vicinity of Osmia nesting boxes. However, despite the fact that pear pollen was also detected in Osmia spp. nest cells, the amount and frequency of pear pollen collection for their nest built-up turned out to be rather low. In the same intensive pear orchard studied for pollination effects, we simultaneously demonstrate the impact of a mixed hedgerow to enhance integrated pest control.

Nursery plays an important role on starting chili cultivation, determining the crop health, fertility from disease, and growth performance. Early-stage germination detection is necessary to minimize nursery failure and improve plant health, but manual detection is challenging for large scale nursery in the greenhouse. The aim of this research was to develop an automatic detection model integrated with a You Only Look Once (YOLO) based deep learning algorithm using RGB camera to monitor the chili germination stages. Method to detect germination was YOLO with several steps, included: (1) early stages chili germination images acquisition, (2) datasets preparations, (3) dataset annotation and labeling, (4) model development using deep learning YOLO algorithms, and (5) model testing and validation. The training of 11,423 images was conducted utilizing the YOLOv5 and YOLOv8 algorithms, which categorized into, three classes (germinated, not germinated, and cotyledon appearance). The model was evaluated using mean Average Precision (mAP), precision, accuracy, and recall with the respective values of 0.697, 73%, 75%, and 73% for YOLOv8, and 0.664, 70%, 73%, and 70% for YOLOv5. Both model achieved high accuracy, but YOLOv8 was better to detect and classify chili seedling growth stages than YOLOv5. This study also demonstrated that model can be implemented in real applications integrated with automatic monitoring system included in the model. Keywords: Chili seedling, Deep learning, Detection system, Germination.

Fruit picking in complex orchard environments is limited by the low detection and recognition accuracy due to clustered background, illumination variation, and partial occlusion. To improve the picking performance, a hierarchical visual grasping architecture based on You Look Only Once (YOLO) algorithm and adaptive error compensation is proposed. The upper layer uses YOLOv5 and inverse kinematics to recognize and localize the target. Ant colony optimization is specifically used for hyperparameter tuning of YOLOv5 to improve the detection accuracy. The middle layer dynamically compensates for the output torque of the joint actuator through the feedback linearization method. The lower layer finishes precise grasping through nonlinear mapping model between the pulse width modulation signal and the servo angle. Experimental results validate that the proposed architecture outperforms traditional methods by reaching 98.2% and 97.7% recognition accuracy in obstacle-free scenarios and complex environments, a higher grasping success rate, and a lower positioning deviation.

Early diagnosis of cerebral palsy in infants has produced promising results using tools like the General Movement Assessment (GMA). Pose estimation of infants lying supine is an important step towards an automated system for GMA. Developing methods for accurate, reliable, fast estimation of the three-dimensional (3D) position of human limbs and proposing motion features suitable for 3D models to classify typical and atypical movements are attracting increasing research interest lately. In this study, we propose a 3D pose estimation method with low training cost suitable for infants in lying positions. The method uses an existing two-dimensional human body keypoint detection method combined with the in-depth information in Red-Green-Blue Depth (RGB-D) data from a Kinect sensor. The method is evaluated using the Moving INfants In RGB-D (MINI-RGBD) open dataset. The results show that the average error of the estimated body part length is 13.76 mm, while the accuracy of the Percentage of Correctly-localized Parts (PCP) and Percentage of Correct Keypoint (PCK) is 80.7 and 86.1%, respectively. The results are comparable to those achieved in the baseline study performed by the researchers who generated this open dataset. The advantage of our method is its low training cost.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

Human Activity Recognition is one of the active research areas in computer vision for various contexts like security surveillance, healthcare and human computer interaction. In this paper, a total of thirty-two recent research papers on sensing technologies used in HAR are reviewed. The review covers three area of sensing technologies namely RGB cameras, depth sensors and wearable devices. It also discusses on the pros and cons of the mentioned sensing technologies. The findings showed that RGB cameras have lower popularity when compared to depth sensors and wearable devices in HAR research.

Five-axis complex surface machining on a Computer Numerically Controlled (CNC) machine needs CAD/CAM system for the cutter location and orientation data. Since these data are defined with respect to the coordinate system of work-piece, they need to be converted to the machine’s control commands in machine coordinate system, through processing so called a CNC post-processing. In this work, a kinematic model method and a efficient algorithm for CNC post-processing have been presented for a typical five-axis milling machine of table rotating/tilting type. The proposed algorithms have been realized in our developing CNC system.

This study presents a vision-based human action recognition system using a deep learning technique. The system can recognize human actions successfully when the camera of a robot is moving toward the target person from various directions. Therefore, the proposed method is useful for the vision system of indoor mobile robots.

Over the last year the need for video conferences has risen significantly due to the ongoing global pandemic. The goal of this project is to improve user experience from having access to only voice and plain 2D image by adding a third spatial dimension, creating a more immersive setting. Azure Kinect Development Kit utilizes multiple cameras, namely the RGB camera and depth camera. Depth camera is based on ToF principle, which uses near-IR to cast modulated illumination onto the scene. The setup uses multiple Azure Kinect devices in sync and offset in space to obtain non-static 3D capture of a person. Unity engine with Azure Kinect SDK is used to process the data gathered by all devices. Firstly, a depth spatial map is created by combining overlaid outputs from each device. Secondly, RGB pixels are mapped onto depth spatial points to provide a final texture to the 3D model. Taking into account the need to export a continuous capture of raw data to a server, body tracking and image processing algorithms are used. Finally, the processed data can be exported and utilized in AR, VR or any other 3D capable interface. This 3D projection aims to enhance sensory experience by utilising non-verbal communication along with classical speech in video conferences.Keywords3D sensingVolumetric alignmentPoint cloud

A Positioning Method for Apple Fruits Based on Image Processing and Information Fusion

Objective To explore the optimal cortical puncture site in deep brain stimulation of subthalamic nucleus (STN-DBS). Methods Fifty patients of Parkinson’s disease (PD) were enrolled into this study who were consecutively admitted to Department of Neurosurgery, General Hospital of Shenyang Military Command from September 2016 to April 2017 and underwent bilateral STN-DBS (100 electrodes implanted) assisted by the ROSA (robotized stereotactic assistance) system. CT angiography and MRI fused images were used to avoid the blood vessels in plan of surgical trajectory. Data of microelectrode recording (MER) in 100 trajectories were documented, out of which 80 trajectories with MER≥5 mm were selected. Cartesian coordinate system was established in the 3D image of ROSA. The distances were measured which were from the cortical entry point to the median sagittal line and from the cortical entry point to projected line of coronal suture of the skull onto the cortex. Moreover, the reference range was calculated and the optimal cortical puncture site was decided. Results There was no case of cerebral hemorrhage in this series. For those electrode implantation with MER ≥5 mm, distances from the entry point to the median sagittal line were decided to be 50.8 mm (maximal), 29.6 mm (minimal) and 37.9±5.0 mm (average). The distances between the cortical entry point to projection of coronal suture were 30.3 mm (maximal), -8.5 mm (minimal) and 9.0±6.5 mm (average). Conclusion In STN-DBS, choosing cortical puncture site within the area (32.9-42.8 mm lateral to the midline and 2.5-15.5 mm anterior to the coronal suture) might have greater chance of achieving satisfactory electrophysiological recordings, which could also be useful in operations assisted by the traditional stereotactic frame. Key words: Deep brain stimulation; Subthalamic nucleus; Robotized stereotactic assistant; Cortical puncture point; Microelectrode recording

The co-registration between an optical tracker and Magnetic Resonance Imaging (MRI) space is an indispensable step for MRI-guided surgery. In this study, with a focus on RGB cameras as the tracker, we introduce an innovative co-registration scheme for tracker-to-MRI integration. Firstly, we design a cube-shaped registration model that is equipped with an ArUco marker on its exterior for RGB camera detection and houses four water blobs inside for MRI calibration. Secondly, we employ a line scan pulse sequence for the localization and reconstruction of the water blobs. Lastly, we establish the transformation relationship between the camera and MRI coordinate systems. Our registration scheme was implemented on a 0.35T MRI system, accompanied by a magnetically shielded RGB camera. In comparison to conventional image domain-based phantom blob reconstruction techniques, the line scanning method showcased lower registration errors and achieved scanning speeds over ten times faster. In needle localization accuracy experiments, the needle tip position, as determined by the ArUco marker on the handle, deviated by a mere 1.008 mm from its actual MRI scan position. Our results highlight the considerable potential for cost-effective RGB cameras in MRI-guided surgeries. Moreover, our registration scheme is not confined to RGB cameras and can be generalized to other optical trackers by simply substituting the corresponding marker. The proposed scheme promises to streamline and automate the co-registration process, thereby reducing surgery preparation time and bolstering the clinical applicability of MRI-guided surgeries.

The development of an unmanned aircraft system (UAS) traffic management (UTM) system for the safe integration of unmanned aerial vehicles (UAVs) requires pre-flight conflict detection and resolution (CDR) methods to provide collision-free flight paths for all UAVs before takeoff. A popular solution consists in adapting multi-agent path finding (MAPF) techniques. However, standard MAPF solvers consider only a fixed takeoff time and fixed uniform speed for each UAV flight path, which can lead to inefficiencies in the resolution of instances. Therefore, in this article, we propose incorporating scheduling elements into MAPF solvers, which allows us to adjust the takeoff times and speeds of each UAV to solve conflicts. We introduce two time-related resolution techniques: 1) takeoff scheduling, whereby the start time of a UAV agent is delayed, and 2) speed adjustment, wherein the speed of a UAV agent is decreased over a segment on its flight path. Importantly, we present a distinction of conflict types, which enables us to combine replanning resolution to the aforementioned temporal resolution techniques. We evaluate our proposed approaches on a realistic, high-density UAV delivery scenario in Tokyo, Japan. We show that the combination of takeoff scheduling, replanning, and speed-adjustment resolution techniques improves the efficiency of route planning by reducing the average delay per flight path and the number of rejected flight paths.

This study presents simple controllers for trajectory tracking of the kinematic model of an Unmanned Aerial Vehicle (UAV). We consider the effects of wind perturbations on an UAV in flight, and consider the applications of an existing controller with a wind compensation term while the control objective is to track a time-varying trajectory which is given in terms of the inertial coordinate system. Application of the approach to UAVs flying in a string-like formation in the presence of additive wind perturbations is also considered. Numerical and simulation results are demonstrated.

The aim of my thesis was to investigate how reaching for visual targets placed in 3D space influences the coordinate frames and the kinematics in non-human and human primates. To this end, I conducted three studies. The first study was conducted on non-human primates to find the predominant reference frame of cells in a specific reach related area of the PPC (area PEc) while reaching towards targets placed at different depths and directions; we tested whether PEc reaching cells displayed hand- and/or body-centered coding of reach targets. We found that the majority of PEc neurons encoded targets in a mixed body/hand-centered reference frame. Our findings highlight a role for area PEc as intermediate node between the visually dominated area V6A and the somatosensory dominated area PE. The second study was conducted on healthy human subjects to find the reference frame used while reaching towards targets placed at different depths and directions. Our results revealed reach error patterns based on both eye- and space-centered coordinate systems: in depth more biased towards a space-centered representation and in direction mixed between space- and eye-centered representation. The third study was conducted on a patient with a parietal cortex lesion who showed optic ataxia symptoms. Optic ataxia patients show deficits in visuo-manual guidance especially when reaching to targets located in the periphery of the visual field. By manipulating gaze position and hand position of visual reaching targets, placed at different depth and directions, we investigated how reaching in peripheral and central viewing conditions influenced the trajectories and reach errors of the patient and controls. Our results suggest that the reaching inaccuracies observed, in particular in the configurations where the direction of gaze and reach differed, are due to a disruption of the online correction mechanism and that the PPC is involved in these automatic corrections.