Mixed-reality for quadruped-robotic guidance in SAR tasks

Frequent natural disasters and man-made catastrophes have threatened the safety of citizens and have attracted much more attention. The rescue mission under disaster environment is very complicated and dangerous for a rescue team. Search and rescue (SAR) robots can not only improve the efficiency of rescue operations but also reduce the casualty of rescuers. Robots can help rescue teams and even replace rescuers to perform dangerous missions. Search and rescue robots will play a more and more important role in the rescue operations. A survey of the research status of search and rescue robots in Japan, USA, China and other countries has been provided. According to current research, experiences and the lessons learned from applications, the five key performances of a search and rescue robot are survivability, mobility, sensing, communicability and operability. Multi-technique fusion and multi-agent intelligent network are considered to be requirements for the future development of the search and rescue robot. Disaster prevention, disaster reduction and disaster rescue are the important parts of national public safety. They are also crucial for the safety of citizens and their estates. Search and rescue robotic technique is an urgent needed, strategic and core technique for national development. It will be important and strategic for national economy and safety.

The Tampen Area contains huge oil and gas fields. The population in the area is more than 2000 persons at work at any time. Statoil is the major operator in the area and responsible for an efficient and well-trained emergency organization, including SAR missions and medical evacuations. Statoil established a Search and Rescue service (SAR) in 1981. Through the years since then the service has undergone major development and improvement and represents a major contribution to the safety and security of personnel working in the area. SAR also represents an important resource for the national Joint Rescue Coordination Centre in Stavanger. Since 1981 several hundred patients has been evacuated to hospitals onshore due to severe medical conditions and injuries. In our earlier review, presented in Jakarta 1994 (SPE 27220)1, the main cause for medical evacuations were diseases or injuries in the muscular-skeletal system followed by problems in the respiratory and gastro-intestinal tract. During the last 10 years the number of evacuations due to cardio-vascular disease has increased. The proportion of evacuations due to this diagnosis has risen from 4 % to approx. 40%. In our paper we will present statistics on medical evacuations from the area, including information on new medical training, equipment and standards. We also describe our standards on treating acute heart disease, prehospital diagnosis through transfer of ECG using telemedicine for obtaining correct medical diagnosis directly from the emergency care units in hospitals. We also present the standards on prehospital trombolysis administrated by nurses in the SAR service and our cooperation with the specialist staff at Haukeland University Hospital. The presentation will contain short case studies covering the most important diagnosis such as myocardial infarction and multi-trauma injuries and treatment.

Visual object recognition is a fundamental challenge for reliable search and rescue (SAR) robots, where vision can be limited by lighting and other harsh environmental conditions in disaster sites. The goal of this paper is to explore the use of thermal and visible light images for automatic object detection in SAR scenes. With this purpose, we have used a new dataset consisting of pairs of thermal infrared (TIR) and visible (RGB) video sequences captured from an all-terrain vehicle moving through several realistic SAR exercises participated by actual first response organisations. Two instances of the open source YOLOv3 convolutional neural network (CNN) architecture are trained from annotated sets of RGB and TIR images, respectively. In particular, frames are labelled with four representative classes in SAR scenes comprising both persons (civilian and first-responder) and vehicles (Civilian-car and response-vehicle). Furthermore, we perform a comparative evaluation of these networks that can provide insight for future RGB/TIR fusion.

Recent developments in unmanned aerial vehicles (UAV), artificial intelligence and miniaturized thermal imaging systems represent a new opportunity for search and rescue (SAR) experts to survey relatively large areas. The system presented in this paper includes thermal image acquisition, the video processing pipeline that performs object detection and classification of people in need of SAR in outdoor environments. The system is tested on thermal video data from ground based and test flight footage and is found to be able to detect all the target people located in the surveyed area. The training dataset is a combination of gathered data and internet sourced data. The initial data procurement utilised online academic thermal databases and also the simulation of the UAV mounted camera environment. Ground based data was collected at Kangaroo Point cliffs in Brisbane, Australia, giving an approximate elevation of 26m. Airborne datasets were collected at South Bribie Island in Queensland, Australia, at a range of heights and vegetation density. These datasets where collected at different times of the day, allowing for a range of contrast levels between background and intended target. Once all data was collected, individual frames where extracted from each image and augmentation and annotation was completed. The images were gaussian blurred, lightened and darkened, once all annotation were completed. A total of 2751 original images were annotated, with the augmented dataset comprising of 10380 images. The YOLOV3 algorithm was selected as the neural network (NN) to be used for this experiment throughout training and classification. The `Experiencor' GitHub pipeline was also used throughout the entirety of this project for data output and analysis purposes. The algorithm training was implemented on the Queensland University of Technology (QUT) High Performance Computing (HPC) cluster. Two main models were trained using different hyperparameters for comparison purposes. The first model had a loss of 3.81, AP of 98.6 after ~88 hours of training, with model two having a loss of 4.73, AP of 97.7 after ~184 hours of training. The comparison shows the importance of the chosen parameters when training detection algorithms of this nature, as minor changes can be the difference between an efficiently trained model and an inefficient failed training attempt. The prediction testing was completed on the test data sets that were not included in the training of the two models. This was done to remove any bias from the system, although it is noted that due to the shared environments of two of the test sets with the training sets, a small amount of bias will exist. Predictions made on the two data sets sharing environmental conditions with the training data, showed good prediction results for both trained models, with very limited false positive and missed detections. The system is flexible in that the user can readily define the types of objects to classify and the object characteristics that should be considered during training.

Length Measurement of Potato Leaf using Depth Camera

Earthquake is one of the natural disaster types that suddenly breaks regular human life. Rescue activities in disasters are one of the most critical stages of modern disaster management. This management stage, as mentioned earlier, includes all the activities that need to be done after the disaster. Search And Rescue (SAR) teams perform one of these most critical activities after the earthquake post-disaster period. Search and rescue teams that will rescue and relief after a disaster are selected according to the criteria selected. Location layout selection problems are NP-Hard, and obtaining hard results is in the class of these problems. One of these criteria is the Risk Pressure Factor (RPF) used in determining the priorities of the risk areas. Determining the level of risk level is very difficult and also these are difficult to predict. In this study, it is aimed to estimate this parametric value by using an artificial neural network (ANN) method which is applied in many fields. And then in this study, a prediction model was constructed by using back propagation method which is a suitable propagation method in ANN method and results are obtained from the MATLAB program. The resulting risk-pressure factor (RPF) value can be used as a parameter in the proposed mathematical model. As a result of the study, the missing parameter of the mathematical model will be found in the estimation of a parameter belonging to the proposed mathematical model.

Empowering industrial robotic training with kinesthetic learning and digital twins in human-centric industrial systems

The development of maritime trade and operations has led to a gradual increase in maritime accidents. Research on the specificity of maritime search and rescue (SAR) missions is essential to improve mission efficiency; however, traditional SAR programs usually use predefined paths to conduct searches, which is difficult to meet the timeliness and uncertainty requirements. To solve the challenge, we propose a Boundary Adaptive Neural Network Coverage Path Planning Scheme based on Target Drift Prediction (BANCP‐TDP). The framework includes three modules: drift trajectory prediction, optimal region determination, and coverage search. First, the Limited Red‐billed Blue Magpie Optimizer Back Propagation drift model is used to predict the drift trajectory of the wrecked target. Subsequently, we use the Multiphysics Monte Carlo Gravitational Search prediction model to determine the distribution of targets at different moments and the optimal SAR region for guiding an autonomous underwater vehicle to carry out SAR missions. Then, we propose a BANCP for the SAR regions with complex boundaries, aiming to minimize the path length and maximize the coverage ratio. The comparative field experiments at Qingdao Jin Cao Gou reservoir and simulation results show that the proposed framework can effectively shorten path length while minimizing the repeated paths.

Fall is the biggest threat to seniors, with significant emotional, physical and financial implications. It is the major cause of serious injuries, disabilities, hospitalizations and even death especially for elderly people living alone. Timely detection could provide immediate medical service to the injured and avoid its harmful consequences. Great number of vision-based techniques has been proposed by installing cameras in several everyday environments. Recently, deep learning has revolutionized these techniques, mostly using convolutional neural networks (CNNs). In this paper, we propose weighted multi-stream deep convolutional neural networks that exploit the rich multimodal data provided by RGB-D cameras. Our method detects automatically fall events and sends a help request to the caregivers. Our contribution is three-fold. We build a new architecture composed of four separate CNN streams, one for each modality. The first modality is based on a single combined RGB and depth image to encode static appearance information. RGB image is used to capture color and texture and depth image deals with illumination variations. In contrast of the first feature that lacks the contextual information about previous and next frames, the second modality characterizes the human shape variations. After background subtraction and person recognition, human silhouette is extracted and stacked to define history of binary motion HBMI. The last two modalities are used to more discriminate the motion information. Stacked amplitude and oriented flow are used in addition to stacked optical flow field to describe respectively the velocity, the direction and the motion displacements. The main motivation behind the use of these multimodal data is to combine complementary information such as motion, shape, RGB and depth appearance to achieve more accurate detection than using only one modality. Our second contribution is the combination of the four streams to generate the final decision for fall detection. We evaluate early and late fusion strategies and we have defined the weight of each modality based on its overall system performance. Weighted score fusion is finally adopted based on our experiments. In the third contribution, transfer learning and data augmentation are applied to increase the amount of training data, avoid over fitting and improve the accuracy. Experiments have been conducted on publicly available standard datasets and demonstrate the effectiveness of the proposed method compared to existing methods.

This paper outlines the design and implementation of novel object manipulation for a social robot, here Pepper by SoftBank Robotics. It is primarily designed for verbal interaction and has therefore not been equipped with object manipulation capabilities. The proposed routine exploits the built-in RGB and 3D cameras. First, semantic segmentation based on the Mini-YOLOv3 neural network is run on the RGB image. Next, 3D sensor data are used to position the hand over the object, implementing a novel routine to grab the object and to scan it for recognition purposes. To preserve patient and location sensitive data, the here-proposed architecture operates automatically and offline, running on the robot’s operating system. Experimental results on 370 grabbing processes showed how the manipulation routine achieves a grabbing success rate of up to 96%. They also proved that the success rate remains unaltered if the target object is positioned in a rectangular area of ± 6 cm × ± 3 cm centered in the nominal position provided by an initial positioning grid. The grabbing success rate remains above 80% even if the object to be grabbed is stored with an angle that ranges between 10° and 45° within the above-reported area.

Beyond traditional, dyadic human-robot interaction, embedding robots into multi-human teams, such as search and rescue (SAR), requires an understanding of fundamental aspects of team composition and dynamics. While considerable work has examined how robot agents influence both taskwork and teamwork, few studies have focused on identifying which factor best explains differences in team outputs. This research investigates the neurodynamics of taskwork and teamwork as SAR teams transition between multi-human (mH) and multi-human-robot (mHR) configurations. Electroencephalogram (EEG) has been a key tool in human teamwork research because of its sensitivity to changes in cognitive states such as mental workload, sustained attention, and engagement. Specifically, EEG power spectral density (PSD), particularly frontal theta activity (4-7 Hz), has been used to assess variations in mental workload and social cognition associated with task performance. EEG hyperscanning, which evaluates interbrain synchrony between two or more individuals, using metrics such as weighted phase lag index (wPLI), has been widely employed to study teamwork among humans. In this study, PSD and EEG hyperscanning were used to analyze taskwork and teamwork in 22 teams comprising a highly engaged SAR team member (mission commander), a less-involved member (safety officer), and a navigator as they searched for victims in a virtual emergency environment. The navigator was either a trained researcher posing as a participant or a virtual robot, with the robot's performance manipulated using the Wizard of Oz technique. Results for taskwork show that the social-cognitive abilities of mission commanders, but not those of safety officers, are adversely impacted by a robot navigator compared with a human navigator, despite the perceived workload remaining stable. Although team trust outcomes were similar, neural synchrony across occipital, parietal, and temporal regions increased in mHR teams relative to mH teams, indicating different neurodynamical patterns of teamwork. The study findings provide evidence that both taskwork and teamwork are fundamentally altered in mHR teams, regardless of the effectiveness of robotic capabilities and functions, compared with mH teams. Therefore, beyond dyadic interactions, multi-human robot teaming must be viewed as a fundamentally distinct team construct rather than simply an extension of human-human teaming.

As the use of flying robots in various types of human-robot collaborative work continues to increase, but their interaction with humans remains somewhat deficient, it has become critical to ensure that flying robots are able to convey information to humans in a timely and accurate manner. Lightspeak is an effective means to achieve this goal by controlling the lights on flying robots to convey information in a specific pattern, color, or frequency. In the field of industrial robotics, specific light patterns can be used to indicate the robot's working status, production progress, or possible malfunctions. Such visual signals can help plant operators quickly recognize robot status and act accordingly, thereby increasing productivity and reducing potential errors or hazards. Although the application of light language in industrial robotics has achieved some success, the research on human-robot interaction for flying robots is still relatively limited. Especially in dynamic and complex environments, how to design efficient light language patterns to adapt to the multi-dimensional motion characteristics of flying robots and to ensure that humans can quickly interpret light language messages remains an urgent challenge. Using virtual reality (VR) technology, this study aims to develop a set of flying robot intention characterization methods based on light signals by simulating flying robot light changes, designing light language rules applicable to the multi-degree-of-freedom motion and warning states of flying robots, and optimizing their performance in user understanding and interaction through experiments. Based on the task analysis of the flying robot, this study identifies seven degrees of freedom of motion and three main warning modes by decomposing the motion and warning states of the flying robot in detail. Based on this, we designed three basic light modes including constant light, blinking, and surge, and optimized their colors, blinking frequencies, and timings. The experiment uses 73 motion state light languages and 8 alarm state light languages to test the characterization effect and user preference of different light languages in multiple contexts through animated video and randomized design. The experimental study shows that in the stationary state, the all-light constant light signal has the highest preference rate, while in the dynamic motion state and emergency alarm state, the flashing signal (e.g., double flashing, long and short flashing) exhibits higher saliency and information transfer efficiency. Users' choices of light language patterns are highly correlated with their visual saliency, clarity of signal meaning, and situational urgency. The red light signal has a significant advantage in the warning state, and the high-frequency flashing signal is the most effective in the “hardware failure” scenario, while the stable constant light signal is more suitable for the “active stop” scenario. In this study, we propose a light language design method that combines perceptual theory, color design, and light animation, which significantly improves the efficiency of the flying robot's message delivery in multi-degree-of-freedom motion and alarm states. Through experimental verification, this set of light language rules provides important support for the safety and human-robot interaction ability of flying robots in complex application scenarios, and lays a foundation for the expansion of the light language in the field of robotics.

In recent years, Unmanned Aerial Vehicles (UAVs) have become essential for important tasks like search and rescue (SAR), environmental monitoring, disaster assessment, and surveillance. The effectiveness of these AI-driven vision systems depends greatly on stable communication links between UAVs and ground stations. Packet loss and transmission issues often reduce the quality of visual data. This, in turn, lowers the accuracy of automated detection and analysis. To address this problem, this paper presents a Hybrid Dual-Domain Attention and Transformer Network (Hybrid DDAC) designed to restore UAV images that are damaged by communication-related distortions. The network combines multi-scale convolutional feature extraction, spatial and channel attention, transformerbased global reasoning, and an Adaptive Communication-Aware Correction Layer (ACCL) to effectively recover local details while keeping global consistency. A dataset containing 2,405 original UAV images and 2,405 corresponding packet-lossdistorted images from an open-source repository was used for training and evaluation. The proposed Hybrid DDAC outperforms traditional models by achieving better PSNR, SSIM, and MAE values. Experimental results show that the Hybrid DDAC significantly improves image clarity and structural integrity, making UAV-based vision systems more robust under unreliable communication conditions.

In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary in low light condition, the multispectral fusion of RGB and NIR images provides a viable solution to low light imaging. In this paper, we propose multispectral fusion of RGB and NIR images using weighted least squares (WLS) and convolution neural networks (CNNs). We combine traditional WLS filtering for layer decomposition and denoising with latest deep learning for image enhancement and texture transfer into the multispectral fusion to take both advantages. We build two networks based on CNN: image enhancement network (IEN) for image enhancement and texture transfer network (TTN) for NIR texture transfer. First, we perform RGB image denoising based on WLS filtering and generate the base layer. We use both RGB and NIR images for WLS filtering as weights to filter out noise in low light RGB images. Second, we conduct IEN to enhance contrast of the base layer. Third, we perform TTN to deliver NIR details completely and naturally to the fusion result. The combination of WLS, TTN and IEN leads to noise reduction, contrast enhancement, and detail preservation in fusion. Experimental results show that the proposed method achieves good performance in both noise reduction and detail transfer as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.

The ability to recognize and interact with a variety of doorknob designs is an important component on the path to true robot adaptability, allowing robotic systems to effectively interact with a variety of environments and objects The problem addressed in this paper is to develop and implement a method for recognizing the position of a door handle by a robot using data from an RGBD camera. To achieve this goal, we propose a revolutionary approach designed for autonomous robots that allows them to identify and manipulate door handles in different environments using data obtained from RGBD cameras. This was achieved by creating and annotating a complete dataset consisting of 5000 images of door handles from different angles, with the coordinates of the vertices of the bounding rectangles labeled. The architectural basis of the proposed approach is based on MobileNetV2, combined with a special decoder that optimally increases the resolution to 448 pixels. A new activation function specially designed for this neural network is implemented to ensure increased accuracy and efficiency of raw data processing. The most important achievement of this study is the model's ability to work in real-time, processing up to 16 images per second. This research paves the way for new advancements in the fields of robotics and computer vision, making a substantial contribution to the practical deployment of autonomous robots in a myriad of life's spheres.