Vision-guided robot hearing

Some robots for human-robot interaction are designed with female or male physical appearance. Other robots are endowed with no gender characteristics, namely genderless robots, such as Pepper and NAO robot. A robot with male or female physical appearance should possess the mapped speech gender style during a natural human-robot interaction, which can be learned from humans' male or female speech. In this paper, we make a new trial to synthesis gender-free speeches for physically genderless robots, which is promising in order to improve a more natural human-robot interaction with genderless robots. Our gender style-controlled speech synthesizer takes the speech text and gender style embedding as inputs to generate speech audio. A speech gender encoder network is used to extract the embedding of the speech gender style with female and male speeches as input. Based on the distribution of the female and male gender style embedding, we explore the gender-free speech style embedding space where we sample some gender-free embedding vectors to generate genderless speech audio. This is a preliminary work where we show how the genderless speech audio wave will be synthesized from text.

Aim/Purpose: Our study is focused on prototyping, development, testing, and deployment of a new knowledge primitive for the humanoid robot assistant NAO, in order to enhance student visual learning by establishing a human-robot interaction. Background: This new primitive, utilizing a convolutional neural network (CNN), enables real-time recognition of handwritten digits captured by the NAO robot, a humanoid robot assistant developed by SoftBank Robotics. It is equipped with advanced capabilities, including a wide range of sensors, cameras, and interactive features. By integrating the proposed primitive, the NAO robot gains the ability to accurately recognize handwritten digits, contributing to improved student visual learning experiences. Methodology: Our developed primitive consists of the use of a convolutional neural network (CNN) so that the robot is able to recognize the handwriting of the digits present in the input image received in real-time. The NAO robot establishes interaction with the learners through a scenario based on a predefined assignment. In this scenario, NAO captures the digit handwritten by the learner via its camera, recognizes the digit using the deep learning model generated by the MNIST dataset, and announces to the learner the handwritten digit in the input image. The prototype is realized using the concept of a distributed system allowing the distribution of tasks in four different computing nodes. Contribution: Our research makes a significant contribution by equipping the humanoid robot NAO with a cognitive intelligence system through the integration of a new knowledge primitive based on handwriting digit recognition (HWDR). Our approach used to create and implement this primitive in the NAO robot is interesting and innovative, and presents a promising provision for enhancing the visual learning experience of children and young students with special needs, based on the use of distributed systems that divide the work using various components distributed over several nodes, coordinating their efforts to perform tasks more efficiently than a single device besides the NAO robot. Findings: We designed our model using specific parameters and a fully convolutional neural network architecture, which includes three residual depthwise separable convolutions, each followed by batch normalization and ReLU activation. To evaluate the performance of our model, we tested it on the MNIST dataset, where we achieved a remarkable accuracy, F1 score, and recall of 99%. An experiment was conducted to test our implemented primitive and see the effectiveness of this invention for enhancing visual learning in children with special needs. We developed a visual learning strategy based on the creation of engaging activities mediated by the NAO robot in an educational context. The results showed that participants achieved a strong commitment to the NAO robot, appreciating its ability to recognize handwritten digits and highlighting its promising potential to enrich visual learning experiences. Participants expressed a strong preference for teaching methods integrating assistive learning technologies, demonstrating the positive impact of our humanoid assistant robot on improving learning and visual intelligence in an educational environment. Recommendations for Practitioners: Encourage creativity and innovation in the field of robotics and special needs. This can lead to new and effective solutions that improve the lives of students with special needs. Recommendation for Researchers: Test and evaluate the proposed robotics solutions to ensure they are effective and making a positive impact. Use feedback from users, educators, and parents to refine and improve your solutions. Also, ensure that the robotics solutions are accessible to students with a range of abilities. This may involve designing solutions that are adjustable or providing alternative means of access. Impact on Society: As there are several ways to educate, there are multiple forms of learning. With the help of this learning procedure and strategy, the human teacher collaborates with the robot assistance NAO to improve visual learning among students. The findings of this research can serve as an application for the implementation of various pedagogical methods that will assist in meeting the needs of the majority of learners. Future Research: Our future research will concentrate on addressing the educational needs of students with special needs, enabling them to overcome their challenges and reach academic excellence in an inclusive environment. To achieve this goal, we plan to leverage the capabilities of social robots, which have emerged as a significant contributor to the field of human-robot interaction, particularly in facilitating inclusive education. These agents have proven to be effective in providing support to students with special needs, thereby enabling them to receive the education they need to succeed.

This study presents a novel approach using Wasserstein Generative Adversarial Networks with Gradient Penalty (WGAN-GP) to generate synthetic electroencephalography (EEG) and electrocardiogram (ECG) waveforms. The synthetic EEG data represent concentration and relaxation mental states, while the synthetic ECG data correspond to normal and abnormal states. By addressing the challenges of limited biophysical data, including privacy concerns and restricted volunteer availability, our model generates realistic synthetic waveforms learned from real data. Combining real and synthetic datasets improved classification accuracy from 92% to 98.45%, highlighting the benefits of dataset augmentation for machine learning performance. The WGAN-GP model achieved 96.84% classification accuracy for synthetic EEG data representing relaxation states and optimal accuracy for concentration states when classified using a fusion of convolutional neural networks (CNNs). A 50% combination of synthetic and real EEG data yielded the highest accuracy of 98.48%. For EEG signals, the real dataset consisted of 60-s recordings across four channels (TP9, AF7, AF8, and TP10) from four individuals, providing approximately 15,000 data points per subject per state. For ECG signals, the dataset contained 1200 real samples, each comprising 140 data points, representing normal and abnormal states. WGAN-GP outperformed a basic generative adversarial network (GAN) in generating reliable synthetic data. For ECG data, a support vector machine (SVM) classifier achieved an accuracy of 98% with real data and 95.8% with synthetic data. Synthetic ECG data improved the random forest (RF) classifier’s accuracy from 97% with real data alone to 98.40% when combined with synthetic data. Statistical significance was assessed using the Wilcoxon signed-rank test, demonstrating the robustness of the WGAN-GP model. Techniques such as discrete wavelet transform, downsampling, and upsampling were employed to enhance data quality. This method shows significant potential in addressing biophysical data scarcity and advancing applications in assistive technologies, human-robot interaction, and mental health monitoring, among other medical applications.

As a promising component for body sensor networks, the wearable sensors for hand gesture recognition have increasingly received great attention in recent years. By interpreting human intentions through hand gestures, the natural human–robot interaction can be realized in the smart home where the youth and the elderly can perform hand gestures to control the household robot or the robotic wheelchair. Here, a wearable wrist-worn camera sensor was shown to recognize hand trajectory gestures. The moving velocity of the user’s hand was deduced from the matched speeded up robust features keypoints of the moving background of the video sequence. Furthermore, the segmentation of continuous gestures was achieved by detecting the predefined gesture starting signal from the hand region of the image, which was segmented by the lazy snapping algorithm. In this paper, 10 types of gestures and 1350 gesture samples collected from 15 subjects at three different scenes were classified by the dynamic time warping algorithm and the results achieved an average recognition accuracy up to 97.6%. Moreover, the practicability of the proposed system was further demonstrated by controlling a cooperative robot to draw letters on paper.

International audience

This paper presents a series of 4 single subject experiments aimed to investigate whether children with autism show more social engagement when interacting with the Nao robot, compared to a human partner in a motor imitation task. The Nao robot imitates gross arm movements of the child in real-time. Different behavioral criteria (i.e. eye gaze, gaze shifting, free initiations and prompted initiations of arm movements, and smile/laughter) were analyzed based on the video data of the interaction. The results are mixed and suggest a high variability in reactions to the Nao robot. The results are as follows: For Child2 and Child3, the results indicate no effect of the Nao robot in any of the target variables. Child1 and Child4 showed more eye gaze and smile/laughter in the interaction with the Nao robot compared to the human partner and Child1 showed a higher frequency of motor initiations in the interaction with the Nao robot compared to the baselines, but not with respect to the human-interaction. The robot proved to be a better facilitator of shared attention only for Child1. Keywords: human-robot interaction; assistive robotics; autism

International audience

This paper provides a comprehensive review of the research status of shoulder joint exoskeleton robots, focusing on analyzing their mechanism design and compliant control methods, and proposing improvement suggestions for existing shortcomings. Currently, shoulder joint exoskeletons primarily adopt three active degrees of freedom (DoFs) to mimic the motion characteristics of the human shoulder ball-and-socket joint, achieving rehabilitation or assistive functions through motor-driven or pneumatic-driven systems. Compliant control systems mainly include position-based impedance control and admittance control to ensure safe and natural human-robot interaction. However, these robots still have numerous shortcomings. Therefore, identifying current limitations and proposing optimization measures for future development by integrating cutting-edge technologies is crucial. This paper addresses the limitations in portability and non-visualized prediction, proposing optimized structural design and mechanical layout to enhance portability, as well as deep integration of multi-system compliant control with intelligent neural algorithms and the introduction of neural network or biomechanics-based predictive algorithms to improve predictability and human-robot interaction performance.

In the context of industry 5.0, human-robot collaboration (HRC) in assembly, a flexible production mode, has been paid increasing attention. In the scenario of HRC in assembly, to perfect the efficiency and fluency of the whole assembly process, the leading point is to develop a more natural human-robot interaction (HRI). In that way, the robot has the access to predict the human's intention earlier. The single process's intention has been mainly focused on human intention prediction, however, is verified against the natural HRI, causing the robot insensitive to turn-taking among the successive process. Therefore, this paper enters a proposal that we can realize early prediction of turn-taking in HRC assembly tasks based on Izhikevich neuron model-based spiking neuron network (SNN). The proposal is also verified in a developed HRC gear assembly scenario. The results express that our method can greatly advance the recognition time of human-robot turn-taking, which improves the efficiency of human-robot collaborative assembly.

Natural Human-Robot Interaction (HRI) is one of the key components for service robots to be able to work in human-centric environments. In such dynamic environments, the robot needs to understand the intention of the user to accomplish a task successfully. Towards addressing this point, we propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. Unlike most grounding methods that tackle the challenge using pre-trained object detectors via a two-stepped process, we develop a single stage zero-shot model that is able to provide predictions in unseen data. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets. Experimental results showed that the proposed model performs well in terms of accuracy and speed, while showcasing robustness to variation in the natural language input.

PurposeThis paper aims to provide a comprehensive bibliometric analysis of AI research focusing on trust, credibility, and related issues in automated systems across diverse fields. This study offers a brief systematic literature review by identifying key themes and trends within the literature, emphasizing the critical role of trust in AI systems such as autonomous robotics, software engineering, and human-agent interactions.Design/methodology/approachUsing a bibliometric approach, data were collected from Scopus spanning the years 1987–2024. The study systematically analyzes publication types, collaborative patterns, subject areas, and citation impact. It also identifies key thematic areas related to trust and credibility in AI applications, such as mobile ad hoc networks (MANETs), peer-to-peer networks, and decision-making in uncertainty.FindingsA total of 111 papers were published between 1987 and 2024, with an average of 2.92 publications annually, 60.36% of which appeared in journals. Collaboration has significantly increased, with an average of 3.48 authors per paper. The period from 2020 to 2024 witnessed a surge in both publications (41 in 2024) and authorship (139 authors). Key contributors, such as Yang from the University of Michigan and high-impact authors from Purdue University, highlight the global scope of AI research. The systematic review identifies central themes such as “trust dynamics,” “credibility assessment,” and “human-robot interaction” as crucial areas within the literature.Research limitations/implicationsThe study emphasizes the need for better visibility for conference proceedings and emerging research topics, such as reinforcement learning. The analysis also reveals the growing significance of trust and credibility in AI-driven systems, especially as AI becomes more integrated into decision-making processes, providing a roadmap for future research.Practical implicationsPublishing in high-impact journals such as the Journal of Management Information Systems significantly enhances research visibility, while other journals may require strategies to improve their citation potential. As AI applications continue to expand, themes like trust and credibility assessment are essential for fostering effective human-AI collaboration across interdisciplinary fields.Originality/valueThis study delivers a unique combination of bibliometric analysis and a systematic literature review, shedding light on key research trends in AI, particularly in the context of trust and credibility. It provides valuable insights into collaborative research patterns, institutional contributions, and the evolution of trust-related themes, positioning it as a key reference for future exploration of AI and trust dynamics.

Homophily, a person’s bias for having ties with people who are similar to themselves in social ways, has a vital role in creating a social connection between people. Studying homophily in human-robot interactions can provide valuable insights for improving those interactions. In this paper, we investigate whether similar interests have a positive effect on a human-robot interaction similar to the positive impact it can have on human-human interaction. We explore whether sharing similar interests can affect trust. This experiment consisted of two NAO robots; each gave differing speeches. For each participant, their national origin was asked in the pre-questionnaire, and during the sessions, one of the robot’s topics was either personalized or not to their national origin. Since one robot shared a familiar topic, we expected to observe bonding between humans and the robot. We gathered data from a post-questionnaire and analyzed them. The results summarize the hypotheses here. We conclude that homophily plays a significant role in human-robot interaction, affecting trust in a robot partner.

The application of intelligent systems in humanoid robots provides research and development alternatives, as is the case with human-robot interaction. This paper focuses on the design and implementation of an intelligent system in the NAO robot to plan and execute moves in the board game known as dominoes. This system uses the NAO robot’s vision to determine the distribution of tiles on the board, as well as those available in hand. The appropriate moves are planned using a computational intelligence technique, and a kinematics model executes them. The results show that the vision system has an average error of 5.62%, in addition to 3.37% for the decision-making system and 7.87% for the kinematics of the robot. This leads to the NAO robot being capable of making successful plays through the proposed system, with an average effectiveness of 83.15%.

The ability of robots to interact in a socially intelligent manner with humans is the core of human-robot interaction (HRI). The quality of this interaction is typically measured in terms of how it is engaging to the users either reflected in duration of time users spend interacting with a robot, or their self-reports on engagement during the interaction. In contrast to existing studies that analyze the influence of robots' ability to mimic affective states (happy or sad) of users on their engagement, in this paper we study the influence of sentiment apprehension by robots (i.e., robot's ability to reason about the user's attitudes such as judgment / liking) on the user engagement. Specifically, we present the findings from our pilot study on the effect of sentiment apprehension in HRI using NAO robot. In this study, we analyzed two versions of mimicry game: in the first, NAO was solely mimicking facial expressions of the users, while in the second he was also providing a feedback based on the sentiment apprehension. A total of 32 participants (7 female, 25 male) were recruited for this experiment, and the results show that the participants in the second group spent more time interacting with the robot and played more rounds of the mimicry game. After experiencing both versions of the game, ratings given by the participants indicate (with 99% confidence) that the game with sentiment apprehension is more engaging than the baseline version.

Human-computer interaction for virtual-real fusion