Using Haptic Feedback in Human-Swarm Interaction

This study explores neural architecture mapping (NAM) in human-swarm interaction (HSI), merging neuroscience, computer science, and robotics to enhance collaboration. It discusses NAM's concepts, challenges, and applications like neuroanatomy mapping, neural interface design, and algorithm development. Despite hurdles like neural complexity, NAM promises real-time monitoring, advanced brain-machine interfaces, and seamless HSI integration. The chapter offers an overview of NAM's aims, methods, and impacts on human-robot symbiosis and swarm systems. Theoretical neuroscience, human factor studies, and swarm robotics inform NAM, integrating neural networks with robotics for AI-driven swarm behavior. Adaptive learning in swarm robotics develops autonomous algorithms, enhancing collaboration. This research aims to improve swarm robotics' efficiency across various domains.

One of the main challenges in human–swarm interactions is the construction of suitable abstractions that make an entire robot team amenable to human control. For such abstractions to be useful, they need to scale gracefully as the number of robots increases. In this work, we consider the use of time-varying density functions to externally influence a robot swarm. Density functions abstract away the size of the robot team and describe instead the concentration of agents over the domain of interest. This allows a human operator to design densities so as to manipulate the robot swarm as a whole, instead of at the individual robot level. We discuss coverage of time-varying density functions as a mechanism to translate densities into robotic movement, and provide a series of control laws that guarantee optimal coverage by the robot team. Distributed approximations allow the solutions to scale with the size of the robot team. This renders coverage a viable choice of method for influencing a robot swarm. Finally, we provide a framework for the design of density functions that shape the swarm to achieve specified geometric configurations within the domain of interest. We show through robotic implementation in two different platforms the viability of human–swarm interactions with the proposed schemes.

We adopt a tripart approach in describing the human-centred challenges with human-swarm interaction. First, the results of large-N laboratory studies will be discussed which found evidence of trust biases (e.g. system-wide trust (SWT)), uncertainties regarding appropriate control strategies and timing, and inaccuracies of perceived swarm reliability. Second, the results of a small-N subject-matter expert study will be discussed, highlighting evidence for component-specific trust (CST) and will highlight concerns from expert operators regarding: understanding appropriate inputs for swarm control, transfer of control, general transparency for swarms including the need for understanding why errors might occur and concerns regarding the effects of command latency. Third, human-centred challenges from the use of swarms in the real world will be discussed. These challenges include lessons learned regarding human-swarm interfaces, the intense challenges of logistics when using swarms in the real world, and otherwise hidden challenges of using swarms for real-world scenarios such as the Defense Advanced Research Project Agency (DARPA) OFFensive Swarm-Enabled Tactics (OFFSET) programme and the use of swarms for monitoring operations for the Air Force Marathon. This article will close with a brief discussion of the opportunity space within human-swarm interaction if the research community is able to close the gap on these human-centred challenges.This article is part of the theme issue 'The road forward with swarm systems'.

In this paper, we present a study that evaluates the effect of visual and haptic feedbacks and their relevance to human performance in a needle insertion task. A virtual needle insertion simulator with a four-layer tissue model (skin, fat, muscle, and bone) and haptic feedback is used for perceptual experiments. The task was to detect the puncture of a particular layer using haptic and visual cues provided by the simulation. The results show that the addition of force feedback reduces error in detection of transitions between tissue layers by at least 55%. The addition of real-time visual feedback (image overlay) improved user performance by at least 87% in scenarios without force feedback. Presentation of both force and visual feedback improved performance by at least 43% in comparison to scenarios without feedback. These results suggest that real-time image overlay provides greater improvement in performance than force feedback.

The haptic interface or force feedback device improves the quality of human computer interaction by accommodating the sense of touch in a computer simulation. A haptic device enables a two way communication between the user and the host computer. The applications of force feedback device cover a wide area such as in medical training, systems for the visually impaired, engineering research, and creative art-work. The role of haptic feedback varies depending on the applications in question and the devices used. This includes haptic as a teaching tool, and a communication aid. Despite the benefits of haptic feedback, its deployment to support interactions in a virtual environment is still slow. This is due to the high cost of the haptic devices in the market. We address this issue by presenting a conceptual framework in designing and implementing a cost-effective haptic device; building on similar concepts obtained from existing technologies. The device is developed using those gadgets that are mainly available in-house. It is projected that haptic feedback generated from the device could benefit teaching abstract concepts to students.

The video presents the first results of a Swiss-funded project focusing on symbiotic peer-to-peer interaction and cooperation between humans and robot swarms. As a first step, we considered human-swarm interaction, and selected the use of hand gestures to let a human communicate with a swarm of relatively simple mobile robots. In our scenario, a hand gesture encodes a command, that the swarm will execute. The robots that we used are the foot-bots, developed in the Swarmanoid project [1].

Haptic feedback is important to improve the operability of virtual objects. However, AR systems with haptic feedback are still not many. Moreover, users cannot perform tasks that need large workspaces in those systems because of physical constraints from haptic feedback devices. In this paper, we combined a wearable haptic device and a marker-less AR technology to develop an AR system in which users can perform activities that need large workspaces. Differences of the operability of virtual objects in an assembly task of large machines are examined to evaluate effects of the haptic feedback in this system. The result indicates that haptic feedback is effective and users can perform activities that need large space in this system.

Swarm robotics and cooperative AI represent a transformative approach in robotics and artificial intelligence, drawing inspiration from collective behaviors observed in natural systems such as ant colonies, bird flocks, and fish schools. By leveraging distributed control, decentralized decision-making, and local communication among multiple agents, swarm robotics enables scalable, robust, and flexible multi-agent systems capable of tackling complex tasks. Cooperative AI integrates algorithms for coordination, task allocation, and collaborative learning to optimize group performance while maintaining individual autonomy. Applications span autonomous exploration, search and rescue, environmental monitoring, logistics, agriculture, and defense. This article explores the principles, architectures, coordination strategies, learning mechanisms, and real-world applications of swarm robotics and cooperative AI. It also addresses challenges including scalability, communication constraints, fault tolerance, ethical considerations, and human-swarm interaction. Future directions involve integrating advanced AI techniques such as reinforcement learning, neuro-symbolic reasoning, and edge intelligence to enhance adaptability, resilience, and efficiency in collective robotic systems. Swarm robotics and cooperative AI promise to redefine automation, multi-agent intelligence, and collective problem-solving in dynamic and uncertain environments.

This chapter examines the use of haptic feedback in a multimedia simulation as a means of conveying information about physical science concepts. The case study presented herein investigates the effects of force feedback on children’s conceptions of gravity, mass, and related concepts following experimentation with a force-feedback-enabled simulation. Two groups of 17 children conducted experiments with the simulation; the experimental group used both visual and force feedback, and the control group used visual feedback only. Evidence of positive gains by the experimental group who used the simulation with force feedback is presented. Guidelines for applying these technologies effectively for educational purposes are discussed. This chapter adds to the limited research on the application of haptic feedback for conceptual learning and provides a basis for further research into the effects of computer-based haptic feedback on children’s cognition.

Camera drones allow exploring remote scenes that are inaccessible or inappropriate to visit in person. However, these exploration experiences are often limited due to insufficient scene information provided by front cameras, where only 2D images or videos are supplied. Combining a camera drone vision with haptic feedback would augment users' spatial understandings of the remote environment. But such designs are usually difficult for users to learn and apply, due to the complexity of the system and unfluent UAV control. In this paper, we present a new telepresence system for remote environment exploration, with a drone agent controlled by a VR mid-air panel. The drone is capable of generating real-time location and landmark details using integrated Simultaneous Location and Mapping (SLAM). The SLAMs' point cloud generations are produced using RGB input, and the results are passed to a Generative Adversarial Network (GAN) to reconstruct 3D remote scenes in real-time. The reconstructed objects are taken advantage of by haptic devices which could improve user experience through haptic rendering. Capable of providing both visual and haptic feedback, our system allows users to examine and exploit remote areas without having to be physically present. An experiment has been conducted to verify the usability of 3D reconstruction result in haptic feedback rendering.

In this paper we categorise swarming into four classes, depending on the manner in which swarm members communicate. We identify two of these classes as ready candidates for the provision of communications within tactical defence networks in which radio-frequency communications are highly contested or denied. We demonstrate the feasibility of a swarm-robotics approach to data ferrying from both of these classes using simulation, emulation, and physical swarm robotic platforms within indoor flight facilities. The results show strong alignment between data dissemination capabilities of the simulated and physical systems; we envisage these techniques providing communications between not only troops, but also other swarm robotic platforms, thereby enabling swarm robotics applications and human-swarm interaction within harsh communications environments.

Swarm robotics is an approach to the coordination of large numbers of robots that has become an increasingly popular field of research in recent years, not least because properly engineered robot swarms are scalable, flexible, and robust, making them an attractive alternative to single-robot systems in many application domains. Since its inception, the field of swarm robotics has grown beyond its roots in purely decentralised control inspired by social insect behaviour, now often utilising hybrid centralised/decentralised control architectures that incorporate human operators who guide swarm actions during tasks such as firefighting, or the localisation of radiation sources. This kind of human-swarm interaction has attracted significant interest from the research community, spawning an entire sub-field of its own that investigates how human operators, supervisors, and team-mates can interact with robot swarms and receive feedback from them. To date, human-swarm control methods such as the use of graphical user interfaces and spatial gestures have received much attention, but there has been little investigation into the potential of controlling swarm robotic systems with an operator’s voice. The few studies that have explored this idea are restricted to the use of specific predefined phrases that the human operator is required to learn, resulting in interactions that are unnatural in comparison to the way a human would normally express themselves in speech. In this paper, we present a novel architecture for conversational human-swarm interaction that addresses these issues, allowing swarm robotic systems to be engineered in such a way that a human operator can guide a swarm using spoken dialogue in a more natural manner.

Haptic feedback provides an essential sensory stimulus crucial for interaction and analyzing three-dimensional spatio-temporal phenomena on surface visualizations. Given its ability to provide enhanced spatial perception and scene maneuverability, virtual reality (VR) catalyzes haptic interactions on surface visualizations. Various interaction modes, encompassing both mid-air and on-surface interactions-with or without the application of assisting force stimuli-have been explored using haptic force feedback devices. In this paper, we evaluate the use of on-surface and assisted on-surface haptic modes of interaction compared to a no-haptic interaction mode. A force-based haptic stylus is used for all three modalities; the on-surface mode uses collision based forces, whereas the assisted on-surface mode is accompanied by an additional snapping force. We conducted a within-subjects user study involving fundamental interaction tasks performed on surface visualizations. Keeping a consistent visual design across all three modes, our study incorporates tasks that require the localization of the highest, lowest, and random points on surfaces; and tasks that focus on brushing curves on surfaces with varying complexity and occlusion levels. Our findings show that participants took almost the same time to brush curves using all the interaction modes. They could draw smoother curves using the on-surface interaction modes compared to the no-haptic mode. However, the assisted on-surface mode provided better accuracy than the on-surface mode. The on-surface mode was slower in point localization, but the accuracy depended on the visual cues and occlusions associated with the tasks. Finally, we discuss participant feedback on using haptic force feedback as a tangible input modality and share takeaways to aid the design of haptics-based tangible interactions for surface visualizations.

Interaction with computers and virtual environments is almost totally based on visual displays, along with auditory feedback. Somesthetic feedback, a necessary feature in all other skilled activity, is usually absent. Haptic interfaces attempt to solve this problem by coupling the human sense of touch with a computer-generated world. Haptics is the science of applying tactile sensation and kinesthesis to interaction with computer applications. By using special input/output devices (joysticks, data gloves, or other devices), users can receive feedback from computer applications in the form of sensations felt in the hand or other parts of the body. These sensations include forces, vibrations and/or motions. The aim of this tutorial is to provide participants with an overview of the technical aspects of haptic feedback and provide information about different haptic feedback devices and applications of these devices using virtual reality and computer simulations for motor rehabilitation, learning and teleoperation. The tutorial will include opportunities for hands-on practice with haptic devices and different environments. The haptic devices available for hands-on practice will include the Novint Falcon, and the SensAble OMNI devices.

Haptic technologies and applications have received enormous attention in the last decade. The incorporation of haptic modality into multimedia applications adds excitement and enjoyment to an application. It also adds a more natural feel to multimedia applications, that otherwise would be limited to vision and audition, by engaging as well the user's sense of touch, giving a more intrinsic feel essential for ambient intelligent applications. However, the improvement of an application's Quality of Experience (QoE) by the addition of haptic feedback is still not completely understood. The research presented in this paper focuses on the effect of haptic feedback and what it potentially adds to the experience of the user as opposed to the traditional visual and auditory feedback. In essence, it investigates certain issues regarding stylus-based haptic education applications and haptic-enhanced entertainment videos. To this end, we used two haptic applications: the haptic handwriting learning tool to experiment with force feedback haptic interaction and the tactile YouTube application for tactile haptic feedback. In both applications, our analysis shows that the addition of haptic feedback will increase the QoE in the absence of fatigue or discomfort for this category of applications. This implies that the incorporation of haptic modality (both force feedback as well as tactile feedback) has positively contributed to the overall QoE for the users.