Motion-Controlled Virtual Reality Robotic Arm With Kinect

This paper reports the design and development of a real-time IMU-Vision-based hybrid control algorithm to interact with a 6-DOF Kinova virtual robotic arm. Human Robot Interaction (HRI) control scheme proposed in this paper utilizes an embedded gyroscopic sensor from a Myo Gesture Control Armband's inertial measurement unit and an 800*600-pixel resolution from a Microsoft HD camera. The algorithm exploits the mathematical features of a numerical discrete time integrator and a mean filter to process the raw angular velocity data from the gyroscope. The processed data further provides the angular displacements to the end-effector of the robotic arm during clockwise or counterclockwise actions along x, y, and z axes from the user. This also facilitates the end effector (gripper) motion which was also controlled simultaneously by the roll action through threshold comparison in the algorithm. A vision based feedback system was designed using a computer vision toolbox and blob analysis technique in order to make the system more reliable and to control the distance of the end-effector while reaching to desired objects. The results demonstrated a significant control of the 6-DOF virtual robotic limb using the gyroscopic information and user inputs. The virtual robotic arm stopped the movement after reaching 320 mm from the desired object as expected. For three different objects, the maximum error between the real and the measured distance was calculated as 15.3 cm for cylindrical object. Due to its smooth control and arm gesture controller, this technology has the potential to assist people with either physical impairments or neurological disorders to perform activities of daily living using assistive robotic arm in the near future.

Industry 4.0 and digital factories leverage key technologies such as virtual reality (VR) to design, simulate, optimize, and interact with physical production systems remotely or collaboratively. Recent advancements in industrial and aerospace applications demonstrate that the integration of VR with versatile robotic control techniques can unlock new opportunities. VR offers an "extended arm" to physical environments by providing a user-friendly human-machine interface, enhancing sustainability, testability, and more in robotic control systems. This research focuses on a widely used and continuously evolving robotic arm, designed to function in an integrated system operating in both real and virtual environments. A 3-degree-of-freedom (3DOF) robotic arm was initially developed and later upgraded to a 5DOF system. The robotic arm integrated with Virtual Reality (VR) interfaces based on the Robot Operating System (ROS) and Unity3D platforms. In the real-world scenario, inverse kinematics applied for the robot's movement mechanism, while gradient descent algorithms used for the virtual model. Forward kinematics using homogeneous transformation matrices employed to verify the accuracy of the robotic arm's movements. VR headsets and controllers used to manage the virtual robotic arm's control. The goal of this study is to develop integrated systems that enable control of real-world operations via remote observation, addressing challenges like prototyping costs and safety risks. By offering an innovative approach to robotic arm control through VR integration, this work contributes to the advancement of Industry 4.0 technologies. The findings suggest potential applications across various industries, opening up new possibilities for remote operation and training in complex manufacturing environments. DOI: 10.61416/ceai.v28i1.9592

Trade-off between human health and environmental health in global diets

We have developed a hybrid model of motor control for a brain-machine interface that is based on a large-scale model of primary motor cortex (M1) based on several mammalian studies (Figure ​(Figure1).1). The M1 model consisted of 10,000 spiking Izhikevich neurons of four types: regular-firing and bursting pyramidal neurons, and fast-spiking and low-threshold-spiking interneurons. Within the M1 population, cell proportions, locations, connectivity and delays were drawn primarily from mouse experimental data. Properties were based on cell body cortical depth (distance from pia to white matter). Synapses included four different receptors: AMPA, NMDA, GABAA and GABAB. The model exhibited realistic physiological properties, including firing rates and local field potential spectra. Figure 1 Model of primary motor cortex showing inputs from PMd biological neurons, and closed-loop interaction with virtual musculoskeletal arm. Cell types are shown with different colors. The output connections of a single L5A neuron are shown to illustrate connectivity. ... Pyramidal tract-projecting neurons in layer 5B were connected to a descending spinal cord neural population, which provided excitation to the muscles of a realistic virtual musculoskeletal arm. Proprioceptive feedback from the arm was encoded in an ascending spinal cord population which then projected to M1 layer 2/3. The virtual arm movements were also followed by a robotic arm [1]. An additional population, which reproduced the spiking patterns recorded from 92 neurons of macaque dorsal premotor cortex (PMd) during a center-out reaching task, was connected to M1 layer 5A providing a modulatory input [2]. The network was trained to drive the virtual arm to reach multiple targets, by combining arm exploratory movements with reinforcement learning and spike-timing dependent plasticity (STDP). Synaptic plasticity occurred between multiple populations, including between the PMd inputs and layer 5A neurons. Tuning of learning metaparameters was achieved by employing parallel evolutionary algorithms in a high-performance computing cluster. This work moves towards a new generation of neuroprosthetic systems where biological brain circuits interact directly with biomimetic cortical models, and employ co-adaptation and learning to achieve a functional task. Such a multiscale approach, ranging from the cellular to the behavioral level, will provide deeper insights into brain dynamics and have applications for the diagnosis and restoration of brain disorders.

Real world virtual world communication and interaction will be a cornerstone of future intelligent manufacturing ecosystems. Human robotic interaction is considered to be the basic element of factories of the future. Despite the advancement of different technologies such as wearables and Augmented Reality (AR), human-robot interaction (HRI) is still extremely challenging. Whilst progress has been made in the development of different mechanisms to support HRI, there are issues with cost, naturalistic and intuitive interaction, and communication across heterogeneous systems. To mitigate these limitations, RoSTAR is proposed. RoSTAR is a novel open-source HRI system based on the Robot Operating System (ROS) and Augmented Reality. An AR Head Mounted Display (HMD) is deployed. It enables the user to interact and communicate through a ROS powered robotic arm. A model of the robot arm is imported directly into the Unity Game engine, and any interactions with this virtual robotic arm are communicated to the ROS robotic arm. This system has the potential to be used for different process tasks, such as robotic gluing, dispensing and arc welding as part of an interoperable, low cost, portable and naturalistically interactive experience.

Computer-Based Social Anxiety Regulation in Virtual Reality Exposure Therapy

Robot arm technology has revolutionized the manufacturing industry, offering worker safety and productivity benefits. However, issues like unintended actions and difficulty in intuitive operation remain. This study proposes a remote-control system for robot arms that combines augmented reality (AR) and a multi-sensor interactive smart ball (PALL0) from Ai2Ai Corp. The AR application allows operators to synchronize the control of a virtual and physical robot arm using their hand and the PALL0 held in the hand.The AR application operates the robot arm by matching the position of its head with the operator's hand and the posture of the head with the acceleration and angular velocity of the PALL0. A button on the PALL0 controls the opening and closing of the gripper, while the physical robot arm also operates. The start and end of the operation, as well as the return of the robot arm to its initial posture, can be performed using buttons in the AR application.Future research should compare the method of controlling robot arms using PALL0 with other methods, such as those that use the movement of the body's upper or lower extremities, and invest resources in building the robot's digital twin to improve system portability.

This study introduces a robotic control system that combines virtual reality integration and inertial measurement units (IMU) using mixed reality (MR) devices. The system integrates three main modules: (1) virtual reality (VR), which simulates remote reality by the Hololens2, (2) the wearable IMUs device, which captures the operator's hand movements; and (3) the robotic arm UR5, which is controlled by the user and the VR environment. The virtual and physical systems communicate via a Message Queuing Telemetry Transport (MQQT) communication architecture to establish communication between modules. To introduce a closed-loop control system for the robotic arm, model predictive control (MPC) was achieved with precise path planning to provide a flexible, intuitive, and reliable method to operate the remote-controlled manipulator. To validate the system integration and functions, two demonstrations were conducted: (a) the offline mode, where the VR module of the robotic arm was controlled by the IMUs device to check correctness, and (b) the online mode, where the control command was transferred to UR5 to complete a target mission via artificial potential field (APF) adjustment. The primary outcome of this study was the development of virtual and real industrial robotic arms to test the developed model in VR and shop floor labs.

Online instruction has become increasingly common as an alternative to face-to-face instruction (Bishop et al., 2013; Maertens et al., 2016). One benefit with online instruction is that it is more easily accessible for students who are not able to fully access the more traditional face-to-face instruction on campus. After the COVID pandemic, online instruction has gained further ground (Zhu, X., & Liu, J. 2020; Kerres & Buchner, 2022; Li, 2022). At the same time, we have seen a rapid increase in new educational technologies, including that of virtual reality (Ding, 2022; Al-Ansi, 2023; Zhang, 2022). Studies show that virtual reality can make the learning process more engaging and interactive (Jackson & Fagan, 2000; Ardiny, 2018; Roopa et al, 2021) and that it can increase reception levels and train collaborative skills (Isik-Ercan, 2010; Petersen et al, 2023). This paper raises the question of how the use of VR in online instruction affects learning experiences. While the participants in this study display a genuine enthusiasm for using VR in an online setting, results display a lack of knowledge in how to use VR to improve student learning. One area of investigation is concentration. Here, results are inconclusive as 50 % of the participants in G1 are unsure of whether VR improves concentration and 50 % of the participants in G2 claim that the use of VR improves their concentration level. Another area of investigation is understanding the topic. Here, the participants from G1 gave higher ratings than those who performed the experiment in G2, which implies that the impact was not as high as expected. In fact, the participants in G2 found that the VR equipment shifted focus from learning to other details in the visual medium. Another area is interactivity. Here, results indicate that VR technology has the didactic potential of engaging students and making them more interactive in the learning situation. The study concludes that while VR technology has the possibility of enhancing learning, a necessary prerequisite is that both students and teachers have skills and knowledge of how to use VR technology in a pedagogical setting along with few technical modifications to the device itself.
 Key words: Online learning, Virtual reality, Learning experiences

As population aging intensifies and automation technology advances, robotic massage has emerged as a significant healthcare innovation. However, participants often experience discomfort and fear when facing cold robot arms. This study proposes and validates a Mixed Reality (MR) solution integrating high-fidelity virtual massage therapists with robotic massage systems to improve participant experience. We evaluated three conditions: real robot arm, virtual robot arm, and virtual massage therapist. Results demonstrate that virtual anthropomorphic representation significantly improves participant trust, positive affect, and engagement while reducing psychological barriers. Despite challenges like increased oculomotor strain, our research confirms MR technology’s potential in enhancing psychological acceptance of robotic massage, opening new avenues for robotic massage services that may extend to other intimate human-robot interaction scenarios.

Virtual reality and augmented reality are being increasingly used in the manufacturing field for different purposes. As a result, these technologies seem to have a very promising future in the manufacturing industry. However, most of the current applications are based on the use of expensive proprietary software packages (such as specific CAD modules). This work presents a methodology that helps programmers to build virtual and augmented reality systems, which is valid for a broad number of industrial plants. Differently to other approaches, ours is based on open technologies, most of them free of cost. The technologies proposed to build the virtual worlds are already mature at the entertainment industry. The presented approach opens new possibilities for creating virtual reality models aimed at two major directions: (1) pure simulation of real processes and products for different purposes (e.g. staff training; prototype designing; manufacturing optimization; marketing) and (2) teleoperation of real processes (e.g. in dangerous environments or in micro and macro scale manufacturing). The application of the methodology is illustrated by means of the creation of a non-immersive virtual world which represents a part-classifying station, allowing both the simulation of the real plant as well as its remote operation.

Comparative studies show that urban coyotes behave differently from rural counterparts. However, these studies often homogenize cities. Cities feature diverse pressures for wildlife, such as variation in human densities and environmental health, two factors known to increase risk-taking. This heterogeneity creates a landscape of risk, which may drive locally adapted behavioral strategies within cities. Yet, the influence of these pressures on coyote behavior remains unclear. To investigate this, we conducted novel object testing at 24 sites across gradients of human density and pollution. We recorded coyote detections, group size, and behavioral responses to the novel object, focusing on time spent alert, time spent close, and total exploration. We found that coyote detections varied with human density and pollution, with markedly lower detections in areas with high human density and pollution. Coyote boldness (time spent alert and close) and exploration were uniformly associated with human density, with coyotes in high human density areas displaying elevated boldness and heightened exploration. We also found that time spent close and exploration increase with group size. In contrast, coyote risk-taking did not vary with pollution. Our results suggest that heterogeneity in human density, environmental health, and social context differentially affects coyote ecology, which may have consequences for human-carnivore coexistence.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-21946-y.

With the rapid advancements in technology, the education system and universities must align their curriculum and subject content to prepare future graduates to work with the latest technologies. Education 5.0 aligned with Industry 4.0, with the new smart technology, helps students to learn virtually (virtual reality, VR) in an augmented way, and with augmented reality (AR) on varied platforms by easily connecting and interacting with experts. This chapter provides insight into these emerging technologies that will create a virtual and interactive environment for better learning in various sectors. The AR and VR technology thus provides an illusion of the presence of physical objects and takes users to a new world of imagination. This chapter also discusses the basic concepts of AR and VR, types of AR and VR, hardware and software requirements, and applications of AR and VR technology in various areas, especially in education and industries.

Within a few decades of development, Internet is been refreshing the world’s awareness of its development and potentials at an unexpected speed. In recent years, virtual reality technology has begun to be applied to more and more fields. Especially in instant transmission of information, the applications of VR are incomparable. Akin to the wide-ranged advantages of the technology in education and health, the applications of VR in the art of artistic image designing need to be addressed. With this research work, a comprehensive study is presented about the applications of virtual reality in the art of image visualization. The research intends to analyze the possibility of intermingling modern image visualization with the emerging VR technology. Following a systematic approach, the history and development of display design are reviewed besides studying the artistry techniques used in the technology of art image visualization design. The interdisciplinary association of the two fields is explored with the help of relevant algorithms. The model proposed in the paper for the design of art image is based on the visual interactive process, rigorous assembly sequence, and appropriate equipment mode. As a result, the whole design process is more controllable and feasible. The suggested design concept is verified dynamically. Through the collection and comparison of feedback information, the hypothesis algorithm of the model is updated to accommodate the dynamic information so that to reflect the complete image design process. To assess performance of the model, evaluation was conducted from various perspectives. The three basic image operations such as illumination change, occlusion change, and color change were performed by using the two technologies such as the traditional image design and VR-based image design. From the comparison of experimental data, it was revealed that the effects of traditional art image visualization products produced by the art image visualization and that of VR technology are quite different. For instance, the visualization score achieved for the lighting change operation using the VR technology was 95.48%, whereas 85.57% was obtained for the same operation using the traditional image design technology. Similarly, while adding visual effects using the VR technology, an average of 9% higher score was obtained for the said image operations. The promising results attained for the VR-based design testify that the technology is feasible and suitable to be intermingled with the visual design of art image. As a whole, the findings justify that the proposed approach is quite applicable in visual design of art image systems. In view of the significance of the research, the study can be extended to related domains of computer-based designing and visualization as well.

In this paper, a review of the implementation of Virtual Reality (VR) with robotics and how it has changed various areas is explored. In general, VR improves efficiency and safety in operational work—especially in the energy sector, where it optimizes substation operations (e.g., increasing accuracy by up to 25% due to a higher level of automation). For mechatronics, VR allows advanced digital design, such as virtual prototyping and collaborative simulation, which significantly increase system efficiency. In education, VR technology has increased student engagement and learning outcomes by up to 40%, improving participation in subjects such as natural sciences. VR integration into robotics also enables the testing of various movements and trajectories for safety, resulting in minimal damages incurred during the design phase and increased productivity. VR also enhances human-robot interaction by enabling intuitive control based on better gesture recognition and a 35% increase in precision. In future applications, the remote operation of robots in dangerous environments will greatly improve emergency rescue efficiency and speed. Using VR for substation operations has translated to 25% better accuracy and a similar gain in terms of operational efficiency. By providing a more connected, efficient future in multiple spheres and delivering about 40% fewer accidents during simulations, the fusion of VR with robotics is an overall successful venture.