Deep Learning–Powered Robust Tactile Perception: Bridging Graphene Electronic Skin and Dynamic Decoding

As the technology moves towards more human-like bionic limbs, it is necessary to develop a feedback system that provides active touch feedback to a user of a prosthetic hand. Most of the contemporary sensory substitution methods comprise simple position and force sensors combined with few discrete stimulation units, and hence they are characterized with a limited amount of information that can be transmitted by the feedback. The present study describes a novel system for tactile feedback integrating advanced multipoint sensing (electronic skin) and stimulation (matrix electrodes). The system comprises a flexible sensing array (16 sensors) integrated on the index finger of a Michelangelo prosthetic hand mockup, embedded interface electronics and multichannel stimulator connected to a flexible matrix electrode (24 pads). The developed system conveys contact information (binary detections) to the user. To demonstrate the feasibility, the system was tested in six able-bodied subjects who were asked to recognize static patterns (contact position) with two different spatial resolutions and dynamic movement patterns (i.e., sliding along and/or across the finger) presented on the electronic skin. The experiments demonstrated that the system successfully translated the mechanical interaction into electrotactile profiles, which the subjects could recognize with good performance. The success rates (mean ± standard deviation) for the static patterns were 91 ± 4% and 58 ± 10% for low and high spatial resolution, respectively, while the success rate for sliding touch was 94 ± 4%. These results demonstrate that the developed system is an important step towards a new generation of tactile feedback interfaces that can provide high-bandwidth connection between the user and his/her bionic limb. Such systems would allow mimicking spatially distributed natural feedback, thereby facilitating the control and embodiment of the artificial device into the user body scheme.

Electronic skin, as a key technology for enabling robots and wearable devices to perceive the external environment and achieve intelligent interaction, faces the core challenge of developing multimodal and highly sensitive comprehensive tactile sensing. In this work, we propose a biomimetic, multimodal, and flexible electronic skin that provides high‐sensitivity discrimination and reliable detection of both pressure and temperature signals. Through the synergistic regulation of the conductive hydrogel and the strain‐sensitive layer, the sensor exhibits excellent performance, with a pressure sensitivity of 1063 kPa − 1 , a detection limit of 1.16 Pa, and a response time of under 30 ms; the temperature sensing range spans 0°C–80°C, offering a broad‐spectrum detection capability. Signals are processed using a discrete wavelet transform (DWT) for multiscale feature extraction, and high‐dimensional feature fusion and classification are achieved by combining support vector machines (SVM) with deep neural networks (DNN), demonstrating the feasibility and potential of the system for distinguishing samples with different surface texture characteristics, with an average recognition accuracy of 93%. The system integrates multimodal detection capabilities for pressure, temperature, and material recognition, demonstrating its broad application potential in robotic tactile perception, wearable health monitoring, and human‐machine interaction.

The rapid advancements in robotics and automation have highlighted the need for robotic systems capable of adapting to the diverse physical properties of objects. Traditional grippers often lack the versatility to handle both hard and soft objects without extensive reprogramming or hardware adjustments. Previous studies have explored various approaches to this challenge, including tactile sensors and force feedback mechanisms to distinguish object properties. For instance, research by Calandra et al. (2018) utilized deep learning with tactile data to enable robotic hands to identify objects and adjust grip accordingly. Similarly, the paper by Li et al. (2020) “Design and performance characterization of a soft robot hand with fingertip haptic feedback for teleoperation”, focuses on designing and characterizing a soft robotic hand with fingertip haptic feedback for teleoperation emphasizing real-time tactile sensing and feedback mechanisms. However, these studies primarily focus on tactile feedback or specialized hardware, limiting their applicability in scenarios where such systems are not available or practical.This study introduces a novel machine learning-based approach, focusing on the use of visual data alone to classify and adapt to the hardness or softness of objects. By leveraging the CIFAR-100 dataset, we trained a deep learning model based on the ResNet50 architecture, achieving significant results in binary classification of hard and soft objects. The CIFAR-100 dataset, consisting of 100 diverse object categories/classes, was reorganized into two classes: hard (39 categories) and soft (61 categories). The ResNet50 model, pre-trained on ImageNet, was fine-tuned specifically for this task, with modifications to the last 210 layers to enhance its adaptability.Data augmentation techniques, including rotations, translations, shearing, zooming, and horizontal flipping, were applied to simulate real-world variations, ensuring robust learning. The model was further refined with additional fully connected layers, dropout, and batch normalization to prevent overfitting. Optimized using the AdamW optimizer, the model achieved a training accuracy of 83.31% and a validation accuracy of 80.25%, with a test accuracy of 80%. The precision, recall, and F1-scores were 0.82, 0.86, and 0.84 for the soft object class, and 0.76, 0.71, and 0.73 for the hard object class, demonstrating the model’s effectiveness in distinguishing between hard and soft objects without the need for specialized sensors. It is also observed that the accuracy of the model in relation to hard objects is significantly lesser as compared to soft objects. It is understood that the CIFAR-100 dataset (comprising of 100 classes) is inadequate for model training, so we are exploring the ILSVRC (ImageNet subset) dataset for model training in future. The research is useful in a variety of fields where the focus lies on object handling. In everyday life, we encounter a wide array of objects with varying degrees of hardness, requiring different levels of care and precision during handling. By relying on visual data and machine learning algorithms, as demonstrated in this research, robotic systems can become more autonomous and versatile, reducing the burden on human operators and improving overall efficiency. This approach can lead to cost savings by reducing the need for specialized hardware, such as tactile sensors, which are often expensive and difficult to integrate.

Tactile perception represents a fundamental sensory modality through which humans engage with the external environment. In virtual and augmented reality (VR/AR) technologies, users can achieve a more immersive and verisimilar interactive experience by perceiving the surface topography of contacted objects. This study addresses the bionic requirements for tactile sensors to identify surface topography features primarily characterized by hardness and surface flatness. We present a novel tactile feature perception and classification method integrating fiber Bragg grating (FBG) tactile sensing technology with deep learning architectures. A three-point synchronous tactile sensor model is employed to collect data from objects with diverse hardness and flatness attributes. Static stress and dynamic vibration signals are extracted via wavelength-swept optical coherence tomography technology, facilitating the construction of a three-point static-dynamic tactile dataset. Based on this dataset, a gated Transformer-based deep learning network is designed to classify 20 distinct combinations of tactile features, encompassing four levels of Shore A hardness and five surface flatness variations. The proposed method yields an average classification accuracy of 93.756% with a single recognition time of approximately 1.621 ms, validating the feasibility of integrating tactile sensing with deep learning networks. This approach offers an efficient solution for biomimetic, real-time, and high-precision tactile perception, demonstrating significant potential for practical applications.

In recent years, extensive research has been conducted on the development of high-performance flexible tactile sensors, pursuing the next generation of highly intelligent electronics with diverse potential applications in self-powered wearable sensors, human-machine interactions, electronic skin, and soft robotics. Among the most promising materials that have emerged in this context are functional polymer composites (FPCs), which exhibit exceptional mechanical and electrical properties, enabling them to be excellent candidates for tactile sensors. Herein, this review provides a comprehensive overview of recent advances in FPCs-based tactile sensors, including the fundamental principle, the necessary property parameter, the unique device structure, and the fabrication process of different types of tactile sensors. Examples of FPCs are elaborated with a focus on miniaturization, self-healing, self-cleaning, integration, biodegradation, and neural control. Furthermore, the applications of FPC-based tactile sensors in tactile perception, human-machine interaction, and healthcare are further described. Finally, the existing limitations and technical challenges for FPCs-based tactile sensors are briefly discussed, offering potential avenues for the development of electronic products.

Electronic skin (e-skin), mimicking the physical-chemical and sensory properties of human skin, is promising to be applied as robotic skins and skin-attachable wearables with multisensory functionalities. To date, most e-skins are dedicated to sensory function development to mimic human skins in one or several aspects, yet advanced e-skin covering all the hyper-attributes (including both the sensory and physical-chemical properties) of human skins is seldom reported. Herein, a water-modulated biomimetic hyper-attribute-gel (Hygel) e-skin with reversible gel-solid transition is proposed, which exhibits all the desired skin-like physical-chemical properties (stretchability, self-healing, biocompatibility, biodegradability, weak acidity, antibacterial activities, flame retardance, and temperature adaptivity), sensory properties (pressure, temperature, humidity, strain, and contact), function reconfigurability, and evolvability. Then the Hygel e-skin is applied as an on-robot e-skin and skin-attached wearable to demonstrate its highly skin-like attributes in capturing multiple sensory information, reconfiguring desired functions, and excellent skin compatibility for real-time gesture recognition via deep learning. This Hygel e-skin may find more applications in advanced robotics and even skin-replaceable artificial skin.

In consumer electronics and industrial electronics today, transistors are common devices. With the rapid development of electronic skin and wearable electronics, there has been an escalating demand to extend thin film transistors as sensors to perceive multiple physiological signals and varieties of external stimuli. However, conventional single-function sensors are limited to detecting only one physical parameter, which is inadequate to meet the growing requirement for multifunctional integration in increasingly complex application scenarios. Consequently, recent efforts have been put forward to develop multimodal sensors for simultaneous detection of multiple physical quantities. To achieve full process perception of varieties of targets from proximity to contact, here in-plane interdigital piezoelectric polymer capacitors were introduced to amorphous oxide thin film transistors as extended gates to construct flexible multimodal sensors, in which the piezoelectric effect endowed the sensors with contact perception capability, and electrostatic induction realized sensitive proximity perception of a charged object, while the fringe capacitance effect further guaranteed noncontact perception of a zero potential and conductive target. The synergistic effect of both the piezoelectric effect and field effect further endowed the devices with effective perception of both static and dynamic mechanical excitation beyond the capability of a piezoelectric capacitor alone. Such devices realized proximity perception of a variety of common objects and tactile perception of pulse, breath, and bending of human joints. This piezoelectric extended gate transistor configuration was further developed as an electronic skin to control the grasping action of a mechanical hand as well as the measurement of the applied force. This work provides a feasible solution to design multimodal sensing systems for applications in electronic skin, wearable devices, and robotic perception.

The method of tactile perception can accurately reflect the contact state by collecting force and torque information, but it is not sensitive to the changes in position and posture between assembly objects. The method of visual perception is very sensitive to changes in pose and posture between assembled objects, but they cannot accurately reflect the contact state, especially since the objects are occluded from each other. The robot will perceive the environment more accurately if visual and tactile perception can be combined. Therefore, this paper proposes the alignment method of combined perception for the peg‐in‐hole assembly with self‐supervised deep reinforcement learning. The agent first observes the environment through visual sensors and then predicts the action of the alignment adjustment based on the visual feature of the contact state. Subsequently, the agent judges the contact state based on the force and torque information collected by the force/torque sensor. And the action of the alignment adjustment is selected according to the contact state and used as a visual prediction label. Whereafter, the network of visual perception performs backpropagation to correct the network weights according to the visual prediction label. Finally, the agent will have learned the alignment skill of combined perception with the increase of iterative training. The robot system is built based on CoppeliaSim for simulation training and testing. The simulation results show that the method of combined perception has higher assembly efficiency than single perception.

The World Health Organization (WHO) has stated two modes of transmission of the coronavirus disease (COVID-19) virus viz., respiratory droplets, and physical contact. Hence, there is a need to take some precautionary measures to avoid the spread of this virus such as social distancing and wearing masks. Between these two precautions, wearing a mask is considered to be the key factor for preventing the spread of the virus as these respiratory droplets can land on any surface. Therefore, keeping track of whether people are wearing masks or not is of utmost importance. Using deep learning, a computer-friendly COVID-19 face mask detector can be efficiently created that can help in identifying people without masks, thus, preventing COVID-19 from further spreading. This paper covers different techniques for detecting whether a person in the camera, picture, or video is wearing a mask or not.

This study explores the use of deep learning to predict stiffness discrimination thresholds using psychophysical experimental data. Haptic technology, integral to various fields, enables realistic simulations and interactions with virtual worlds. Traditional methods for assessing tactile sensitivity such as ability to discriminate stiffness differences are laborintensive, requiring numerous trials. This research investigates deep learning as a solution to reduce experimental demands while maintaining reliable assessments. Thirteen participants engaged in experiments using a haptic device to measure their stiffness discrimination thresholds. Data on palpation force, fingertip velocity, and Weber fractions were collected and analyzed. The study employed binary and multiclass classification, as well as regression, to predict Weber fractions using deep learning models. A comparison of the results indicated that deep learning models outperformed traditional machine learning algorithms for the same prediction task, demonstrating higher accuracy. The findings suggest that deep learning can effectively streamline the assessment of stiffness discrimination during haptic exploration, offering a promising tool for future research and applications in exploration of virtual worlds using haptics.

Electronic skin (e-skin) has attracted considerable interests to endow robots the abundant perception abilities like human skin. However, multifunctional tactile feedbacks and poor wearability hinder the practical application towards compliant robot manipulations and friendly human-robot interaction (HRI). Here, a 3D-shaped e-skin integrated with hierarchically distributed flexible pressure, proximity, humidity sensors, and digital conversion circuits has been developed via the parting casting method. It can perfectly cover the non-developable and time-dynamic curved surface of a bionic hand and shows excellent wearability. The sensing data are collected by multichannel acquisition circuits and then sent to processing center to establish the mapping relations between multiple stimuli and movements of the bionic hand and arm. The multifunctional tactile feedbacks have been successfully demonstrated, which include non-contact obstacle avoidance, grasping force control, slip detection, and voice feedback. The e-skin shows great potential in the fields of HRI, artificial limbs, health care robots, and service robots.

Robust all-fabric e-skin with high-temperature and corrosion tolerance for self-powered tactile sensing

Tactile sensors with visible light feedback functions, such as wearable displays and electronic skin and biomedical devices, are becoming increasingly important in various fields. However, existing methods cannot meet the application requirements for the tactile perception of intensity feedback and extended intersection due to their limited light-mapping performance and insufficient portability. Herein, a freely constructible self-powered visual tactile sensor is proposed, which consists of a high-output triboelectric nanogenerator (TENG) and a visual light source. The transferred charge of the TENG is enhanced to 746 nC by the structural design of the triboelectric material and device, which can easily drive the light source to generate a light signal with a brightness of 9.8 cd m-2 . Notably, the application of the TENG enables to realization visual sensing of the palm-grasp state and strength feedback without an external power supply. This visual feedback and power-free tactile sensors are expected to have potential application in the field of artificial intelligence as a new interactive medium for smart protective clothing and robotics.

The progress from intelligent interactions requires electronic skin (E‐skin) to shift from single‐functional perception to multisensory capabilities. However, the intuitive and interference‐free reading of multiple sensory signals without involving complex algorithms is a critical challenge. Herein, we propose a flexible multisensory E‐skin by developing a highly homogeneous dispersion of BaTiO3 nanoparticles in polydimethylsiloxane dielectric layer. The E‐skin is sensitive to externally applied pressure as well as temperature and can distinguish dual synergetic stimuli by the time decoupling effect. The pressure and temperature perception was achieved in an individual device, which greatly reduced the structural complexity compared with multifunctional integrated devices. The sensitivity of E‐skin for pressure detection is as high as 0.0724 kPa−1 and the detection range reaches as wide as 15.625–10 MPa. The sensitivity to temperature detection is as high as −1.34°C−1 and the detection range reaches 20–200°C. More importantly, by equipping with a multilayer neural network, the evolution from tactile perception to advanced intelligent tactile cognition is demonstrated.

Human skin plays a critical role in a person communicating with his or her environment through diverse activities such as touching or deforming an object. Various electronic skin (E‐skin) devices have been developed that show functional or geometrical superiority to human skin. However, research into stretchable E‐skin that can simultaneously distinguish materials and textures has not been established yet. Here, the first approach to achieving a stretchable multimodal device is reported, that operates on the basis of various electrical properties of piezoelectricity, triboelectricity, and piezoresistivity and that exceeds the capabilities of human tactile perception. The prepared E‐skin is composed of a wrinkle‐patterned silicon elastomer, hybrid nanomaterials of silver nanowires and zinc oxide nanowires, and a thin elastomeric dielectric layer covering the hybrid nanomaterials, where the dielectric layer exhibits high surface roughness mimicking human fingerprints. This versatile device can identify and distinguish not only mechanical stress from a single stimulus such as pressure, tensile strain, or vibration but also that from a combination of multiple stimuli. With simultaneous sensing and analysis of the integrated stimuli, the approach enables material discrimination and texture recognition for a biomimetic prosthesis when the multifunctional E‐skin is applied to a robotic hand.