Multi-modal vision-based deformable perception for in-finger manipulation with soft active surfaces

A novel robotic gripper with soft surfaces and underactuated joints was proposed. The soft surface was fabricated from a deformable rubber bag filled with incompressible fluid and a microgripper inside the fluid. A ratchet was installed at the underactuated joint so that the joint's rotation caused by contact with an environment, such as a supporting surface, can be preserved, and the actions of scooping and enveloping an object are realized. With one actuator, the gripper realized three modes, i.e., parallel gripper, pinching, and enveloping. The range of graspable objects was wide and included soft, rigid, deformable, fragile, small (boundary length less than 30 mm), large (more than 80 mm long), thin (less than 0.5 mm), and heavy (more than 3 kg) objects.

Three-dimensional object detection is a key task in the field of autonomous driving that is aimed at identifying the position and category of objects in the scene. Due to the 3D nature of data generated by LiDAR, most models use it as input data for detection. However, the low scanning resolution of LiDAR for distant objects has inherent limitations to the method, and multimodal fusion 3D object detection methods have attracted widespread attention, mostly using both LiDAR and camera data as inputs for detection. Certainly, multimodal methods can also lead to many problems, the two main ones being the incomplete utilization of camera features and rough fusion methods. In this study, we proposed a novel multimodal 3D object detection method named VirtualFilter, which uses 3D point clouds and 2D images as inputs. In order to better utilize camera features, VirtualFilter utilizes the image semantic segmentation model to generate image semantic features and uses the semantic information to filter the virtual point cloud data during the virtual point cloud generation process to enhance the data accuracy of the virtual cloud. In addition, VirtualFilter utilizes a better RoI feature fusion strategy named 3D-DGAF (3D Distance-based Grid Attentional Fusion), which employs a attention mechanism based on distance gridding to better fuse the RoI features of the original and virtual point clouds. The experimental results on the authoritative autonomous driving dataset KITTI show that this multimodal 3D object detection method outperforms the baseline method in several evaluation metrics.

Dr Damien Vivet, of the University of Orléans, talks about the story behind their paper ‘Access control based on gait analysis and face recognition’ page 751. Dr Damien Vivet My field focuses on multi-modal perception and its interpretation for an efficient use of sensor measurements, whatever the modality (radar, vision, laser…). I worked on various applications such as localisation and mapping for robots, autonomous or assisted driving and even for traffic analysis. Even though robotics is a very interesting research field, I wanted to apply such perception technics in a different context with the objective to assist people and analyse their behaviours. Our Letter presents a human recognition application for access control, but it has also been applied in a medical context to distinguish patients in nursing homes. I think home care applications have a big future in the current context of an aging population. For example, in France, by 2035, it is estimated that 31% of people will be over 60 years old and 13.6% will be over 75. In this situation, applications to improve home care take a very important dimension. It seems natural that detection and localisation technics used to measure activities, will be essential to increase the autonomy of these populations. This research area is wide, but we try to make a contribution by proposing descriptors in order to detect and understand human behaviour. In this article, we deal with gait, but within our research team other works are in progress on human 3D pose detection, emotion analysis and even abnormal event detection. We have proposed a people identification technic based on gait analysis and face recognition, applied and evaluated in an access control context. Our system is based on a unique low-cost camera and can be easily positioned in a room to analyse the gait of the detected people. The system is able to detect people and their faces automatically and to process both static (for faces) and dynamic (for gait) descriptors. Based on previous learning about the characteristics, the system is able to identify detected people very accurately even if the face is hidden. Existing recognition systems are mainly based on face recognition. Their main problem is when the face is hidden, for example, when people are wearing a hat, or when the detected faces are of poor quality (small image size, noisy image), detection cannot be done. Our system overcomes these limitations as, in such conditions, it can identify people using only their gait, even from the back. Of course very different clothing can also cause problems for our gait descriptor. This is why we chose to use both. In my opinion, human analysis research has a bright future. The need for behaviour analysis is increasing, whether for security or access control, home care application or even for intelligent management of building energy use in response to occupant activities. Nevertheless, we should be aware of the ethical problems of such systems. For me, the main challenge to overcome is the public's acceptance of camera sensors in a housing environment or even on the streets. Respect for private life is becoming more and more important so such visual sensors could become unsuitable. Even if we certify that no image is transmitted outside the system, people are still reticent to have such technology at home. If we want this development to happen and to be integrated in future intelligent houses, new non-intrusive imaging sensors have to be developed. Part of our group is exploring scene analysis with the objective of detecting if something that happens is abnormal or not. For this purpose, we are developing new descriptors to characterise scenes. For example, in addition to face, emotion or gait descriptors, we are currently trying to extract the 3D human pose using only a single camera, in order to easily detect and recognise activities. Moreover, we don’t just consider people, but also deal with traffic monitoring to detect infractions or accidents using a camera network.

With the increasing awareness of high-quality life, there is a growing need for health monitoring devices running robust algorithms in home environment. Health monitoring technologies enable real-time analysis of users' health status, offering long-term healthcare support and reducing hospitalization time. The propose of this work is twofold, the software focuses on the analysis of gait, which is widely adopted for joint correction and assessing any lower limb, or spinal problem. On the hardware side, a novel marker-less gait analysis device using a low-cost RGB camera mounted on a mobile tele-robot is designed. As gait analysis with a single camera is much more challenging compared to previous works utilizing multi-cameras, a RGB-D camera or wearable sensors, we propose using vision-based human pose estimation approaches. More specifically, based on the out-put of state-of-the-art human pose estimation models, we devise measurements for four bespoke gait parameters: inversion/eversion, dorsiflexion/plantarflexion, ankle and foot progression angles. We thereby classify walking patterns into normal, supination, pronation and limp. We also illustrate how to run the proposed machine learning models in low-resource environments such as a single entry-level CPU. Experiments show that our single RGB camera method achieves competitive performance compared to multi-camera motion capture systems, at smaller hardware costs.

To improve the application of behavior detection technology in college education, the study proposes a new model built on deep CNN, which is used for student behavior detection and analysis in college labor education courses. The study first analyzed the target detection algorithm, and optimized the selected You Only Look Once version 5 (YOLOv5) algorithm and its network structure with a series of improvements, and based on this, embedded the attention module into the algorithm structure to finally obtain a new model, namely YOLOv5-O. After a series of experiments, YOLOv5-O reached an average accuracy of 90.1% on the test set, while the application test in the actual teaching environment showed that its average accuracy was 86.7%. This result is obviously superior to the existing technology, which proves the validity of the study and provides strong data support for the automatic detection of student behavior. In addition, in the teaching experiment, YOLOv5-O assisted teaching achieved the most significant teaching effect, and students’ achievement improved the most. The feasibility of this method is verified.

Satellite image classification is crucial in various applications such as urban planning, environmental monitoring, and land use analysis. In this study, the authors present a comparative analysis of different supervised and unsupervised learning methods for satellite image classification, focusing on a case study in Casablanca using Landsat 8 imagery. This research aims to identify the most effective machine-learning approach for accurately classifying land cover in an urban environment. The methodology used consists of the pre-processing of Landsat imagery data from Casablanca city, the authors extract relevant features and partition them into training and test sets, and then use random forest (RF), SVM (support vector machine), classification, and regression tree (CART), gradient tree boost (GTB), decision tree (DT), and minimum distance (MD) algorithms. Through a series of experiments, the authors evaluate the performance of each machine learning method in terms of accuracy, and Kappa coefficient. This work shows that random forest is the best-performing algorithm, with an accuracy of 95.42% and 0.94 Kappa coefficient. The authors discuss the factors of their performance, including data characteristics, accurate selection, and model influencing.

In this paper, a new method for vehicle discrimination and recognition on the basis of combination of multiple features based vehicle face is proposed. The color difference, vehicle face pattern difference and logo matching degree are getting together to improve the performance of vehicle discrimination. This method is assessed on a set of 200 images that belong to five distinctive vehicle manufacturers. A series of experiments are conducted, splitting the 300 pairs of images to a training set and a test set, respectively. It is shown that the enhanced feature combination approach (CMN) proposed in this paper boosts the recognition accuracy compared with the CM and CN method. The reported results indicate a high classification rate in similar or different vehicles and a fast processing time, making it suitable for real-time applications.

Random cross-validation (CV) is often used to evaluate geospatial machine learning models, particularly when a limited amount of sample data are available, and collecting an extra test set is unfeasible. However, the prediction locations can be substantially different from the available sample, leading to over-optimistic evaluation results. This has fostered the development of spatial CV methods. Yet these methods only focus on spatial autocorrelation and cannot sufficiently guarantee that the validation subset is a good proxy of the test set with significant differences. In this paper, we propose the spatial+ cross-validation (SP-CV) method. This method, which considers both the geographic and feature spaces, is composed of two stages. The first stage addresses spatial autocorrelation issues by using agglomerative hierarchical clustering to divide the available sample into blocks. The second stage deals with multiple sources of differences. It uses cluster ensembles to split the blocks into training and validation folds based on the locations of the sample data and the values of the covariates and target variable. The proposed method is compared against random and block CV methods in a series of experiments with Amazon basin above ground biomass and California houseprice datasets. Our results show that SP-CV provided the smallest error differences with respect to the reference error. This means that SP-CV produced more representative splits and led to more reliable model evaluations. It suggests that a reliable model evaluation requires to consider both the geographic and the feature spaces in a comprehensive manner.

Existing single image super-resolution (SISR) methods usually focus on Low-Resolution (LR) images which are artificially generated from High-Resolution (HR) images by a down-sampling process, but are not robust for unmatched training set and testing set. This paper proposes a GAN Flexible Lmser (GFLmser) network that bidirectionally learns the High-to-Low (H2L) process that degrades HR images to LR images and the Low-to-High (L2H) process that recovers the LR images back to HR images. The two directions share the same architecture, added with the gated skip connections from the H2L-net to the L2H-net in order to enhance information transferring for super-resolution. In comparison with several related state-of-the-art methods, experiments demonstrate that not only GFLmser is the most robust method on images of unmatched training set and testing set, but also its performance on real-world face LR images is best in PSNR and reasonably good in FID.

Visual, haptic and cross-modal recognition of objects and scenes

Emulating a human-like somatosensory system in instruments such as robotic hands and surgical grippers has the potential to revolutionize these domains. Using a combination of different sensing modalities is problematic due to the limited space and incompatibility of these sensing principles. Therefore, in contrast to the natural world, it is currently difficult to concurrently measure the force, geometry, and temperature of contact in conventional tactile sensing. To this end, here we present a soft multifunctional tactile sensing principle. The temperature is estimated using a thermochromic liquid crystal ink layer which exhibits colour variation under temperature change. The shape and force of contact is estimated through the 3D reconstruction of a deformed soft silicone surface. Our experiments have demonstrated high accuracy in all three modalities, which can be measured at the same time. The resolution of the distributed force and temperature sensing was found to be 0.7N and 0.4 °C respectively.

Virtual Screening (VS) of large compound libraries using Artificial Intelligence (AI) models is a highly effective approach for early drug discovery. Data splitting is crucial for benchmarking the performance of such AI models. Traditional random data splits often result in structurally similar molecules in both training and test sets, which conflict with the reality of VS libraries that typically contain structurally diverse compounds. To tackle this challenge, scaffold split, which groups molecules by shared core structure, and Butina clustering, which clusters molecules by chemotypes, have long been used. However, we show that these methods still introduce high similarities between training and test sets, leading to overestimated model performance. Our study examined four representative AI models across 60 NCI-60 datasets, each comprising approximately 33,000–54,000 molecules tested on different cancer cell lines. Each dataset was split in four ways: random, scaffold, Butina clustering and the more realistic Uniform Manifold Approximation and Projection (UMAP) clustering. Using Linear Regression, Random Forest, Transformer-CNN, and GEM, we trained a total of 8400 models and evaluated under four splitting methods. These comprehensive results show that UMAP split provides more challenging and realistic benchmarks for model evaluation, followed by Butina splits, then scaffold splits and closely after random splits. Consequently, we recommend using UMAP splits instead of overly optimistic Butina splits and especially scaffold splits for molecular property prediction, including VS. Lastly, we illustrate how misaligned ROC AUC is with VS goals, despite its common use. The code and datasets for reproducibility are available at https://github.com/Rong830/UMAP_split_for_VS and archived in https://zenodo.org/records/14736486. Scientific contribution This work advances the field by introducing UMAP clustering as a robust splitting method for molecular datasets, improving over traditional methods like Butina clustering and especially scaffold splits. It offers a new evaluation framework to benchmark AI models under more realistic conditions, fostering progress in molecular property prediction. The findings also show how inappropriate the use of ROC AUC for virtual screening (VS) continues to be, despite its popularity, emphasizing the need for context-specific evaluation metrics.

MENet: Map-enhanced 3D object detection in bird’s-eye view for LiDAR point clouds

We propose an efficient real-time solution for tracking rigid objects in 3D using a single camera that can handle large camera displacements, drastic aspect changes, and partial occlusions. While commercial products are already available for offline camera registration, robust online tracking remains an open issue because many real-time algorithms described in the literature still lack robustness and are prone to drift and jitter. To address these problems, we have formulated the tracking problem in terms of local bundle adjustment and have developed a method for establishing image correspondences that can equally well handle short and wide-baseline matching. We then can merge the information from preceding frames with that provided by a very limited number of keyframes created during a training stage, which results in a real-time tracker that does not jitter or drift and can deal with significant aspect changes.

In this paper, we propose a blind watermarking algorithm for 3D meshes. The proposed algorithm embeds spectral domain constraints in segmented patches. After aligning the 3D object using volumetric moments, the patches are extracted using a robust segmentation method which ensures that they have equal areas. One bit is then embedded in each patch by enforcing specific constraints in the distribution of its spectral coefficients by using Principal Component Analysis (PCA). A series of experiments and comparisons with state-of-the-art in 3D graphics watermarking have been performed; they show that the proposed scheme provides a very good robustness against both geometry and connectivity attacks, while introducing a low level of dis torsion.