Point cloud deformation modeling for particle selection following cryo-EM 2D classification.

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications [1,2].METHODS & RESULTS: While new approaches for deep neural networks are rapidly developing [1], the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ [3], can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments [1], purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo [5], enables to perform ML for change analysis in multitemporal point clouds [6]. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo). REFERENCES:[1] Esmorís Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.[2] Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.112772 [3] HELIOS++: https://github.com/3dgeo-heidelberg/helios[4] VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d[5] py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo[6] Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034

Learning on (3D) point clouds is vital for a broad range of emerging applications such as autonomous driving, robot perception, augmented reality, gaming, and security. Such needs have increased recently due to the prevalence of 3D sensors such as LiDAR, 3D camera, and RGB-D. Point clouds consist of thousands to millions of points; They contain rich information and are complementary to the traditional 2D cameras that we have been working on for years in the multimedia (or vision) community. 3D learning algorithms on point cloud data are new, and exciting, for numerous core problems such as 3D classification, detection, semantic segmentation, and face recognition. Covers the requirements of point cloud data, the background of capturing the data, 3D representations, emerging applications, core problems, state-of-the art learning algorithms, and future research opportunities.

Recently, the researches on 3D model analysis begin to come into focus. With the development of 3D data capturing devices, people can obtain 3D data much easier than before. At present, many existing models for 3D classification are constructed with convolution operator which can benefit from improvements on convolution. Meanwhile, attention mechanisms have demonstrated their utility across quite a few tasks including sequence learning, understanding in images. Some lightweight and general modules based on the attention mechanism are designed to improve the performance of network based on CNN architectures. Inspired by the research on image representations based on the attention mechanism, this paper applies the mechanism to address the problem of 3D shape classification. In this work, a channel attention module named 3DSE block is used to improve the performance of some representative models for 3D point cloud classification. The dependency relationship between the different channels is explicitly modeled to obtain the channel descriptor of the feature map, and it is used to refine the original feature map, so as to sift through the information from the feature and enhance the ability to feature extraction. This paper uses 3DSE module to improve the performance of the existing typical 3D classification networks. Then some experiments are performed on popular shape classification benchmark to test our ideas. The result demonstrates the effectiveness of our method.

Tobacco mosaic virus (TMV) is a helical particle, the structure of which has been determined by X-ray diffraction. We chose it as a test specimen to investigate whether cryo-electron microscopy and computer image processing could reveal internal structure at high resolution in a non-crystalline specimen.Low dose images of TMV embedded in a thin layer of vitreous ice were recorded at 60,000 magnification in a JEOL100CX electron microscope equipped with a cold stacje at -148°C. The best images were selected for processing on the basis of their optical diffractograms. The defocus was determined from the image of the carbon film adjacent to the hole over which TMV particles were suspended. The defocus values of processed images ranged from 5000 to -10000 Å. The initial processing steps included masking out the selected segment of a particle, interpolating the image to make the axis of a selected particle segment parallel to the transform array, straightening the particle segment by a real space correlation and re-interpolation procedure. The computed structure factors from the straightened particle segment were corrected for the effects of defocus by applying a Wiener filter function.

In recent years, there has been significant growth in the ubiquity and popularity of three-dimensional (3D) point clouds, with an increasing focus on the classification of 3D point clouds. To extract richer features from point clouds, many researchers have turned their attention to various point set regions and channels within irregular point clouds. However, this approach has limited capability in attending to crucial regions of interest in 3D point clouds and may overlook valuable information from neighboring features during feature aggregation. Therefore, this paper proposes a novel 3D point cloud classification method based on global attention and adaptive graph convolution (Att-AdaptNet). The method consists of two main branches: the first branch computes attention masks for each point, while the second branch employs adaptive graph convolution to extract global features from the point set. It dynamically learns features based on point interactions, generating adaptive kernels to effectively and precisely capture diverse relationships among points from different semantic parts. Experimental results demonstrate that the proposed model achieves 93.8% in overall accuracy and 90.8% in average accuracy on the ModeNet40 dataset.

This article presents a new series of invariant moments, called Fractional-order Generalized Laguerre Moment Invariants (FGLMI), based on Fractional-order Generalized Laguerre polynomials (FGLPs). To begin, we provide the relations and the properties necessary to define the fractional-order generalized Laguerre moments. Then, we present the theoretical framework to derive invariants from fractional-order moments with respect to the change in orientation, size and position based on the algebraic relationships between FGLM and fractional-order geometric moments. In addition, a fast and precise algorithm has been proposed for the calculation of FGLM in order to speed up the calculation time and ensure the numerical stability of the invariant moments. Numerical experiments are carried out to demonstrate the efficiency of FGLM and their proposed invariants compared to existing methods, with regard to the reconstruction of 2D and 3D images, the computation time, the global entity extraction capacity and image localization, invariability property and 2D / 3D image classification performance on different 2D and 3D image databases. The theoretical and experimental results presented clearly show the efficiency of the descriptors proposed for the representation and classification of 2D and 3D images by other types of orthogonal moments.

Various classifications of hemifacial microsomia (HFM) have been described previously. Although some of these classifications are used widely, others use external outlines of reference organs, even in three-dimensional (3D) images. The purpose of this study was to investigate the 3D characteristics of the mandibular condyle in HFM and to update the Pruzansky and Kaban classification as a 3D classification. Fifty-three patients with HFM were classified according to the Pruzansky and Kaban classification (type I, IIA, IIB, and III) using computed tomographic scan images. 3D images of the mandible were isolated, and the 3D characteristics were observed; furthermore, the angle of inclination of the mandibular condyle was measured in 3D. Subtypes of retroflexed mandibular condyle in 3D were observed in the Pruzansky and Kaban classification type IIA and IIB, termed as type IIAβ (33.3% in type IIA) and type IIBβ (100% in type IIB). Although some differences were observed in the inclination of the mandibular condyle between the control and the affected sides in type I and IIAα, multiple differences were observed in type IIAβ and IIBβ. To the best of our knowledge, this is the first report that identified the retroflexed mandibular condyle as subtypes type IIAβ and IIBβ. Notably, this could not be identified in the two-dimensional images (lateral cephalogram) yet. We proposed to update the Pruzansky and Kaban classification as a 3D classification with a new 3D subtype. The angle of the retroflexed mandibular condyle may predict mandibular growth in HFM.

Laser imaging offers potential for observation, for 3D terrain-mapping and classification as well as for target identification, including behind vegetation, camouflage or glass windows, at day and night, and under all-weather conditions. First generation systems deliver 3D point clouds. The threshold detection is largely affected by the local opto-geometric characteristics of the objects, leading to inaccuracies in the distances measured, and by partial occultation, leading to multiple echos. Second generation systems circumvent these limitations by recording the temporal waveforms received by the system, so that data processing can improve the telemetry and the point cloud better match the reality. Future algorithms may exploit the full potential of the 4D full-waveform data. Hence, being able to simulate point-cloud (3D) and full-waveform (4D) laser imaging is key. We have developped a numerical model for predicting the output data of 3D or 4D laser imagers. The model does account for the temporal and transverse characteristics of the laser pulse (i.e. of the "laser bullet") emitted by the system, its propagation through turbulent and scattering atmosphere, its interaction with the objects present in the field of view, and the characteristics of the optoelectronic reception path of the system.

Laser three-dimensional (3D) image is a novel non-cooperative target 3D image acquisition technology, and the improvements in detection capability and imaging accuracy of the system are critically dependent on efficient echo-signal processing technique and 3D reconstruction method. The registration process is an essential step in array 3D imaging laser point cloud data processing. Registration of point clouds is an effective method that solves the problem caused by the target self-occlusion in the laser 3D imaging system. The accurate registration result will help provide better support for subsequent applications, such as object reconstruction and target recognition. In this study, a set of thresholds in the iterative closest point (ICP) algorithm is analysed on the basis of the characteristics of the laser array 3D imaging system and is combined with the range error and visual lateral resolution of the system, which are both important parameters in the imaging system. To improve the accuracy and speed of registration, the stop threshold of the iterative algorithm and the corresponding point-distance threshold in the algorithm are established in a novel way based on the range error and visual lateral resolution of the system. This forms the foundation, based on which an adaptive threshold ICP algorithm is proposed. The principal idea of the algorithm is to improve the threshold set that has a considerable effect on the accuracy and speed of registration. At first, the characteristics of the imaging point clouds of the laser array 3D imaging system are analysed in the algorithm. Based on this analysis, the distance between the two point clouds and corresponding points with ideal registrations are estimated theoretically, according to the range error and visual lateral resolution of the system. The simulation results show that the theoretically estimated results and actual results have the same variation tendency, thus providing a theoretical basis for subsequent improvements. Next, the estimated results are added according to the iterative closest point algorithm. This implies that the registration thresholds are capable of changing and adapting under different iterations and imaging systems, thus improving the speed and accuracy of registrations. This phenomenon is not seen in other algorithms. Experiments involving laser array imaging of a point cloud and laser scanning of depth imaging data show that the algorithm is practical and effective for both imaging types of point clouds and can improve the speed and accuracy of registration notably. The effectiveness and feasibility of the proposed algorithm are thus verified. In addition, for its full consideration of the imaging system, the basic idea of the proposed algorithm can be used for designing future applications as required.

The cryo-EM structure of ETB-Gi complex, obtained using the Fusion-G system, provides insights into the distinct activation mechanism and G-protein promiscuity of ETB, aiding in the design of ETB agonists for therapeutic applications.

Cryogenic transmission electron microscopy (cryo-TEM) combined with single particle analysis (SPA) is an emerging imaging approach for soft materials. However, the accuracy of SPA-reconstructed nanostructures, particularly those formed by synthetic polymers, remains uncertain due to potential packing heterogeneity of the nanostructures. In this study, the combination of molecular dynamics (MD) simulations and image simulations is utilized to validate the accuracy of cryo-TEM 3D reconstructions of self-assembled polypeptoid fibril nanostructures. Using CryoSPARC software, image simulations, 2D classifications, ab initio reconstructions, and homogenous refinements are performed. By comparing the results with atomic models, the recovery of molecular details is assessed, heterogeneous structures are identified, and the influence of extraction location on the reconstructions is evaluated. These findings confirm the fidelity of single particle analysis in accurately resolving complex structural characteristics and heterogeneous structures, exhibiting its potential as a valuable tool for detailed structural analysis of synthetic polymers and soft materials.

In a brief period, the rapid increase in interest in Computer Vision and Deep Learning has led to a plethora of different applications on text, images, and videos. Those applications range from simple problems such as “motion detection in static cameras” or “Spam Filtering” to more complex ones such as “Robot Object Grasping through Vision” or “Image Caption Generation.” With the expeditious development of Computer Graphics, and 3D models acquisition technologies, Deep Learning applications on 3D Object Models have attracted increased attention. Sensors can provide us with 3D data with rich geometry, shape, and scale information, and accompanied by 2D images, can grant us with a better understanding of a certain environment. Three-Dimensional Data can be represented in a variety of ways, one of the most commonly used being point clouds (PC), which preserve the primary geometric information in 3D space without any discretization. However, even Deep Learning on Point Clouds is still in its preliminary stages due to challenges encountered when processing PCs with Deep Neural Networks. One of the main challenges is that decent quality PC Models are hard to obtain and often miss parts of data. A way to solve this problem is to create a 3D Shape Completion Model capable of restoring those missing parts. In this dissertation, we will research a family of models, with a variety of applications in Computer Vision, called Deep Generative Models. Specifically, we will expand indepth on Generative Adversarial Networks (GANs) as well as Variational Autoencoders (VAE). GANs create new data based on a set of given training data, with which they have the same statistics, whereas VAEs ensure that their encodings distribution is regularized during the training so that the latent space is of sufficient quality, to generate new data. The main objective is to create a point-based shape completion network based on Deep Generative Models that can complete a partial scheme with reasonable results.

In computer-vision-based human computer interaction (HCI), higher-quality signal leads to better system performance. In this paper, we develop a real-time high-resolution 3D object scanning system based on structured light illumination (SLI). Our system fuses depth information with RGB texture to reconstruct high-resolution 3D point cloud. The point cloud preserves accurate surface geometry of the object (e.g., finger postures of hands, facial expressions, etc). Respectively, for a 640 × 480 video stream, our system can generate phase and texture video at 1500 frames per second (fps) and produce full 3D point clouds at 300 fps. For gesture recognition, we propose to combine the module of robust face recognition with the module of 3D point cloud classification. Moreover, rather than extracting sophisticated features, we leverage the accurate reconstruction and classify each point cloud by directly matching the whole 3D surface geometry with the templates of different classes. The proposed recognition system is robust to the scaling, translation, rotation and texture of objects. Finally, utilizing the system, we contribute to the research community two large-scale high-resolution 3D point cloud databases, i.e., SLI 3D Hand Gesture Database and SLI 3D Face Database. The proposed point cloud recognition approach achieves recognition rates up to 98.0% over the gesture database and 88.2% over the face database in our pilot study.

3D LiDAR point clouds, obtained through scanning by LiDAR devices, contain rich information such as 3D coordinates (X, Y, Z), color, classification values, intensity values, and time. However, the original collected 3D LiDAR point clouds often exhibit significant disparities in instance counts, which can hinder the effectiveness of point cloud segmentation. PolarMix, a data augmentation algorithm for 3D LiDAR point cloud datasets, addresses this issue by rotating and pasting selected class instances around the Z axis multiple times to enrich the distribution of the point cloud. However, PolarMix does not adequately consider the substantial variations in instance counts within the original point clouds, leading to an imbalance in the dataset. To address this limitation, we propose a modified version of PolarMix's instance-level rotation and pasting method that dynamically adjusts the number of rotations and pastes based on the proportion of each instance's point cloud count relative to the total. This adaptive class-balancing approach ensures a more balanced distribution of instances across the entire dataset. We term our new algorithm Dynamic Adaptive Class-Balanced PolarMix (DACB-PolarMix). Experimental results demonstrate the effectiveness of DACB-PolarMix in balancing class distribution and enhancing model performance. The results on the SemanticKitti dataset are particularly significant. Under the MinkNet model, our method improved the mIoU from 65% to 67.9%, and under the SPVCNN model, our method increased the mIoU from 66.2% to 67.5%.

We propose a novel deep architecture for semantic labeling of 3D point clouds referred to as Global and Local Streams Network (GLSNet) which is designed to capture both global and local structures and contextual information for large scale 3D point cloud classification. Our GLSNet tackles a hard problem - large differences of object sizes in large-scale point cloud segmentation including extremely large objects like water, and small objects like buildings and trees, and we design a two-branch deep network architecture to decompose the complex problem to separate processing problems at global and local scales and then fuse their predictions. GLSNet combines the strength of Submanifold Sparse Convolutional Network [1] for learning global structure with the strength of PointNet++ [2] for incorporating local information. The first branch of GLSNet processes a full point cloud in the global stream, and it captures long range information about the geometric structure by using a U-Net structured Submanifold Sparse Convolutional Network (SSCN-U) architecture. The second branch of GLSNet processes a point cloud in the local stream, and it partitions 3D points into slices and processes one slice of the cloud at a time by using the PointNet ++ architecture. The two streams of information are fused by max pooling over their classification prediction vectors. Our results on the IEEE GRSS Data Fusion Contest Urban Semantic 3D, Track 4 (DFT4) [3] [4] [5] point cloud classification dataset have shown that GLSNet achieved performance gains of almost 4% in mIOU and 1% in overall accuracy over the individual streams on the held-back testing dataset.