Tracing low-level structures in cryo-electron tomography

Cryogenic electron tomography (cryo-ET) visualizes the 3D spatial distribution of macromolecules at nanometer resolution inside native cells. However, automated identification of macromolecules inside cellular tomograms is challenged by noise and reconstruction artifacts, as well as the presence of many molecular species in the crowded volumes. Here, we present DeepFinder, a computational procedure that uses artificial neural networks to simultaneously localize multiple classes of macromolecules. Once trained, the inference stage of DeepFinder is faster than template matching and performs better than other competitive deep learning methods at identifying macromolecules of various sizes in both synthetic and experimental datasets. On cellular cryo-ET data, DeepFinder localized membrane-bound and cytosolic ribosomes (roughly 3.2 MDa), ribulose 1,5-bisphosphate carboxylase-oxygenase (roughly 560 kDa soluble complex) and photosystem II (roughly 550 kDa membrane complex) with an accuracy comparable to expert-supervised ground truth annotations. DeepFinder is therefore a promising algorithm for the semiautomated analysis of a wide range of molecular targets in cellular tomograms.

As cryo-electron microscopy (cryo-EM) and cryo-electron tomography (cryo-ET) continue to advance, the ability to visualize cellular and organismal structures with unprecedented clarity is redefining the landscape of structural biology. Breakthroughs in imaging technology, sample preparation and image processing now enable the detailed elucidation of cellular architecture, macromolecular organization, and dynamic biological processes at sub-nanometer resolution. Recent methodological advances have propelled the field to new frontiers, facilitating the investigation of complex biological questions across scales-from macromolecular complexes to organism-wide structural insights. This review explores rapidly emerging trends, highlights key innovations that are pushing the boundaries of in situ structural biology, and addresses persistent challenges in expanding the applicability of cryo-EM and cryo-ET across diverse biological systems.

Road geometric information and a digital model based on light detection and ranging (LiDAR) can perform accurate geometric inventories and three-dimensional (3D) descriptions for as-built roads and infrastructures. However, unorganized point clouds and complex road scenarios would reduce the accuracy of geometric information extraction and digital modeling. There is a standardization need for information extraction and 3D model construction that integrates point cloud processing and digital modeling. This paper develops a framework from semantic segmentation to geometric information extraction and digital modeling based on LiDAR data. A semantic segmentation network is improved for the purpose of dividing the road surface and infrastructure. The road boundary and centerline are extracted by the alpha-shape and Voronoi diagram methods based on the semantic segmentation results. The road geometric information is obtained by a coordinate transformation matrix and the least square method. Subsequently, adaptive road components are constructed using Revit software. Thereafter, the road route, road entity model, and various infrastructure components are generated by the extracted geometric information through Dynamo and Revit software. Finally, a detailed digital model of the road scenario is developed. The Toronto-3D and Semantic3D datasets are utilized for analysis through training and testing. The overall accuracy (OA) of the proposed net for the two datasets is 95.3 and 95.0%, whereas the IoU of segmented road surfaces is 95.7 and 97.9%. This indicates that the proposed net could accomplish superior performance for semantic segmentation of point clouds. The mean absolute errors between the extracted and manually measured geometric information are marginal. This demonstrates the effectiveness and accuracy of the proposed extraction methods. Thus, the proposed framework could provide a reference for accurate extraction and modeling from LiDAR data.

Three-dimensional (3D) semantic segmentation based on point clouds has become a critical technology in the field of intelligent spatial perception and scene understanding. However, most existing methods cannot fully exploit the spatial feature of 3D point cloud data and the orderliness of two-dimensional (2D) image information, resulting in an inability to effectively improve segmentation accuracy. This study proposes a novel 2D image and 3D point cloud fusion method named shared multi-layer perceptron fusion semantic segmentation (SMFusionSeg) for semantic segmentation. The rich features of 2D image segmentation and spatial information of 3D point clouds are integrated by designing a fusion architecture that facilitates the transfer of 2D semantics to 3D semantics. Specifically, the fusion architecture concatenates the features from both 2D and 3D domains, which are then further integrated through a shared multi-layer perceptron (MLP) and optimized using an attention mechanism to enhance feature effectiveness and distinctiveness. Moreover, a knowledge distillation strategy is introduced to improve the learning capability and segmentation accuracy in parsing complex scenes. Experimental results on the SemanticKITTI dataset demonstrate that the proposed SMFusionSeg achieves a mean Intersection over Union ( mIoU ) of 65.4%, significantly outperforming traditional single-modal methods and many existing segmentation techniques, thereby effectively enhancing the precision and robustness of geographic feature recognition.

With the emergence of deep learning, computer vision has witnessed extensive advancement and has seen immense applications in multiple domains. Specifically, image captioning has become an attractive focal direction for most machine learning experts, which includes the prerequisite of object identification, location, and semantic understanding. In this paper, semantic segmentation and image captioning are comprehensively investigated based on traditional and state‐of‐the‐art methodologies. In this survey, we deliberate on the use of deep learning techniques on the segmentation analysis of both 2D and 3D images using a fully convolutional network and other high‐level hierarchical feature extraction methods. First, each domain’s preliminaries and concept are described, and then semantic segmentation is discussed alongside its relevant features, available datasets, and evaluation criteria. Also, the semantic information capturing of objects and their attributes is presented in relation to their annotation generation. Finally, analysis of the existing methods, their contributions, and relevance are highlighted, informing the importance of these methods and illuminating a possible research continuation for the application of semantic image segmentation and image captioning approaches.

Zernike Phase Contrast Cryo-Electron Microscopy and Tomography for Structure Determination at Nanometer and Subnanometer Resolutions

The present systematic review and meta-analysis aimed to compare the efficacy of three-dimensional (3D) imaging techniques in terms of accuracy and precision for periodontal disease assessment. A literature search was conducted across multiple databases (PubMed, Scopus, Web of Science, Google Scholar, and ScienceDirect) following Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) protocols. The primary outcomes focused on comparing the accuracy and precision of 3D versus two-dimensional (2D) imaging techniques. Furthermore, it assessed their performance in determining periodontal diseases. Quality assessment was performed using the risk of bias (RoB)-2 for randomized controlled trials (RCTs) and ROB in nonrandomized studies-Intervention (ROBINS-I) for non-RCTs. Meta-analysis was conducted using RevMan 5.4 with a significance level set at 0.01. While meta-regression was performed using OpenMEE. After screening, 22 studies met the eligibility criteria for qualitative and quantitative analysis. Qualitatively, 3D imaging, particularly cone-beam computed tomography (CBCT), showed superior accuracy and precision over 2D techniques. The meta-analysis revealed significant differences in several areas: overall (p = 0.00001, Mean Difference (MD) = -0.36, 95% confidence interval [CI]: -0.96 to 0.24, I² = 93%), horizontal measurements (p = 0.00001, MD = -0.75, 95% CI: -2 to -0.49, I² = 92%), and vertical measurements (p = 0.00001, MD = -0.59, 95% CI: -2.40 to 1.23, I² = 92%). Nonsignificant differences were found for furcation height, width, and depth. Most studies showed good quality with a low risk of bias. Age of the participants and study quality were found to be the sources of heterogeneity. Consistent trends highlight the advantages of 3D imaging in assessing both periodontal and nonperiodontal diseases. However, given the nonsignificant differences in furcation height, width, and depth, the recommended approach is to combine CBCT with digital intraoral radiography for more comprehensive periodontal bone assessment.

Cryogenic electron tomography (cryo-ET) combined with subtomogram averaging, allows in situ visualization and structure determination of macromolecular complexes at subnanometre resolution. Cryogenic focused ion beam (cryo-FIB) micromachining is used to prepare a thin lamella-shaped sample out of a frozen-hydrated cell for cryo-ET imaging, but standard cryo-FIB fabrication is blind to the precise location of the structure or proteins of interest. Fluorescence-guided focused ion beam (FIB) milling at target locations requires multiple sample transfers prone to contamination, and relocation and registration accuracy is often insufficient for 3D targeting. Here, we present in situ fluorescence microscopy-guided FIB fabrication of a frozen-hydrated lamella to address this problem: we built a coincident three-beam cryogenic correlative microscope by retrofitting a compact cryogenic microcooler, custom positioning stage, and an inverted widefield fluorescence microscope (FM) on an existing FIB scanning electron microscope. We show FM controlled targeting at every milling step in the lamella fabrication process, validated with transmission electron microscope tomogram reconstructions of the target regions. The ability to check the lamella during and after the milling process results in a higher success rate in the fabrication process and will increase the throughput of fabrication for lamellae suitable for high-resolution imaging.

Cryogenic electron tomography (cryoET) allows visualization of cellular structures in situ. However, anisotropic resolution arising from the intrinsic “missing-wedge” problem has presented major challenges in visualization and interpretation of tomograms. Here, we have developed IsoNet, a deep learning-based software package that iteratively reconstructs the missing-wedge information and increases signal-to-noise ratio, using the knowledge learned from raw tomograms. Without the need for sub-tomogram averaging, IsoNet generates tomograms with significantly reduced resolution anisotropy. Applications of IsoNet to three representative types of cryoET data demonstrate greatly improved structural interpretability: resolving lattice defects in immature HIV particles, establishing architecture of the paraflagellar rod in Eukaryotic flagella, and identifying heptagon-containing clathrin cages inside a neuronal synapse of cultured cells. Therefore, by overcoming two fundamental limitations of cryoET, IsoNet enables functional interpretation of cellular tomograms without sub-tomogram averaging. Its application to high-resolution cellular tomograms should also help identify differently oriented complexes of the same kind for sub-tomogram averaging.

With the recent growth of urban mapping and autonomous driving efforts, there has been an explosion of raw 3D data collected from terrestrial platforms with lidar scanners and color cameras. However, due to high labeling costs, ground-truth 3D semantic segmentation annotations are limited in both quantity and geographic diversity, while also being difficult to transfer across sensors. In contrast, large image collections with ground-truth semantic segmentations are readily available for diverse sets of scenes. In this paper, we investigate how to use only those labeled 2D image collections to supervise training 3D semantic segmentation models. Our approach is to train a 3D model from pseudo-labels derived from 2D semantic image segmentations using multi-view fusion. We address several novel issues with this approach, including how to select trusted pseudo-labels, how to sample 3D scenes with rare object categories, and how to decouple input features from 2D images from pseudo-labels during training. The proposed network architecture, 2D3DNet, achieves significantly better performance (+6.2-11.4 mIoU) than baselines during experiments on a new urban dataset with lidar and images captured in 20 cities across 5 continents.

With the continuous advancement of the construction of smart cities, the availability of large-scale and semantically enriched datasets is essential for enhancing the machine’s ability to understand urban scenes. Mesh data have a distinct advantage over point cloud data for large-scale scenes, as they can provide inherent geometric topology information and consume less memory space. However, existing publicly available large-scale scene mesh datasets are limited in scale and semantic richness and do not cover a wide range of urban semantic information. The development of 3D semantic segmentation algorithms depends on the availability of datasets. Moreover, existing large-scale 3D datasets lack various types of official annotation data, which hinders the widespread applicability of benchmark applications and may cause label errors during data conversion. To address these issues, we present a comprehensive urban-scale semantic segmentation benchmark dataset. It is suitable for various research pursuits on semantic segmentation methodologies. This dataset contains finely annotated point cloud and mesh data types for 3D, as well as high-resolution original 2D images with detailed 2D semantic annotations. It is constructed from a 3D reconstruction of 10,840 UVA aerial images and spans a vast area of approximately 2.85 square kilometers that covers both urban and rural scenes. The dataset is composed of 152,298,756 3D points and 289,404,088 triangles. Each 3D point, triangular mesh, and the original 2D image in the dataset are carefully labeled with one of the ten semantic categories. Six typical 3D semantic segmentation methods were compared on the CUS3D dataset, with KPConv demonstrating the highest overall performance. The mIoU is 59.72%, OA is 89.42%, and mAcc is 97.88%. Furthermore, the experimental results on the impact of color information on semantic segmentation suggest that incorporating both coordinate and color features can enhance the performance of semantic segmentation. The current limitations of the CUS3D dataset, particularly in class imbalance, will be the primary target for future dataset enhancements.

Three-dimensional (3D) imaging systems are being researched extensively for purposes of sensing and visualization in fields as diverse as defense, medical, art, and entertainment. When compared to traditional 2D imaging techniques, 3D imaging offers advantages in ranging, robustness to scene occlusion, and target recognition performance. Amongst the myriad 3D imaging techniques, 3D multiperspective imaging technologies have received recent attention due to the technologies’ relatively low cost, scalability, and passive sensing capabilities. Multiperspective 3D imagers collect 3D scene information by recording 2D intensity information from multiple perspectives, thus retaining both ray intensity and angle information. Three novel developments in 3D sensing, imaging, and visualization systems are presented: 3D imaging with axially distributed sensing, 3D optical profilometry, and occluded 3D object tracking.

Cryo-electron tomography (cryo-ET) enables the (noisy) observation of individual macromolecular complexes executing their biological functions within a near-native vitrified cellular environment of interest.Subsequent subtomogram averaging (STA) can reveal macromolecular structure at sub-nanometer resolution, though typically at the cost of coarsely discretizing or "averaging out" potentially important heterogeneity.We have previously introduced tomoDRGN, a deep learning tool to resolve ensembles of highly heterogeneous structures imaged via cryo-ET [1].Here we present updates to tomoDRGN aimed at facilitating its use and interaction with modern STA software, and illustrate the resulting insights gained from applying tomoDRGN to experimental cryo-ET datasets.

Wall segmentation is a special case of semantic segmentation, and the task is to classify each pixel into one of two classes: wall and no-wall. The segmentation model returns a mask showing where objects like windows and furniture are located, as well as walls. This article proposes the module's structure for semantic segmentation of walls in 2D images, which can effectively address the problem of wall segmentation. The proposed model achieved higher accuracy and faster execution than other solutions. An encoder-decoder architecture of the segmentation module was used. Dilated ResNet50/101 network was used as an encoder, representing ResNet50/101 network in which dilated convolutional layers replaced the last convolutional layers. The ADE20K dataset subset containing only interior images, was used for model training, while only its subset was used for model evaluation. Three different approaches to model training were analyzed in the research. On the validation dataset, the best approach based on the proposed structure with the ResNet101 network resulted in an average accuracy at the pixel level of 92.13% and an intersection over union (IoU) of 72.58%. Moreover, all proposed approaches can be applied to recognize other objects in the image to solve specific tasks.

Uncrewed aerial vehicles (UAVs) aerial remote sensing (RS) images offer higher spatial resolution compared with satellite RS images, providing more detailed scene information. The flexibility of UAV-mounted sensors allows for the simultaneous acquisition of multimodal data, including RGB images and depth maps. Depth information, serving as a supplementary cue, has been extensively utilized in semantic segmentation research. Nevertheless, the current investigations on red, green, blue-depth (RGB-D) image semantic segmentation predominantly concentrate on indoor or road scenarios. In UAV RS scenes, there exist numerous objects featuring similar visual texture manifestations. The introduction of depth geometric information can efficaciously enhance the semantic segmentation accuracy of complex ground objects. Concurrently, high-resolution RS images also present challenges to the modeling of global information in large-scale space. To address these challenges, we propose the first framework for semantic segmentation of RGB-D images in UAV RS, namely, SkyMamba. This framework effectively integrates visual texture information and geometric depth information. Simultaneously, it leverages the long-sequence modeling capability of the state-space model to efficiently capture global context information in the RS space, achieving efficient semantic segmentation of RS scenes. Furthermore, we introduce a novel benchmark dataset for semantic segmentation of UAV RS RGB-D images, named comprehensive urban-scale depth, which is divided into two versions based on coarse-grained and fine-grained semantic annotation strategies to provide diverse challenges. Experimental results demonstrate that the proposed method exhibits state-of-the-art performance on both versions of the dataset, particularly in handling imbalanced scene semantic class distributions. The code for this article is available at the following URL: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CapyLin-G/CUS-Depth</uri>