DeepETPicker: Fast and accurate 3D particle picking for cryo-electron tomography using weakly supervised deep learning

BackgroundCryo-electron tomography is an important tool to study structures of macromolecular complexes in close to native states. A whole cell cryo electron tomogram contains structural information of all its macromolecular complexes. However, extracting this information remains challenging, and relies on sophisticated image processing, in particular for template-free particle extraction, classification and averaging. To develop these methods it is crucial to realistically simulate tomograms of crowded cellular environments, which can then serve as ground truth models for assessing and optimizing methods for detection of complexes in cell tomograms.ResultsWe present a framework to generate crowded mixtures of macromolecular complexes for realistically simulating cryo electron tomograms including noise and image distortions due to the missing-wedge effects. Simulated tomograms are then used for assessing the template-free Difference-of-Gaussian (DoG) particle-picking method to detect complexes of different shapes and sizes under various crowding and noise levels. We identified DoG parameter settings that maximize precision and recall for detecting particles over a wide range of sizes and shapes. We observed that medium sized DoG scaling factors showed the overall best performance. To further improve performance, we propose a combination strategy for integrating results from multiple parameter settings. With increasing macromolecular crowding levels, the precision of particle picking remained relatively high, while the recall was dramatically reduced, which limits the detection of sufficient copy numbers of complexes in a crowded environment. Over a wide range of increasing noise levels, the DoG particle picking performance remained stable, but dramatically reduced beyond a specific noise threshold.ConclusionsAutomatic and reference-free particle picking is an important first step in a visual proteomics analysis of cell tomograms. However, cell cytoplasm is highly crowded, which makes particle detection challenging. It is therefore important to test particle-picking methods in a realistic crowded setting. Here, we present a framework for simulating tomograms of cellular environments at high crowding levels and assess the DoG particle picking method. We determined optimal parameter settings to maximize the performance of the DoG particle-picking method.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-016-1283-3) contains supplementary material, which is available to authorized users.

Hierarchical detection and analysis of macromolecular complexes in cryo-electron tomograms using Pyto software.

The resolution of subtomogram averages calculated from cryo-electron tomograms (cryo-ET) of crowded cellular environments is often limited due to signal-loss in, and misalignment of the subtomograms. In contrast, single-particle cryo-electron microscopy (SP-cryo-EM) routinely reaches near-atomic resolution of isolated complexes. We have developed a method called “TomographY-Guided 3D REconstruction of Subcellular Structures” (TYGRESS) that is a hybrid of cryo-ET and SP-cryo-EM, and is able to achieve close-to-nanometer resolution of complexes inside crowded cellular environments. TYGRESS combines the advantages of SP-cryo-EM (images with good signal-to-noise ratio/contrast and minimal radiation damage) and subtomogram averaging (3D-alignment of macromolecules in a complex sample). Using TYGRESS, we determined the structure of the intact ciliary axoneme with up to 12 Å resolution. These results reveal many structural details that were not visible by cryo-ET. TYGRESS is generally applicable to cellular complexes that are amenable to subtomogram averaging, bringing us a step closer to (pseudo-)atomic models of cells.

Generating Chromosome Geometries at the Single-Cell Level from Cryo-Electron Tomograms

Conical Fourier shell correlation applied to electron tomograms.

Cryogenic electron tomography (cryo-ET) visualizes the 3D spatial distribution of macromolecules at nanometer resolution inside native cells. However, automated identification of macromolecules inside cellular tomograms is challenged by noise and reconstruction artifacts, as well as the presence of many molecular species in the crowded volumes. Here, we present DeepFinder, a computational procedure that uses artificial neural networks to simultaneously localize multiple classes of macromolecules. Once trained, the inference stage of DeepFinder is faster than template matching and performs better than other competitive deep learning methods at identifying macromolecules of various sizes in both synthetic and experimental datasets. On cellular cryo-ET data, DeepFinder localized membrane-bound and cytosolic ribosomes (roughly 3.2 MDa), ribulose 1,5-bisphosphate carboxylase-oxygenase (roughly 560 kDa soluble complex) and photosystem II (roughly 550 kDa membrane complex) with an accuracy comparable to expert-supervised ground truth annotations. DeepFinder is therefore a promising algorithm for the semiautomated analysis of a wide range of molecular targets in cellular tomograms.

Electron cryotomography (cryo-ET) is an imaging technique uniquely suited to the study of bacterial ultrastructure and cell biology. Recent years have seen a surge in structural and cell biology research on bacteria using cryo-ET. This research has driven major technical developments in the field, with applications emerging to address a wide range of biological questions. In this review, we explore the diversity of cryo-ET approaches used for structural and cellular microbiology, with a focus on in situ localization and structure determination of macromolecules. The first section describes strategies employed to locate target macromolecules within large cellular volumes. Next, we explore methods to study thick specimens by sample thinning. Finally, we review examples of macromolecular structure determination in a cellular context using cryo-ET. The examples outlined serve as powerful demonstrations of how the cellular location, structure, and function of any bacterial macromolecule of interest can be investigated using cryo-ET.

Quantifying resolution limiting factors in subtomogram averaged cryo-electron tomography using simulations.

Deep learning advanced to one of the most important technologies in almost all medical fields. Especially in areas, related to medical imaging it plays a big role. However, in interventional radiotherapy (brachytherapy) deep learning is still in an early phase. In this review, first, we investigated and scrutinised the role of deep learning in all processes of interventional radiotherapy and directly related fields. Additionally, we summarised the most recent developments. For better understanding, we provide explanations of key terms and approaches to solving common deep learning problems. To reproduce results of deep learning algorithms both source code and training data must be available. Therefore, a second focus of this work is on the analysis of the availability of open source, open data and open models. In our analysis, we were able to show that deep learning plays already a major role in some areas of interventional radiotherapy, but is still hardly present in others. Nevertheless, its impact is increasing with the years, partly self-propelled but also influenced by closely related fields. Open source, data and models are growing in number but are still scarce and unevenly distributed among different research groups. The reluctance in publishing code, data and models limits reproducibility and restricts evaluation to mono-institutional datasets. The conclusion of our analysis is that deep learning can positively change the workflow of interventional radiotherapy but there is still room for improvements when it comes to reproducible results and standardised evaluation methods.

The conventional seismic inversion approach is practical for operational work, as it only uses simple linearized algorithms and assumptions, but may be less applicable when dealing with a complex geological setting, especially in the Malay basin fields, as it may introduce non-linear noises and non-unique solutions. In the Malay basin, we also frequently struggle with a scarcity of reliable well data when performing seismic inversions. This makes finding an accurate prior model for inversion challenging and contributes to high uncertainty in properties’ estimation. Implementation of deep learning for seismic inversion has become routine and has shown increasing capability in addressing nonlinearity in inverse problems. In this work, we develop a robust approach to deep learning-based seismic inversion to predict elastic properties from seismic data. The approach incorporates synthetic well and seismic data generation from a set of rock physics knowledge called the rock physics library, which plays a significant role in dataset input for network training, validation, and testing to improve elastic properties in this field. The deep learning network architecture comprising UNET and RESNET-18 with weak supervision networks has proven to be useful to enhance computational work efficiency and prediction accuracy while handling the non-linearity of the data and the non-uniqueness of the solutions. We successfully validated the proposed method on actual field data from a clastic fluvial-dominated field in the Malay basin. Upon comparative analysis with the conventional method, both inversion results are comparable and capable of identifying the reservoir occurrence and distribution. The conventional method exposed the presence of scattered amplitude noises and prominent seismic imprints masking the reservoir. Meanwhile, the proposed method showed more stable, clearer definition and fewer noise inversion results but with a faster turn-around time and a more efficient workflow. There are substantial improvements of up to 31% in correlation accuracy achieved upon implementing the proposed method for elastic properties prediction compared to the conventional. The result implies that the proposed method can provide a good elastic properties prediction framework while addressing data limitations and sparsity issues in typical deep learning-based inversions.

Recent interest in geospatial artificial intelligence (GeoAI) has fostered a wide range of applications using artificial intelligence (AI), especially deep learning for geospatial problem solving. Major challenges, however, such as a lack of training data and ignorance of spatial principles and spatial effects in AI model design remain, significantly hindering the in-depth integration of AI with geospatial research. This article reports our work in developing a cutting-edge deep learning model that enables object detection, especially of natural features, in a weakly supervised manner. Our work has made three innovative contributions: First, we present a novel method of object detection using only weak labels. This is achieved by developing a spatially explicit model according to Tobler’s first law of geography to enable weakly supervised object detection. Second, we integrate the idea of an attention map into the deep learning–based object detection pipeline and develop a multistage training strategy to further boost detection performance. Third, we have successfully applied this model for the automated detection of Mars impact craters, the inspection of which often involved tremendous manual work prior to our solution. Our model is generalizable for detecting both natural and man-made features on the surface of the Earth and other planets. This research has made a major contribution to the enrichment of the theoretical and methodological body of knowledge of GeoAI.

The cuneiform script provides a glimpse into our ancient history. However, reading age-old clay tablets is time-consuming and requires years of training. To simplify this process, we propose a deep-learning based sign detector that locates and classifies cuneiform signs in images of clay tablets. Deep learning requires large amounts of training data in the form of bounding boxes around cuneiform signs, which are not readily available and costly to obtain in the case of cuneiform script. To tackle this problem, we make use of existing transliterations, a sign-by-sign representation of the tablet content in Latin script. Since these do not provide sign localization, we propose a weakly supervised approach: We align tablet images with their corresponding transliterations to localize the transliterated signs in the tablet image, before using these localized signs in place of annotations to re-train the sign detector. A better sign detector in turn boosts the quality of the alignments. We combine these steps in an iterative process that enables training a cuneiform sign detector from transliterations only. While our method works weakly supervised, a small number of annotations further boost the performance of the cuneiform sign detector which we evaluate on a large collection of clay tablets from the Neo-Assyrian period. To enable experts to directly apply the sign detector in their study of cuneiform texts, we additionally provide a web application for the analysis of clay tablets with a trained cuneiform sign detector.

Overhead depth map measurements capture sufficient amount of information to enable human experts to track pedestrians accurately. However, fully automating this process using image analysis algorithms can be challenging. Even though hand-crafted image analysis algorithms are successful in many common cases, they fail frequently when there are complex interactions of multiple objects in the image. Many of the assumptions underpinning the hand-crafted solutions do not hold in these cases and the multitude of exceptions are hard to model precisely. Deep Learning (DL) algorithms, on the other hand, do not require hand crafted solutions and are the current state-of-the-art in object localization in images. However, they require exceeding amount of annotations to produce successful models. In the case of object localization these annotations are difficult and time consuming to produce. In this work we present an approach for developing pedestrian localization models using DL algorithms with efficient weak supervision from an expert. We circumvent the need for annotation of large corpus of data by annotating only small amount of patches and relying on synthetic data augmentation as a vehicle for injecting expert knowledge in the model training. This approach of weak supervision through expert selection of representative patches, suitable transformations and synthetic data augmentations enables us to successfully develop DL models for pedestrian localization efficiently.

Structures of macromolecules in their native state provide unique unambiguous insights into their functions. Cryo-electron tomography combined with subtomogram averaging demonstrated the power to solve such structures in situ at resolutions in the range of 3 Angstrom for some macromolecules. In order to be applicable to the structural determination of the majority of macromolecules observable in cells in limited amounts, processing of tomographic data has to be performed in a high-throughput manner. Here we present TomoBEAR—a modular configurable workflow engine for streamlined processing of cryo-electron tomographic data for subtomogram averaging. TomoBEAR combines commonly used cryo-EM packages with reasonable presets to provide a transparent (“white box”) approach for data management and processing. We demonstrate applications of TomoBEAR to two data sets of purified macromolecular targets, to an ion channel RyR1 in a membrane, and the tomograms of plasma FIB-milled lamellae and demonstrate the ability to produce high-resolution structures. TomoBEAR speeds up data processing, minimizes human interventions, and will help accelerate the adoption of in situ structural biology by cryo-ET. The source code and the documentation are freely available.

Osteochondrodysplasia, affecting 2–3% of newborns globally, is a group of bone and cartilage disorders that often result in head malformations, contributing to childhood morbidity and reduced quality of life. Current research on this disease using mouse models faces challenges since it involves accurately segmenting (precisely delineating) the developing cartilage in 3D micro-CT images of embryonic mice. Tackling this segmentation task with deep learning (DL) methods is laborious due to the big burden of manual image annotation, expensive due to the high acquisition costs of 3D micro-CT images, and difficult due to embryonic cartilage’s complex and rapidly changing shapes. While DL approaches have been proposed to automate cartilage segmentation, most such models have limited accuracy and generalizability, especially across data from different embryonic age groups. To address these limitations, we propose novel DL methods that can be adopted by any DL architectures—including Convolutional Neural Networks (CNNs), Transformers, or hybrid models—which effectively leverage age and spatial information to enhance model performance. Specifically, we propose two new mechanisms, one conditioned on discrete age categories and the other on continuous image crop locations, to enable an accurate representation of cartilage shape changes across ages and local shape details throughout the cranial region. Extensive experiments on multi-age cartilage segmentation datasets show significant and consistent performance improvements when integrating our conditional modules into popular DL segmentation architectures. On average, we achieve a 1.7% Dice score increase with minimal computational overhead and a 7.5% improvement on unseen data. These results highlight the potential of our approach for developing robust, universal models capable of handling diverse datasets with limited annotated data, a key challenge in DL-based medical image analysis.