TomoAlign: A novel approach to correcting sample motion and 3D CTF in CryoET

Quantifying resolution limiting factors in subtomogram averaged cryo-electron tomography using simulations.

BackgroundThe transmission electron microscope is used to acquire structural information of macromolecular complexes. However, as any other imaging device, it introduces optical aberrations that must be corrected if high-resolution structural information is to be obtained. The set of all aberrations are usually modeled in Fourier space by the so-called Contrast Transfer Function (CTF). Before correcting for the CTF, we must first estimate it from the electron micrographs. This is usually done by estimating a number of parameters specifying a theoretical model of the CTF. This estimation is performed by minimizing some error measure between the theoretical Power Spectrum Density (PSD) and the experimentally observed PSD. The high noise present in the micrographs, the possible local minima of the error function for estimating the CTF parameters, and the cross-talking between CTF parameters may cause errors in the estimated CTF parameters.ResultsIn this paper, we explore the effect of these estimation errors on the theoretical CTF. For the CTF model proposed in [1] we show which are the most sensitive CTF parameters as well as the most sensitive background parameters. Moreover, we provide a methodology to reveal the internal structure of the CTF model (which parameters influence in which parameters) and to estimate the accuracy of each model parameter. Finally, we explore the effect of the variability in the detection of the CTF for CTF phase and amplitude correction.ConclusionWe show that the estimation errors for the CTF detection methodology proposed in [1] does not show a significant deterioration of the CTF correction capabilities of subsequent algorithms. All together, the methodology described in this paper constitutes a powerful tool for the quantitative analysis of CTF models that can be applied to other models different from the one analyzed here.

Automatic CTF correction for single particles based upon multivariate statistical analysis of individual power spectra

Images taken by transmission electron microscopes are usually affected by lens aberrations and image defocus, among other factors. These distortions can be modeled in reciprocal space using the contrast transfer function (CTF). Accurate estimation and correction of the CTF is essential for restoring the high-resolution signal in cryogenic electron microscopy (cryoEM). Previously, we described the implementation of algorithms for this task in the cisTEM software package (Grant et al., 2018). Here we show that taking sample characteristics, such as thickness and tilt, into account can improve CTF estimation. This is particularly important when imaging cellular samples, where measurement of sample thickness and geometry derived from accurate modeling of the Thon ring pattern helps judging the quality of the sample. This improved CTF estimation has been implemented in CTFFIND5, a new version of the cisTEM program CTFFIND. We evaluated the accuracy of these estimates using images of tilted aquaporin crystals and eukaryotic cells thinned by focused ion beam milling. We estimate that with micrographs of sufficient quality CTFFIND5 can measure sample tilt with an accuracy of 3° and sample thickness with an accuracy of 5 nm.

We introduce IsoNet2, an end-to-end self-supervised deep-learning method that directly reconstructs high-quality 3D densities from cryogenic electron tomograms. A unified network simultaneously performs denoising, contrast transfer function correction, and missing-wedge restoration, achieving ~20 Å resolution without averaging. A feature-rich GUI enables rapid, dataset-specific fine-tuning for end-users. IsoNet2 resolves domain organization in HIV capsid proteins, tRNA occupancy in individual ribosomes, and in situ architectures of mitochondrial respiration-related complexes, enabling atomic-level interpretation of cellular environments.

Motion estimation is a vital function of video coders that use motion compensated prediction to exploit the temporal redundancy of video signals. The reference images used in the motion estimation process can be either original images (open‐loop architecture) or reconstructed images (closed‐loop architecture). While the closed‐loop architecture is intuitively superior to the open‐loop architecture, closed‐loop motion estimation is more difficult to implement in a real‐time video encoder. A compromise is to perform open‐loop integer‐pel accurate motion estimation followed by closed‐loop half‐pel accurate estimation. In this article we present a comparative study of the performance of half‐pel closed‐loop versus open‐loop motion estimation for coding of high definition television with the MPEG algorithm. Simulation results indicate that on average, closed‐loop half‐pel accurate motion estimation provides slightly better performance than open‐loop half‐pel accurate estimation. This improvement depends on the bit rate of the coded video biststream. For relatively high bit rates, the closed‐loop implementation often provides no improvement, and sometimes an actual degradation of performance. An analysis of why a closed‐loop implementation does not guarantee a better performance is presented.©1994 John Wiley & Sons Inc

Automatic Alignment of Transmission Electron Microscope Tilt Series without Fiducial Markers

The cryo-EM structure of ETB-Gi complex, obtained using the Fusion-G system, provides insights into the distinct activation mechanism and G-protein promiscuity of ETB, aiding in the design of ETB agonists for therapeutic applications.

This paper demonstrates how active contour segmentation and optical flow estimation techniques can be combined to create a seamless mosaic of frames from a video sequence. This virtual scene forms the basis of a number of post-production effects such as creating new image trajectories for the shot, or special effects which include new foreground elements. The optical flow method produces highly accurate global motion estimates by drawing upon an appropriate 3D geometrical model of camera motion. The active contour technique has been extended to include user selected edge data to guide crudely specified initial contours to their final location. The resultant rotoscoping tool dramatically improves the performance of traditional tools by using edge chain matching to relocate the active contour in subsequent frames.

Accurate image alignment is needed for computing three-dimensional reconstructions from transmission electron microscope tilt series. So far, the best alignment results have been obtained by using colloidal gold beads as fiducial markers. If their use has not been possible for some reason, the only option has been the automatic 2D cross-correlation-based registration methods. However, since the actual motion is three dimensional, this approach is inaccurate and inappropriate for the whole problem. Conversely, we propose a novel method that uses the actual 3D motion model without any fiducial markers in the images. The method is based on matching and tracking corner features by first solving the underlying geometrical constraint of consecutive images in the tilt series. The results show that our method is competitive with the gold marker alignment in the level of accuracy and hence opens the way for new opportunities in the analysis of electron tomography reconstructions.

Conventional motion estimation algorithms in video coding consider only translations, which can be readily characterized by displacement motion vectors. We might be able to achieve more accurate motion estimation by enlarging the search windows, at the expense of drastically increased computational complexity. Alternatively, we could also increase the accuracy of motion estimation by taking into account potential non-translation motions of objects in a video scene, without increasing the search window size. In this paper, we conduct a comparative study on the computational complexity of these two methods. We demonstrate that our motion estimation method based on the Lie operators can significantly increase the accuracy of motion estimation, while requiring much lower complexity than the method of enlarged search windows.

Temporal segmentation of a video sequence into different shots is fundamental to a number of video retrieval and analysis applications. Motion estimation has been wildly used in many applications of video processing, since it provides the most essential information for an image sequence. In this paper, we explore the possibility to exploit motion and illumination estimation in a video sequence to detect various types of shot changes. Optical flow is the motion vector computed at each pixel in an image sequence from intensity variation. Traditionally, optical flow computational algorithms were derived from the brightness constancy assumption. In this paper, we employ a generalized optical flow constraint that includes an illumination parameter to model local illumination changes. An iterative optical flow and illumination estimation algorithm is developed in this paper to refine the generalized optical flow constraints step by step, thus leading to a very accurate estimation of the optical flow and illumination parameters. Two robust measures are defined from the mean and standard deviation of the estimated intensity compensation values for all the blocks in the same image. Either of these two measures corresponds significantly to various types of shot changes. We show the usefulness of these two measures through experiments.

Cardiac x-ray CT imaging is still challenging due to heart motion, which cannot be ignored even with the current rotation speed of the equipment. In response, many algorithms have been developed to compensate remaining motion artifacts by estimating the motion using projection data or reconstructed images. In these algorithms, accurate motion estimation is critical to the compensated image quality. In addition, since the scan range is directly related to the radiation dose, it is preferable to minimize the scan range in motion estimation. In this paper, the authors propose a novel motion estimation and compensation algorithm using a sinogram with a rotation angle of less than 360°. The algorithm estimates the motion of the whole heart area using two opposite 3D partial angle reconstructed (PAR) images and compensates the motion in the reconstruction process. A CT system scans the thoracic area including the heart over an angular range of 180° + α + β, where α and β denote the detector fan angle and an additional partial angle, respectively. The obtained cone-beam projection data are converted into cone-parallel geometry via row-wise fan-to-parallel rebinning. Two conjugate 3D PAR images, whose center projection angles are separated by 180°, are then reconstructed with an angular range of β, which is considerably smaller than a short scan range of 180° + α. Although these images include limited view angle artifacts that disturb accurate motion estimation, they have considerably better temporal resolution than a short scan image. Hence, after preprocessing these artifacts, the authors estimate a motion model during a half rotation for a whole field of view via nonrigid registration between the images. Finally, motion-compensated image reconstruction is performed at a target phase by incorporating the estimated motion model. The target phase is selected as that corresponding to a view angle that is orthogonal to the center view angles of two conjugate PAR images. To evaluate the proposed algorithm, digital XCAT and physical dynamic cardiac phantom datasets are used. The XCAT phantom datasets were generated with heart rates of 70 and 100 bpm, respectively, by assuming a system rotation time of 300 ms. A physical dynamic cardiac phantom was scanned using a slowly rotating XCT system so that the effective heart rate will be 70 bpm for a system rotation speed of 300 ms. In the XCAT phantom experiment, motion-compensated 3D images obtained from the proposed algorithm show coronary arteries with fewer motion artifacts for all phases. Moreover, object boundaries contaminated by motion are well restored. Even though object positions and boundary shapes are still somewhat different from the ground truth in some cases, the authors see that visibilities of coronary arteries are improved noticeably and motion artifacts are reduced considerably. The physical phantom study also shows that the visual quality of motion-compensated images is greatly improved. The authors propose a novel PAR image-based cardiac motion estimation and compensation algorithm. The algorithm requires an angular scan range of less than 360°. The excellent performance of the proposed algorithm is illustrated by using digital XCAT and physical dynamic cardiac phantom datasets.

PurposeOur purpose was to perform an in vivo validation of ultrasound imaging for intrafraction motion estimation using the Elekta Clarity Autoscan system during prostate radiation therapy. The study was conducted as part of the Clarity-Pro trial (NCT02388308).Methods and MaterialsInitial locations of intraprostatic fiducial markers were identified from cone beam computed tomography scans. Marker positions were translated according to Clarity intrafraction 3-dimensional prostate motion estimates. The updated locations were projected onto the 2-dimensional electronic portal imager plane. These Clarity-based estimates were compared with the actual portal-imaged 2-dimensional marker positions. Images from 16 patients encompassing 80 fractions were analyzed. To investigate the influence of intraprostatic markers and image quality on ultrasound motion estimation, 3 observers rated image quality, and the marker visibility on ultrasound images was assessed.ResultsThe median difference between Clarity-defined intrafraction marker locations and portal-imaged marker locations was 0.6 mm (with 95% limit of agreement at 2.5 mm). Markers were identified on ultrasound in only 3 of a possible 240 instances. No linear relationship between image quality and Clarity motion estimation confidence was identified. The difference between Clarity-based motion estimates and electronic portal–imaged marker location was also independent of image quality. Clarity estimation confidence was degraded in a single fraction owing to poor probe placement.ConclusionsThe accuracy of Clarity intrafraction prostate motion estimation is comparable with that of other motion-monitoring systems in radiation therapy. The effect of fiducial markers in the study was deemed negligible as they were rarely visible on ultrasound images compared with intrinsic anatomic features. Clarity motion estimation confidence was robust to variations in image quality and the number of ultrasound-imaged anatomic features; however, it was degraded as a result of poor probe placement.

Videogrammetry and photogrammetry are increasingly being used in marine science for unsupervised data collection. The camera systems employed are complex, in contrast to “consumer” digital cameras and smartphones carried by potential citizen scientists. However, using consumer cameras in photogrammetry will introduce unknown length estimation errors through both the image acquisition process and lens distortion. This study presents a methodology to achieve accurate 2-dimensional (2-D) total length (TL) estimates of fish without specialist equipment or proprietary software. Photographs of fish were captured with an action camera using a background fiducial marker, a foreground fiducial marker and a laser marker. The geometric properties of the lens were modelled with OpenCV to correct image distortion. TL estimates were corrected for parallax effects using an algorithm requiring only the initial length estimate and known fish morphometric relationships. Correcting image distortion decreased RMSE by 96% and the percentage mean bias error (%MBE) by 50%. Correcting for parallax effects achieved a %MBE of −0.6%. This study demonstrates that the morphometric measurement of different species can be accurately estimated without the need for complex camera equipment, making it particularly suitable for deployment in citizen science and other volunteer-based data collection endeavours.