Learning geometry-aware joint latent space for simultaneous multimodal shape generation

Three-dimensional to two-dimensional (3D-2D) image registration is a key to fusion and simultaneous visualization of valuable information contained in 3D pre-interventional and 2D intra-interventional images with the final goal of image guidance of a procedure. In this paper, the authors focus on 3D-2D image registration within the context of intracranial endovascular image-guided interventions (EIGIs), where the 3D and 2D images are generally acquired with the same C-arm system. The accuracy and robustness of any 3D-2D registration method, to be used in a clinical setting, is influenced by (1) the method itself, (2) uncertainty of initial pose of the 3D image from which registration starts, (3) uncertainty of C-arm's geometry and pose, and (4) the number of 2D intra-interventional images used for registration, which is generally one and at most two. The study of these influences requires rigorous and objective validation of any 3D-2D registration method against a highly accurate reference or "gold standard" registration, performed on clinical image datasets acquired in the context of the intervention. The registration process is split into two sequential, i.e., initial and final, registration stages. The initial stage is either machine-based or template matching. The latter aims to reduce possibly large in-plane translation errors by matching a projection of the 3D vessel model and 2D image. In the final registration stage, four state-of-the-art intrinsic image-based 3D-2D registration methods, which involve simultaneous refinement of rigid-body and C-arm parameters, are evaluated. For objective validation, the authors acquired an image database of 15 patients undergoing cerebral EIGI, for which accurate gold standard registrations were established by fiducial marker coregistration. Based on target registration error, the obtained success rates of 3D to a single 2D image registration after initial machine-based and template matching and final registration involving C-arm calibration were 36%, 73%, and 93%, respectively, while registration accuracy of 0.59 mm was the best after final registration. By compensating in-plane translation errors by initial template matching, the success rates achieved after the final stage improved consistently for all methods, especially if C-arm calibration was performed simultaneously with the 3D-2D image registration. Because the tested methods perform simultaneous C-arm calibration and 3D-2D registration based solely on anatomical information, they have a high potential for automation and thus for an immediate integration into current interventional workflow. One of the authors' main contributions is also comprehensive and representative validation performed under realistic conditions as encountered during cerebral EIGI.

GoDec+ shows its robustness in low-rank matrix decomposition but only deals with single-view data. This paper extends GoDec+ to multi-view data by jointly learning latent space and multi-view fusion feature. The proposed method factorizes the low-rank matrix in GoDec+ into the product of a basis matrix of the latent space and a shared representation given by a transformation matrix. By constraining the basis matrix to be group sparse, the proposed method treats the effects of different views differently. Extensive experiments show that the proposed method learns a good fusion feature and outperforms the compared methods in image classification and annotation.

The direct extension of 2D image learning to three-dimensional space is 3D point cloud learning. Recently, point cloud learning has shown significant results in 3D shape estimation and semantic segmentation. Despite these advancements, fundamental problems in point cloud learning still pose significant challenges. These problems include representation learning, shape generation, shape segmentation, and shape matching. In this paper, we propose a cognitive self-attention based learning approach to aggregate global representation of 3D shapes from point cloud data. We also integrate 3D point data with a binary tree structure to build a point cloud generator. We further design a novel Generative Adversarial Network (GAN) architecture to generate point clouds resembling the ground truth that could be used for unsupervised learning of 3D shapes. Relying on a self-attention mechanism, our framework, called SAPCGAN, aggregates the global graph features to correct the structural information of 3D shapes in the latent space. Finally, we compare the performance of two gradient penalty methods used in stabilizing the training of our GAN system. We show that our framework has high training efficiency and can provide state-of-the-art results in 3D point cloud generation. The performance of our is demonstrated with both quantitative and qualitative experimental evaluations. Furthermore, the generated 3D point clouds can be segmented into their natural semantic parts, such as, for example the four legs of a chair, the wings of an air plane, or the four wheels of a car.

Judgement of nasolabial aesthetics in cleft lip and palate (CLP) is a vital component of assessment of treatment outcome. It is usually performed based on two-dimensional (2D) facial photographs. An increasing use of three-dimensional (3D) imaging warrants an assessment if 3D images can substitute 2D photographs during aesthetic evaluation. The aim of this study was to compare reliability of rating nasolabial appearance on 3D images and standard 2D photographs in prepubertal children. Forty subjects (age: 8.8-12) with unilateral CLP treated according to a standardized protocol, who had 2D and 3D facial images were selected. Eight lay raters assessed nasal form, nasal deviation, vermilion border, and nasolabial profile on cropped 2D and 3D images using a 100-mm visual analogue scale (VAS). Additionally, raters answer two questions: 1. Do 2D or 3D images provide more information on nasolabial aesthetics? and 2. Is aesthetic evaluation easier on 2D or 3D images? Intrarater agreement demonstrated a better reliability of ratings performed on 3D images than 2D images (correlation coefficients for 3D images ranged from 0.733 to 0.857; for 2D images from 0.151 to 0.611). The mean scores showed, however, no difference between 2D and 3D formats (>0.05). 3D images were regarded more informative than 2D images (P = 0.001) but probably more difficult to evaluate (P = 0.06). Basal view of the nose was not assessed. 3D images seem better than 2D images for rating nasolabial aesthetics but raters should familiarize themselves with them prior to rating.

This study investigates and compares the reliability and reproducibility of two facial anthropometric methods about external nasal angles, 3D imaging and conventional 2D photography. 2D photograph images and 3D images about external nose of 30 volunteers were taken using digital camera and Morpheus 3D scanner. To evaluate intra-rater reliability, each images were taken over two different days for each subject by the same researcher. To evaluate inter-rater reliability, another researcher took each images for each subject on the first day. The reliability of each method for measuring 4 external nasal angle is obtained using intraclass correlation coefficient (ICC) and compared. Inter-rater and intra-rater reliability of both 3D imaging and 2D photography had excellent agreement in all 4 nasal angles. In the nasofacial angular parameter, Inter-rater ICC, 2D photography was significantly higher than 3D imaging. Result of intra-rater ICC also showed both 3D imaging and 2D photography had good reliability in all 4 nasal angles. Similar to those of inter-rater ICC, nasofacial angular parameter showed statistically significant differences between 3D imaging and 2D photography. In terms of reliability, both 2D and 3D showed appropriate anthropometric results and considering its own advantage, each methods can be used complementarily.

Point clouds are fundamental in the representation of 3D objects. However, they can also be highly unstructured and irregular. This makes it difficult to directly extend 2D generative models to three-dimensional space. In this article, we cast the problem of point cloud generation as a topological representation learning problem. In order to capture the representative features of 3D shapes in the latent space, we propose a hierarchical mixture model that integrates self-attention with an inference tree structure for constructing a point cloud generator. Based on this, we design a novel Generative Adversarial Network (GAN) architecture that is capable of generating recognizable point clouds in an unsupervised manner. The proposed adversarial framework (SG-GAN) relies on self-attention mechanism and Graph Convolution Network (GCN) to hierarchically infer the latent topology of 3D shapes. Embedding and transferring the global topology information in a tree framework allows our model to capture and enhance the structural connectivity. Furthermore, the proposed architecture endows our model with partially generating 3D structures. Finally, we propose two gradient penalty methods to stabilize the training of SG-GAN and overcome the possible mode collapse of GAN networks. To demonstrate the performance of our model, we present both quantitative and qualitative evaluations and show that SG-GAN is more efficient in training and it exceeds the state-of-the-art in 3D point cloud generation.

Artificial intelligence assistance for fetal head biometry: Assessment of automated measurement software.

CT Colonography evolution

Performance analysis of real-coded evolutionary algorithms under a computationally expensive optimization scenario: 3D–2D Comparative Radiography

Background Clinicians leverage information from multiple data sources, such as the electrocardiogram (ECG) and echocardiogram to diagnose and manage cardiovascular disease. Deep learning models have shown immense potential to enhance these clinical processes by capturing digital biomarkers of cardiovascular disease from diagnostic modalities. However, the development of multimodal algorithms capable of making joint inferences from multiple modalities, mirroring the clinicians’ decision-making, has remained an unmet need. Purpose We aimed to develop a multimodal foundation model of ECG and echocardiogram as an integrative deep learning approach to evaluating cardiovascular conditions. Methods Twelve-lead ECGs and transthoracic echocardiograms performed at a large academic health system between 2001-2021 were collected. Individuals with paired ECGs and echocardiograms recorded within 3 years were identified and the data was split at the individual level into training, validation, and test sets (80/10/10%). Using paired ECGs and echocardiograms, we developed a deep learning model in a self-supervised setting with contrastive loss and merged embeddings in a joint latent space (Figure 1). We evaluated the performance of this foundation model after fine-tuning for left ventricular systolic dysfunction (LVSD defined as left ventricular ejection fraction of 40% or less), atrial fibrillation, age, and sex as auxiliary tasks. The performance of the multimodal model was compared with those of supervised unimodal ECG-only and echocardiography-only models for each task. Results In total, 4,391,610 videos from 129,424 echocardiography studies paired with 1,094,656 ECGs from 53,876 unique individuals were included (67 ± 14 years of age, 43% female, Figure 1). The multimodal algorithm demonstrated excellent discrimination for disease classification in the test set. In classification of LVSD, for example, it achieved an area under the receiver operating characteristic (AUROC) curve of 0.94, area under precision recall curve (AUPRC) of 0.99 among individuals with LV ejection fraction greater than 40%, and AUPRC of 0.72 among those with LV ejection fraction of 40% or less. The multimodal model consistently outperformed the unimodal algorithms, including the ECG-only and echocardiography-only models in classification of LVSD (AUROC ECG+Echo 0.94, ECG 0.89, Echo 0.89), atrial fibrillation (AUROC ECG+Echo 0.95, ECG, 0.94, Echo 0.75), sex (AUROC ECG+Echo 0.93, ECG 0.87, Echo 0.78) and age (R for variance ECG+Echo 0.75, ECG 0.67, Echo 0.58, Figure 2). Conclusion We have developed a multimodal foundation model for joint inference from ECGs and echocardiograms for diagnosis of cardiovascular conditions. This novel approach outperforms the conventional unimodal models in disease diagnosis and represents an advance in integrating the clinical applications of artificial intelligence.Figure 1 Figure 2

Pose estimation is a widely explored problem, enabling many robotic tasks such as grasping and manipulation. In this paper, we tackle the problem of pose estimation for objects that exhibit rotational symmetry, which are common in man-made and industrial environments. In particular, our aim is to infer poses for objects not seen at training time, but for which their 3D CAD models are available at test time. Previous work has tackled this problem by learning to compare captured views of real objects with the rendered views of their 3D CAD models, by embedding them in a joint latent space using neural networks. We show that sidestepping the issue of symmetry in this scenario during training leads to poor performance at test time. We propose a model that reasons about rotational symmetry during training by having access to only a small set of symmetry-labeled objects, whereby exploiting a large collection of unlabeled CAD models. We demonstrate that our approach significantly outperforms a naively trained neural network on a new pose dataset containing images of tools and hardware.

This article introduces a two-phase deep feature engineering framework for efficient learning of semantics enhanced joint embedding, which clearly separates the deep feature engineering in data preprocessing from training the text-image joint embedding model. We use the Recipe1M dataset for the technical description and empirical validation. In preprocessing, we perform deep feature engineering by combining deep feature engineering with semantic context features derived from raw text-image input data. We leverage LSTM to identify key terms, deep NLP models from the BERT family, TextRank, or TF-IDF to produce ranking scores for key terms before generating the vector representation for each key term by using Word2vec. We leverage Wide ResNet50 and Word2vec to extract and encode the image category semantics of food images to help semantic alignment of the learned recipe and image embeddings in the joint latent space. In joint embedding learning, we perform deep feature engineering by optimizing the batch-hard triplet loss function with soft-margin and double negative sampling, taking into account also the category-based alignment loss and discriminator-based alignment loss. Extensive experiments demonstrate that our SEJE approach with deep feature engineering significantly outperforms the state-of-the-art approaches.

In this article, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a variational autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individual component evidence lower bounds (MELBO) on the log-likelihood of the given training samples. The mixing coefficients in the mixture model control the contributions of each expert in the global representation. These are sampled from a Dirichlet distribution whose parameters are determined through nonparametric estimation during lifelong learning. The model can learn new tasks fast when these are similar to those previously learned. The proposed lifelong mixture of VAE (L-MVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used when fed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert is used during the inference. The L-MVAE inference model is able to perform interpolations in the joint latent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation.

Photocurable three-dimensional (3D) printing is a stepwise layer-by-layer fabrication process widely used in the manufacture of highly specialized objects. Current 3D printing techniques are easily implemented; however, the build rate is slow and the surface quality is less than ideal. Holographic 3D display (3DHD) technology makes it possible to reform planar wavefronts into a 3D intensity distribution, which appears as a 3D image in space. This paper examined the application of holographic imaging technology to 3D printing based on photocurable polymers. The proposed system uses a 3DHD diffractive optics system based on a liquid-crystal-on-silicon spatial light modulator (LCoS-SLM), wherein a 3D layered image is created in the optical near field, based on a computer-generated hologram (CGH) optimized using the iterative angular spectrum algorithm (IASA) and a circular IASA. From a single CGH, multiple 2D sliced images are created in space to form a 3D optical image used to initiate the photopolymerization of photocurable resin to form 3D objects. In experiments, the proposed 3D printing system was used to create five polymer objects with a maximum axial length of 25 mm and minimum feature width of 149 μm. The phase-only CGH reformed the incident light into a distribution of optical intensity with high diffraction efficiency suitable for photocuring. Despite limitations pertaining to fabrication area and axial complexity in this initial study, the proposed method demonstrated high light efficiency, high resolution in the lateral direction, rapid fabrication, and good object continuity.

This article presents a new series of invariant moments, called Fractional-order Generalized Laguerre Moment Invariants (FGLMI), based on Fractional-order Generalized Laguerre polynomials (FGLPs). To begin, we provide the relations and the properties necessary to define the fractional-order generalized Laguerre moments. Then, we present the theoretical framework to derive invariants from fractional-order moments with respect to the change in orientation, size and position based on the algebraic relationships between FGLM and fractional-order geometric moments. In addition, a fast and precise algorithm has been proposed for the calculation of FGLM in order to speed up the calculation time and ensure the numerical stability of the invariant moments. Numerical experiments are carried out to demonstrate the efficiency of FGLM and their proposed invariants compared to existing methods, with regard to the reconstruction of 2D and 3D images, the computation time, the global entity extraction capacity and image localization, invariability property and 2D / 3D image classification performance on different 2D and 3D image databases. The theoretical and experimental results presented clearly show the efficiency of the descriptors proposed for the representation and classification of 2D and 3D images by other types of orthogonal moments.