Video-based face recognition and image synthesis from rotating head frames using nonlinear manifold learning by neural networks

In this paper, we propose a highly automatic approach for 3D photorealistic face reconstruction from a single frontal image. The key point of our work is the implementation of adaptive manifold learning approach. Beforehand, an active appearance model (AAM) is trained for automatic feature extraction and adaptive locally linear embedding (ALLE) algorithm is utilized to reduce the dimensionality of the 3D database. Then, given an input frontal face image, the corresponding weights between 3D samples and the image are synthesized adaptively according to the AAM selected facial features. Finally, geometry reconstruction is achieved by linear weighted combination of adaptively selected samples. Radial basis function (RBF) is adopted to map facial texture from the frontal image to the reconstructed face geometry. The texture of invisible regions between the face and the ears is interpolated by sampling from the frontal image. This approach has several advantages: (1) Only a single frontal face image is needed for highly automatic face reconstruction; (2) Compared with former works, our reconstruction approach provides higher accuracy; (3) Constraint based RBF texture mapping provides natural appearance for reconstructed face.

This paper proposes a method for discovering the primary objects in single images by learning from videos in a purely unsupervised manner—the learning process is based on videos, but the generated network is able to discover objects from a single input image. The rough idea is that an image typically consists of multiple object instances (like the foreground and background) that have spatial transformations across video frames and they can be sparsely represented. By exploring the sparsity representation of a video with a neural network, one may learn the features of each object instance without any labels, which can be used to discover, recognize, or distinguish object instances from a single image. In this paper, we consider a relatively simple scenario, where each image roughly consists of a foreground and a background. Our proposed method is based on encoder-decoder structures to sparsely represent the foreground, background, and segmentation mask, which further reconstruct the original images. We apply the feed-forward network trained from videos for object discovery in single images, which is different from the previous co-segmentation methods that require videos or collections of images as the input for inference. The experimental results on various object segmentation benchmarks demonstrate that the proposed method extracts primary objects accurately and robustly, which suggests that unsupervised image learning tasks can benefit from the sparsity of images and the inter-frame structure of videos.

Facial pose synthesis has many useful applications in practice. How to synthesize facial pose images robustly and simply is still a challenging problem. In this paper we proposed a tensor-based subspace learning method (TSL) that makes possible the synthesis of human multi-pose facial images from a single 2D image. We organize 2D multi-pose images in a tensor form and apply tensor decomposition to build a projection subspace. An input 2D image is projected into the projection subspace to get a corresponding identity vector. The identity vector is used to generate the novel pose images. The experiments are performed on MaVIC (KAO-Ritsumeikan Multi-angle View, Illumination and Cosmetic Facial Database) and preliminary experimental results show the effectiveness of our proposed method.

To investigate the effect of image averaging on qualitative and quantitative assessments of optical coherence tomography angiography (OCTA) images from eyes of patients with branch retinal vein occlusion (BRVO). Macular OCTA images of 33 eyes of 33 patients with BRVO were obtained using the HS100 HR-SD-OCT system (Canon, Inc.). For each eye, five OCTA cube scans were obtained with a 3 × 3mm scan protocol, and the data were averaged and compounded into a single high image quality cube data using built-in software. Pre- and post-averaging images were compared qualitatively and quantitatively in superficial capillary plexus (SCP) and deep capillary plexus (DCP) OCTA image slabs. After averaging, all OCTA images showed marked improvement in image quality with less background noise and better vessel continuity. The number of microaneurysms in both the SCP and DCP was larger in single images than in averaged images. A significant increase in the detection rate of capillary telangiectasia in the DCP was noted after image averaging. The number of eyes with disrupted foveal avascular zone (FAZ) decreased significantly after averaging (P = .0253). Five eyes (15.2%) with a disrupted FAZ on the single image showed an intact FAZ after averaging. Vessel length density (VLD) and fractal dimension (FD) significantly decreased and vessel diameter index (VDI) increased after averaging, while significant changes were not observed in vessel density (VD) in both the SCP and DCP. In the SCP, lower VD, VLD, and fractal dimension were significantly correlated with worse visual acuity. OCTA averaging has a significant effect on qualitative and quantitative assessments in eyes with BRVO.

BackgroundAccurate Hounsfield units (HU) are critical for dose calculation and anatomical visualization, but are often affected by dental artifacts in head and neck (H&N) cancer patients. The HyperSight cone‐beam computed tomography (CBCT) platform provides improved image quality over previous CBCT platforms and offers metal artifact reduction (iCBCT MAR) reconstruction.PurposeThis study evaluates the quality of HyperSight CBCT images compared to current clinical standards: TrueBeam CBCT for image guidance and fan‐beam CT (FBCT) from a CT simulator for treatment planning, using images captured during H&N cancer treatment.MethodsImages for 30 H&N cancer patients were acquired on a HyperSight CBCT, conventional TrueBeam and FBCT, with 24 patients exhibiting metal dental artifacts. The HyperSight images were reconstructed using iCBCT MAR and iterative (iCBCT Acuros) algorithms. The four image sets were rigidly registered and compared using the artifact index (AI) measured in the oral cavity and the percentile range (PR) measured in the oral cavity, brain, brainstem and eyes to assess image non‐uniformity. The HU accuracy was calculated relative to FBCT (baseline) for soft tissues (oral cavity, brainstem, submandibular and parotid glands), and bone (mandible). The contrast relative to baseline was evaluated between the oral cavity and nearby structures. Image‐based metrics were computed relative to FBCT including structural similarity index measure (SSIM), mean‐square error (MSE) and peak signal‐to‐noise ratio (PSNR).ResultsThe HyperSight iCBCT MAR images showed a significant reduction in AI values compared to the other images (p < 0.0004), but higher PR values indicating decreased HU uniformity compared to HyperSight iCBCT Acuros and FBCT (p < 0.0002). The soft‐tissue HU and contrast values were significantly closer to baseline in both HyperSight images compared to TrueBeam (HU: p < 0.001, contrast: p < 0.001). For soft‐tissue the HU mean absolute deviation (MAD) from baseline was 16 ± 10 HU for HyperSight iCBCT Acuros, 15 ± 10 HU for HyperSight iCBCT MAR, and 35 ± 22 HU for TrueBeam. For bone, the HU MAD from baseline was 153 ± 233 HU, 185 ± 268 HU, and 214 ± 212 HU, respectively. The HyperSight iCBCT Acuros algorithm achieved significantly superior SSIM, MSE, and PSNR metrics compared to TrueBeam and HyperSight iCBCT MAR in regions with large amounts of bone and air.ConclusionsHyperSight iCBCT MAR significantly reduced artifacts compared to HyperSight iCBCT Acuros, TrueBeam and FBCT, making it particularly beneficial for patients with metal implants. Both HyperSight reconstructions demonstrated improved soft‐tissue HU accuracy and contrast compared to TrueBeam, however the iCBCT Acuros algorithm may be preferred when metal‐induced artifacts are not a concern. These results support the suitability of HyperSight images in adaptive treatment workflows requiring accurate image quality, even with severe metal artifacts.

Plane wave (PW) imaging is fast, but limited by poor imaging quality. Coherent PW compounding (CPWC) improves image quality but decrease frame rate. In this study, we propose a modified CycleGAN model that combines a residual attention module with a space-frequency dual-domain discriminator, termed RADD-CycleGAN, to rapidly reconstruct high-quality ultrasound images. To enhance the ability to reconstruct image details, we specially design a process of hybrid dynamic and static channel selection followed by the frequency domain discriminator. The low-quality images are generated by the 3-angle CPWC, while the high-quality images are generated as real images (ground truth) by the 75-angle CPWC. The training set includes unpaired images, whereas the images in the test set are paired to verify the validity and superiority of the proposed model. Finally, we respectively design ablation and comparison experiments to evaluate the model performance. Compared with the basic CycleGAN, our proposed method reaches a better performance, with a 7.8% increase in the peak signal-to-noise ratio and a 22.2% increase in the structural similarity index measure. The experimental results show that our method achieves the best unsupervised reconstruction from low quality images in comparison with several state-of-the-art methods.

Blurring facial images can significantly degrade the performance of face recognition and video surveillance applications. Therefore, an effective image restoration method is essential to address this issue. However, existing methods struggle with varying levels of blur severity, limiting their effectiveness. This study proposes a facial image restoration approach that integrates blur severity classification using a Convolutional Neural Network (CNN) with a U-Net-based deblurring model to overcome these challenges. This method ensures that each blurred image is processed using the most suiTable deblurring model, optimizing the restoration process. The dataset used in this study is Flickr-Faces-HQ (FFHQ), to which a Gaussian blur is applied and categorized into five levels: very low, low, medium, high, and very high. They employ the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) as quantitative metrics to evaluate the model's performance. Experimental results show that the proposed method consistently outperforms fixed kernel size and multi-kernel size deblurring approaches across all blur severities. Specifically, our method achieves a PSNR of 40.001 dB for very low blur severity and an SSIM of 0.990. For low severity, it attains a PSNR of 31.104 dB and an SSIM of 0.946. For medium severity, the results are 27.995 dB (PSNR) and 0.874 (SSIM). At high severity, our model achieves a PSNR of 28.896 dB and an SSIM of 0.855. Finally, for very high severity, the PSNR drops to 26.566 dB, with an SSIM of 0.812. These results demonstrate the effectiveness of the proposed method in enhancing image clarity across different blur levels while preserving facial details. This research contributes to the development of more adaptive and efficient image restoration techniques, particularly for applications that require high-quality facial images, such as face recognition and video surveillance.

Context. Removing the undesirable consequences of rain effects from single images is an actual problem in many computer vision tasks, because rain streaks can significantly degrade the visual quality of images and seriously interfere with the operation of various intelligent systems, which are used for their processing and further analysis.&#x0D; Objective. The goal of the work is to develop a method for detecting and removing undesirable effects of the rain from single images, which is based on the using of a convolutional neural network with a recurrent structure.&#x0D; Method. The main component of the proposed method is a convolutional neural network, which has a recurrent multi-stage structure. A feature of this network architecture is the use of repeated blocks (layers), at the output of which you can get an intermediate result of «cleaning» the original image. Moreover at the output of each next layer of the network we get an image with less influence of rain components than on the previous one. Each network layer contains two independent sub-networks (branches) for parallel image processing. The main branch is designed to detect and remove the effect of rain from the image and the attention branch is used to improve and speed up the process of detecting undesirable rain components (for rain attention map formation).&#x0D; Results. An approach has been developed to automatically detect and remove the rain effect from single images. The process of “cleaning” the original image is based on the use of a convolutional neural network with a recurrent structure, which was trained on the Rain100H and Rain100L datasets. The results of computer experiments, which testifies to the effectiveness and expediency of using the proposed method for solving practical tasks of pre-processing “contaminated” images are presented.&#x0D; Conclusions. The advantage of the developed method for removing undesirable components of rain from images is that the recurrent multi-stage network architecture, on which it is based allows it to be potentially applied to solving tasks under conditions of limited computing resources. The proposed method can be successfully used in the development of intelligent systems for area monitoring with surveillance cameras, autonomous vehicles control, processing aerial photography results, etc. In the future, it should be considered the possibility of forming a separate sub-network to eliminate blurring in the image and train the network on datasets that contain image samples with different components of rain, which will make the method more «resistant» to different forms of the rain effect and increase the quality of image “cleaning”.

Ultrafast ultrasound imaging, characterized by high frame rates, generates low-quality images. Convolutional neural networks (CNNs) have demonstrated great potential to enhance image quality without compromising the frame rate. However, CNNs have been mostly trained on simulated or phantom images, leading to suboptimal performance on in vivo images. In this study, we present a method to enhance the quality of single plane wave (PW) acquisitions using a CNN trained on in vivo images. Our contribution is twofold. Firstly, we introduce a training loss function that accounts for the high dynamic range of the radio frequency data and uses the Kullback-Leibler divergence to preserve the probability distributions of the echogenicity values. Secondly, we conduct an extensive performance analysis on a large new in vivo dataset of 20,000 images, comparing the predicted images to the target images resulting from the coherent compounding of 87 PWs. Applying a volunteer-based dataset split, the peak signal-to-noise ratio and structural similarity index measure increase, respectively, from 16.466 ± 0.801 dB and 0.105 ± 0.060, calculated between the single PW and target images, to 20.292 ± 0.307 dB and 0.272 ± 0.040, between predicted and target images. Our results demonstrate significant improvements in image quality, effectively reducing artifacts.

Application of Self2Self self-supervised denoising framework to airborne gamma-ray spectrometry data.

We present a stochastic filtering approach to perform albedo estimation from a single non-frontal face image. Albedo estimation has far reaching applications in various computer vision tasks like illumination-insensitive matching, shape recovery, etc. We extend the formulation proposed in [3] that assumes face in known pose and present an algorithm that can perform albedo estimation from a single image even when pose information is inaccurate. 3D pose of the input face image is obtained as a byproduct of the algorithm. The proposed approach utilizes class-specific statistics of faces to iteratively improve albedo and pose estimates. Illustrations and experimental results are provided to show the effectiveness of the approach. We highlight the usefulness of the method for the task of matching faces across variations in pose and illumination. The facial pose estimates obtained are also compared against ground truth.

The civil engineering industry faces a critical need for innovative non-destructive evaluation methods, particularly for ageing critical infrastructure, such as bridges, where current techniques fall short. Muography, a non-invasive imaging technique, constructs three-dimensional density maps by detecting the interactions of naturally occurring cosmic-ray muons within the scanned volume. Cosmic-ray muons offer both deep penetration capabilities due to their high momenta and inherent safety due to their natural source. However, the technology’s reliance on this natural source results in a constrained muon flux, leading to prolonged acquisition times, noisy reconstructions, and challenges in image interpretation. To address these limitations, we developed a two-model deep learning approach. First, we employed a conditional Wasserstein Generative Adversarial Network with Gradient Penalty (cWGAN-GP) to perform predictive upsampling of undersampled muography images. Using the Structural Similarity Index Measure (SSIM), 1-day sampled images were able to match the perceptual qualities of a 21-day image, while the Peak Signal-to-Noise Ratio (PSNR) indicated a noise improvement to that of 31 days worth of sampling. A second cWGAN-GP model, trained for semantic segmentation, was developed to quantitatively assess the upsampling model’s impact on each of the features within the concrete samples. This model was able to achieve segmentation of rebar grids and tendon ducts embedded in the concrete, with respective Dice–Sørensen accuracy coefficients of 0.8174 and 0.8663. This model also revealed an unexpected capability to mitigate—and in some cases entirely remove—z-plane smearing artifacts caused by the muography’s inherent inverse imaging problem. Both models were trained on a comprehensive dataset generated through Geant4 Monte Carlo simulations designed to reflect realistic civil infrastructure scenarios. Our results demonstrate significant improvements in both acquisition speed and image quality, marking a substantial step toward making muography more practical for reinforced concrete infrastructure monitoring applications.

PurposeKilovoltage cone beam computed tomography (kVCBCT)‐guided adaptive radiation therapy (ART) uses daily deformed CT (dCT), which is generated automatically through deformable registration methods. These registration methods may perform poorly in reproducing volumes of the target organ, rectum, and bladder during treatment. We analyzed the registration errors between the daily kVCBCTs and corresponding dCTs for these organs using the default optical flow algorithm and two registration procedures. We validated the effectiveness of these registration methods in replicating the geometry for dose calculation on kVCBCT for ART.MethodsWe evaluated three deformable image registration (DIR) methods to assess their registration accuracy and dose calculation effeciency in mapping target and critical organs. The DIR methods include (1) default intensity‐based deformable registration, (2) hybrid deformable registration, and (3) a two‐step deformable registration process. Each technique was applied to a computerized imaging reference system (CIRS) phantom (Model 062 M) and to five patients who received volumetric modulated arc therapy to the prostate. Registration accuracy was assessed using the 95% Hausdorff distance (HD95) and Dice similarity coefficient (DSC), and each method was compared with the intensity‐based registration method. The improvement in the dCT image quality of the CIRS phantom and five patients was assessed by comparing dCT with kVCBCT. Image quality quantitative metrics for the phantom included the signal‐to‐noise ratio (SNR), uniformity, and contrast‐to‐noise ratio (CNR), whereas those for the patients included the mean absolute error (MAE), mean error, peak signal‐to‐noise ratio (PSNR), and structural similarity index measure (SSIM). To determine dose metric differences, we used a dose‐volume histogram (DVH) and 3.0%/0.3 mm gamma analysis to compare planning computed tomography (pCT) and kVCBCT recalculations with restimulated CT images used as a reference.ResultsThe dCT images generated by the hybrid (dCTH) and two‐step (dCTC) registration methods resulted in significant improvements compared to kVCBCT in the phantom model. Specifically, the SNR improved by 107% and 107.2%, the uniformity improved by 90% and 75%, and the CNR improved by 212.2% and 225.6 for dCTH and dCTC methods, respectively. For the patient images, the MAEs improved by 98% and 94%, the PSNRs improved by 16.3% and 22.9%, and the SSIMs improved by 1% and 1% in the dCTH and dCTC methods, respectively. For the geometric evaluation, only the two‐step registration method improved registration accuracy. The dCTH method yielded an average HD95 of 12 mm and average DSC of 0.73, whereas dCTC yielded an average HD95 of 2.9 mm and average DSC of 0.902. The DVH showed that the dCTC‐based dose calculations differed by <2% from the expected results for treatment targets and volumes of organs at risk. Additionally, gamma indices for dCTC‐based treatment plans were >95% at all points, whereas they were <95% for kVCBCT‐based treatment plans.ConclusionThe two‐step registration method outperforms the intensity‐based and hybrid registration methods. While the hybrid and two‐step‐based methods improved the image quality of kVCBCT in a linear accelerator, only the two‐step method improved the registration accuracy of the corresponding structures among the pCT and kVCBCT datasets. A two‐step registration process is recommended for applying kVCBCT to ART, which achieves better registration accuracy for local and global image structures. This method appears to be beneficial for radiotherapy dose calculation in patients with pelvic cancer.

To propose an acceleration method for 3D variable flip-angle (VFA) T1 mapping based on a technique called shift undersampling improves parametric mapping efficiency and resolution (SUPER). The proposed method incorporates strategies of SUPER, controlled aliasing in volumetric parallel imaging (CAIPIRINHA), and total variation-based regularization to accelerate 3D VFA T1 mapping. The k-space sampling grid of CAIPIRINHA is internally undersampled with SUPER along the contrast dimension. A proximal algorithm was developed to preserve the computational efficiency of SUPER in the presence of regularization. The regularized SUPER-CAIPIRINHA (rSUPER-CAIPIRINHA) was compared with lowrank plus sparsity (L + S), reconstruction of principal component coefficient maps (REPCOM), and other SUPER-based methods via simulations and in vivo brain T1 mapping. The results were assessed quantitatively with NRMSE and structural similarity index measure (SSIM), and qualitatively by two experienced reviewers. rSUPER-CAIPIRINHA achieved a lower NRMSE and higher SSIM than L + S (0.11 ± 0.01 vs. 0.19 ± 0.03, p < 0.001; 0.66 ± 0.05 vs. 0.37 ± 0.03, p < 0.001) and REPCOM (0.16 ± 0.02, p < 0.001; 0.46 ± 0.04, p < 0.001). The reconstruction time of rSUPER-CAIPIRINHA was 6% of L + S and 2% of REPCOM. For the qualitative comparison, rSUPER-CAIPIRINHA showed improvement of overall image quality and reductions of artifacts and blurring, although with a lower apparent SNR. Compared with 2D SUPER-SENSE, rSUPER-CAIPIRINHA significantly reduced NRMSE (0.11 ± 0.01 vs. 0.23 ± 0.04, p < 0.001) and generated less noisy reconstructions. By incorporating SUPER, CAIPIRINHA, and regularization, rSUPER-CAIPIRINHA mitigated noise amplification, reduced artifacts and blurring, and achieved faster reconstructions compared with L + S and REPCOM. These advantages render 3D rSUPER-CAIPIRINHA VFA T1 mapping potentially useful for clinical applications.

Purpose:Fusing information from multiple correlated realizations (e.g., 4DCT) can improve image quality. This process often involves ill‐conditioned and asymmetric nonlinear registration and the proper selection of a reference image is important. This work proposes to examine post‐registration variation indirectly for such selection, and develops further insights to reduce the number of cross‐registrations needed.Methods:We consider each individual scan as a noisy point in the vicinity of an image manifold, related by motion. Nonrigid registration “transports” a scan along the manifold to the reference neighborhood, and the residual is a surrogate for local variation. To test this conjecture, 10 thoracic scans from the same session were reconstructed from a recently developed low‐dose helical 4DCT protocol. Pairwise registration was repeated bi‐directionally (81 times) and fusion was performed with each candidate reference. The fused image quality was assessed with SNR and CNR. Registration residuals in SSD, harmonic energy, and deformation Jacobian behavior were examined. The semi‐symmetry is further utilized to reduce the number of registration needed.Results:The comparison of image quality between single image and fused ones identified reduction of local intensity variance as the major contributor of image quality, boosting SNR and CNR by 5 to 7 folds. This observation further suggests the criticality of good agreement across post‐registration images. Triangle inequality on the SSD metric provides a proficient upper‐bound and surrogate on such disagreement. Empirical observation also confirms that fused images with high residual SSD have lower SNR and CNR than the ones with low or intermediate SSDs. Registration SSD is structurally close enough to symmetry for reduced computation.Conclusion:Registration residual is shown to be a good predictor of post‐fusion image quality and can be used to identify good reference centers. Semi‐symmetry of the registration residual further reduces computation cost.Supported by in part by NIH R01 CA096679.