Robust and Efficient Facial Landmark Localization

Comunicació presentada a la 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), celebrada els dies 30 de maig a 3 de juny de 2017 a Washington DC, EUA.

Head pose estimation methods can be broadly classified into learning-based methods and model-based methods. The learning based methods use machine learning techniques to directly predict the pose from image appearance, while the model-based methods link the 2D observation (e.g. facial landmarks) and 3D model through the projection model for pose estimation. However, both methods may have difficulty on images with very low quality (e.g. low resolution, occlusion, and noisy images). For example, there would be limited appearance information to generate accurate landmark detection on lowquality images for reliable face pose estimation. To tackle pose estimation on low-quality images, we propose to combine the learning and model based methods. Specifically, we first build the relationship between facial landmark locations and image appearance using the Restricted Boltzmann Machine (RBM) model. Then, we link the landmark locations and 3D model analytically using the projection model. By combining the RBM model with the projection model, without explicit landmark detection, we predict the head pose with a KL-divergence based method and a gradient-based method. Experimental results demonstrate the effectiveness of the proposed method.

In this paper we describe a multi-modal ear and face biometric system. The system is comprised of two components: a 3D ear recognition component and a 2D face recognition component. For the 3D ear recognition, a series of frames is extracted from a video clip and the region of interest (i.e., ear) in each frame is independently reconstructed in 3D using Shape From Shading. The resulting 3D models are then registered using the iterative closest point algorithm. We iteratively consider each model in the series as a reference model and calculate the similarity between the reference model and every model in the series using a similarity cost function. Cross validation is performed to assess the relative fidelity of each 3D model. The model that demonstrates the greatest overall similarity is determined to be the most stable 3D model and is subsequently enrolled in the database. For the 2D face recognition, a set of facial landmarks is extracted from frontal facial images using the Active Shape Model. Then, the response of facial images to a series of Gabor filters at the locations of facial landmarks are calculated. The Gabor features (attributes) are stored in the database as the face model for recognition. The similarity between the Gabor features of a probe facial image and the reference models are utilized to determine the best match. The match scores of the ear recognition and face recognition modalities are fused to boost the overall recognition rate of the system. Experiments are conducted using a gallery set of 402 video clips and a probe of 60 video clips (images). As a result, a rank-one identification rate of 100% was achieved using the weighted sum technique for fusion.

Facial analysis in videos, including head pose estimation and facial landmark localization, is key for many applications such as facial animation capture, human activity recognition, and human-computer interaction. In this paper, we propose to use a recurrent neural network (RNN) for joint estimation and tracking of facial features in videos. We are inspired by the fact that the computation performed in an RNN bears resemblance to Bayesian filters, which have been used for tracking in many previous methods for facial analysis from videos. Bayesian filters used in these methods, however, require complicated, problem-specific design and tuning. In contrast, our proposed RNN-based method avoids such tracker-engineering by learning from training data, similar to how a convolutional neural network (CNN) avoids feature-engineering for image classification. As an end-to-end network, the proposed RNN-based method provides a generic and holistic solution for joint estimation and tracking of various types of facial features from consecutive video frames. Extensive experimental results on head pose estimation and facial landmark localization from videos demonstrate that the proposed RNN-based method outperforms frame-wise models and Bayesian filtering. In addition, we create a large-scale synthetic dataset for head pose estimation, with which we achieve state-of-the-art performance on a benchmark dataset.

In recent years, pattern recognition has garnered widespread attention, especially in the domain of face recognition. Traditional face recognition methods have certain limitations in unconstrained environments due to factors such as lighting, facial expressions, and poses. Deep learning can be used to address these challenges. This paper proposes a comprehensive approach to face occlusion recognition based on a two-stage Convolutional Neural Network (CNN). Face verification aims at verifying whether two face images belong to the same individual, and it is a more fundamental task compared to face recognition. The process of face recognition essentially involves multiple instances of face verification, sequentially validating different individuals to ultimately determine the corresponding individual for each face. The primary steps in this research include facial detection, image preprocessing, facial landmark localization, facial landmark extraction, feature matching recognition, and 2D image-assisted 3D face reconstruction. A novel two-stage CNN was designed for facial detection and alignment. The first stage of the network is dedicated to the search for facial windows and regressing vector boundaries. The second stage utilizes 2D images to assist in 3D face reconstruction and perform secondary recognition for cases not identified in the first stage. This method demonstrated excellent performance in handling facial occlusions, achieving high accuracy on datasets such as AFW and FDDB. On the test dataset, face recognition accuracy reached 97.3%, surpassing the original network accuracy of 89.1%. This method outperforms traditional algorithms and general CNN approaches. This study achieved efficient face validation and further handling of unrecognized situations, contributing to the enhancement of face recognition system performance.

Heatmap regression has become the mainstream methodology for deep learning-based semantic landmark localization, including in facial landmark localization and human pose estimation. Though heatmap regression is robust to large variations in pose, illumination, and occlusion in unconstrained settings, it usually suffers from a sub-pixel localization problem. Specifically, considering that the activation point indices in heatmaps are always integers, quantization error thus appears when using heatmaps as the representation of numerical coordinates. Previous methods to overcome the sub-pixel localization problem usually rely on high-resolution heatmaps. As a result, there is always a trade-off between achieving localization accuracy and computational cost, where the computational complexity of heatmap regression depends on the heatmap resolution in a quadratic manner. In this paper, we formally analyze the quantization error of vanilla heatmap regression and propose a simple yet effective quantization system to address the sub-pixel localization problem. The proposed quantization system induced by the randomized rounding operation 1) encodes the fractional part of numerical coordinates into the ground truth heatmap using a probabilistic approach during training; and 2) decodes the predicted numerical coordinates from a set of activation points during testing. We prove that the proposed quantization system for heatmap regression is unbiased and lossless. Experimental results on popular facial landmark localization datasets (WFLW, 300W, COFW, and AFLW) and human pose estimation datasets (MPII and COCO) demonstrate the effectiveness of the proposed method for efficient and accurate semantic landmark localization. Code is available at http://github.com/baoshengyu/H3R.

Recent advances in deep learning have resulted in human-level performances on popular unconstrained face datasets including Labeled Faces in the Wild and YouTube Faces. To further advance research, IJB-A benchmark was recently introduced with more challenges especially in the form of extreme head poses. Registration of such faces is quite demanding and often requires laborious procedures like facial landmark localization. In this paper, we propose a Convolutional Neural Networks based data-driven approach which learns to simultaneously register and represent faces. We validate the proposed scheme on template based unconstrained face identification. Here, a template contains multiple media in the form of images and video frames. Unlike existing methods which synthesize all template media information at feature level, we propose to keep the template media intact. Instead, we represent gallery templates by their trained one-vs-rest discriminative models and then employ a Bayesian strategy which optimally fuses decisions of all medias in a query template. We demonstrate the efficacy of the proposed scheme on IJB-A, YouTube Celebrities and COX datasets where our approach achieves significant relative performance boosts of 3.6%, 21.6% and 12.8% respectively.

Deep-Orga: An improved deep learning-based lightweight model for intestinal organoid detection

The current robotic arm assembly mainly relies on vision to provide behavioral guidance. However, there are a large number of assembly tasks that cannot be completed by vision alone or have insufficient precision, such as the snap-fit of the Flexible Printed Cable and the insertion and removal of multi-pin plugs. These tasks require the guidance of force information, which we call haptics, generated during assembly. However, the current force simulation tools are difficult to achieve real-time accurate simulation, and usually occupy a lot of computational cost. In this paper, a method is proposed, which uses the collision system of Unity to model the local force on the contact surface, designs the collider array according to the contour structure of the surface. To verify the effectiveness of this representation, we train a simple network to estimate the attitude of irregular surface contact. we use the collected local force information as a feature. The position and pose of the head are used as the output, and a nonlinear neural network is trained, which can estimate the position and pose of the male head based on the collected features. It has been verified that the estimation accuracy of most of the outputs exceeds 95%.KeywordsRobot arm assemblyHaptic simulationPose estimationVirtual environment

BackgroundCerebral arteriovenous malformation (AVM) is a cerebrovascular disorder posing a risk for intracranial hemorrhage. However, there are few reliable quantitative indices to predict hemorrhage risk accurately. This study aimed to...

This study aimed to develop a multiregional radiomics-based model using multisequence MRI to predict microvascular invasion in hepatocellular carcinoma. We enrolled 141 patients with hepatocellular carcinoma, including 61 with microvascular invasion, who were diagnosed between March 2017 and July 2022. Clinical data were compared using the Wilcoxon rank-sum test or χ 2 test. Patients were randomly divided into training (n=112, 80%) and test (n=29, 20%) data sets. Four MRI sequences-including T2-weighted imaging, T2-weighted imaging with fat suppression, arterial phase-contrast enhancement, and portal venous phase contrast enhancement-were used to build the radiomics model. The tumor volumes of interest were manually delineated, and the expand-5mm and expand-10mm volumes of interest were automatically generated. A total of 1409 radiomic features were extracted from each volume of interest. Feature selection was performed using the least absolute shrinkage and selection operator and Spearman correlation analysis. Three logistic regression models (Tumor, Tumor-Expand5, and Tumor-Expand10) were established based on the radiomic features. Model performance was assessed using receiver operating characteristic analysis and Delong's test. Maximum tumor diameter, hepatitis B virus DNA, and aspartate aminotransferase levels were significantly different between the groups. The Tumor-Expand5mm model exhibited the best performance among the 3 models, with areas under the curve of 0.90 and 0.84 in the training and test data sets. The Tumor-Expand5 model based on multisequence MRI shows great potential for predicting microvascular invasion in patients with hepatocellular carcinoma, and may further contribute to personal clinical decision-making.

This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images, which can be trained without the gaze angle and the head pose annotations. Common gaze-correction methods usually require annotating training data with precise gaze, and head pose information. Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels. To address this issue, we first create two new portrait datasets: CelebGaze ( 256 ×256 ) and high-resolution CelebHQGaze ( 512 ×512 ). Second, we formulate the gaze correction task as an image inpainting problem, addressed using a Gaze Correction Module (GCM) and a Gaze Animation Module (GAM). Moreover, we propose an unsupervised training strategy, i.e., Synthesis-As-Training, to learn the correlation between the eye region features and the gaze angle. As a result, we can use the learned latent space for gaze animation with semantic interpolation in this space. Moreover, to alleviate both the memory and the computational costs in the training and the inference stage, we propose a Coarse-to-Fine Module (CFM) integrated with GCM and GAM. Extensive experiments validate the effectiveness of our method for both the gaze correction and the gaze animation tasks in both low and high-resolution face datasets in the wild and demonstrate the superiority of our method with respect to the state of the art.

ObjectiveRecent studies have suggested that deep-learning models can satisfactorily assist in fracture diagnosis. We aimed to evaluate the performance of two of such models in wrist fracture detection.MethodsWe collected image data of patients who visited with wrist trauma at the emergency department. A dataset extracted from January 2018 to May 2020 was split into training (90%) and test (10%) datasets, and two types of convolutional neural networks (i.e., DenseNet-161 and ResNet-152) were trained to detect wrist fractures. Gradient-weighted class activation mapping was used to highlight the regions of radiograph scans that contributed to the decision of the model. Performance of the convolutional neural network models was evaluated using the area under the receiver operating characteristic curve. ResultsFor model training, we used 4,551 radiographs from 798 patients and 4,443 radiographs from 1,481 patients with and without fractures, respectively. The remaining 10% (300 radiographs from 100 patients with fractures and 690 radiographs from 230 patients without fractures) was used as a test dataset. The sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of DenseNet-161 and ResNet-152 in the test dataset were 90.3%, 90.3%, 80.3%, 95.6%, and 90.3% and 88.6%, 88.4%, 76.9%, 94.7%, and 88.5%, respectively. The area under the receiver operating characteristic curves of DenseNet-161 and ResNet-152 for wrist fracture detection were 0.962 and 0.947, respectively.ConclusionWe demonstrated that DenseNet-161 and ResNet-152 models could help detect wrist fractures in the emergency room with satisfactory performance.

The US heart allocation system prioritizes medically urgent candidates with a high risk of dying without transplant. The current therapy-based 6-status system is susceptible to manipulation and has limited rank ordering ability. To develop and validate a candidate risk score that incorporates current clinical, laboratory, and hemodynamic data. A registry-based observational study of adult heart transplant candidates (aged ≥18 years) from the US heart allocation system listed between January 1, 2019, and December 31, 2022, split by center into training (70%) and test (30%) datasets. Adult candidates were listed between January 1, 2019, and December 31, 2022. A US candidate risk score (US-CRS) model was developed by adding a predefined set of predictors to the current French Candidate Risk Score (French-CRS) model. Sensitivity analyses were performed, which included intra-aortic balloon pumps (IABP) and percutaneous ventricular assist devices (VAD) in the definition of short-term mechanical circulatory support (MCS) for the US-CRS. Performance of the US-CRS model, French-CRS model, and 6-status model in the test dataset was evaluated by time-dependent area under the receiver operating characteristic curve (AUC) for death without transplant within 6 weeks and overall survival concordance (c-index) with integrated AUC. A total of 16 905 adult heart transplant candidates were listed (mean [SD] age, 53 [13] years; 73% male; 58% White); 796 patients (4.7%) died without a transplant. The final US-CRS contained time-varying short-term MCS (ventricular assist-extracorporeal membrane oxygenation or temporary surgical VAD), the log of bilirubin, estimated glomerular filtration rate, the log of B-type natriuretic peptide, albumin, sodium, and durable left ventricular assist device. In the test dataset, the AUC for death within 6 weeks of listing for the US-CRS model was 0.79 (95% CI, 0.75-0.83), for the French-CRS model was 0.72 (95% CI, 0.67-0.76), and 6-status model was 0.68 (95% CI, 0.62-0.73). Overall c-index for the US-CRS model was 0.76 (95% CI, 0.73-0.80), for the French-CRS model was 0.69 (95% CI, 0.65-0.73), and 6-status model was 0.67 (95% CI, 0.63-0.71). Classifying IABP and percutaneous VAD as short-term MCS reduced the effect size by 54%. In this registry-based study of US heart transplant candidates, a continuous multivariable allocation score outperformed the 6-status system in rank ordering heart transplant candidates by medical urgency and may be useful for the medical urgency component of heart allocation.

The process of extracting meaningful rules from big and complex data is called data mining. Data mining has an increasing popularity in every field today. Data units are established in customer-oriented industries such as marketing, finance and telecommunication to work on the customer churn and acquisition, in particular. Among the data mining methods, classification algorithms are used in studies conducted for customer acquisition to predict the potential customers of the company in question in the related industry. In this study, bank marketing data set in UCI Machine Learning Data Set was used by creating models with the same classification algorithms in different data mining programs. Accuracy, precision and f- measure criteria were used to test performances of the classification models. When creating the classification models, the test and training data sets were randomly divided by the holdout method to evaluate the performance of the data set. The data set was divided into training and test data sets with the 60-40%, 75­25% and 80-20% separation ratios. Data mining programs used for these processes are the R, Knime, RapidMiner and WEKA. And, classification algorithms commonly used in these platforms are the k-nearest neighbor (k-nn), Naive Bayes, and C4.5 decision tree.