A Self-supervised Deep Framework for Reference Bony Shape Estimation in Orthognathic Surgical Planning

The purpose of this study was to reduce the experience dependence during the orthognathic surgical planning that involves virtually simulating the corrective procedure for jaw deformities. We introduce a geometric deep learning framework for generating reference facial bone shape models for objective guidance in surgical planning. First, we propose a surface deformation network to warp a patient's deformed bone to a set of normal bones for generating a dictionary of patient-specific normal bony shapes. Subsequently, sparse representation learning is employed to estimate a reference shape model based on the dictionary. We evaluated our method on a clinical dataset containing 24 patients, and compared it with a state-of-the-art method that relies on landmark-based sparse representation. Our method yields significantly higher accuracy than the competing method for estimating normal jaws and maintains the midfaces of patients' facial bones as well as the conventional way. Experimental results indicate that our method generates accurate shape models that meet clinical standards.

Deep learning is employed in many applications, such as computer vision, natural language processing, robotics, and recommender systems. Large and complex neural networks lead to high accuracy; however, they adversely affect many aspects of deep learning performance, such as training time, latency, throughput, energy consumption, and memory usage in the training and inference stages. To solve these challenges, various optimization techniques and frameworks have been developed for the efficient performance of deep learning models in the training and inference stages. Although optimization techniques such as quantization have been studied thoroughly in the past, less work has been done to study the performance of frameworks that provide quantization techniques. In this paper, we have used different performance metrics to study the performance of various quantization frameworks, including TensorFlow automatic mixed precision and TensorRT. These performance metrics include training time and memory utilization in the training stage along with latency and throughput for graphics processing units (GPUs) in the inference stage. We have applied the automatic mixed precision (AMP) technique during the training stage using the TensorFlow framework, while for inference we have utilized the TensorRT framework for the post-training quantization technique using the TensorFlow TensorRT (TF-TRT) application programming interface (API).We performed model profiling for different deep learning models, datasets, image sizes, and batch sizes for both the training and inference stages, the results of which can help developers and researchers to devise and deploy efficient deep learning models for GPUs.

This paper presents a novel method for face alignment under unknown head poses and nonrigid warp, within the framework of markov random field. The proposed method learns a 3D face shape model comprised of 31 facial features and a texture model for each facial feature from a 3D face database. The models are combined to serve as the unary, pairwise and high order constraints of the markov random field. The facial features are located by minimizing the potential function of markov random field, which is solved with dual decomposition. The main contribution of this paper is composed of three aspects. First, Random Project Tree is utilized to learn the manifold structure of the facial feature appearance under different view points. Second, a 3D face shape model is learned to capture the linear part of face shape distribution due to the change of identity and expression, which is served as the high order constraints. Third, Markov random field is introduced to model the nonlinear part of the face shape distribution, and also to deal with occlusion of facial features due to head pose variation or ornaments. Experiments was taken on the Texas 3D face database and face images downloaded from the Internet, results shows capability of adapting large head pose variations of the method.KeywordsFace ImageFacial FeatureMarkov Random FieldFace ShapeTexture ModelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We propose a method that can reconstruct both a 3D facial shape and camera poses from freehand multi-viewpoint snapshots. This method is based on Active Shape Model (ASM) using a facial shape database. Most ASM methods require an image in which the camera pose is known, but our method does not require this information. First, we choose an initial shape by selecting the model from the database which is most suitable to input images. Then, we improve the model by morphing it to fit the input images. Next, we estimate the camera poses using the morphed model. Finally we repeat the process, improving both the facial shape and the camera poses until the error between the input images and the computed result is minimized. Through experimentation, we show that our method reconstructs the facial shape within 3.5 mm of the ground truth.KeywordsFeature PointInput ImageFacial ShapeActive Shape ModelWorld Coordinate SystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We propose a method that can reconstruct both a 3D facial shape and camera poses from freehand multi-viewpoint snapshots. This method is based on Active Shape Model (ASM) using a facial shape database. Most ASM methods require an image in which the camera pose is known, but our method does not require this information. First, we choose an initial shape by selecting the model from the database which is most suitable to input images. Then, we improve the model by morphing it to fit the input images. Next, we estimate the camera poses using the morphed model. Finally we repeat the process, improving both the facial shape and the camera poses until the error between the input images and the computed result is minimized. Through experimentation, we show that our method reconstructs the facial shape within 3.5 mm of the ground truth.

Orthognathic surgical outcomes rely heavily on the quality of surgical planning. Automatic estimation of a reference facial bone shape significantly reduces experience-dependent variability and improves planning accuracy and efficiency. We propose an end-to-end deep learning framework to estimate patient-specific reference bony shape models for patients with orthognathic deformities. Specifically, we apply a point-cloud network to learn a vertex-wise deformation field from a patient's deformed bony shape, represented as a point cloud. The estimated deformation field is then used to correct the deformed bony shape to output a patient-specific reference bony surface model. To train our network effectively, we introduce a simulation strategy to synthesize deformed bones from any given normal bone, producing a relatively large and diverse dataset of shapes for training. Our method was evaluated using both synthetic and real patient data. Experimental results show that our framework estimates realistic reference bony shape models for patients with varying deformities. The performance of our method is consistently better than an existing method and several deep point-cloud networks. Our end-to-end estimation framework based on geometric deep learning shows great potential for improving clinical workflows.

Cascaded regression has recently become the method of choice for solving non-linear least squares problems such as deformable image alignment. Given a sizeable training set, cascaded regression method learns a set of generic rules that are sequentially applied to solve the least squares problem. Despite the success of cascaded regression for problems such as face alignment and head pose estimation, there are several shortcomings arising in the strategies proposed thus far. For example, the result is sensitive to the initial shape and the storage of the model is too large, and no robustness to self-occlusion. This paper proposes an approach to solve that problem. First, we reconstruct the 3D face from a 2D face image with arbitrary poses and expressions. Second, we project the 3D face shape model to the 2D plane, then we can get a 2D face shape. Third, we use the projected 2D face shape as the initial shape for the cascaded regression instead of the generic mean shape. Comparing to the existing methods the proposed method can increase the accuracy of the face alignment under confirming appropriate performance.

Three-dimensional (3D) shape models have recently been used in various fields including machine designing, internet mall, and medical application. Therefore, efficient management and reuse of 3D shape models is becoming important with research on object searching of 3D shape models. 3D shape model searching requires taking into consideration rotation, scaling, and translation of models. This paper proposes a method to search 3D objects to include and account for the above conditions. The proposed method acquires 2D range images by rendering a 3D object from many viewpoints, and discretely approximating two of the three degrees of rotational freedom by a finite number of viewpoints. Using the phase correlation method on the range image one can estimate the rotation around one axis, include scaling and translation, and compare the similarities between the input with models in a dictionary. Experimental results show that whether models match or not match can be distinguished even when the supposed viewpoints are limited to a finite number.

In this paper, we present face pose estimate and multi-pose synthesis technique. Through combining composite principal component analysis (CPCA) of the shape feature and context feature respectively in eigenspace, we can get new eigenvectors to represent the human face pose. Support vector machine (SVM) has the optimal hyperplane that the expected classification error for unseen test samples is minimized. We utilize CPCA-SVM technology to get face pose discrimination. As for pose synthesis, the face shape model and the texture model are established through statistical learning. Using these two models and Delaunay triangular, we can match a face image with a parameter vector, the shape model, and the texture model. The synthesized image contains much more personal details, which improve its reality. Accurate pose discrimination and multi-pose synthesis helps to get optimal face and improve recognition rate

In this paper, we present a novel strategy to design disentangled 3D face shape representation. Specifically, a given 3D face shape is decomposed into identity part and expression part, which are both encoded and decoded in a nonlinear way. To solve this problem, we propose an attribute decomposition framework for 3D face mesh. To better represent face shapes which are usually nonlinear deformed between each other, the face shapes are represented by a vertex based deformation representation rather than Euclidean coordinates. The experimental results demonstrate that our method has better performance than existing methods on decomposing the identity and expression parts. Moreover, more natural expression transfer results can be achieved with our method than existing methods.

Brain-inspired architectures are gaining increased attention, especially for edge devices to perform cognitive tasks utilizing its limited energy budget and computing resources. Hyperdimensional computing (HDC) paradigm is an emerging framework inspired by an abstract representation of neuronal circuits’ attributes in the human brain. That includes a fully holographic random representation, high-dimension vectors representing data, and robustness to uncertainty. The basic HDC pipeline consists of an encoding, training and comparison stages. The encoding algorithm maps different representations of inputs into a single class and stores them in the associative memory (AM) throughout the training stage. Later, during the inference stage, the similarity is computed between the query vector, which is encoded using the same encoding model, and the stored classes in the AM. HDC has shown promising results for 1D applications using less power, and lower latency than state-of-the-art digital neural networks (DNN). While in 2D applications, convolutional neural network (CNN) still achieves higher classification accuracy at the expense of more computations. In this paper, a comprehensive study on the HDC paradigm, main algorithms, and its implementation is presented. Moreover, the main state-of-the-art HDC architectures for 1D and 2D applications are highlighted. The article also analyzes two competing paradigms, namely, HDC and CNN, in terms of accuracy, complexity, and the number of operations. The paper concluded by highlighting challenges and recommendations for future directions on the HDC framework.

Text classification is a fundamental task in natural language processing (NLP). Deep-learning-based text classification methods usually have two stages: training and inference. However, the training dataset is only used in the training stage. To make full use of the training dataset in the inference stage in order to improve model performance, we propose a k-nearest neighbors retrieval augmented method (KRA) for deep-learning-based text classification models. KRA works by first constructing a storage system that stores the embeddings of the training samples during the training stage. During the inference stage, the model retrieves the top k-nearest neighbors of the testing text from the storage. Then, we use text augmentation methods to expand the retrieved neighbors, including traditional augmentation methods and a large language model (LLM)-based method. Next, the method weights the augmented neighbors based on their distances from the target text and incorporates their labels into the inference of the final results accordingly. We evaluate our KRA method on six benchmark datasets using four commonly used deep learning models: CNN, LSTM, BERT, and RoBERTa. The results demonstrate that KRA significantly improves the classification performance of these models, with an average accuracy improvement of 0.3% for BERT and up to 0.4% for RoBERTa. These improvements highlight the effectiveness and generalizability of KRA across different models and datasets, making it a valuable enhancement for a wide range of text classification tasks.

This paper addresses the problem of facial landmark localization and tracking from a single camera. We present a two-stage cascaded deformable shape model to effectively and efficiently localize facial landmarks with large head pose variations. For face detection, we propose a group sparse learning method to automatically select the most salient facial landmarks. By introducing 3D face shape model, we use procrustes analysis to achieve pose-free facial landmark initialization. For deformation, the first step uses mean-shift local search with constrained local model to rapidly approach the global optimum. The second step uses component-wise active contours to discriminatively refine the subtle shape variation. Our framework can simultaneously handle face detection, pose-free landmark localization and tracking in real time. Extensive experiments are conducted on both laboratory environmental face databases and face-in-the-wild databases. All results demonstrate that our approach has certain advantages over state-of-the-art methods in handling pose variations.

Orthognathic surgery corrects jaw deformities to improve aesthetics and functions. Due to the complexity of the craniomaxillofacial (CMF) anatomy, orthognathic surgery requires precise surgical planning, which involves predicting postoperative changes in facial appearance. To this end, most conventional methods involve simulation with biomechanical modeling methods, which are labor intensive and computationally expensive. Here we introduce a learning-based framework to speed up the simulation of postoperative facial appearances. Specifically, we introduce a facial shape change prediction network (FSC-Net) to learn the nonlinear mapping from bony shape changes to facial shape changes. FSC-Net is a point transform network weakly-supervised by paired preoperative and postoperative data without point-wise correspondence. In FSC-Net, a distance-guided shape loss places more emphasis on the jaw region. A local point constraint loss restricts point displacements to preserve the topology and smoothness of the surface mesh after point transformation. Evaluation results indicate that FSC-Net achieves 15× speedup with accuracy comparable to a state-of-the-art (SOTA) finite-element modeling (FEM) method.

It is not only interesting to predict how an individual of a relatively young age will look in the future but also to reconstruct the facial appearance in the past during childhood. It can be even more desirable when different circumstances, behavior and lifestyle and their impacts on the facial shape appearance as a consequence are taken into account. Such may be applicable for many practical reasons in healthcare, forensics psychology, missing people and children, etc. This paper presents the 3D Face Time Machine Matrix (FT2M), a 3D Dynamic Shape Model which is a fusion of two models of ageing and rejuvenation with facial shape variations due to lifestyle and behavioral factors. This dynamic model is learned from a database of three dimensional facial images which is built by ten individual age groups between 3 to 75 years old. 3D facial aging modeling is a complex process since it affects both the shape and texture of the face. We propose a Dynamic face model to transform the given input face to his youthful or adulthood appearance by taking into account his lifestyle and behavioral traits and the probable changes may occur in perceptible appearance by altering its shape and texture simultaneously.