Combining Stochastic and Deterministic Search for Pose-Invariant Facial Expression Recognition

최근 얼굴 표정 인식에 있어 영상 기반의 방법의 하나로서 ULBP 블록 히스토그램 피쳐와 SVM을 분류기로 사용한 연구가 수행되었다. Ojala 등에 의해 소개된 LBP는 높은 식별력과 조명의 변화에 대한 내구성과 간단한 연산 때문에 영상 인식 분야에 많이 사용되고 있다. 본 논문에서는 ULBP 블록 히스토그램을 계산함에 있어 분할 영역의 이동, 크기 변화에 더하여 미세한 특징 요소를 표현할 수 있도록 <TEX>$LBP_{8,2}$</TEX>과 <TEX>$LBP_{8,1}$</TEX>를 결합하였다. <TEX>$LBP_{8,1}$</TEX> 660개, <TEX>$LBP_{8,2}$</TEX> 550개의 분할 창으로부터 1210개의 ULBP 히스토그램 피쳐를 추출하고 이로부터 AdaBoost를 이용하여 50개의 약 분류기를 생성하였다. <TEX>$LBP_{8,1}$</TEX>와 <TEX>$LBP_{8,2}$</TEX>가 결합된 하이브리드 형태의 ULBP 블록 히스토그램 피쳐와 SVM 분류기를 이용함으로써 표정 인식률을 향상시킬 수 있었으며 다양한 실험을 통하여 이를 확인하였다. 본 논문에서 제안한 하이브리드 Boosted ULBP 히스토그램의 경우에 표정의 인식률이 96.3%로 가장 높은 결과를 보였으며 제안한 방법의 우수성을 확인하였다. Recently, as one of images based methods in facial expression recognition, the research which used ULBP block histogram feature and SVM classifier was performed. Due to the properties of LBP introduced by Ojala, such as highly distinction capability, durability to the illumination changes and simple operation, LBP is widely used in the field of image recognition. In this paper, we combined <TEX>$LBP_{8,2}$</TEX> and <TEX>$LBP_{8,1}$</TEX> to describe micro features in addition to shift, size change in calculating ULBP block histogram. From sub-windows of 660 of <TEX>$LBP_{8,1}$</TEX> and 550 of <TEX>$LBP_{8,2}$</TEX>, ULBP histogram feature of 1210 were extracted and weak classifiers of 50 were generated using AdaBoost. By using the combined <TEX>$LBP_{8,1}$</TEX> and <TEX>$LBP_{8,2}$</TEX> hybrid type of ULBP histogram feature and SVM classifier, facial expression recognition rate could be improved and it was confirmed through various experiments. Facial expression recognition rate of 96.3% by hybrid boosted ULBP block histogram showed the superiority of the proposed method.

This paper presents a new method for facial expression recognition using divided local binary pattern features. Motivated by Fourier’s parity decomposition, we decompose a LBP descriptor into two subdescriptors. This method reduces the amount of data of the feature extraction, reducing the data dimension effectively. Experimental results on JAFFE facial expression database show that the proposed method can get significant performance improvement in recognition rate, reduce the recognition time greatly.

We propose a novel approach using Complete Local Binary Pattern feature generation method for facial expression recognition with the help of Multi-Class Support Vector Machine. Complete Local Binary Pattern method is an extended version of Local Binary Pattern method with a little difference. LBP feature considers only signs of local differences, whereas CLBP feature considers both signs and magnitude of local differences as well as original center gray level value. CLBP and LBP have same computational complexity while CLBP performs better facial expression recognition over LBP using SVM training and multiclass classification with binary SVM classifiers. The experimental result demonstrate the average efficiency of recognition of propose method (35 images) with CLBP is 86.4%, while with LBP and CCV is 84.1255% and 75.83% in the JAFFE database.

This paper presents a novel approach for estimating 3D head pose in single-view video sequences acquired by an uncalibrated camera. Following the initialization by a face detector, a tracking technique localizes the faces in each frame in the video sequence. Head pose estimation is performed by using a structure from motion and self-calibration technique in a sequential way. The proposed method was applied to the IDIAP database that contains head pose ground truth data. The obtained results demonstrate that the method can estimate the head pose with satisfying accuracy.

This paper presents an analysis of hand-crafted and learned feature extraction methods for real-time facial expression recognition (FER). Our analysis focuses on methods capable of running on mobile devices, including traditional algorithms such as Gabor transform, HOG, LBP, as well as two compact CNN models, named Mobilenet V1 and V2. Additionally, we test the performance of MOTIF, a highly efficient texture feature extractor algorithm. Furthermore, we analyze the contribution of the mouth and front-eyes regions for recognizing the seven basic facial expressions. Experimental results are evaluated on two publicly available datasets. KDEF database which was captured under controlled conditions and RAF database which represents more naturalistic expressions captured in-the-wild. Under the same experimental conditions, MOTIF presents the fastest performance by sacrificing accuracy, while Mobilenet V2 presents the highest results with considerable speed and model size.

This article presents a multimodal approach to head pose estimation of individuals in environments equipped with multiple cameras and microphones, such as SmartRooms or automatic video conferencing. Determining the individuals head orientation is the basis for many forms of more sophisticated interactions between humans and technical devices and can also be used for automatic sensor selection (camera, microphone) in communications or video surveillance systems. The use of particle filters as a unified framework for the estimation of the head orientation for both monomodal and multimodal cases is proposed. In video, we estimate head orientation from color information by exploiting spatial redundancy among cameras. Audio information is processed to estimate the direction of the voice produced by a speaker making use of the directivity characteristics of the head radiation pattern. Furthermore, two different particle filter multimodal information fusion schemes for combining the audio and video streams are analyzed in terms of accuracy and robustness. In the first one, fusion is performed at a decision level by combining each monomodal head pose estimation, while the second one uses a joint estimation system combining information at data level. Experimental results conducted over the CLEAR 2006 evaluation database are reported and the comparison of the proposed multimodal head pose estimation algorithms with the reference monomodal approaches proves the effectiveness of the proposed approach.

A Facial Expression Recognition (FER) method based on Conv3D-ConvLSTM-SEnet in an online education environment is proposed to address the issue of low accuracy in current classroom expression recognition (FER) methods. Firstly, ConvLSTM is used to integrate the local feature extraction ability of CNN and the temporal modeling ability of LSTM, and based on this, the rich spatial features of the image are characterized. Then, by introducing Depth Separated Convolution (DSC) to change the number of output channels and separate each channel, the Feature Maps (F-M) are sequentially concatenated to obtain a multi-channel output F-M. Finally, based on ConvLSTM and SEnet modules, a Conv3D ConvLSTM-based FER model was proposed by redistributing the extracted abstract features and basic texture features, achieving high accuracy classroom FER. The proposed classroom FER method and the other five methods were compared and analyzed through simulation experiments using the CK[Formula: see text], FER2013 and JAFFE datasets, respectively. The results indicate that the proposed method achieves the highest accuracy, precision, recall, and F1 score, with improvements of at least 1.85%, 2.41%, 1.18% and 2.05%, respectively, compared to the other five methods on the FER2013 dataset. The proposed method can perform FER on students in online classroom teaching, detect their emotions in real-time, and help teachers make better course teaching adjustments based on this.

Head pose estimation is an important problem in the field of computer vision and facial analysis. We model the problem of head pose estimation as a regression problem, where the three rotation angles (yaw, pitch, roll) are functions of the face appearance. We make use of that fact and learn the appearance of the face using a tree cascade of sparse Multi-Variate Relevance Vector Machines (MVRVM). Our method is fast and suitable for real-time applications as it is not computationally expensive. Our method learns the face appearance to estimate the head rotation angles. We evaluated our approach on two challenging datasets, the YouTube Faces and the Point and Shoot Challenging (PaSC) dataset. We achieved results of head pose estimation (yaw, pitch, roll) with mean error less than 5\(\circ \) and with error tolerance less than ±4 on the PaSC dataset. In terms of speed, one prediction takes around 6 milliseconds, which is suitable for real-time applications and also with high frame rate.

Although the ability to estimate the face pose and recognise its identity are common human abilities, they are still a challenge in computer vision context. In this study, the authors aim to overcome these difficulties by learning a non‐linear tensor‐based model based on multi‐linear decomposition. Proposed model maps the high‐dimensional image space into low‐dimensional pose manifold. For preserving the actual distance along the manifold shape, a graph‐based distance measure is proposed. Also, to compensate for the limited number of training poses, mirrored images are added to training ones to improve the recognition accuracy. For performance evaluation of the proposed method, experiments are run on three famous face databases using three different manifold shapes and two different distance measures. Eight training data modes are chosen such that the influential parameters are studied comprehensively. The obtained results confirm the effectiveness of proposed model in achieving high accuracy in pose estimation and multi‐view face recognition, even with different training poses for different identities.

A robust and efficient visual target tracking algorithm using particle filtering is proposed. Particle filtering has been proven very successful in estimating non-Gaussian and non-linear problems. In this paper, particle filter with color feature estimated the target state with time. Color feature being scale and rotational invariant, have showed robustness to partial occlusion and computationally efficient. The performance is made more robust by choosing the different (YIQ) color scheme. Tracking has been performed by comparison of chrominance histograms of target and candidate positions (particles). The Color based particle filter tracking often leads to inaccurate results when light intensity changes during a video stream. Furthermore, background subtraction has been used for size estimation of target. The qualitative evaluation of proposed algorithm is performed on several real world videos. The experimental results demonstrated that the proposed algorithm can track the moving objects well under illumination changes, occlusion and moving background.

Facial expression recognition has been applied in many fields such as human computer interaction, patient monitoring, neurology, social robot… Although facial expression recognition has gained some encourage results, there are still many challenges such as the change of illumination, blur, etc. Especially, recognizing between sadness and anger expression raises more barriers. In this paper, we proposed a framework using only salient facial regions, but it would be able to improve the accuracy of facial expression recognition. In this framework, one of the most state-of-the-art descriptor, called Pyramid of Local Phase Quantization descriptor (PLPQ) was used to robust with respect to image blur. The experiment achieved 97.7% accuracy recognition rate on the extend Cohn-Canade (CK+) database, and outperformed than other state-of-the-art facial expression recognition methods.

A support vector machine-based multi-view face detection and recognition framework is described. Face detection is carried out by constructing several detectors, each of them in charge of one specific view. The symmetrical property of face images is employed to simplify the complexity of the modelling. The estimation of head pose, which is achieved by using the support vector regression technique, provides crucial information for choosing the appropriate face detector. This helps to improve the accuracy and reduce the computation in multi-view face detection compared to other methods. For video sequences, further computational reduction can be achieved by using a pose change smoothing strategy. When face detectors find a face in frontal view, a support vector machine-based multi-class classifier is activated for face recognition. All the above issues are integrated under a support vector machine framework. Test results on four video sequences are presented, among them the detection rate is above 95%, recognition accuracy is above 90%, average pose estimation error is around 10/spl deg/, and the full detection and recognition speed is up to 4 frames/second on a Pentium II 300 PC.

Perception of head pose is useful for many face-related tasks such as face recognition, gaze estimation, and emotion analysis. In this paper, we propose a novel random forest based method for estimating head pose angles from single face images. In order to improve the effectiveness of the constructed head pose predictor, we introduce feature weighting and tree screening into the random forest training process. In this way, the features with more discriminative power are more likely to be chosen for constructing trees, and each of the trees in the obtained random forest usually has high pose estimation accuracy, while the diversity or generalization ability of the forest is not deteriorated. The proposed method has been evaluated on four public databases as well as a surveillance dataset collected by ourselves. The results show that the proposed method can achieve state-of-the-art pose estimation accuracy. Moreover, we investigate the impact of pose angle sampling intervals and heterogeneous face images on the effectiveness of trained head pose predictors.

Facial expression methods play a vital role in human-computer interaction and other fields, but there are factors such as occlusion, illumination, and pose changes in wild facial recognition, as well as category imbalances between different datasets, that result in large variations in recognition rates and low accuracy rates for different categories of facial expression datasets. This study introduces RCL-Net, a method of recognizing wild facial expressions that is based on an attention mechanism and LBP feature fusion. The structure consists of two main branches, namely the ResNet-CBAM residual attention branch and the local binary feature (LBP) extraction branch (RCL-Net). First, by merging the residual network and hybrid attention mechanism, the residual attention network is presented to emphasize the local detail feature information of facial expressions; the significant characteristics of facial expressions are retrieved from both channel and spatial dimensions to build the residual attention classification model. Second, we present a locally improved residual network attention model. LBP features are introduced into the facial expression feature extraction stage in order to extract texture information on expression photographs in order to emphasize facial feature information and enhance the recognition accuracy of the model. Lastly, experimental validation is performed using the FER2013, FERPLUS, CK+, and RAF-DB datasets, and the experimental results demonstrate that the proposed method has superior generalization capability and robustness in the laboratory-controlled environment and field environment compared to the most recent experimental methods.

Head pose estimation is important for driver attention monitoring as well as for various human computer interaction tasks. In this paper, an adaptive head pose estimation method is proposed to overcome difficulties of existing approaches. The proposed method is based on the analysis of two approaches for head pose estimation from an image sequence, that is, principal component analysis (PCA) and 3D motion estimation. The algorithm performs accurate pose estimation by learning the subject appearance on-line. Depth information is used effectively in the algorithm to segment the head region even in a cluttered scene and to perform 3D head motion estimation based on optical flow constraints.