MODIFIED LOCAL TERNARY PATTERN WITH CONVOLUTIONAL NEURAL NETWORK FOR FACE EXPRESSION RECOGNITION
Abstract
Facial expression recognition (FER) on images with illumination variation and noises is a challenging problem in the computer vision field. We solve this using deep learning approaches that have been successfully applied in various fields, especially in uncontrolled input conditions. We apply a sequence of processes including face detection, normalization, augmentation, and texture representation, to develop FER based on Convolutional Neural Network (CNN). The combination of TanTriggs normalization technique and Adaptive Gaussian Transformation Method is used to reduce light variation. The number of images is augmented using a geometric augmentation technique to prevent overfitting due to lack of training data. We propose a representation of Modified Local Ternary Pattern (Modified LTP) texture image that is more discriminating and less sensitive to noise by combining the upper and lower parts of the original LTP using the logical AND operation followed by average calculation. The Modified LTP texture images are then used to train a CNN-based classification model. Experiments on the KDEF dataset show that the proposed approach provides a promising result with an accuracy of 81.15%.
Downloads
References
Katsaggelos, Aggelos K, "IEEE Signal Processing Magazine: Farewell," IEEE Signal Processing Magazine, pp. 2-4, 2002.
Assari, Mohammad Amin, and M. Rahmati, "Driver drowsiness detection using face expression recognition," in 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pp. 337-341, 2011. https://doi.org/10.1109/ICSIPA.2011.6144162.
A. Kapoor, W. Burleson, R.W. Picard, "Automatic prediction of frustration," Int. J. Hum. Comput. Stud, pp. 724–736, 2007. https://doi.org/10.1016/j.ijhcs.2007.02.003.
Fei, Zixiang, Erfu Yang, David Day-Uei Li, Stephen Butler, Winifred Ijomah, Xia Li, and Huiyu Zhou, "Deep convolution network based emotion analysis towards mental health care," Neurocomputing 388, pp. 212-227, 2020. https://doi.org/10.1016/j.neucom.2020.01.034.
Levi, Gil, and Tal Hassner, "Emotion recognition in the wild via convolutional neural networks and mapped binary patterns," in Proceedings of the 2015 ACM on international conference on multimodal interaction, pp. 503-510. 2015. https://doi.org/10.1145/2818346.2830587.
Tan, Xiaoyang, and Bill Triggs, "Enhanced local texture feature sets for face recognition under difficult lighting conditions," IEEE Transactions on Image Processing 19, no. 6, pp. 1635-1650, 2010. https://doi.org/10.1109/TIP.2010.2042645.
Murala, Subrahmanyam, R. P. Maheshwari, and R. Balasubramanian, "Local tetra patterns: a new feature descriptor for content-based image retrieval," IEEE Transactions on Image Processing 21, no. 5, pp. 2874-2886, 2012. https://doi.org/10.1109/TIP.2012.2188809.
R.P. Holder, J.R. Tapamo, "Improved gradient local ternary patterns for facial expression recognition," Eurasip J. Image Video Process, 2017. https://doi.org/10.1186/s13640-017-0190-5.
Y. Huang, F. Chen, S. Lv, X. Wang, "Facial expression recognition: A survey," Symmetry (Basel), 11, 2019. https://doi.org/10.3390/sym11101189.
Y.-L. Tian, T. Kanade, J.F. Cohn, "Facial Expression Analysis," in Handbook of Face Recognition, 2005. https://doi.org/10.1007/0-387-27257-7_12.
C.P. Papageorgiou, M. Oren, T. Poggio, "General framework for object detection," in Proc. IEEE Int. Conf. Comput. Vis., pp. 555–562, 1998 https://doi.org/10.1109/iccv.1998.710772.
P. Viola, M. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, no. 1, 2001. https://doi.org/10.1109/cvpr.2001.990517.
S. Yan, S. Shan, X. Chen, W. Gao, "Locally Assembled Binary (LAB) feature with feature-centric cascade for fast and accurate face detection," in Proc. 26th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR, 2008. https://doi.org/10.1109/CVPR.2008.4587802.
S. Wu, M. Kan, Z. He, S. Shan, X. Chen, "Funnel-structured cascade for multi-view face detection with alignment-awareness," Neurocomputing no. 221, pp. 138–145, 2017. https://doi.org/10.1016/j.neucom.2016.09.072.
D.E. King, "Dlib-ml: A machine learning toolkit", J. Mach. Learn. Res. 10, pp. 1755-1758, 2009.
T. Ojala, M. Pietikäinen, D. Harwood, "A comparative study of texture measures with classification based on featured distributions", Pattern Recognit. 29, pp. 51–59, 1996. https://doi.org/10.1016/0031-3203(95)00067-4.
L. Liu, P. Fieguth, G. Zhao, M. Pietikäinen, D. Hu, "Extended local binary patterns for face recognition", Inf. Sci. (Ny). no. 358–359, pp. 56-72, 2016. https://doi.org/10.1016/j.ins.2016.04.021.
W. Huang, H. Yin, "Robust face recognition with structural binary gradient patterns", Pattern Recognit no. 68, pp. 126–140, 2017. https://doi.org/10.1016/j.patcog.2017.03.010.
Z. Li, N. Yang, B. Xie, J. Zhang, "A two-phase face recognition method in frequency domain", Optik (Stuttg). no. 124, pp. 6333–6337, 2013. https://doi.org/10.1016/j.ijleo.2013.05.057.
E. khadiri I, C. A, E. merabet Y, R. Y, T. R, "Local directional ternary pattern: A New texture descriptor for texture classification", Comput. Vis. Image Underst. no. 169, pp. 14–27, 2018. https://doi.org/https://dx.doi.org/10.1016/j.cviu.2018.01.004.
D. Huang, C. Shan, M. Ardabilian, Y. Wang, L. Chen, "Local binary patterns and its application to facial image analysis: A survey", IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. no. 41, pp. 765–781, 2011. https://doi.org/10.1109/TSMCC.2011.2118750.
A. Kurniawardhani, N. Suciati, I. Arieshanti, "Klasifikasi Citra Batik Menggunakan Metode Ekstraksi Ciri yang Invariant terhadap Rotasi", JUTI J. Ilm. Teknol. Inf. no. 12, pp. 48, 2014. https://doi.org/10.12962/j24068535.v12i2.a322.
D. E.Lundqvist, A. Flykt, A. Öhman, "The Karolinska Directed Emotional Faces - KDEF", CD ROM from Department of Clinical Neuroscience, in: Psychol. Sect., Psychology, Karolinska Institutet, 1998.
D.E. King, "Max-Margin Object Detection", Comput. Vis. Pattern Recognit. 2015. http://arxiv.org/abs/1502.00046.
Y.-H. Lee, S. Zhang, M. Li, X. He, "Blind Inverse Gamma Correction with Maximized Differential Entropy", ArXiv. abs/2007.0, pp. 1-12, 2020.
M.A. Farooque, J. S.Rohankar, "Survey on Various Noises and Techniques for Denoising the Color Image", Int. J. Appl. or Innov. Eng. Manag. no. 2, pp. 217-221, 2013.
W. Yang, X. Zhang, J. Li, "A local multiple patterns feature descriptor for face recognition", Neurocomputing. no. 373, pp. 109-122, 2020. https://doi.org/10.1016/j.neucom.2019.09.102.
J. C. Russ, "The Image Processing Handbook, Fifth Edition (Image Processing Handbook, Fifth Edit", CRC Press, Inc., United State, 2006.
Downloads
Published
Issue
Section
License
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in JUTI unless they receive approval for doing so from the Editor-in-Chief.
JUTI open access articles are distributed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
