Database Development and Recognition of Facial Expression using Deep Learning
DOI:
https://doi.org/10.47839/ijc.23.4.3760Keywords:
Image classification, Convolutional Neural Network, Facial expression recognition, E2E NetworkAbstract
Facial expressions serve as a means of conveying human emotions and individual intentions. The ability to perceive and interpret facial emotions is a relatively effortless job for humans, although it poses significant challenges when attempting to replicate this capability using a computer. Facial expressions can be detected from static photos, video, webcam data, or real-time photographs. The primary focus of this study is to create the SMM Facial Expression dataset and to develop a Convolutional Neural Network (CNN) model for accurately recognizing and classifying facial expressions. This model utilizes End-to-end (E2E) networks to analyze individual frames or clusters of frames received from the camera. The analysis is conducted through various layers of convolutional and pooling operations. The proposed model is evaluated on two benchmark datasets, add long form Cohn-Kanade (CK+) and Facial Expression Recognition 2013 Dataset (FER2013) for facial expression recognition (FER). The obtained accuracy rates are 85.27% and 82.18% for CK+ and FER2013, respectively. This study demonstrates that the SMM Facial dataset is comparable in quality to previously benchmarked datasets, and the proposed model holds potential for real-time facial expression recognition.
References
M. Albert, Nonverbal Communication, Routledge, 2017, https://doi.org/10.4324/9781351308724.
Y. L. Tian, T. Kanade, and J. F. Cohn, “Recognizing lower face action units for facial expression analysis,” Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), Grenoble, France, 2000, pp. 484-490, https://doi.org/10.1109/AFGR.2000.840678.
I. M. Revina and W. R. S. Emmanuel, “A survey on human face expression recognition techniques,” Journal of King Saud University – Computer and Information Sciences, vol. 33, issue 6, pp. 619–628, 2021, https://doi.org/10.1016/j.jksuci.2018.09.002.
V. Umarani and M. Srilakshmi, “Face emotion recognition: A brief review,” International Journal of Creative Research Thoughts, vol. 9, issue 11, pp. 766–771, 2021, [Online]. Availablenat: www.ijcrt.org.
P. Tarnowski, M. Kołodziej, A. Majkowski, and R. J. Rak, “Emotion recognition using facial expressions,” Procedia Computer Science, vol. 108, pp. 1175–1184, 2017, https://doi.org/10.1016/j.procs.2017.05.025.
M. H. Siddiqi et al., “A Brief Review of Facial Emotion Recognition Based on Visual Information,” Proceedings of the 2018 JEEMA CeTechNxT, vol. 5, issue 1, pp. 196–201, 2018.
Y. Xu, Q. Zhu, Z. Fan, D. Zhang, J. Mi, and Z. Lai, “Using the idea of the sparse representation to perform coarse-to-fine face recognition,” Information Sciences, vol. 238, pp. 138–148, 2013, https://doi.org/10.1016/j.ins.2013.02.051.
S. Nithya Roopa, “Emotion recognition from facial expression using deep learning,” International Journal of Engineering and Advanced Technology, vol. 8, special issue 6, pp. 91–95, 2019, https://doi.org/10.35940/ijeat.F1019.0886S19.
A. Hernandez-Matamoros, A. Bonarini, E. Escamilla-Hernandez, M. Nakano-Miyatake, and H. Perez-Meana, “Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach,” Knowledge-Based Systems, vol. 110, pp. 1–14, 2016, https://doi.org/10.1016/j.knosys.2016.07.011.
P. L. C. Courville, A. Goodfellow, I. J. M. Mirza, Y. Bengio, FER-2013 Face Database. Universite de Montreal: Montréal, QC, Canada, 2013.
Z. Zhang and M. Li, “Research on facial expression recognition based on neural network,” Proceedings of the 2020 International Conference on Computer Network, Electronic and Automation (ICCNEA), Xi'an, China, 2020, pp. 78-81, https://doi.org/10.1109/ICCNEA50255.2020.00025.
P. Lucey et al., “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression Patrick,” Proceedings of the 2010 IEEE Computer Society Conference Workshops on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 94-101, https://doi.org/10.1109/CVPRW.2010.5543262.
N. Mehendale, “Facial emotion recognition using convolutional neural networks (FERC),” SN Applied Sciences, vol. 2, issue 3, pp. 1–8, 2020, https://doi.org/10.1007/s42452-020-2234-1.
M. J. Cossetin, J. C. Nievola, and A. L. Koerich, “Facial expression recognition using a pairwise feature selection and classification approach,” Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada, 2016, pp. 5149-5155, https://doi.org/10.1109/IJCNN.2016.7727879.
M. Shao, S. Xia, and Y. Fu, “Genealogical face recognition based on UB KinFace database,” Proceedings of the Workshops on Computer Vision and Pattern Recognition CVPR 2011, Colorado Springs, CO, USA, 2011, pp. 60-65, https://doi.org/10.1109/CVPRW.2011.5981801.
Z. Yue, F. Yanyan, Z. Shangyou, P, Bing, “Facial expression recognition based on convolutional neural network,” Proceedings of the 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), 2019, pp. 410-413, https://doi.org/10.1109/ICSESS47205.2019.9040730.
N. Christou and N. Kanojiya, “Human facial expression recognition with convolution neural networks,” In: X.S. Yang, S. Sherratt, N. Dey, A. Joshi (Eds.), Proceedings of the Third International Congress on Information and Communication Technology, Advances in Intelligent Systems and Computing, Springer, Singapore, 2019, vol. 797. https://doi.org/10.1007/978-981-13-1165-9_49.
H. Zhang, A. Jolfaei, and M. Alazab, “A face emotion recognition method using convolutional neural network and image edge computing,” IEEE Access, vol. 7, pp. 159081–159089, 2019, https://doi.org/10.1109/ACCESS.2019.2949741.
N. Aifanti, C. Papachristou, and A. Delopoulos, “The MUG facial expression database,” Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, Desenzano del Garda, Italy, 2010, pp. 1-4.
R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-PIE,” Image and Vision Computing, vol. 28, issue 5, pp. 807–813, 2010, https://doi.org/10.1016/j.imavis.2009.08.002.
M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-based database for facial expression analysis,” Proceedings of the IEEE International Conference on Multimedia and Expo, ICME 2005, vol. 2005, issue July, pp. 317–321, 2005, https://doi.org/10.1109/ICME.2005.1521424.
M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, “The first facial expression recognition and analysis challenge,” Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA, 2011, pp. 921-926, https://doi.org/10.1109/FG.2011.5771374.
A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 2011, pp. 2106-2112, https://doi.org/10.1109/ICCVW.2011.6130508.
A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A database for facial expression, Valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, issue 1, pp. 18–31, 2019, https://doi.org/10.1109/TAFFC.2017.2740923.
S. Li, W. Deng, and J. P. Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 2584-2593, https://doi.org/10.1109/CVPR.2017.277.
C. Shan, S. Gong, P.W. McOwan, “Facial expression recognition based on local binary patterns: A comprehensive study,” Image and Vision Computing, vol. 27, issue 6, pp. 803–816, 2009, https://doi.org/10.1016/j.imavis.2008.08.005.
T. Jabid, M. H. Kabir, and O. Chae, “Robust facial expression recognition based on local directional pattern,” ETRI Journal, vol. 32, issue 5, pp. 784–794, 2010, https://doi.org/10.4218/etrij.10.1510.0132.
H. Wang, J. Gao, L. Tong, and L. Yu, “Facial expression recognition based on PHOG feature and sparse representation,” Proceedings of the 2016 35th Chinese Control Conference (CCC), Chengdu, China, 2016, pp. 3869-3874, https://doi.org/10.1109/ChiCC.2016.7553957.
Y. Gao and M. K. H. Leung, “Face recognition using line edge map,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 764-779, 2002, https://doi.org/10.1109/TPAMI.2002.1008383.
M. H. Alkawaz, D. Mohamad, A. H. Basori, and T. Saba, “Blend shape interpolation and FACS for realistic avatar,” 3D Research, vol. 6, issue 1, 2015, https://doi.org/10.1007/s13319-015-0038-7.
M. H. Siddiqi, M. Alruwaili, J. H. Bang, and S. Y. Lee, “Real time human facial expression recognition system using smartphone,” IJCSNS International Journal of Computer Science and Network Security, vol. 17, issue 10, pp. 223–230, 2017.
N. Zeng, H. Zhang, B. Song, W. Liu, Y. Li, and A. M. Dobaie, “Facial expression recognition via learning deep sparse autoencoders,” Neurocomputing, vol. 273, pp. 643–649, 2018, https://doi.org/10.1016/j.neucom.2017.08.043.
S. H. Wang, P. Phillips, Z. C. Dong, and Y. D. Zhang, “Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm,” Neurocomputing, vol. 272, pp. 668–676, 2018, https://doi.org/10.1016/j.neucom.2017.08.015.
M. Asad, S. O. Gilani, and M. Jamil, “Emotion detection through facial feature recognition,” International Journal of Multimedia and Ubiquitous Engineering, vol. 12, issue 11, pp. 21–30, 2017, https://doi.org/10.14257/ijmue.2017.12.11.03.
M. Merlin Steffi and J. John Raybin Jose, “Comparative analysis of facial recognition involving feature extraction techniques,” International Journal of Computer Sciences and Engineering, vol. 6, issue 2, pp. 81–86, 2018, [Online]. Available at: http://www.ijcseonline.org/spl_pub_paper/NCTT-2018-18.pdf.
V. M. Alvarez, R. Velazquez, S. Gutierrez, and J. Enriquez-Zarate, “A method for facial emotion recognition based on interest points,” Proceedings of the 2018 International Conference on Research in Intelligent and Computing in Engineering (RICE), San Salvador, El Salvador, 2018, pp. 1-4, https://doi.org/10.1109/RICE.2018.8509055.
S. Minaee, M. Minaei, and A. Abdolrashidi, “Deep-emotion: Facial expression recognition using attentional convolutional network,” Sensors, vol. 21, issue 9, 3046, 2021, https://doi.org/10.3390/s21093046.
D. Goren and H. R. Wilson, “Quantifying facial expression recognition across viewing conditions,” Vision Research, vol. 46, issue 8–9, pp. 1253–1262, 2006, https://doi.org/10.1016/j.visres.2005.10.028.
N. Samadiani, G. Huang, B. Cai, W. Luo, C. Chi, Y. Xiang, and J. He, “A review on automatic facial expression recognition systems assisted by multimodal sensor data,” Sensors, vol. 19, issue 8, 1863, 2019, https://doi.org/10.3390/s19081863.
Z. Song, “Facial expression emotion recognition model integrating philosophy and machine learning theory,” Frontiers in Psychology, vol. 12, 2021, https://doi.org/10.3389/fpsyg.2021.759485.
K. Lekdioui, Y. Ruichek, R. Messoussi, Y. Chaabi, and R. Touahni, “Facial expression recognition using face-regions,” Proceedings of the 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Fez, Morocco, 2017, pp. 1-6, https://doi.org/10.1109/ATSIP.2017.8075517.
K. R. Kulkarni and S. B. Bagal, “Facial expression recognition,” Proceedings of the 12th 2015 International Conference on Information Processing (ICIP), 2015, pp. 535-539. https://doi.org/10.1109/INFOP.2015.7489442.
G. N. Matre and S. K. Shah, “Facial expression detection,” Proceedings of the 2013 IEEE International Conference on Computational Intelligence and Computing Research, Enathi, India, 2013, pp. 1-3, https://doi.org/10.1109/ICCIC.2013.6724242.
J. Avanija, K. R. Madhavi, G. Sunitha, S. C. Sangapu, and S. Raju, “Facial expression recognition using convolutional neural network,” Proceedings of the 2022 First International Conference on Artificial Intelligence Trends and Pattern Recognition (ICAITPR), 2022, pp. 336–341, https://doi.org/10.1109/ICAITPR51569.2022.9844221.
T. Debnath, M. M. Reza, A. Rahman, A. Beheshti, S. S. Band, and H. Alinejad-Rokny, “Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity,” Scientific Reports, vol. 12, issue 1, pp. 1–18, 2022, https://doi.org/10.1038/s41598-022-11173-0.
A. Khopkar and A. A. Saxena, “Facial expression recognition using CNN with Keras,” Bioscience Biotechnology Research Communications, vol. 14, issue 5, pp. 47–50, 2021, https://doi.org/10.21786/bbrc/14.5/10.
F. Yao, and L. Qiu, “Facial expression recognition based on convolutional neural network fusion SIFT features of mobile virtual reality,” Wireless Communications and Mobile Computing, vol. 14, issue 1–6, pp. 253–267, 2021, https://doi.org/10.1155/2021/5763626.
V. Tümen, Ö. F. Söylemez, and B. Ergen, “Facial emotion recognition on a dataset using convolutional neural network,” Proceedings of the 2017 International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Turkey, 2017, pp. 1-5, https://doi.org/10.1109/IDAP.2017.8090281.
Y.-K. Zhai and J. Liu, “Facial Expression Recognition based on Transfer ring Convolutional Neural Network,” Journal of Signal Processing, pp. 729–738, 2018.
Y. Fang, Research of Facial Expression Recognition Based on Convolutional Neural Networks, Dissertation, Xidian University, 2017.
A. Mollahosseini, D. Chan, and M. H. Mahoor, “Going deeper in facial expression recognition using deep neural networks,” Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, 2016, pp. 1-10, https://doi.org/10.1109/WACV.2016.7477450.
P. D. M. Fernandez, F. A. G. Pena, T. I. Ren, and A. Cunha, “FERAtt: Facial expression recognition with attention net,” Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019, pp. 837-846, https://doi.org/10.1109/CVPRW.2019.00112.
K. Shan, J. Guo, W. You, D. Lu, and R. Bie, “Automatic facial expression recognition based on a deep convolutional-neural-network structure,” Proceedings of the 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), London, UK, 2017, pp. 123-128, https://doi.org/10.1109/SERA.2017.7965717.
S. Sawardekar and S. R. Naik, “Facial expression recognition using efficient LBP and CNN,” International Research Journal of Engineering and Technology (IRJET), issue June, pp. 2273–2277, 2018.
S. Rifai, Y. Bengio, A. Courville, P. Vincent, M. Mirza, “Disentangling factors of variation for facial expression recognition,” In: A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, C. Schmid, (Eds.), Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol. 7577, 2012, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33783-3_58.
M. Liu, S. Li, S. Shan, R. Wang, X. Chen, “Deeply learning deformable facial action parts model for dynamic expression analysis,” Proceedings of the Asian Conference on Computer Vision, 2014, 143-157. https://doi.org/10.1007/978-3-642-33783-3_58.
F. Adjailia, et al., “Integration of 2D textural and 3D geometric features for robust facial expression recognition,” Computing & Informatics, vol. 40, issue 5, pp. 988–1007, 2021. https://doi.org/10.31577/cai_2021_5_988.
J. Haddad, O. Lezoray, P. Hamel, “3D-CNN for facial emotion recognition in videos,” In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science, vol. 12510, 2020. Springer, Cham. https://doi.org/10.1007/978-3-030-64559-5_23.
M. M. Oliver, and E. A. Alcover, “UIBVFED: Virtual facial expression dataset,” Plos one, vol. 15, issue 4, e0231266, 2020. https://doi.org/10.1371/journal.pone.0231266.
Y. Lin, and H. Xie, “Face gender recognition based on face recognition feature vectors,” Proceedings of the 2020 IEEE 3rd International Conference on Information Systems and Computer Aided Education (ICISCAE), 2020. https://doi.org/10.1109/ICISCAE51034.2020.9236905.
M. K. Scheuerman, K. Wade, C. Lustig, and J. R. Brubaker, “How we've taught algorithms to see identity: Constructing race and gender in image databases for facial analysis,” Proceedings of the ACM Hum.-Comput. Interact, vol. 4, issue CSCW1, Article 58, pp. 1-35, 2020. https://doi.org/10.1145/3392866.
I. O. Ertugrul, J. F. Cohn, L. A. Jeni, Z. Zhang, L. Yin, Q. Ji, “Cross-domain AU detection: Domains, learning approaches, and measures,” Proceedings of the Int Conf Autom Face Gesture Recognit, May 2019. https://doi.org/10.1109/FG.2019.8756543. Epub 2019 Jul 11. PMID: 31749665; PMCID: PMC6867108.
M. Olszanowski, et al., “Warsaw set of emotional facial expression pictures: A validation study of facial display photographs,” Frontiers in Psychology, vol. 5, Frontiers Media SA, 2015. https://doi.org/10.3389/fpsyg.2014.01516.
P. Mehta, A. Kumar and S. Bhattacharjee, “Fire and gun violence based anomaly detection system using deep neural networks,” Proceedings of the2020 International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India, 2020, pp. 199-204, https://doi.org/10.1109/ICESC48915.2020.9155625.
S. Tripathi, S. Tripathi, and H. Beigi, “Multi-modal emotion recognition on IEMOCAP dataset using deep learning,” arXiv preprint arXiv:1804.05788, 2018.
Downloads
Published
How to Cite
Issue
Section
License
International Journal of Computing is an open access journal. Authors who publish with this journal agree to the following terms:• Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
• Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
• Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
